CN116631619A - Postoperative leg bending training monitoring system and method thereof - Google Patents

Postoperative leg bending training monitoring system and method thereof Download PDF

Info

Publication number
CN116631619A
CN116631619A CN202310530445.XA CN202310530445A CN116631619A CN 116631619 A CN116631619 A CN 116631619A CN 202310530445 A CN202310530445 A CN 202310530445A CN 116631619 A CN116631619 A CN 116631619A
Authority
CN
China
Prior art keywords
scale
gait
gait energy
feature
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310530445.XA
Other languages
Chinese (zh)
Inventor
彭亚文
柴伟
白杨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
First Medical Center of PLA General Hospital
Original Assignee
First Medical Center of PLA General Hospital
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by First Medical Center of PLA General Hospital filed Critical First Medical Center of PLA General Hospital
Priority to CN202310530445.XA priority Critical patent/CN116631619A/en
Publication of CN116631619A publication Critical patent/CN116631619A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
    • G16H10/60—ICT specially adapted for the handling or processing of patient-related medical or healthcare data for patient-specific data, e.g. for electronic patient records

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Public Health (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Primary Health Care (AREA)
  • Epidemiology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Human Computer Interaction (AREA)
  • Pathology (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

A post-operative leg bending training monitoring system and method thereof are disclosed. Firstly, extracting a plurality of bending training monitoring key frames in a gait cycle from a bending training monitoring video of an object to be monitored, then, processing the bending training monitoring key frames to obtain a gait energy image, then, respectively passing the gait energy image through a convolutional neural network model and a ViT model to obtain a first-scale gait energy characteristic image and a second-scale gait energy characteristic image, then, fusing the first-scale gait energy characteristic image and the second-scale gait energy characteristic image to obtain a classification characteristic image, and finally, passing the classification characteristic image through a classifier to obtain a classification result for indicating whether the gait of the object to be monitored is normal. In this way, the gait evaluation of the patient can be accurately performed, and abnormalities can be found in time to facilitate corresponding treatments.

Description

Postoperative leg bending training monitoring system and method thereof
Technical Field
The application relates to the field of intelligent monitoring, in particular to a postoperative leg bending training monitoring system and a postoperative leg bending training monitoring method.
Background
Functional exercise is a necessary means for ensuring joint function, recovering muscle strength and accelerating rehabilitation process. For patients who do post-operative leg bending exercises, appropriate activities can promote blood circulation, alleviating the risk of thrombosis. However, postoperative leg bending patients may experience knee flexion while walking, resulting in gait abnormalities that affect the postoperative rehabilitation of the patient.
Accordingly, an optimized post-operative leg bending training monitoring system is desired that can evaluate the gait of the patient, and discover anomalies in time for corresponding treatment.
Disclosure of Invention
The present application has been made to solve the above-mentioned technical problems. The embodiment of the application provides a postoperative leg bending training monitoring system and a postoperative leg bending training monitoring method. Firstly, extracting a plurality of bending training monitoring key frames in a gait cycle from a bending training monitoring video of an object to be monitored, then, processing the bending training monitoring key frames to obtain a gait energy image, then, respectively passing the gait energy image through a convolutional neural network model and a ViT model to obtain a first-scale gait energy characteristic image and a second-scale gait energy characteristic image, then, fusing the first-scale gait energy characteristic image and the second-scale gait energy characteristic image to obtain a classification characteristic image, and finally, passing the classification characteristic image through a classifier to obtain a classification result for indicating whether the gait of the object to be monitored is normal. In this way, the gait evaluation of the patient can be accurately performed, and abnormalities can be found in time to facilitate corresponding treatments.
According to one aspect of the present application, there is provided a post-operative leg bending training monitoring system comprising:
the video data acquisition module is used for acquiring a bending training monitoring video of an object to be monitored;
the key frame extraction module is used for extracting a plurality of key frames for the bending training monitoring in a gait cycle from the bending training monitoring video of the object to be monitored;
the gait energy diagram generation module is used for processing the plurality of bending training monitoring key frames to obtain a gait energy diagram;
the first scale image feature extraction module is used for enabling the gait energy image to pass through a convolutional neural network model serving as a feature extractor to obtain a first scale gait energy feature image;
the second scale image feature extraction module is used for enabling the gait energy diagram to pass through a ViT model to obtain a second scale gait energy feature diagram;
the multi-scale image feature association module is used for fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; and
and the gait detection module is used for passing the classification characteristic diagram through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the gait of the object to be monitored is normal or not.
In the above postoperative leg bending training monitoring system, the first scale image feature extraction module is configured to:
each layer of the convolutional neural network model serving as the feature extractor is respectively carried out in forward transfer of the layer:
carrying out convolution processing on input data to obtain a first scale convolution characteristic diagram;
carrying out mean pooling treatment on the first scale convolution feature map to obtain a first scale pooling feature map; and
non-linear activation is carried out on the first scale pooling feature map so as to obtain a first scale activation feature map;
the output of the last layer of the convolutional neural network model serving as the feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model serving as the feature extractor is the gait energy map.
In the above postoperative leg bending training monitoring system, the second scale image feature extraction module includes:
the image blocking unit is used for carrying out image blocking processing on the gait energy diagram to obtain a sequence of image blocks;
an embedded coding unit, configured to use an embedded layer of the ViT model to perform embedded coding on each image block in the sequence of image blocks to obtain a sequence of image block embedded vectors;
An image context Wen Yuyi association unit for inputting the sequence of image block embedding vectors into the converter structure of the ViT model to obtain a sequence of image block context semantic feature vectors; and
and the dimension reconstruction unit is used for arranging the sequence of the image block context semantic feature vectors according to the corresponding spatial positions during the image block processing so as to obtain the second-scale gait energy feature map.
In the above-mentioned postoperative leg bending training monitored control system, the embedded coding unit is used for:
arranging pixel values of all pixel positions in each image block in the sequence of image blocks into one-dimensional vectors; and
and performing full-connection coding on the one-dimensional vector by using a full-connection layer of the ViT model according to the following full-connection coding formula to extract high-dimensional implicit characteristics of characteristic values of all positions in the one-dimensional vector, wherein the full-connection coding formula is as follows:wherein X is the one-dimensional vector, Y is the output vector, W is the weight matrix, B is the bias vector,>representing a matrix multiplication.
In the above postoperative leg bending training monitoring system, the multi-scale image feature association module is configured to:
performing scale cognition-based Gaussian joint density fusion on the first scale gait energy feature map and the second scale gait energy feature map by using the following optimization formula to obtain the classification feature map;
Wherein, the optimization formula is:
f 1 ∈F 1 and f 2 ∈F 2
wherein F is 1 Representing the first scale gait energy characteristic diagram, F 2 Representing the second scale gait energy profile, f 1 Is the characteristic value of each position in the first scale gait energy characteristic diagram, f 2 Is the characteristic value of each position in the second scale gait energy characteristic diagram, μ and σ are the mean and variance of the characteristic set of all characteristic values of the first scale gait energy characteristic diagram and the second scale gait energy characteristic diagram, W, H and C are the width, height and channel number of the characteristic diagram, respectively, and f' is the characteristic value of each position of the classification characteristic diagram.
In the above-mentioned postoperative leg bending training monitored control system, the gait detection module includes:
the feature map unfolding unit is used for unfolding the classification feature map into classification feature vectors according to row vectors or column vectors;
the full-connection coding unit is used for carrying out full-connection coding on the classification characteristic vectors by using a full-connection layer of the classifier so as to obtain coded classification characteristic vectors; and
and the classification unit is used for inputting the coding classification feature vector into a Softmax classification function of the classifier to obtain the classification result.
According to another aspect of the present application, there is provided a post-operative leg bending training monitoring method comprising:
acquiring a bending training monitoring video of an object to be monitored;
extracting a plurality of bending training monitoring key frames in a gait cycle from the bending training monitoring video of the object to be monitored;
processing the plurality of leg bending training monitoring key frames to obtain a gait energy diagram;
the gait energy image is passed through a convolutional neural network model serving as a feature extractor to obtain a first-scale gait energy feature image;
passing the gait energy diagram through a ViT model to obtain a second-scale gait energy characteristic diagram;
fusing the first scale gait energy feature map and the second scale gait energy feature map to obtain a classification feature map; and
and the classification characteristic diagram is passed through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the gait of the object to be monitored is normal or not.
In the above method for monitoring postoperative leg bending training, the step of obtaining a first scale step energy feature map by using the step energy map as a convolutional neural network model of a feature extractor includes:
each layer of the convolutional neural network model serving as the feature extractor is respectively carried out in forward transfer of the layer:
Carrying out convolution processing on input data to obtain a first scale convolution characteristic diagram;
carrying out mean pooling treatment on the first scale convolution feature map to obtain a first scale pooling feature map; and
non-linear activation is carried out on the first scale pooling feature map so as to obtain a first scale activation feature map;
the output of the last layer of the convolutional neural network model serving as the feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model serving as the feature extractor is the gait energy map.
In the above method for monitoring postoperative leg bending training, the step of passing the gait energy map through a ViT model to obtain a second scale gait energy feature map includes:
performing image blocking processing on the gait energy diagram to obtain a sequence of image blocks;
using the embedding layer of the ViT model to respectively carry out embedded coding on each image block in the sequence of the image blocks so as to obtain a sequence of image block embedded vectors;
inputting the sequence of image block embedded vectors into a converter structure of the ViT model to obtain a sequence of image block context semantic feature vectors; and
and arranging the sequence of the image block context semantic feature vectors according to the corresponding spatial positions during the image block processing to obtain the second-scale gait energy feature map.
In the above method for monitoring post-operative leg bending training, the embedding layer of the ViT model is used to respectively perform embedded coding on each image block in the sequence of image blocks to obtain a sequence of image block embedded vectors, including:
arranging pixel values of all pixel positions in each image block in the sequence of image blocks into one-dimensional vectors; and
and performing full-connection coding on the one-dimensional vector by using a full-connection layer of the ViT model according to the following full-connection coding formula to extract high-dimensional implicit characteristics of characteristic values of all positions in the one-dimensional vector, wherein the full-connection coding formula is as follows:wherein X is the one-dimensional vector, Y is the output vector, W is the weight matrix, B is the bias vector,>representing a matrix multiplication.
Compared with the prior art, the postoperative bending training monitoring system and the postoperative bending training monitoring method provided by the application have the advantages that firstly, a plurality of bending training monitoring key frames in a gait cycle are extracted from bending training monitoring videos of an object to be monitored, then, the plurality of bending training monitoring key frames are processed to obtain a gait energy diagram, then, the gait energy diagram is respectively passed through a convolutional neural network model and a ViT model to obtain a first-scale gait energy characteristic diagram and a second-scale gait energy characteristic diagram, then, the first-scale gait energy characteristic diagram and the second-scale gait energy characteristic diagram are fused to obtain a classification characteristic diagram, and finally, the classification characteristic diagram is passed through a classifier to obtain a classification result for indicating whether the gait of the object to be monitored is normal. In this way, the gait evaluation of the patient can be accurately performed, and abnormalities can be found in time to facilitate corresponding treatments.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort to a person of ordinary skill in the art. The following drawings are not intended to be drawn to scale, emphasis instead being placed upon illustrating the principles of the application.
Fig. 1 is an application scenario diagram of a post-operative leg bending training monitoring system according to an embodiment of the present application.
Fig. 2 is a block diagram schematic of a post-operative leg bending training monitoring system in accordance with an embodiment of the present application.
Fig. 3 is a block diagram of the second scale image feature extraction module in the postoperative bending training monitor system according to an embodiment of the present application.
Fig. 4 is a block diagram schematic diagram of the gait detection module in the postoperative swing training monitoring system according to the embodiment of the present application.
Fig. 5 is a flow chart of a post-operative leg curl training monitoring method according to an embodiment of the application.
Fig. 6 is a schematic diagram of a system architecture of a post-operative leg bending training monitoring method according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are also within the scope of the application.
As used in the specification and in the claims, the terms "a," "an," "the," and/or "the" are not specific to a singular, but may include a plurality, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that the steps and elements are explicitly identified, and they do not constitute an exclusive list, as other steps or elements may be included in a method or apparatus.
Although the present application makes various references to certain modules in a system according to embodiments of the present application, any number of different modules may be used and run on a user terminal and/or server. The modules are merely illustrative, and different aspects of the systems and methods may use different modules.
A flowchart is used in the present application to describe the operations performed by a system according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in order precisely. Rather, the various steps may be processed in reverse order or simultaneously, as desired. Also, other operations may be added to or removed from these processes.
Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.
As described above, post-operative leg bending monitoring requires attention to several monitoring items: activity, for postoperative patients with leg bending, proper activity can promote blood circulation and mitigate thrombosis risk. Therefore, the patient needs to pay close attention to the activity condition, and proper activities are performed according to the doctor instruction; pain assessment, the patient may have a feeling of pain discomfort after surgery. The pain of the patient needs to be evaluated and treated according to the medication advice of the doctor; gait assessment, postoperative leg bending patients may develop knee flexion while walking, resulting in abnormal gait. The gait of the patient needs to be evaluated, and abnormal conditions are found in time so as to be correspondingly processed; lower limb activity assessment, monitoring patient lower limb activity, can help monitor muscle traction and joint range, increase lower limb circulation, and reduce risk of deep vein thrombosis in the limb. The monitoring items require professional personnel to scientifically monitor the patient, timely discover abnormal conditions, and timely adopt corresponding processing methods so as to ensure that the patient can be better recovered.
Further, for patients who do post-operative leg bending exercises, appropriate activities can promote blood circulation, alleviating the risk of thrombosis. However, postoperative leg bending patients may experience knee flexion while walking, resulting in gait abnormalities that affect the postoperative rehabilitation of the patient. Accordingly, an optimized post-operative leg bending training monitoring system is desired that can evaluate the gait of the patient, and discover anomalies in time for corresponding treatment.
Accordingly, considering that the patient is in the process of performing postoperative bending training, in order to be able to evaluate the gait of the patient and to find abnormalities in time, it is desirable to perform analysis on the bending training monitoring video of the object to be monitored. That is, gait feature information on the patient during post-operative leg bending training exists in the leg bending training monitoring video of the object to be monitored, and therefore gait monitoring during post-operative leg bending training of the patient is achieved based on analysis of the video. However, because there is a large amount of information in the monitor video for the leg bending training, gait feature information about the post-operation leg bending training of the patient is a small-scale implicit feature in the monitor video, and it is difficult to perform sufficient feature expression. Therefore, in this process, it is difficult to fully extract gait implicit characteristic distribution information about postoperative leg bending training of a patient in the leg bending training monitoring video, so as to accurately perform gait evaluation of the patient, and to find abnormalities in time to facilitate corresponding processing.
In recent years, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, text signal processing, and the like. The development of deep learning and neural networks provides a new solution idea and scheme for mining gait implicit characteristic distribution information about postoperative leg bending training of patients in the leg bending training monitoring video.
Specifically, in the technical scheme of the application, firstly, a bending training monitoring video of an object to be monitored is obtained. Next, considering that in the bending training monitor video, the gait change characteristics of the object to be monitored may be represented by the difference between adjacent monitor frames in one gait cycle in the bending training monitor video, that is, the gait change condition during the patient's postoperative bending training is represented by the image representation of the adjacent image frames. However, in consideration of the fact that the difference between adjacent frames in the monitor video is small, there is a large amount of data redundancy, so in order to reduce the amount of calculation and avoid the adverse effect of the data redundancy on detection, the knee training monitor video is subjected to key frame sampling at a predetermined sampling frequency to extract a plurality of knee training monitor key frames in one gait cycle from the knee training monitor video of the object to be monitored.
Then, considering that the gait energy diagram (Gait Engery Image, GEI) is a very common gait feature in gait detection, it can well represent the feature information of the speed, morphology, etc. of the gait. Therefore, in the technical scheme of the application, the plurality of bending training monitoring key frames are processed to obtain the gait energy diagram. In particular, in one specific example of the present application, gait energy pattern generation mainly includes: cutting the outline of the patient on the original outline map, namely in each bending training monitoring key frame, and taking the top of the head of the patient as a central position; then, the images after clipping are synthesized, and a gait energy diagram is synthesized by using the images of one gait cycle.
Further, feature mining of the gait energy map is performed by using a convolutional neural network model which is used as a feature extractor and has excellent performance in the aspect of implicit feature extraction of images, so that implicit feature distribution information such as gait speed, morphology and the like of a patient in the postoperative leg bending training process is extracted, and a first-scale gait energy feature map is obtained.
Then, given the inherent limitations of convolution operations, the pure CNN approach has difficulty learning explicit global and remote semantic information interactions. In addition, because hidden characteristics of the gait energy diagram, such as gait speed, morphology and the like of a patient in the postoperative leg bending training process, are small-scale fine characteristics, capturing and extracting are difficult to carry out. Therefore, in order to improve the expression capability of the subtle features of the gait energy diagram on the hidden small scale of the gait of the patient, so as to improve the accurate assessment of the gait of the patient, in the technical scheme of the application, the gait energy diagram is encoded by a ViT model so as to extract hidden global context semantic association feature distribution information of the gait energy diagram on the gait speed and the morphology of the patient in the postoperative leg bending training process, thereby obtaining a second scale gait energy feature diagram. It should be understood that the small-scale implicit features on the gait speed and morphology of the patient during the postoperative leg bending training process in each image block after the image blocking processing of the gait energy map are no longer small-scale feature information, so as to be beneficial to the subsequent detection of whether the gait of the patient is normal or not. In particular, here, the embedding layer linearly projects the individual image blocks as one-dimensional embedding vectors via a learnable embedding matrix. The embedding process is realized by firstly arranging pixel values of all pixel positions in each image block into one-dimensional vectors, and then carrying out full-connection coding on the one-dimensional vectors by using a full-connection layer. And, here, the ViT model may directly process the image blocks through a self-attention mechanism like a transducer, so as to extract global implicit context semantic association characteristic information based on the gait energy diagram and related to the gait speed and the morphology of the patient in the postoperative leg bending training process in the image blocks.
And then, further fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map, so as to fuse multi-scale relevance feature distribution information based on long-distance dependence correlation and short-distance dependence correlation of hidden features such as gait speed, morphology and the like of a patient in the training process in the bending training monitoring video. And then, carrying out classification processing on the classification characteristic diagram in a classifier to obtain a classification result for indicating whether the gait of the object to be monitored is normal.
That is, in the technical solution of the present application, the label of the classifier includes a gait normal (first label) of the object to be monitored and a gait abnormal (second label) of the object to be monitored, wherein the classifier determines to which classification label the classification feature map belongs through a soft maximum function. It should be noted that the first tag p1 and the second tag p2 do not include a human-set concept, and in fact, during the training process, the computer model does not have a concept of "whether the gait of the object to be monitored is normal", which is simply that there are two kinds of classification tags and the probability that the output feature is under the two kinds of classification tags, that is, the sum of p1 and p2 is one. Therefore, the classification result of whether the gait of the object to be monitored is normal is actually converted into the classified probability distribution conforming to the natural rule through classifying the tags, and the physical meaning of the natural probability distribution of the tags is essentially used instead of the language text meaning of whether the gait of the object to be monitored is normal. It should be understood that, in the technical solution of the present application, the classification label of the classifier is an evaluation detection label for whether the gait of the object to be monitored is normal, so after the classification result is obtained, the gait evaluation of the patient during the postoperative leg bending training process can be accurately performed based on the classification result, and the abnormality can be found in time to facilitate the corresponding processing.
Particularly, in the technical scheme of the application, when the first-scale gait energy feature map and the second-scale gait energy feature map are fused to obtain the classification feature map, as the first-scale gait energy feature map and the second-scale gait energy feature map respectively express the image semantics of the gait energy maps under different scales, namely, the first-scale gait energy feature map expresses the locally associated image semantics of the feature extractor under the convolution kernel scale, and the second-scale gait energy feature map expresses the context image semantics of the ViT model under the image block scale, the fusion of the classification feature map to the different scales of the first-scale gait energy feature map and the second-scale gait energy feature map is expected to improve the fusion effect of the classification feature map on the first-scale gait energy feature map and the second-scale gait energy feature map.
Based on this, the applicant of the present application performed on the first scale gait energy profile F 1 And the second scale gait energy profile F 2 The gaussian joint density fusion based on scale cognition was performed as:
f 1 ∈F 1 and f 2 ∈F 2
mu and sigma are respectivelyIs the first scale gait energy feature map F 1 And the second scale gait energy profile F 2 Feature set of all feature values of (a), W, H and C are the width, height, and number of channels of the feature map, respectively, and f' is the feature value of the classified feature map.
Here, the scale-cognition-based gaussian joint density fusion considers the scale optimal expression characteristic of the feature to be fused, and in order to improve the effectiveness of feature fusion and generalization performance relative to the feature to be fused, the classification performance difference (performance gap) of the feature distribution based on the mean and variance is strategically expressed (policy representation) by taking the scale-cognition-based gaussian joint density as a dominance function (advantage function), so that the feature-scale self-dependency of the feature fusion is improved, and the fusion effect of the classification feature map on the first-scale gait energy feature map and the second-scale gait energy feature map is improved. Therefore, gait detection and evaluation of the patient in the postoperative leg bending training process can be accurately carried out, abnormality can be found timely, and corresponding treatment is facilitated.
Fig. 1 is an application scenario diagram of a post-operative leg bending training monitoring system according to an embodiment of the present application. As shown in fig. 1, in this application scenario, first, a bending training monitoring video (e.g., D shown in fig. 1) of an object to be monitored may be acquired through a device such as a monitoring camera (e.g., C shown in fig. 1), and then the bending training monitoring video of the object to be monitored is input to a server (e.g., S shown in fig. 1) in which a post-operation bending training monitoring algorithm is deployed, where the server can process the bending training monitoring video of the object to be monitored using the post-operation bending training monitoring algorithm to obtain a classification result for indicating whether gait of the object to be monitored is normal.
Having described the basic principles of the present application, various non-limiting embodiments of the present application will now be described in detail with reference to the accompanying drawings.
Fig. 2 is a block diagram schematic of a post-operative leg bending training monitoring system in accordance with an embodiment of the present application. As shown in fig. 2, a post-operative leg bending training monitoring system 100 according to an embodiment of the present application includes: the video data acquisition module 110 is used for acquiring a bending training monitoring video of an object to be monitored; a key frame extracting module 120, configured to extract a plurality of key frames for training and monitoring in a gait cycle from the video for training and monitoring the bending leg of the object to be monitored; a gait energy diagram generation module 130, configured to process the plurality of training monitor key frames for leg bending to obtain a gait energy diagram; a first scale image feature extraction module 140, configured to pass the gait energy graph through a convolutional neural network model serving as a feature extractor to obtain a first scale gait energy feature graph; a second scale image feature extraction module 150, configured to pass the gait energy graph through a ViT model to obtain a second scale gait energy feature graph; a multi-scale image feature correlation module 160, configured to fuse the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; and a gait detection module 170, configured to pass the classification feature map through a classifier to obtain a classification result, where the classification result is used to indicate whether the gait of the object to be monitored is normal. Therefore, in order to be able to evaluate the gait of a patient and to find abnormalities in time, it is desirable to do so by analyzing a crimping training monitoring video of the object to be monitored. That is, gait feature information on the patient during post-operative leg bending training exists in the leg bending training monitoring video of the object to be monitored, and therefore gait monitoring during post-operative leg bending training of the patient is achieved based on analysis of the video. However, because there is a large amount of information in the monitor video for the leg bending training, gait feature information about the post-operation leg bending training of the patient is a small-scale implicit feature in the monitor video, and it is difficult to perform sufficient feature expression. Therefore, in the technical scheme of the application, gait implicit characteristic distribution information about postoperative leg bending training of a patient in the leg bending training monitoring video is mined, so that gait evaluation of the patient is accurately performed, and abnormality is found in time so as to be convenient for corresponding processing.
More specifically, in the embodiment of the present application, the video data acquisition module 110 is configured to acquire a bending training monitoring video of an object to be monitored. For patients who do post-operative leg bending exercises, appropriate activities can promote blood circulation, alleviating the risk of thrombosis. However, postoperative leg bending patients may experience knee flexion while walking, resulting in gait abnormalities that affect the postoperative rehabilitation of the patient.
More specifically, in an embodiment of the present application, the key frame extracting module 120 is configured to extract a plurality of key frames for training and monitoring in a gait cycle from the video for training and monitoring of the bending leg of the subject to be monitored. In the bending training monitoring video, gait change characteristics of the object to be monitored can be represented by differences between adjacent monitoring frames in one gait cycle in the bending training monitoring video, that is, gait change conditions during the postoperative bending training of a patient are represented by image representations of the adjacent image frames. However, in consideration of the fact that the difference between adjacent frames in the monitor video is small, there is a large amount of data redundancy, so in order to reduce the amount of calculation and avoid the adverse effect of the data redundancy on detection, the knee training monitor video is subjected to key frame sampling at a predetermined sampling frequency to extract a plurality of knee training monitor key frames in one gait cycle from the knee training monitor video of the object to be monitored.
More specifically, in an embodiment of the present application, the gait energy graph generating module 130 is configured to process the plurality of training monitor keyframes for leg bending to obtain a gait energy graph. The gait energy diagram is a very common gait feature in gait detection, and can well represent the feature information of the speed, the form and the like of the gait. Therefore, in the technical scheme of the application, the plurality of bending training monitoring key frames are processed to obtain the gait energy diagram. In particular, in one specific example of the present application, gait energy pattern generation mainly includes: cutting the outline of the patient on the original outline map, namely in each bending training monitoring key frame, and taking the top of the head of the patient as a central position; then, the images after clipping are synthesized, and a gait energy diagram is synthesized by using the images of one gait cycle.
More specifically, in an embodiment of the present application, the first scale image feature extraction module 140 is configured to pass the gait energy graph through a convolutional neural network model as a feature extractor to obtain a first scale gait energy feature graph. Feature mining of the gait energy map is performed by using a convolutional neural network model which is used as a feature extractor and has excellent performance in the aspect of implicit feature extraction of images, so that implicit feature distribution information such as gait speed, morphology and the like of a patient in the postoperative leg bending training process is extracted, and a first-scale gait energy feature map is obtained.
More specifically, in an embodiment of the present application, the first scale image feature extraction module 140 is configured to: each layer of the convolutional neural network model serving as the feature extractor is respectively carried out in forward transfer of the layer: carrying out convolution processing on input data to obtain a first scale convolution characteristic diagram; carrying out mean pooling treatment on the first scale convolution feature map to obtain a first scale pooling feature map; performing nonlinear activation on the first scale pooling feature map to obtain a first scale activation feature map; the output of the last layer of the convolutional neural network model serving as the feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model serving as the feature extractor is the gait energy map.
It should be appreciated that convolutional neural network (Convolutional Neural Network, CNN) is an artificial neural network and has wide application in the fields of image recognition and the like. The convolutional neural network may include an input layer, a hidden layer, and an output layer, where the hidden layer may include a convolutional layer, a pooling layer, an activation layer, a full connection layer, etc., where the previous layer performs a corresponding operation according to input data, outputs an operation result to the next layer, and obtains a final result after the input initial data is subjected to a multi-layer operation. The convolutional neural network model has excellent performance in the aspect of image local feature extraction by taking a convolutional kernel as a feature filtering factor, and has stronger feature extraction generalization capability and fitting capability compared with the traditional image feature extraction algorithm based on statistics or feature engineering.
More specifically, in an embodiment of the present application, the second scale image feature extraction module 150 is configured to pass the gait energy graph through a ViT model to obtain a second scale gait energy feature graph. Given the inherent limitations of convolution operations, pure CNN methods have difficulty learning explicit global and remote semantic information interactions. In addition, because hidden characteristics of the gait energy diagram, such as gait speed, morphology and the like of a patient in the postoperative leg bending training process, are small-scale fine characteristics, capturing and extracting are difficult to carry out. Therefore, in order to improve the expression capability of the subtle features of the gait energy diagram on the hidden small scale of the gait of the patient, so as to improve the accurate assessment of the gait of the patient, in the technical scheme of the application, the gait energy diagram is encoded by a ViT model so as to extract hidden global context semantic association feature distribution information of the gait energy diagram on the gait speed and the morphology of the patient in the postoperative leg bending training process, thereby obtaining a second scale gait energy feature diagram.
It should be understood that the small-scale implicit features on the gait speed and morphology of the patient during the postoperative leg bending training process in each image block after the image blocking processing of the gait energy map are no longer small-scale feature information, so as to be beneficial to the subsequent detection of whether the gait of the patient is normal or not. In particular, here, the embedding layer linearly projects the individual image blocks as one-dimensional embedding vectors via a learnable embedding matrix. The embedding process is realized by firstly arranging pixel values of all pixel positions in each image block into one-dimensional vectors, and then carrying out full-connection coding on the one-dimensional vectors by using a full-connection layer. And, here, the ViT model may directly process the image blocks through a self-attention mechanism like a transducer, so as to extract global implicit context semantic association characteristic information based on the gait energy diagram and related to the gait speed and the morphology of the patient in the postoperative leg bending training process in the image blocks.
More specifically, in an embodiment of the present application, as shown in fig. 3, the second scale image feature extraction module 150 includes: an image blocking unit 151, configured to perform image blocking processing on the gait energy map to obtain a sequence of image blocks; an embedded encoding unit 152, configured to perform embedded encoding on each image block in the sequence of image blocks by using an embedded layer of the ViT model to obtain a sequence of image block embedded vectors; an image context Wen Yuyi association unit 153 for inputting the sequence of image block embedding vectors into the converter structure of the ViT model to obtain a sequence of image block context semantic feature vectors; and a dimension reconstruction unit 154, configured to arrange the sequence of the context semantic feature vectors of the image block according to a spatial position corresponding to the image block processing to obtain the second-scale gait energy feature map.
More specifically, in the embodiment of the present application, the embedded encoding unit 152 is configured to: arranging pixel values of all pixel positions in each image block in the sequence of image blocks into one-dimensional vectors; and performing full-join encoding on the one-dimensional vector by using a full-join layer of the ViT model according to the following full-join encoding formula to extract high-dimensional implicit features of feature values of each position in the one-dimensional vector, wherein the full-join encoding formula is as follows: Wherein X is the one-dimensional vector, Y is the output vector, W is the weight matrix, B is the bias vector,>representing a matrix multiplication.
More specifically, in an embodiment of the present application, the multi-scale image feature association module 160 is configured to fuse the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map. Therefore, multiscale relevance feature distribution information based on long-distance dependence and short-distance dependence of hidden features such as gait speed, morphology and the like of a patient in the training process in the bending training monitoring video is fused.
Particularly, in the technical scheme of the application, when the classification characteristic map is obtained by fusing the first-scale gait energy characteristic map and the second-scale gait energy characteristic map, the first-scale gait energy characteristic map and the second-scale gait energy characteristic map respectively express different valuesThe image semantics of the gait energy map under the scale, namely the first scale gait energy feature map expresses the local correlation image semantics under the convolution kernel scale of the feature extractor, and the second scale gait energy feature map expresses the context image semantics under the image block scale of the ViT model, so that fusion of the first scale gait energy feature map and the second scale gait energy feature map is expected aiming at different scales of the first scale gait energy feature map and the second scale gait energy feature map so as to improve the fusion effect of the classification feature map on the first scale gait energy feature map and the second scale gait energy feature map. Based on this, the applicant of the present application performed on the first scale gait energy profile F 1 And the second scale gait energy profile F 2 And carrying out Gaussian joint density fusion based on scale cognition.
More specifically, in an embodiment of the present application, the multi-scale image feature association module 160 is configured to: performing scale cognition-based Gaussian joint density fusion on the first scale gait energy feature map and the second scale gait energy feature map by using the following optimization formula to obtain the classification feature map; wherein, the optimization formula is:
f 1 ∈F 1 and f 2 ∈F 2
wherein F is 1 Representing the first scale gait energy characteristic diagram, F 2 Representing the second scale gait energy profile, f 1 Is the characteristic value of each position in the first scale gait energy characteristic diagram, f 2 Is the characteristic value of each position in the second scale gait energy characteristic diagram, mu and sigma are the mean value and variance of the characteristic set of all characteristic values of the first scale gait energy characteristic diagram and the second scale gait energy characteristic diagram respectively,w, H and C are the width, height and number of channels, respectively, of the feature map, and f' is the respective position feature value of the classification feature map.
Here, the scale-cognition-based gaussian joint density fusion considers the scale optimal expression characteristic of the feature to be fused, and in order to improve the effectiveness of feature fusion and generalization performance relative to the feature to be fused, the classification performance difference (performance gap) of the feature distribution based on the mean and variance is strategically expressed (policy representation) by taking the scale-cognition-based gaussian joint density as a dominance function (advantage function), so that the feature-scale self-dependency of the feature fusion is improved, and the fusion effect of the classification feature map on the first-scale gait energy feature map and the second-scale gait energy feature map is improved. Therefore, gait detection and evaluation of the patient in the postoperative leg bending training process can be accurately carried out, abnormality can be found timely, and corresponding treatment is facilitated.
More specifically, in the embodiment of the present application, the gait detection module 170 is configured to pass the classification feature map through a classifier to obtain a classification result, where the classification result is used to indicate whether the gait of the object to be monitored is normal. That is, in the technical solution of the present application, the label of the classifier includes a gait normal (first label) of the object to be monitored and a gait abnormal (second label) of the object to be monitored, wherein the classifier determines to which classification label the classification feature map belongs through a soft maximum function. It should be understood that, in the technical solution of the present application, the classification label of the classifier is an evaluation detection label for whether the gait of the object to be monitored is normal, so after the classification result is obtained, the gait evaluation of the patient during the postoperative leg bending training process can be accurately performed based on the classification result, and the abnormality can be found in time to facilitate the corresponding processing.
It should be appreciated that the role of the classifier is to learn the classification rules and classifier using a given class, known training data, and then classify (or predict) the unknown data. Logistic regression (logistics), SVM, etc. are commonly used to solve the classification problem, and for multi-classification problems (multi-class classification), logistic regression or SVM can be used as well, but multiple bi-classifications are required to compose multiple classifications, but this is error-prone and inefficient, and the commonly used multi-classification method is the Softmax classification function.
More specifically, in an embodiment of the present application, as shown in fig. 4, the gait detection module 170 includes: a feature map expansion unit 171 for expanding the classification feature map into classification feature vectors according to row vectors or column vectors; a full-connection encoding unit 172, configured to perform full-connection encoding on the classification feature vector by using a full-connection layer of the classifier to obtain an encoded classification feature vector; and a classification unit 173, configured to input the encoded classification feature vector into a Softmax classification function of the classifier to obtain the classification result.
In summary, the postoperative leg bending training monitoring system 100 according to the embodiment of the present application is illustrated, firstly, a plurality of leg bending training monitoring key frames in a gait cycle are extracted from a leg bending training monitoring video of an object to be monitored, then, the plurality of leg bending training monitoring key frames are processed to obtain a gait energy map, then, the gait energy map is respectively passed through a convolutional neural network model and a ViT model to obtain a first-scale gait energy feature map and a second-scale gait energy feature map, then, the first-scale gait energy feature map and the second-scale gait energy feature map are fused to obtain a classification feature map, and finally, the classification feature map is passed through a classifier to obtain a classification result for indicating whether the gait of the object to be monitored is normal. In this way, the gait evaluation of the patient can be accurately performed, and abnormalities can be found in time to facilitate corresponding treatments.
As described above, the post-operative bending training monitoring system 100 according to the embodiment of the present application may be implemented in various terminal devices, for example, a server having the post-operative bending training monitoring algorithm according to the embodiment of the present application, and the like. In one example, the post-operative leg rest training monitoring system 100 according to embodiments of the present application may be integrated into the terminal device as a software module and/or hardware module. For example, the post-operative leg bending training monitoring system 100 according to an embodiment of the present application may be a software module in the operating system of the terminal device, or may be an application developed for the terminal device; of course, the post-operative leg bending training monitoring system 100 according to an embodiment of the present application may also be one of a plurality of hardware modules of the terminal device.
Alternatively, in another example, the post-operative leg rest training monitoring system 100 and the terminal device according to the embodiment of the present application may be separate devices, and the post-operative leg rest training monitoring system 100 may be connected to the terminal device through a wired and/or wireless network and transmit interactive information according to an agreed data format.
Fig. 5 is a flow chart of a post-operative leg curl training monitoring method according to an embodiment of the application. As shown in fig. 5, the postoperative leg bending training monitoring method according to the embodiment of the present application includes: s110, acquiring a bending training monitoring video of an object to be monitored; s120, extracting a plurality of bending training monitoring key frames in a gait cycle from the bending training monitoring video of the object to be monitored; s130, processing the plurality of leg bending training monitoring key frames to obtain a gait energy diagram; s140, the gait energy image is passed through a convolutional neural network model serving as a feature extractor to obtain a first-scale gait energy feature image; s150, passing the gait energy diagram through a ViT model to obtain a second-scale gait energy characteristic diagram; s160, fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; and S170, enabling the classification characteristic diagram to pass through a classifier to obtain a classification result, wherein the classification result is used for indicating whether gait of the object to be monitored is normal or not.
Fig. 6 is a schematic diagram of a system architecture of a post-operative leg bending training monitoring method according to an embodiment of the present application. As shown in fig. 6, in the system architecture of the postoperative bending training monitoring method, first, a bending training monitoring video of an object to be monitored is acquired; then extracting a plurality of bending training monitoring key frames in a gait cycle from the bending training monitoring video of the object to be monitored; then, processing the plurality of leg bending training monitoring key frames to obtain a gait energy diagram; then, the gait energy image is passed through a convolutional neural network model serving as a feature extractor to obtain a first-scale gait energy feature image; then, the gait energy diagram is passed through a ViT model to obtain a second-scale gait energy characteristic diagram; then, fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; and finally, the classification characteristic diagram is passed through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the gait of the object to be monitored is normal or not.
In a specific example, in the above method for monitoring post-operative leg bending training, the step of passing the gait energy graph through a convolutional neural network model as a feature extractor to obtain a first scale gait energy feature graph includes: each layer of the convolutional neural network model serving as the feature extractor is respectively carried out in forward transfer of the layer: carrying out convolution processing on input data to obtain a first scale convolution characteristic diagram; carrying out mean pooling treatment on the first scale convolution feature map to obtain a first scale pooling feature map; performing nonlinear activation on the first scale pooling feature map to obtain a first scale activation feature map; the output of the last layer of the convolutional neural network model serving as the feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model serving as the feature extractor is the gait energy map.
In a specific example, in the above method for monitoring post-operative leg bending training, the step of passing the gait energy graph through a ViT model to obtain a second-scale gait energy feature graph includes: performing image blocking processing on the gait energy diagram to obtain a sequence of image blocks; using the embedding layer of the ViT model to respectively carry out embedded coding on each image block in the sequence of the image blocks so as to obtain a sequence of image block embedded vectors; inputting the sequence of image block embedded vectors into a converter structure of the ViT model to obtain a sequence of image block context semantic feature vectors; and arranging the sequence of the image block context semantic feature vectors according to the corresponding spatial positions during the image block processing to obtain the second-scale gait energy feature map.
In a specific example, in the above method for monitoring post-operative leg bending training, the embedding layer of the ViT model is used to respectively perform embedded encoding on each image block in the sequence of image blocks to obtain a sequence of image block embedded vectors, including: arranging pixel values of all pixel positions in each image block in the sequence of image blocks into one-dimensional vectors; and performing full-join encoding on the one-dimensional vector by using a full-join layer of the ViT model according to the following full-join encoding formula to extract high-dimensional implicit features of feature values of each position in the one-dimensional vector, wherein the full-join encoding formula is as follows:wherein X is the one-dimensional vector, Y is the output vector, W is the weight matrix, B is the bias vector,>representing a matrix multiplication.
In a specific example, in the above postoperative leg bending training monitoring method, fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map includes: performing scale cognition-based Gaussian joint density fusion on the first scale gait energy feature map and the second scale gait energy feature map by using the following optimization formula to obtain the classification feature map; wherein, the optimization formula is:
f 1 ∈F 1 and f 2 ∈F 2
Wherein F is 1 Representing the first scale gait energy characteristic diagram, F 2 Representing the second scale gait energy profile, f 1 Is the characteristic value of each position in the first scale gait energy characteristic diagram, f 2 Is the characteristic value of each position in the second scale gait energy characteristic diagram, μ and σ are the mean and variance of the characteristic set of all characteristic values of the first scale gait energy characteristic diagram and the second scale gait energy characteristic diagram, W, H and C are the width, height and channel number of the characteristic diagram, respectively, and f' is the characteristic value of each position of the classification characteristic diagram.
In a specific example, in the above method for monitoring post-operation leg bending training, the classifying feature map is passed through a classifier to obtain a classifying result, where the classifying result is used to indicate whether gait of the object to be monitored is normal, and the method includes: expanding the classification characteristic map into classification characteristic vectors according to row vectors or column vectors; performing full-connection coding on the classification feature vectors by using a full-connection layer of the classifier to obtain coded classification feature vectors; and inputting the coding classification feature vector into a Softmax classification function of the classifier to obtain the classification result.
Here, it will be understood by those skilled in the art that the specific operations of the respective steps in the above-described post-operative leg bending training monitoring method have been described in detail in the above description of the post-operative leg bending training monitoring system 100 with reference to fig. 1 to 4, and thus, repetitive descriptions thereof will be omitted.
According to another aspect of the present application there is also provided a non-volatile computer readable storage medium having stored thereon computer readable instructions which when executed by a computer can perform a method as described above.
Program portions of the technology may be considered to be "products" or "articles of manufacture" in the form of executable code and/or associated data, embodied or carried out by a computer readable medium. A tangible, persistent storage medium may include any memory or storage used by a computer, processor, or similar device or related module. Such as various semiconductor memories, tape drives, disk drives, or the like, capable of providing storage functionality for software.
All or a portion of the software may sometimes communicate over a network, such as the internet or other communication network. Such communication may load software from one computer device or processor to another. For example: a hardware platform loaded from a server or host computer of the video object detection device to a computer environment, or other computer environment implementing the system, or similar functioning system related to providing information needed for object detection. Thus, another medium capable of carrying software elements may also be used as a physical connection between local devices, such as optical, electrical, electromagnetic, etc., propagating through cable, optical cable, air, etc. Physical media used for carrier waves, such as electrical, wireless, or optical, may also be considered to be software-bearing media. Unless limited to a tangible "storage" medium, other terms used herein to refer to a computer or machine "readable medium" mean any medium that participates in the execution of any instructions by a processor.
The application uses specific words to describe embodiments of the application. Reference to "a first/second embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic is associated with at least one embodiment of the application. Thus, it should be emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various positions in this specification are not necessarily referring to the same embodiment. Furthermore, certain features, structures, or characteristics of one or more embodiments of the application may be combined as suitable.
Furthermore, those skilled in the art will appreciate that the various aspects of the application are illustrated and described in the context of a number of patentable categories or circumstances, including any novel and useful procedures, machines, products, or materials, or any novel and useful modifications thereof. Accordingly, aspects of the application may be performed entirely by hardware, entirely by software (including firmware, resident software, micro-code, etc.) or by a combination of hardware and software. The above hardware or software may be referred to as a "data block," module, "" engine, "" unit, "" component, "or" system. Furthermore, aspects of the application may take the form of a computer product, comprising computer-readable program code, embodied in one or more computer-readable media.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The foregoing is illustrative of the present invention and is not to be construed as limiting thereof. Although a few exemplary embodiments of this invention have been described, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of this invention. Accordingly, all such modifications are intended to be included within the scope of this invention as defined in the following claims. It is to be understood that the foregoing is illustrative of the present invention and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the appended claims. The invention is defined by the claims and their equivalents.

Claims (10)

1. A post-operative leg bending training monitoring system, comprising:
the video data acquisition module is used for acquiring a bending training monitoring video of an object to be monitored;
the key frame extraction module is used for extracting a plurality of key frames for the bending training monitoring in a gait cycle from the bending training monitoring video of the object to be monitored;
the gait energy diagram generation module is used for processing the plurality of bending training monitoring key frames to obtain a gait energy diagram;
the first scale image feature extraction module is used for enabling the gait energy image to pass through a convolutional neural network model serving as a feature extractor to obtain a first scale gait energy feature image;
the second scale image feature extraction module is used for enabling the gait energy diagram to pass through a ViT model to obtain a second scale gait energy feature diagram;
the multi-scale image feature association module is used for fusing the first-scale gait energy feature map and the second-scale gait energy feature map to obtain a classification feature map; and
and the gait detection module is used for passing the classification characteristic diagram through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the gait of the object to be monitored is normal or not.
2. The post-operative leg curl training monitoring system of claim 1, wherein said first scale image feature extraction module is to:
each layer of the convolutional neural network model serving as the feature extractor is respectively carried out in forward transfer of the layer:
carrying out convolution processing on input data to obtain a first scale convolution characteristic diagram;
carrying out mean pooling treatment on the first scale convolution feature map to obtain a first scale pooling feature map; and
non-linear activation is carried out on the first scale pooling feature map so as to obtain a first scale activation feature map;
the output of the last layer of the convolutional neural network model serving as the feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model serving as the feature extractor is the gait energy map.
3. The post-operative leg curl training monitoring system of claim 2, wherein said second scale image feature extraction module comprises:
the image blocking unit is used for carrying out image blocking processing on the gait energy diagram to obtain a sequence of image blocks;
an embedded coding unit, configured to use an embedded layer of the ViT model to perform embedded coding on each image block in the sequence of image blocks to obtain a sequence of image block embedded vectors;
An image context Wen Yuyi association unit for inputting the sequence of image block embedding vectors into the converter structure of the ViT model to obtain a sequence of image block context semantic feature vectors; and
and the dimension reconstruction unit is used for arranging the sequence of the image block context semantic feature vectors according to the corresponding spatial positions during the image block processing so as to obtain the second-scale gait energy feature map.
4. A post-operative leg curl training monitoring system as defined in claim 3, wherein said embedded coding unit is configured to:
arranging pixel values of all pixel positions in each image block in the sequence of image blocks into one-dimensional vectors; and
and performing full-connection coding on the one-dimensional vector by using a full-connection layer of the ViT model according to the following full-connection coding formula to extract high-dimensional implicit characteristics of characteristic values of all positions in the one-dimensional vector, wherein the full-connection coding formula is as follows:wherein X is the one-dimensional vector, Y is the output vector, W is the weight matrix, B is the bias vector,>representing a matrix multiplication.
5. The post-operative leg curl training monitoring system of claim 4, wherein said multi-scale image feature correlation module is for:
Performing scale cognition-based Gaussian joint density fusion on the first scale gait energy feature map and the second scale gait energy feature map by using the following optimization formula to obtain the classification feature map;
wherein, the optimization formula is:
f 1 ∈F 1 and f 2 ∈F 2
wherein F is 1 Representing the first scale gait energy characteristic diagram, F 2 Representing the second scale gait energy profile, f 1 Is the characteristic value of each position in the first scale gait energy characteristic diagram, f 2 Is the characteristic value of each position in the second scale gait energy characteristic diagram, μ and σ are the mean and variance of the characteristic set of all characteristic values of the first scale gait energy characteristic diagram and the second scale gait energy characteristic diagram, W, H and C are the width, height and channel number of the characteristic diagram, respectively, and f' is the characteristic value of each position of the classification characteristic diagram.
6. The post-operative leg curl training monitoring system of claim 5, wherein said gait detection module comprises:
the feature map unfolding unit is used for unfolding the classification feature map into classification feature vectors according to row vectors or column vectors;
the full-connection coding unit is used for carrying out full-connection coding on the classification characteristic vectors by using a full-connection layer of the classifier so as to obtain coded classification characteristic vectors; and
And the classification unit is used for inputting the coding classification feature vector into a Softmax classification function of the classifier to obtain the classification result.
7. A method for monitoring post-operative leg bending training, comprising:
acquiring a bending training monitoring video of an object to be monitored;
extracting a plurality of bending training monitoring key frames in a gait cycle from the bending training monitoring video of the object to be monitored;
processing the plurality of leg bending training monitoring key frames to obtain a gait energy diagram;
the gait energy image is passed through a convolutional neural network model serving as a feature extractor to obtain a first-scale gait energy feature image;
passing the gait energy diagram through a ViT model to obtain a second-scale gait energy characteristic diagram;
fusing the first scale gait energy feature map and the second scale gait energy feature map to obtain a classification feature map; and
and the classification characteristic diagram is passed through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the gait of the object to be monitored is normal or not.
8. The post-operative leg curl training monitoring method of claim 7, wherein passing said gait energy pattern through a convolutional neural network model as a feature extractor to obtain a first scale gait energy pattern comprises:
Each layer of the convolutional neural network model serving as the feature extractor is respectively carried out in forward transfer of the layer:
carrying out convolution processing on input data to obtain a first scale convolution characteristic diagram;
carrying out mean pooling treatment on the first scale convolution feature map to obtain a first scale pooling feature map; and
non-linear activation is carried out on the first scale pooling feature map so as to obtain a first scale activation feature map;
the output of the last layer of the convolutional neural network model serving as the feature extractor is the first-scale gait energy feature map, and the input of the first layer of the convolutional neural network model serving as the feature extractor is the gait energy map.
9. The post-operative leg curl training monitoring method of claim 8, wherein passing said gait energy pattern through a ViT model to obtain a second scale gait energy profile comprises:
performing image blocking processing on the gait energy diagram to obtain a sequence of image blocks;
using the embedding layer of the ViT model to respectively carry out embedded coding on each image block in the sequence of the image blocks so as to obtain a sequence of image block embedded vectors;
inputting the sequence of image block embedded vectors into a converter structure of the ViT model to obtain a sequence of image block context semantic feature vectors; and
And arranging the sequence of the image block context semantic feature vectors according to the corresponding spatial positions during the image block processing to obtain the second-scale gait energy feature map.
10. The post-operative leg curl training monitoring method of claim 9, wherein each image block in said sequence of image blocks is respectively insert-encoded using an insert layer of said ViT model to obtain a sequence of image block insert vectors, comprising:
arranging pixel values of all pixel positions in each image block in the sequence of image blocks into one-dimensional vectors; and
and performing full-connection coding on the one-dimensional vector by using a full-connection layer of the ViT model according to the following full-connection coding formula to extract high-dimensional implicit characteristics of characteristic values of all positions in the one-dimensional vector, wherein the full-connection coding formula is as follows:wherein X is the one-dimensional vector, Y is the output vector, W is the weight matrix,b is the bias vector, ">Representing a matrix multiplication.
CN202310530445.XA 2023-05-12 2023-05-12 Postoperative leg bending training monitoring system and method thereof Pending CN116631619A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310530445.XA CN116631619A (en) 2023-05-12 2023-05-12 Postoperative leg bending training monitoring system and method thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310530445.XA CN116631619A (en) 2023-05-12 2023-05-12 Postoperative leg bending training monitoring system and method thereof

Publications (1)

Publication Number Publication Date
CN116631619A true CN116631619A (en) 2023-08-22

Family

ID=87620551

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310530445.XA Pending CN116631619A (en) 2023-05-12 2023-05-12 Postoperative leg bending training monitoring system and method thereof

Country Status (1)

Country Link
CN (1) CN116631619A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117437459A (en) * 2023-10-08 2024-01-23 昆山市第一人民医院 Method for realizing user knee joint patella softening state analysis based on decision network

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117437459A (en) * 2023-10-08 2024-01-23 昆山市第一人民医院 Method for realizing user knee joint patella softening state analysis based on decision network
CN117437459B (en) * 2023-10-08 2024-03-22 昆山市第一人民医院 Method for realizing user knee joint patella softening state analysis based on decision network

Similar Documents

Publication Publication Date Title
Khan et al. Human gait analysis for osteoarthritis prediction: A framework of deep learning and kernel extreme learning machine
Che et al. Hybrid multimodal fusion with deep learning for rolling bearing fault diagnosis
Narayan et al. A novel recommender system based on FFT with machine learning for predicting and identifying heart diseases
Chen et al. Automated pain detection from facial expressions using facs: A review
Wu et al. Combining attention-based multiple instance learning and gaussian processes for CT hemorrhage detection
CN117598674B (en) Multi-parameter heart function monitoring system and method
CN116631619A (en) Postoperative leg bending training monitoring system and method thereof
Nicolle et al. Real-time facial action unit intensity prediction with regularized metric learning
Liu et al. Information recovery-driven deep incomplete multiview clustering network
Guo et al. A tree-structure-guided graph convolutional network with contrastive learning for the assessment of parkinsonian hand movements
Pradhan et al. Optimizing CNN‐LSTM hybrid classifier using HCA for biomedical image classification
Rahman et al. [Retracted] Automated Detection of Rehabilitation Exercise by Stroke Patients Using 3‐Layer CNN‐LSTM Model
Mogan et al. Advances in vision-based gait recognition: From handcrafted to deep learning
CN118051861A (en) Patient movement function analysis system and method based on multi-mode data
Gu et al. Cross-domain EEG signal classification via geometric preserving transfer discriminative dictionary learning
Soundrapandiyan et al. AI-based wavelet and stacked deep learning architecture for detecting coronavirus (COVID-19) from chest X-ray images
Das et al. AESPNet: attention enhanced stacked parallel network to improve automatic diabetic foot ulcer identification
Anbalagan et al. Deep learning model using ensemble based approach for walking activity recognition and gait event prediction with grey level co-occurrence matrix
CN116431004B (en) Control method and system for interactive behavior of rehabilitation robot
Mahalakshmi et al. Few-shot learning-based human behavior recognition model
Zuo et al. Unsupervised statistical feature-guided diffusion model for sensor-based human activity recognition
Albert et al. Data augmentation of kinematic time-series from rehabilitation exercises using GANs
Abdulmunem et al. Deep learning based masked face recognition in the era of the COVID-19 pandemic
Xu et al. A deep attention transformer network for pain estimation with facial expression video
Yin et al. Predicting age-related macular degeneration progression with contrastive attention and time-aware lstm

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination