Disclosure of Invention
In order to overcome the defects in the background technology, the embodiment of the invention provides a visual education teaching analysis method, a visual education teaching analysis system and a storage medium, which can effectively solve the problems related to the background technology.
The aim of the invention can be achieved by the following technical scheme: the first aspect of the invention provides a visual education teaching analysis method, which comprises the following steps: s1, collecting hand actions and lip actions of a deaf-mute student: and collecting hand action videos and lip action videos of the deaf-mute student when the deaf-mute student answers the questions each time through the mobile equipment.
S2, analysis of expression text of the deaf-mute students: and respectively analyzing corresponding sign language expression text and lip language expression text of the deaf-mute students when answering the questions for each time based on the collected hand action videos and lip action videos of the deaf-mute students when answering the questions for each time.
S3, analyzing the text matching degree of the expression of the deaf-mute students: and analyzing the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time based on the sign language expression text and the lip language expression text corresponding to the deaf-mute student when answering the questions for each time, and further analyzing the standard expression language corresponding to the deaf-mute student when answering the questions for each time.
S4, analyzing the quality of the answer questions of the deaf-mute students: and evaluating the quality coefficient corresponding to each time of questions answering by the deaf-mute student according to the standard expression language and the standard answer corresponding to each time of questions answering by the deaf-mute student.
In one possible design, the sign language expression text corresponding to each time the deaf-mute student answers the questions comprises the following specific analysis method: s211: extracting hand motion videos corresponding to the sign language expression texts from the cloud database, dividing the hand motion videos corresponding to the sign language expression texts into hand motion sub-pictures according to the preset video frame number, and further obtaining the hand motion sub-pictures corresponding to the sign language expression texts.
S212: and extracting hand contours according to the hand motion sub-pictures corresponding to the sign language expression texts, and acquiring the joint points of the hand contours of the hand motion sub-pictures corresponding to the sign language expression texts.
S213: connecting hand contour lines connected with all joint points of hand contours of all hand motion sub-images corresponding to all sign language expression texts to obtain all relations of hand contours of all hand motion sub-images corresponding to all sign language expression textsThe bending angle of the node is marked as TQ pjx Where p=1, 2,..q, j is denoted as the number of each hand action sub-picture, j=1, 2,..k, x is denoted as the number of each joint point, x=1, 2,..y.
S214: the method comprises the steps of uniformly dividing collected hand action videos of the deaf-mute students when answering questions for each time into hand action videos of each section, dividing each section of hand action video into hand pictures according to set video frame numbers, marking the hand pictures as pictures to be analyzed, and further obtaining pictures to be analyzed, to which each section of hand action videos belong, when the deaf-mute students answer questions for each time.
S215: similarly, the bending angles of all joint points of hand outlines in all pictures to be analyzed, to which all hand action videos belong, of the deaf-mute students when answering questions for each time are obtained, and the bending angles are marked as WQ iamx Where i is a number when each question is answered, i=1, 2,..n, a is a number of each hand action video, a=1, 2,..b, m is a number of each picture to be analyzed, m=1, 2,..l.
S216: comparing the bending angle of each joint point of the hand outline in each picture to be analyzed, which each section of hand action video belongs to when the deaf-mute student answers the questions, with the bending angle of each joint point of the hand outline, which each hand action sub-picture belongs to, corresponding to each sign language expression text, so as to analyze the matching degree of each picture to be analyzed, which each section of hand action video belongs to when the deaf-mute student answers the questions, and each hand action sub-picture corresponding to each sign language expression text, wherein the calculation formula is as follows:wherein->And the matching degree of the mth picture to be analyzed, which belongs to the mth hand motion video, and the jth hand motion sub picture corresponding to the p sign language expression text when the deaf-mute student answers the question for the ith time is expressed, and y is expressed as the number of the articulation points.
S217: each section of hand action video of each deaf-mute student when answering questions each time is subjected to each waitingThe matching degree of each hand action sub-picture corresponding to the analysis picture and each sign language expression text is respectively compared with a preset matching degree threshold value, so that the number of target pictures of each hand action video when the deaf-mute student answers the questions each time is counted, and the target pictures are marked as SL ia 。
S218: counting the total number of hand action sub-pictures corresponding to each sign language expression text, and analyzing the matching coefficient of each hand action video and each sign language expression text of each deaf-mute student when each answer questions by combining the number of target pictures of each hand action video when each deaf-mute student answers the questions, the matching degree of each picture to be analyzed, which each hand action video belongs to, and each hand action sub-picture corresponding to each sign language expression text when each deaf-mute student answers the questions, wherein the calculation formula is as follows: Wherein->The matching coefficient of the a-th hand motion video and the p-th sign language expression text when the i-th question is answered by the deaf-mute student is represented, SS p The total number of the hand action sub-pictures corresponding to the p-th sign language expression text is represented, AD is represented as a preset matching degree threshold value, k is represented as the number of the hand action sub-pictures, and l is represented as the number of the pictures to be analyzed.
S219: based on the matching coefficients of the hand action videos and the sign language expression texts when the deaf-mute students answer the questions each time, the sign language expression text of the maximum matching coefficient corresponding to the hand action videos of the deaf-mute students when the questions are answered each time is screened, and the sign language expression text is used as the sign language expression text of the hand action videos of the deaf-mute students when the questions are answered each time.
S2110: and splicing sign language expression texts of the hand action videos of each section when the deaf-mute student answers the questions each time, so as to obtain the corresponding sign language expression texts when the deaf-mute student answers the questions each time.
In one possible design, the specific analysis method of the lip language corresponding to each time the deaf-mute student answers the questions is as follows: s221: extracting lip action videos corresponding to the lip expression texts from the cloud database, dividing the lip action videos corresponding to the lip expression texts into lip action sub-pictures according to the preset video frame number, and further obtaining the lip action sub-pictures corresponding to the lip expression texts.
S222: and extracting lip outlines from lip action sub-pictures corresponding to the lip expression texts.
S223: the lip action videos acquired when the deaf-mute student answers the questions for each time are uniformly divided into lip action videos of each section, the lip action videos of each section are divided into lip action pictures according to the set video frame number, the lip action pictures are marked as pictures to be analyzed, and then the pictures to be analyzed, to which the lip action videos of each section belong, of the deaf-mute student answers the questions for each time are obtained.
S224: and (3) carrying out equal proportion processing on the lip outline in each picture to be analyzed, to which each section of lip action video belongs, when the deaf-mutes answer the questions each time.
S225: and (3) overlapping and comparing the lip outline in each picture to be analyzed, which each section of lip action video belongs to when the deaf-mute student answers the questions for each time after the equal proportion processing, with the lip outline in each lip action sub-picture corresponding to each lip expression text.
S226: acquiring the overlapping area of the lip outline in each picture to be analyzed, which is attributed to each section of lip action video, and the lip outline in each lip action sub-picture corresponding to each lip expression text when the deaf-mute student answers the questions in each time after the equal proportion processing, and marking the overlapping area as Wherein a is denoted as the number of each lip motion video, a=1, 2,..m, B, M is denoted as the number of each picture to be parsed, m=1, 2,..l, U is denoted as the number of each lip expression text, u=1, 2,..r, J is denoted as the number of each lip motion sub-picture, j=1, 2,..k.
Obtaining lip contour areas in each picture to be analyzed to which each section of lip action video belongs when the deaf-mute students answer questions each time after the equal proportion processing,and marks it as MJ UJ 。
Therefore, the matching coefficient of each section of lip action video and each lip expression text when the deaf-mute student answers the questions each time is analyzed, and the calculation formula is as follows:wherein->The matching coefficient of the A-th section lip motion video and the U-th lip expression text when the deaf-mute student answers the question for the i-th time is represented, and L, K is represented as the number of pictures to be analyzed and the number of lip motion sub-pictures respectively.
S227: based on the matching coefficient of each section of lip motion video and each lip expression text when the deaf-mute student answers the questions each time, the lip expression text of each section of lip motion video corresponding to the largest matching coefficient when the deaf-mute student answers the questions each time is screened, and the lip expression text is used as the lip expression text of each section of lip motion video when the deaf-mute student answers the questions each time.
S228: and splicing lip expression texts of the lip action videos of each section when the deaf-mute student answers the questions each time, so as to obtain the corresponding lip expression texts when the deaf-mute student answers the questions each time.
In one possible design, the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions each time is as follows: s311: constructing a sign language expression text keyword set and a lip language expression text keyword set corresponding to each question answering by the deaf-mute student according to the sign language expression text and the lip language expression text when the deaf-mute student answers the questions each time, and marking the sign language expression text keyword set and the lip language expression text keyword set as E respectively i 、F i 。
S312: matching and comparing the corresponding sign language expression text keyword set and lip language expression text keyword set when the deaf-mutes answer questions for each time, and analyzing the matching degree of the sign language expression text and the lip language expression text when the deaf-mutes answer questions for each time according to the matching degree, wherein the calculation formula is as follows:wherein->The matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the question for the ith time is expressed.
In one possible design, the analysis method for the standard expression language corresponding to each time the deaf-mute student answers the questions is as follows: s321: comparing the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time with a preset threshold value of the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time, and taking the sign language expression text when the deaf-mute student answers the questions for each time as a corresponding standard expression language when the deaf-mute student answers the questions for each time if the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time is greater than or equal to the threshold value of the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time.
S322: if the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for a certain time is smaller than the matching degree threshold of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions, the following analysis is carried out: the method comprises the steps of obtaining the matching coefficients of hand action videos and sign language expression texts of all segments when a deaf-mutes student answers questions at the time, and the matching coefficients of lip action videos and lip language expression texts of all segments, analyzing the average matching coefficients of hand action videos and sign language expression texts and the average matching coefficients of lip action videos and lip language expression texts of the deaf-mutes student at the time according to the matching coefficients, and obtaining the corresponding standard expression language of the deaf-mutes student at the time when the deaf-mutes student answers questions at the time.
S323: and obtaining a standard expression language corresponding to each time of answering the questions by the deaf-mute students.
In one possible design, the specific analysis method of the quality coefficient corresponding to each time the deaf-mute student answers the questions is as follows: according to the standard expression language corresponding to each time of questions answering of deaf-mute students and each time of returns stored in the cloud databaseStandard answers when answering questions construct a standard expression language keyword set corresponding to each time the deaf-mute student answers questions and a standard answer keyword set when each time answers questions, and mark the keyword sets as QA respectively i 、DA i And further analyzing the corresponding quality coefficient of the deaf-mute students when answering the questions each time, wherein the calculation formula is as follows:wherein ZL i Expressed as the corresponding quality coefficient when the deaf-mute student answers the question for the ith time.
A second aspect of the present invention provides a visual educational instruction analysis system comprising: the deaf-mute student hand action and lip action acquisition module comprises: and collecting hand action videos and lip action videos of the deaf-mute student when the deaf-mute student answers the questions each time through the mobile equipment.
Deaf-mute student expression text analysis module: and respectively analyzing corresponding sign language expression text and lip language expression text of the deaf-mute students when answering the questions for each time based on the collected hand action videos and lip action videos of the deaf-mute students when answering the questions for each time.
The deaf-mute student expresses the text matching degree analysis module: and analyzing the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time based on the sign language expression text and the lip language expression text corresponding to the deaf-mute student when answering the questions for each time, and further analyzing the standard expression language corresponding to the deaf-mute student when answering the questions for each time.
The quality analysis module of the questions answered by the deaf-mute students: and evaluating the quality coefficient corresponding to each time of questions answering by the deaf-mute student according to the standard expression language and the standard answer corresponding to each time of questions answering by the deaf-mute student.
Cloud database: and storing hand action videos corresponding to the sign language expression texts, storing lip action videos corresponding to the lip language expression texts, and storing standard answers when answering the questions for each time.
The third aspect of the invention provides a visual education teaching analysis storage medium, wherein the storage medium is burnt with a computer program, and the computer program realizes the visual education teaching analysis method when running in the memory of a server.
Compared with the prior art, the embodiment of the invention has at least the following advantages or beneficial effects: (1) According to the quality analysis of the deaf-mute student answer questions, the sign language expression text of the deaf-mute student is analyzed, the lip language expression text of the deaf-mute student is also analyzed, the defect of single analysis dimension of the center in the prior art is overcome, the similarity of the sign language expression text and the lip language expression text of the deaf-mute student is further analyzed through analyzing the sign language expression text and the lip language expression text of the deaf-mute student, the phenomenon that the sign language expression text of the deaf-mute student is inaccurate or the lip language expression text is inaccurate can be effectively avoided, the error rate of taking the single sign language expression text or the single lip language expression text of the deaf-mute student as the expression language of the deaf-mute student is further reduced, the reliability of the quality analysis of the deaf-mute student answer questions is improved, and in conclusion, the quality analysis method can provide powerful data support for the quality analysis of the deaf-mute student answer questions.
(2) When analyzing the sign language expression text of the deaf-mute student, the bending angle of the hand joint points of the deaf-mute student is analyzed, so that the phenomenon that the sign language expression text analysis of the deaf-mute student is inaccurate due to inconsistent hand size, shape and fat and thin of the deaf-mute student is avoided, the accuracy of the sign language expression text analysis result of the deaf-mute student is improved, the problem that the sign language expression text analysis of the deaf-mute student is not strong in pertinence in the prior art is solved, a solid foundation is laid for the subsequent analysis of the sign language expression text analysis result of the deaf-mute student, and the accuracy of the answer question quality analysis result of the deaf-mute student is improved.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, a first aspect of the present invention provides a visual education teaching analysis method including the steps of: s1, collecting hand actions and lip actions of a deaf-mute student: and collecting hand action videos and lip action videos of the deaf-mute student when the deaf-mute student answers the questions each time through the mobile equipment.
S2, analysis of expression text of the deaf-mute students: and respectively analyzing corresponding sign language expression text and lip language expression text of the deaf-mute students when answering the questions for each time based on the collected hand action videos and lip action videos of the deaf-mute students when answering the questions for each time.
In a specific embodiment of the invention, the sign language expression text corresponding to each time the deaf-mute student answers the questions comprises the following specific analysis method: s211: extracting hand motion videos corresponding to the sign language expression texts from the cloud database, dividing the hand motion videos corresponding to the sign language expression texts into hand motion sub-pictures according to the preset video frame number, and further obtaining the hand motion sub-pictures corresponding to the sign language expression texts.
S212: and extracting hand contours according to the hand motion sub-pictures corresponding to the sign language expression texts, and acquiring the joint points of the hand contours of the hand motion sub-pictures corresponding to the sign language expression texts.
S213: connecting hand contour lines connected with all joint points of hand contours of all hand motion sub-images corresponding to all sign language expression texts to further obtain all joints of hand contours of all hand motion sub-images corresponding to all sign language expression textsThe bending angle of the dot is marked as TQ pjx Where p=1, 2,..q, j is denoted as the number of each hand action sub-picture, j=1, 2,..k, x is denoted as the number of each joint point, x=1, 2,..y.
S214: the method comprises the steps of uniformly dividing collected hand action videos of the deaf-mute students when answering questions for each time into hand action videos of each section, dividing each section of hand action video into hand pictures according to set video frame numbers, marking the hand pictures as pictures to be analyzed, and further obtaining pictures to be analyzed, to which each section of hand action videos belong, when the deaf-mute students answer questions for each time.
S215: similarly, the bending angles of all joint points of hand outlines in all pictures to be analyzed, to which all hand action videos belong, of the deaf-mute students when answering questions for each time are obtained, and the bending angles are marked as WQ iamx Where i is a number when each question is answered, i=1, 2,..n, a is a number of each hand action video, a=1, 2,..b, m is a number of each picture to be analyzed, m=1, 2,..l.
S216: comparing the bending angle of each joint point of the hand outline in each picture to be analyzed, which each section of hand action video belongs to when the deaf-mute student answers the questions, with the bending angle of each joint point of the hand outline, which each hand action sub-picture belongs to, corresponding to each sign language expression text, so as to analyze the matching degree of each picture to be analyzed, which each section of hand action video belongs to when the deaf-mute student answers the questions, and each hand action sub-picture corresponding to each sign language expression text, wherein the calculation formula is as follows:wherein->And the matching degree of the mth picture to be analyzed, which belongs to the mth hand motion video, and the jth hand motion sub picture corresponding to the p sign language expression text when the deaf-mute student answers the question for the ith time is expressed, and y is expressed as the number of the articulation points.
S217: dividing each waiting point of each hand action video when each deaf-mute student answers questionsComparing the matching degree of each hand action sub-picture corresponding to the analysis picture and each sign language expression text with a preset matching degree threshold value respectively, further counting the number of target pictures to which each segment of hand action video belongs when the deaf-mute student answers the questions each time, and marking the target pictures as SL ia 。
It should be noted that if the matching degree of a picture to be analyzed, to which a certain hand action video belongs, and a hand action sub-picture corresponding to a sign language when a deaf-mute student answers a question for a certain time is greater than or equal to the matching degree threshold of the picture to be analyzed, to which the hand action video belongs, and the hand sub-picture corresponding to the sign language when the deaf-mute student answers the question for a certain time, the picture to be analyzed, to which the hand action video belongs, is marked as a target picture when the deaf-mute student answers the question for the certain time.
S218: counting the total number of hand action sub-pictures corresponding to each sign language expression text, and analyzing the matching coefficient of each hand action video and each sign language expression text of each deaf-mute student when each answer questions by combining the number of target pictures of each hand action video when each deaf-mute student answers the questions, the matching degree of each picture to be analyzed, which each hand action video belongs to, and each hand action sub-picture corresponding to each sign language expression text when each deaf-mute student answers the questions, wherein the calculation formula is as follows:wherein->The matching coefficient of the a-th hand motion video and the p-th sign language expression text when the i-th question is answered by the deaf-mute student is represented, SS p The total number of the hand action sub-pictures corresponding to the p-th sign language expression text is represented, AD is represented as a preset matching degree threshold value, k is represented as the number of the hand action sub-pictures, and l is represented as the number of the pictures to be analyzed.
S219: based on the matching coefficients of the hand action videos and the sign language expression texts when the deaf-mute students answer the questions each time, the sign language expression text of the maximum matching coefficient corresponding to the hand action videos of the deaf-mute students when the questions are answered each time is screened, and the sign language expression text is used as the sign language expression text of the hand action videos of the deaf-mute students when the questions are answered each time.
S2110: and splicing sign language expression texts of the hand action videos of each section when the deaf-mute student answers the questions each time, so as to obtain the corresponding sign language expression texts when the deaf-mute student answers the questions each time.
When analyzing the sign language expression text of the deaf-mute student, the bending angle of the hand joint points of the deaf-mute student is analyzed, so that the phenomenon that the sign language expression text analysis of the deaf-mute student is inaccurate due to inconsistent hand size, shape and fat and thin of the deaf-mute student is avoided, the accuracy of the sign language expression text analysis result of the deaf-mute student is improved, the problem that the sign language expression text analysis of the deaf-mute student is not strong in pertinence in the prior art is solved, a solid foundation is laid for the subsequent analysis of the sign language expression text analysis result of the deaf-mute student, and the accuracy of the answer question quality analysis result of the deaf-mute student is improved.
In a specific embodiment of the invention, the lip expression text corresponding to each time the deaf-mute student answers the questions comprises the following specific analysis method: s221: extracting lip action videos corresponding to the lip expression texts from the cloud database, dividing the lip action videos corresponding to the lip expression texts into lip action sub-pictures according to the preset video frame number, and further obtaining the lip action sub-pictures corresponding to the lip expression texts.
S222: and extracting lip outlines from lip action sub-pictures corresponding to the lip expression texts.
S223: the lip action videos acquired when the deaf-mute student answers the questions for each time are uniformly divided into lip action videos of each section, the lip action videos of each section are divided into lip action pictures according to the set video frame number, the lip action pictures are marked as pictures to be analyzed, and then the pictures to be analyzed, to which the lip action videos of each section belong, of the deaf-mute student answers the questions for each time are obtained.
S224: and (3) carrying out equal proportion processing on the lip outline in each picture to be analyzed, to which each section of lip action video belongs, when the deaf-mutes answer the questions each time.
The specific method for carrying out the equal proportion processing on the lip outline in each picture to be analyzed, which each section of lip action video belongs to when the deaf-mute student answers the questions each time, is as follows: extracting lip contours from each picture to be analyzed, which each section of lip action video belongs to, when a deaf-mute student answers questions each time, comparing the lip contours in each picture to be analyzed, which each section of lip action video belongs to, with the lip contours in each lip action sub-picture corresponding to each lip language, if the lip contours in a picture to be analyzed, which each section of lip action video belongs to, when a deaf-mute student answers questions each time, are larger than the lip contours in a picture to be analyzed, which corresponds to a lip language, carrying out equal proportion reduction processing on the lip contours in the picture to be analyzed, which each section of lip action video belongs to, when a deaf-mute student answers questions each time, if the lip contours in a picture to be analyzed, which each section of lip action video belongs to, are smaller than the lip contours in a picture to be analyzed, which each section of lip action sub-picture corresponds to, and carrying out equal proportion amplification processing on the lip contours in the picture to be analyzed, which each section of lip action video belongs to, when a deaf-mute student answers questions each time.
S225: and (3) overlapping and comparing the lip outline in each picture to be analyzed, which each section of lip action video belongs to when the deaf-mute student answers the questions for each time after the equal proportion processing, with the lip outline in each lip action sub-picture corresponding to each lip expression text.
S226: acquiring the overlapping area of the lip outline in each picture to be analyzed, which is attributed to each section of lip action video, and the lip outline in each lip action sub-picture corresponding to each lip expression text when the deaf-mute student answers the questions in each time after the equal proportion processing, and marking the overlapping area asWherein a is denoted as the number of each lip motion video, a=1, 2,..m, B, M is denoted as the number of each picture to be parsed, m=1, 2,..l, U is denoted as the number of each lip expression text, u=1, 2,..r, J is denoted as the number of each lip motion sub-picture, j=1, 2,..k.
Obtaining lip contour areas in each picture to be analyzed, to which each section of lip action video belongs, of each section of lip action video when each deaf-mute student answers questions after equal proportion processing, and marking the lip contour areas as MJ UJ 。
Therefore, the matching coefficient of each section of lip action video and each lip expression text when the deaf-mute student answers the questions each time is analyzed, and the calculation formula is as follows:wherein->The matching coefficient of the A-th section lip motion video and the U-th lip expression text when the deaf-mute student answers the question for the i-th time is represented, and L, K is represented as the number of pictures to be analyzed and the number of lip motion sub-pictures respectively.
S227: based on the matching coefficient of each section of lip motion video and each lip expression text when the deaf-mute student answers the questions each time, the lip expression text of each section of lip motion video corresponding to the largest matching coefficient when the deaf-mute student answers the questions each time is screened, and the lip expression text is used as the lip expression text of each section of lip motion video when the deaf-mute student answers the questions each time.
S228: and splicing lip expression texts of the lip action videos of each section when the deaf-mute student answers the questions each time, so as to obtain the corresponding lip expression texts when the deaf-mute student answers the questions each time.
According to the method, the phenomenon that the sign language of the deaf-mute student is inaccurate or the lip language is inaccurate can be effectively avoided, the error rate of taking the single sign language or the single lip language of the deaf-mute student as the expression language of the deaf-mute student is reduced, the reliability of analyzing the quality of the questions answered by the deaf-mute student is improved, and the method can provide powerful data support for the quality analysis of the questions answered by the deaf-mute student.
S3, analyzing the text matching degree of the expression of the deaf-mute students: and analyzing the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time based on the sign language expression text and the lip language expression text corresponding to the deaf-mute student when answering the questions for each time, and further analyzing the standard expression language corresponding to the deaf-mute student when answering the questions for each time.
In a specific embodiment of the invention, the matching degree of the sign language expression text and the lip language expression text when the deaf-mute students answer the questions each time is as follows: s311: constructing a sign language expression text keyword set and a lip language expression text keyword set corresponding to each question answering by the deaf-mute student according to the sign language expression text and the lip language expression text when the deaf-mute student answers the questions each time, and marking the sign language expression text keyword set and the lip language expression text keyword set as E respectively i 、F i 。
S312: matching and comparing the corresponding sign language expression text keyword set and lip language expression text keyword set when the deaf-mutes answer questions for each time, and analyzing the matching degree of the sign language expression text and the lip language expression text when the deaf-mutes answer questions for each time according to the matching degree, wherein the calculation formula is as follows:wherein->The matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the question for the ith time is expressed.
In a specific embodiment of the invention, the method for analyzing the standard expression language corresponding to each time of answering the questions by the deaf-mute students comprises the following steps: the specific analysis method for analyzing the standard expression language corresponding to each time of answering the questions by the deaf-mute students comprises the following steps: s321: comparing the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time with a preset threshold value of the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time, and taking the sign language expression text when the deaf-mute student answers the questions for each time as a corresponding standard expression language when the deaf-mute student answers the questions for each time if the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time is greater than or equal to the threshold value of the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time.
S322: if the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for a certain time is smaller than the matching degree threshold of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions, the following analysis is carried out: the method comprises the steps of obtaining the matching coefficients of hand action videos and sign language expression texts of all segments when a deaf-mutes student answers questions at the time, and the matching coefficients of lip action videos and lip language expression texts of all segments, analyzing the average matching coefficients of hand action videos and sign language expression texts and the average matching coefficients of lip action videos and lip language expression texts of the deaf-mutes student at the time according to the matching coefficients, and obtaining the corresponding standard expression language of the deaf-mutes student at the time when the deaf-mutes student answers questions at the time.
It should be noted that, the specific calculation formula of the average matching coefficient of the hand action video and the sign language expression text when the deaf-mute student answers the question this time is as follows:wherein PJ is expressed as the average matching coefficient of hand motion video and sign language expression text when the deaf-mute student answers the question this time,>the matching coefficients of the a-th hand motion video and the p-th sign language expression text are expressed when the deaf-mute student answers the question, and b and q are respectively expressed as the number of the hand motion videos and the number of the sign language expression texts.
It should be noted that, the specific calculation formula of the average matching coefficient of the lip action video and the lip expression text when the deaf-mute student answers the question this time is as follows:wherein PA is expressed as lip action when the deaf-mute student answers the questionAverage matching coefficient of video and lip expression text, < ->The matching coefficients of the A-th hand motion video and the R-th sign language expression text when the deaf-mute students answer the questions are expressed, and B, R are expressed as the number of lip motion videos and the number of lip language expression texts respectively.
It should be noted that, the specific analysis method of the corresponding standard expression language when the deaf-mute student answers the question this time is as follows: comparing the average matching coefficient of the hand motion video and the sign language expression text when the deaf-mute student answers the questions for the time with the average matching coefficient of the lip motion video and the lip language expression text as far as possible, and if the average matching coefficient of the hand motion video and the sign language expression text when the deaf-mute student answers the questions for the time is larger than or equal to the average matching coefficient of the lip motion video and the lip language expression text, taking the sign language expression text when the deaf-mute student answers the questions for the time as the standard expression language corresponding to the questions for the time, otherwise, taking the lip language expression text when the deaf-mute student answers the questions for the time as the standard expression language corresponding to the questions for the time.
S323: and obtaining a standard expression language corresponding to each time of answering the questions by the deaf-mute students.
S4, analyzing the quality of the answer questions of the deaf-mute students: and evaluating the quality coefficient corresponding to each time of questions answering by the deaf-mute student according to the standard expression language and the standard answer corresponding to each time of questions answering by the deaf-mute student.
In a specific embodiment of the invention, the specific analysis method of the quality coefficient corresponding to each time of answering the questions by the deaf-mute student comprises the following steps: constructing a standard expression language keyword set corresponding to each question answering time of the deaf-mute student and a standard answer keyword set corresponding to each question answering time of the deaf-mute student according to the standard expression language corresponding to each question answering time of the deaf-mute student and standard answers stored in a cloud database, and marking the standard expression language keyword set and the standard answer keyword set as QA (quality assurance) respectively i 、DA i And further analyzing the corresponding quality coefficient of the deaf-mute students when answering the questions each time, wherein the calculation formula is as follows:wherein ZL i Expressed as the corresponding quality coefficient when the deaf-mute student answers the question for the ith time.
Referring to fig. 2, a second aspect of the present invention provides a visual education teaching analysis system including: the deaf-mute student hand action and lip action acquisition module comprises: and collecting hand action videos and lip action videos of the deaf-mute student when the deaf-mute student answers the questions each time through the mobile equipment.
Deaf-mute student expression text analysis module: and respectively analyzing corresponding sign language expression text and lip language expression text of the deaf-mute students when answering the questions for each time based on the collected hand action videos and lip action videos of the deaf-mute students when answering the questions for each time.
The deaf-mute student expresses the text matching degree analysis module: and analyzing the matching degree of the sign language expression text and the lip language expression text when the deaf-mute student answers the questions for each time based on the sign language expression text and the lip language expression text corresponding to the deaf-mute student when answering the questions for each time, and further analyzing the standard expression language corresponding to the deaf-mute student when answering the questions for each time.
The quality analysis module of the questions answered by the deaf-mute students: and evaluating the quality coefficient corresponding to each time of questions answering by the deaf-mute student according to the standard expression language and the standard answer corresponding to each time of questions answering by the deaf-mute student.
Cloud database: and storing hand action videos corresponding to the sign language expression texts, storing lip action videos corresponding to the lip language expression texts, and storing standard answers when answering the questions for each time.
The system comprises a deaf-mute student hand action and lip action acquisition module, a deaf-mute student expression text analysis module, a deaf-mute student expression text matching degree analysis module, a deaf-mute student answer question quality analysis module and a cloud database, wherein the deaf-mute student hand action and lip action acquisition module is connected with the deaf-mute student expression text analysis module, the deaf-mute student expression text analysis module is connected with the deaf-mute student expression text matching degree analysis module, the deaf-mute student answer question quality analysis module is connected with the deaf-mute student answer question quality analysis module, and the cloud database is respectively connected with the deaf-mute student expression text analysis module and the deaf-mute student answer question quality analysis module.
The third aspect of the invention provides a visual education teaching analysis storage medium, wherein the storage medium is burnt with a computer program, and the computer program realizes the visual education teaching analysis method when running in the memory of a server.
The foregoing is merely illustrative of the structures of this invention and various modifications, additions and substitutions for those skilled in the art can be made to the described embodiments without departing from the scope of the invention or from the scope of the invention as defined in the accompanying claims.