CN103151042B - Full-automatic oral evaluation management and points-scoring system and methods of marking thereof - Google Patents
Full-automatic oral evaluation management and points-scoring system and methods of marking thereof Download PDFInfo
- Publication number
- CN103151042B CN103151042B CN201310034371.7A CN201310034371A CN103151042B CN 103151042 B CN103151042 B CN 103151042B CN 201310034371 A CN201310034371 A CN 201310034371A CN 103151042 B CN103151042 B CN 103151042B
- Authority
- CN
- China
- Prior art keywords
- feature
- examinee
- model
- answer
- recognition result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Machine Translation (AREA)
Abstract
The present invention relates to Computer Assisted Instruction (CAI) field, a kind of full-automatic oral evaluation management and points-scoring system are provided, it comprise connect successively client, land server and the webserver, described examinee result of answering comprises one in the spoken evaluating result of the spoken evaluating result of reading aloud topic and spontaneous spoken statement topic, the described server that lands also comprises scoring apparatus, this scoring apparatus comprises identification module and grading module, and described identification module comprises acoustics submodule, language submodule and recognin module; Institute's scoring module comprises feature extraction submodule and scoring submodule.The present invention also provides a kind of full-automatic oral evaluation management and methods of marking.Full-automatic oral evaluation management of the present invention and points-scoring system and methods of marking thereof can completely by computer controlled automatic examinee preparation Reaction time with answer the time, without the need to manual intervention, and more more accurate than manual time-keeping, further ensure that the fairness of evaluation and test.
Description
Technical field
The present invention relates to Computer Assisted Instruction (CAI) field, particularly relate to a kind of full-automatic oral evaluation management and points-scoring system and methods of marking thereof.
Background technology
Along with popularizing of China's English teaching, the problem of Dumb English is also more and more serious, and therefore the teaching of Oral English Practice also comes into one's own day by day with evaluation and test.The oral evaluation of English is carried out in multiple provinces and cities of China, and what wherein Guangdong Province had brought common college entrance examination into spoken English evaluating must examine scope.Due to the singularity of oral evaluation mode and content, the scoring of current Oral English Practice mainly or by the mode of manually appraising solves.Artificial scoring has a lot of very formidable shortcoming:
Teacher and the face-to-face oral evaluation of student lack fairness and accuracy.Different teacher is difficult to ensure completely the same for the scoring of same examinee, same teacher is also difficult to ensure completely the same for the standards of grading of different examinee, everything all will reduce the fairness of evaluation and test greatly.Native system formulates a set of just unified machine automatic scoring standard that everybody approves, thus can avoid the diversity of values that causes because of artificial phoneme, ensure that the fairness of evaluation and test.
Artificial scoring oral evaluation scoring efficiency is very low.The a set of full-automatic oral evaluation management of the present invention's design and points-scoring system, artificial scoring is replaced with machine scoring, from the distribution of evaluation and test paper, to beginning and the end of evaluation and test, to the preservation of examinee's answer audio frequency, arrive automatic scoring more all automatically to be completed by machine, scoring efficiency improves greatly.
Traditional oral evaluation organization and administration are very complicated.Traditional Human To Human's oral evaluation, owing to adopting the form of interview, needs in a large number through the spoken examiner of professional training; Meanwhile, every evaluation and test can only be checked and rated several students, if will to annual level even the study English student of course of whole school carry out an oral evaluation, not only time-consuming effort also exists to leak out the examination questions and lets out the risk of topic.The a set of full-automatic oral evaluation management of the present invention's design is with points-scoring system can for annual level, even whole school student carries out once justice, safety, efficiently Oral English Practice level are evaluated and tested simultaneously simultaneously, and can greatly use manpower and material resources sparingly, that also can avoid bringing because of evaluating and testing in batches lets out topic risk.
Tradition oral evaluation needs manual control to evaluate and test the time starting and terminate, and change exercise question, even the little topic of per pass all needs reclocking, bothers very much at every turn.
Summary of the invention
The present invention is for solving the problems of the technologies described above, there is provided a kind of without the need to manual intervention, timing is more accurate and a step guarantees full-automatic oral evaluation management and the points-scoring system of evaluating and testing fairness, it comprises the client connected successively, land server and the webserver, wherein, the arrangement of webserver primary responsibility evaluation result, the distribution of collection and paper, land server primary responsibility machine automatic scoring, client primary responsibility is evaluated and tested, paper is distributed to client from the webserver by landing server, examinee's result of answering uploads to the webserver from client by landing server, described examinee result of answering comprises one in the spoken evaluating result of the spoken evaluating result of reading aloud topic and spontaneous spoken statement topic, the described server that lands also comprises scoring apparatus, this scoring apparatus comprises identification module and grading module, described identification module comprises acoustics submodule, language submodule and recognin module, described acoustics submodule extracts the answer acoustic feature of audio frequency of examinee and obtains acoustic model, described language submodule obtains language model according to topic information and training text, described recognin module is carried out decoding by acoustic model and language model obtain recognition result to examinee's audio frequency of answering, institute's scoring module comprises feature extraction submodule and scoring submodule, described feature extraction submodule is for extracting the flow comprehensive characteristics in described recognition result, described flow comprehensive characteristics comprises the feature in the feature in pronouncing accuracy direction in spoken test and appraisal, the feature in fluency direction and text semantic similarity direction, described scoring submodule carries out scoring training to flow comprehensive characteristics, obtain Rating Model, and according to Rating Model, recognition result is marked.
Preferably, the feature in described pronouncing accuracy direction is obtained by following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
Recognition result and correct text are carried out pressure align, calculate the pressure alignment score of each phoneme;
Build single-tone element decoded model and each phoneme of decoding, calculate the maximum likelihood score of each phoneme;
Utilize the feature of forcing the difference of alignment score and maximum likelihood score to obtain pronouncing accuracy direction.
Preferably, the feature in described fluency direction comprises word speed feature and the duration characteristics that pauses in short-term, and described word speed feature is obtained by following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
The frame number that in voice identification result, each phoneme is corresponding is counted according to recognition result;
The ratio of the duration of the total number of phoneme and all phonemes is utilized to obtain word speed feature;
The described duration characteristics of pause is in short-term obtained by following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
Frame number that in voice identification result, each phoneme is corresponding and the total frame number of audio frequency is counted according to recognition result;
The summation of the duration utilizing all phonemes to pause in short-term and the ratio of total pronunciation duration are paused duration characteristics in short-term.
Preferably, the feature in described text semantic similarity direction comprises semantic relevancy feature and syntactic structure similarity feature.
Preferably, described semantic relevancy feature comprises the following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
Calculate the semantic similarity score of each word in each word in recognition result and model answer;
Calculate the semantic similarity score of each sentence in each word in recognition result and model answer;
Calculate semantic similarity score maximal value in recognition result in each word and model answer in each sentence or mean value as the similarity score between word and sentence;
Calculate the similarity score between examinee's answer and model answer.
Preferably, described syntactic structure similarity feature comprises the following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
Syntax sequence vector set up in each sentence being respectively recognition result;
Obtain the syntactic structure similarity score of each sentence in recognition result and each sentence in model answer respectively, to get in recognition result each Sentence Grammar structural similarity score maximal value as the syntactic structure similarity score of this sentence;
By to the syntactic structure similarity feature in recognition result between each the weighted average calculation examinee's answer of Sentence Grammar structural similarity score and model answer.
Preferably, the webserver comprises scheduler module, for dispatching evaluation and test information landing between server and the webserver; Adopt networking evaluation and test pattern, the communication of server of landing between the examination hall making to be distributed in different location is managed by described webserver scheduler module United Dispatching.
Preferably, described system comprises the role of three kinds of different rights: examinee, teacher and keeper, and examinee's primary responsibility is evaluated and tested and answered; Teacher's primary responsibility system volume, issue are evaluated and tested, manage evaluation and test, check evaluation result and the work of scoring, the method combined that scoring aspect employing system is marked and teacher marks; The management of keeper's primary responsibility evaluation and test and the time of welltesting software are controlled.
Preferably, role playing is inscribed to the Key for Reference that the little topic of per pass can be utilized the to have provided referenced text as text semantic similarity analysis, and then expand other Key for Reference; The transcription text of audio content can be utilized as the referenced text of text semantic similarity analysis for repetition topic, and then expand other Key for Reference.
The present invention provides a kind of full-automatic oral evaluation management and methods of marking in addition, and it comprises following several step:
A0, choose the process that some examinees carry out as described in steps A 1 ~ A5, automatic scoring model training is carried out in combination of then described flow comprehensive characteristics and teacher being marked, and forms Rating Model;
A1, collect examinee and to answer audio frequency;
A2, extract the answer acoustic feature of audio frequency of examinee and obtain acoustic model, and obtain language model according to topic information and training text;
A3, according to the acoustic model set up and language model, examinee's audio frequency of answering is carried out to decoding and obtains recognition result;
A4, the flow comprehensive characteristics extracted in recognition result, described flow comprehensive characteristics comprises the feature in the feature in pronouncing accuracy direction in spoken test and appraisal, the feature in fluency direction and text semantic similarity direction;
A5, according to flow comprehensive characteristics formed Rating Model carry out automatic scoring.
Full-automatic oral evaluation management of the present invention and points-scoring system and methods of marking thereof: inscribe the feature spoken basis of testing and assessing adding text semantic similarity direction in traditional reading aloud, thus can test and assess for spontaneous spoken statement; Devise unique high in the clouds evaluating system framework, guarantee that evaluation and test is efficiently carried out, take full advantage of the resource of whole system simultaneously, greatly improve and organize oral evaluation efficiency, saved a large amount of manpower and materials; Oral evaluation can be organized on a large scale simultaneously, meet within the scope of provinces and cities even nationwide and organize the demand of oral evaluation simultaneously; The form of oral evaluation is also by diversification more, more comprehensively just to the investigation of examinee's spoken language proficiency.
Accompanying drawing explanation
Fig. 1 is the full-automatic oral evaluation management of one embodiment of the invention and methods of marking schematic flow sheet;
Fig. 2 is the full-automatic oral evaluation management of one embodiment of the invention and points-scoring system configuration diagram;
Fig. 3 is one embodiment of the invention semantic relevancy feature extraction schematic flow sheet;
Fig. 4 is another embodiment of the present invention semantic relevancy feature extraction schematic flow sheet;
Fig. 5 is one embodiment of the invention syntactic structure similarity feature extraction process flow diagram.
Embodiment
Below in conjunction with accompanying drawing and specific embodiment, the present invention is described in further detail.
Embodiment:
The invention provides a kind of full-automatic oral evaluation management and points-scoring system, it comprises the client connected successively, land server and the webserver, wherein, the arrangement of webserver primary responsibility evaluation result, the distribution of collection and paper, land server primary responsibility machine automatic scoring, client primary responsibility is evaluated and tested, paper is distributed to client from the webserver by landing server, examinee's result of answering uploads to the webserver from client by landing server, described examinee result of answering comprises one in the spoken evaluating result of the spoken evaluating result of reading aloud topic and spontaneous spoken statement topic, the described server that lands also comprises scoring apparatus, this scoring apparatus comprises identification module and grading module, described identification module comprises acoustics submodule, language submodule and recognin module, described acoustics submodule extracts the answer acoustic feature of audio frequency of examinee and obtains acoustic model, described language submodule obtains language model according to topic information and training text, described recognin module is carried out decoding by acoustic model and language model obtain recognition result to examinee's audio frequency of answering, institute's scoring module comprises feature extraction submodule and scoring submodule, described feature extraction submodule is for extracting the flow comprehensive characteristics in described recognition result, described flow comprehensive characteristics comprises the feature in the feature in pronouncing accuracy direction in spoken test and appraisal, the feature in fluency direction and text semantic similarity direction, described scoring submodule carries out scoring training to flow comprehensive characteristics, obtain Rating Model, and according to Rating Model, recognition result is marked.
As shown in Figure 1, be the full-automatic oral evaluation management of one embodiment of the invention and methods of marking schematic flow sheet, it comprises following several step:
A0, choose the process that some examinees carry out as described in steps A 1 ~ A5, automatic scoring model training is carried out in combination of then described flow comprehensive characteristics and teacher being marked, and forms Rating Model;
A1, collect examinee and to answer audio frequency;
A2, extract the answer acoustic feature of audio frequency of examinee and obtain acoustic model, and obtain language model according to topic information and training text;
A3, according to the acoustic model set up and language model, examinee's audio frequency of answering is carried out to decoding and obtains recognition result;
A4, the flow comprehensive characteristics extracted in recognition result;
A5, according to flow comprehensive characteristics formed Rating Model carry out automatic scoring.
In the above-described embodiments, described language model comprises one or more of topic related text, written related text and colloquial style text.So-called topic related text refers to determines a topic, and model answer can be seen to examinee or not see to examinee, the text formed according to topic and model answer by examinee when evaluating and testing; So-called written related text refers to the speech text that examinee is formed according to model answer; So-called colloquial style text refers to not with reference to any model answer, the speech text formed freely is played completely by examinee, described teacher scoring refers to engages the teacher of specialty to mark to examinee's audio frequency of answering, carry out two commenting, namely with two teachers that mark examination papers to the final result being equally divided into this examinee of achieving the result, if the achievement that two teachers that mark examination papers provide differs more than 3 points, three are taked to comment mode.
As shown in Figure 2, be the full-automatic oral evaluation management of one embodiment of the invention and points-scoring system configuration diagram, it is mainly divided into following three parts:
Client: computer, panel computer, high-end smartphones all can as clients of the present invention, and client must possess independently audio frequency and video playing function and independently audio input device.Described client comprises evaluation and test module and release module, described evaluation and test module is used for examinee and carries out oral evaluation, comprise examination question issue, evaluation and test, rolling etc., and process and transmission examinee answer audio frequency to landing server, to answer audio frequency for examinee, evaluation and test module is also provided with first processing module for the format conversion of audio frequency of answering to examinee and feature extraction.After evaluation and test terminates, the evaluation and test achievement of examinee also will be issued on the client by release module.
Land server: the server that lands refers to the high-performance computer being arranged on examination hall, the server that lands comprises communication module, acquisition module and grading module, described communication module is used for being received in the evaluation and test information transmission of the webserver to client, it provides paper to client at special time and controls the evaluation and test time, described acquisition module is for audio frequency of answering from client collection examinee, institute's scoring module is used for identifying examinee's test paper, decode, marking, and after having marked, evaluation result is fed back to client in time by communication module.According to examination hall scale and calculation task amount, the server that lands can select multiple stage high-performance computer to set up the form of computer cluster, to accelerate scoring and the speed of decoding.
The webserver: webserver primary responsibility centralized control is dispersed in the server that lands in each examination hall, it comprises scheduler module, analysis module and enquiry module, the webserver comprises scheduler module, for dispatching evaluation and test information landing between server and the webserver, adopt networking evaluation and test pattern, the communication of server of landing between the examination hall making to be distributed in different location is managed by described webserver scheduler module United Dispatching, unified welltesting software is landed server control the time that its evaluation and test starts and terminate to each, simultaneously the object done like this is convenient to organize oral evaluation on a large scale, so often organize and once evaluate and test, just without the need to preparing a examination paper for each examinee in advance, examination paper can be distributed to each and land server by the webserver, examination paper can be distributed to each client by the server that lands again, significantly reduce the risk of letting out topic, described analysis module terminates to do concentrated analysis and treament to answer information and its scoring event of examinee afterwards for evaluating and testing, by Information Statistics such as examinee's total score, individual event score and ranks out, described enquiry module inquires about the information such as examinee's total score, individual event score and rank at any time for academics and students.
Described system comprises the role of three kinds of different rights: examinee, teacher and keeper, and examinee's primary responsibility is evaluated and tested and answered; Teacher's primary responsibility system volume, issue are evaluated and tested, manage evaluation and test, check evaluation result and the work of scoring, the method combined that scoring aspect employing system is marked and teacher marks; The time control of the management that keeper's primary responsibility is evaluated and tested, welltesting software and the maintenance of overall evaluating system.
In the present embodiment, identification module described in scoring apparatus adopts the decode system based on extensive continuous speech recognition, and acoustic model adopts the acoustic model based on Hidden Markov Model (HMM), and what language model adopted is based on the grammatical language model of N unit.During multipass decoding, because unknown examinee answers content so adopt the process such as non-supervisory self-adaptation, decode in two phases directly decoding, return based on maximum linear likelihood.Many encoding and decoding techniques can be adopted to improve discrimination during decoding in addition.After having decoded, extracted the flow comprehensive characteristics of needs by grading module from recognition result, training Rating Model, finally carries out machine automatic scoring according to Rating Model to new audio frequency.
The management of the present embodiment full-automatic oral evaluation and the core algorithm of points-scoring system adopt the machine learning method based on pattern-recognition, and the described machine learning based on pattern-recognition adopts the study of linear regression method, case similarity assessment, correlation rule learns, one in neural network or support vector machine.
The feature in the direction of pronouncing accuracy described in above-described embodiment is obtained by following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
Recognition result and correct text are carried out pressure align, calculate the pressure alignment score of each phoneme;
Build single-tone element decoded model and each phoneme of decoding, calculate the maximum likelihood score of each phoneme;
Utilize the feature of forcing the difference of alignment score and maximum likelihood score to obtain pronouncing accuracy direction.
The feature specific embodiment in described pronouncing accuracy direction:
Represent etic acoustic feature with O, p represents the pronunciation phonemes corresponding to phonetics, and Q represents all set of phonemes.Under the prerequisite of then given acoustic feature O, the posterior probability of phoneme p is:
PP＝P(p|o)＝P(o|p)P(p)/∑
q∈QP(Q|q)P(q)
Conveniently calculate, make following hypothesis: all phonemes are all that equiprobability occurs, existing P (p)=P (q), and the summation in denominator can use maximum estimated value, so just obtains the definition of pronouncing accuracy:
GOP(p)＝log(P(o|p)/MAX
q∈Q
P(o|q))。
Feature extraction: pronouncing accuracy can be obtained by following formula
GOP＝S
FA-S
PL。
The feature in the direction of fluency described in above-described embodiment comprises word speed feature and the duration characteristics that pauses in short-term, and described word speed feature is obtained by following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
The frame number that in voice identification result, each phoneme is corresponding is counted according to recognition result;
The ratio of the duration of the total number of phoneme and all phonemes is utilized to obtain word speed feature;
The described duration characteristics of pause is in short-term obtained by following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
Frame number that in voice identification result, each phoneme is corresponding and the total frame number of audio frequency is counted according to recognition result;
The summation of the duration utilizing all phonemes to pause in short-term and the ratio of total pronunciation duration are paused duration characteristics in short-term.
About described word speed feature specific embodiment:
In second language learning, word speed feature (ROS) well can characterize the fluency and spoken proficient of speaking, especially true for beginner.Use T
srepresent the duration of all phonemes, N
prepresent the total number of phoneme, then have:
ROS＝N
P/T
S。
About the described specific embodiment of pause feature in short-term:
Pause in short-term in voice between word and word also reflects pronunciation fluency.In general, the dead time is longer, and corresponding fluency is also poorer.Use D
sPrepresentative accounts for the proportion of overall length when pausing, T represents total pronunciation often, T
irepresent i-th to pause often, N
sPrepresent total number of pausing in short-term, then have:
The feature in the similarity of text semantic described in above-described embodiment direction comprises semantic relevancy feature and syntactic structure similarity feature.
Described semantic relevancy feature comprises the following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
Calculate the semantic similarity score of each word in each word in recognition result and model answer;
Calculate the semantic similarity score of each sentence in each word in recognition result and model answer;
Calculate semantic similarity score maximal value in recognition result in each word and model answer in each sentence as the similarity score between word and sentence;
Calculate the similarity score between examinee's answer and model answer.
Described semantic similarity feature can also be obtained by following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
Calculate the semantic similarity score of each word in each word in recognition result and model answer;
Calculate the semantic similarity score of each sentence in each word in recognition result and model answer;
To calculate in recognition result in each word and model answer each sentence semantic similarity score averages as the similarity score between word and sentence;
Calculate the similarity score between examinee's answer and model answer.
Described syntactic structure similarity feature comprises the following steps:
For examinee answer audio frequency carry out decoding identify, obtain recognition result;
Syntax sequence vector set up in each sentence being respectively recognition result;
Obtain the syntactic structure similarity score of each sentence in recognition result and each sentence in model answer respectively, to get in recognition result each Sentence Grammar structural similarity score maximal value as the syntactic structure similarity score of this sentence;
By to the syntactic structure similarity feature in recognition result between each the weighted average calculation examinee's answer of Sentence Grammar structural similarity score and model answer.
Described semantic relevancy feature specific embodiment:
As shown in Figure 3, for one embodiment of the invention semantic relevancy feature extraction process flow diagram, semantic relevancy can greatly react the semantic degree of correlation between examinee's answer and model answer, and the semantic relevancy of two words is higher in general, and the answer of examinee is also more near the mark answer.Semantic similarity between word and word can be calculated by word network (WordNet), and WordNet is the English glossary semantic net of a coverage broadness.Noun, verb, adjective and adverbial word are organized into a synon network separately, each TongYiCi CiLin represents a basic semantic concept, and is also connected (polysemant will appear in its TongYiCi CiLin of each meaning) by various relation between these set.Calculate the semantic similarity between two words for the method expanding semantic covering, such as we will calculate the semantic similarity of drawing and decal in English, and first we will obtain the semanteme of two words by WordNet:
drawing:paperthatisspeciallypreparedforuseindrafting。
decal:theartoftransferringdesignsfromspeciallypreparedpapertoawoodorglassormetalsurface。
The semanteme having three words in the semanteme of two words covers, wherein paper be unitary word we long-pending 1 point, and speciallyprepared be a binary phrase we long-pending 4 points, the semantic similarity between drawing and decal is 5 points.We just define a marking mechanism like this:
1) occur in the semanteme between two words unitary word we long-pending one point, occurs our long-pending n of a n unit phrase square point because the probability of an appearance n conjunction will far away than occurring that the probability of n unitary word is low simultaneously.Usually in WordNet, each word can comprise multiple semanteme.
With ω, ω ' respectively represent two word S (ω) represent word ω all semantemes in word network, c
irepresent i-th semanteme of word ω in word network, the semanteme between two words getting similarity score maximal value between two words when the similarity of calculating two words.
SCORE
lesk(ω，ω')＝MAX
ci∈s(ω')c
j∈s(ω′)rel(c
ic
j)。
After having had the semantic similarity between word and word, just the similarity score between word and sentence can be calculated, get similar score averages between word to word in sentence as the similarity score between word and sentence, with | U| represents the number of the word in sentence U.
Finally just can calculate the similarity score between examinee's answer U and model answer P, represent the duplicate removal word number of examinee's answer with uniq (U), then the similarity score between U and P is:
As shown in Figure 4, for another embodiment of the present invention semantic relevancy feature extraction process flow diagram, semantic relevancy can greatly react the semantic degree of correlation between examinee's answer and model answer, and the semantic relevancy of two words is higher in general, and the answer of examinee is also more near the mark answer.With ω, ω ' respectively represent two word S (ω) represent word ω all semantemes in word network, c
irepresent i-th semanteme of word ω in word network, the semanteme between two words getting similarity score maximal value between two words when the similarity of calculating two words.
After having had the semantic similarity between word and word, just the similarity score between word and sentence can be calculated.Get similar score maximal value between word to word in sentence as the similarity score between word and sentence, represent sentence with U.
SCORE
W(ω,U)＝max
ωi∈
uSCORE(ω,ωi)。
Finally just can calculate the similarity score between examinee's answer U and model answer P.
Described syntactic structure similarity feature specific embodiment:
As shown in Figure 5, be one embodiment of the invention syntactic structure similarity feature extraction process flow diagram.
Consider following two sentences:
T
1:ω
1ω
2ω
3ω
4ω
5ω
6ω
7ω
8ω
9。
T
2:ω
1ω
2ω
3ω
9ω
5ω
6ω
7ω
8ω
4。
Sentence T
1and T
2the word comprised is the same, and carry out semantic similarity analysis if simple by the word comprised in two words, the result so drawn will be T
1with T
2the meaning of one's words expressed by two words is duplicate.The word order of two sentences is different, and here by a kind of scheme of being carried out syntactic structure similarity analysis by word order of proposition, we are to T
1and T
2in each word renumber, such as T
1the numbering of middle dog is the numbering of 4, over is 6.Definition sentence T
1with T
2syntax sequence vector is r
1and r
2then have:
r
1＝(123456789)
r
2＝(123956784)
So sentence T
1with T
2syntactic structure similarity can be calculated by following formula:
T in actual conditions
1with T
2the word comprised might not be duplicate, need the word rule of correspondence between definition sentence here.
If at T
2in a word ω
ibe also contained in T
1in the middle of, we by this word at sentence T
1the numbering that middle first time occurs as it at T
2in numbering.Otherwise we will at T
1middle searching and ω
isemantic immediate word
If ω
iwith
similarity exceed the threshold value preset, so ω
iat T
1in numbering be just set to
at T
2in numbering.
If first two search all have failed, so ω
inumbering be just set to 0.
According to above rule, we can extract and obtain syntactic structure similarity feature.
Utilize formula
obtain every syntactic structure similarity talked about in recognition result every word and model answer respectively, to get in recognition result every language method structural similarity maximal value as its syntactic structure similarity.
Syntactic structure similarity S between final examinee's answer recognition result and model answer
rECcan be expressed as
Wherein M is the sentence number of each examinee's recognition result, and REC represents recognition result, and REF represents Key for Reference.
After obtaining above flow comprehensive characteristics, we need adjustment feature weight to make Rating Model effect best, can carry out feature weight optimization based on expectation-maximization algorithm.
In above any embodiment, the problem of fixed reference answer is not had for objective test, we have proposed a kind of answer expansion scheme based on text semantic similarity analysis: role playing is inscribed to the Key for Reference that the little topic of per pass can be utilized the to have provided referenced text as text semantic similarity analysis, and then expand other Key for Reference; The transcription text of audio content can be utilized as the referenced text of text semantic similarity analysis for repetition topic, and then expand other Key for Reference, examinee not necessarily answers by Key for Reference completely, so can not carry out pressure alignment.Decode procedure comprises the first decoding of row, speaker adaptation and decode in two phases identification; From recognition result, flow comprehensive characteristics is extracted after decode in two phases identification completes.Then flow comprehensive characteristics and teacher are marked and carry out Rating Model training, obtain Rating Model, carry out automatic scoring.
Full-automatic oral evaluation management of the present invention and points-scoring system to be answered text without the need to predicting examinee, and examinee can carry out the statement of spontaneity spoken language according to topic content, need to know that examinee's topic related text of answering can be marked before scoring; Read aloud the spoken test and appraisal of topic relative to tradition, the present invention not only can carry out the spoken language test and appraisal of reading aloud topic, can also carry out the spoken language test and appraisal of spontaneous spoken statement topic; Mark more comprehensively just, pronouncing accuracy and the fluency of examinee can be investigated when the statement of examinee's spontaneity, more can reflect the spoken language proficiency of examinee's reality; Examinee's text of answering no longer is restricted, and automatic scoring evaluation and test topic type also will no longer be only limitted to read aloud topic, so just can investigate examinee when spontaneous spoken statement to the understanding of language, utilization and ability to express; Semantic relevancy feature and syntactic structure similarity feature is added in flow comprehensive characteristics, the semantic dependency that examinee's spoken language uses can be investigated like this, the syntactical level of examinee's spoken language can be investigated again, devise unique high in the clouds evaluating system framework, guarantee that evaluation and test is efficiently carried out, take full advantage of the resource of whole system simultaneously, greatly improve and organize oral evaluation efficiency, save a large amount of manpower and materials; Oral evaluation can be organized on a large scale simultaneously, meet within the scope of provinces and cities even nationwide and organize the demand of oral evaluation simultaneously; The form of oral evaluation is also by diversification more, more comprehensively just to the investigation of examinee's spoken language proficiency.
Be understandable that, for the person of ordinary skill of the art, other various corresponding change and distortion can be made by technical conceive according to the present invention, and all these change the protection domain that all should belong to the claims in the present invention with distortion.
Claims (5)
1. a full-automatic oral evaluation management and points-scoring system, it comprises the client connected successively, land server and the webserver, wherein, the arrangement of webserver primary responsibility evaluation result, the distribution of collection and paper, land server primary responsibility machine automatic scoring, client primary responsibility is evaluated and tested, paper is distributed to client from the webserver by landing server, examinee's result of answering uploads to the webserver from client by landing server, described examinee result of answering comprises one in the spoken evaluating result of the spoken evaluating result of reading aloud topic and spontaneous spoken statement topic, it is characterized in that: described in land server and also comprise scoring apparatus, this scoring apparatus comprises identification module and grading module, described identification module comprises acoustics submodule, language submodule and recognin module, described acoustics submodule extracts the answer acoustic feature of audio frequency of examinee and obtains acoustic model, described language submodule obtains language model according to topic information and training text, described recognin module is carried out decoding by acoustic model and language model obtain recognition result to examinee's audio frequency of answering, institute's scoring module comprises feature extraction submodule and scoring submodule, described feature extraction submodule is for extracting the flow comprehensive characteristics in described recognition result, described flow comprehensive characteristics comprises the feature in the feature in pronouncing accuracy direction in spoken test and appraisal, the feature in fluency direction and text semantic similarity direction, and the feature in described text semantic similarity direction comprises semantic relevancy feature and syntactic structure similarity feature,
Described feature extraction submodule, for extracting described semantic relevancy feature, comprising: the semantic similarity score calculating each word in each word in recognition result and model answer; Calculate the semantic similarity score of each sentence in each word in recognition result and model answer; Calculate semantic similarity score maximal value in recognition result in each word and model answer in each sentence or mean value as the similarity score between word and sentence; Calculate the similarity score between examinee's answer and model answer;
Described feature extraction submodule, for extracting described syntactic structure similarity feature, comprising: syntax sequence vector set up in each sentence being respectively recognition result; Obtain the syntactic structure similarity score of each sentence in recognition result and each sentence in model answer respectively, to get in recognition result each Sentence Grammar structural similarity score maximal value as the syntactic structure similarity score of this sentence; By to the syntactic structure similarity feature in recognition result between each the weighted average calculation examinee's answer of Sentence Grammar structural similarity score and model answer;
Described identification module adopts the decode system based on extensive continuous speech recognition, described acoustic model adopts based on Hidden Markov Model (HMM), described language model adopts the language model based on unit's syntax, adopt based on multipass decoding technique when decoding, the decoding of described multipass comprise directly decode, the non-supervisory self-adaptation that returns based on maximum linear likelihood and decode in two phases; Described scoring submodule carries out scoring training to flow comprehensive characteristics, obtains Rating Model, and marks to recognition result according to Rating Model.
2. full-automatic oral evaluation management as claimed in claim 1 and points-scoring system, is characterized in that: described feature extraction submodule also for extracting the feature in described pronouncing accuracy direction, comprising:
Recognition result and correct text are carried out pressure align, calculate the pressure alignment score of each phoneme;
Build single-tone element decoded model and each phoneme of decoding, calculate the maximum likelihood score of each phoneme;
Utilize the feature of forcing the difference of alignment score and maximum likelihood score to obtain pronouncing accuracy direction.
3. oral evaluation management as claimed in claim 1 full-automatic and points-scoring system, is characterized in that: the webserver comprises scheduler module, for dispatching evaluation and test information landing between server and the webserver; Adopt networking evaluation and test pattern, the communication of server of landing between the examination hall making to be distributed in different location is managed by described webserver scheduler module United Dispatching.
4. full-automatic oral evaluation management as claimed in claim 1 and points-scoring system, it is characterized in that: described system comprises the role of three kinds of different rights: examinee, teacher and keeper, examinee's primary responsibility is evaluated and tested and is answered; Teacher's primary responsibility system volume, issue are evaluated and tested, manage evaluation and test, check evaluation result and the work of scoring, the method combined that scoring aspect employing system is marked and teacher marks; The management of keeper's primary responsibility evaluation and test and the time of welltesting software are controlled.
5. full-automatic oral evaluation management and a methods of marking, is characterized in that: it comprises following several step:
A0, choose the process that some examinees carry out as described in steps A 1 ~ A5, automatic scoring model training is carried out in combination of then flow comprehensive characteristics and teacher being marked, and forms Rating Model;
A1, collect examinee and to answer audio frequency;
A2, extract the answer acoustic feature of audio frequency of examinee and obtain acoustic model, and obtain language model according to topic information and training text;
A3, according to the acoustic model set up and language model, examinee's audio frequency of answering is carried out to decoding and obtains recognition result;
A4, the flow comprehensive characteristics extracted in recognition result, described flow comprehensive characteristics comprises the feature in the feature in pronouncing accuracy direction in spoken test and appraisal, the feature in fluency direction and text semantic similarity direction; The feature in described text semantic similarity direction comprises semantic relevancy feature and syntactic structure similarity feature;
Obtain described semantic relevancy feature to comprise the following steps: for examinee answer audio frequency carry out decoding identify, obtain recognition result; Calculate the semantic similarity score of each word in each word in recognition result and model answer; Calculate the semantic similarity score of each sentence in each word in recognition result and model answer; Calculate semantic similarity score maximal value in recognition result in each word and model answer in each sentence or mean value as the similarity score between word and sentence; Calculate the similarity score between examinee's answer and model answer;
Obtain described syntactic structure similarity feature to comprise the following steps: for examinee answer audio frequency carry out decoding identify, obtain recognition result; Syntax sequence vector set up in each sentence being respectively recognition result; Obtain the syntactic structure similarity score of each sentence in recognition result and each sentence in model answer respectively, to get in recognition result each Sentence Grammar structural similarity score maximal value as the syntactic structure similarity score of this sentence; By to the syntactic structure similarity feature in recognition result between each the weighted average calculation examinee's answer of Sentence Grammar structural similarity score and model answer;
Adopt the decode system based on extensive continuous speech recognition to carry out decoding to identify, described acoustic model adopts based on Hidden Markov Model (HMM), described language model adopts the language model based on unit's syntax, adopt based on multipass decoding technique when decoding, the decoding of described multipass comprise directly decode, the non-supervisory self-adaptation that returns based on maximum linear likelihood and decode in two phases;
A5, according to flow comprehensive characteristics formed Rating Model carry out automatic scoring.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310034371.7A CN103151042B (en) | 2013-01-23 | 2013-01-29 | Full-automatic oral evaluation management and points-scoring system and methods of marking thereof |
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013100276765 | 2013-01-23 | ||
| CN201310027676.5 | 2013-01-23 | ||
| CN201310027676 | 2013-01-23 | ||
| CN201310034371.7A CN103151042B (en) | 2013-01-23 | 2013-01-29 | Full-automatic oral evaluation management and points-scoring system and methods of marking thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103151042A CN103151042A (en) | 2013-06-12 |
| CN103151042B true CN103151042B (en) | 2016-02-24 |
Family
ID=48549064
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310034371.7A Active CN103151042B (en) | 2013-01-23 | 2013-01-29 | Full-automatic oral evaluation management and points-scoring system and methods of marking thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103151042B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106782603A (en) * | 2016-12-22 | 2017-05-31 | 上海语知义信息技术有限公司 | Intelligent sound evaluating method and system |
Families Citing this family (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104347071B (en) * | 2013-08-02 | 2020-02-07 | 科大讯飞股份有限公司 | Method and system for generating reference answers of spoken language test |
| CN103761975B (en) * | 2014-01-07 | 2017-05-17 | 苏州驰声信息科技有限公司 | Method and device for oral evaluation |
| CN103985391A (en) * | 2014-04-16 | 2014-08-13 | 柳超 | Phonetic-level low power consumption spoken language evaluation and defect diagnosis method without standard pronunciation |
| CN103985392A (en) * | 2014-04-16 | 2014-08-13 | 柳超 | Phoneme-level low-power consumption spoken language assessment and defect diagnosis method |
| CN104505089B (en) * | 2014-12-17 | 2018-05-18 | 福建网龙计算机网络信息技术有限公司 | Spoken error correction method and equipment |
| CN106156905A (en) * | 2015-03-25 | 2016-11-23 | 肖圣林 | Mobile interchange learning platform |
| CN104810017B (en) * | 2015-04-08 | 2018-07-17 | 广东外语外贸大学 | Oral evaluation method and system based on semantic analysis |
| CN105632488A (en) * | 2016-02-23 | 2016-06-01 | 深圳市海云天教育测评有限公司 | Voice evaluation method and device |
| CN108010513B (en) * | 2016-10-28 | 2021-05-14 | 北京回龙观医院 | Voice processing method and device |
| CN108154735A (en) * | 2016-12-06 | 2018-06-12 | 爱天教育科技（北京）有限公司 | Oral English Practice assessment method and device |
| CN106847263B (en) * | 2017-01-13 | 2020-09-11 | 科大讯飞股份有限公司 | Speech level evaluation method, device and system |
| CN106652622B (en) * | 2017-02-07 | 2019-09-17 | 广东小天才科技有限公司 | Text training method and device |
| CN107316255A (en) * | 2017-04-07 | 2017-11-03 | 苏州清睿教育科技股份有限公司 | A kind of efficient competition method competed online that shuttles |
| CN109686383B (en) * | 2017-10-18 | 2021-03-23 | 腾讯科技（深圳）有限公司 | Voice analysis method, device and storage medium |
| CN109785698B (en) * | 2017-11-13 | 2021-11-23 | 上海流利说信息技术有限公司 | Method, device, electronic equipment and medium for oral language level evaluation |
| CN108090047B (en) * | 2018-01-10 | 2022-05-24 | 华南师范大学 | Text similarity determination method and equipment |
| CN108363687A (en) * | 2018-01-16 | 2018-08-03 | 深圳市脑洞科技有限公司 | Subjective item scores and its construction method, electronic equipment and the storage medium of model |
| CN108428382A (en) * | 2018-02-14 | 2018-08-21 | 广东外语外贸大学 | It is a kind of spoken to repeat methods of marking and system |
| CN108447504A (en) * | 2018-03-20 | 2018-08-24 | 校宝在线（杭州）科技股份有限公司 | A kind of oral evaluation method based on brain wave |
| CN108766415B (en) * | 2018-05-22 | 2020-11-24 | 清华大学 | Voice evaluation method |
| CN108831212B (en) * | 2018-06-28 | 2020-10-23 | 深圳语易教育科技有限公司 | Auxiliary device and method for oral teaching |
| AU2019316649A1 (en) * | 2018-08-10 | 2021-03-11 | Actively Learn, Inc. | Short answer grade prediction |
| CN109036464B (en) * | 2018-09-17 | 2022-02-22 | 腾讯科技（深圳）有限公司 | Pronunciation error detection method, apparatus, device and storage medium |
| CN109256152A (en) * | 2018-11-08 | 2019-01-22 | 上海起作业信息科技有限公司 | Speech assessment method and device, electronic equipment, storage medium |
| CN109523194B (en) * | 2018-12-14 | 2021-06-01 | 深圳市小柠檬教育科技有限公司 | Chinese reading ability evaluation method and device and readable storage medium |
| CN109410984B (en) * | 2018-12-20 | 2022-12-27 | 广东小天才科技有限公司 | Reading scoring method and electronic equipment |
| CN109829155B (en) * | 2019-01-18 | 2024-03-22 | 平安科技（深圳）有限公司 | Keyword determination method, automatic scoring method, device, equipment and medium |
| CN109858877A (en) * | 2019-01-22 | 2019-06-07 | 北京五岳鑫信息技术股份有限公司 | A kind of test questions input method and device |
| CN109545243B (en) * | 2019-01-23 | 2022-09-02 | 北京猎户星空科技有限公司 | Pronunciation quality evaluation method, pronunciation quality evaluation device, electronic equipment and storage medium |
| CN110085256B (en) * | 2019-03-21 | 2021-11-19 | 视联动力信息技术股份有限公司 | Information processing method and device |
| CN109830230B (en) * | 2019-03-27 | 2020-09-01 | 深圳平安综合金融服务有限公司上海分公司 | Data labeling method and device based on self-learning algorithm |
| CN110164422A (en) * | 2019-04-03 | 2019-08-23 | 苏州驰声信息科技有限公司 | A kind of the various dimensions appraisal procedure and device of speaking test |
| CN109979256A (en) * | 2019-04-24 | 2019-07-05 | 上海乂学教育科技有限公司 | English pronunciation teaching system |
| CN110491369A (en) * | 2019-07-24 | 2019-11-22 | 北京大米科技有限公司 | Appraisal procedure, device, storage medium and the electronic equipment of spoken grade |
| CN110570838B (en) * | 2019-08-02 | 2022-06-07 | 北京葡萄智学科技有限公司 | Voice stream processing method and device |
| CN110797010A (en) * | 2019-10-31 | 2020-02-14 | 腾讯科技（深圳）有限公司 | Question-answer scoring method, device, equipment and storage medium based on artificial intelligence |
| CN111047932A (en) * | 2020-01-02 | 2020-04-21 | 上海健坤教育科技有限公司 | Voice interactive teaching system |
| CN111383495A (en) * | 2020-03-04 | 2020-07-07 | 苏州驰声信息科技有限公司 | In-class explanation system, method, device and medium for spoken language teaching |
| CN111599234A (en) * | 2020-05-19 | 2020-08-28 | 黑龙江工业学院 | Automatic English spoken language scoring system based on voice recognition |
| CN112767932A (en) * | 2020-12-11 | 2021-05-07 | 北京百家科技集团有限公司 | Voice evaluation system, method, device, equipment and computer readable storage medium |
| CN112951207B (en) * | 2021-02-10 | 2022-01-07 | 网易有道信息技术（北京）有限公司 | Spoken language evaluation method and device and related product |
| CN112992184B (en) * | 2021-04-20 | 2021-09-10 | 北京世纪好未来教育科技有限公司 | Pronunciation evaluation method and device, electronic equipment and storage medium |
| CN113626768A (en) * | 2021-08-12 | 2021-11-09 | 上海好学网络科技有限公司 | Multi-score multi-intelligent-score joint-scoring method for spoken language examination |
| CN115497455B (en) * | 2022-11-21 | 2023-05-05 | 山东山大鸥玛软件股份有限公司 | Intelligent evaluating method, system and device for oral English examination voice |
| CN118035711B (en) * | 2024-04-11 | 2024-06-14 | 国家计算机网络与信息安全管理中心江苏分中心 | Evaluation method and system for large model content security capability |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101246685A (en) * | 2008-03-17 | 2008-08-20 | 清华大学 | Pronunciation quality evaluation method of computer auxiliary language learning system |
| CN101393103A (en) * | 2008-10-31 | 2009-03-25 | 中国科学院力学研究所 | Hydrate microscopic seepage experimental apparatus |
| CN101551947A (en) * | 2008-06-11 | 2009-10-07 | 俞凯 | Computer system for assisting spoken language learning |
| CN101740024A (en) * | 2008-11-19 | 2010-06-16 | 中国科学院自动化研究所 | Method for automatic evaluation based on generalized fluent spoken language fluency |
| CN102034475A (en) * | 2010-12-08 | 2011-04-27 | 中国科学院自动化研究所 | Method for interactively scoring open short conversation by using computer |
| CN102509483A (en) * | 2011-10-31 | 2012-06-20 | 苏州思必驰信息科技有限公司 | Distributive automatic grading system for spoken language test and method thereof |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8775184B2 (en) * | 2009-01-16 | 2014-07-08 | International Business Machines Corporation | Evaluating spoken skills |
-
2013
- 2013-01-29 CN CN201310034371.7A patent/CN103151042B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101246685A (en) * | 2008-03-17 | 2008-08-20 | 清华大学 | Pronunciation quality evaluation method of computer auxiliary language learning system |
| CN101551947A (en) * | 2008-06-11 | 2009-10-07 | 俞凯 | Computer system for assisting spoken language learning |
| CN101393103A (en) * | 2008-10-31 | 2009-03-25 | 中国科学院力学研究所 | Hydrate microscopic seepage experimental apparatus |
| CN101740024A (en) * | 2008-11-19 | 2010-06-16 | 中国科学院自动化研究所 | Method for automatic evaluation based on generalized fluent spoken language fluency |
| CN102034475A (en) * | 2010-12-08 | 2011-04-27 | 中国科学院自动化研究所 | Method for interactively scoring open short conversation by using computer |
| CN102509483A (en) * | 2011-10-31 | 2012-06-20 | 苏州思必驰信息科技有限公司 | Distributive automatic grading system for spoken language test and method thereof |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106782603A (en) * | 2016-12-22 | 2017-05-31 | 上海语知义信息技术有限公司 | Intelligent sound evaluating method and system |
| CN106782603B (en) * | 2016-12-22 | 2020-08-11 | 云知声（上海）智能科技有限公司 | Intelligent voice evaluation method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103151042A (en) | 2013-06-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103151042B (en) | Full-automatic oral evaluation management and points-scoring system and methods of marking thereof | |
| CN105741831B (en) | A kind of oral evaluation method and system based on syntactic analysis | |
| CN101740024B (en) | Method for automatic evaluation of spoken language fluency based on generalized fluency | |
| CN101739867B (en) | Method for scoring interpretation quality by using computer | |
| CN109785698B (en) | Method, device, electronic equipment and medium for oral language level evaluation | |
| CN103065626B (en) | Automatic grading method and automatic grading equipment for read questions in test of spoken English | |
| CN101751919B (en) | Spoken Chinese stress automatic detection method | |
| CN103559892B (en) | Oral evaluation method and system | |
| CN103594087B (en) | Improve the method and system of oral evaluation performance | |
| CN105845134A (en) | Spoken language evaluation method through freely read topics and spoken language evaluation system thereof | |
| CN103559894A (en) | Method and system for evaluating spoken language | |
| CN104464423A (en) | Calibration optimization method and system for speaking test evaluation | |
| CN109697988B (en) | Voice evaluation method and device | |
| Delais-Roussarie et al. | Learner corpora and prosody: From the COREIL corpus to principles on data collection and corpus design | |
| CN109658918A (en) | A kind of intelligence Oral English Practice repetition topic methods of marking and system | |
| KR100995847B1 (en) | Language training method and system based sound analysis on internet | |
| CN110598041A (en) | FlACS real-time analysis method and device | |
| CN113205729A (en) | Foreign student-oriented speech evaluation method, device and system | |
| CN110010123A (en) | English phonetic word pronunciation learning evaluation system and method | |
| Wang et al. | LAIX Corpus of Chinese Learner English: Towards a Benchmark for L2 English ASR. | |
| Zechner et al. | Using automatic speech recognition to assess the reading proficiency of a diverse sample of middle school students | |
| CN115116474A (en) | Spoken language scoring model training method, scoring method, device and electronic equipment | |
| CN114241835A (en) | Student spoken language quality evaluation method and device | |
| Jing et al. | The speech evaluation method of English phoneme mobile learning system | |
| CN111599234A (en) | Automatic English spoken language scoring system based on voice recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |