CN117635381B

CN117635381B - Method and system for evaluating computing thinking quality based on man-machine conversation

Info

Publication number: CN117635381B
Application number: CN202311477552.7A
Authority: CN
Inventors: 詹泽慧; 钟煊妍
Original assignee: South China Normal University
Current assignee: South China Normal University
Priority date: 2023-11-07
Filing date: 2023-11-07
Publication date: 2024-06-25
Anticipated expiration: 2043-11-07
Also published as: CN117635381A

Abstract

The application discloses a method and a system for evaluating computing thinking quality based on man-machine conversation, wherein the method comprises the following steps: constructing a computer agent, wherein the computer agent is used for providing a plurality of questions and solving prompts of the plurality of questions in a man-machine dialogue mode; obtaining answering information generated in the process that a learner answers through a computer agency, wherein the answering information comprises behavior information and text information; constructing a computing thinking test database which comprises question information and answer information; the question information comprises a plurality of questions and corresponding question solving prompts; according to a preset automatic evaluation algorithm and a calculation thinking test database, abnormal data which do not accord with preset conditions in answer information are removed, and then the current calculation thinking quality of a learner is determined according to the answer information and corresponding question information. The application can quickly and accurately acquire the evaluation result of the computing thinking quality of the learner through learning analysis, and can be widely applied to the technical field of computer information.

Description

Method and system for evaluating computing thinking quality based on man-machine conversation

Technical Field

The application relates to the technical field of computer information, in particular to a method and a system for evaluating the quality of computing thinking based on man-machine interaction.

Background

The computational thinking is one of the core literacy of the information technology subject, and with new lesson innovation and age development, the computational thinking becomes an indispensable position in the core literacy culture process more and more. The evaluation of the computing thinking has important significance in promoting the cultivation of the computing thinking, in particular to the evaluation of the computing thinking quality, namely the evaluation of the performance of the learner in each sub-dimension of the computing thinking, which is helpful for teachers or learners to improve the training more pertinently.

However, the present stage of calculation thinking evaluation mainly adopts a work analysis method and a scale investigation method. The work analysis method is to measure the calculated thinking level by scoring the student works, namely, the performance of each sub-dimension of the calculated thinking cannot be evaluated, and an evaluation result of the calculated thinking quality is obtained. Although the scale investigation method measures from different dimensions, the method is too subjective, and in addition, the computing thinking is a procedural capability, and the method lacks support of procedural data and processing of procedural data, so that even if the scale investigation method is divided into dimensions during scale production, the learner performance in different sub-dimensions of the computing thinking is difficult to score accurately through simple investigation. With the development of technology, people are gradually aware of the significance of computing thinking quality by using learning analysis technology. However, due to the singleness of data collection in the current assessment method, in order to ensure the accuracy of the assessment result, it has to be required for the learner to complete a large number of questions to acquire more data of the learner. This not only requires a long measurement time, but also causes a great cognitive load on the learner and even affects the answering situation of the learner.

Disclosure of Invention

In view of the above, the present application provides a method and a system for evaluating the quality of computing thinking based on human-computer interaction, so as to quickly and accurately obtain the evaluation result of computing thinking quality of a learner through learning analysis.

One aspect of the present application provides a method for evaluating computational thinking quality based on human-machine conversation, comprising:

Constructing a computer agent, wherein the computer agent is used for providing a plurality of questions and solving questions prompts of the plurality of questions in a man-machine dialogue mode;

obtaining answering information generated in the process that a learner answers through the computer agency, wherein the answering information comprises behavior information and text information; the behavior information comprises behavior steps of answering the questions of the learner, time stamps of answering the questions and answering results of interaction questions; the text information comprises man-machine dialogue content of the learner and the computer agent and a answering result of a text question;

constructing a computing thinking test database, wherein the computing thinking test database comprises question information and answer information; the title information comprises a plurality of the titles and corresponding problem solving prompts;

And removing abnormal data which do not meet preset conditions in the answer information according to a preset automatic evaluation algorithm and the calculated thinking test database, and determining the current calculated thinking quality of the learner according to the answer information and the corresponding question information.

Optionally, the obtaining answer information generated in the process that the learner answers through the computer agent includes:

Transmitting the questions input by the learner to the computer agent through a WebSocket interface;

Performing intention recognition on the questions input by the learner by using a natural language understanding model constructed by a natural language understanding framework through the computer agent to obtain question intention of the learner;

Determining a response mode and a corresponding response text according to the questioning intention and the context of the current man-machine conversation through a conversation strategy of a conversation manager; the responding mode comprises the steps of generating responding text to respond by utilizing a generated dialogue model and responding by using a predefined responding text; the dialogue strategy comprises a story defined for questioning intents in different fields and a defined slot, wherein the story is used for enabling the computer agent to understand the input questions, and the slot is used for storing state information and topic IDs of the current man-machine dialogue;

The response text is obtained through a WebSocket interface and displayed on a display interface of the learner in a chat frame mode, so that the learner can answer according to the response text;

obtaining answer information generated in the process that the learner answers according to the answer text;

wherein, the natural language understanding model and the generated dialogue model both adopt a transducer model.

Optionally, the constructing a computing thinking test database includes:

constructing a computing thinking test database;

Storing the topic content, the topic answers, the topic IDs, the examined capability labels of the computing thinking sub-capability and the horizontal labels corresponding to the computing thinking sub-capability of each topic as topic information in a first data set;

Taking the current questions which are currently answered by the learner as current questions, for each current question, acquiring the question ID of the current question and the behavior information in the answering process through the log file of the computer agent, and storing the question ID, the answering time stamp, the answering correctness label and the behavior information in the answering process of the current question as the answering information in a second data set after the learner finishes the current questions;

Storing the first data set and the second data set in the computational thinking test database.

Optionally, the removing abnormal data in the answer information, which does not meet the preset condition, according to the preset automatic evaluation algorithm and the calculated thinking test database, and determining the current calculated thinking quality of the learner according to the answer information and the corresponding question information includes:

Obtaining answer information and question information from the computing thinking test database;

Carrying out standardization processing on each capacity label and each horizontal label by using the preset automatic evaluation algorithm through Robust standardization;

grouping a plurality of said topics according to each of said capability tags;

and determining the computing thinking sub-capability which is correctly answered in each group and has the highest level label as the current computing thinking quality of the learner according to the answer correctness labels in the second data set.

Optionally, the method further comprises:

constructing a potential computing thinking quality assessment model, extracting characteristics from the behavior information by using the potential computing thinking quality assessment model and identifying a behavior mode of answering by the learner to obtain potential computing thinking quality;

Semantic analysis and topic classification are carried out on the text information through a preset text analysis model, and the sub-dimension which cannot be solved by the learner in the answering process is determined and used as the thinking sub-dimension of the doubtful calculation;

Determining recommended topics for the learner based on the potential computational thinking qualities and the in-doubt computational thinking sub-dimensions.

Optionally, the constructing the potential computing thinking quality assessment model includes:

Constructing a cyclic neural network model for capturing behavior mode and sequence information in behavior information of the tester;

taking behavior information of different calculation thinking quality testers as training samples, and taking calculation thinking quality of each tester as training labels to train the cyclic neural network model;

In the training process, the parameters of the cyclic neural network model are adjusted by using a cross entropy loss function and a back propagation algorithm of the classification problem, and the cyclic neural network model after the training is finished is used as the potential computing thinking quality evaluation model;

Wherein the recurrent neural network model comprises:

The embedded layer is used for converting the answer step sequence from a word or symbol sequence into a continuous vector representation;

The circulating layer adopts a long-short-time memory network and is used for capturing time dependence in the sequence data;

And the full-connection layer is used for outputting predicted calculated thinking quality, measuring the difference between the predicted label and the actual label by using the cross entropy loss function in the training process of the full-connection layer, and training the cyclic neural network model through the back propagation algorithm so as to improve the accuracy of the predicted label.

Optionally, the performing semantic analysis and topic classification on the text information through a preset text analysis model, determining a sub-dimension that cannot be solved by the learner in the answering process, as a thinking sub-dimension of the in-doubt calculation, includes:

Converting the current man-machine dialogue content and the current answering title ID in the text information into semantic vectors through a word embedding model;

inputting the current man-machine conversation content and the current answering question ID in the form of semantic vectors into the preset text analysis model to obtain the computational thinking sub-dimension related to the current man-machine conversation content;

Extracting data corresponding to the currently answered topic ID from the computational thinking test database, and determining the difficulty level of the currently answered topic ID on the computational thinking sub-dimension through a preset mapping function, thereby obtaining the doubtful computational thinking sub-dimension;

The preset text analysis model is a convolutional neural network which is obtained by training by taking event elements marked with relation labels as training data.

Optionally, the determining the recommended topic of the learner based on the potential computational thinking quality and the in-doubt computational thinking sub-dimension includes:

Constructing a queue, wherein the queue is used for storing questions to be answered by the learner;

Randomly acquiring a plurality of questions from a preset question bank according to a self-evaluation result preset by the learner, and storing the randomly acquired plurality of questions in the queue;

When the learner finishes one question to be answered, selecting a question matched with the thinking sub-dimension of the in-doubt calculation from the preset question bank, and adding the question into the queue;

comparing the potential computing thinking quality with the current computing thinking quality, and if the comparison result is not matched, selecting a question matched with the potential computing thinking quality from the preset question bank and adding the question into the queue;

And taking the topics in the queue as the recommended topics.

In another aspect, the present application also provides a system for evaluating the quality of computing thinking based on man-machine conversation, comprising:

The first module is used for constructing a computer agent, and the computer agent is used for providing a plurality of questions and solving questions prompts of the plurality of questions in a man-machine dialogue mode;

The second module is used for obtaining answering information generated in the process that the learner answers the questions through the computer agency, and the answering information comprises behavior information and text information; the behavior information comprises behavior steps of answering the questions of the learner, time stamps of answering the questions and answering results of interaction questions; the text information comprises man-machine dialogue content of the learner and the computer agent and a answering result of a text question;

The third module is used for constructing a computing thinking test database, and the computing thinking test database comprises question information and answer information; the title information comprises a plurality of the titles and corresponding problem solving prompts;

and a fourth module, configured to remove abnormal data in the answer information, where the abnormal data does not meet a preset condition, according to a preset automatic evaluation algorithm and the calculated thinking test database, and then determine the current calculated thinking quality of the learner according to the answer information and the corresponding question information.

Another aspect of the application also provides an electronic device comprising a processor and a memory;

the memory is used for storing programs;

The processor executes the program to implement the method described above.

Another aspect of the present application also provides a computer-readable storage medium storing a program that is executed by a processor to implement the foregoing method.

The application also discloses a computer program product or a computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of an electronic device, the computer instructions being executed by the processor to cause the electronic device to perform the aforementioned method.

Firstly, constructing a computer agent, wherein the computer agent is used for providing a plurality of questions and questions solving prompts of the plurality of questions in a man-machine conversation mode, so that a learner can answer questions in the man-machine conversation mode through the computer agent; further obtaining answering information generated in the process that the learner answers the questions through the computer agency; then adding the answer information, the questions and the question prompts of each question into a computing thinking test database; and then removing abnormal data which do not accord with preset conditions in the answer information according to a preset automatic evaluation algorithm and a calculation thinking test database, and further determining the current calculation thinking quality of the learner according to the answer information and the corresponding question information. The method improves the convenience and accuracy of computing thinking evaluation and solves the problems of the existing computing thinking evaluation process that the computing thinking evaluation is cumbersome and the evaluation result is inaccurate.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flow chart of a method for evaluating quality of computing thinking based on man-machine interaction according to an embodiment of the present application;

FIG. 2 is a schematic flow chart of a man-machine conversation between a learner and a computer agent according to an embodiment of the present application;

FIG. 3 is a block diagram of a text analysis model according to an embodiment of the present application;

FIG. 4 is a block diagram of a system for evaluating quality of computing thinking based on human-computer interaction according to an embodiment of the present application;

fig. 5 is a block diagram of another system for evaluating quality of computing thinking based on man-machine interaction according to an embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

It should be noted that although functional block division is performed in a device diagram and a logic sequence is shown in a flowchart, in some cases, the steps shown or described may be performed in a different order than the block division in the device, or in the flowchart.

The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the application only and is not intended to be limiting of the application.

Aiming at the defects of single data source and evaluation index, inaccurate evaluation result, lack of dimension division, long evaluation time, large topic quantity, difficulty in personalized adjustment according to the situation of learners and the like in the existing research method, the application provides a method and a system for evaluating the computing thinking quality based on man-machine conversation.

Referring to fig. 1, an embodiment of the present application provides a method for evaluating quality of computing thinking based on human-machine interaction, which includes steps S100 to S130, specifically as follows:

S100: a computer agent is constructed for providing a plurality of topics and a plurality of questions solving prompts in the form of a human-machine conversation.

Specifically, the learner can answer questions through the man-machine conversation by the computer agent.

S110: obtaining answering information generated in the process that a learner answers through the computer agency, wherein the answering information comprises behavior information and text information; the behavior information comprises behavior steps of answering the questions of the learner, time stamps of answering the questions and answering results of interaction questions; the text information includes the content of human-machine conversations of the learner with the computer agent and the answering result of the text questions.

Specifically, the learner may generate various answer information, such as behavior information and text information, during the answer process, and obtaining the answer information may be used to evaluate the computational thinking quality of the learner.

Further, S110 may include:

S120: constructing a computing thinking test database, wherein the computing thinking test database comprises question information and answer information; the question information comprises a plurality of questions and corresponding question solving prompts.

Specifically, considering that a learner generates a large amount of answer information when answering, the computer agent can provide a plurality of questions and questions solving prompts of the questions, and the data volume is also huge, so that the embodiment can construct a computing thinking test database to store the information.

Further, S120 may include:

constructing a computing thinking test database;

Specifically, in this embodiment, the capability label and the horizontal label in the first data set may be labeled in a manual labeling manner, and the corresponding label may be obtained after the manual labeling.

The step of obtaining the answer correctness labels in the second data set may specifically include:

The behavior information and text information representing the answer result are collected through a log collection system associated with a log server, and stored in a log cache cluster; and obtaining the question ID of the current answer of the student and the answer result of the student from the log cache cluster, comparing the answer result of the learner in the log file with a standard answer, and storing whether the answer is correct or not as a label in a second data set of the computer thinking test database according to the comparison result, wherein the label is used for recording the answer correctness of the student.

S130: and removing abnormal data which do not meet preset conditions in the answer information according to a preset automatic evaluation algorithm and the calculated thinking test database, and determining the current calculated thinking quality of the learner according to the answer information and the corresponding question information.

Specifically, the abnormal data which does not conform to the preset condition in the answer information may refer to data which does not conform to the behavior information or the text information, and after the abnormal information is removed, the present embodiment may determine the current calculated thinking quality of the learner by using the answer information and the corresponding question information in the calculated thinking test database and combining with a preset automatic evaluation algorithm.

Further, S130 may include:

grouping a plurality of said topics according to each of said capability tags;

In consideration of the fact that the more questions the learner makes, the more accurate the evaluation result is, but this may require the learner to make a lot of questions. Assuming that the computing thinking is classified into three levels 1, 2 and 3, if the learner's level is sub-level 3, the learner has to do from the first level of questions, which is time-consuming and labor-consuming. For learners with different calculated thinking levels, the paths of doing the same topic are different, and if the topic doing path of a certain learner is identified as level 3, the topic of level 3 can be directly recommended.

Accordingly, the present embodiment may further include a step of determining a learner recommended topic, including S140 to S160:

s140: and constructing a potential computing thinking quality evaluation model, and extracting characteristics from the behavior information and identifying the behavior mode of the learner answer by using the potential computing thinking quality evaluation model to obtain the potential computing thinking quality.

Further, the step of constructing the potential computational thinking quality assessment model may include:

Wherein the recurrent neural network model comprises:

S150: and carrying out semantic analysis and topic classification on the text information through a preset text analysis model, and determining the sub-dimension which cannot be solved by the learner in the answering process as the thinking sub-dimension of the doubtful calculation.

Further, S150 may include:

S160: determining recommended topics for the learner based on the potential computational thinking qualities and the in-doubt computational thinking sub-dimensions.

Further, S160 may include:

And taking the topics in the queue as the recommended topics.

Computing thinking is a kind of thinking and ability focused on "processes", which emphasizes the use of basic concepts of computer science to solve problems, design systems, and understand human behavior, which can be divided into three sub-dimensions of computing concepts, computing practices, and computing concepts. The embodiment of the application provides a computing thinking quality assessment method and a computing thinking quality assessment system based on man-machine dialogue, which provide help in the answer process for learners through the man-machine dialogue, avoid burnout and helplessness caused by long-time confusion of the learners in the test, analyze the answer process and the result of the learners in real time based on natural language understanding and machine learning, realize the high efficiency of the assessment process, the individuation of test question recommendation and the real-time of updating the computing thinking assessment result, solve the problems of large task and inaccurate result of the current computing thinking assessment to a great extent, promote the convenience and pertinence of computing thinking assessment, and promote the optimization computing thinking culture by assessment as guiding.

In order that the application may be more clearly understood, the application will be described in the following in alternative specific embodiments.

The embodiment discloses a computing thinking evaluation method based on man-machine conversation, which comprises the following steps:

Acquiring information of computational thinking test questions, marking one or more computational thinking sub-capabilities and difficulty levels of the computational thinking sub-capabilities inspected by each question in a manual marking mode, and storing the inspected capabilities and capability level labels, the content of the questions, answers of the questions and question IDs in a first data set;

Obtaining the answering condition information of the learner, obtaining the current question ID of the learner and the behavior information in the answering process in real time through the log file, and storing the current question ID, the answering time stamp, the correctness of the answering and the behavior information in the answering process in a second data set after the learner finishes the questions;

constructing a computing thinking test database, and storing the contents of the first data set and the second data set into the computing thinking test database;

analyzing the current computing thinking quality of the learner based on the answer condition information and the corresponding question information of the target user in the computing thinking test database:

Specifically:

extracting a question ID, a question answering time stamp, a question answering correctness label, a question investigation capability label and a capability level label of the completion of the student from a calculation thinking test database;

performing standardization processing on the labels with different capacity levels through Robust standardization;

Grouping topics according to capability tags in each of the topic history information, wherein one topic history can exist in a plurality of groups;

The record in each group that is correctly answered and the capacity level under investigation is the highest is selected and taken as the current capacity level of the student.

In this embodiment, a computer agent is constructed to perform a man-machine conversation, and the questionable computational thinking dimension is obtained by analyzing the content of the man-machine conversation, and a specific example flow may refer to fig. 2, and the target object in fig. 2 is referred to as a learner.

Specifically, the problem raised by the target object is transmitted to the computer agent through the WebSocket interface; the computer agent uses a trained natural language understanding model to analyze the student's problem, the model will help the agent determine the intent of the problem and extract relevant entities; helping a dialog agent to determine a response means according to intention and context of a question through a dialog policy of a dialog manager (Rasa Core); the dialog strategy includes generating an answer based on an answer to the generated dialog model or a predefined answer template; and finally, transmitting the generated answer to a computing thinking test system through a WebSocket interface.

Specifically, the present embodiment may train and integrate a natural language understanding model and a generated answer model. The construction of the natural language understanding model can be realized through the following steps:

step 1: training data is created, preparing a training data set containing the question text and corresponding intent tags. Each training sample should include a question and an intent tag corresponding to the question.

Step 2: the Rasa NLU model is configured and a configuration file (config. Yml) is created to specify the parameters of the model and the NLU pipeline. In the configuration file spaCy is used as a tagging and feature extraction tool and the pipeline is configured to include the intent classifier.

An example configuration file is as follows:

pipeline:

"WhitespaceTokenizer"// for segmenting the input text into words or tokens;

"LanguageModelFeaturizer"// converting text information into semantic vectors using a pre-trained Bert model;

model_name:"bert-base-chinese"

model_weights:"bert-base-chinese"

cache_dir:null

"EntitySynonymMapper"// map synonyms for entities to standardized entity tags;

name- "SKLEARNINTENTCLASSIFIER"// component for intent classification, which can map entered text to predefined intent tags.

The configuration file is clear, and the natural language model adopts a Bert model for converting text information into semantic vectors and classifying intentions. The Bert model is composed of multiple transducer layers, each of which in turn includes the following two sublayers:

Self-attention mechanism (Self-Attention Mechanism): the sub-layer assigns a weight to each word or sub-word in the input text to understand word-to-word relationships. This sub-layer can learn the relevance between different words in the text.

Feedforward neural network (Feedforward Neural Network): receives the output of the self-attention mechanism and proceeds through the neural network layer for further processing. The help model understands contextual information and semantic relationships between words.

In addition, to improve training stability and performance of the model. In both sub-layers of the Transformer of the Bert model, residual connection (Residual Connection) and layer normalization (Layer Normalization) are added. The residual connection is to add the output of each sub-layer to its input, which helps to alleviate the gradient vanishing problem and make training more stable. Layer normalization is a regularization technique that helps to speed training convergence and improve model performance in deep networks, which operates as shown in equation (1). Where ε (epsilon) is a small positive number to prevent zero variance, p is the input value, a is the input mean, b is the input variance, h is the normalized value, gamma is a learnable scaling parameter, betas is a learnable translation parameter.

Final output = gamma x h + beta (1)

Step 3: the training commands of the Rasa NLU are run to train the model, during which the model will learn to map the input questions to the corresponding intent tags.

Step 4: and integrating the trained Rasa NLU model into a computer agent. In the computer proxy configuration file, the NLU component is configured and ensures that the natural language understanding model can work in concert with the dialog manager (Rasa Core).

The dialog manager determines the response mode based on the result of the intention recognition output by the natural language understanding model. In the configuration file of the dialog manager, mapping Policy is used to map specific intents to answer templates. Meanwhile, fallback Policy is configured to handle default choice generative answer models when there is no matching mapping.

The model construction of the generated answer can be realized by the following steps:

Step 1: dialogue data including questions, topic IDs, and reference answers associated therewith are collected. The data is organized in the form of a CSV file, with each line including input questions, question IDs, and corresponding answer samples.

Step 2: text data is processed by word segmentation, encoding, normalization and the like by using a text preprocessing tool (tokenizers library) and is converted into vectors.

Step 3: the pre-trained Seq2Seq (Sequence-to-Sequence) model is loaded using Hugging Face Transformers libraries, the Seq2Seq model is composed of an encoder (Encoder) and a Decoder (Decoder). Wherein the encoder is responsible for encoding the input sequence (text) into a semantic representation and capturing context information of the input sequence. The key layers of the encoder are:

Embedding layer (Embedding Layer): words or subwords in the input sequence will first be converted by the embedding layer into a continuous vector representation so that the model can process them. This layer maps discrete words to a continuous vector space.

Transformer encoder: this is the core of the encoder, which receives the embedded input sequence and progressively captures the context information. For a transducer encoder, it consists of multiple self-attention layers and a feed-forward neural network for processing the entire input sequence simultaneously.

Context vector: finally, the encoder encodes the entire input sequence into a fixed-size context vector or matrix that contains the semantic information of the input sequence. This context vector is passed to the decoder to help generate the output sequence.

The decoder receives the context vector delivered by the encoder and decodes it into an output sequence that generates text to answer the results of the target object question. The following are key layers of the decoder:

Transformer decoder: a decoder layer comprising a plurality of transducers. It gradually generates words or subwords of the output sequence. As with the transducer encoder, the transducer decoder is also composed of multiple self-attention layers and a feedforward neural network;

Attention mechanism (Attention Mechanism): the attention mechanism focuses on different parts of the input sequence to determine the importance of each word that generates the output sequence. This helps the model to focus better on the relevant information of the input sequence when it is generated.

Generating an output sequence: words or subwords of the output sequence are generated one by one with the help of the context vector, the generated partial sequence and the attention mechanism until a complete sequence is generated. In generating the sequence, a particular start tag is typically used, and then a cluster search strategy is used to consider multiple alternatives to select the most appropriate word.

Step 4: a loss function (cross entropy loss) and an optimizer (Adam) are defined.

Further, determining a sub-dimension which cannot be solved by individuals in the answering process based on semantic analysis and topic classification of text information of a dialogue between the computer agent and the target object;

The computational thinking quality of the doubt is determined based on the sub-dimensions and corresponding question IDs that cannot be solved by the person during the answering process.

Specifically, the steps are as follows:

step 1, obtaining text information of man-machine conversation, and converting the text data into semantic vectors which can be understood by a model through a Word embedding model (Word 2 Vec);

Step 2, extracting text data and topic ID of a man-machine dialogue in the process of answering the current topic as input of a text analysis model;

Step 3, obtaining the dimension of the computing thinking related to the dialogue through a text analysis model;

Step 4, extracting records corresponding to the topic ID from a computational thought test database, and determining the difficulty level of the topic on the computational thought sub-dimension through a mapping function so as to obtain the in-doubt computational thought sub-dimension;

Further, training of the text analysis model is further included, specifically:

Data collection and preparation: sufficient human-machine conversation data is collected, including conversation text and corresponding topic IDs. The data is divided into a training set, a validation set and a test set.

Text preprocessing: the dialog text is pre-processed, including text word segmentation, stop word removal, word embedding (Word Embedding), etc., to convert the text to an input format acceptable to the model.

Model training: the Convolutional Neural Network (CNN) is trained using a training set. During training, the model will attempt to learn how to extract information about the computational thought sub-dimension from the text data. The goal of the training is to minimize the loss function, which measures the gap between the model's predictions and the actual labels.

Referring to fig. 3, the present embodiment provides a schematic structural diagram of a text analysis model, which is a convolutional neural network and may include an input layer, a convolutional layer, an activation function layer, a pooling layer, and a fully-connected layer. The input layer mainly outputs word embedding vectors processed by the word embedding model, and stacks the word embedding vectors into a two-dimensional matrix, wherein each row represents one word embedding vector; the convolutional layer is the core of the convolutional neural network for capturing local features in the text. The activation function layer then adds an activation function ReLU (RECTIFIED LINEAR Unit) after the convolution layer, as in equation (2), where x is the value of the input, and if x is equal to or greater than 0, then the output is equal to x, otherwise the output is 0 to introduce nonlinear properties.

ReLU(x) ＝ max(0, x) (2)

The pooling layer is used for reducing the size of the feature map, reducing the calculation complexity and extracting the most important features; the full connection layer is located on top of the network for mapping the features extracted by the rolling and pooling layers to the final output class.

In the embodiment, behavior information representing a student answering process in a computing thinking test database is converted into a vector sequence, and a cyclic neural network model is utilized to capture mode and sequence information in the behavior information, so that potential computing thinking quality corresponding to the behavior mode is predicted.

The key of the cyclic neural network model is a cyclic layer and a full-connection layer, and the cyclic layer uses a long-short-time memory network (LSTM) for capturing time dependence in sequence data. By means of a plurality of LSTM units in the LSTM, each behavior input can be processed time step by time step, and a feature vector is generated in each time step, wherein the feature vector comprises important features of behavior information and also comprises information of the previous time step. Finally, the loop layer outputs a sequence of feature vectors. The sequence will be the input to the full join layer, which will flatten the sequence of vectors output by the LSTM, joining multiple vectors into one long vector. The long vector is mapped to the intermediate representation by a plurality of neurons, where the last three neurons, each outputting a scalar, correspond to three different sub-dimensions of the computational thought. These three scalars are further applied to the activation function Softmax to convert their quantities into three sets of probability distributions, each set of probability distributions being randomly sampled to obtain a sample value, which will correspond to the level of the dimension. The activation function Softmax is shown in equation (3), where e ^z1、e^z2、e^z3 is the output value of the three scalars, respectively.

P₁＝e^z1/(e^z1+e^z2+e^z3)

P₂＝e^z2/(e^z1+e^z2+e^z3) (3)

P₂＝e^z3/(e^z1+e^z2+e^z3)

Further, training the cyclic neural network model is further included, specifically:

Obtaining behaviors of learners with different computing thinking qualities in the answering process, obtaining the level of each sub-dimension of the computing thinking corresponding to each behavior information in a manual labeling mode, and constructing the behavior information representing the answering process and the corresponding computing thinking quality into a training set;

Training the cyclic neural network model by using a training set, and capturing the sequence information of answering behaviors and the correlation between different sub-dimension levels. During the training process, the model will attempt to learn how to extract features from the behavioral information and map these features to the level of each computational thought child dimension. The goal of the training is to minimize the loss function, which measures the gap between the model's predictions and the actual labels. The system is classified into multiple labels according to the requirements, and the loss function adopts a binary cross entropy loss function, specifically:

Wherein, Is a loss function representing the actual label y and the predictive label/>, of the modelThe gap between them; n is the number of training samples; m is the number of sub-dimensions (3 sub-dimensions of the computational thinking according to the setup of this example); y _ij is the value of the jth sub-dimension of the ith sample in the actual label, typically 0 or 1, indicating whether it belongs to that sub-dimension; /(I)Is a predictive value of the model representing the level of the jth sub-dimension of the ith sample.

During training, an optimization algorithm (Adam) is used to continually adjust the weights and bias of the model to minimize the loss function. The training model is iterated repeatedly on the training set, so that the understanding capability of the relation between the answer behaviors and the calculated thinking quality is gradually improved.

Further, the personalized recommendation topic for the target object according to the potential computing thinking quality and the suspected computing thinking sub-dimension specifically comprises the following steps:

a queue is constructed to store the questions to be completed, some of the questions are randomly selected from the computer thought test database, and they are added to the queue of questions to be completed. The number of the questions and the difficulty level can be determined according to the self-evaluation of the students, so that the adaptability of the questions is ensured.

The learner completes the topics in the queue one by one. After each question is completed, behavior information and text information representing the answering process of the learner are analyzed, and potential computing thinking quality and suspected computing thinking sub-dimension are obtained.

The topics in the queue are dynamically adjusted according to the potential computational thinking quality and the in-doubt computational thinking sub-dimension. That is, if a learner encounters difficulty on a topic and requests the help of a computer agent, a doubtful computational thought sub-dimension is obtained through analysis of human-computer conversation content, and a query is written to retrieve topic records matching these parameters (computational thought sub-dimension and difficulty level) from a computational thought test database. One or more topics are selected from the matched topic records and added to the learner's topic queue. If the learner correctly completes the answering of the questions, the current computing thinking quality is obtained according to the behavior information or text information analysis of the answer representing results, the potential computing thinking quality is obtained according to the behavior information of the standard answer results, the potential computing thinking quality and the current computing thinking quality can be compared, if the potential computing thinking quality and the current computing thinking quality are not matched, the questions with corresponding difficulties are found in the computing thinking test database according to the potential computing thinking quality, and the questions are brought into a queue of the questions to be completed.

Referring to fig. 4, an embodiment of the present invention provides a system for evaluating quality of computing thinking based on man-machine conversation, comprising:

The embodiment of the computing thinking quality evaluation system shown in fig. 4 is substantially the same as the embodiment of the computing thinking quality evaluation method described above, and will not be described herein.

Referring to fig. 5, another system for evaluating quality of computing thinking based on man-machine conversation may be provided according to an embodiment of the present application, including:

A computational thought test database module configured to: constructing a database for storing question information and answer condition information of a target object (i.e. a learner), wherein the database comprises a question ID, a question inspected capability label, a question inspected capability level label, a question content, a question answer, a question ID finished by the learner, an answer time stamp, an answer correctness and behavior information of an answer process;

A human-machine conversation module configured to: the method comprises the steps of carrying out natural language understanding on a learner question by utilizing a Rasa NLU model, namely a natural language understanding model, mapping the question to a specific intention and entity, selecting a proper response by utilizing a Rasa Core, namely a dialog manager according to the current dialog state and the intention of the learner, and generating a response by utilizing a generated dialog model or a predefined response template;

The current computing thinking quality computing module is configured to: comparing the information of the characterization completion result with the preset question answers, marking the behavior information of the students by taking the correctness of the completion as a label, and storing the behavior information into a computing thinking test database. If the answer is correct, the corresponding question investigation capability and level are obtained according to the question ID, and the current calculation thinking quality of the learner is obtained;

A potential computational thinking quality prediction module configured to: analyzing behavior information representing a learner answering process by using a cyclic neural network, capturing modes and sequence information in the behavior information, and further predicting potential computing thinking quality corresponding to the behavior modes;

An in-doubt computing thinking sub-dimension analysis module configured to: acquiring text information of a man-machine conversation and a topic ID corresponding to the conversation content, determining a computational thought subdimension designed by the man-machine conversation content by using a text analysis model, extracting a corresponding record from a computational thought test database according to the topic ID corresponding to the section of man-machine conversation content, determining the difficulty level of the topic examined on the computational thought subdimension by using a mapping function, and further obtaining the in-doubt computational thought subdimension;

A personalized recommendation module configured to: and constructing a queue of questions to be completed, and randomly extracting 10 questions with corresponding levels from the calculation thinking test database to be incorporated into the queue according to the self-evaluation result of the learner. And then obtaining the current calculated thinking quality, the in-doubt calculated thinking sub-dimension and the potential calculated thinking quality according to the answering condition of the learner, and dynamically adjusting the queue according to the current calculated thinking quality, the in-doubt calculated thinking sub-dimension and the potential calculated thinking quality. Comparing the potential computing thinking quality with the current computing thinking quality, if the potential computing thinking quality and the current computing thinking quality are not matched, finding out the questions corresponding to the potential computing thinking quality from a computing thinking test database, and incorporating the questions into a queue of questions to be completed; if the learner has man-machine interaction in the answering process, obtaining the suspicious computing thinking sub-dimension according to the man-machine dialogue content, finding the questions corresponding to the suspicious computing thinking sub-dimension in the computing thinking test database, and incorporating the questions into a queue of questions to be completed.

The embodiment of the application also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the computing thinking quality evaluation method when executing the computer program.

Specifically, the electronic device may be a user terminal or a server.

The embodiment of the application also provides a computer readable storage medium, which stores a computer program, and the computer program realizes the method for evaluating the computing thinking quality when being executed by a processor.

The memory, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory remotely located relative to the processor, the remote memory being connectable to the processor through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Embodiments of the present application also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read by a processor of an electronic device from a computer-readable storage medium and executed by the processor to cause the electronic device to perform the method shown in fig. 1.

In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present application are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.

Furthermore, while the application is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the described functions and/or features may be integrated in a single physical device and/or software module or one or more functions and/or features may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present application. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be apparent to those skilled in the art from consideration of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the application as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the application, which is to be defined in the appended claims and their full scope of equivalents.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing an electronic device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

While embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the application, the scope of which is defined by the claims and their equivalents.

While the preferred embodiment of the present application has been described in detail, the present application is not limited to the embodiments described above, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope of the present application as defined in the appended claims.

Claims

1. A method for evaluating the quality of a computational thought based on a human-machine conversation, comprising:

obtaining answering information generated in the process of answering by a learner through the computer agent by utilizing the computer agent, wherein the answering information comprises behavior information and text information; the behavior information comprises behavior steps of answering the questions of the learner, time stamps of answering the questions and answering results of interaction questions; the text information comprises man-machine dialogue content of the learner and the computer agent and a answering result of a text question;

2. The method for evaluating the quality of computing thinking based on man-machine conversation according to claim 1, wherein the step of obtaining answer information generated in the process of answering by the learner through the computer agent comprises the following steps:

3. A method for evaluating the quality of a computational thinking based on a human-machine conversation as claimed in claim 1, wherein said constructing a computational thinking test database comprises:

constructing a computing thinking test database;

4. The method for evaluating the quality of the computing thinking based on the human-computer interaction according to claim 3, wherein the step of removing abnormal data which do not meet the preset condition from the answer information according to the preset automatic evaluation algorithm and the computing thinking test database, and determining the current computing thinking quality of the learner according to the answer information and the corresponding question information comprises the following steps:

grouping a plurality of said topics according to each of said capability tags;

5. A method for evaluating the quality of a computational thought based on a human-machine conversation according to claim 1, characterized in that it further comprises:

6. A method for evaluating the quality of a computational thought based on a human-machine conversation according to claim 5, wherein said constructing a model for evaluating the quality of a potential computational thought comprises:

Wherein the recurrent neural network model comprises:

7. The method for evaluating the quality of computing thinking based on man-machine conversation according to claim 5, wherein the steps of performing semantic analysis and topic classification on the text information through a preset text analysis model, determining a sub-dimension which cannot be solved by the learner in the answering process as an in-doubt computing thinking sub-dimension, and comprising:

8. A method for evaluating computational thinking quality based on human-machine conversation as claimed in claim 5, wherein said determining recommended subjects of said learner from said potential computational thinking quality and said in-doubt computational thinking sub-dimension comprises:

And taking the topics in the queue as the recommended topics.

9. A human-machine conversation-based computing thought quality assessment system, comprising:

The second module is used for obtaining answering information generated in the process that the learner answers the questions through the computer agent by utilizing the computer agent, wherein the answering information comprises behavior information and text information; the behavior information comprises behavior steps of answering the questions of the learner, time stamps of answering the questions and answering results of interaction questions; the text information comprises man-machine dialogue content of the learner and the computer agent and a answering result of a text question;

10. An electronic device comprising a processor and a memory;

the memory is used for storing programs;

the processor executing the program implements the method of any one of claims 1 to 8.