CN118116009B - Teaching aid control method and system based on local area network - Google Patents
Teaching aid control method and system based on local area network Download PDFInfo
- Publication number
- CN118116009B CN118116009B CN202410529634.XA CN202410529634A CN118116009B CN 118116009 B CN118116009 B CN 118116009B CN 202410529634 A CN202410529634 A CN 202410529634A CN 118116009 B CN118116009 B CN 118116009B
- Authority
- CN
- China
- Prior art keywords
- handwriting
- teaching
- point
- video
- gesture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000011156 evaluation Methods 0.000 claims abstract description 22
- 210000000707 wrist Anatomy 0.000 claims description 14
- 238000001514 detection method Methods 0.000 claims description 11
- 230000006870 function Effects 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 claims description 3
- 210000004247 hand Anatomy 0.000 description 5
- 230000008859 change Effects 0.000 description 3
- 210000001513 elbow Anatomy 0.000 description 3
- 210000002832 shoulder Anatomy 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention belongs to the technical field of teaching aids, and particularly relates to a teaching aid control method and system based on a local area network, wherein the method comprises the following steps: determining the similarity of the handwriting gesture according to the relative position characteristics of each concerned gesture point in all frame images of the handwriting exercise video and the handwriting teaching video, determining the handwriting matching degree according to the matching result of the handwriting word graph and the template word graph, determining the handwriting horizontal evaluation value of a handwriting exerciser according to the similarity of the handwriting gesture, the handwriting matching degree, the difference of the number of frame images contained in the handwriting exercise video and the handwriting teaching video and the difference of pressure data of each first pixel point matched in the handwriting word graph and the template binary image, and controlling the digital soft pen handwriting teaching tool to recommend teaching resources for the handwriting exerciser according to the handwriting horizontal evaluation value of the handwriting exerciser. The invention improves the accuracy of evaluating the exercise level of the handwriting exercise person and personally recommends teaching resources suitable for the exercise level and the requirement of the handwriting exercise person.
Description
Technical Field
The invention relates to the technical field of teaching aids. More particularly, the invention relates to a teaching aid control method and system based on a local area network.
Background
Aiming at the problem that the current offline soft-pen handwriting training courses are limited by time, place, energy and the like, the digital soft-pen handwriting teaching aid combined with intelligent hardware and digital interaction technology is generated, and a platform for online learning and handwriting practice without limitation of time, place, energy and the like is provided for soft-pen handwriting lovers.
In order to better meet the learning requirements of different handwriting exercisers, the learning effect is improved, and the digital soft pen handwriting teaching aid can analyze and evaluate the exercise conditions of the handwriting exercisers, so that teaching resources suitable for the level and the requirements of the handwriting exercisers are recommended individually.
In the prior art, the exercise level of a handwriting exerciser is evaluated mainly according to a soft pen handwriting copying similarity evaluation method, wherein the soft pen handwriting copying similarity evaluation method only evaluates according to the similarity of single strokes, and the evaluation accuracy is not high.
Disclosure of Invention
The invention provides a method for evaluating the similarity of handwriting copying of a digital soft pen, which aims to solve the technical problems that the prior method for evaluating the similarity of handwriting copying of the soft pen only evaluates the exercise level of a handwriting exerciser according to the similarity of single strokes, and cannot accurately evaluate the exercise level of the handwriting exerciser, so that teaching resources recommended by a digital soft pen handwriting teaching tool are not suitable for the handwriting exerciser.
In a first aspect, the present invention provides a teaching aid control method based on a local area network, including:
Collecting handwriting exercise videos and handwriting word graphs of handwriting exercisers, handwriting teaching videos and template word graphs of handwriting teaching students and pressure data of pixel points at all positions on the handwriting word graphs and the template word graphs through a digital soft pen handwriting teaching tool;
Human body posture detection is carried out on each frame of images of the handwriting exercise video and the handwriting teaching video, and attention posture points and reference points in each frame of images are obtained; determining the relative position characteristics of each concerned gesture point in each frame image according to the position relation between each concerned gesture point and the reference point in each frame image; determining the similarity of pen moving gestures according to the distances between the relative position features of all the concerned gesture points in all the frame images of the handwriting exercise video and the relative position features of all the concerned gesture points in the frame images of the handwriting teaching video;
obtaining a binary image corresponding to a handwriting word graph and a binary image corresponding to a template word graph through an Ojin method, respectively serving as the handwriting binary image and the template binary image, matching all first pixel points in the handwriting binary image with all first pixel points in the template binary image according to neighborhood characteristics and coordinates of first pixel points in the handwriting binary image and the template binary image, and determining handwriting matching degree according to a matching result;
Determining pressure data of each first pixel point in the handwriting binary image and the template binary image according to the pressure data of the pixel points at each position on the handwriting word graph and the template word graph;
Determining a handwriting horizontal evaluation value of a handwriting exerciser according to the similarity of the pen-handling gesture, the handwriting matching degree, the difference of the number of frame images contained in the handwriting exercise video and the handwriting teaching video and the difference of pressure data of each first pixel point in the handwriting binary image and each first pixel point in the template binary image;
and controlling the digital soft pen handwriting teaching tool to recommend teaching resources for the handwriting trainer according to the handwriting level evaluation value of the handwriting trainer.
In one embodiment, the handwriting level evaluation value of the handwriting trainer satisfies the expression:
；
Wherein S represents the handwriting level, G represents the pen-handling gesture similarity, P represents the handwriting matching degree, K represents the number of frame images contained in the handwriting exercise video, A represents the number of frame images contained in the handwriting teaching video, j represents the serial number of the first pixel point, Pressure data representing the j first pixel point in the handwriting binary image,Representing pressure data of a j-th first pixel point in the handwriting binary image in a corresponding first pixel point in the template binary image, N represents the number of the first pixel points in the handwriting binary image,The representation takes the absolute value of the value,An exponential function based on a natural constant is represented.
In one embodiment, the determining the similarity of the pen-handling gesture according to the distance between the relative position features of each focus gesture point in all frame images of the handwriting exercise video and the relative position features of each focus gesture point in the frame images of the handwriting teaching video includes:
For any concerned gesture point, a sequence formed by the relative position features of the concerned gesture point in all frame images of the handwriting exercise video according to time sequence is used as a first feature sequence of the concerned gesture point; the relative position features of the concerned gesture points in all frame images of the handwriting teaching video are used as a second feature sequence of the concerned gesture points according to a sequence formed by time sequences;
The similarity of pen-handling gestures satisfies the expression:
；
wherein G represents the similarity of pen-carrying gestures, A first feature sequence representing an ith pose point of interest,A second feature sequence representing an ith pose point of interest, i representing a sequence number of the pose point of interest,Representing DTW distance.
In one embodiment, the determining the relative position feature of each focus pose point in each frame image according to the position relation between each focus pose point in each frame image and the reference point includes:
The relative position characteristics of each concerned gesture point in each frame image of the handwriting exercise video satisfy the expression:
；
In the method, in the process of the invention, Representing the relative position characteristics of the ith concerned gesture point in the kth frame image of the handwriting exercise video, K representing the sequence number of the frame image of the handwriting exercise video, K taking all integers in the range of [1, K ], K representing the number of the frame images contained in the handwriting exercise video, i representing the sequence number of the concerned gesture point, i taking all integers in the range of [1,5],Representing the abscissa of the ith pose point of interest in the kth frame of image of the handwriting exercise video,Representing the ordinate of the ith pose point of interest in the kth frame of image of the handwriting exercise video,The abscissa representing the reference point in the kth frame image of the handwriting exercise video,Representing the ordinate of the reference point in the kth frame image of the handwriting exercise video,Representing an arctangent function;
The relative position characteristics of each concerned gesture point in each frame image of the handwriting teaching video satisfy the expression:
；
In the method, in the process of the invention, Representing the relative position characteristics of the ith concerned gesture point in the a-frame image of the handwriting teaching video, a represents the sequence number of the frame image of the handwriting teaching video, a takes all integers in the range of [1, A ], A represents the number of the frame images contained in the handwriting teaching video,Representing the abscissa of the ith focus gesture point in the a-frame image of the handwriting teaching video,Representing the ordinate of the ith focus gesture point in the a-frame image of the handwriting teaching video,Representing the abscissa of the reference point in the a-th frame image of the handwriting teaching video,And the ordinate of the reference point in the a-frame image of the handwriting teaching video is represented.
In one embodiment, the handwriting match satisfies the expression:
；
wherein P represents the matching degree of handwriting, Representing the neighborhood characteristic of the j first pixel point in the handwriting binary image,Representing the neighborhood characteristics of the jth first pixel point in the handwriting binary image in the corresponding first pixel point in the template binary image,Representing the Euclidean distance between the j first pixel point in the handwriting binary image and the corresponding first pixel point in the template binary image, wherein N represents the number of the first pixel points in the handwriting binary image;
the first pixel point is a pixel point with a gray value of 0 in the binary image.
In one embodiment, the controlling the digital soft pen handwriting teaching tool to recommend teaching resources for the handwriting exercise according to the handwriting level evaluation value of the handwriting exercise includes:
The handwriting exerciser performs copying exercise for one Chinese character each time, and teaching resources of the Chinese character comprise handwriting teaching videos and template word graphs of the Chinese character, and for any Chinese character:
(1) When the handwriting level of the handwriting exerciser is smaller than a preset first threshold value during exercise of the Chinese character, controlling the digital soft pen handwriting teaching aid to continuously recommend the handwriting teaching video and the template word graph of the Chinese character for the handwriting exerciser;
(2) When the handwriting level of the handwriting trainer is not less than a preset first threshold value and not more than a preset second threshold value during the training of the Chinese character, controlling the digital soft pen handwriting teaching tool to recommend the handwriting teaching video and the template word diagram of other Chinese characters in the combination of the Chinese character for the handwriting trainer;
(3) When the handwriting level of the handwriting exerciser is greater than a preset second threshold value during exercise of the Chinese character, controlling the digital soft pen handwriting teaching tool to recommend the handwriting teaching video and the template word graph of the Chinese character in other combinations except the combination where the Chinese character is located for the handwriting exerciser.
In one embodiment, the detecting the human body posture of each frame of image of the handwriting exercise video and the handwriting teaching video to obtain the attention posture point and the reference point in each frame of image includes:
Respectively carrying out human body posture detection on each frame of images of the handwriting exercise video and the handwriting teaching video through MEDIAPIPE POSE models to obtain all posture points in each frame of images of the handwriting exercise video and the handwriting teaching video;
For all the gesture points in any frame image, the gesture point corresponding to the shoulder on the right side of the human body is used as a reference point in the frame image, and the gesture points corresponding to the elbow on the right side of the human body, the wrist on the right side of the human body and the hand on the right side of the human body are used as concerned gesture points in the frame image.
In one embodiment, the collecting, by the digital soft pen handwriting teaching tool, handwriting exercise videos and handwriting word graphs of a handwriting exerciser, handwriting teaching videos and template word graphs of a handwriting learner, and pressure data of pixels at each position on the handwriting word graphs and the template word graphs includes:
the digital soft pen handwriting teaching tool comprises a high-speed shooting instrument, a capacitance pen and a writing board;
Collecting a handwriting teaching video of a handwriting teaching person through a high-speed photo instrument when the handwriting teaching person performs teaching display, and collecting a template word graph of the handwriting teaching person through the high-speed photo instrument when the teaching is finished;
when a handwriting exerciser exercises, acquiring handwriting exercise videos of the handwriting exerciser through a high-speed shooting instrument, and acquiring handwriting word graphs of the handwriting exerciser through the high-speed shooting instrument when the exercise is finished;
When a handwriting learner performs teaching on the writing board, recording the pressure from the capacitance pen received by each position on the writing board as pressure data of pixel points of each position on a template word graph;
when a handwriting exerciser exercises on the writing board, the pressure from the capacitance pen on each position on the writing board is recorded and used as pressure data of pixel points on each position on a handwriting chart.
In a second aspect, the present invention provides a teaching aid control system based on a local area network, which adopts the following technical scheme:
a teaching aid control system based on a local area network, comprising: the teaching aid control system comprises a processor and a memory, wherein the memory stores computer program instructions which are executed by the processor to realize the teaching aid control method based on the local area network.
By adopting the technical scheme, the teaching aid control method based on the local area network generates the computer program, and the computer program is stored in the memory to be loaded and executed by the processor, so that the terminal equipment is manufactured according to the memory and the processor, and the teaching aid control method based on the local area network is convenient to use.
The invention has the beneficial effects that: according to the invention, human body posture detection is carried out on each frame image of the handwriting exercise video and the handwriting teaching video, the similarity of the pen-carrying posture is determined according to the distance between the relative position characteristic of each concerned posture point in all frame images of the handwriting exercise video and the relative position characteristic of each concerned posture point in the frame images of the handwriting teaching video, the matching degree of handwriting is determined according to the matching result of all first pixel points in the handwriting binary image and all first pixel points in the template binary image, the consistency of writing speeds of a handwriting exerciser and a handwriting learner is represented through the difference of the number of frame images contained in the handwriting exercise video and the handwriting teaching video, the consistency of pen pressures of the handwriting exerciser and the handwriting learner is reflected through the difference of the pressure data of the pixel points of each position on the handwriting word graph and the template word graph, the similarity of the pen-carrying posture of the handwriting exerciser and the handwriting exerciser is further synthesized, the matching degree of handwriting and the consistency in the writing speeds and the pressure aspect are determined, the accuracy of the handwriting exerciser is improved, the requirements of the handwriting exerciser on the handwriting exerciser and the handwriting exerciser are well-level, and the accuracy of the handwriting exerciser is well-trained, and the requirements of the handwriting exerciser are well-trained, and the handwriting exerciser are well-trained.
Drawings
The above, as well as additional purposes, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description when read in conjunction with the accompanying drawings. In the drawings, embodiments of the invention are illustrated by way of example and not by way of limitation, and like reference numerals refer to similar or corresponding parts and in which:
FIG. 1 is a flow chart schematically illustrating a method of teaching aid control over a local area network in accordance with the present invention;
fig. 2 is a schematic diagram showing 33 posture points and positions thereof obtained by the MEDIAPIPE POSE model in the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
The embodiment of the invention discloses a teaching aid control method based on a local area network, which comprises the following steps of S1-S5 with reference to FIG. 1:
S1, acquiring handwriting exercise videos and handwriting word graphs of handwriting exercisers and handwriting teaching videos and template word graphs of handwriting teaching exercisers through digital soft pen handwriting teaching tools, and acquiring pressure data of pixel points at all positions on the handwriting word graphs and the template word graphs.
In order to better meet the learning requirements of different handwriting exercisers and improve the learning effect, the digital soft pen handwriting teaching tool analyzes and evaluates the exercise conditions of the handwriting exercisers, so that teaching resources suitable for the level and the requirements of the handwriting exercisers are recommended individually; in the prior art, the soft pen handwriting copying similarity evaluation method for evaluating the exercise level of a handwriting exerciser only evaluates according to the similarity of single strokes, and the evaluation accuracy is not high.
It should be further noted that the pen-carrying gesture, writing, pen pressure and writing speed during handwriting copying practice all affect the quality of soft pen handwriting works, in order to improve the accuracy of evaluating the exercise level of the handwriting exerciser, the invention integrates the pen-carrying gesture, writing, pen pressure and writing speed of the handwriting exerciser during handwriting copying practice, and the consistency of the pen-carrying gesture, writing, pen pressure and writing speed of the handwriting exerciser during teaching is used for evaluating the exercise level of the handwriting exerciser; therefore, the handwriting exercise video and the handwriting word graph of the handwriting exerciser and the handwriting teaching video and the template word graph of the handwriting learner as well as the pressure data of the pixel points at each position on the handwriting word graph and the template word graph are acquired, and the characteristics of the handwriting exerciser and the handwriting exerciser such as pen carrying posture, writing, pen pressure, writing speed and the like are acquired from the data and are used for evaluating the exercise level of the handwriting exerciser.
Specifically, the digital soft pen handwriting teaching aid consists of a high-speed shooting instrument for shooting, a liquid crystal screen for displaying teaching videos, a capacitance pen for handwriting exercise and a writing board.
1. The high-speed shooting instrument consists of a rotatable bracket and a camera arranged on the rotatable bracket, and is used for collecting handwriting teaching videos of the handwriting teaching staff when the handwriting teaching staff performs teaching display, collecting template word graphs of the handwriting teaching staff when the teaching is finished, collecting handwriting exercise videos of the handwriting training staff when the handwriting training staff performs exercise, and collecting handwriting word graphs of the handwriting training staff when the exercise is finished;
2. The liquid crystal screen is used for displaying corresponding handwriting teaching videos and template word graphs to a handwriting exerciser;
3. The essence of the writing board is a capacitive touch screen, when a handwriting learner performs teaching display, paper is paved on the writing board, the handwriting learner performs handwriting copying or handwriting copying on the paper of the writing board through a capacitive pen, and pressure from the capacitive pen on each position on the writing board is recorded and used as pressure data of pixel points on each position on a template character graph; when a handwriting exerciser exercises, firstly, paper is laid on the writing board, the handwriting exerciser performs handwriting tracing or handwriting copying on the paper of the writing board through the capacitance pen, and the pressure from the capacitance pen on each position on the writing board is recorded and used as pressure data of pixel points on each position on a handwriting chart.
S2, determining the similarity of the pen-handling gesture according to the relative position characteristics of each concerned gesture point in all frame images of the handwriting exercise video and the handwriting teaching video.
In order to improve the accuracy of evaluating the exercise level of a handwriting exercise person, the invention obtains all gesture points in each frame image of the handwriting exercise video and the handwriting teaching video by carrying out gesture recognition on the handwriting exercise video and the handwriting teaching video, obtains a first characteristic sequence and a second characteristic sequence of each concerned gesture point according to the relative position characteristics of each concerned gesture point in each frame image, and obtains the track coincidence degree of each concerned gesture point according to the distance between the first characteristic sequence and the second characteristic sequence of each concerned gesture point so as to represent whether the pen-carrying gesture of the handwriting exercise person is consistent with the pen-carrying gesture of the handwriting teaching person when carrying out handwriting copying exercise, thereby being used for evaluating the exercise level of the handwriting exercise person.
Specifically, human body posture detection is respectively carried out on each frame of image of the handwriting exercise video through a MEDIAPIPE POSE model (human body posture detection model), so that all posture points in each frame of image of the handwriting exercise video are obtained; and carrying out human body posture detection on each frame of image of the handwriting teaching video through a MEDIAPIPE POSE model (human body posture detection model) to obtain all posture points in each frame of image of the handwriting teaching video.
The MEDIAPIPE POSE model is a model for high-fidelity body posture tracking, and includes Pose detection model (human body detection model) and Pose landmarker model (human body labeling model), and 33 posture points of the human body and coordinates of the posture points can be obtained from each frame of image of the video.
For example, please refer to fig. 2, which shows 33 gesture points and their positions obtained by MEDIAPIPE POSE model.
In the case of handwriting copying, the handwriting is mainly performed by the shoulders, elbows, wrists and hands of the human body, so that the gestures of the handwriting exerciser can be reflected by the shoulders, elbows, wrists and hands, and only the gestures corresponding to the shoulders, elbows, wrists and hands are analyzed.
Illustratively, in fig. 2, the posture point denoted by 11 is a posture point corresponding to a shoulder on the right side of the human body, the posture point denoted by 13 is a posture point corresponding to an elbow on the right side of the human body, the posture point denoted by 15 is a posture point corresponding to a wrist on the right side of the human body, and the posture points denoted by 17, 19, 21 are posture points corresponding to hands on the right side of the human body; the posture point with the reference numeral 12 is a posture point corresponding to the shoulder on the left side of the human body, the posture point with the reference numeral 14 is a posture point corresponding to the elbow on the left side of the human body, the posture point with the reference numeral 16 is a posture point corresponding to the wrist on the left side of the human body, and the posture points with the reference numerals 18, 20 and 22 are posture points corresponding to the hands on the left side of the human body.
Further, for all gesture points in any frame of image of the handwriting exercise video, the gesture point corresponding to the shoulder on the right side of the human body is used as a reference point in the frame of image, and the gesture points corresponding to the elbow on the right side of the human body, the wrist on the right side of the human body and the hand on the right side of the human body are used as concerned gesture points in the frame of image; therefore, each frame of image of the handwriting exercise video comprises 1 reference point and 5 concerned gesture points; for all gesture points in any frame of image of the handwriting teaching video, the gesture point corresponding to the shoulder on the right side of the human body is used as a reference point in the frame of image, and the gesture points corresponding to the elbow on the right side of the human body, the wrist on the right side of the human body and the hand on the right side of the human body are used as concerned gesture points in the frame of image; therefore, each frame of image of the handwriting teaching video comprises 1 reference point and 5 concerned gesture points.
In each frame of image of the handwriting exercise video and the handwriting teaching video, a rectangular coordinate system is constructed by taking a pixel point at the upper left corner as an original point, taking the horizontal right of the original point as the positive X-axis direction and taking the vertical downward of the original point as the positive Y-axis direction.
Specifically, according to the position relation between each concerned gesture point and the reference point in each frame image of the handwriting exercise video, determining the relative position characteristic of each concerned gesture point in each frame image of the handwriting exercise video, wherein the relative position characteristic meets the expression:
；
In the method, in the process of the invention, Representing the relative position characteristics of the ith concerned gesture point in the kth frame image of the handwriting exercise video, K representing the sequence number of the frame image of the handwriting exercise video, K taking all integers in the range of [1, K ], K representing the number of the frame images contained in the handwriting exercise video, i representing the sequence number of the concerned gesture point, i taking all integers in the range of [1,5],Representing the abscissa of the ith pose point of interest in the kth frame of image of the handwriting exercise video,Representing the ordinate of the ith pose point of interest in the kth frame of image of the handwriting exercise video,The abscissa representing the reference point in the kth frame image of the handwriting exercise video,Representing the ordinate of the reference point in the kth frame image of the handwriting exercise video,Representing an arctangent function.
According to the position relation between each concerned gesture point and the reference point in each frame image of the handwriting teaching video, determining the relative position characteristic of each concerned gesture point in each frame image of the handwriting teaching video, wherein the relative position characteristic satisfies the expression:
；
In the method, in the process of the invention, Representing the relative position characteristics of the ith concerned gesture point in the a-frame image of the handwriting teaching video, a represents the sequence number of the frame image of the handwriting teaching video, a takes all integers in the range of [1, A ], A represents the number of the frame images contained in the handwriting teaching video,Representing the abscissa of the ith focus gesture point in the a-frame image of the handwriting teaching video,Representing the ordinate of the ith focus gesture point in the a-frame image of the handwriting teaching video,Representing the abscissa of the reference point in the a-th frame image of the handwriting teaching video,And the ordinate of the reference point in the a-frame image of the handwriting teaching video is represented.
The right shoulder of the calligraphic learner when performing calligraphic education and the right shoulder of the calligraphic exerciser when performing calligraphic exercise are not necessarily located at the same position, so that the right elbow, the right wrist and the right hand of the calligraphic learner when performing calligraphic education are not necessarily located at the same position as the gesture points corresponding to the right elbow, the right wrist and the right hand of the calligraphic exerciser when performing calligraphic exercise, and therefore, the consistency of the pen carrying gesture of the calligraphic exerciser and the calligraphic learner cannot be accurately represented directly according to the consistency of the positions of the gesture points corresponding to the right shoulder, the right elbow, the right wrist and the right hand in the calligraphic exercise video and the calligraphic teaching video; considering that when the pen-handling gestures are consistent, the relative positions of the right elbow, the right wrist and the right hand are consistent compared with the relative positions of the right shoulder, so that the gesture points corresponding to the right shoulder of the human body are used as reference points in the frame image, the gesture points corresponding to the right elbow, the right wrist and the right hand of the human body are used as concerned gesture points in the frame image, the included angle formed between the connecting line of each concerned gesture point and the reference point and the horizontal direction is used as the relative position characteristic of each concerned gesture point in each frame image, and the sequence formed by the relative position characteristics of each concerned gesture point in each frame image can reflect the gesture change condition of each concerned gesture point in the pen-handling process.
Further, for any concerned gesture point, a sequence formed by the relative position features of the concerned gesture point in all frame images of the handwriting exercise video according to time sequence is used as a first feature sequence of the concerned gesture point; and taking the relative position characteristics of the concerned gesture point in all frame images of the handwriting teaching video as a second characteristic sequence of the concerned gesture point according to a sequence formed by time sequences.
Further, according to the distance between the first feature sequence and the second feature sequence of each concerned gesture point, calculating the track coincidence degree of each concerned gesture point, and according to the track coincidence degree of each concerned gesture point, determining the pen-moving gesture similarity, and satisfying the expression:
；
wherein G represents the similarity of pen-carrying gestures, A first feature sequence representing an ith pose point of interest,A second feature sequence representing an ith pose point of interest, i representing a sequence number of the pose point of interest,Represents an exponential function with a base of a natural constant,The distance of DTW is indicated as such,The DTW distances of the first feature sequence and the second feature sequence representing the ith pose point of interest,Representing the track coincidence degree of the ith concerned attitude point; the DTW distance is obtained using a DTW algorithm (dynamic time warping algorithm).
It should be noted that, because the number of frame images included in the handwriting exercise video and the handwriting teaching video is different, the lengths of the first feature sequence and the second feature sequence of each concerned gesture point may be different, so that the distance between the first feature sequence and the second feature sequence of each concerned gesture point is calculated by adopting the DTW algorithm; the DTW algorithm is a well-known technique and will not be described in detail here.
The closer the DTW distance between the first feature sequence and the second feature sequence of each concerned gesture point is, the more similar the gesture change condition of each concerned gesture point in the handwriting exercise video in the pen transporting process is to the gesture change condition of each concerned gesture point in the handwriting teaching video in the pen transporting process, the larger the track overlap ratio is.
And S3, determining the handwriting matching degree according to the matching result of the handwriting word graph and the template word graph.
In order to improve the accuracy of evaluating the exercise level of a handwriting exerciser, the invention determines the handwriting matching degree by matching the neighborhood characteristics of the first pixel points which are pixel points belonging to handwriting in the handwriting word graph and the template binary image according to the difference and Euclidean distance between the neighborhood characteristics of the matching result of each first pixel point in the handwriting binary image of the handwriting exerciser and each first pixel point in the handwriting binary image of the handwriting exerciser.
Specifically, threshold segmentation is carried out on a handwriting word graph and a template word graph through an Ojin method, so that a binary image corresponding to the handwriting word graph and a binary image corresponding to the template word graph are obtained and respectively recorded as a handwriting binary image and a template binary image; in the binary image, a pixel having a gray level of 0 is referred to as a first pixel, and a pixel having a gray level of 255 is referred to as a second pixel.
In the handwriting binary image and the template binary image, a rectangular coordinate system is constructed by taking a pixel point at the upper left corner as an original point, taking the horizontal right of the original point as an X-axis positive direction and taking the vertical downward of the original point as a Y-axis positive direction, and the coordinates of all the pixel points are determined in the rectangular coordinate system.
Determining pressure data of each first pixel point in the handwriting binary image and the template binary image according to the pressure data of the pixel points at each position on the handwriting word graph and the template word graph, wherein the method comprises the following steps: the pressure data of each first pixel point in the handwriting binary image refers to the pressure data of the pixel point at the position of the first pixel point in the handwriting word graph; the pressure data of each first pixel point in the template binary image refers to the pressure data of the pixel point at the position of the first pixel point in the template word graph.
Further, regarding any one first pixel point, taking the difference between the number of the first pixel points and the number of the second pixel points contained in the neighborhood of the first pixel point as the neighborhood characteristic of each first pixel point; according to the neighborhood characteristics and the position coordinates of each first pixel point, all the first pixel points in the handwriting binary image and all the first pixel points in the template binary image are matched through a FLANN characteristic matching algorithm, and a matching result is obtained, wherein the matching result refers to the first pixel points corresponding to each first pixel point in the handwriting binary image in the template binary image.
It should be noted that, the FLANN feature matching algorithm is a known technique, and will not be described here.
In this embodiment, the neighborhood size is 5×5, and in other embodiments, the neighborhood size may be set according to the actual application scenario and requirements.
Further, determining a handwriting matching degree according to the matching result, wherein the handwriting matching degree meets the expression:
；
wherein P represents the matching degree of handwriting, Representing the neighborhood characteristic of the j first pixel point in the handwriting binary image,Representing the neighborhood characteristics of the j first pixel point in the handwriting binary image corresponding to the first pixel point in the template binary image, j representing the serial number of the first pixel point,Represents the Euclidean distance between the j first pixel point in the handwriting binary image and the corresponding first pixel point in the template binary image, N represents the number of the first pixel points in the handwriting binary image,The representation takes the absolute value of the value,An exponential function based on a natural constant is represented.
It should be noted that, the first pixel points represent the handwriting of the handwriting word graph and the template word graph, so that the smaller the difference between the neighborhood feature and the position coordinate, the more similar the handwriting of the handwriting word graph and the template word graph is, and the larger the matching degree of the handwriting is.
S4, determining a handwriting horizontal evaluation value of the handwriting exerciser according to the similarity of the pen-handling gesture, the handwriting matching degree, the difference of the number of frame images contained in the handwriting exercise video and the handwriting teaching video and the difference of pressure data of each first pixel point in the handwriting binary image and the corresponding first pixel point in the template binary image.
In order to improve the accuracy of evaluating the exercise level of the handwriting exercise machine, the invention integrates the pen-carrying gesture, writing, pen pressure and writing speed of the handwriting exercise machine when the handwriting exercise machine performs handwriting copying exercise, and evaluates the exercise level of the handwriting exercise machine according to the consistency of the pen-carrying gesture, writing, pen pressure and writing speed of the handwriting teaching machine when the handwriting teaching machine performs teaching; the number of frame images contained in the handwriting exercise video and the handwriting teaching video respectively represents the time for completing handwriting exercise and handwriting teaching, so that the writing speeds of a handwriting exerciser and a handwriting learner can be reflected; the pressure data of the pixel points at each position on the handwriting word graph and the template word graph represent the pen pressure of the handwriting exerciser and the handwriting teaching staff; therefore, the invention combines the frame image quantity contained in the handwriting exercise video and the handwriting teaching video, the pressure data of the pixel points at each position on the handwriting word graph and the template word graph, and the similarity of the pen-handling gesture and the handwriting matching degree to obtain the evaluation result of the exercise level of the handwriting exerciser.
Specifically, determining a handwriting horizontal evaluation value of a handwriting exerciser according to the similarity of pen-handling gestures, the handwriting matching degree, the difference of the number of frame images contained in handwriting exercise videos and handwriting teaching videos, and the difference of pressure data of each first pixel point in a handwriting binary image and a corresponding first pixel point in a template binary image; the handwriting level evaluation value of the handwriting exerciser satisfies the expression:
；
Wherein S represents the handwriting level, G represents the pen-handling gesture similarity, P represents the handwriting matching degree, K represents the number of frame images contained in the handwriting exercise video, A represents the number of frame images contained in the handwriting teaching video, j represents the serial number of the first pixel point, Pressure data representing the j first pixel point in the handwriting binary image,Representing pressure data of a j-th first pixel point in the handwriting binary image in a corresponding first pixel point in the template binary image, N represents the number of the first pixel points in the handwriting binary image,The representation takes the absolute value of the value,An exponential function based on a natural constant is represented.
Note that, the difference in the number of frame images included in the handwriting exercise video and the handwriting teaching videoThe smaller the writing speed is, the more consistent the writing speeds of the handwriting exerciser and the handwriting learner are, and the difference between the pressure data of each first pixel point in the handwriting binary image and the corresponding first pixel point in the template binary image isThe smaller the pen pressure is, the more consistent the handwriting exerciser and the handwriting teaching student are when copying; the higher the similarity of the pen-handling posture and the handwriting matching degree, and the more consistent the writing speeds and the pen pressures of the handwriting trainers and the handwriting trainers, the higher the handwriting horizontal evaluation value of the handwriting trainers can reach the handwriting level of the handwriting trainers, namely the higher the handwriting horizontal evaluation value of the handwriting trainers.
S5, controlling the digital soft pen handwriting teaching tool to recommend teaching resources for the handwriting exerciser according to the handwriting level evaluation value of the handwriting exerciser.
Specifically, the handwriting exerciser performs copying exercise for one Chinese character at a time, for any one Chinese character:
(1) When the handwriting level of the handwriting exerciser is smaller than a preset first threshold value during exercise of the Chinese character, the handwriting exerciser is not mastered to copy the handwriting of the Chinese character, and the Chinese character still needs to be exercised at the moment, so that the digital soft pen handwriting teaching tool is controlled to continuously recommend the handwriting teaching video and the template word graph of the Chinese character for the handwriting exerciser;
(2) When the handwriting level of the handwriting trainer is not less than a preset first threshold value and not more than a preset second threshold value during the training of the Chinese character, the handwriting trainer is informed that part of the handwriting of the Chinese character is copied, and at the moment, the training of other Chinese characters of the type can be tried, so that the digital soft pen handwriting teaching tool is controlled to recommend handwriting teaching videos and template word patterns of other Chinese characters in the combination of the Chinese character for the handwriting trainer;
(3) When the handwriting level of the handwriting trainer during the training of the Chinese character is larger than a preset second threshold value, the handwriting trainer is fully mastered to copy the handwriting of the Chinese character, and other Chinese characters of other types can be tried to be trained at the moment, so that the digital soft pen handwriting teaching tool is controlled to recommend handwriting teaching videos and template word patterns of the Chinese characters in other combinations except the combination of the Chinese characters for the handwriting trainer.
The combination is divided by related handwriting specialists according to the structure of the Chinese characters and the writing difficulty.
The embodiment of the invention also discloses a teaching aid control system based on the local area network, which comprises a processor and a memory, wherein the memory stores computer program instructions, and the teaching aid control method based on the local area network is realized when the computer program instructions are executed by the processor.
The above system further comprises other components well known to those skilled in the art, such as a communication bus and a communication interface, the arrangement and function of which are known in the art and therefore are not described in detail herein.
In the description of the present specification, the meaning of "a plurality", "a number" or "a plurality" is at least two, for example, two, three or more, etc., unless explicitly defined otherwise.
While various embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Many modifications, changes, and substitutions will now occur to those skilled in the art without departing from the spirit and scope of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention.
Claims (9)
1. The teaching aid control method based on the local area network is characterized by comprising the following steps of:
Collecting handwriting exercise videos and handwriting word graphs of handwriting exercisers, handwriting teaching videos and template word graphs of handwriting teaching students and pressure data of pixel points at all positions on the handwriting word graphs and the template word graphs through a digital soft pen handwriting teaching tool;
Human body posture detection is carried out on each frame of images of the handwriting exercise video and the handwriting teaching video, and attention posture points and reference points in each frame of images are obtained; determining the relative position characteristics of each concerned gesture point in each frame image according to the position relation between each concerned gesture point and the reference point in each frame image; determining the similarity of pen moving gestures according to the distances between the relative position features of all the concerned gesture points in all the frame images of the handwriting exercise video and the relative position features of all the concerned gesture points in the frame images of the handwriting teaching video;
obtaining a binary image corresponding to a handwriting word graph and a binary image corresponding to a template word graph through an Ojin method, respectively serving as the handwriting binary image and the template binary image, matching all first pixel points in the handwriting binary image with all first pixel points in the template binary image according to neighborhood characteristics and coordinates of first pixel points in the handwriting binary image and the template binary image, and determining handwriting matching degree according to a matching result;
Determining pressure data of each first pixel point in the handwriting binary image and the template binary image according to the pressure data of the pixel points at each position on the handwriting word graph and the template word graph;
Determining a handwriting horizontal evaluation value of a handwriting exerciser according to the similarity of the pen-handling gesture, the handwriting matching degree, the difference of the number of frame images contained in the handwriting exercise video and the handwriting teaching video and the difference of pressure data of each first pixel point in the handwriting binary image and each first pixel point in the template binary image;
and controlling the digital soft pen handwriting teaching tool to recommend teaching resources for the handwriting trainer according to the handwriting level evaluation value of the handwriting trainer.
2. The teaching aid control method based on the local area network according to claim 1, wherein the handwriting level evaluation value of the handwriting trainer satisfies the expression:
；
Wherein S represents the handwriting level, G represents the pen-handling gesture similarity, P represents the handwriting matching degree, K represents the number of frame images contained in the handwriting exercise video, A represents the number of frame images contained in the handwriting teaching video, j represents the serial number of the first pixel point, Pressure data representing the j first pixel point in the handwriting binary image,Representing pressure data of a j-th first pixel point in the handwriting binary image in a corresponding first pixel point in the template binary image, N represents the number of the first pixel points in the handwriting binary image,The representation takes the absolute value of the value,An exponential function based on a natural constant is represented.
3. The method according to claim 2, wherein determining the similarity of the pen-carrying gesture according to the distance between the relative position features of each focus gesture point in all frame images of the handwriting exercise video and the relative position features of each focus gesture point in the frame images of the handwriting teaching video comprises:
For any concerned gesture point, a sequence formed by the relative position features of the concerned gesture point in all frame images of the handwriting exercise video according to time sequence is used as a first feature sequence of the concerned gesture point; the relative position features of the concerned gesture points in all frame images of the handwriting teaching video are used as a second feature sequence of the concerned gesture points according to a sequence formed by time sequences;
The similarity of pen-handling gestures satisfies the expression:
；
wherein G represents the similarity of pen-carrying gestures, A first feature sequence representing an ith pose point of interest,A second feature sequence representing an ith pose point of interest, i representing a sequence number of the pose point of interest,Representing DTW distance.
4. The teaching aid control method based on the local area network according to claim 3, wherein determining the relative position feature of each focus posture point in each frame image according to the position relation between each focus posture point in each frame image and the reference point comprises:
The relative position characteristics of each concerned gesture point in each frame image of the handwriting exercise video satisfy the expression:
；
In the method, in the process of the invention, Representing the relative position characteristics of the ith concerned gesture point in the kth frame image of the handwriting exercise video, K representing the sequence number of the frame image of the handwriting exercise video, K taking all integers in the range of [1, K ], K representing the number of the frame images contained in the handwriting exercise video, i representing the sequence number of the concerned gesture point, i taking all integers in the range of [1,5],Representing the abscissa of the ith pose point of interest in the kth frame of image of the handwriting exercise video,Representing the ordinate of the ith pose point of interest in the kth frame of image of the handwriting exercise video,The abscissa representing the reference point in the kth frame image of the handwriting exercise video,Representing the ordinate of the reference point in the kth frame image of the handwriting exercise video,Representing an arctangent function;
The relative position characteristics of each concerned gesture point in each frame image of the handwriting teaching video satisfy the expression:
；
In the method, in the process of the invention, Representing the relative position characteristics of the ith concerned gesture point in the a-frame image of the handwriting teaching video, a represents the sequence number of the frame image of the handwriting teaching video, a takes all integers in the range of [1, A ], A represents the number of the frame images contained in the handwriting teaching video,Representing the abscissa of the ith focus gesture point in the a-frame image of the handwriting teaching video,Representing the ordinate of the ith focus gesture point in the a-frame image of the handwriting teaching video,Representing the abscissa of the reference point in the a-th frame image of the handwriting teaching video,And the ordinate of the reference point in the a-frame image of the handwriting teaching video is represented.
5. The teaching aid control method based on the local area network according to claim 2, wherein the handwriting matching degree satisfies an expression:
；
wherein P represents the matching degree of handwriting, Representing the neighborhood characteristic of the j first pixel point in the handwriting binary image,Representing the neighborhood characteristics of the jth first pixel point in the handwriting binary image in the corresponding first pixel point in the template binary image,Representing the Euclidean distance between the j first pixel point in the handwriting binary image and the corresponding first pixel point in the template binary image, wherein N represents the number of the first pixel points in the handwriting binary image;
the first pixel point is a pixel point with a gray value of 0 in the binary image.
6. The method for controlling a teaching aid based on a local area network according to claim 2, wherein the controlling the digital soft pen handwriting teaching aid to recommend teaching resources for the handwriting exerciser according to the handwriting level evaluation value of the handwriting exerciser comprises:
The handwriting exerciser performs copying exercise for one Chinese character each time, and teaching resources of the Chinese character comprise handwriting teaching videos and template word graphs of the Chinese character, and for any Chinese character:
(1) When the handwriting level of the handwriting exerciser is smaller than a preset first threshold value during exercise of the Chinese character, controlling the digital soft pen handwriting teaching aid to continuously recommend the handwriting teaching video and the template word graph of the Chinese character for the handwriting exerciser;
(2) When the handwriting level of the handwriting trainer is not less than a preset first threshold value and not more than a preset second threshold value during the training of the Chinese character, controlling the digital soft pen handwriting teaching tool to recommend the handwriting teaching video and the template word diagram of other Chinese characters in the combination of the Chinese character for the handwriting trainer;
(3) When the handwriting level of the handwriting exerciser is greater than a preset second threshold value during exercise of the Chinese character, controlling the digital soft pen handwriting teaching tool to recommend the handwriting teaching video and the template word graph of the Chinese character in other combinations except the combination where the Chinese character is located for the handwriting exerciser.
7. The method for controlling teaching aid based on local area network according to claim 4, wherein the detecting human body posture of each frame of image of handwriting exercise video and handwriting teaching video to obtain the attention posture point and reference point in each frame of image comprises:
Respectively carrying out human body posture detection on each frame of images of the handwriting exercise video and the handwriting teaching video through MEDIAPIPE POSE models to obtain all posture points in each frame of images of the handwriting exercise video and the handwriting teaching video;
For all the gesture points in any frame image, the gesture point corresponding to the shoulder on the right side of the human body is used as a reference point in the frame image, and the gesture points corresponding to the elbow on the right side of the human body, the wrist on the right side of the human body and the hand on the right side of the human body are used as concerned gesture points in the frame image.
8. The method for controlling teaching aid based on local area network according to claim 1, wherein the collecting, by the digital soft pen handwriting teaching aid, handwriting exercise video and handwriting word graph of a handwriting exerciser, handwriting teaching video and template word graph of a handwriting learner, and pressure data of pixel points at each position on the handwriting word graph and the template word graph comprises:
the digital soft pen handwriting teaching tool comprises a high-speed shooting instrument, a capacitance pen and a writing board;
Collecting a handwriting teaching video of a handwriting teaching person through a high-speed photo instrument when the handwriting teaching person performs teaching display, and collecting a template word graph of the handwriting teaching person through the high-speed photo instrument when the teaching is finished;
when a handwriting exerciser exercises, acquiring handwriting exercise videos of the handwriting exerciser through a high-speed shooting instrument, and acquiring handwriting word graphs of the handwriting exerciser through the high-speed shooting instrument when the exercise is finished;
When a handwriting learner performs teaching on the writing board, recording the pressure from the capacitance pen received by each position on the writing board as pressure data of pixel points of each position on a template word graph;
when a handwriting exerciser exercises on the writing board, the pressure from the capacitance pen on each position on the writing board is recorded and used as pressure data of pixel points on each position on a handwriting chart.
9. A teaching aid control system based on a local area network, comprising: a processor and a memory storing computer program instructions which, when executed by the processor, implement a method of controlling a lan-based teaching aid according to any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410529634.XA CN118116009B (en) | 2024-04-29 | 2024-04-29 | Teaching aid control method and system based on local area network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410529634.XA CN118116009B (en) | 2024-04-29 | 2024-04-29 | Teaching aid control method and system based on local area network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118116009A CN118116009A (en) | 2024-05-31 |
| CN118116009B true CN118116009B (en) | 2024-07-05 |
Family
ID=91219641
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410529634.XA Active CN118116009B (en) | 2024-04-29 | 2024-04-29 | Teaching aid control method and system based on local area network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118116009B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112908355A (en) * | 2021-01-18 | 2021-06-04 | 江苏师范大学 | System and method for quantitatively evaluating teaching skills of teacher and teacher |
| CN114241595A (en) * | 2021-11-03 | 2022-03-25 | 橙狮体育(北京)有限公司 | Data processing method and device, electronic equipment and computer storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20240104739A1 (en) * | 2022-09-26 | 2024-03-28 | Ron Nachum | Computer vision for analyzing handwriting kinematics |
-
2024
- 2024-04-29 CN CN202410529634.XA patent/CN118116009B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112908355A (en) * | 2021-01-18 | 2021-06-04 | 江苏师范大学 | System and method for quantitatively evaluating teaching skills of teacher and teacher |
| CN114241595A (en) * | 2021-11-03 | 2022-03-25 | 橙狮体育(北京)有限公司 | Data processing method and device, electronic equipment and computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118116009A (en) | 2024-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8358320B2 (en) | Interactive transcription system and method | |
| CN103164022B (en) | Many fingers touch method and device, portable terminal | |
| CN105536205A (en) | Upper limb training system based on monocular video human body action sensing | |
| CN111626297A (en) | Character writing quality evaluation method and device, electronic equipment and recording medium | |
| CN108958636B (en) | Intelligent copybook device and copybook writing detection method | |
| Johnson et al. | Detecting hand posture in piano playing using depth data | |
| CN110718280A (en) | Fitness action accuracy determining method and device, electronic equipment and storage medium | |
| CN107146478A (en) | A kind of drills to improve one's handwriting systems, devices and methods | |
| CN104407696B (en) | The virtual ball simulation of mobile device and the method for control | |
| CN112270853A (en) | Calligraphy teaching system and method | |
| CN112699840A (en) | Writing process evaluation and feedback method, writing process evaluation and feedback system and computer storage medium | |
| CN108628455A (en) | A kind of virtual husky picture method for drafting based on touch-screen gesture identification | |
| CN111985184A (en) | Auxiliary writing font copying method, system and device based on AI vision | |
| CN118116009B (en) | Teaching aid control method and system based on local area network | |
| CN115346640B (en) | Intelligent monitoring method and system for closed-loop feedback of functional rehabilitation training | |
| CN116630992A (en) | Copybook grid text intelligent matching method and system | |
| CN109358799A (en) | The method of the hand annotation information of user's input is added on handwriting equipment | |
| CN107831894A (en) | It is a kind of suitable for mobile terminal every empty-handed gesture writing on the blackboard method | |
| CN110766997A (en) | Copy display method, device and storage medium | |
| CN114863558A (en) | Method and system for recognizing and counting human body training actions | |
| Shi | Stage performance characteristics of minority dance based on human motion recognition | |
| Gurieva et al. | Augmented reality for personalized learning technique: Climbing gym case study | |
| Albu et al. | QUALITY EVALUATION APPROACHES OF THE FIRST GRADE CHILDREN'S HANDWRITTING. | |
| Jeong et al. | Prototype of learning tool with augmented reality and natural hand interaction using depth sensing camera | |
| Leung et al. | Web-based handwriting education with animated virtual teacher |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |