WO2016117564A1 - プログラム、情報記憶媒体及び認識装置 - Google Patents
プログラム、情報記憶媒体及び認識装置 Download PDFInfo
- Publication number
- WO2016117564A1 WO2016117564A1 PCT/JP2016/051457 JP2016051457W WO2016117564A1 WO 2016117564 A1 WO2016117564 A1 WO 2016117564A1 JP 2016051457 W JP2016051457 W JP 2016051457W WO 2016117564 A1 WO2016117564 A1 WO 2016117564A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- stroke
- specific gesture
- character string
- mathematical expression
- range specified
- Prior art date
Links
- 230000014509 gene expression Effects 0.000 claims description 102
- 238000000034 method Methods 0.000 description 63
- 238000012545 processing Methods 0.000 description 32
- 238000010586 diagram Methods 0.000 description 23
- 230000006870 function Effects 0.000 description 13
- 230000002452 interceptive effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000004816 latex Substances 0.000 description 1
- 230000000149 penetrating effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012916 structural analysis Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/18—Extraction of features or characteristics of the image
- G06V30/1801—Detecting partial patterns, e.g. edges or contours, or configurations, e.g. loops, corners, strokes or intersections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
Definitions
- the present invention relates to a program, an information storage medium, and a recognition device.
- Mathematical expressions consist of the relationship between symbols and the position and size of symbols representing the four arithmetic operators, parentheses, fractional symbols and roots, integral symbols, and powers.
- a handwritten mathematical expression recognition system it is normal to assume an alphanumeric symbol as a symbol.
- a character string other than an alphanumeric symbol is often written as a symbol.
- Formulas containing these strings often appear in textbooks and are often written on a blackboard.
- the present invention has been made in view of the above problems, and an object of the present invention is to provide a program, an information storage medium, and a recognition device that can recognize mathematical expressions including character strings. is there.
- the present invention is a program for recognizing a mathematical expression including a character string from a handwritten input stroke string, and the input stroke or stroke string is a specific gesture for distinguishing a character string from a mathematical expression.
- a registration unit for registering information regarding a range specified by the specific gesture and a specified by the specific gesture among the input strokes when it is determined that the specific gesture is satisfied.
- a category in which a stroke included in the range to be set is set as a stroke corresponding to one of the character string and the mathematical expression, and a stroke not included in the range specified by the specific gesture is set as a stroke corresponding to the other of the character string and the mathematical expression Text and the stroke set as the stroke corresponding to the character string.
- the present invention also relates to an information storage medium that can be read by a computer and stores a program for causing the computer to function as each of the above-described units.
- the present invention also relates to a recognition device including the above-described units.
- the stroke included in the range specified by the specific gesture is set as a stroke corresponding to one of the character string and the mathematical expression
- a stroke that is not included in the range specified by the specific gesture is set as a stroke corresponding to the other of the character string and the mathematical expression
- the character string is recognized by the character string recognition engine from the stroke set as the stroke corresponding to the character string.
- the classification unit is designated by the specific gesture when it is determined that the input stroke or stroke sequence corresponds to the specific gesture.
- the stroke included in the range is set as a stroke corresponding to one of the character string and the mathematical expression, and it is determined that the input stroke does not correspond to the specific gesture, the input stroke is specified by the specific gesture It is possible to determine whether the input stroke is included in the range specified by the specific gesture, and the input stroke may be set as a stroke corresponding to one of the character string and the mathematical expression.
- the sorting unit sets a stroke included in a range specified by the specific gesture as a stroke corresponding to a character string. And set a stroke that is not included in the range specified by the specific gesture as a stroke corresponding to the mathematical expression, and the recognition unit uses a stroke included in the range specified by the one specific gesture as one symbol. It may be handled and a mathematical expression is recognized by a mathematical expression recognition engine.
- a stroke included in a range specified by one specific gesture (a stroke set as a stroke corresponding to a character string) is treated as one symbol, and a mathematical expression is recognized by a mathematical expression recognition engine.
- a mathematical expression including a character string can be recognized using a mathematical expression recognition engine that cannot recognize the character string.
- the computer further functions as a display control unit that performs control to display the input stroke sequence on the display unit, and the display control unit corresponds to the specific gesture. Then, control may be performed to display the determined stroke or stroke sequence as an image representing a shape corresponding to the specific gesture.
- the recognition apparatus further includes a display control unit that performs control to display the input stroke sequence on the display unit, and the display control unit displays the stroke or the stroke sequence determined to correspond to the specific gesture.
- the image may be displayed as an image representing the shape corresponding to the specific gesture.
- the present invention it is possible to cause the user to recognize that the specific gesture has been correctly determined by displaying the stroke determined to correspond to the specific gesture as an image representing the shape corresponding to the specific gesture. Convenience can be improved.
- the display control unit does not include a stroke included in a range specified by the specific gesture within a range specified by the specific gesture. You may control to display with a color different from a stroke.
- the stroke included in the range specified by the specific gesture is displayed in a color different from the stroke not included in the range specified by the specific gesture, whereby the character string and the mathematical expression are displayed by the specific gesture. It is possible to make the user recognize that it is correctly classified, and to improve the convenience for the user.
- the registration unit corresponds to the specific gesture. Then, it may be determined.
- FIG. 1 is an example of a functional block diagram of the recognition apparatus of the present embodiment.
- FIG. 2A is a diagram illustrating an example of writing in which mathematical expressions, character strings, and specific gestures are mixed.
- FIG. 2B is a diagram illustrating an example of the specific gesture table.
- FIG. 3A is a diagram for explaining the order in which specific gestures are written.
- FIG. 3B is a diagram for explaining the order in which specific gestures are written.
- FIG. 3C is a diagram for explaining the order in which the specific gesture is written.
- FIG. 3D is a diagram for explaining the order in which specific gestures are written.
- FIG. 4A is a diagram illustrating an example of writing of a rectangular specific gesture.
- FIG. 4B is a diagram illustrating an example of writing of a rectangular specific gesture.
- FIG. 5 is a flowchart illustrating an example of processing in the sequential method.
- FIG. 6A is a diagram illustrating a specific display example in the sequential method.
- FIG. 6B is a diagram illustrating a specific display example in the sequential method.
- FIG. 6C is a diagram illustrating a specific display example in the sequential method.
- FIG. 6D is a diagram illustrating a specific display example in the sequential method.
- FIG. 6E is a diagram illustrating a specific display example in the sequential method.
- FIG. 6F is a diagram illustrating a specific display example in the sequential method.
- FIG. 7 is a flowchart showing an example of processing in the batch method.
- FIG. 8A is a diagram illustrating a specific display example in the batch method.
- FIG. 8B is a diagram illustrating a specific display example in the batch method.
- FIG. 8C is a diagram illustrating a specific display example in the batch method.
- FIG. 1 shows an example of a functional block diagram of the recognition apparatus of the present embodiment.
- the recognition apparatus of this embodiment is good also as a structure which abbreviate
- the character input unit 160 is for the user to input handwritten characters with a writing medium (pen, fingertip, etc.), and the function can be realized by a writing surface such as a tablet or a touch panel.
- the character input unit 160 detects coordinate data representing the position of the writing medium from when the writing medium touches the writing surface until it leaves, and strokes (strokes) the detected coordinate data string (coordinate point series). Is output to the processing unit 100.
- a vector from the end point of the stroke to the start point of the next stroke is called an off stroke (handwriting vector), and a series of strokes and off strokes is called a stroke sequence.
- the storage unit 170 stores a program and various data for causing the computer to function as each unit of the processing unit 100, and also functions as a work area of the processing unit 100.
- the function can be realized by a hard disk, a RAM, or the like.
- the display unit 190 outputs the image generated by the processing unit 100, and its function can be realized by a display such as a touch panel, LCD, or CRT that also functions as the character input unit 160.
- the processing unit 100 performs processing such as recognition processing and display control based on the coordinate data and program from the character input unit 160.
- the processing unit 100 performs various processes using the main storage unit in the storage unit 170 as a work area.
- the functions of the processing unit 100 can be realized by hardware such as various processors (CPU, DSP, etc.), ASIC (gate array, etc.), and programs.
- the processing unit 100 includes a registration unit 110, a sorting unit 112, a recognition unit 114, and a display control unit 120.
- the registration unit 110 determines whether or not the input stroke or stroke sequence corresponds to a specific gesture for distinguishing a character string from a mathematical expression, and determines that the input gesture corresponds to the specific gesture. A process for registering information related to the range (area) specified by. The registered information is stored in the storage unit 170.
- the registration unit 110 may determine that the stroke or stroke sequence corresponds to the specific gesture.
- the sorting unit 112 sets a stroke included in the range specified by the specific gesture among the input strokes as a stroke corresponding to one of the character string and the mathematical expression, and is included in the range specified by the specific gesture.
- a stroke that does not exist is set as a stroke corresponding to the other of the character string and the mathematical expression. That is, the sorting unit 112 sets a stroke included in the range specified by the specific gesture among the input strokes as a stroke corresponding to the character string, and is not included in the range specified by the specific gesture. May be set as a stroke corresponding to the mathematical formula, or vice versa.
- the recognition unit 114 recognizes a character string by the character string recognition engine from the stroke set as a stroke corresponding to the character string by the classification unit 112, and uses the mathematical expression recognition engine from the stroke set by the classification unit 112 as a stroke corresponding to the mathematical expression.
- the process of recognizing the mathematical formula is performed.
- the classification unit 112 sets a stroke included in the range specified by the specific gesture as a stroke corresponding to a character string
- the recognition unit 114 is specified by one specific gesture and the specific gesture.
- the stroke included in the range may be handled as one symbol, and the mathematical expression may be recognized by the mathematical expression recognition engine.
- the character strings “area”, “height”, and “width” are treated as one symbol such as “x”, “y”, and “z”, respectively, and applied to the mathematical expression recognition engine. It may be recognized.
- the display control unit 120 performs control to display the input stroke sequence on the display unit 190.
- the display control unit 120 may perform control to display the stroke or the stroke sequence determined to correspond to the specific gesture as an image representing a shape corresponding to the specific gesture (for example, an image representing a rectangle). Good.
- the display control unit 120 may perform control to display a stroke included in a range specified by the specific gesture in a color different from a stroke not included in the range specified by the specific gesture.
- the display control unit 120 performs control to display a mathematical expression (recognition result) including a character string recognized by the recognition unit 114 on the display unit 190.
- a gesture for distinguishing a character string and a mathematical expression is introduced.
- This is called a specific gesture.
- the specific gesture means an operation of specifying a stroke included in a range specified by the gesture and performing a process corresponding to the stroke other than a character to be recognized or a symbol other than a mathematical expression.
- a rectangle surrounding a character string is adopted as the specific gesture. That is, when a stroke (or stroke sequence) input by handwriting forms a substantially rectangular shape, it is determined that the stroke corresponds to the specific gesture, and the character string is recognized from the stroke included in the range specified by the specific gesture. The mathematical expression is recognized from the stroke not included in the range specified by the specific gesture.
- the reason for adopting the rectangle as the shape of the specific gesture is that the rectangle does not appear in an ordinary mathematical formula, and it is natural for humans to write the rectangle to specify the range. Note that squares and rectangles may be used as symbols in certain areas of mathematics. In this case, symbols having a different shape may be adopted as specific gestures instead of rectangles.
- the specific gesture may be any shape as long as the shape is unique (easily distinguishable from other symbols and symbols) and the range indicated by the specific gesture is clear.
- the shape of the specific gesture may be one that writes one or both of the diagonal lines of the rectangle from right to left (because writing from left to right makes it indistinguishable from “X”), or both of the diagonal lines of the rectangle What is continuously written (a shape in which “ ⁇ ” is inverted) may be used.
- a specific gesture is used to distinguish a character string included in a mathematical expression, but a specific gesture (for example, a rectangle surrounding the mathematical expression) may be used to distinguish a mathematical expression included in the character string.
- FIG. 2A is a diagram illustrating an example of writing in which mathematical expressions, character strings, and specific gestures are mixed.
- the registration unit 110 monitors whether or not the stroke is a specific gesture.
- the registration unit 110 determines the stroke order of the specific gesture (the stroke of the stroke corresponding to the specific gesture). (Order) and coordinates (rectangular rectangular coordinates or rectangular opposing rectangular coordinates) are registered in the specific gesture table.
- FIG. 2B is a diagram showing a specific gesture table corresponding to the writing example of FIG. 2A. For example, in the specific gesture table shown in FIG.
- the specific gesture BG 1 is written in the fifth stroke after the 4-stroke character string “area”, and the coordinates of the upper left corner and the lower right corner (with the specific gesture).
- An example of information relating to the designated range) is registered, and for the specific gesture BG 2 is written in the 13th screen after the first “x”, and the coordinates of its upper left corner and lower right corner are registered,
- the specific gesture BG 3 is written in the 18th screen after the second “x”, and the coordinates of its upper left corner and lower right corner are registered.
- the classification unit 112 refers to the specific gesture table as illustrated in FIG. 2B, classifies the stroke surrounded by any specific gesture as a part of the character string (stroke corresponding to the character string), and other than that The stroke is divided as a part of the formula (stroke corresponding to the formula).
- the case where the specific gesture is written in one stroke is shown, but it may be allowed to write the specific gesture in multiple strokes.
- the maximum number of strokes for example, 4 strokes
- the stroke corresponding to a mathematical expression or character is written from the start of writing the specific gesture to the end of writing.
- the stroke order of the first stroke and the stroke order of the last stroke constituting the specific gesture are registered in the specific gesture table.
- the “adjusted stroke order” in the specific gesture table of FIG. 2B will be described later.
- FIG. 3A shows a case where a specific gesture BG is first written, and then a character string is written in the specific gesture BG.
- FIG. 3B first writes a character string, and then,
- FIG. 3C shows a case where a specific gesture BG surrounding the character string is written, and
- FIG. 3A shows a case where a specific gesture BG is first written, and then a character string is written in the specific gesture BG.
- FIG. 3C shows a case where the character string is first written and then another stroke is written, and then the specific gesture BG surrounding the character string is written.
- FIG. 3D shows a case where a character string is added to a specific gesture BG surrounding a character string already written.
- Fig. 4 shows an example of writing a rectangular specific gesture.
- the user starts writing from the upper left corner of the rectangle, proceeds clockwise (see FIG. 4A) or counterclockwise (see FIG. 4B), and returns to the upper left corner.
- the specific gesture includes at least a part (for example, 1/3 or more) of brushstrokes constituting the input stroke
- the sorting unit 112 is surrounded by the specific gesture (specified by the specific gesture). To be included in the range to be included. This is because the specific gesture written by the user may cross another stroke (see FIG. 3).
- an online recognition method that uses time-series information of a stroke and an offline recognition method that uses a feature as an image of a stroke are used together to recognize the specific gesture.
- the recognition target when recognizing a specific gesture is a mathematical expression (a symbol such as an English character, an arithmetic operator, an operator such as a fractional symbol, a root or an integral symbol, a parenthesis) and a specific gesture (a surrounding rectangle).
- MRF Markov Random Field
- MQDF Modified Quadrantic Discriminant Function
- the stroke can be easily converted into an image by connecting the strokes in chronological order. Although the time-series information is lost when the stroke is converted into an image, the influence of the stroke order difference, overwriting, or the like can be eliminated by converting it into an image.
- Gradient features are extracted from this image by the Sobel filter and projected in eight directions.
- the original image is 64 ⁇ 64 pixels and is divided into 8 ⁇ 8 sections of the same size.
- a Gaussian filter of 18 ⁇ 18 pixels is applied to blur.
- the direction feature close to the center greatly contributes, the contribution of the periphery becomes small, and some contribution is also received from the adjacent section, which makes it strong against misalignment.
- features in 8 directions are extracted for each 8 ⁇ 8 section, and 512-dimensional features are obtained as a whole.
- dimensions are compressed by Fisher's discriminant analysis to obtain 256-dimensional features.
- MQDF is used for identification.
- a threshold value is set, and if the distance to the rectangle of the input pattern is the shortest, it is rejected if it is larger than the threshold value.
- the online recognition method gives a logarithmic value of a probability value (a negative value, but a larger value is better), and the offline recognition method gives a distance (a smaller value is better). Therefore, it is necessary to match these scales. Therefore, the integrated score is calculated by multiplying the online recognition score by ⁇ 1 to make it a positive value (thus, the smaller the better) and taking the weighted sum.
- the integrated score score combination is calculated by the following equation.
- score online and score offline are the score of the online recognition method and the score of the offline recognition method, respectively.
- the registration unit 110 recognizes the smallest gesture combination as the specific gesture among the recognition targets.
- the recognition device is equipped with an interactive device that displays time-series writing points in real time and provides recognition feedback for each stroke (stroke), as in a tablet PC, and a memory for time-series writing points.
- an interactive device that displays time-series writing points in real time and provides recognition feedback for each stroke (stroke), as in a tablet PC, and a memory for time-series writing points.
- a non-interactive device in which recording is performed with a tablet or an electronic pen, and reading and processing of the data is performed collectively by a PC or the like.
- Sequential processing In the sequential method adopted in interactive devices, the user designates the language (for example, English or Japanese) for recognizing the character string, starts up the system (program), and enters the input section such as a touch panel. Start writing mathematical formulas and strings. The system recognizes a specific gesture and classifies a character string and a mathematical expression each time a stroke is written, and recognizes a character string and a numerical string and displays a recognition result after writing.
- FIG. 5 is a flowchart illustrating an example of processing in the sequential method.
- the processing unit 100 acquires a stroke input to the character input unit 160 (step S10).
- the display control unit 120 performs control to display the acquired stroke on the display unit 190.
- the registration unit 110 determines whether or not the latest stroke up to the maximum number of strokes (for example, 4 strokes) of the specific gesture corresponds to the specific gesture (configures the specific gesture) from the acquired stroke (Ste S12). For example, the registration unit 110 calculates an integrated score score combination for each of a recent stroke of one stroke, a stroke sequence of two strokes, a stroke sequence of three strokes, a stroke sequence of four strokes (maximum number of strokes), When the specific combination gives the smallest score combination among the recognition targets, it is recognized as a specific gesture. However, for a stroke rejected by either the online recognition method or the offline recognition method, the score combination is not calculated, and if any stroke is rejected, it is determined that the stroke does not correspond to the specific gesture.
- the registration unit 110 specifies the information of the specific gesture (stroke order and opposing two-dimensional coordinates) from the stroke information determined to be applicable to the specific gesture. Register in the gesture table (step S14).
- the registration unit 110 cancels the setting when the stroke determined to correspond to the specific gesture has been previously set as a stroke corresponding to the mathematical expression.
- the display control unit 120 controls to display an image of a stroke determined to be a specific gesture as an image representing a rectangle (a shape corresponding to the specific gesture) of a color (for example, red) different from other strokes. I do. Thereby, it can be fed back to the user that the specific gesture has been correctly recognized.
- the sorting unit 112 refers to the specific gesture table, and sets a stroke included in a range specified by the registered specific gesture as a stroke corresponding to the character string (step S16).
- the display control unit 120 performs control to change the color of the stroke included in the range specified by the specific gesture (for example, change from black to blue). Thereby, it can be fed back to the user that the character string has been correctly classified by the specific gesture. If there is no stroke included in the range specified by the specific gesture, the process of step S16 is skipped.
- the sorting unit 112 refers to the specific gesture table, and determines whether or not the acquired stroke is included in the range specified by any specific gesture. Is determined (step S18).
- the sorting unit 112 sets the acquired stroke as a stroke corresponding to the character string (Step S18).
- the display control unit 120 performs control to change the color of the acquired stroke (for example, change from black to blue).
- the sorting unit 112 sets the acquired stroke as a stroke corresponding to the mathematical expression (Step S18). S22).
- step S24 the processing unit 100 determines whether or not there is an input of a stroke (step S24). If there is an input (Y in step S24), the processing unit 100 proceeds to the process of step S10, and writing by the user is completed. Until this is done, the processing from step S10 is repeated.
- the recognition unit 114 recognizes the character string by the character string recognition engine from the stroke set as the stroke corresponding to the character string in steps S16 and S20.
- the mathematical expression recognition engine recognizes the mathematical expression from the stroke set as the stroke corresponding to the mathematical expression in step S22 (step S26). Then, the display control unit 120 performs control for causing the display unit 190 to display the recognition result (the mathematical expression including the character string).
- the mathematical expression recognition engine does not understand the stroke corresponding to the specific gesture or the stroke corresponding to the character string, but there is no information on the character string surrounded by the specific gesture (that is, from the stroke information corresponding to the mathematical expression). Alone), the structure of the formula cannot be analyzed. Therefore, in the method of the present embodiment, one specific gesture and a character string surrounded by the specific gesture are treated as one symbol (one symbol in the mathematical expression), and the coordinate information and mathematical expression of the specific gesture handled as the symbol are supported. Recognize mathematical formulas from stroke information. Therefore, the stroke order of the specific gesture is adjusted.
- the earlier stroke order of the specific gesture is set as the stroke order after adjustment of the specific gesture.
- “1”, “9”, and “12” are registered as the adjusted stroke order of the specific gestures BG 1 , BG 2 , and BG 3 , respectively.
- the recognition unit 114 treats the specific gesture BG 1 and the character string “area” included in the specific gesture BG 1 as one symbol written in the first stroke by referring to the specific gesture table,
- the specific gesture BG 2 and the character string “height” included therein are treated as one symbol written in the ninth stroke, and the specific gesture BG 3 and the character string “width” included therein are written in the 12th stroke.
- the structure of the mathematical formula can be analyzed from the relationship between the position and size of these symbols and the stroke corresponding to the mathematical formula.
- the user may make a mistake in writing, so it is desirable to have an Undo function.
- the specific gesture and its registration information are deleted and included in the range specified by the specific gesture.
- the stroke (the stroke set as the stroke corresponding to the character string) is reset as the stroke corresponding to the mathematical expression.
- the latest undo stroke is a stroke corresponding to a character string or a mathematical expression, the stroke is simply deleted.
- FIG. 6 is a diagram illustrating a specific display example in the sequential method.
- the user first writes a character string “value” and a mathematical expression, and then writes a specific gesture BG 1 surrounding the character string “value” (FIG. 6A).
- the registration unit 110 recognizes the specific gesture BG 1 , and the display control unit 120 displays the specific gesture BG 1 in red as an image BI 1 (specific gesture image) that represents a rectangle having the same size as the specific gesture BG 1. It was replaced to change the color of a character string enclosed by a particular gesture BG 1 "value" in blue (Fig. 6B).
- the user writes a new specific gesture BG 2 (FIG. 6C).
- the registration unit 110 recognizes the specific gesture BG 2 and the display control unit 120 replaces the specific gesture BG 2 with the specific gesture image BI 2 (FIG. 6D).
- the display control unit 120 displays the character string “result” in blue every time a stroke is acquired (FIG. 6E).
- the recognition unit 114 recognizes the character string and the mathematical expression, and the display control unit 120 displays the recognition result (FIG. 6F).
- the specific gesture images BI 1 and BI 2 are removed and the recognition result is displayed.
- FIG. 7 is a flowchart showing an example of processing of the latter method in the batch method.
- the processing unit 100 acquires the stroke sequence input to the character input unit 160 (step S30). Next, the processing unit 100 sets 1 to the variable i (step S32). Next, the registration unit 110 determines whether or not the most recent stroke from the i-th (i-th stroke) stroke to the maximum number of strokes (for example, 4 strokes) of the specific gesture corresponds to the specific gesture (step) S34).
- the registration unit 110 specifies the information of the specific gesture (stroke order and opposing two-dimensional coordinates) from the stroke information determined to be applicable to the specific gesture. Register in the gesture table (step S36). If it does not correspond to the specific gesture (N in step S34), the i-th stroke is set as a non-gesture stroke (stroke not corresponding to the specific gesture) (step S38).
- the processing unit 100 increases the value of the variable i by 1 (step S40), and whether or not the value of the variable i is n (n is the total number of strokes included in the acquired stroke sequence) or not. Is determined (step S42). When the value of the variable i is n or less (Y in step S42), the process proceeds to step S34, and the processing from step S34 is repeated until the value of the variable i exceeds n.
- the processing unit 100 sets 1 to the variable i (step S44).
- the sorting unit 112 refers to the specific gesture table to determine whether or not the i-th non-gesture stroke set in step S38 is included in a range specified by any specific gesture (Ste S46).
- the sorting unit 112 sets the i-th non-gesture stroke as a stroke corresponding to the character string. Set (step S48).
- the sorting unit 112 sets the i-th non-gesture stroke as a stroke corresponding to the mathematical expression. (Step S50).
- the processing unit 100 increases the value of the variable i by 1 (step S52), and determines whether or not the value of the variable i is equal to or less than m (m is the total number of non-gesture strokes) (step S54). ). When the value of the variable i is less than or equal to m (Y in step S54), the process proceeds to step S46, and the processes in and after step S46 are repeated until the value of the variable i exceeds m.
- the recognition unit 114 recognizes the character string by the character string recognition engine from the stroke set as the stroke corresponding to the character string in step S48,
- the mathematical expression is recognized by the mathematical expression recognition engine from the specific gesture table (coordinate information of the specific gesture treated as a symbol) and the stroke set as the stroke corresponding to the mathematical expression in step S50 (step S56).
- the display control unit 120 performs control for causing the display unit 190 to display the recognition result (the mathematical expression including the character string).
- FIG. 8 is a diagram illustrating a specific display example in the batch method.
- all strokes written by the user and taken into the recognition device are displayed (FIG. 8A), and when the user performs an operation to instruct execution of recognition, recognition of a specific gesture, classification of character strings and mathematical expressions, The recognition of the character string and the numerical sequence is executed, and the recognition result is displayed (FIG. 8C).
- FIG. 8B After performing recognition of a specific gesture and classification of a character string and a mathematical expression, and before performing recognition of a character string and a numerical sequence, as shown in FIG. 8B, the recognition result of the specific gesture and the classification of the mathematical expression and the character string The result may be displayed.
- FIG. 8A all strokes written by the user and taken into the recognition device
- FIG. 8C the recognition result is displayed.
- FIG. 8B after performing recognition of a specific gesture and classification of a character string and a mathematical expression, and before performing recognition of a character string and a numerical sequence, as shown in FIG. 8B,
- the specific gestures BG 1 , BG 2 , and BG 3 are replaced with the specific gesture images BI 1 , BI 2 , and BI 3 , respectively, and the characters surrounded by the specific gestures BG 1 , BG 2 , and BG 3 are used.
- the colors of the columns “area”, “height”, and “width” are changed to blue.
- a specific gesture for dividing a character string and a mathematical expression is introduced, a stroke surrounded by the specific gesture is divided into a character string, and a stroke not surrounded by the specific gesture is divided into a mathematical expression,
- the character string recognition engine and the mathematical expression recognition engine By recognizing each by the character string recognition engine and the mathematical expression recognition engine, it is possible to clearly distinguish the character string and the mathematical expression included in the mathematical expression. Further, when recognizing a mathematical expression, by treating the specific gesture and the stroke surrounded by the specific gesture as one symbol, the structural analysis of the mathematical expression including the character string can be performed using the mathematical expression recognition engine.
- the stroke surrounded by the specific gesture is processed into a character string (step S16 in FIG. 5), and the stroke is also written.
- the process of dividing the stroke into a character string is performed (step S20 in FIG. 5), so that the character string and the mathematical expression are divided regardless of the timing at which the specific gesture is written. It is possible to improve user convenience. That is, according to the present embodiment, when writing a character string after writing a specific gesture (FIG. 3A), writing a specific gesture after writing a character string (FIG. 3B, FIG. 3C), Even when a character string is added to the specific gesture (FIG. 3D), the process of dividing the character string and the mathematical expression can be performed.
- the present invention includes configurations that are substantially the same as the configurations described in the embodiments (for example, configurations that have the same functions, methods, and results, or configurations that have the same objects and effects).
- the invention includes a configuration in which a non-essential part of the configuration described in the embodiment is replaced.
- the present invention includes a configuration that exhibits the same operational effects as the configuration described in the embodiment or a configuration that can achieve the same object.
- the invention includes a configuration in which a known technique is added to the configuration described in the embodiment.
- processing unit 110 registration unit, 112 sorting unit, 114 recognition unit, 120 display control unit, 160 character input unit, 170 storage unit, 190 display unit
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Character Discrimination (AREA)
Abstract
Description
図1に本実施形態の認識装置の機能ブロック図の一例を示す。なお本実施形態の認識装置は図1の構成要素(各部)の一部を省略した構成としてもよい。
次に本実施形態の手法について図面を用いて説明する。
次に、本実施形態の認識装置の処理の流れについて説明する。認識装置には、タブレット型PCのように、時系列の筆点を実時間で表示して認識のフィードバックを筆画(ストローク)毎に行える対話型装置と、時系列の筆点をメモリを備えたタブレットや電子ペンで記録し、そのデータの読み込みと処理はPC等で一括してなされる非対話型装置がある。
対話型装置で採用される逐次方式では、ユーザは、文字列を認識する言語(例えば、英語か日本語)を指定してシステム(プログラム)を立ち上げ、タッチパネル等の入力部に数式や文字列を筆記し始める。システムは、特定ジェスチャの認識、文字列及び数式の区分をストロークが筆記される毎に行い、筆記終了後に、文字列及び数列の認識と認識結果の表示を行う。図5は、逐次方式における処理の一例を示すフローチャートである。
非対話型装置で採用される一括方式では、ユーザは、システムを立ち上げ、タブレット等のインターフェースに数式や文字列を筆記する。そして、筆記後に、ストローク列のデータをPC等(認識装置)に読み込ませ、その際に、文字列を認識する言語を指定する。一括方式では、特定ジェスチャの認識、文字列及び数式の区分、文字列及び数列の認識と認識結果の表示の3つの処理は、全てのストローク列のデータがPC等に送られてから実行される。このとき、入力されたストロークを順次処理する際に、逐次方式と同様の処理を行うこともできる。また、一括方式では、特定ジェスチャの認識と、文字列及び数式の区分とを実時間で行う必要がないことから、全ての特定ジェスチャを認識した後に、それ以外のストロークについて文字列と数式とに区分し、最後にそれらを認識する方法を採用してもよい。図7は、一括方式における後者の方法の処理の一例を示すフローチャートである。
Claims (8)
- 手書き入力されたストローク列から文字列を含む数式を認識するためのプログラムであって、
入力されたストローク又はストローク列が、文字列と数式とを区分するための特定ジェスチャに該当するか否かを判定し、前記特定ジェスチャに該当すると判定した場合に、前記特定ジェスチャで指定される範囲に関する情報を登録する登録部と、
入力されたストロークのうち、前記特定ジェスチャで指定される範囲に含まれるストロークを文字列及び数式の一方に対応するストロークとして設定し、前記特定ジェスチャで指定される範囲に含まれないストロークを文字列及び数式の他方に対応するストロークとして設定する区分部と、
文字列に対応するストロークとして設定されたストロークから文字列認識エンジンにより文字列を認識し、数式に対応するストロークとして設定されたストロークから数式認識エンジンにより数式を認識する認識部としてコンピュータを機能させることを特徴とするプログラム。 - 請求項1において、
前記区分部は、
入力されたストローク又はストローク列が前記特定ジェスチャに該当すると判定された場合に、当該特定ジェスチャで指定される範囲に含まれるストロークを文字列及び数式の一方に対応するストロークとして設定し、入力されたストロークが前記特定ジェスチャに該当しないと判定された場合に、当該入力されたストロークが前記特定ジェスチャで指定される範囲に含まれるか否かを判定し、前記特定ジェスチャで指定される範囲に含まれる場合に、当該入力されたストロークを文字列及び数式の一方に対応するストロークとして設定することを特徴とするプログラム。 - 請求項1又は2において、
前記区分部は、
入力されたストロークのうち、前記特定ジェスチャで指定される範囲に含まれるストロークを文字列に対応するストロークとして設定し、前記特定ジェスチャで指定される範囲に含まれないストロークを数式に対応するストロークとして設定し、
前記認識部は、
1つの前記特定ジェスチャで指定される範囲に含まれるストロークを1つのシンボルとして扱って、数式認識エンジンにより数式を認識することを特徴とするプログラム。 - 請求項1乃至3のいずれか1項において、
入力されたストローク列を表示部に表示させる制御を行う表示制御部として更にコンピュータを機能させ、
前記表示制御部は、
前記特定ジェスチャに該当すると判定されたストローク又はストローク列を、前記特定ジェスチャに対応する形状を表す画像として表示させる制御を行うことを特徴とするプログラム。 - 請求項4において、
前記表示制御部は、
前記特定ジェスチャで指定される範囲に含まれるストロークを、前記特定ジェスチャで指定される範囲に含まれないストロークとは異なる色で表示させる制御を行うことを特徴とするプログラム。 - 請求項1乃至5のいずれか1項において、
前記登録部は、
入力されたストローク又はストローク列が略矩形を形成する場合に、当該ストローク又はストローク列が前記特定ジェスチャに該当すると判定することを特徴とするプログラム。 - コンピュータ読み取り可能な情報記憶媒体であって、請求項1乃至6のいずれか1項のプログラムを記憶したことを特徴とする情報記憶媒体。
- 手書き入力されたストローク列から文字列を含む数式を認識する認識装置であって、
入力されたストローク又はストローク列が、文字列と数式とを区分するための特定ジェスチャに該当するか否かを判定し、前記特定ジェスチャに該当すると判定した場合に、前記特定ジェスチャで指定される範囲に関する情報を登録する登録部と、
入力されたストロークのうち、前記特定ジェスチャで指定される範囲に含まれるストロークを文字列及び数式の一方に対応するストロークとして設定し、前記特定ジェスチャで指定される範囲に含まれないストロークを文字列及び数式の他方に対応するストロークとして設定する区分部と、
文字列に対応するストロークとして設定されたストロークから文字列認識エンジンにより文字列を認識し、数式に対応するストロークとして設定されたストロークから数式認識エンジンにより数式を認識する認識部とを含むことを特徴とする認識装置。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201680006635.XA CN107209862B (zh) | 2015-01-21 | 2016-01-19 | 识别装置和信息存储介质 |
JP2016570658A JP6694638B2 (ja) | 2015-01-21 | 2016-01-19 | プログラム、情報記憶媒体及び認識装置 |
KR1020177018253A KR101909485B1 (ko) | 2015-01-21 | 2016-01-19 | 프로그램, 정보기억매체 및 인식장치 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2015009207 | 2015-01-21 | ||
JP2015-009207 | 2015-01-21 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2016117564A1 true WO2016117564A1 (ja) | 2016-07-28 |
Family
ID=56417105
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2016/051457 WO2016117564A1 (ja) | 2015-01-21 | 2016-01-19 | プログラム、情報記憶媒体及び認識装置 |
Country Status (4)
Country | Link |
---|---|
JP (1) | JP6694638B2 (ja) |
KR (1) | KR101909485B1 (ja) |
CN (1) | CN107209862B (ja) |
WO (1) | WO2016117564A1 (ja) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019191815A (ja) * | 2018-04-23 | 2019-10-31 | 国立大学法人東京農工大学 | プログラム、情報記憶媒体及びクラスタリング装置 |
CN111539383A (zh) * | 2020-05-22 | 2020-08-14 | 浙江蓝鸽科技有限公司 | 公式知识点识别方法及装置 |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111079622A (zh) * | 2019-12-10 | 2020-04-28 | 黄淮学院 | 一种在统一识别框架下小型化手写体文本识别器的方法 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH11203412A (ja) * | 1998-01-16 | 1999-07-30 | Toshiba Corp | 文書画像処理装置、文書画像処理方法及び文書画像処理プログラムを記録したコンピュータ読み取り可能な記録媒体 |
JP2009080615A (ja) * | 2007-09-26 | 2009-04-16 | Univ Of Tokyo | オンライン手書き数式認識方法及び装置 |
JP2014127188A (ja) * | 2012-12-27 | 2014-07-07 | Toshiba Corp | 整形装置及び方法 |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3150468B2 (ja) * | 1993-01-07 | 2001-03-26 | 株式会社東芝 | オンライン文字認識装置 |
JPH09319828A (ja) * | 1996-05-30 | 1997-12-12 | Nec Corp | オンライン文字認識装置 |
JP3827453B2 (ja) * | 1998-09-07 | 2006-09-27 | 積水樹脂株式会社 | 吸音構造体 |
US20060001667A1 (en) * | 2004-07-02 | 2006-01-05 | Brown University | Mathematical sketching |
JP5098504B2 (ja) * | 2007-08-09 | 2012-12-12 | 富士通株式会社 | 文字認識プログラム、文字認識装置および文字認識方法 |
CN101388068B (zh) * | 2007-09-12 | 2011-06-22 | 汉王科技股份有限公司 | 数学公式识别编码方法 |
CN101685497B (zh) * | 2008-09-28 | 2011-10-12 | 汉王科技股份有限公司 | 手写信息处理方法和装置 |
JP2011053986A (ja) * | 2009-09-02 | 2011-03-17 | Sharp Corp | 計算装置、計算プログラムおよびこれが記録された記録媒体 |
CN103679164A (zh) * | 2012-09-21 | 2014-03-26 | 阿里巴巴集团控股有限公司 | 一种基于移动终端的标志识别处理方法和*** |
CN103295008B (zh) * | 2013-05-22 | 2017-04-05 | 华为终端有限公司 | 一种文字识别方法及用户终端 |
CN104063063B (zh) * | 2014-06-05 | 2018-10-09 | 北京搜狗科技发展有限公司 | 输入法的数学计算输入方法和输入法*** |
CN104268118A (zh) * | 2014-09-23 | 2015-01-07 | 赵方 | 一种包括触摸屏的数学式计算器及数学公式识别的方法 |
-
2016
- 2016-01-19 JP JP2016570658A patent/JP6694638B2/ja active Active
- 2016-01-19 KR KR1020177018253A patent/KR101909485B1/ko active IP Right Grant
- 2016-01-19 WO PCT/JP2016/051457 patent/WO2016117564A1/ja active Application Filing
- 2016-01-19 CN CN201680006635.XA patent/CN107209862B/zh not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH11203412A (ja) * | 1998-01-16 | 1999-07-30 | Toshiba Corp | 文書画像処理装置、文書画像処理方法及び文書画像処理プログラムを記録したコンピュータ読み取り可能な記録媒体 |
JP2009080615A (ja) * | 2007-09-26 | 2009-04-16 | Univ Of Tokyo | オンライン手書き数式認識方法及び装置 |
JP2014127188A (ja) * | 2012-12-27 | 2014-07-07 | Toshiba Corp | 整形装置及び方法 |
Non-Patent Citations (4)
Title |
---|
ATSUSHI MURASE ET AL.: "A System for Recognizing Handwritten Mathematical Expressions", IPSJ SIG NOTES, vol. 91, no. 34, 9 May 1991 (1991-05-09), pages 1 - 8 * |
KEISUKE MOCHIDA ET AL.: "A Method for Recognizing On-line Handwritten Patterns mixed of Text and Formulas", IEICE TECHNICAL REPORT, vol. 104, no. 741, 10 March 2005 (2005-03-10), pages 43 - 48 * |
MASAKI NAKAGAWA ET AL.: "Fast Algorithms to Determine Object Inclusion within a Pen Movement Enclosure on a Display Integrated Tablet", IEICE TECHNICAL REPORT, vol. J77-D-II, 25 August 1994 (1994-08-25), pages 1630 - 1639 * |
SO I ET AL.: "On Line Recognition of Handwritten Mathematical Formulae Using Stroke Relations", IEICE TECHNICAL REPORT, vol. 98, no. 489, 17 December 1998 (1998-12-17), pages 9 - 16 * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019191815A (ja) * | 2018-04-23 | 2019-10-31 | 国立大学法人東京農工大学 | プログラム、情報記憶媒体及びクラスタリング装置 |
JP7104965B2 (ja) | 2018-04-23 | 2022-07-22 | 国立大学法人東京農工大学 | プログラム及びクラスタリング装置 |
CN111539383A (zh) * | 2020-05-22 | 2020-08-14 | 浙江蓝鸽科技有限公司 | 公式知识点识别方法及装置 |
CN111539383B (zh) * | 2020-05-22 | 2023-05-05 | 浙江蓝鸽科技有限公司 | 公式知识点识别方法及装置 |
Also Published As
Publication number | Publication date |
---|---|
JP6694638B2 (ja) | 2020-05-20 |
CN107209862B (zh) | 2021-03-09 |
JPWO2016117564A1 (ja) | 2017-10-26 |
KR20170100541A (ko) | 2017-09-04 |
CN107209862A (zh) | 2017-09-26 |
KR101909485B1 (ko) | 2018-10-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110232311B (zh) | 手部图像的分割方法、装置及计算机设备 | |
US10360473B2 (en) | User interface creation from screenshots | |
US8542928B2 (en) | Information processing apparatus and control method therefor | |
JP4712613B2 (ja) | 情報処理装置、情報処理方法およびプログラム | |
US10769427B1 (en) | Detection and definition of virtual objects in remote screens | |
US20140143721A1 (en) | Information processing device, information processing method, and computer program product | |
US9122353B2 (en) | Kind of multi-touch input device | |
CN109214471A (zh) | 评价练字字帖中的书写字的方法及*** | |
US9099007B1 (en) | Computerized processing of pictorial responses in evaluations | |
KR20180064371A (ko) | 다수의 객체 입력을 인식하기 위한 시스템 및 그 방법 및 제품 | |
CN111626297A (zh) | 文字书写质量评价方法、装置、电子设备和记录介质 | |
JP6877446B2 (ja) | 多重オブジェクト構造を認識するためのシステムおよび方法 | |
WO2016117564A1 (ja) | プログラム、情報記憶媒体及び認識装置 | |
JP5231159B2 (ja) | 人物検出装置及び方法、学習モデル作成装置及び方法、並びにプログラム | |
Suganya et al. | Detection and recognition of hand gestures to control the system Applications by neural networks | |
US20210073552A1 (en) | Information processing apparatus and non-transitory computer readable medium storing program | |
CN114241486A (zh) | 一种提高识别试卷学生信息准确率的方法 | |
Dey et al. | Algorithm For Multi-Hand Finger Counting: An Easy Approach | |
JP5674535B2 (ja) | 画像処理装置、方法、及びプログラム | |
WO2020090356A1 (ja) | インクデータ生成装置、方法及びプログラム | |
US20240184494A1 (en) | Method, electronic device, and storage medium for adjusting document style | |
KR100983779B1 (ko) | 도서 정보 서비스 장치 및 방법 | |
Chen et al. | Design of Interactive Teaching System Based on Computer Vision Technology | |
Suresh | Air Canvas: Expressing Creativity in the Real World with OpenCV and Python | |
Shukla et al. | CHARACTER IDENTIFICATION AND MOUSE CONTROL USING HAND GESTURES: A SURVEY |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16740171 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 20177018253 Country of ref document: KR Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2016570658 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 16740171 Country of ref document: EP Kind code of ref document: A1 |