CN111539405A - Reading assisting method and device, electronic equipment and storage medium - Google Patents

Reading assisting method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN111539405A
CN111539405A CN202010301997.XA CN202010301997A CN111539405A CN 111539405 A CN111539405 A CN 111539405A CN 202010301997 A CN202010301997 A CN 202010301997A CN 111539405 A CN111539405 A CN 111539405A
Authority
CN
China
Prior art keywords
page
reading
user
reading position
text
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010301997.XA
Other languages
Chinese (zh)
Other versions
CN111539405B (en
Inventor
贾光辉
王晓斐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Anhui Toycloud Technology Co Ltd
Original Assignee
Anhui Toycloud Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Anhui Toycloud Technology Co Ltd filed Critical Anhui Toycloud Technology Co Ltd
Priority to CN202010301997.XA priority Critical patent/CN111539405B/en
Publication of CN111539405A publication Critical patent/CN111539405A/en
Application granted granted Critical
Publication of CN111539405B publication Critical patent/CN111539405B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/22—Character recognition characterised by the type of writing
    • G06V30/224—Character recognition characterised by the type of writing of printed characters having additional code marks or containing code marks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Facsimile Scanning Arrangements (AREA)
  • Image Input (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The embodiment of the invention provides an auxiliary reading method, an auxiliary reading device, electronic equipment and a storage medium, wherein the method comprises the following steps: determining a page reading position of a user finger on a page image to be read; determining a rough reading position of the page reading position mapped in the reference page image based on the page reading position and the coordinate mapping relation between the page image to be read and the corresponding reference page image; determining a fine reading position in a reference page image based on a page interested area of the page reading position in the page image to be read and a reference interested area of the rough reading position in the reference page image; and performing auxiliary reading based on the fine reading position. The auxiliary reading method, the auxiliary reading device, the electronic equipment and the storage medium provided by the embodiment of the invention improve the convenience and accuracy of determining the to-be-read area.

Description

Reading assisting method and device, electronic equipment and storage medium
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an auxiliary reading method and apparatus, an electronic device, and a storage medium.
Background
In recent years, as people pay more attention to the improvement of reading level, various auxiliary reading methods are emerging continuously, and early auxiliary reading products, namely 'reading pens' with high cost and narrow book coverage are replaced gradually.
In the current reading assistance method, a user is required to set content to be read in advance, or after the user indicates a region to be read using a finger, the content to be read by the user is identified based on an OCR (Optical Character Recognition) method. However, if the content to be read is preset, once the user wants to replace the reading area, the user needs to reset the reading area, which is complicated to operate; if the content to be read is identified based on the OCR method, the content of the indication area cannot be accurately identified due to the occlusion of the finger.
Disclosure of Invention
The embodiment of the invention provides an auxiliary reading method, an auxiliary reading device, electronic equipment and a storage medium, which are used for solving the problems of complex and inaccurate operation of determining contents to be read by a user in the conventional auxiliary reading method.
In a first aspect, an embodiment of the present invention provides an assistant reading method, including:
determining a page reading position of a user finger on a page image to be read;
determining a rough reading position of the page reading position mapped in the reference page image based on the page reading position and a coordinate mapping relation between the page image to be read and the corresponding reference page image;
determining a fine reading position in the reference page image based on a page interesting region of the page reading position in the page image to be read and a reference interesting region of the rough reading position in the reference page image;
and performing auxiliary reading based on the fine reading position.
Optionally, the determining the fine reading position in the reference page image based on the page interested region of the page reading position in the page image to be read and the reference interested region of the coarse reading position in the reference page image specifically includes:
matching each characteristic point of the page area of interest with each characteristic point of the reference area of interest to obtain a page area characteristic point set and a reference area characteristic point set matched with the page area characteristic point set;
and determining the fine reading position based on the page area characteristic point set and the reference area characteristic point set.
Optionally, the determining the fine reading position based on the page area feature point set and the reference area feature point set specifically includes:
determining each page neighborhood point corresponding to the page reading position based on the page area feature point set;
determining each reference neighborhood point corresponding to the rough reading position based on each page neighborhood point and the reference region feature point set;
and determining the fine reading position based on each page neighborhood point and each reference neighborhood point.
Optionally, the determining, based on the page region feature point set, each page neighborhood point corresponding to the page reading position specifically includes:
gradually increasing the range of the neighborhood corresponding to the page reading position until the number of the feature points in the neighborhood, which belong to the page area feature point set, is greater than a preset threshold;
and taking the characteristic points contained in the neighborhood and belonging to the page area characteristic point set as page neighborhood points corresponding to the page reading positions.
Optionally, the determining the fine reading position based on each page neighborhood point and each reference neighborhood point specifically includes:
determining the weight of each page neighborhood point based on the page reading position and the position of each page neighborhood point;
and determining the fine reading position based on each reference neighborhood point and the weight of the page neighborhood point corresponding to each reference neighborhood point.
Optionally, the performing auxiliary reading based on the fine reading position specifically includes:
determining a user reading text based on the user reading voice;
determining a text to be read based on the fine reading position;
matching the user reading text with the corresponding text in the text to be read to obtain the similarity between the user reading text and the corresponding text in the text to be read;
and prompting errors occurring in the reading process to the user based on the similarity between the user reading text and the corresponding text in the text to be read and the current reading state of the user.
Optionally, the prompting, based on the similarity between the text read by the user and the corresponding text in the text to be read and the current reading state of the user, an error occurring in the reading process to the user specifically includes:
if the similarity between the user reading text and the corresponding text in the text to be read is smaller than a first preset similarity threshold, calculating the similarity between the user reading text and the text before or after the corresponding text in the text to be read;
and if the similarity between the text read by the user and the text before or after the corresponding text in the text to be read is smaller than a second preset similarity threshold value, or the user pause time is longer than the preset time, prompting the user of an error occurring in the reading process based on the current reading state of the user.
Optionally, the current reading state of the user is determined based on at least one of difficulty of reading the content, speed change rate of reading, pause time of the user, accuracy rate of reading of the user and historical prompting times.
Optionally, the determining a page reading position of the finger of the user on the page image to be read specifically includes:
determining the stable state of the finger of the user based on the finger coordinate position of the finger of the user in the continuous multi-frame acquisition page images;
and if the stable state of the user finger is stable, determining the page reading position based on the finger coordinate position in the continuous multi-frame collected page images.
In a second aspect, an embodiment of the present invention provides an auxiliary reading device, including:
the page reading position positioning unit is used for determining the page reading position of the finger of the user on the page image to be read;
the rough reading position positioning unit is used for determining the rough reading position of the page reading position mapped in the reference page image based on the page reading position and the coordinate mapping relation between the page image to be read and the corresponding reference page image;
the fine reading position positioning unit is used for determining a fine reading position in the reference page image based on a page interested region of the page reading position in the page image to be read and a reference interested region of the rough reading position in the reference page image;
and the auxiliary reading unit is used for performing auxiliary reading based on the fine reading position.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a bus, where the processor and the communication interface, the memory complete mutual communication through the bus, and the processor may call a logic command in the memory to perform the steps of the method provided in the first aspect.
In a fourth aspect, an embodiment of the present invention provides a non-transitory computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the steps of the method as provided in the first aspect.
According to the auxiliary reading method, the auxiliary reading device, the electronic equipment and the storage medium, the rough reading position in the reference page image is calculated based on the page reading position of the finger of the user on the page image to be read, the fine reading position in the reference page image is further obtained, the area to be read is accurately determined based on the fine reading position, and convenience and accuracy of determining the area to be read are improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a flowchart illustrating an assistant reading method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a method for determining a fine reading position according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a fine reading position determining method according to another embodiment of the present invention;
fig. 4 is a schematic flowchart of a page neighborhood point determination method according to an embodiment of the present invention;
fig. 5 is a flowchart illustrating a reading prompting method according to an embodiment of the present invention;
FIG. 6 is a flowchart illustrating a reading prompt method according to another embodiment of the present invention;
FIG. 7 is a flowchart illustrating a method for detecting a stable state of a finger according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an auxiliary reading device according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In recent years, people gradually pay more attention to the improvement of reading level, and various auxiliary reading methods are continuously emerged.
The current auxiliary reading method mainly has two modes when determining the content to be read by a user: one is that the content to be read is preset by the user; the other is that the user indicates an area to be read by using a finger, and then recognizes the content of the area indicated by the user as the content to be read by the user based on the OCR technology. For the first mode, if the user wants to change the reading content after the setting is completed, the content to be read must be reset, and the operation is complex; for another mode, when the finger indicates the reading area, part of characters in the reading area are blocked, so that the content accuracy of the indicated area identified by the OCR technology is poor.
Accordingly, the embodiment of the invention provides an auxiliary reading method. Fig. 1 is a schematic flow chart of an assistant reading method according to an embodiment of the present invention, as shown in fig. 1, the method includes:
step 110, determining the page reading position of the user finger on the page image to be read.
Specifically, after entering the reading state, the user may perform a finger reading operation on the page to be read using a finger to indicate an area to be read, i.e., the area to be read. Here, the page to be read is a book page to be read by a user, and the page to be read may be a page in an entity book or a page in an electronic book. And then, shooting to obtain a page image to be read corresponding to the page to be read, and obtaining the page reading position of the user finger on the page image to be read. Here, the page reading position is a reading position of a user finger in the page image to be read, and may be used to determine a region to be read in the page image to be read.
And step 120, determining a rough reading position of the page reading position mapped in the reference page image based on the page reading position and the coordinate mapping relation between the page image to be read and the corresponding reference page image.
Here, the reference page image corresponding to the page image to be read is a page image of the same page of the same book as the page image to be read. Alternatively, a plurality of reference images may be obtained by photographing or scanning pages of a plurality of books in advance. And inputting the page image to be read and each reference image into an image classifier, wherein the image classifier is used for matching the page image to be read and each reference image and then obtaining the reference page image output by the image classifier.
Based on the coordinate mapping relation between the page image to be read and the reference page image, the rough reading position of the page reading position mapped in the reference page image can be determined. Here, the rough reading position is a reading position corresponding to the page reading position in the reference page image, and is used to determine a region to be read in the reference page image.
For example, let f (x, y) denote the coordinate mapping relationship between the page image to be read and the reference page image, TPWhen the page reading position is indicated, the rough reading position H can be obtained by using the following formulaP：
HP＝f(x,y)TP
Optionally, the coordinate mapping relationship between the page image to be read and the reference page image may be established as follows: respectively extracting the characteristic points of the page image to be read and the reference page image, and then matching the characteristic points of the page image to be read with the characteristic points of the reference page image to obtain a page characteristic point set to be read and a reference page characteristic point set matched with the page characteristic point set; and then, according to the one-to-one corresponding coordinates of the characteristic points of the page to be read and the coordinates of the characteristic points of the reference page, determining a coordinate mapping relation between the page image to be read and the reference page image.
The feature points of the page image to be read are matched with the feature points of the reference page image, and the following method can be adopted: and calculating the distance between each characteristic point of the page image to be read and each characteristic point of the reference page image, and then determining the characteristic point of the reference page image matched with the characteristic point of the page image to be read based on the distance between any characteristic point of the page image to be read and each characteristic point of the reference page image.
Step 130, determining the fine reading position in the reference page image based on the page interesting region of the page reading position in the page image to be read and the reference interesting region of the rough reading position in the reference page image.
Because the finger tip has a certain length and width in the page image to be read, the determined page reading position cannot accurately represent the actual reading position, and here, the actual reading position is the position in the page image to be read where the user actually wants to read. Accordingly, the rough reading position obtained based on the page reading position is not accurate enough.
Aiming at the problems, the embodiment of the invention uses the page reading position as the center, and defines an area in the page image to be read as the page interested area; similarly, a region is defined as a reference region of interest in the reference page image with the rough finger reading position as the center. Here, the page area of interest includes an actual reading position, and accordingly, the reference area of interest includes a reading position corresponding to the actual reading position. In addition, the shapes and sizes of the page interesting region and the reference interesting region may be set according to actual situations, which is not specifically limited in the embodiment of the present invention.
Then, based on the coordinate points in the page interesting area, the importance degree of each coordinate point in determining the actual reading position is analyzed. And determining a reading position corresponding to the actual reading position in the reference interested area as a fine reading position based on the coordinate point corresponding to the page interested area coordinate point in the reference interested area and the importance degree of the page interested area coordinate point. Here, the fine finger reading position can be used to determine the region to be read in the reference page image.
And 140, performing auxiliary reading based on the fine reading position.
Specifically, based on the fine reading position, the region to be read in the reference page image can be determined, and the user can be assisted to read on the basis. For example, a user may be fed back a problem that the user has during reading, or the reading condition of the user may be scored after the user finishes reading, which is not specifically limited by the embodiment of the present invention.
The user can determine the region to be read of the user after performing finger reading operation by using a finger, so that the user does not need to set the region to be read in advance, and can change the region to be read at any time, and complicated operation of repeated setting is avoided. In addition, the fine reading position corresponds to the actual reading position, so that the area to be read in the reference page image determined according to the fine reading position can accurately reflect the area which the user actually wants to read without being influenced by finger shielding.
According to the method provided by the embodiment of the invention, the rough reading position in the reference page image is calculated based on the page reading position of the finger of the user on the page image to be read, so that the fine reading position in the reference page image is obtained, the area to be read is accurately determined based on the fine reading position, and the convenience and the accuracy of determining the area to be read are improved.
Based on the foregoing embodiment, fig. 2 is a schematic flowchart of a fine reading position determining method according to an embodiment of the present invention, and as shown in fig. 2, step 130 specifically includes:
and 131, matching each characteristic point of the page area of interest with each characteristic point of the reference area of interest to obtain a page area characteristic point set and a reference area characteristic point set matched with the page area characteristic point set.
Specifically, in consideration of the fact that the feature points in the page area of interest are more important in determining the actual reading position relative to other coordinate points, the feature points of the page area of interest and the feature points of the reference area of interest are extracted, and then each feature point of the page area of interest and each feature point of the reference area of interest are matched to obtain a page area feature point set and a reference area feature point set matched with the page area feature point set. Here, the page area feature point set is composed of feature points of the page area of interest, and the reference area feature point set is composed of feature points in the reference area of interest that are matched with the feature points of the page area of interest. And the feature points in the page area feature point set can be used for positioning the actual reading position in the page image to be read, and correspondingly, the feature points in the reference area feature point set can be used for positioning the reading position corresponding to the actual reading position in the reference page image.
When each feature point of the page interesting region is matched with each feature point of the reference interesting region, a matching mode between the feature point of the page image to be read and the feature point of the reference page image in the above embodiment can be adopted. Further, a Random Sample Consensus (RANSAC) algorithm may be used to further screen out correctly matched feature points of the page region of interest and reference region of interest, so as to obtain a page region feature point set and a reference region feature point set.
Step 132, determining the fine reading position based on the page area feature point set and the reference area feature point set.
Specifically, based on the feature points in the page area feature point set, the importance degree of each feature point in determining the actual reading position is analyzed. And then determining the fine reading position in the reference region of interest based on the feature points in the reference region feature point set matched with the feature points in the page region feature point set and the importance degree of the feature points in the page region feature point set.
The method provided by the embodiment of the invention determines the fine reading position based on the page area characteristic point set and the reference area characteristic point set matched with the page area characteristic point set, can position the reading position corresponding to the actual reading position, and is favorable for improving the accuracy of determining the area to be read by the user.
Based on any of the above embodiments, fig. 3 is a schematic flowchart of a fine reading position determining method according to another embodiment of the present invention, as shown in fig. 3, step 132 specifically includes:
step 1321, determining each page neighborhood point corresponding to the page reading position based on the page area feature point set.
Specifically, considering that the distance between the actual reading position and the page reading position is relatively close, that is, the feature points near the page reading position are more important when determining the actual reading position, the feature points closer to the page reading position are screened out from the page area feature point set to serve as the page neighborhood points corresponding to the page reading position.
Step 1322 is to determine each reference neighborhood point corresponding to the rough reading position based on each page neighborhood point and the reference region feature point set.
Specifically, since there is a matching relationship between the feature points in the page area feature point set and the feature points in the reference area feature point set, the feature points corresponding to the page neighborhood points can be selected from the reference area feature point set according to the matching relationship and used as the reference neighborhood points.
Step 1323, determining a fine reading position based on each page neighborhood point and each reference neighborhood point.
Specifically, the importance level of each page neighborhood point in determining the actual reading position is analyzed first. Considering that there is a corresponding relationship between the page neighborhood point and the reference neighborhood point, the importance degree of any reference neighborhood point in determining the fine reading position can be considered to be the same as the importance degree of the corresponding page neighborhood point in determining the actual reading position. Thus, the fine reading position can be determined based on the degree of importance of each page neighborhood point in determining the actual reading position, and each reference neighborhood point.
According to the method provided by the embodiment of the invention, the importance degree of each page neighborhood point in the actual reading position is determined by determining each page neighborhood point corresponding to the page reading position and each reference neighborhood point corresponding to the rough reading position, so that the fine reading position is determined based on the importance degrees of each reference neighborhood point and each corresponding page neighborhood point, and the accuracy of the fine reading position is further improved.
Based on any of the above embodiments, fig. 4 is a schematic flowchart of a method for determining a page neighborhood point according to an embodiment of the present invention, and as shown in fig. 4, step 1321 specifically includes:
and step 1321-1, gradually increasing the range of the neighborhood corresponding to the page reading position until the number of the feature points in the neighborhood, which belong to the page area feature point set, is greater than a preset threshold value.
And step 1321-2, taking the feature points in the feature point set belonging to the page area and contained in the neighborhood as page neighborhood points corresponding to the page reading position.
Specifically, the initial neighborhood corresponding to the page reading position may be determined by taking the page reading position as a center and taking the preset distance as a radius. And judging whether the number of the feature points belonging to the page area feature point set and contained in the current neighborhood is greater than a preset threshold value, if not, gradually increasing the range of the domain until the number of the feature points belonging to the page area feature point set and contained in the current neighborhood is greater than the preset threshold value. And then, taking the feature points in the feature point set belonging to the page area and contained in the neighborhood as page neighborhood points corresponding to the page reading positions.
According to the method provided by the embodiment of the invention, the range of the neighborhood corresponding to the page reading position is gradually increased, and the feature points in the page region feature point set contained in the neighborhood range are used as the page neighborhood points, so that a basis is provided for determining the fine reading position.
Based on any of the above embodiments, step 1323 specifically includes:
step 1323-1, determining the weight of each page neighborhood point based on the page reading position and the position of each page neighborhood point.
Specifically, the weight of each page neighborhood point is determined based on the page reading position and the position of each page neighborhood point. The closer the distance between the position of the page neighborhood point and the page reading position is, the higher the corresponding weight is, and the higher the importance degree of the page neighborhood point in determining the actual reading position is represented.
And step 1323-2, determining the fine reading position based on each reference neighborhood point and the weight of the page neighborhood point corresponding to each reference neighborhood point.
Specifically, after determining the weight of each page neighborhood point, the fine reading position may be determined based on each reference neighborhood point and the weight of the page neighborhood point corresponding to each reference neighborhood point.
For example, the fine reading position is calculated by the following formula:
Figure BDA0002454354790000111
wherein H _ P _ LAST is the fine finger read position,
Figure BDA0002454354790000112
the position of the reference neighborhood point corresponding to the ith page neighborhood point,
Figure BDA0002454354790000113
is the weight of the ith page neighborhood point.
According to the method provided by the embodiment of the invention, the fine reading position is determined by determining the weight of each page neighborhood point and based on each reference neighborhood point and the weight of the page neighborhood point corresponding to each reference neighborhood point, so that the accuracy of the fine reading position can be further improved.
Based on any of the above embodiments, fig. 5 is a schematic flow chart of the reading prompting method according to the embodiment of the present invention, and as shown in fig. 5, step 140 specifically includes:
step 141, determining a user reading text based on the user reading voice;
142, determining a text to be read based on the fine reading position;
step 143, matching the user reading text with the corresponding text in the text to be read to obtain the similarity between the user reading text and the corresponding text in the text to be read;
and 144, prompting the error in the reading process to the user based on the similarity between the reading text of the user and the corresponding text in the text to be read and the current reading state of the user.
Specifically, based on the fine reading position, the region to be read in the reference page image may be determined, and on this basis, the text in the region to be read is identified as the text to be read. And after the user starts reading, recording to obtain the reading voice of the user. Determining that the user reads the text based on the user reading voice. Here, the user-read text is the text that the user actually reads at present. Meanwhile, matching the user reading text with the corresponding text in the text to be read in real time, and determining the similarity between the user reading text and the corresponding text in the text to be read. Here, the text to be read corresponds to the text actually read by the user.
The current auxiliary reading method usually feeds back errors occurring in the reading process to a user in real time in the reading process of the user, or feeds back the errors occurring in the reading process to the user after the reading is finished. The method comprises the following steps that real-time feedback is carried out in the reading process, and when a user reads a wrong word, the error can be fed back to the user, so that the user is easily interrupted, and the use experience of the user is poor; and after reading, the user can not be corrected in time by feeding back.
Aiming at the problem, the embodiment of the invention prompts the error in the reading process to the user based on the current reading state of the user and the similarity between the reading text of the user and the corresponding text in the text to be read. Here, the current reading state of the user may be determined based on factors such as a reading speed change rate or a reading accuracy rate in the reading process of the user, which is not specifically limited in the embodiment of the present invention. When the current reading state of the user is better, errors are not fed back to the user, so that the reading of the user is prevented from being interrupted, and the effect of auxiliary reading is reduced; when the current reading state of the user is poor, the user is indicated to be stopped currently to correct errors, errors occurring in the reading process are prompted to the user based on the similarity between the reading text of the user and the corresponding text in the text to be read, the error correction effect can be improved while timely error correction in the reading process is achieved, and the use experience of the user is improved.
According to the method provided by the embodiment of the invention, based on the similarity between the reading text of the user and the corresponding text in the text to be read and the current reading state of the user, the error occurring in the reading process is prompted to the user, so that the error can be corrected in time in the reading process, the error correction effect is improved, and the use experience of the user is improved.
Based on any of the above embodiments, fig. 6 is a schematic flow chart of a reading prompting method according to another embodiment of the present invention, as shown in fig. 6, step 144 specifically includes:
step 1441, if the similarity between the user reading text and the corresponding text in the text to be read is smaller than a first preset similarity threshold, calculating the similarity between the user reading text and the text before or after the corresponding text in the text to be read;
step 1442, if the similarity between the text read by the user and the text before or after the corresponding text in the text to be read is smaller than a second preset similarity threshold, or the user pause time is longer than a preset time, prompting the user of an error occurring in the reading process based on the current reading state of the user.
Specifically, if the similarity between the user reading text and the corresponding text in the text to be read is smaller than a first preset similarity threshold, it indicates that the similarity between the user reading text and the corresponding text in the text to be read is low in the current reading process. However, the reason why the similarity between the text read by the user and the corresponding text in the text to be read is low includes that the user has more errors in the current reading process, and also includes that the user has missed reading or repeated reading when reading. If the similarity between the reading text of the user and the corresponding text in the text to be read is low only because the user misses reading or rereads part of the characters, the user does not need to be prompted with errors in the reading process, so that the reading of the user is interrupted, and the effect of auxiliary reading is reduced.
Therefore, in order to distinguish the situations that a user has more errors in the current reading process and the situations that the user has missed reading or rereading when reading, when the similarity between the user reading text and the corresponding text in the text to be read is smaller than a first preset similarity threshold, the similarity between the user reading text and the text before or after the corresponding text in the text to be read is calculated. Here, the text before the corresponding text in the text to be read, that is, the text that the user has read; and reading the text after the corresponding text in the text to be read, namely the text to be read by the user.
If the similarity between the text read by the user and the text before or after the corresponding text in the text to be read is greater than or equal to the second preset similarity threshold, it indicates that the user may have the phenomenon of re-reading or missing reading, and at this time, the user is not prompted about the error occurring in the reading process. And if the similarity between the text read by the user and the text before or after the corresponding text in the text to be read is smaller than a second preset similarity threshold, which indicates that the user really has more errors in the reading process, prompting the user of the errors in the reading process based on the current reading state of the user. In addition, if the user pause time is longer than the preset time, which indicates that the user pauses for a long time, a good opportunity for prompting the user of errors occurring in the reading process is provided, and the errors occurring in the reading process can be prompted to the user based on the current reading state of the user.
The method provided by the embodiment of the invention determines whether the prompt is needed to be given to the user or not based on the similarity between the reading text of the user and the corresponding text in the text to be read and the similarity between the reading text of the user and the text before or after the corresponding text in the text to be read, and prompts the error in the reading process to the user based on the current reading state of the user when the prompt is needed, so that the error in the reading process can be prevented from being prompted to the user when the user misses or rereads, the error correction effect is improved, and the use experience of the user is improved.
Based on any embodiment, the current reading state of the user is determined based on at least one of the difficulty of reading the content, the speed change rate of reading, the pause time of the user, the reading accuracy rate of the user and the historical prompting times.
Specifically, the difficulty of reading the content, the rate of change of the reading speed, the pause time of the user, the reading accuracy of the user, and the number of historical prompts all affect the current reading state of the user, so that the current reading state of the user can be determined based on one or more of the difficulty of reading the content, the rate of change of the reading speed, the pause time of the user, the reading accuracy of the user, and the number of historical prompts.
For example, the following formula may be used to determine the current reading status of the user:
SR＝-w0Dif+w1Crt-w2Num-k0Spd-k1Tim
wherein S isRIs the current reading state of the user, Dif is the difficulty of reading the content, Crt is the correct rate of reading by the user, Num is the number of history prompts, Spd is the rate of change of the speed of reading, Tim is the pause time of the user, w0、w1、w2、k0And k1The preset reading content difficulty, the reading accuracy of the user, the historical prompting times, the reading speed change rate and the corresponding weight of the user pause time are respectively.
The method provided by the embodiment of the invention determines the current reading state of the user based on at least one of the difficulty of reading the content, the speed change rate of reading, the pause time of the user, the reading accuracy of the user and the historical prompting times, and provides a basis for judging whether to prompt the user of errors occurring in the reading process.
Based on any of the above embodiments, step 110 specifically includes:
determining the stable state of the finger of the user based on the finger coordinate position of the finger of the user in the continuous multi-frame acquisition page images; and if the finger stable state of the user is stable, determining the page reading position based on the finger coordinate position in the continuous multi-frame acquisition page image.
In particular, the user's finger steady state may indicate whether the user has performed a finger reading operation. If the user carries out finger reading operation, fingers of the user stay at the same position of the page to be read for a long time, so that whether the same finger appears in the continuous multi-frame collected page images or not can be judged based on the finger coordinate position of the finger of the user in the continuous multi-frame collected page images, and the change of the finger coordinate position is within a preset range. When the finger stable state of the user is stable, the user can be considered to be currently performing finger reading operation, and at this time, the page image to be read and the page finger reading position of the user finger in the page image to be read can be determined based on the continuous multi-frame acquisition of the page image and the finger coordinate position thereof. For example, the currently acquired page image may be selected as the page image to be read, and the finger coordinate position of the user finger in the currently acquired page image may be used as the page reading position.
Optionally, fig. 7 is a schematic flowchart of a method for detecting a stable state of a finger according to an embodiment of the present invention, and as shown in fig. 7, in the method, a finger detection model is used to sequentially detect whether a finger exists in consecutive multi-frame acquired page images, and if no finger is detected, a counter and a recorded finger coordinate position are cleared, and whether a finger exists in a next acquired page image is detected. If the finger is detected, adding one to the Timer by the counter, recording the coordinate position of the finger in the current acquisition page, and judging whether the following two conditions are met at the same time: the value of the counter Timer is greater than a first preset threshold value; and the variance of the recorded finger coordinate positions is less than a second preset threshold value. If the two conditions are met simultaneously, determining that the stable state of the finger of the user is stable, and ending the method; otherwise, continuously detecting whether the next acquired page image has the finger.
According to the method provided by the embodiment of the invention, by judging the stable state of the finger of the user, when the stable state of the finger of the user is stable, namely the user carries out finger reading operation currently, the page finger reading position is determined based on the finger coordinate position in the continuous multi-frame collected page image, and the accuracy of the page finger reading position can be improved.
Based on any one of the above embodiments, another embodiment of the present invention provides an assistant reading method, including:
first, after a user enters a reading phase, a finger steady state of the user is detected. Here, the user can obtain the page of the book to be read through catalog selection or voice search, and enter the reading stage by clicking a start button and the like; if the user does not select the page of the book to be read, the page image can be collected, the collected page image is matched with a plurality of reference images collected in advance, and if the matching is successful, the user is considered to enter a reading stage.
And if the finger stable state of the user is stable, taking the current acquisition page image as a page image to be read, taking the finger coordinate position of the finger of the user in the current acquisition page image as a page reading position, and then inputting the page image to be read and each pre-collected reference image into an image classifier. And the image classifier matches the page image to be read with each reference image to obtain the matching score of each reference image, judges whether the highest matching score is greater than a preset score threshold value or not, and outputs the reference image corresponding to the highest matching score as the reference page image if the highest matching score is greater than the preset score threshold value. Here, the image classifier may employ a feature extraction algorithm, such as orb (organized FAST and Rotated brief), SIFT (Scale-innovative feature Transform) algorithm, and an image classification algorithm, such as KNN (K-Nearest Neighbor), SVM (Support Vector Machine) algorithm, to realize matching of the page image to be read and each reference image.
And after the page image to be read and the corresponding reference page image are obtained, constructing a coordinate mapping relation between the page image to be read and the reference page image. And respectively extracting the feature points of the page image to be read and the reference page image by using a feature extraction algorithm, such as ORB, SIFT and the like. And calculating the distance, such as Euclidean distance, between each feature point of the page image to be read and each feature point of the reference page image. And aiming at any characteristic point of the page image to be read, arranging the distance between the characteristic point and each characteristic point of the reference page image from small to large, and calculating the ratio of the minimum two distances. If the ratio is smaller than the preset ratio threshold, the characteristic point of the reference page image corresponding to the minimum distance is matched with the characteristic point. Finally, the page feature point set to be read and the reference page feature point set matched with the page feature point set can be screened out. And further, based on the one-to-one corresponding coordinates of the characteristic points of the page to be read and the coordinates of the characteristic points of the reference page, determining a coordinate mapping relation between the page image to be read and the reference page image by using a vector field consistency algorithm (VFC).
Based on the page reading position and the coordinate mapping relation between the page image to be read and the corresponding reference page image, the rough reading position of the page reading position mapped in the reference page image can be determined.
Then, with the page reading position as the center, a square area is defined in the page image to be read by a preset diameter to serve as a page interesting area; similarly, a square area is defined in the reference page image with a preset diameter as a reference region of interest centered on the rough finger reading position. Extracting feature points of the page interesting region and feature points of the reference interesting region by using a feature extraction algorithm, such as a fast (features from accessed Segment test) algorithm, and then matching each feature point of the page interesting region with each feature point of the reference interesting region. Here, each feature point of the page area of interest and each feature point of the reference area of interest may be matched in a matching manner between the feature point of the page image to be read and the feature point of the reference page image. Further, a random sampling consistency algorithm can be used for further screening correctly matched feature points of the page area of interest and reference area of interest to obtain a page area feature point set and a reference area feature point set.
And determining an initial neighborhood corresponding to the page reading position by taking the page reading position as a center and taking the preset distance as a radius. And gradually increasing the range of the field until the number of the feature points in the feature point set belonging to the page area contained in the current neighborhood is greater than a preset threshold value. And then, taking the feature points in the feature point set belonging to the page area and contained in the neighborhood as page neighborhood points corresponding to the page reading positions. And selecting feature points corresponding to the page neighborhood points from the reference region feature point set based on the matching relationship between the feature points in the page region feature point set and the feature points in the reference region feature point set, and taking the feature points as the reference neighborhood points.
And determining the weight of each page neighborhood point based on the page reading position and the position of each page neighborhood point. For example, the following formula can be used to calculate the weight of any page neighborhood point:
Figure BDA0002454354790000171
wherein p is the coordinate of the neighborhood point of the page, p0For page pointing to read position, | p-p0And | represents the distance between the neighborhood point of the page and the page reading position, and β is a constant coefficient.
Then, the weight of each page neighborhood point is normalized, and the fine reading position is determined based on each reference neighborhood point and the normalized weight of the page neighborhood point corresponding to each reference neighborhood point.
And determining a region to be read in the reference page image by taking the fine reading position as a starting point, and identifying the text in the region to be read as the text to be read on the basis. After the user starts reading, recording is carried out, the reading voice of the user is obtained, the reading text of the user is determined, and then the similarity between the reading text of the user and the corresponding text in the text to be read is calculated in real time. And if the similarity between the user reading text and the corresponding text in the text to be read is smaller than a first preset similarity threshold, calculating the similarity between the user reading text and the text before or after the corresponding text in the text to be read. And if the similarity between the text read by the user and the text before or after the corresponding text in the text to be read is smaller than a second preset similarity threshold value, or the user pause time is longer than the preset time, prompting the user of the error in the reading process based on the current reading state of the user.
The current reading state of the user is determined based on the difficulty of reading the content, the speed change rate of reading, the pause time of the user, the reading accuracy rate of the user and the historical prompting times. The difficulty of reading the content may be determined based on one or more of the degree of commonalities of the word to be read, the number of strokes, and the difficulty of pronunciation. For example, the following formula can be used to determine the difficulty of reading the content:
Dif＝a0Fre+a1Str+a2Pro
wherein Dif is the difficulty of reading the content, Fre is the degree of commonness of the word, Str is the number of strokes of the word, Pro is the difficulty of pronunciation of the word, a0、a1And a2Respectively, the preset common degree of the character, the stroke number and the weight corresponding to the pronunciation difficulty.
And after the reading of the user is finished, scoring the reading level of the user, and displaying the score in real time. The user may also play back the audio of the reading, reviewing the reading process.
Based on any of the above embodiments, fig. 8 is a schematic structural diagram of an auxiliary reading device according to an embodiment of the present invention, as shown in fig. 8, the device includes a page reading position locating unit 810, a rough reading position locating unit 820, a fine reading position locating unit 830, and an auxiliary reading unit 840.
The page reading position locating unit 810 is configured to determine a page reading position of a finger of a user on the page image to be read;
the rough reading position positioning unit 820 is configured to determine a rough reading position where the page reading position is mapped in the reference page image based on the page reading position and a coordinate mapping relationship between the page image to be read and the corresponding reference page image;
the fine reading position locating unit 830 is configured to determine a fine reading position in the reference page image based on a page interesting region of the page reading position in the page image to be read and a reference interesting region of the rough reading position in the reference page image;
the auxiliary reading unit 840 is used for performing auxiliary reading based on the fine reading position.
According to the device provided by the embodiment of the invention, the rough reading position in the reference page image is calculated based on the page reading position of the finger of the user on the page image to be read, so that the fine reading position in the reference page image is obtained, the area to be read is accurately determined based on the fine reading position, and the convenience and the accuracy of determining the area to be read are improved.
Based on any of the above embodiments, the fine reading position locating unit 830 is specifically configured to:
matching each characteristic point of the page area of interest with each characteristic point of the reference area of interest to obtain a page area characteristic point set and a reference area characteristic point set matched with the page area characteristic point set;
and determining the fine reading position based on the page area characteristic point set and the reference area characteristic point set.
The device provided by the embodiment of the invention determines the fine reading position based on the page area characteristic point set and the reference area characteristic point set matched with the page area characteristic point set, can position the reading position corresponding to the actual reading position, and is favorable for improving the accuracy of determining the area to be read by the user.
Based on any of the above embodiments, determining the fine reading position based on the page area feature point set and the reference area feature point set specifically includes:
determining each page neighborhood point corresponding to the page reading position based on the page area feature point set;
determining each reference neighborhood point corresponding to the rough reading position based on each page neighborhood point and the reference region feature point set;
and determining the fine reading position based on each page neighborhood point and each reference neighborhood point.
According to the device provided by the embodiment of the invention, the importance degree of each page neighborhood point in the actual reading position is determined by determining each page neighborhood point corresponding to the page reading position and each reference neighborhood point corresponding to the rough reading position, so that the fine reading position is determined based on the importance degrees of each reference neighborhood point and each corresponding page neighborhood point, and the accuracy of the fine reading position is further improved.
Based on any of the above embodiments, determining each page neighborhood point corresponding to the page reading position based on the page region feature point set specifically includes:
gradually increasing the range of the neighborhood corresponding to the page reading position until the number of the feature points in the neighborhood, which belong to the feature point set of the page area, is greater than a preset threshold value;
and taking the characteristic points which belong to the page area characteristic point set and are contained in the neighborhood as page neighborhood points corresponding to the page reading positions.
According to the device provided by the embodiment of the invention, the range of the neighborhood corresponding to the page reading position is gradually increased, and the feature points in the page region feature point set in the neighborhood range are used as the page neighborhood points, so that a basis is provided for determining the fine reading position.
Based on any of the above embodiments, determining the fine reading position based on each page neighborhood point and each reference neighborhood point specifically includes:
determining the weight of each page neighborhood point based on the page reading position and the position of each page neighborhood point;
and determining the fine reading position based on each reference neighborhood point and the weight of the page neighborhood point corresponding to each reference neighborhood point.
The device provided by the embodiment of the invention determines the fine reading position by determining the weight of each page neighborhood point and based on each reference neighborhood point and the weight of the page neighborhood point corresponding to each reference neighborhood point, so that the accuracy of the fine reading position can be further improved.
Based on any of the above embodiments, the auxiliary reading unit 840 is specifically configured to:
determining a user reading text based on the user reading voice;
determining a text to be read based on the fine reading position;
matching the user reading text with the corresponding text in the text to be read to obtain the similarity between the user reading text and the corresponding text in the text to be read;
and prompting errors occurring in the reading process to the user based on the similarity between the reading text of the user and the corresponding text in the text to be read and the current reading state of the user.
According to the device provided by the embodiment of the invention, based on the similarity between the reading text of the user and the corresponding text in the text to be read and the current reading state of the user, the error occurring in the reading process is prompted to the user, so that the error can be corrected in time in the reading process, the error correction effect is improved, and the use experience of the user is improved.
Based on any of the embodiments, based on the similarity between the text read by the user and the corresponding text in the text to be read, and the current reading state of the user, the error occurring in the reading process is prompted to the user, which specifically includes:
if the similarity between the user reading text and the corresponding text in the text to be read is smaller than a first preset similarity threshold, calculating the similarity between the user reading text and the text before or after the corresponding text in the text to be read;
and if the similarity between the text read by the user and the text before or after the corresponding text in the text to be read is smaller than a second preset similarity threshold value, or the user pause time is longer than the preset time, prompting the user of the error in the reading process based on the current reading state of the user.
The device provided by the embodiment of the invention determines whether the prompt is needed to be given to the user or not based on the similarity between the reading text of the user and the corresponding text in the text to be read and the similarity between the reading text of the user and the text before or after the corresponding text in the text to be read, and prompts the error in the reading process to the user based on the current reading state of the user when the prompt is needed, so that the error in the reading process can be prevented from being prompted to the user when the user misses or rereads, the error correction effect is improved, and the use experience of the user is improved.
Based on any embodiment, the current reading state of the user is determined based on at least one of the difficulty of reading the content, the speed change rate of reading, the pause time of the user, the reading accuracy rate of the user and the historical prompting times.
The device provided by the embodiment of the invention determines the current reading state of the user based on at least one of the difficulty of reading the content, the change rate of the reading speed, the pause time of the user, the reading accuracy of the user and the historical prompting times, and provides a basis for judging whether to prompt the user of errors occurring in the reading process.
Based on any of the above embodiments, the page reading position locating unit 810 is specifically configured to:
determining the stable state of the finger of the user based on the finger coordinate position of the finger of the user in the continuous multi-frame acquisition page images;
and if the finger stable state of the user is stable, determining the page reading position based on the finger coordinate position in the continuous multi-frame acquisition page image.
According to the device provided by the embodiment of the invention, by judging the stable state of the finger of the user, when the stable state of the finger of the user is stable, namely the user carries out finger reading operation currently, the page finger reading position is determined based on the finger coordinate position in the continuous multi-frame collected page image, and the accuracy of the page finger reading position can be improved.
Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 9, the electronic device may include: a processor (processor)910, a communication Interface (Communications Interface)920, a memory (memory)930, and a communication bus 940, wherein the processor 910, the communication Interface 920, and the memory 930 communicate with each other via the communication bus 940. Processor 910 may invoke logical commands in memory 930 to perform the following method: determining a page reading position of a user finger on a page image to be read; determining a rough reading position of the page reading position mapped in the reference page image based on the page reading position and a coordinate mapping relation between the page image to be read and the corresponding reference page image; determining a fine reading position in the reference page image based on a page interesting region of the page reading position in the page image to be read and a reference interesting region of the rough reading position in the reference page image; and performing auxiliary reading based on the fine reading position.
In addition, the logic commands in the memory 930 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic commands are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of commands for enabling a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Embodiments of the present invention further provide a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the method provided in the foregoing embodiments when executed by a processor, and the method includes: determining a page reading position of a user finger on a page image to be read; determining a rough reading position of the page reading position mapped in the reference page image based on the page reading position and a coordinate mapping relation between the page image to be read and the corresponding reference page image; determining a fine reading position in the reference page image based on a page interesting region of the page reading position in the page image to be read and a reference interesting region of the rough reading position in the reference page image; and performing auxiliary reading based on the fine reading position.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes commands for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (12)

1. An assistive reading method, comprising:
determining a page reading position of a user finger on a page image to be read;
determining a rough reading position of the page reading position mapped in the reference page image based on the page reading position and a coordinate mapping relation between the page image to be read and the corresponding reference page image;
determining a fine reading position in the reference page image based on a page interesting region of the page reading position in the page image to be read and a reference interesting region of the rough reading position in the reference page image;
and performing auxiliary reading based on the fine reading position.
2. The reading assistance method according to claim 1, wherein the determining the fine reading position in the reference page image based on the page area of interest of the page reading position in the page image to be read and the reference area of interest of the coarse reading position in the reference page image comprises:
matching each characteristic point of the page area of interest with each characteristic point of the reference area of interest to obtain a page area characteristic point set and a reference area characteristic point set matched with the page area characteristic point set;
and determining the fine reading position based on the page area characteristic point set and the reference area characteristic point set.
3. The reading assistance method according to claim 2, wherein the determining the fine reading position based on the page area feature point set and the reference area feature point set specifically comprises:
determining each page neighborhood point corresponding to the page reading position based on the page area feature point set;
determining each reference neighborhood point corresponding to the rough reading position based on each page neighborhood point and the reference region feature point set;
and determining the fine reading position based on each page neighborhood point and each reference neighborhood point.
4. The reading assistance method according to claim 3, wherein the determining each page neighborhood point corresponding to the page reading position based on the page area feature point set specifically comprises:
gradually increasing the range of the neighborhood corresponding to the page reading position until the number of the feature points in the neighborhood, which belong to the page area feature point set, is greater than a preset threshold;
and taking the characteristic points contained in the neighborhood and belonging to the page area characteristic point set as page neighborhood points corresponding to the page reading positions.
5. The reading aid method according to claim 3, wherein the determining the fine reading position based on each page neighborhood point and each reference neighborhood point comprises:
determining the weight of each page neighborhood point based on the page reading position and the position of each page neighborhood point;
and determining the fine reading position based on each reference neighborhood point and the weight of the page neighborhood point corresponding to each reference neighborhood point.
6. The reading aid method according to any one of claims 1 to 5, wherein the reading aid based on the fine reading position comprises:
determining a user reading text based on the user reading voice;
determining a text to be read based on the fine reading position;
matching the user reading text with the corresponding text in the text to be read to obtain the similarity between the user reading text and the corresponding text in the text to be read;
and prompting errors occurring in the reading process to the user based on the similarity between the user reading text and the corresponding text in the text to be read and the current reading state of the user.
7. The reading aid method according to claim 6, wherein the prompting the user of the error occurring in the reading process based on the similarity between the text read by the user and the corresponding text in the text to be read and the current reading state of the user specifically comprises:
if the similarity between the user reading text and the corresponding text in the text to be read is smaller than a first preset similarity threshold, calculating the similarity between the user reading text and the text before or after the corresponding text in the text to be read;
and if the similarity between the text read by the user and the text before or after the corresponding text in the text to be read is smaller than a second preset similarity threshold value, or the user pause time is longer than preset time, prompting the user of errors occurring in the reading process based on the current reading state of the user.
8. The reading aid method according to claim 7, wherein the current reading status of the user is determined based on at least one of difficulty of reading the content, rate of change of reading speed, user's pause time, user's reading accuracy, and historical prompting times.
9. The reading aid method according to any one of claims 1 to 5, wherein the determining a page reading position of a finger of the user on the page image to be read specifically includes:
determining the stable state of the finger of the user based on the finger coordinate position of the finger of the user in the continuous multi-frame acquisition page images;
and if the stable state of the user finger is stable, determining the page reading position based on the finger coordinate position in the continuous multi-frame collected page images.
10. An assistive reading device, comprising:
the page reading position positioning unit is used for determining the page reading position of the finger of the user on the page image to be read;
the rough reading position positioning unit is used for determining the rough reading position of the page reading position mapped in the reference page image based on the page reading position and the coordinate mapping relation between the page image to be read and the corresponding reference page image;
the fine reading position positioning unit is used for determining a fine reading position in the reference page image based on a page interested region of the page reading position in the page image to be read and a reference interested region of the rough reading position in the reference page image;
and the auxiliary reading unit is used for performing auxiliary reading based on the fine reading position.
11. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the assistive reading method according to any of claims 1 to 9 are implemented when the processor executes the program.
12. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the reading-assist method according to any one of claims 1 to 9.
CN202010301997.XA 2020-04-16 2020-04-16 Auxiliary reading method, auxiliary reading device, electronic equipment and storage medium Active CN111539405B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010301997.XA CN111539405B (en) 2020-04-16 2020-04-16 Auxiliary reading method, auxiliary reading device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010301997.XA CN111539405B (en) 2020-04-16 2020-04-16 Auxiliary reading method, auxiliary reading device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN111539405A true CN111539405A (en) 2020-08-14
CN111539405B CN111539405B (en) 2023-05-30

Family

ID=71976788

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010301997.XA Active CN111539405B (en) 2020-04-16 2020-04-16 Auxiliary reading method, auxiliary reading device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN111539405B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112230876A (en) * 2020-10-13 2021-01-15 华南师范大学 Artificial intelligence reading accompanying method and reading accompanying robot
CN112988012A (en) * 2021-03-25 2021-06-18 百度在线网络技术（北京）有限公司 Image display method, device, equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130152014A1 (en) * 2011-12-12 2013-06-13 Qualcomm Incorporated Electronic reader display control
US20170228018A1 (en) * 2014-10-13 2017-08-10 Thomason Licensig Method for controlling the displaying of text for aiding reading on a display device, and apparatus adapted for carrying out the method, computer program, and computer readable storage medium
CN110991371A (en) * 2019-12-09 2020-04-10 深圳一块互动网络技术有限公司 Intelligent reading learning method based on coordinate recognition

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130152014A1 (en) * 2011-12-12 2013-06-13 Qualcomm Incorporated Electronic reader display control
US20170228018A1 (en) * 2014-10-13 2017-08-10 Thomason Licensig Method for controlling the displaying of text for aiding reading on a display device, and apparatus adapted for carrying out the method, computer program, and computer readable storage medium
CN110991371A (en) * 2019-12-09 2020-04-10 深圳一块互动网络技术有限公司 Intelligent reading learning method based on coordinate recognition

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
邢玲;马建国;李幼平;刘志文;: "一种基于UCL的中文网页信息过滤方法" *
马慧;王科俊;: "采用旋转校正的指静脉图像感兴趣区域提取方法" *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112230876A (en) * 2020-10-13 2021-01-15 华南师范大学 Artificial intelligence reading accompanying method and reading accompanying robot
CN112988012A (en) * 2021-03-25 2021-06-18 百度在线网络技术（北京）有限公司 Image display method, device, equipment and storage medium

Also Published As

Publication number Publication date
CN111539405B (en) 2023-05-30

Similar Documents

Publication Publication Date Title
CN109817046B (en) Learning auxiliary method based on family education equipment and family education equipment
US8194934B2 (en) Apparatus for and method of using reliability information to produce and update image recognition data
TWI321294B (en) Method and device for determining at least one recognition candidate for a handwritten pattern
JP2004518198A (en) Method, device and computer program for recognizing handwritten characters
CN111539405B (en) Auxiliary reading method, auxiliary reading device, electronic equipment and storage medium
JP2003308480A (en) On-line handwritten character pattern recognizing editing device and method, and computer-aided program to realize method
CN111401318B (en) Action recognition method and device
US20220036112A1 (en) Word recognition method, apparatus and storage medium
CN113688273B (en) Graphic question answering and judging method and device
EP3828685A1 (en) Gesture stroke recognition in touch-based user interface input
CN113723420A (en) Scanning method and related equipment thereof
CN111507925B (en) Repair processing method, device, equipment and storage medium
JPH10307889A (en) Character recognition method, its device and recording medium recording character recognition program
CN110795918A (en) Method, device and equipment for determining reading position
CN113970996B (en) Character gesture recognition method and system
CN112560728B (en) Target object identification method and device
CN111553365B (en) Question selection method and device, electronic equipment and storage medium
CN111563511B (en) Method and device for intelligent frame questions, electronic equipment and storage medium
CN111563497B (en) Frame question method and device based on moving track, electronic equipment and storage medium
CN111582281B (en) Picture display optimization method and device, electronic equipment and storage medium
CN109766413B (en) Searching method applied to family education equipment and family education equipment
CN111104886A (en) Gesture recognition method, device, equipment and storage medium
JP2761679B2 (en) Online handwritten character recognition device
CN111563498B (en) Method and device for collecting questions, electronic equipment and storage medium
CN115859907B (en) Reading annotation zoom display method, system and readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information
CB02 Change of applicant information

Address after: 230031 China (Anhui) pilot Free Trade Zone, Hefei, Anhui province 6 / F and 23 / F, scientific research building, building 2, zone a, China sound Valley, No. 3333 Xiyou Road, high tech Zone, Hefei

Applicant after: Anhui taoyun Technology Co.,Ltd.

Address before: 230031 9th floor, building 1, tianyuandike science and Technology Park, 66 Qianshui East Road, high tech Zone, Hefei City, Anhui Province

Applicant before: ANHUI TAOYUN TECHNOLOGY Co.,Ltd.

GR01 Patent grant
GR01 Patent grant