CN116935416A - Nib positioning method and device, terminal equipment and storage medium - Google Patents
Nib positioning method and device, terminal equipment and storage medium Download PDFInfo
- Publication number
- CN116935416A CN116935416A CN202210397872.0A CN202210397872A CN116935416A CN 116935416 A CN116935416 A CN 116935416A CN 202210397872 A CN202210397872 A CN 202210397872A CN 116935416 A CN116935416 A CN 116935416A
- Authority
- CN
- China
- Prior art keywords
- nib
- region
- coordinate
- size
- frame image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000001514 detection method Methods 0.000 claims abstract description 131
- 238000004590 computer program Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 description 20
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 210000000707 wrist Anatomy 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The embodiment of the application discloses a nib positioning method, a nib positioning device, terminal equipment and a storage medium, wherein the method can comprise the following steps: performing nib detection on the current frame image by using a first nib detection model to obtain a first nib coordinate and a first nib region; determining a second nib region according to the position relation between the first nib coordinate and the first nib region; the size of the second pen point area is a first size which is alpha times of the size of the current frame image; and performing nib detection on the second nib region through the second nib detection model so as to obtain target nib coordinates corresponding to the current frame image. By implementing the method, the precision of the nib positioning can be improved.
Description
Technical Field
The present application relates to the field of terminal devices, and in particular, to a method and apparatus for positioning a pen tip, a terminal device, and a storage medium.
Background
The current terminal equipment mostly has the functions of point reading, question searching, note sorting and the like. In practice, it is found that when the terminal device implements these functions, it is often necessary to obtain information such as handwriting and area written by the user using the pen, and it is important how to accurately locate the position of the pen point during the writing process of the user.
Disclosure of Invention
The embodiment of the application provides a nib positioning method, a nib positioning device, terminal equipment and a storage medium, which can realize accurate nib positioning.
An embodiment of the present application provides a method for positioning a pen tip, including:
performing nib detection on the current frame image by using a first nib detection model to obtain a first nib coordinate and a first nib region; the first pen point detection model is trained by a plurality of first sample images, and each first sample image carries corresponding calibration pen point coordinates and a first calibration pen point area;
determining a second nib region according to the position relation between the first nib coordinate and the first nib region; the size of the second pen point area is a first size which is alpha times the size of the current frame image, and alpha is larger than 0 and smaller than 1;
performing nib detection on the second nib region through a second nib detection model to obtain target nib coordinates corresponding to the current frame image; the second nib detection model is trained by a plurality of second sample images, each second sample image is an image of a second calibration nib region cut from each first sample image, and each second sample image carries corresponding calibration nib coordinates; the second sample image has a second size that is a multiple of the corresponding first sample image size.
As an optional implementation manner, in the first aspect of the embodiment of the present application, the determining the second nib area according to the positional relationship between the first nib coordinate and the first nib area includes:
if the first nib coordinate is in the first nib region, determining a second nib region according to a distance value between the first nib coordinate and a central point coordinate of the first nib region;
and if the first nib coordinate is not in the first nib region, determining a second nib region taking the first nib coordinate as a central point coordinate.
In a first aspect of the present embodiment, the determining the second nib region according to the distance value between the first nib coordinate and the center point coordinate of the first nib region includes:
if the distance value between the first nib coordinate and the central point coordinate of the first nib region is smaller than or equal to a width threshold value, determining a second nib region taking the first nib coordinate as the central point coordinate, wherein the width threshold value is beta times of the width of the current frame image, and beta is larger than 0 and smaller than 1;
If the distance value between the first nib coordinate and the central point coordinate of the first nib region is larger than the width threshold, determining a third nib region which takes the first nib coordinate as the central coordinate and has the same region size as the first nib region, determining a union region of the third nib region and the first nib region, and determining a second nib region according to the union region.
As an optional implementation manner, in the first aspect of the embodiment of the present application, the determining the second nib area according to the union area includes:
acquiring an circumscribed rectangular area corresponding to the union area;
if the size of the circumscribed rectangular area is smaller than the first size, the circumscribed rectangular area is expanded in equal proportion to obtain a second nib area with the area size being the first size;
and if the size of the circumscribed rectangular area is larger than the first size, the circumscribed rectangular area is reduced in an equal proportion to obtain a second nib area with the area size of the first size.
As an optional implementation manner, in the first aspect of the embodiment of the present application, the first nib detection model includes a first backbone network, a first detector, and a first regressor; the method for detecting the pen point of the current frame image by using the first pen point detection model to obtain the first pen point coordinate and the first pen point area comprises the following steps:
Extracting image features corresponding to the current frame image by using the first backbone network;
processing the image features by using the first detector to obtain a first nib region;
and carrying out regression processing on the image features by using the first regression device to obtain a first nib coordinate.
In an optional implementation manner, in a first aspect of the embodiment of the present application, after the performing, by using a second nib detection model, nib detection on the second nib area to obtain the target nib coordinate corresponding to the current frame image, the method further includes:
calculating a movement distance value corresponding to the pen point according to the target pen point coordinate corresponding to the current frame image and the target pen point coordinate corresponding to the previous frame image;
and if the moving distance value is larger than the moving distance threshold value, determining that the target nib coordinate corresponding to the current frame image is an invalid coordinate.
As an optional implementation manner, in the first aspect of the embodiment of the present application, after determining that the target pen point coordinate corresponding to the current frame image is an invalid coordinate, the method further includes:
acquiring target nib coordinates corresponding to a plurality of target frame images; the target frame image is an image acquired before the current frame image, and the coordinates of a target pen point corresponding to the target frame image are effective coordinates;
And processing target nib coordinates corresponding to each target frame image by using a preset tracking algorithm to acquire a motion curve of the nib, and predicting effective target nib coordinates corresponding to the current frame image according to the motion curve.
In a second aspect, an embodiment of the present application provides a pen tip positioning device, including
The first nib detection unit is used for carrying out nib detection on the current frame image by utilizing the first nib detection model so as to obtain a first nib coordinate and a first nib region; the first pen point detection model is trained by a plurality of first sample images, and each first sample image carries corresponding calibration pen point coordinates and a first calibration pen point area;
a nib region determining unit configured to determine a second nib region according to a positional relationship between the first nib coordinates and the first nib region; the size of the second pen point area is a first size which is alpha times the size of the current frame image, and alpha is larger than 0 and smaller than 1;
the second nib detection unit is used for carrying out nib detection on the second nib region through a second nib detection model so as to obtain target nib coordinates corresponding to the current frame image; the second nib detection model is trained by a plurality of second sample images, each second sample image is an image of a second calibration nib region cut from each first sample image, and each second sample image carries corresponding calibration nib coordinates; the second sample image has a second size that is a multiple of the corresponding first sample image size.
A third aspect of an embodiment of the present application provides a terminal device, which may include:
a memory storing executable program code;
and a processor coupled to the memory;
the processor invokes the executable program code stored in the memory, which when executed by the processor causes the processor to implement the method according to the first aspect of the embodiment of the present application.
A fourth aspect of the embodiments of the present application provides a computer readable storage medium having stored thereon executable program code which, when executed by a processor, implements a method according to the first aspect of the embodiments of the present application.
A fifth aspect of an embodiment of the application discloses a computer program product which, when run on a computer, causes the computer to perform any of the methods disclosed in the first aspect of the embodiment of the application.
A sixth aspect of the embodiments of the present application discloses an application publishing platform for publishing a computer program product, wherein the computer program product, when run on a computer, causes the computer to perform any of the methods disclosed in the first aspect of the embodiments of the present application.
From the above technical solutions, the embodiment of the present application has the following advantages:
in the embodiment of the application, the first nib detection model is utilized to carry out nib detection on the current frame image so as to obtain a first nib coordinate and a first nib region; determining a second nib region according to the position relation between the first nib coordinate and the first nib region; the size of the second pen point area is a first size which is alpha times of the size of the current frame image; and performing nib detection on the second nib region through the second nib detection model so as to obtain target nib coordinates corresponding to the current frame image.
According to the method, first-level detection is carried out on the nib coordinates in the current frame image through the first nib detection model to obtain first nib coordinates and a first nib region, then the position relation between the first nib coordinates and the first nib region is integrated, a more effective nib region, namely a second nib region, can be determined, second-level detection is carried out on the nib coordinates in the second nib region through the second nib detection model, and high-precision nib coordinates can be obtained. Furthermore, the pen point is positioned based on image acquisition and identification, and other additional auxiliary hardware is not needed, so that the cost is reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the description of the embodiments and the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings.
FIG. 1A is a schematic view of a pen tip positioning method according to an embodiment of the present application;
FIG. 1B is a diagram of a current frame image disclosed in an embodiment of the present application;
FIG. 1C is a flow chart of a method for positioning a pen tip according to an embodiment of the present application;
FIG. 2A is a flow chart of another nib positioning method according to an embodiment of the present application;
FIG. 2B is a diagram of a first nib region in a current frame image in accordance with an embodiment of the present application;
FIG. 2C is a diagram of a third nib region in a current frame image in accordance with an embodiment of the present application;
FIG. 2D is a diagram of a circumscribed rectangular region in a current frame image in accordance with an embodiment of the present application;
FIG. 2E is a diagram of a second nib region in a current frame image according to an embodiment of the present application;
FIG. 3 is a block diagram of a nib positioning apparatus according to an embodiment of the present application;
Fig. 4 is a block diagram of a terminal device according to an embodiment of the present application.
Detailed Description
The embodiment of the application provides a nib positioning method, a nib positioning device, terminal equipment and a storage medium, which can realize accurate nib positioning.
In order that those skilled in the art will better understand the present application, reference will now be made to the accompanying drawings in which embodiments of the application are illustrated, it being apparent that the embodiments described are only some, but not all, of the embodiments of the application. Based on the embodiments of the present application, it should be understood that the present application is within the scope of protection.
It will be appreciated that the terminal devices involved in embodiments of the present application may include general handheld, on-screen electronic terminal devices, such as cell phones, smart phones, portable terminals, personal digital assistants (PersonalDigitalAssistant, PDA), portable multimedia player (PersonalMediaPlayer, PMP) devices, notebook computers, notePad (NotePad), wireless broadband (Wibro) terminals, tablet computers (PersonalComputer, PC), smart PCs, sales terminals (PointofSales, POS), and vehicle-mounted computers.
The terminal device may also comprise a wearable device. The wearable device may be worn directly on the user or be a portable electronic device integrated into the user's clothing or accessories. The wearable device is not only a hardware device, but also can realize powerful intelligent functions through software support, data interaction and cloud server interaction, such as: the mobile phone terminal has the advantages of calculating function, positioning function and alarming function, and can be connected with mobile phones and various terminals. Wearable devices may include, but are not limited to, wrist-supported watch types (e.g., watches, wrist products, etc.), foot-supported shoes (e.g., shoes, socks, or other leg wear products), head-supported Glass types (e.g., glasses, helmets, headbands, etc.), and smart apparel, school bags, crutches, accessories, etc. in various non-mainstream product forms.
It should be noted that, the execution main body of the nib positioning method disclosed in the embodiment of the present application may be a terminal device or a nib positioning device, which is not limited by the embodiment of the present application. The following embodiments mainly take terminal devices as examples.
Referring to fig. 1A, fig. 1A is a schematic view of a pen tip positioning method according to an embodiment of the application. The schematic view of the scene shown in fig. 1A may include a terminal device 10, a writing surface 20, a writing pen 30, and a pen-holding hand 40. Wherein, the terminal device 10 includes a camera 11, and in the process of controlling the writing pen 30 to write on the writing surface 20 by the pen holding hand 40, the terminal device 10 can control the front camera 11 to collect the writing picture in real time, so as to position the pen tip of the writing pen 30 on the writing surface 20 in real time based on the collected writing picture.
The following describes a method of positioning the pen tip of the writing pen 30, taking the current frame image as an example. Referring to fig. 1B, fig. 1B is a schematic diagram of a current frame image according to an embodiment of the present application. The current frame image shown in fig. 1B includes a writing pen 30, a writing surface 20, and a pen-holding hand 40. The terminal device 10 first performs a first level detection on the nib coordinates in the current frame image through the first nib detection model to obtain first nib coordinates and a first nib region, then synthesizes the position relationship between the first nib coordinates and the first nib region, can determine a more effective nib region, namely a second nib region, and finally performs a second level detection on nib coordinates in the second nib region through the second nib detection model, so as to obtain high-precision nib coordinates. Furthermore, the terminal device 10 realizes the positioning of the pen point based on image acquisition and recognition, and no additional auxiliary hardware is needed, thereby being beneficial to reducing the cost.
Referring to fig. 1C, fig. 1C is a flow chart of a pen tip positioning method according to an embodiment of the application. The nib positioning method as shown in fig. 1C may include the steps of:
101. and performing nib detection on the current frame image by using the first nib detection model so as to obtain a first nib coordinate and a first nib region.
The first pen point detection model is trained by a plurality of first sample images, and each first sample image carries corresponding calibration pen point coordinates and a first calibration pen point area.
In an embodiment of the present application, the first nib detection model may be a regression model of a multitasking architecture, and the first nib detection model may include nib coordinate regression branches and nib region detection branches. The nib coordinate regression branch is used for roughly regressing nib coordinates in the current frame image to obtain first nib coordinates, and the nib region detection branch is used for roughly regressing nib regions in the current frame image to obtain first nib regions.
Based on this, the learning of the first nib detection model includes not only learning nib coordinate regression branches using the plurality of first sample images and the calibrated nib coordinates corresponding to the respective first sample images, but also learning nib region detection branches using the plurality of first sample images and the first calibrated nib regions corresponding to the respective first sample images. The model learning method of the multitasking architecture can enable the first nib detection model to learn more effective features, and is beneficial to improving the detection precision of the first nib detection model.
The sizes of the different first sample images may be the same or different, which is not limited in the embodiment of the present application. If the plurality of first sample images include a plurality of sizes, the learned first pen point detection model is applicable to the images with the plurality of sizes, and has strong universality.
The first nib coordinate refers to the position of the nib deduced by the first nib detection model in the current frame image, the first nib area refers to the area of the nib deduced by the first nib detection model in the current frame image, and the number of pixels contained in the first nib area is smaller than that in the current frame image.
In some embodiments, performing nib detection on the current frame image using the first nib detection model to obtain the first nib coordinate and the first nib region may include: and extracting image features corresponding to the current frame image by using the first nib detection model, and determining first nib coordinates and a first nib region according to the image features corresponding to the current frame image.
In some embodiments, the image features corresponding to the current frame image include at least any one of: the characteristics of the pen shaft, the characteristics of the hand of the pen holder, the characteristics of the first knuckle of the pen-holding finger, etc. Wherein the first knuckle of the pen-holding finger is the knuckle closest to the pen tip. The characteristics of the pen holder, the pen holder hand and the pen holding finger are obvious, and the accuracy of the detection result obtained by taking the obvious characteristics as the reference is higher.
102. And determining a second nib region according to the position relation between the first nib coordinate and the first nib region.
The size of the second nib area is a first size, the first size is alpha times of the size of the current frame image, and alpha is larger than 0 and smaller than 1. The positional relationship between the first nib coordinate and the first nib region may include the first nib coordinate being within the first nib region, the first nib coordinate not being within the first nib region, and the like. If the first nib coordinate is not in the first nib region, the fact that the detection result of the first nib detection model has larger deviation from the real nib region is indicated, and the detection result is inaccurate; otherwise, if the first nib coordinate is in the first nib region, it is indicated that the detection result of the first nib detection model has smaller deviation from the real nib region, and the detection result is more accurate.
In some embodiments, determining the second nib region from the positional relationship between the first nib coordinate and the first nib region may include, but is not limited to, the following:
mode 1, when a first nib coordinate is within a first nib region, determining a second nib region having the first nib coordinate as a center point coordinate;
Mode 2, when the first nib coordinate is in the first nib region and the first nib region is not the first size, adjusting the size of the first nib region to the first size, and taking the first nib region with the adjusted size as the second nib region;
mode 3, when the first nib coordinate is within the first nib region and the first nib region is of the first size, the first nib region is taken as the second nib region;
mode 4, determining a second nib region having the first nib coordinate as a center point coordinate when the first nib coordinate is not located in the first nib region;
mode 5, when the first nib coordinate is not within the first nib region, determining a fourth nib region having the first nib coordinate as a center coordinate and a region size of a third size, determining a union region of the fourth nib region and the first nib region, and when the size of the union region is larger than the first size, reducing the union region in an equal ratio to obtain a second nib region having the region size of the first size, and when the size of the union region is smaller than the first size, enlarging the union region in an equal ratio to obtain a second nib region having the region size of the first size.
In some embodiments, the size of the third dimension may be related to a distance value between the first nib coordinate and the center point coordinate of the first nib region, the larger the distance value, the larger the third dimension, and vice versa, the smaller the third dimension.
103. And performing nib detection on the second nib region through the second nib detection model so as to obtain target nib coordinates corresponding to the current frame image.
The second nib detection model is trained by a plurality of second sample images, each second sample image is an image of a second calibration nib region cut from each first sample image, and each second sample image carries corresponding calibration nib coordinates. The second sample image has a second size that is a multiple of the corresponding first sample image size.
In some embodiments, performing nib detection on the second nib region by using the second nib detection model to obtain the target nib coordinate corresponding to the current frame image may include: and extracting image features corresponding to the second nib region through the second nib detection model, and determining target nib coordinates corresponding to the current frame image according to the image features corresponding to the second nib region. For the description of the image features corresponding to the second nib region, please refer to the description of the image features corresponding to the current frame image, which is not repeated herein.
In the embodiment of the application, compared with the pixels of the current frame image, the number of the pixels of the image in the second nib region is smaller, and the background interference is less. Based on this, in some embodiments, the second nib detection model may be a lightweight regression model, which helps to improve the detection efficiency of the second nib detection model, and thus helps to improve the positioning efficiency of the nib.
By implementing the method, first-level detection is performed on the nib coordinates in the current frame image through the first nib detection model to obtain the first nib coordinates and the first nib region, then a more accurate nib region, namely a second nib region, is determined by integrating the position relationship between the first nib coordinates and the first nib region, and finally second-level detection is performed on the nib coordinates in the second nib region through second nib detection, so that the nib coordinates with high precision can be obtained. Furthermore, the pen point is positioned based on image acquisition and identification, and other additional auxiliary hardware is not needed, so that the cost is reduced.
Referring to fig. 2A, fig. 2A is a flow chart of another nib positioning method according to an embodiment of the application. The nib positioning method as shown in fig. 2A may include the steps of:
201. And performing nib detection on the current frame image by using the first nib detection model so as to obtain a first nib coordinate and a first nib region.
In some embodiments, the first nib detection model may include a first backbone network, a first detector, and a first regressor. The first backbone network and the first detector form a nib region detection branch, and the first backbone network and the first regressor form a nib coordinate regression branch.
In some embodiments, performing nib detection on the current frame image using the first nib detection model to obtain the first nib coordinate and the first nib region may include: extracting image features corresponding to the current frame image by using a first backbone network; processing the image features by using a first detector to obtain a first nib region; and carrying out regression processing on the image features by using a first regression device to obtain a first nib coordinate.
202. And if the first nib coordinate is in the first nib region, determining a second nib region according to the distance value between the first nib coordinate and the central point coordinate of the first nib region.
In some embodiments, determining the second nib region from the distance value between the first nib coordinate and the center point coordinate of the first nib region may include, but is not limited to, the following:
Mode 1, if the distance value between the first nib coordinate and the center point coordinate of the first nib region is smaller than or equal to a width threshold, determining a second nib region with the first nib coordinate as the center point coordinate, wherein the width threshold is β times the width of the current frame image, and β is greater than 0 and smaller than 1;
mode 2, if the distance value between the first nib coordinate and the center point coordinate of the first nib region is smaller than or equal to the width threshold, and the size of the first calibration nib region is the first size, taking the first calibration nib region as the second nib region;
mode 3, if the distance value between the first nib coordinate and the center point coordinate of the first nib region is greater than the width threshold, determining a third nib region having the same region size as the first nib region with the first nib coordinate as the center coordinate, determining a union region of the third nib region and the first nib region, and determining a second nib region according to the union region.
In some embodiments, the distance value between the first nib coordinate and the center point coordinate of the first nib region may include, but is not limited to, a euclidean distance value between the first nib coordinate and the center point coordinate of the first nib region, and/or a distance value of the first nib coordinate and the center point coordinate of the first nib region in a horizontal direction.
In some embodiments, determining the second nib region from the union region may include: acquiring an circumscribed rectangular area corresponding to the union area; if the size of the circumscribed rectangular area is smaller than the first size, the circumscribed rectangular area is expanded in equal proportion to obtain a second nib area with the area size being the first size; if the size of the circumscribed rectangular area is larger than the first size, the circumscribed rectangular area is reduced in an equal proportion, and a second nib area with the area size being the first size is obtained. The union region refers to a minimum region including the first nib region and the second nib region, and the circumscribed rectangular region of the union region refers to a minimum rectangular region including the first nib region and the second nib region.
By way of example, the determination of the second nib region is described below in connection with fig. 2B-2E. Wherein: the current frame image as shown in fig. 2B includes a first nib region S1 and a first nib coordinate P. The current frame image shown in fig. 2C includes a third nib region S2 having the same region size as the first nib region S1 with the first nib coordinate P as a center coordinate. The current frame image as shown in fig. 2D includes a circumscribed rectangular region S3 and a first nib coordinate P. The current frame image as shown in fig. 2E includes a second nib region S4 having a region size of a first size and a first nib coordinate P.
By implementing the method, when a small deviation exists between the detection result of the first nib detection model and the real nib region, the division granularity of the deviation degree can be further improved based on the distance value between the first nib coordinate and the central point coordinate of the first nib region (if the distance value between the first nib coordinate and the central point coordinate of the first nib region is smaller than or equal to the width threshold value, the deviation degree of the detection junction of the first nib detection model is extremely small, if the distance value between the first nib coordinate and the central point coordinate of the first nib region is larger than the width threshold value, the detection result of the first nib detection model is proved to have a certain deviation, finally, the second nib region is determined by adopting different modes according to different deviation degrees, and the effectiveness of the second nib region can be further improved.
203. If the first nib coordinate is not in the first nib region, determining a second nib region taking the first nib coordinate as a central point coordinate.
204. And performing nib detection on the second nib region through the second nib detection model so as to obtain target nib coordinates corresponding to the current frame image.
In some embodiments, after step 204, a moving distance value corresponding to the pen tip may be calculated according to the target pen tip coordinate corresponding to the current frame image and the target pen tip coordinate corresponding to the previous frame image; and when the moving distance value is larger than the moving distance threshold value, determining that the target nib coordinate corresponding to the current frame image is an invalid coordinate.
The moving distance value may refer to a moving distance between a point of time of acquisition of a previous frame image and a point of time of acquisition of a current frame image. The moving distance threshold value can be obtained by counting and analyzing moving distance values of the pen point between a plurality of adjacent two image acquisition time points.
In the embodiment of the application, if the moving distance value is smaller than or equal to the moving distance threshold value, determining the target nib coordinate corresponding to the current frame image as the effective coordinate.
In some embodiments, after determining that the target nib coordinate corresponding to the current frame image is an invalid coordinate, the target nib coordinates corresponding to the plurality of target frame images may also be obtained; the target frame image is an image acquired before the current frame image, and the coordinates of a target pen point corresponding to the target frame image are effective coordinates; and processing target nib coordinates corresponding to each target frame image by using a preset tracking algorithm to acquire a nib motion curve, and predicting to acquire effective target nib coordinates corresponding to the current frame image according to the motion curve.
In some embodiments, the preset tracking algorithm may include, but is not limited to, a Kalman filtering algorithm, a Camshift algorithm, and the like.
By implementing the method, the validity of the target nib coordinate corresponding to the current frame image obtained by the nib detection model can be detected, and under the condition that the target nib coordinate corresponding to the current frame image is an invalid coordinate, the target nib coordinate corresponding to the current frame image can be obtained by prediction according to the target nib coordinates corresponding to a plurality of images before the current frame image, so that the error degree of nib positioning can be reduced.
By implementing the method, first-level detection is performed on the nib coordinates in the current frame image through the first nib detection model to obtain the first nib coordinates and the first nib region, and then the second nib region is determined by integrating the first nib region and the first nib coordinates under the condition that the first nib coordinates are in the first nib region, so that the effectiveness of the second nib region can be further ensured, and the positioning accuracy of the nib can be further improved. Furthermore, the pen point is positioned based on image acquisition and identification, and other additional auxiliary hardware is not needed, so that the cost is reduced.
Referring to fig. 3, fig. 3 is a block diagram illustrating a pen tip positioning device according to an embodiment of the application. The nib positioning apparatus as shown in fig. 3 may include a first nib detecting unit 301, a nib region determining unit 302, and a second nib detecting unit 303; wherein:
A first nib detection unit 301, configured to perform nib detection on the current frame image by using a first nib detection model, so as to obtain a first nib coordinate and a first nib region; the first pen point detection model is trained by a plurality of first sample images, and each first sample image carries corresponding calibration pen point coordinates and a first calibration pen point area;
a nib region determining unit 302 for determining a second nib region according to a positional relationship between the first nib coordinates and the first nib region; the size of the second pen point area is a first size which is alpha times the size of the current frame image, and alpha is more than 0 and less than 1;
a second nib detection unit 303, configured to perform nib detection on the second nib region through a second nib detection model, so as to obtain a target nib coordinate corresponding to the current frame image; the second nib detection model is trained by a plurality of second sample images, each second sample image is an image of a second calibration nib region cut from each first sample image, and each second sample image carries corresponding calibration nib coordinates; the second sample image has a second size that is a multiple of the corresponding first sample image size.
In some embodiments, the manner in which the nib region determining unit 302 is configured to determine the second nib region according to the positional relationship between the first nib coordinate and the first nib region may specifically include: a nib region determining unit 302, configured to determine, if the first nib coordinate is within the first nib region, a second nib region according to a distance value between the first nib coordinate and a center point coordinate of the first nib region; if the first nib coordinate is not in the first nib region, determining a second nib region taking the first nib coordinate as a central point coordinate.
In some embodiments, if the first nib coordinate is within the first nib region, the nib region determining unit 302 is configured to determine the second nib region according to the distance value between the first nib coordinate and the center point coordinate of the first nib region, where the method specifically includes: a nib region determining unit 302, configured to determine, if a distance value between the first nib coordinate and a center point coordinate of the first nib region is less than or equal to a width threshold, a second nib region having the first nib coordinate as the center point coordinate, the width threshold being β times the width of the current frame image, β being greater than 0 and less than 1; if the distance value between the first nib coordinate and the central point coordinate of the first nib region is larger than the width threshold, determining a third nib region which takes the first nib coordinate as the central coordinate and has the same region size as the first nib region, determining a union region of the third nib region and the first nib region, and determining a second nib region according to the union region.
In some embodiments, the manner in which the nib region determining unit 302 determines the second nib region according to the union region may specifically include: a nib region determining unit 302, configured to obtain an circumscribed rectangular region corresponding to the union region; if the size of the circumscribed rectangular area is smaller than the first size, the circumscribed rectangular area is expanded in equal proportion to obtain a second nib area with the area size being the first size; if the size of the circumscribed rectangular area is larger than the first size, the circumscribed rectangular area is reduced in an equal proportion, and a second nib area with the area size being the first size is obtained.
In some embodiments, the first nib detection model includes a first backbone network, a first detector, and a first regressor. The manner in which the second nib detection unit 303 is configured to perform nib detection on the second nib region through the second nib detection model to obtain the target nib coordinate corresponding to the current frame image may specifically include: a second nib detecting unit 303, configured to extract an image feature corresponding to the current frame image by using the first backbone network; processing the image features by using a first detector to obtain a first nib region; and carrying out regression processing on the image features by using a first regression device to obtain a first nib coordinate.
In some embodiments, the second nib detection unit 303 is further configured to perform nib detection on the second nib region through the second nib detection model, so as to obtain a target nib coordinate corresponding to the current frame image, and calculate a movement distance value corresponding to the nib according to the target nib coordinate corresponding to the current frame image and the target nib coordinate corresponding to the previous frame image; and if the moving distance value is larger than the moving distance threshold value, determining that the target nib coordinate corresponding to the current frame image is an invalid coordinate.
In some embodiments, the second nib detection unit 303 is further configured to obtain target nib coordinates corresponding to the plurality of target frame images after determining that the target nib coordinates corresponding to the current frame image are invalid coordinates; the target frame image is an image acquired before the current frame image, and the coordinates of a target pen point corresponding to the target frame image are effective coordinates; and processing target nib coordinates corresponding to each target frame image by using a preset tracking algorithm to acquire a nib motion curve, and predicting effective target nib coordinates corresponding to the current frame image according to the motion curve.
Referring to fig. 4, fig. 4 is a block diagram of a terminal device according to an embodiment of the present application. The terminal device as shown in fig. 4 may include: a processor 401, a memory 402 coupled to the processor 401, wherein the memory 402 may store one or more computer programs.
Processor 401 may include one or more processing cores. The processor 401 connects the various parts within the overall terminal device using various interfaces and lines, performs various functions of the terminal device and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 402, and invoking data stored in the memory 402. Alternatively, the processor 401 may be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 401 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), a modem, etc. The CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for being responsible for rendering and drawing of display content; the modem is used to handle wireless communications. It will be appreciated that the modem may not be integrated into the processor 401 and may be implemented by a single communication chip.
The Memory 402 may include a random access Memory (Random Access Memory, RAM) or a Read-Only Memory (ROM). Memory 402 may be used to store instructions, programs, code sets, or instruction sets. The memory 402 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (e.g., a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, and the like. The storage data area may also store data created by the terminal device in use, etc.
In an embodiment of the present application, the processor 401 further has the following functions:
performing nib detection on the current frame image by using a first nib detection model to obtain a first nib coordinate and a first nib region; the first pen point detection model is trained by a plurality of first sample images, and each first sample image carries corresponding calibration pen point coordinates and a first calibration pen point area;
determining a second nib region according to the position relation between the first nib coordinate and the first nib region; the size of the second pen point area is a first size which is alpha times the size of the current frame image, and alpha is more than 0 and less than 1;
Performing nib detection on the second nib region through a second nib detection model to obtain target nib coordinates corresponding to the current frame image; the second nib detection model is trained by a plurality of second sample images, each second sample image is an image of a second calibration nib region cut from each first sample image, and each second sample image carries corresponding calibration nib coordinates; the second sample image has a second size that is a multiple of the corresponding first sample image size.
In an embodiment of the present application, the processor 401 further has the following functions:
if the first nib coordinate is in the first nib region, determining a second nib region according to a distance value between the first nib coordinate and a central point coordinate of the first nib region;
if the first nib coordinate is not in the first nib region, determining a second nib region taking the first nib coordinate as a central point coordinate.
In an embodiment of the present application, the processor 401 further has the following functions:
if the distance value between the first nib coordinate and the center point coordinate of the first nib region is smaller than or equal to a width threshold value, determining a second nib region taking the first nib coordinate as the center point coordinate, wherein the width threshold value is beta times of the width of the current frame image, and beta is larger than 0 and smaller than 1;
If the distance value between the first nib coordinate and the central point coordinate of the first nib region is larger than the width threshold, determining a third nib region which takes the first nib coordinate as the central coordinate and has the same region size as the first nib region, determining a union region of the third nib region and the first nib region, and determining a second nib region according to the union region.
In an embodiment of the present application, the processor 401 further has the following functions:
acquiring an circumscribed rectangular area corresponding to the union area;
if the size of the circumscribed rectangular area is smaller than the first size, the circumscribed rectangular area is expanded in equal proportion to obtain a second nib area with the area size being the first size;
if the size of the circumscribed rectangular area is larger than the first size, the circumscribed rectangular area is reduced in an equal proportion, and a second nib area with the area size being the first size is obtained.
In the embodiment of the application, a first nib detection model comprises a first backbone network, a first detector and a first regressor; the processor 401 also has the following functions:
extracting image features corresponding to the current frame image by using a first backbone network;
processing the image features by using a first detector to obtain a first nib region;
And carrying out regression processing on the image features by using a first regression device to obtain a first nib coordinate.
In an embodiment of the present application, the processor 401 further has the following functions:
calculating a moving distance value corresponding to the pen point according to the target pen point coordinate corresponding to the current frame image and the target pen point coordinate corresponding to the previous frame image;
and if the moving distance value is larger than the moving distance threshold value, determining that the target nib coordinate corresponding to the current frame image is an invalid coordinate.
In an embodiment of the present application, the processor 401 further has the following functions:
acquiring target nib coordinates corresponding to a plurality of target frame images; the target frame image is an image acquired before the current frame image, and the coordinates of a target pen point corresponding to the target frame image are effective coordinates;
and processing target nib coordinates corresponding to each target frame image by using a preset tracking algorithm to acquire a nib motion curve, and predicting effective target nib coordinates corresponding to the current frame image according to the motion curve.
The embodiment of the application discloses a computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, causes the processor to implement part or all of the steps executed by a terminal device in the above embodiment.
The embodiment of the application discloses a computer program product which enables a computer to execute part or all of the steps executed by terminal equipment in the embodiment when the computer program product runs on the computer.
The embodiment of the application discloses an application release platform which is used for releasing a computer program product, wherein when the computer program product runs on a computer, the computer is caused to execute part or all of the steps executed by terminal equipment in the embodiment.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, magnetic disk, tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk SolidStateDisk (SSD)), etc.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable magnetic disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or the like, which can store program codes.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims (10)
1. A nib positioning method, comprising:
performing nib detection on the current frame image by using a first nib detection model to obtain a first nib coordinate and a first nib region; the first pen point detection model is trained by a plurality of first sample images, and each first sample image carries corresponding calibration pen point coordinates and a first calibration pen point area;
determining a second nib region according to the position relation between the first nib coordinate and the first nib region; the size of the second pen point area is a first size which is alpha times the size of the current frame image, and alpha is larger than 0 and smaller than 1;
Performing nib detection on the second nib region through a second nib detection model to obtain target nib coordinates corresponding to the current frame image; the second nib detection model is trained by a plurality of second sample images, each second sample image is an image of a second calibration nib region cut from each first sample image, and each second sample image carries corresponding calibration nib coordinates; the second sample image has a second size that is a multiple of the corresponding first sample image size.
2. The method of claim 1, wherein determining a second nib region based on the positional relationship between the first nib coordinate and the first nib region comprises:
if the first nib coordinate is in the first nib region, determining a second nib region according to a distance value between the first nib coordinate and a central point coordinate of the first nib region;
and if the first nib coordinate is not in the first nib region, determining a second nib region taking the first nib coordinate as a central point coordinate.
3. The method of claim 2, wherein the determining a second nib region from the distance value between the first nib coordinate and the center point coordinate of the first nib region comprises:
if the distance value between the first nib coordinate and the central point coordinate of the first nib region is smaller than or equal to a width threshold value, determining a second nib region taking the first nib coordinate as the central point coordinate, wherein the width threshold value is beta times of the width of the current frame image, and beta is larger than 0 and smaller than 1;
if the distance value between the first nib coordinate and the central point coordinate of the first nib region is larger than the width threshold, determining a third nib region which takes the first nib coordinate as the central coordinate and has the same region size as the first nib region, determining a union region of the third nib region and the first nib region, and determining a second nib region according to the union region.
4. A method according to claim 3, wherein said determining a second nib region from said union region comprises:
acquiring an circumscribed rectangular area corresponding to the union area;
If the size of the circumscribed rectangular area is smaller than the first size, the circumscribed rectangular area is expanded in equal proportion to obtain a second nib area with the area size being the first size;
and if the size of the circumscribed rectangular area is larger than the first size, the circumscribed rectangular area is reduced in an equal proportion to obtain a second nib area with the area size of the first size.
5. The method of any one of claims 1-4, wherein the first nib detection model comprises a first backbone network, a first detector, and a first regressor; the method for detecting the pen point of the current frame image by using the first pen point detection model to obtain the first pen point coordinate and the first pen point area comprises the following steps:
extracting image features corresponding to the current frame image by using the first backbone network;
processing the image features by using the first detector to obtain a first nib region;
and carrying out regression processing on the image features by using the first regression device to obtain a first nib coordinate.
6. The method according to any one of claims 1 to 4, wherein after performing nib detection on the second nib region by using a second nib detection model to obtain a target nib coordinate corresponding to the current frame image, the method further comprises:
Calculating a movement distance value corresponding to the pen point according to the target pen point coordinate corresponding to the current frame image and the target pen point coordinate corresponding to the previous frame image;
and if the moving distance value is larger than the moving distance threshold value, determining that the target nib coordinate corresponding to the current frame image is an invalid coordinate.
7. The method of claim 6, wherein after determining that the target pen point coordinates corresponding to the current frame image are invalid coordinates, the method further comprises:
acquiring target nib coordinates corresponding to a plurality of target frame images; the target frame image is an image acquired before the current frame image, and the coordinates of a target pen point corresponding to the target frame image are effective coordinates;
and processing target nib coordinates corresponding to each target frame image by using a preset tracking algorithm to acquire a motion curve of the nib, and predicting effective target nib coordinates corresponding to the current frame image according to the motion curve.
8. A pen point detection device is characterized by comprising
The first nib detection unit is used for carrying out nib detection on the current frame image by utilizing the first nib detection model so as to obtain a first nib coordinate and a first nib region; the first pen point detection model is trained by a plurality of first sample images, and each first sample image carries corresponding calibration pen point coordinates and a first calibration pen point area;
A nib region determining unit configured to determine a second nib region according to a positional relationship between the first nib coordinates and the first nib region; the size of the second pen point area is a first size which is alpha times the size of the current frame image, and alpha is larger than 0 and smaller than 1;
the second nib detection unit is used for carrying out nib detection on the second nib region through a second nib detection model so as to obtain target nib coordinates corresponding to the current frame image; the second nib detection model is trained by a plurality of second sample images, each second sample image is an image of a second calibration nib region cut from each first sample image, and each second sample image carries corresponding calibration nib coordinates; the second sample image has a second size that is a multiple of the corresponding first sample image size.
9. A terminal device, comprising
A memory storing executable program code;
and a processor coupled to the memory;
the processor invoking the executable program code stored in the memory, which when executed by the processor, causes the processor to implement the method of any of claims 1-7.
10. A computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, causes the processor to implement the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210397872.0A CN116935416A (en) | 2022-04-07 | 2022-04-07 | Nib positioning method and device, terminal equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210397872.0A CN116935416A (en) | 2022-04-07 | 2022-04-07 | Nib positioning method and device, terminal equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116935416A true CN116935416A (en) | 2023-10-24 |
Family
ID=88381601
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210397872.0A Pending CN116935416A (en) | 2022-04-07 | 2022-04-07 | Nib positioning method and device, terminal equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116935416A (en) |
-
2022
- 2022-04-07 CN CN202210397872.0A patent/CN116935416A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020253657A1 (en) | Video clip positioning method and apparatus, computer device, and storage medium | |
| CN109584276B (en) | Key point detection method, device, equipment and readable medium | |
| US10416740B2 (en) | Upsampling sensors to auto-detect a fitness activity | |
| CN106951484B (en) | Picture retrieval method and device, computer equipment and computer readable medium | |
| CN107368820B (en) | Refined gesture recognition method, device and equipment | |
| CN109189879B (en) | Electronic book display method and device | |
| CN110431563B (en) | Method and device for correcting image | |
| CN112857268A (en) | Object area measuring method, device, electronic device and storage medium | |
| CN111832579B (en) | Map interest point data processing method and device, electronic equipment and readable medium | |
| CN111783626A (en) | Image recognition method and device, electronic equipment and storage medium | |
| CN111708477B (en) | Key identification method, device, equipment and storage medium | |
| CN115039015A (en) | Pose tracking method, wearable device, mobile device and storage medium | |
| CN114463856B (en) | Method, device, equipment and medium for training attitude estimation model and attitude estimation | |
| CN112912893A (en) | Detection method and device for wearing mask, terminal equipment and readable storage medium | |
| CN111382701B (en) | Motion capture method, motion capture device, electronic equipment and computer readable storage medium | |
| WO2020231575A1 (en) | System and method for selecting relevant content in an enhanced view mode | |
| WO2023186009A1 (en) | Step counting method and apparatus | |
| CN116935416A (en) | Nib positioning method and device, terminal equipment and storage medium | |
| CN116935415A (en) | Nib positioning method and device, terminal equipment and storage medium | |
| CN116295466A (en) | Map generation method, map generation device, electronic device, storage medium and vehicle | |
| CN111968030B (en) | Information generation method, apparatus, electronic device and computer readable medium | |
| US10191553B2 (en) | User interaction with information handling systems using physical objects | |
| CN114648556A (en) | Visual tracking method and device and electronic equipment | |
| CN111401182A (en) | Image detection method and device for feeding fence | |
| CN111580649B (en) | Deep learning-based aerial handwriting interaction method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |