CN116935415A

CN116935415A - Nib positioning method and device, terminal equipment and storage medium

Info

Publication number: CN116935415A
Application number: CN202210379235.0A
Authority: CN
Inventors: 王丰焱; 胡东平; 杨宗武
Original assignee: Guangdong Genius Technology Co Ltd
Current assignee: Guangdong Genius Technology Co Ltd
Priority date: 2022-04-12
Filing date: 2022-04-12
Publication date: 2023-10-24

Abstract

The embodiment of the application discloses a nib positioning method, a nib positioning device, terminal equipment and a storage medium, wherein the nib positioning method can comprise the following steps: identifying a pen holder hand contained in the current frame image, and determining a first image from the current frame image; wherein the first image comprises a nib region and at least a partial region of a pen-holder hand; performing nib detection on the first image by using a nib detection model to obtain a target nib coordinate corresponding to the current frame image; the pen point detection model is trained by a plurality of first sample images and calibrated pen point coordinates corresponding to the first sample images, wherein the first sample images comprise a pen point area and at least part of the area of a pen holder hand. By implementing the method, real-time and accurate nib positioning can be realized.

Description

Nib positioning method and device, terminal equipment and storage medium

Technical Field

The present application relates to the field of terminal devices, and in particular, to a method and apparatus for positioning a pen tip, a terminal device, and a storage medium.

Background

The current terminal equipment mostly has the functions of point reading, question searching, note sorting and the like. In practice, it is found that when the terminal device implements these functions, it is often necessary to obtain information such as handwriting and area written by the user using the pen, so it is important how to efficiently and accurately locate the position of the pen point in the writing process of the user.

Disclosure of Invention

The embodiment of the application provides a nib positioning method, a nib positioning device, terminal equipment and a storage medium, which can realize real-time and accurate nib positioning.

An embodiment of the present application provides a method for positioning a pen tip, including:

identifying a pen holder hand contained in a current frame image, and determining a first image from the current frame image; wherein the first image includes a nib region and at least a partial region of the pen-holding hand;

performing nib detection on the first image by using a nib detection model to obtain a target nib coordinate corresponding to the current frame image; the pen point detection model is trained by a plurality of first sample images and calibrated pen point coordinates corresponding to the first sample images, and the first sample images comprise a pen point area and at least part of the area of a pen holder hand.

As an optional implementation manner, in the first aspect of the embodiment of the present application, the identifying a pen-holding person hand included in the current frame image and determining a first image from the current frame image includes:

identifying a pen-holding finger contained in a current frame image through a pen-holding area detection model, extracting the characteristic of a first knuckle in the pen-holding finger, and determining a target pen area according to the characteristic of the first knuckle; the pen region detection model is trained by a plurality of second sample images and calibration pen regions corresponding to the second sample images, wherein the second sample images comprise a pen point region and the whole hand region of a pen holder hand;

Intercepting the target stylus region from the current frame image to obtain a first image;

the first sample image is an image in a calibration pen area corresponding to the second sample image, and the first sample image comprises a pen point area and a corresponding area of a first knuckle of at least a part of a pen holding finger.

identifying a first hand region of a pen-holding person hand contained in the current frame image;

expanding the first hand region according to a preset first size adjustment parameter to determine a first image region including a pen point;

intercepting the first image area from the current frame image to obtain a first image;

wherein the first sample image includes a nib region and an entire hand region of a pen-holder hand.

As an optional implementation manner, in the first aspect of the embodiment of the present application, the identifying a first hand area of a pen-holding person hand included in the current frame image includes:

acquiring a second hand region of the pen-holding hand contained in the previous frame of image;

And adjusting the position of the second hand region according to a preset displacement parameter to determine a first hand region of the pen holder hand contained in the current frame image.

In an optional implementation manner, in a first aspect of the embodiment of the present application, after the tip detection is performed on the first image by using a tip detection model to obtain the target tip coordinate corresponding to the current frame image, the method further includes:

calculating a movement distance value corresponding to the pen point according to the target pen point coordinate corresponding to the current frame image and the target pen point coordinate corresponding to the previous frame image;

and if the moving distance value is larger than the moving distance threshold value, determining that the target nib coordinate corresponding to the current frame image is an invalid coordinate.

As an optional implementation manner, in the first aspect of the embodiment of the present application, after determining that the target pen point coordinate corresponding to the current frame image is an invalid coordinate, the method further includes:

acquiring target nib coordinates corresponding to a plurality of target frame images; the target frame image is an image acquired before the current frame image, and the coordinates of a target pen point corresponding to the target frame image are effective coordinates;

And processing target nib coordinates corresponding to each target frame image by using a preset tracking algorithm to acquire a motion curve of the nib, and predicting to acquire effective target nib coordinates corresponding to the current frame image according to the motion curve.

In an optional implementation manner, in a first aspect of the embodiment of the present application, the nib detection model includes a lightweight backbone network and a regressor, and the nib detection is performed on the first image by using the nib detection model to obtain a target nib coordinate corresponding to the current frame image, where the method includes:

extracting image features of the first image through the lightweight backbone network;

and carrying out regression processing on the image features through the regressive device so as to determine the target nib coordinates corresponding to the current frame image.

In a second aspect, an embodiment of the present application provides a pen tip positioning device, including

An image extraction unit, configured to identify a pen-holding person hand included in a current frame image, and determine a first image from the current frame image; wherein the first image includes a nib region and at least a partial region of the pen-holding hand;

the nib detection unit is used for carrying out nib detection on the first image by utilizing a nib detection model so as to obtain a target nib coordinate corresponding to the current frame image; the pen point detection model is trained by a plurality of first sample images and calibrated pen point coordinates corresponding to the first sample images, and the first sample images comprise a pen point area and at least part of the area of a pen holder hand.

A third aspect of an embodiment of the present application provides a terminal device, which may include:

a memory storing executable program code;

and a processor coupled to the memory;

the processor invokes the executable program code stored in the memory, which when executed by the processor causes the processor to implement the method according to the first aspect of the embodiment of the present application.

A fourth aspect of the embodiments of the present application provides a computer readable storage medium having stored thereon executable program code which, when executed by a processor, implements a method according to the first aspect of the embodiments of the present application.

A fifth aspect of an embodiment of the application discloses a computer program product which, when run on a computer, causes the computer to perform any of the methods disclosed in the first aspect of the embodiment of the application.

A sixth aspect of the embodiments of the present application discloses an application publishing platform for publishing a computer program product, wherein the computer program product, when run on a computer, causes the computer to perform any of the methods disclosed in the first aspect of the embodiments of the present application.

From the above technical solutions, the embodiment of the present application has the following advantages:

in the embodiment of the application, a pen holder hand contained in a current frame image is identified, and a first image is determined from the current frame image; wherein the first image comprises a nib region and at least a partial region of a pen-holder hand; performing nib detection on the first image by using a nib detection model to obtain a target nib coordinate corresponding to the current frame image; the pen point detection model is trained by a plurality of first sample images and calibrated pen point coordinates corresponding to the first sample images, wherein the first sample images comprise a pen point area and at least part of the area of a pen holder hand.

By implementing the method, firstly, the pen point contained in the current frame image is roughly positioned by identifying the pen-holding hand in the current frame image, so that the rough position of the pen point, namely the area corresponding to the first image, is obtained. And then on the basis of the rough position of the nib, the nib detection model is used for carrying out fine positioning of the nib by taking a pen holding hand as a reference, so that the positioning precision of the nib is greatly improved. Furthermore, as the pixels of the first image are fewer than those of the current frame image, the nib detection efficiency of the nib detection model is improved, and the real-time performance of nib positioning is improved. Still further, realize the location to the nib based on image acquisition and discernment, need not its extra auxiliary hardware, be favorable to the cost reduction.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the description of the embodiments and the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings.

FIG. 1A is a schematic view of a pen tip positioning method according to an embodiment of the present application;

FIG. 1B is a diagram of a current frame image disclosed in an embodiment of the present application;

FIG. 1C is a flow chart of a method for positioning a pen tip according to an embodiment of the present application;

FIG. 1D is a diagram of a first image disclosed in an embodiment of the present application;

FIG. 1E is another illustration of a first image disclosed in an embodiment of the present application;

FIG. 2 is a flow chart of another nib positioning method according to an embodiment of the present application;

FIG. 3 is a flow chart of another nib positioning method according to an embodiment of the present application;

FIG. 4 is a block diagram of a nib positioning apparatus according to an embodiment of the present application;

fig. 5 is a block diagram of a terminal device according to an embodiment of the present application.

Detailed Description

In order that those skilled in the art will better understand the present application, reference will now be made to the accompanying drawings in which embodiments of the application are illustrated, it being apparent that the embodiments described are only some, but not all, of the embodiments of the application. Based on the embodiments of the present application, it should be understood that the present application is within the scope of protection.

It will be appreciated that the terminal devices involved in embodiments of the present application may include general hand-held, on-screen electronic terminal devices such as cell phones, smart phones, portable terminals, personal digital assistants (Personal Digital Assistant, PDA), portable multimedia player (Personal Media Player, PMP) devices, notebook computers, notebook (Note Pad), wireless broadband (Wireless Broadband, wibro) terminals, tablet computers (Personal Computer, PC), smart PCs, point of sale (POS), and car computers, among others.

The terminal device may also comprise a wearable device. The wearable device may be worn directly on the user or be a portable electronic device integrated into the user's clothing or accessories. The wearable device is not only a hardware device, but also can realize powerful intelligent functions through software support, data interaction and cloud server interaction, such as: the mobile phone terminal has the advantages of calculating function, positioning function and alarming function, and can be connected with mobile phones and various terminals. Wearable devices may include, but are not limited to, wrist-supported watch types (e.g., watches, wrist products, etc.), foot-supported shoes (e.g., shoes, socks, or other leg wear products), head-supported Glass types (e.g., glasses, helmets, headbands, etc.), and smart apparel, school bags, crutches, accessories, etc. in various non-mainstream product forms.

It should be noted that, the execution main body of the nib positioning method disclosed in the embodiment of the present application may be a terminal device or a nib positioning device, which is not limited by the embodiment of the present application. The following embodiments mainly take terminal devices as examples.

Referring to fig. 1A, fig. 1A is a schematic view of a pen tip positioning method according to an embodiment of the application. The schematic view of the scene shown in fig. 1 may include a terminal device 10, a writing surface 20, a writing pen 30, and a pen-holding hand 40. Wherein, the terminal device 10 includes a camera 11, and in the process of controlling the writing pen 30 to write on the writing surface 20 by the pen holding hand 40, the terminal device 10 can control the front camera 11 to collect the writing picture in real time, so as to position the pen tip of the writing pen 30 on the writing surface 20 in real time based on the collected writing picture.

The following describes a method of positioning the pen tip of the writing pen 30, taking the current frame image as an example. Referring to fig. 1B, fig. 1B is a schematic diagram of a current frame image according to an embodiment of the present application. The current frame image shown in fig. 1B includes a writing pen 30, a writing surface 20, and a pen-holding hand 40. The terminal device 10 can roughly position the pen point contained in the current frame image by identifying the pen-holding hand 40 in the current frame image so as to obtain the rough position of the pen point, namely the area corresponding to the first image, and further finely position the pen point by taking the pen-holding hand as a reference through the pen point detection model on the basis of the rough position of the pen point, thereby improving the positioning precision of the pen point. Furthermore, as the pixels of the first image are fewer than those of the current frame image, the nib detection efficiency of the nib detection model is improved, and the real-time performance of nib positioning is improved. Still further, realize the location to the nib based on image acquisition and discernment, need not its extra auxiliary hardware, be favorable to the cost reduction.

Referring to fig. 1C, fig. 1C is a flow chart of a pen tip positioning method according to an embodiment of the application. The nib positioning method as shown in fig. 1C may include the steps of:

101. identifying a pen holder hand contained in the current frame image, and determining a first image from the current frame image; wherein the first image comprises a nib region and at least a partial region of a pen-holder hand.

In practice, it is found that when a hand holds a pen for writing, the relative positional relationship between the hand of the pen holder and the pen, or the relative positional relationship between the first knuckle of the finger of the pen holder and the pen is often relatively fixed. Based on this, the position of the pen tip in the current frame image can be assisted to be located based on the position of the pen-holding hand contained in the current frame image, or the position of the first knuckle of the pen-holding finger.

In some embodiments, identifying the pen-hold person's hand contained in the current frame image and determining the first image from the current frame image may include, but is not limited to, the following:

mode 1, a whole pen-holding hand included in a current frame image is identified, a first target area including a whole hand area and a pen tip area of the pen-holding hand is determined according to the position of the whole pen-holding hand in the current frame image and a first relative position relation between the pen-holding hand and a pen, and the first target area is intercepted from the current frame image to obtain a first image. The whole pen holder hand comprises all pen holding fingers, and the first relative position relation can be obtained by counting and analyzing a large number of relative position relations between the pen holder hand and the pen.

For a representation of the first image determined by way 1, please refer to fig. 1D. Fig. 1D includes the entire hand region 50 and the pen tip region 60.

Mode 2, a first knuckle of a pen-hold finger included in a current frame image is identified, a second target area including a corresponding area of the first knuckle of the pen-hold finger and a pen tip area is determined according to a position of the first knuckle of the pen-hold finger in the current frame image and a second relative positional relationship between the first knuckle of the pen-hold finger and a pen, and the second target area is cut from the current frame image to obtain a first image. Wherein the second relative positional relationship may be obtained by counting and analyzing the relative positional relationship of the first knuckle of the plurality of pen-holding fingers and the pen. The first knuckle of the pen-holding finger is the knuckle of the pen-holding finger closest to the pen tip.

For a representation of the first image determined by way 1, please refer to fig. 1E. Wherein fig. 1E includes a corresponding region 70 of the first knuckle and nib region 60.

102. And performing nib detection on the first image by using a nib detection model to obtain target nib coordinates corresponding to the current frame image.

The pen point detection model is trained by a plurality of first sample images and calibrated pen point coordinates corresponding to the first sample images, wherein the first sample images comprise a pen point area and at least part of the area of a pen holder hand. Note that, for the first image obtained by using a different manner of recognizing the pen-holding hand, since the content of the image included in the first image is different, the pen-tip detection model may also be trained using a different first sample image. If the first image is obtained in the above-described mode 1, the first sample image includes the nib region and the entire region of the pen-holding person's hand. If the first image is obtained in the above-described manner 2, the first sample image includes the nib region and the corresponding region of the first knuckle of the pen-holding finger. It can be seen that the two ways of determining the first image correspond to different nib detection models respectively.

In some embodiments, performing nib detection on the first image by using a nib detection model to obtain a target nib coordinate corresponding to the current frame image may include: and performing nib detection on the first image by using a nib detection model to obtain the coordinates of the nib on the first image, determining a coordinate transformation relation corresponding to the nib coordinates according to the position relation between the first image and the current frame image, and performing coordinate transformation on the coordinates of the nib on the first image according to the coordinate transformation relation to obtain the target nib coordinates corresponding to the current frame image. The target nib coordinate corresponding to the current frame image reflects the nib position in the current frame image.

In some embodiments, the pen tip detection model may be a lightweight regression model or a complex regression model, and embodiments of the present application are not limited.

Referring to fig. 2, fig. 2 is a flow chart of another nib positioning method according to an embodiment of the application. The nib positioning method as shown in fig. 2 may include the steps of:

201. identifying a pen holding finger contained in the current frame image through the pen area detection model, extracting the characteristic of a first knuckle in the pen holding finger, and determining a target pen area according to the characteristic of the first knuckle.

The first image may be taken from the current frame image, including the image of the nib region and the first knuckle of the pen-holding finger. The pen area detection model is trained by a plurality of second sample images and calibration pen areas corresponding to the second sample images, wherein the second sample images comprise pen point areas and the whole hand area of a pen holder hand, and the calibration pen areas are calibrated by taking a first knuckle of the pen holder finger in the second sample images as a reference, and can comprise the pen point areas and corresponding areas of the first knuckle of at least part of the pen holder finger.

It is understood that the target stylus region may include a nib region and a corresponding region of a first knuckle of at least a portion of the pen-holding finger.

202. And intercepting the target stylus region from the current frame image to obtain a first image.

203. And performing nib detection on the first image by using a nib detection model to obtain target nib coordinates corresponding to the current frame image.

The pen point detection model is obtained by training an image in a calibration pen area corresponding to the second sample image and corresponding calibration pen point coordinates. The nib detection model is nib-localized in combination with features of the first knuckle. Note that, in the current frame image, the number of pixels of the pen tip is usually relatively small, so that the accuracy of positioning the pen tip is often not high when the current frame image is directly returned. The pen point detection model can use the first knuckle as a reference to carry out regression on the pen point coordinate, and the positioning accuracy of the pen point can be effectively improved.

In some embodiments, where the nib detection model is a lightweight regression model, the nib detection model may include a lightweight backbone network and a regressor. Optionally, performing nib detection on the first image by using a nib detection model to obtain a target nib coordinate corresponding to the current frame image may include: extracting image features of a first image through a lightweight backbone network; and carrying out regression processing on the image characteristics through a regressive device so as to determine the target nib coordinates corresponding to the current frame image. By implementing the method, the coordinate of the nib is regressed by using the lightweight regression model, so that the calculation complexity can be simplified, and the positioning efficiency of the nib can be improved.

In some embodiments, after step 203, a moving distance value corresponding to the pen tip may be calculated according to the target pen tip coordinate corresponding to the current frame image and the target pen tip coordinate corresponding to the previous frame image; and if the moving distance value is larger than the moving distance threshold value, determining that the target nib coordinate corresponding to the current frame image is an invalid coordinate.

The moving distance value may refer to a moving distance between a point of time of acquisition of a previous frame image and a point of time of acquisition of a current frame image. The moving distance threshold value can be obtained by counting and analyzing moving distance values of the pen point between a plurality of adjacent two image acquisition time points.

In the embodiment of the application, if the moving distance value is smaller than or equal to the moving distance threshold value, determining the target nib coordinate corresponding to the current frame image as the effective coordinate.

In some embodiments, after determining that the target nib coordinate corresponding to the current frame image is an invalid coordinate, the target nib coordinates corresponding to the plurality of target frame images may also be obtained; the target frame image is an image acquired before the current frame image, and the coordinates of a target pen point corresponding to the target frame image are effective coordinates; and processing target nib coordinates corresponding to each target frame image by using a preset tracking algorithm to acquire a nib motion curve, and predicting to acquire effective target nib coordinates corresponding to the current frame image according to the motion curve.

In some embodiments, the preset tracking algorithm may include, but is not limited to, a Kalman filtering algorithm, a Camshift algorithm, and the like.

By implementing the method, the validity of the target nib coordinate corresponding to the current frame image obtained by the nib detection model can be detected, and under the condition that the target nib coordinate corresponding to the current frame image is an invalid coordinate, the target nib coordinate corresponding to the current frame image can be obtained by prediction according to the target nib coordinates corresponding to a plurality of images before the current frame image, so that the error degree of nib positioning can be reduced.

According to the method, firstly, the handwriting area in the current frame image is identified through the handwriting area detection model so as to realize coarse positioning of the nib, and then the nib coordinate is returned by taking the first knuckle as a reference on the basis of coarse positioning through the nib detection model, so that the positioning accuracy of the nib is improved.

Referring to fig. 3, fig. 3 is a flowchart illustrating a method for positioning a pen tip according to another embodiment of the application. The nib positioning method as shown in fig. 3 may include the steps of:

301. a first hand region of a pen-holding person's hand contained in the current frame image is identified.

In some embodiments, identifying the first hand region of the pen-holding human hand contained in the current frame image may include, but is not limited to, the following:

mode 1, a pen-hold hand included in a current frame image is recognized by using a pen-hold hand detection model, so as to obtain a first hand region of the pen-hold hand. The pen-holding hand detection model can be obtained by training the third sample images and hand areas of pen-holding hands contained in the third sample images.

Mode 2, acquiring a second hand region of a pen holder hand contained in a previous frame of image; and adjusting the position of the second hand region according to a preset displacement parameter to determine a first hand region of the pen holder hand contained in the current frame image.

In some embodiments, the preset displacement parameter may be obtained by counting and analyzing the displacement of the pen-holding hand between a plurality of adjacent two image acquisition time points. By implementing the method, the hand area of the pen holder hand contained in the current frame image is determined based on the hand area of the pen holder hand contained in the previous frame image, complex image recognition is not needed, and the real-time performance of pen point positioning is further improved.

302. The first hand region is enlarged according to a preset first size adjustment parameter to determine a first image region including the pen tip.

The first size adjustment parameter may include an adjustment parameter corresponding to a horizontal direction and/or an adjustment parameter corresponding to a vertical direction. The first sizing parameter may be obtained by statistically analyzing the relative relationship between a plurality of pen holders' hands and the pen tip.

303. And cutting out the first image area from the current frame image to obtain a first image.

304. And performing nib detection on the first image by using a nib detection model to obtain target nib coordinates corresponding to the current frame image.

Wherein the first sample image for training the pen tip detection model comprises a pen tip region and the whole hand region of the pen holder hand. The pen point detection model uses the pen holding hand as a reference to position the pen point, and compared with the method of directly returning the current frame image, the pen point detection model has higher precision of pen point positioning by using the pen holding hand as a reference.

According to the method, firstly, the pen-holding hand in the current frame image is identified to realize coarse positioning of the pen point, and then the whole pen-holding hand is used as a reference to carry out regression on the coordinates of the pen point on the basis of coarse positioning through the pen point detection model, so that the positioning accuracy of the pen point is improved.

Referring to fig. 4, fig. 4 is a block diagram illustrating a pen tip positioning device according to an embodiment of the application. The nib positioning apparatus as shown in fig. 4 may include: an image extraction unit 401 and a pen tip detection unit 402; wherein:

an image extraction unit 401 for identifying a pen-holding person's hand contained in the current frame image, and determining a first image from the current frame image; wherein the first image comprises a nib region and at least a partial region of a pen-holder hand;

a nib detection unit 402, configured to perform nib detection on the first image by using a nib detection model, so as to obtain a target nib coordinate corresponding to the current frame image; the pen point detection model is trained by a plurality of first sample images and calibrated pen point coordinates corresponding to the first sample images, wherein the first sample images comprise a pen point area and at least part of the area of a pen holder hand.

In some embodiments, the manner in which the image extraction unit 401 is configured to identify a pen-holding person included in the current frame image and determine the first image from the current frame image may specifically include: an image extracting unit 401, configured to identify a pen-holding finger included in the current frame image through a pen-holding region detection model, extract a feature of a first knuckle in the pen-holding finger, and determine a target pen-holding region according to the feature of the first knuckle; the pen region detection model is trained by a plurality of second sample images and calibrated pen regions corresponding to the second sample images, wherein the second sample images comprise a pen point region and the whole hand region of a pen holder hand; intercepting a target stylus region from a current frame image to obtain a first image; the first sample image is an image in a calibration pen area corresponding to the second sample image, and the first sample image comprises a pen point area and a corresponding area of a first knuckle of at least a part of a pen holding finger.

In some embodiments, the manner in which the image extraction unit 401 is configured to identify a pen-holding person included in the current frame image and determine the first image from the current frame image may specifically include: an image extraction unit 401 for identifying a first hand region of a pen-holding person hand included in the current frame image; expanding the first hand region according to a preset first size adjustment parameter to determine a first image region including the pen point; a first image area is intercepted from a current frame image to obtain a first image; wherein the first sample image includes a nib region and an entire hand region of the pen-holder hand.

In some embodiments, the manner in which the image extraction unit 401 is used to identify the first hand region of the pen-holding person's hand included in the current frame image may specifically include: an image extraction unit 401, configured to acquire a second hand region of the pen-holding hand included in the previous frame image; and adjusting the position of the second hand region according to a preset displacement parameter to determine a first hand region of the pen holder hand contained in the current frame image.

In some embodiments, the nib detection unit 402 is further configured to perform nib detection on the first image by using a nib detection model, so as to obtain a target nib coordinate corresponding to the current frame image, and calculate a movement distance value corresponding to the nib according to the target nib coordinate corresponding to the current frame image and the target nib coordinate corresponding to the previous frame image; and if the moving distance value is larger than the moving distance threshold value, determining that the target nib coordinate corresponding to the current frame image is an invalid coordinate.

In some embodiments, the pen tip detection unit 402 is further configured to obtain target pen tip coordinates corresponding to a plurality of target frame images after determining that the target pen tip coordinates corresponding to the current frame image are invalid coordinates; the target frame image is an image acquired before the current frame image, and the coordinates of a target pen point corresponding to the target frame image are effective coordinates; and processing target nib coordinates corresponding to each target frame image by using a preset tracking algorithm to acquire a nib motion curve, and predicting to acquire effective target nib coordinates corresponding to the current frame image according to the motion curve.

In some embodiments, the nib detection model includes a lightweight backbone network and a regressor.

Optionally, the manner in which the nib detection unit 402 is configured to perform nib detection on the first image by using the nib detection model to obtain the target nib coordinate corresponding to the current frame image may specifically include: a nib detection unit 402, configured to extract image features of the first image through a lightweight backbone network; and carrying out regression processing on the image characteristics through a regressive device so as to determine the target nib coordinates corresponding to the current frame image.

Referring to fig. 5, fig. 5 is a block diagram of a terminal device according to an embodiment of the present application. The terminal device as shown in fig. 5 may include: a processor 501, a memory 502 coupled to the processor 501, wherein the memory 502 may store one or more computer programs.

The processor 501 may include one or more processing cores. The processor 501 connects various parts within the overall terminal device using various interfaces and lines, performs various functions of the terminal device and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 502, and invoking data stored in the memory 502. Alternatively, the processor 501 may be implemented in hardware in at least one of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 501 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), and a modem, etc. The CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for being responsible for rendering and drawing of display content; the modem is used to handle wireless communications. It will be appreciated that the modem may not be integrated into the processor 501 and may be implemented solely by a single communication chip.

The Memory 502 may include a random access Memory (Random Access Memory, RAM) or a Read-Only Memory (ROM). Memory 502 may be used to store instructions, programs, code sets, or instruction sets. The memory 502 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, and the like. The storage data area may also store data created by the terminal device in use, etc.

In an embodiment of the present application, the processor 501 also has the following functions:

identifying a pen holder hand contained in the current frame image, and determining a first image from the current frame image; wherein the first image comprises a nib region and at least a partial region of a pen-holder hand;

performing nib detection on the first image by using a nib detection model to obtain a target nib coordinate corresponding to the current frame image; the pen point detection model is trained by a plurality of first sample images and calibrated pen point coordinates corresponding to the first sample images, wherein the first sample images comprise a pen point area and at least part of the area of a pen holder hand.

identifying a pen holding finger contained in the current frame image through a pen area detection model, extracting the characteristic of a first knuckle in the pen holding finger, and determining a target pen area according to the characteristic of the first knuckle; the pen region detection model is trained by a plurality of second sample images and calibrated pen regions corresponding to the second sample images, wherein the second sample images comprise a pen point region and the whole hand region of a pen holder hand;

intercepting a target stylus region from a current frame image to obtain a first image;

expanding the first hand region according to a preset first size adjustment parameter to determine a first image region including the pen point;

a first image area is intercepted from a current frame image to obtain a first image;

Wherein the first sample image includes a nib region and an entire hand region of the pen-holder hand.

calculating a moving distance value corresponding to the pen point according to the target pen point coordinate corresponding to the current frame image and the target pen point coordinate corresponding to the previous frame image;

and processing target nib coordinates corresponding to each target frame image by using a preset tracking algorithm to acquire a nib motion curve, and predicting to acquire effective target nib coordinates corresponding to the current frame image according to the motion curve.

extracting image features of a first image through a lightweight backbone network;

and carrying out regression processing on the image characteristics through a regressive device so as to determine the target nib coordinates corresponding to the current frame image.

The embodiment of the application discloses a computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, causes the processor to implement part or all of the steps executed by a terminal device in the above embodiment.

The embodiment of the application discloses a computer program product which, when run on a computer, causes the computer to execute part or all of the steps executed by the terminal device in the embodiment.

The embodiment of the application discloses an application release platform which is used for releasing a computer program product, wherein when the computer program product runs on a computer, the computer is enabled to execute part or all of the steps executed by terminal equipment in the embodiment.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable magnetic disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or the like, which can store program codes.

The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A nib positioning method, comprising:

2. The method of claim 1, wherein the identifying the pen-hold grip contained in the current frame image and determining the first image from the current frame image comprises:

3. The method of claim 1, wherein the identifying the pen-hold grip contained in the current frame image and determining the first image from the current frame image comprises:

4. A method according to claim 3, wherein said identifying a first hand region of a pen-holding person's hand contained in the current frame image comprises:

5. The method according to any one of claims 1-4, wherein after performing nib detection on the first image by using a nib detection model to obtain target nib coordinates corresponding to the current frame image, the method further comprises:

6. The method of claim 5, wherein after determining that the target pen point coordinates corresponding to the current frame image are invalid coordinates, the method further comprises:

7. The method according to any one of claims 1-4, wherein the nib detection model includes a lightweight backbone network and a regressor, and wherein the nib detection of the first image using the nib detection model to obtain target nib coordinates corresponding to the current frame image includes:

8. A pen point detection device is characterized by comprising

9. A terminal device, comprising

A memory storing executable program code;

and a processor coupled to the memory;

the processor invoking the executable program code stored in the memory, which when executed by the processor, causes the processor to implement the method of any of claims 1-7.

10. A computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, causes the processor to implement the method of any one of claims 1-7.