CN115100676A - Writing posture tracking method and device, electronic equipment and storage medium - Google Patents
Writing posture tracking method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115100676A CN115100676A CN202210592919.9A CN202210592919A CN115100676A CN 115100676 A CN115100676 A CN 115100676A CN 202210592919 A CN202210592919 A CN 202210592919A CN 115100676 A CN115100676 A CN 115100676A
- Authority
- CN
- China
- Prior art keywords
- human body
- target
- image
- writing
- visual field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 87
- 230000000007 visual effect Effects 0.000 claims abstract description 80
- 230000008569 process Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012549 training Methods 0.000 claims description 13
- 210000000746 body region Anatomy 0.000 claims description 9
- 238000006073 displacement reaction Methods 0.000 claims description 5
- 230000011218 segmentation Effects 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 4
- 230000036544 posture Effects 0.000 description 112
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 210000002310 elbow joint Anatomy 0.000 description 4
- 238000003062 neural network model Methods 0.000 description 4
- 210000000323 shoulder joint Anatomy 0.000 description 4
- 210000000707 wrist Anatomy 0.000 description 4
- 201000010099 disease Diseases 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 210000001503 joint Anatomy 0.000 description 3
- 208000001491 myopia Diseases 0.000 description 3
- 230000004379 myopia Effects 0.000 description 3
- 241000196324 Embryophyta Species 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 206010028836 Neck pain Diseases 0.000 description 1
- 241000190070 Sarracenia purpurea Species 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a writing posture tracking method, a writing posture tracking device, electronic equipment and a storage medium, wherein the writing posture tracking method comprises the following steps: acquiring a first image in a visual field range, and identifying key points of a human body on the first image; obtaining a target central point position of a target object corresponding to the human body key point based on the successful recognition result of the human body key point and a preset central position recognition model; and adjusting the position of a central point of the visual field to ensure that the adjusted position of the central point of the visual field is superposed with the position of the central point of the target so as to track the writing posture of the target object. The invention can realize the purpose of tracking the target object without depending on the manual operation of the writing posture tracking device, improves the practicability and flexibility of tracking the writing posture, can realize the purpose of accurately tracking the writing posture without manually moving the writing posture tracking device, and further improves the intelligence and reliability of tracking the human body writing posture by the writing posture tracking device.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a writing posture tracking method and device, electronic equipment and a storage medium.
Background
At present, the myopia rate and the incidence rate of waist and neck diseases of teenagers and children are higher and higher, the reason for the myopia rate and the incidence rate is not only because the functions of electronic products such as mobile phones and the like are increasingly enriched and diversified, but also the writing posture of the electronic products is related to the writing posture of the electronic products during the learning process, and if the writing posture is not standard for a long time, other common diseases such as glasses myopia, waist and neck pain and the like can be caused. Therefore, in order to prevent common diseases, it is important to accurately track the writing postures of teenagers and children.
In the related art, before or during writing by a young child using a desk at home or a desk at school, the young child puts the small book robot at a designated position where the whole writing posture can be photographed through the young child himself or a guardian thereof, thereby realizing tracking of the writing posture of the young child.
However, the existing writing gesture tracking method relies on that the robot of the small book child is manually placed to a specified position to track the writing gesture, so that the flexibility and the intelligence of the writing gesture tracking are not high.
Disclosure of Invention
The invention provides a writing posture tracking method, a writing posture tracking device, electronic equipment and a storage medium, which are used for overcoming the defects that the flexibility and the intelligence of the writing posture tracking are not high due to the fact that the writing posture can be tracked only after a small schoolchild robot is manually placed to a specified position for tracking the existing writing posture, and the purpose of flexibly and intelligently tracking the writing posture of a human body is achieved.
The invention provides a writing posture tracking method, which comprises the following steps:
acquiring a first image in a visual field range, and identifying key points of a human body on the first image;
obtaining a target central point position of a target object corresponding to the human key point based on the successful recognition result of the human key point and a preset central position recognition model;
and adjusting the position of a central point of the visual field to ensure that the adjusted position of the central point of the visual field is superposed with the position of the central point of the target so as to track the writing posture of the target object.
The invention also provides a writing tracking method, after the human key point identification is carried out on the first image, the method further comprises the following steps:
adjusting the visual field range based on the recognition failure result of the human body key points;
acquiring a second image based on the new visual field range obtained by the visual field range adjustment;
and identifying key points of the human body on the second image.
The invention also provides a writing tracking method, after the human key point identification is carried out on the first image, the method further comprises the following steps:
based on the recognition failure result of the key points of the human body, carrying out position movement according to a preset path direction;
acquiring a third image based on a new visual field range obtained by the position movement;
and identifying key points of the human body on the third image.
The present invention also provides a writing tracking method, wherein after the position is moved in the preset path direction, the method further comprises:
detecting whether an obstacle object exists in a preset distance or not in the position moving process;
when the obstacle object exists in the preset distance range, changing the direction of the preset path and continuing to move the position;
and when the obstacle object is determined not to exist in the preset distance range, continuing to move the position.
The present invention also provides a writing tracking method, after the position movement is continued, the method further comprising:
and stopping the position movement when determining that the new visual field range generated by the position movement reaches a preset position movement stopping condition.
The invention further provides a writing tracking method, wherein the obtaining of the target central point position of the target object corresponding to the human body key point based on the recognition success result of the human body key point and the preset central position recognition model comprises the following steps:
determining the positions of the identified human key points based on the successful identification result of the human key points;
constructing target contour data of a target object corresponding to the positions of the human body key points on the basis of the positions of the human body key points;
based on the target contour data, a target center point position of the target object is determined.
The invention also provides a writing tracking method, which further comprises the following steps:
when the writing posture of the target object is determined to be changed, acquiring a new target central point position after displacement;
and adjusting the position of the holder based on the new target central point position, so that the new view central point position obtained by the holder position adjustment coincides with the new target central point position.
The invention also provides a writing tracking method, and the training process of the preset central position recognition model comprises the following steps:
acquiring a sample image carrying an area label, wherein the area label comprises a human body area label, a desktop area label, a background area label and a book area label, and the human body area label comprises a human body key point label;
and carrying out training of human body region segmentation, contour data construction and center position determination by using the sample image to obtain a preset center position identification model.
The present invention also provides a writing posture tracking device, comprising:
the key point identification module is used for acquiring a first image in a visual field range and identifying the key points of a human body of the first image;
the position determining module is used for obtaining the position of a target central point of a target object corresponding to the human body key point based on the successful recognition result of the human body key point and a preset central position recognition model;
and the posture tracking module is used for adjusting the position of a central point of a visual field, so that the position of the central point of the visual field after adjustment is superposed with the position of the central point of the target, and the writing posture of the target object is tracked.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the writing gesture tracking method according to any one of the above methods.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a writing gesture tracking method as described in any of the above.
The present invention also provides a computer program product comprising a computer program which, when executed by a processor, implements a writing gesture tracking method as described in any one of the above.
According to the writing posture tracking method, the device, the electronic equipment and the storage medium, when the key points of the human body are identified and successfully identified in the first image acquired in the visual field range, the position of the target center point of the target object corresponding to the key points of the human body is determined through the preset center position identification model, and then the position of the center point of the visual field is adjusted, so that the aim of tracking the target object can be fulfilled without manually operating the writing posture tracking equipment, and the practicability and the flexibility of tracking the writing posture are improved. Furthermore, the purpose of adjusting the position of the center point of the self-vision of the writing posture tracking equipment is to coincide with the position of the target center point of the target object, so that the writing posture of the target object can be tracked subsequently, the purpose of accurately tracking the writing posture can be realized without manually moving the writing posture tracking equipment, and the reliability and the stability of tracking the human body writing posture by the writing posture tracking equipment are further improved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram of a writing gesture tracking method provided by the present invention;
FIG. 2 is a schematic diagram of the locations of key points in a human body provided by the present invention;
FIG. 3 is a schematic diagram of a writing gesture tracking device provided by the present invention;
fig. 4 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The writing gesture tracking method, the writing gesture tracking device, the electronic device and the storage medium according to the present invention are described below with reference to fig. 1 to 4, wherein an executing subject of the writing gesture tracking method may be a writing gesture tracking device having at least an image acquisition function, an image processing function, a position moving function, a data storage function and a view angle adjusting function. Illustratively, the writing gesture tracking device may be a small-sized book robot or an Artificial Intelligence (AI) small-sized book. Also, the method embodiments described below are described with the execution subject as the writing gesture tracking device as an example.
Fig. 1 is a schematic flow chart of a writing posture tracking method provided by the present invention, and as shown in fig. 1, the writing posture tracking method includes the following steps:
and 110, acquiring a first image in a visual field range, and identifying key points of a human body on the first image.
Wherein the first image may include a two-dimensional color image including RGB three channels and a depth image similar to a gray scale image. The human body key points may include, but are not limited to, head key points, hand key points, arm key points, and shoulder key points, the head key points may include, but are not limited to, a vertex, a left eye, a right eye, a left ear, a right ear, a left mouth corner, a right mouth corner, and a bridge of the nose; the hand key points can include, but are not limited to, the fingertips of 5 fingers of each hand and joints of phalanges of each finger; for example, the human body key points may be determined with reference to fig. 2 but not limited to fig. 2.
Specifically, when the writing posture tracking device is started based on the starting instruction, a preset camera can be used for collecting a first image in the visual field range of the writing posture tracking device, and the first image is subjected to human body key point identification. The starting instruction can be triggered and generated based on a virtual starting key on a terminal device connected with the writing gesture tracking device through manual touch, and can also be triggered and generated based on a physical starting key on the writing gesture tracking device through manual pressing. And is not particularly limited herein. In addition, the virtual start key can be manually touched or the physical start key can be manually pressed by the teenager or the parent or the teacher of the teenager or the teacher can manually touch the virtual start key or manually press the physical start key. And is not particularly limited herein.
And 120, obtaining a target central point position of the target object corresponding to the human body key point based on the successful recognition result of the human body key point and a preset central position recognition model.
The successful recognition result can represent that the first image contains all human key points required for determining the position of the target central point; the target object may be a target teenager or a target teenager with a writing gesture to be tracked, and the number of the target teenager or the target teenager may be 1 or more. And is not particularly limited herein.
Specifically, when the recognition result obtained by recognizing the first image by the writing posture tracking device is that the first image has the human body key point, the first image containing the human body key point may be input into the preset central position recognition model, and the target central point position of the target object corresponding to the human body key point, that is, the human body central point coordinate of the target teenager and child, is obtained.
Specifically, when the writing posture tracking device determines the target center point position of the target object, the center point position of the visual field of the writing posture tracking device can be adjusted by adjusting the holder of the writing posture tracking device, namely, the holder of the writing posture tracking device is adjusted until the center point position of the visual field of the camera in the writing posture tracking device after adjustment is superposed with the target center point position, namely, the coordinate of the center point of the visual field of the camera after adjustment is superposed with the coordinate of the human body center point of the target teenager, at the moment, the tracking of the target object from entering the visual field range of the camera to the human body posture in the process of sitting in front of a desk or a desk is realized, so that the writing posture of the target teenager can be tracked in real time in the process of writing before the target teenager sits in the desk or the desk.
According to the writing posture tracking method provided by the invention, when the key points of the human body are identified and successfully identified in the first image collected in the visual field range, the position of the target center point of the target object corresponding to the key points of the human body is determined through the preset center position identification model, and then the position of the center point of the visual field is adjusted, so that the aim of tracking the target object can be realized without depending on manually operating a writing posture tracking device, and the practicability and flexibility of tracking the writing posture are improved. Furthermore, the purpose of adjusting the position of the center point of the self-vision of the writing posture tracking equipment is to coincide with the position of the target center point of the target object so as to track the writing posture of the target object subsequently, so that the purpose of accurately tracking the writing posture can be realized without manually moving the writing posture tracking equipment, and the reliability and the stability of tracking the human body writing posture by the writing posture tracking equipment are further improved.
In the actual processing process, when an obstacle exists in the visual field range of the writing posture tracking device or only part of the human body key points of the target teenager child enter the visual field range of the writing posture tracking device although the obstacle does not exist in the visual field range, the target center point position of the target teenager child cannot be accurately positioned, namely, the human body key points are failed to be identified, and at the moment, the visual field range of the writing posture tracking device needs to be adjusted. Based on this, after step 110, the method may further include:
firstly, adjusting the visual field range based on the recognition failure result of the human body key points; thirdly, adjusting the obtained new visual field range based on the visual field range, and acquiring a second image; and then carrying out human body key point identification on the second image.
The identification failure result can represent that the first image does not contain the human key points and the first image contains partial human key points. For example, when a camera of the writing posture tracking device faces an obstacle such as a wall, a first image containing a key point of a human body cannot be acquired; for example, when there is no obstacle in front of the camera of the writing gesture tracking device but the target object does not completely enter the visual field of the camera, only the first image containing part of the key points of the human body can be acquired.
Specifically, when the recognition result obtained by the writing gesture tracking device recognizing the first image is that no human key point or a part of human key points exist in the first image, it may be considered that the recognition of the human key point with respect to the first image fails, and at this time, the writing gesture tracking device may adjust the visual field range of the writing gesture tracking device, for example, adjust the visual field angle of the writing gesture tracking device or track the moving position of the pan-tilt head of the writing gesture tracking device. And further acquiring a second image in the new visual field range according to the new visual field range obtained after the visual field range is adjusted, and performing human body key point identification on the second image so as to identify human body key points for tracking the human body writing posture from the second image.
According to the writing gesture tracking method provided by the invention, when an obstacle exists in front of the visual field of the writing gesture tracking equipment or only part of a target object enters the visual field range of the writing gesture tracking equipment, the image is re-acquired and key points of a human body are identified by adjusting the visual field range, so that the flexibility and the reliability of the writing gesture tracking equipment are improved.
It should be noted that when the camera of the writing posture tracking device is directly facing to an obstacle such as a small potted plant, the first image containing the key point of the human body cannot be completely collected, and at this time, the position of the writing posture tracking device needs to be moved and then the image is collected again. Based on this, after step 110, the method may further include:
firstly, based on the recognition failure result of the key points of the human body, carrying out position movement according to a preset path direction; thirdly, acquiring a third image based on a new visual field range obtained by the position movement; and then carrying out human body key point identification on the third image.
The preset path direction may be a path direction perpendicular to or parallel to a visual field direction of the writing gesture tracking apparatus.
Specifically, when the recognition result obtained by the writing posture tracking device recognizing the first image is that part of the human key points exist in the first image, the recognition of the human key points aiming at the first image can also be considered to be failed, at the moment, the writing posture tracking device can move in the position according to the preset path direction so as to cross obstacles similar to small potted plants and enable the target object to completely enter the visual field range of the target object, and then the third image is acquired based on the new visual field range obtained by the position movement so that the new visual field range of the camera avoids and completely acquires the human key points of the target object.
According to the writing posture tracking method provided by the invention, when the collected key points of the human body are incomplete due to the existence of obstacles in the visual field of the writing posture tracking equipment, the image is collected again and the key points of the human body are identified in a mode of moving the position of the writing posture tracking equipment, so that the application flexibility and the functional diversity of the writing posture tracking equipment are further improved.
Optionally, when the writing gesture tracking device moves according to the preset path direction, it is also ensured that other obstacles are avoided or the writing gesture tracking device does not move to the edge of the table in the moving process. Based on this, after the position moving in the preset path direction, the method further comprises:
firstly, detecting whether an obstacle object exists in a preset distance or not in the position moving process; thirdly, when the obstacle object exists in the preset distance range, changing the direction of the preset path and continuing to move the position; and finally, when the obstacle object is determined not to exist in the preset distance range, continuing to move the position.
The obstacle object can be an obstacle like a water cup, a pen container and the like, and can also be a desk or a desk edge of a desk. And the preset distance range can be a distance range in which a camera of the writing posture tracking device monitors the front of the path.
Specifically, in the process that the writing posture tracking device moves according to the preset path direction, the camera of the writing posture tracking device can also monitor whether an obstacle object exists in the preset distance range, if the obstacle object exists in the preset distance range, the current preset path direction needs to be changed, and the position movement is continued after the change, for example, the position movement is continued after the view angle is rotated by 90 degrees; on the contrary, if no obstacle object exists in the preset distance range, the position can be continuously moved according to the original preset path direction, so that the obstacle object can be avoided based on the new view range obtained by the position movement.
According to the writing posture tracking method provided by the invention, whether the writing posture tracking equipment continues to move after changing the direction or keeps the original direction to continue to move is judged by monitoring whether an obstacle object exists in front or not in the position moving process of the writing posture tracking equipment, so that the intelligence and the flexibility of the writing posture tracking equipment in position movement are improved.
Optionally, when the writing posture tracking device performs the position movement, even if there is no obstacle object in front or has avoided the obstacle object, the writing posture tracking device may not perform the position movement continuously, but may move to a certain position. Based on this, after the proceeding with the position movement, the method further includes:
and when determining that the new visual field range generated by the position movement reaches a preset position movement stop condition, stopping the position movement.
The preset position movement stop condition can represent that a camera of the writing posture tracking equipment is over against a target object (namely the target object is in a visual field range of the camera) and is not shielded.
Specifically, when the writing posture tracking device performs position movement, the camera thereof may also monitor whether a target object exists in the visual field range thereof to determine whether to stop the position movement. That is, when the target object is detected to exist in the visual field range in the position moving process, the position moving is stopped; on the contrary, when the target object is not monitored or only part of the target object is detected in the position moving process, the position moving can be continued until the preset position moving stop condition is reached on the premise of ensuring that no obstacle object exists in the preset range.
According to the writing posture tracking method provided by the invention, when the writing posture tracking device judges that the target object completely enters the visual field range of the writing posture tracking device in the position moving process, the position moving is stopped, so that the intelligence and the flexibility of the writing posture tracking device can be further improved.
Optionally, in step 120, based on the successful recognition result of the human body keypoint and the recognition model of the preset central position, a target central position of the target object corresponding to the human body keypoint is obtained, and the process may include:
firstly, determining the positions of the recognized human key points based on the recognition success results of the human key points; secondly, constructing target contour data of the target object corresponding to the positions of the human body key points on the basis of the positions of the human body key points; and determining the position of the target center point of the target object based on the target contour data.
The human body key point position is a three-dimensional coordinate position and can include but is not limited to a head key point position, a hand key point position, an arm key point position and a shoulder key point position, and the head key point position can include but is not limited to a vertex coordinate, a left eye coordinate, a right eye coordinate, a left ear coordinate, a right ear coordinate, a left mouth corner coordinate, a right mouth corner coordinate and a nose bridge bone coordinate; the positions of the key points of the hand can include, but are not limited to, the fingertip coordinates of 5 fingers of each hand and the coordinates of joints of phalanges of each section; the position of the arm key point can include but is not limited to a left wrist node coordinate, a right wrist node coordinate, a left arm elbow joint point coordinate and a right arm elbow joint point coordinate; the shoulder keypoint locations may include, but are not limited to, left shoulder joint point coordinates and right shoulder joint point coordinates.
Specifically, the successful recognition result can represent that the first image contains all human key points required for determining the center point position of the target human body, so that the first image can be input into a preset center position recognition model to extract the human body region and identify each human key point position aiming at the extracted human body region based on the successful recognition result, target contour data of the target object is constructed based on the identified human key point positions, the target contour data represents coordinate data of all human key points on the contour of the target object, the target center point position of the target object is further determined based on the target contour data, and the coordinates of the human center point of the target teenager and children are also calculated.
It should be noted that an algorithm for calculating the position of the central point based on the human body key point is stored in the preset central position recognition model in advance, and the algorithm can be obtained based on cloud big data and can also be obtained based on data training. And is not particularly limited herein.
According to the writing posture tracking method provided by the invention, when the first image acquired by the writing posture tracking device contains each key point for determining the position of the target center point, the reliability and stability of determining the coordinates of the human body center point of the target object are improved by using a preset center position recognition model to perform human body region extraction, human body key point position recognition and target center point position determination on the first image, and reliable guarantee is provided for accurately tracking the writing posture of the target object subsequently.
Optionally, when the writing posture tracking device determines that the position of the center point of the visual field coincides with the position of the target center point of the target object, the writing posture tracking device may track the writing posture of the target object. Based on this, after step 130, the method may include:
firstly, when the writing posture of the target object is determined to be changed, acquiring a new target central point position after displacement; and then, based on the new target central point position, adjusting the position of the holder, so that the new view central point position obtained by the holder position adjustment coincides with the new target central point position.
Wherein the writing posture comprises a sitting posture and a pen holding posture of the target object.
Specifically, when the writing posture tracking device determines that the position of the center point of the visual field of the writing posture tracking device coincides with the position of the target center point of the target object, the writing posture tracking of the target object in the writing scene can be performed. For example, when at least one target teenager child sits on a desk or desk for writing, the writing posture tracking device may collect an image in the visual field range in real time or periodically, identify key points of the human body according to the image, determine whether the key points of the human body identified this time are consistent with the key points of the human body identified the previous time, determine that the writing posture of the target object has changed if at least one of the key points of the human body identified this time is inconsistent with the key points of the human body identified the previous time, that is, the sitting posture and/or the pen holding posture of the target teenager child has changed, at this time, acquire a recognition model based on the key points of the human body identified this time and a preset center position, determine a new center point position of the target after the displacement, and adjust the position of the pan/tilt head based on the new center point position of the target, for example, the writing gesture tracking device holder can be adjusted up or down in height, and can also be rotated left or right. Therefore, the new view central point position obtained by the position adjustment of the holder and the new target central point position are always coincided.
According to the writing posture tracking method provided by the invention, when the writing posture tracking equipment tracks the writing posture of the target object under the condition that the visual field center position of the writing posture tracking equipment is superposed with the target center position of the target object and the writing posture of the target object is determined to be changed, the new visual field center position is always superposed with the new target center position in a manner of adjusting the position of the holder, so that the effective stability and the flexible reliability of tracking the writing posture are improved.
Optionally, the training process of the preset central position recognition model includes:
firstly, obtaining a sample image carrying an area label, wherein the area label comprises a human body area label, a desktop area label, a background area label and a book area label, and the human body area label comprises a human body key point label; and then, training of human body region segmentation, contour data construction and center position determination is carried out by using the sample image, and a preset center position identification model is obtained.
The sample image can be three-dimensional data obtained by sample objects in a writing scene, and the number of the sample objects can be 1 or more; when the number of sample objects is 1, the target object may be also included. Moreover, the number of sample images can be multiple and all carry area labels, the writing scene can be a scene of writing operation of sample objects sitting on a desk or in front of the desk, and the sample objects can be different teenagers and children when the number of the sample objects is multiple.
Specifically, when each sample image carries an area label, it can be considered that each sample image can identify at least 4 areas, and the 4 areas are a background area, a desktop area, a book area, and a human body area; when human body key point labels are contained in each human body region, the human body region of each sample image can be considered to identify the vertex, the left eye, the right eye, the left ear, the right ear, the left mouth corner, the right mouth corner, the bridge of the nose, the fingertips of 5 fingers of each hand, the joints of the phalanges, the left wrist node, the right wrist node, the left arm elbow joint point, the right arm elbow joint point, the left shoulder joint point and the right shoulder joint point. Moreover, each sample image carries a reference center point position of the corresponding sample object, and the reference center point position is also the human body center point coordinate of the corresponding sample object.
Based on the above, the neural network model is trained by using the sample images carrying the area labels and the human key point labels, so that the neural network model performs human area segmentation, human contour data extraction and center position determination on the human area in each sample image, the determined center point position is compared with the corresponding reference center point position, if the determined center point position is the same as the corresponding reference center point position, another sample image is changed for training until the number of times that the continuously determined center point positions are the same as the corresponding reference center point position reaches a preset number threshold, the training is stopped, and the corresponding neural network model when the training is stopped is determined as a preset center position identification model.
According to the writing posture tracking method, the preset central position recognition model for determining the position of the target central point is obtained in a mode of training the neural network model by using the sample image carrying the area label and the human body key point label, so that the aim of accurately recognizing the position of the target central point of the target object is fulfilled by combining the neural network learning method, and the accuracy of tracking the writing posture is improved.
The following describes the writing posture tracking device provided by the present invention, and the writing posture tracking device described below and the writing posture tracking method described above can be referred to in correspondence with each other.
Referring to fig. 3, a schematic structural diagram of a writing posture tracking device provided by the present invention is shown in fig. 3, the writing posture tracking device includes:
the key point identification module 310 is configured to acquire a first image in a visual field range, and perform human body key point identification on the first image; a position determining module 320, configured to obtain a target center point position of the target object corresponding to the human body key point based on a successful recognition result of the human body key point and a preset center position recognition model; the gesture tracking module 330 is configured to adjust a position of a center point of a field of view, so that the adjusted position of the center point of the field of view coincides with the position of the center point of the target, so as to track a writing gesture of the target object.
Optionally, the key point identification module 310 may be further configured to adjust the view range based on a result of failed identification of the human key point; acquiring a second image based on the new visual field range obtained by the visual field range adjustment; and identifying key points of the human body on the second image.
Optionally, the key point identifying module 310 may be further configured to perform position movement according to a preset path direction based on the identification failure result of the human body key point; acquiring a third image based on a new visual field range obtained by the position movement; and identifying key points of the human body on the third image.
Optionally, the apparatus may further include an adjusting module, which is specifically configured to detect whether an obstacle object exists within a preset distance in the process of moving the position; when the obstacle object exists in the preset distance range, changing the direction of the preset path and continuing to move the position; and when the obstacle object is determined not to exist in the preset distance range, continuing to move the position.
Optionally, the adjusting module may be further specifically configured to stop the position movement when it is determined that a new field of view generated by the position movement reaches a preset position movement stop condition.
Optionally, the position determining module 320 may be specifically configured to determine the positions of the identified human body key points based on the successful identification result of the human body key points; constructing target contour data of a target object corresponding to the positions of the human body key points on the basis of the positions of the human body key points; based on the target contour data, a target center point position of the target object is determined.
Optionally, the gesture tracking module 330 may be further configured to obtain a new target center point position after displacement when the writing gesture of the target object changes; and adjusting the position of the holder based on the new target central point position, so that the new view central point position obtained by the holder position adjustment coincides with the new target central point position.
Optionally, the device may further include a training module, which may be specifically configured to obtain a sample image carrying an area label, where the area label includes a human body area label, a desktop area label, a background area label, and a book area label, and the human body area label includes a human body key point label; and carrying out training of human body region segmentation, contour data construction and center position determination by using the sample image to obtain a preset center position identification model.
Fig. 4 illustrates a physical structure diagram of an electronic device, and as shown in fig. 4, the electronic device 400 may include: a processor (processor)410, a communication interface (communications interface)420, a memory (memory)430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 are in communication with each other via the communication bus 440. Processor 410 may invoke logic instructions in memory 430 to perform a writing gesture tracking method comprising:
acquiring a first image in a visual field range, and identifying key points of a human body on the first image;
obtaining a target central point position of a target object corresponding to the human body key point based on the successful recognition result of the human body key point and a preset central position recognition model;
and adjusting the position of a central point of the visual field to enable the adjusted position of the central point of the visual field to be coincident with the position of the central point of the target so as to track the writing posture of the target object.
Furthermore, the logic instructions in the memory 430 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer-readable storage medium, the computer program, when executed by a processor, being capable of executing the writing gesture tracking method provided by the above methods, the method comprising:
acquiring a first image in a visual field range, and identifying key points of a human body on the first image;
obtaining a target central point position of a target object corresponding to the human body key point based on the successful recognition result of the human body key point and a preset central position recognition model;
and adjusting the position of a central point of the visual field to enable the adjusted position of the central point of the visual field to be coincident with the position of the central point of the target so as to track the writing posture of the target object.
In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a writing gesture tracking method provided by the above methods, the method comprising:
acquiring a first image in a visual field range, and identifying key points of a human body on the first image;
obtaining a target central point position of a target object corresponding to the human body key point based on the successful recognition result of the human body key point and a preset central position recognition model;
and adjusting the position of a central point of the visual field to enable the adjusted position of the central point of the visual field to be coincident with the position of the central point of the target so as to track the writing posture of the target object.
The above-described embodiments of the apparatus are merely illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, i.e. may be located in one place, or may also be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on the understanding, the above technical solutions substantially or otherwise contributing to the prior art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the various embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (12)
1. A writing gesture tracking method, comprising:
acquiring a first image in a visual field range, and identifying key points of a human body on the first image;
obtaining a target central point position of a target object corresponding to the human body key point based on the successful recognition result of the human body key point and a preset central position recognition model;
and adjusting the position of a central point of the visual field to enable the adjusted position of the central point of the visual field to be coincident with the position of the central point of the target so as to track the writing posture of the target object.
2. The writing gesture tracking method of claim 1, wherein after the human keypoint recognition of the first image, the method further comprises:
adjusting the visual field range based on the recognition failure result of the human body key points;
acquiring a second image based on the new visual field range obtained by the visual field range adjustment;
and identifying key points of the human body on the second image.
3. The writing gesture tracking method of claim 1, wherein after the human keypoint recognition of the first image, the method further comprises:
based on the recognition failure result of the key points of the human body, carrying out position movement according to a preset path direction;
acquiring a third image based on a new visual field range obtained by the position movement;
and identifying key points of the human body on the third image.
4. The writing gesture tracking method of claim 3, wherein after the position movement in the preset path direction, the method further comprises:
detecting whether an obstacle object exists in a preset distance or not in the position moving process;
when the obstacle object exists in the preset distance range, changing the direction of the preset path and continuing to move the position;
and when the obstacle object is determined not to exist in the preset distance range, continuing to move the position.
5. The writing gesture tracking method of claim 4, wherein after the continuing the position movement, the method further comprises:
and when determining that the new visual field range generated by the position movement reaches a preset position movement stop condition, stopping the position movement.
6. The writing gesture tracking method according to claim 1, wherein obtaining the target central point position of the target object corresponding to the human body key point based on the successful recognition result of the human body key point and a preset central position recognition model comprises:
determining the positions of the identified human key points based on the successful identification result of the human key points;
constructing target contour data of a target object corresponding to the positions of the human body key points on the basis of the positions of the human body key points;
determining a target center point position of the target object based on the target contour data.
7. The writing gesture tracking method of claim 1, further comprising:
when the writing posture of the target object is determined to be changed, acquiring a new target central point position after displacement;
and adjusting the position of the holder based on the new target central point position, so that the new view central point position obtained by the holder position adjustment coincides with the new target central point position.
8. The writing gesture tracking method according to any one of claims 1 to 7, wherein the training process of the preset central position recognition model includes:
acquiring a sample image carrying an area label, wherein the area label comprises a human body area label, a desktop area label, a background area label and a book area label, and the human body area label comprises a human body key point label;
and carrying out training of human body region segmentation, contour data construction and center position determination by using the sample image to obtain a preset center position identification model.
9. A writing gesture tracking apparatus, comprising:
the key point identification module is used for acquiring a first image in a visual field range and identifying the key points of a human body of the first image;
the position determining module is used for obtaining the position of a target central point of a target object corresponding to the human body key point based on the successful recognition result of the human body key point and a preset central position recognition model;
and the gesture tracking module is used for adjusting the position of a central point of a visual field, so that the adjusted position of the central point of the visual field is superposed with the position of the central point of the target, and the writing gesture of the target object is tracked.
10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the writing gesture tracking method of any one of claims 1 to 8 when executing the program.
11. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the writing gesture tracking method of any one of claims 1 to 8.
12. A computer program product comprising a computer program, wherein the computer program when executed by a processor implements the writing gesture tracking method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210592919.9A CN115100676A (en) | 2022-05-27 | 2022-05-27 | Writing posture tracking method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210592919.9A CN115100676A (en) | 2022-05-27 | 2022-05-27 | Writing posture tracking method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115100676A true CN115100676A (en) | 2022-09-23 |
Family
ID=83288854
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210592919.9A Pending CN115100676A (en) | 2022-05-27 | 2022-05-27 | Writing posture tracking method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115100676A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110045364A (en) * | 2019-03-01 | 2019-07-23 | 上海大学 | Dynamic target tracking and static object detection system and method based on the identification of gradual fault image feature |
| WO2019179441A1 (en) * | 2018-03-21 | 2019-09-26 | 北京猎户星空科技有限公司 | Focus tracking method and device of smart apparatus, smart apparatus, and storage medium |
| CN110290351A (en) * | 2019-06-26 | 2019-09-27 | 广东康云科技有限公司 | A kind of video target tracking method, system, device and storage medium |
| CN110855891A (en) * | 2019-11-26 | 2020-02-28 | 爱菲力斯(深圳)科技有限公司 | Method and device for adjusting camera shooting angle based on human body posture and robot |
| CN112653844A (en) * | 2020-12-28 | 2021-04-13 | 珠海亿智电子科技有限公司 | Camera holder steering self-adaptive tracking adjustment method |
-
2022
- 2022-05-27 CN CN202210592919.9A patent/CN115100676A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019179441A1 (en) * | 2018-03-21 | 2019-09-26 | 北京猎户星空科技有限公司 | Focus tracking method and device of smart apparatus, smart apparatus, and storage medium |
| CN110045364A (en) * | 2019-03-01 | 2019-07-23 | 上海大学 | Dynamic target tracking and static object detection system and method based on the identification of gradual fault image feature |
| CN110290351A (en) * | 2019-06-26 | 2019-09-27 | 广东康云科技有限公司 | A kind of video target tracking method, system, device and storage medium |
| CN110855891A (en) * | 2019-11-26 | 2020-02-28 | 爱菲力斯(深圳)科技有限公司 | Method and device for adjusting camera shooting angle based on human body posture and robot |
| CN112653844A (en) * | 2020-12-28 | 2021-04-13 | 珠海亿智电子科技有限公司 | Camera holder steering self-adaptive tracking adjustment method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102209595B1 (en) | Detailed eye shape model for robust biometric applications | |
| CN106846403B (en) | Method and device for positioning hand in three-dimensional space and intelligent equipment | |
| US9047507B2 (en) | Upper-body skeleton extraction from depth maps | |
| WO2020125499A1 (en) | Operation prompting method and glasses | |
| Maisto et al. | An accurate algorithm for the identification of fingertips using an RGB-D camera | |
| CN107273869B (en) | Gesture recognition control method and electronic equipment | |
| Ghani et al. | GazePointer: A real time mouse pointer control implementation based on eye gaze tracking | |
| CN110148092A (en) | The analysis method of teenager's sitting posture based on machine vision and emotional state | |
| CN111367415B (en) | Equipment control method and device, computer equipment and medium | |
| WO2010142455A2 (en) | Method for determining the position of an object in an image, for determining an attitude of a persons face and method for controlling an input device based on the detection of attitude or eye gaze | |
| CN113191200A (en) | Push-up test counting method, device, equipment and medium | |
| CN114092985A (en) | Terminal control method, device, terminal and storage medium | |
| US11676357B2 (en) | Modification of projected structured light based on identified points within captured image | |
| EP2261772A1 (en) | Method for controlling an input device based on the detection of attitude or eye gaze | |
| CN115100676A (en) | Writing posture tracking method and device, electronic equipment and storage medium | |
| CN112149517A (en) | Face attendance checking method and system, computer equipment and storage medium | |
| EP2261857A1 (en) | Method for determining the position of an object in an image, for determining an attitude of a persons face and method for controlling an input device based on the detection of attitude or eye gaze | |
| CN112527103B (en) | Remote control method and device for display equipment, equipment and computer readable storage medium | |
| CN113093907B (en) | Man-machine interaction method, system, equipment and storage medium | |
| CN114723659A (en) | Acupuncture point detection effect determining method and device and electronic equipment | |
| CN110705605B (en) | Method, device, system and storage medium for establishing feature database and identifying actions | |
| JP5688514B2 (en) | Gaze measurement system, method and program | |
| Miyoshi et al. | Detection of Dangerous Behavior by Estimation of Head Pose and Moving Direction | |
| Kim et al. | Gaze tracking based on pupil estimation using multilayer perception | |
| Nam et al. | Mouth tracking for hands-free robot control systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |