CN106042005A - Bionic eye positioning tracking system and working method thereof - Google Patents
Bionic eye positioning tracking system and working method thereof Download PDFInfo
- Publication number
- CN106042005A CN106042005A CN201610379179.5A CN201610379179A CN106042005A CN 106042005 A CN106042005 A CN 106042005A CN 201610379179 A CN201610379179 A CN 201610379179A CN 106042005 A CN106042005 A CN 106042005A
- Authority
- CN
- China
- Prior art keywords
- cloud terrace
- module
- speaker
- sub
- coordinate system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
- B25J19/02—Sensing devices
- B25J19/04—Viewing devices
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J11/00—Manipulators not otherwise provided for
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
- B25J19/02—Sensing devices
- B25J19/026—Acoustical sensing devices
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Human Computer Interaction (AREA)
- Processing Or Creating Images (AREA)
- Image Processing (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention discloses a bionic eye positioning tracking system and a working method thereof. The system is mainly composed of six parts including a bionic eye pan-tilt module, a voice acquisition sensor array, a camera module, a pan-tilt movement control module, a controller module and an upper computer module. According to the bionic eye positioning tracking system and the working method of the bionic eye positioning tracking system, the three-dimensional space acoustic source localization technology and the visual processing algorithm are combined with the automatic control technology through the above system, the bionic eye positioning tracking system and the working method of the bionic eye positioning tracking system can be achieved, a bionic eye can face the center of the contour of the face of a speaker in real time, a bionic device can be more vivid, lively and friendly when communicating and interacting with people, and more interestingness is added.
Description
Technical field
The present invention relates to various bionic eye to the location tracking system of target and method of work, particularly relate to bionic eye
Binocular fixation is to the bionic eye location tracking system of face of speaking and method of work.
Background technology
At present, the various bio-robots (e.g., humanoid, the imitative animal shape etc.) sensory perceptual system to surrounding, mainly leads to
Cross bionic eye and bionic ear to obtain external information.
Wherein, in the design of bionic eye, mainly serve as bionic eye by two photographic head of installation and obtain video information,
To bionic eye overall structure founding mathematical models, some algorithms of application visual processes, it is achieved the three-dimensional information to target object
Calculate and recognition of face etc..Technology related to this comparative maturity, and it is widely applied to industry, military affairs, medical treatment etc.
Numerous areas.
But, the existing photographic head serving as bionic eye or be fixedly mounted on certain position, or can be by the behaviour of people
Control travel direction adjusts, and typically cannot realize the automatic to regarding and tracking of bionic eye and current speaker, lack lively and friendly
Communicative effect.
In the sound localization technology being similar to bionic ear, there is the sound signal collecting devices such as application mike and come real
The application of existing sound localization, this technology is techniques known, e.g., following document: Zheng Zhenzhen, Feng Huajun, Shen Changyu etc.
Three-dimensional acoustic localization algorithm [J] based on coordinate system transformation. journal of Zhejiang university, 2008,42 (2): 341-343 (under be called for short literary composition
Offer 1), the most just refer to a kind of sound localization technology based on sodar time difference (TDOA), and give a kind of based on coordinate
The Three-dimensional acoustic localization algorithm of system's conversion.
But, existing sound localization method is only to rely on the sound signal collecting devices such as mike to collect signal,
Information source is single, stability is low, error is big, when sound-source signal is disturbed by extraneous factor, this sound localization method without
Method realizes the precise positioning to speaker.
In exhibition or public occasion, under everybody surrounds and watches situation, how can be lively, accurately by the two of bionic eye photographic head
Ground watches speaker attentively on the face, and can automatically follow the movement of speaker and rotate eyes (two photographic head), allows and talks with people
Face can be in the picture centre that camera collection arrives in real time, there is no the research of this type of technology and method at present.
Summary of the invention
Cannot realize automatically watching attentively to solve existing bionic eye and follow the tracks of the technological deficiency of face of speaking, the present invention based on
Bionic eye and the prior art of bionic ear, it is provided that a kind of bionic eye location tracking system, to realize two shootings of bionic eye
Head is automatically watched attentively and traces into face of speaking.
Present invention simultaneously provides the method for work of this location tracking system.
For reaching above-mentioned purpose, the bionic eye location tracking system of the present invention is mainly made up of following six parts: bionic eye cloud
Platform module, sound collection sensor array, camera module, The Cloud Terrace motion-control module, controller module, upper computer module;
Wherein:
Described bionic eye The Cloud Terrace module includes: a The Cloud Terrace base, the sub-The Cloud Terrace of a carrying platform and two, left and right, its
In, it is connected, by carrying platform on The Cloud Terrace base by a horizontal rotation joint between The Cloud Terrace base and carrying platform
Horizontally rotate to imitate the neck rotation of bio-robot;Carrying platform is provided with two the sub-The Cloud Terraces in left and right, two, left and right
Sub-The Cloud Terrace is respectively arranged with a horizontal rotating frame and a pitching swivel mount respectively, so, every sub-The Cloud Terrace all have horizontal rotation and
Pitching rotates two cradle heads, and two photographic head are separately mounted on the pitching swivel mount in the middle of the sub-The Cloud Terrace in two, left and right, fill
When the eyes of bionic eye, by the respective horizontal rotation of sub-The Cloud Terrace of two, left and right and the rotation of elevation rotary joint, can distinguish
Adjust the photographic head being arranged on two the sub-The Cloud Terraces in left and right to point to, imitate the rotation of eyes, such bionic eye The Cloud Terrace module
A total of five freedoms of motion, the center position on carrying platform, bind a neck that can horizontally rotate with carrying platform
Portion's coordinate system.
Described sound collection sensor array serves as bionic ear in the present system, by mike identical for four characteristics,
Centered by bionic eye The Cloud Terrace base fulcrum, it is arranged on around The Cloud Terrace base center by the space geometry relation of positive tetrahedron, and
The Cloud Terrace base is propped up dot center's initial point as bionic eye The Cloud Terrace base coordinate system, so, by the acoustical signal collected and
Sound localization algorithm (sound localization method that the document 1 etc. that can refer to mention in background technology is mentioned), obtains speaker's mouth
D coordinates value relative to base coordinate system.
Described camera module is made up of two independent photographic head, is separately mounted in the middle of the sub-The Cloud Terrace in two, left and right
On pitching swivel mount, serve as the eyes of bionic eye in the present system, be used for obtaining video information, and video data passed to upper
Machine module.
Described The Cloud Terrace motion-control module includes: the horizontal rotation being separately mounted on two the sub-The Cloud Terraces in left and right drives electricity
The horizontal rotation of machine and pitching rotary drive motor, a carrying platform drives motor, data transmission module.The Cloud Terrace motor control
Module controls the driving motor of corresponding joint according to each driving electric machine rotation signal of the controller module output received
Rotate.
Described controller module, on the one hand gathers the output signal of sound collection sensor array, runs sound localization
Algorithm, it is thus achieved that speaker's mouth is relative to the D coordinates value of The Cloud Terrace base coordinate system, and by the three-dimensional seat of this speaker's mouth
Mark passes to upper computer module;On the other hand receive control instruction and data message that upper computer module sends, control each and drive
Dynamic electric machine rotation respective angles, the most also returns the kinestate of system to upper computer module.
Described upper computer module includes: the eye level corner α of bionic eye The Cloud Terrace1And β1Derivation program, to taking
About carrying platform horizontal rotation the derivation program of rotational angle theta, bionic eye The Cloud Terrace module, two sub-The Cloud Terrace joint spaces solves
Operation program, vision processing algorithm program and the signal procedure of controller.
The method of work of the present invention mainly comprises the steps of:
Step one, system initialization: system electrification, sound collection sensor array enters duty, upper computer module
The video image that display camera module obtains, controller module controls all driving reset motors to initial angle.
Step 2, sound localization: controller module reads the voice data that sound collection sensor array gathers, and runs
Sound localization algorithm obtains speaker's mouth D coordinates value relative to The Cloud Terrace base coordinate system, passes to upper by this coordinate figure
Machine module.
Step 3, eye level corner solve: i.e. to two sub-The Cloud Terrace each level angles about bionic eye The Cloud Terrace module
Solve.
Upper computer module relative to the D coordinates value of The Cloud Terrace base coordinate system, is converted to speaker according to speaker's mouth
Mouth is relative to the D coordinates value of cervical region coordinate system, when, under the non-rotation status of forward neck portion, solving two eyes and turn respectively
Level angle required when speaker's mouth;If wherein the level angle of any one exceedes ± and 30 degree (set point), this
Time need cervical region is carried out to rotate and adjust, then turn to step 4;If the feathering angle of solve out two eyes does not all surpass
Cross ± 30 degree (set point), at this moment need not carry out cervical region rotating adjust, then turn to step 5.
Step 4, neck joint is carried out rotate adjust: i.e. to carrying platform horizontal rotation corner solve.
Upper computer module first according to speaker's mouth relative to the D coordinates value of cervical region coordinate system, to meet carrying platform
Turning to honest is condition towards speaker's mouth, and the anti-feathering angle solved needed for cervical region, upper computer module is by the water of cervical region
The flat anglec of rotation issues controller module, controller module control the horizontal rotary drive motor of cervical region and rotate respective angles, it is achieved
Carrying platform turns to honest in the face of speaker's mouth direction, and so, cervical region coordinate system also turns to honest in the face of speaker mouth side
To, then turn to step 5, more relatively current cervical region coordinate system carries out the pass of two sub-The Cloud Terraces about bionic eye The Cloud Terrace module
Joint space derivation, so that two eyes turn to speaker's mouth direction respectively.
About step 5, bionic eye The Cloud Terrace module, two sub-The Cloud Terrace joint spaces solve: upper computer module is according to speaker
The relation of mouth relatively current cervical region coordinate system carries out two sub-The Cloud Terrace joint spaces about bionic eye The Cloud Terrace module and solves fortune
Calculate, the most counter solve the pitching rotary drive motor of two the sub-The Cloud Terraces in left and right, horizontal rotation drives the angle that should rotate of motor
Information, is then issued to controller module, by controller module control about two sub-The Cloud Terrace respective pitching rotary drive motors,
Horizontal rotation drives electric machine rotation respective angles, to realize the eyes of bionic eye tentatively to the mouth of speaker.
So far, complete the process of two photographic head (eyes) tentatively alignment speaker's mouth.
Step 6, eyes adjustment in direction: i.e. to two each rotary joint corners of sub-The Cloud Terrace about bionic eye The Cloud Terrace module
Correction.Upper computer module runs face recognition algorithms, obtains speaker face rectangular peripheral profile and asks for its centre bit
Put the two-dimensional coordinate value in video image coordinate system, according to the side-play amount of this two-dimensional coordinate value Yu video image center,
Obtain the pitching rotary drive motor of two the sub-The Cloud Terraces in left and right, the correction value of the horizontal rotation driving motor anglec of rotation respectively, so
After above-mentioned anglec of rotation correction value is issued controller module again, by controller module control about two sub-The Cloud Terraces respective
Pitching rotary drive motor, horizontal rotation drive motor fine setting respective angles, to realize to two eyes of bionic eye the most precisely
Face mask center position towards speaker so that speaker face rectangular profile center occurs in the video of photographic head identification
Picture centre.
If step 7 controller module detects when new speaker occurs, then circulation performs step 2, step 3, step
Rapid four, step 5, step 6, carry out real-time tracking to emerging face of speaking;If not having new speaker to occur, then circulate
Perform step 6 so that the corresponding degree of freedom of eyes adjusts according to the movement of face location of currently speaking.
Step 8, detection whole system the need of working on, the most then turn to step 7, it may be judged whether have new
Speaker occurs, if it is not, shutdown of then deactivating.
Below in conjunction with operation principle advantages of the present invention is described:
Two photographic head serving as bionic eye are arranged on a The Cloud Terrace with five degree of freedom by the present invention;By three-dimensional
Spatial sound source location technology, it is achieved the preliminary three-dimensional localization to speaker's mouth position;Solve according to The Cloud Terrace joint space
Operation method, obtains and the corresponding corner in each joint of rotary platform, bionic eye is stared at the direction of Primary Location;In conjunction with face
Recognition methods, finds out the accurate orientation at speaker's face mask center, and the corner of fine-adjustment tripod head associated joint, with by speaker
Face mask move to serve as two camera collections of bionic eye picture centre till;When speaker walks about, bionic eye
The change in location following this face of speaking is rotated adjustment automatically, to ensure that the face of speaker can be in photographic head in real time and adopt
The picture centre of collection;During until another person lifts up one's voice again, bionic eye just can be noted again according to aforesaid work process again
Depending on new face of speaking.
In a word: three dimensions sound localization technology, vision processing algorithm are combined by the present invention with automatic control technology, can
To realize a kind of bionic eye location tracking system and method for work so that bionic eye can be in real time towards speaker's face mask
Center, can allow bionic device when with people's exchange and interdynamic, it appears more lively, active and friendly, increase more interest.
Accompanying drawing explanation
Fig. 1 is bionic eye location tracking system overall relation block diagram;
Fig. 2 is bionic eye location tracking system population structure schematic diagram;
Fig. 3 is bionic eye The Cloud Terrace modular structure schematic diagram;
Fig. 4 is The Cloud Terrace motion-control module structural representation;
Spatial relationship schematic diagram between Fig. 5 speaker's face mask center and two camera image coordinate systems;
Each coordinate system relation schematic diagram when Fig. 6 is initial position;
Fig. 7 is bionic eye location tracking system workflow diagram.
Marginal data:
Fig. 1: 100-sound collection sensor array;200-camera module;300-bionic eye The Cloud Terrace module;400-The Cloud Terrace
Motion-control module;500-controller module;600-upper computer module;
Fig. 2: 100-sound collection sensor array;101-1 mike;102-2 mike;103-3 mike;
104-4 mike;200-camera module;The left photographic head of 201-;The right photographic head of 202-;300-bionic eye The Cloud Terrace module;
Fig. 3: 301-The Cloud Terrace base;302-carrying platform;The left sub-The Cloud Terrace of 303-;The right sub-The Cloud Terrace of 304-;The left sub-The Cloud Terrace of 305-
Horizontal rotating frame;The pitching swivel mount of the left sub-The Cloud Terrace of 306-;The horizontal rotating frame of the right sub-The Cloud Terrace of 307-;The right sub-The Cloud Terrace of 308-
Pitching swivel mount;
The horizontal rotation of Fig. 4: 401-carrying platform drives motor;The horizontal rotation of the left sub-The Cloud Terrace of 402-drives motor;
The pitching rotary drive motor of the left sub-The Cloud Terrace of 403-;The horizontal rotation of the right sub-The Cloud Terrace of 404-drives motor;The right sub-The Cloud Terrace of 405-
Pitching rotary drive motor.
Detailed description of the invention
Below in conjunction with the accompanying drawings, bionic eye location tracking system and the method for work of the present invention are described.
The bionic eye location tracking system of the present invention is mainly made up of following six part: sound collection sensor array
100, camera module 200, bionic eye The Cloud Terrace module 300, The Cloud Terrace motion-control module 400, controller module 500, host computer
Module 600;As it is shown in figure 1, wherein:
Described sound collection sensor array 100 serves as bionic ear in the present system, respectively by No. 1 mike 101,2
Number mike 103, No. 4 mikes 104 of 102, No. 3 mikes, as in figure 2 it is shown, in bionic eye The Cloud Terrace base 301 fulcrum being
The heart, is arranged on around The Cloud Terrace base center according to the space geometry relation of positive tetrahedron, and 301 dot center of The Cloud Terrace base is made
For bionic eye The Cloud Terrace base coordinate system O0Initial point, as shown in Figure 6, so, by the acoustical signal collected and sound localization
Algorithm, available speaker's mouth is relative to the D coordinates value of base coordinate system.
Described camera module 200 is made up of left photographic head 201 and right photographic head 202, as in figure 2 it is shown, by left shooting
201 are arranged on the pitching swivel mount 306 of left sub-The Cloud Terrace 303, and right photographic head 201 is arranged on the pitching of right sub-The Cloud Terrace 304
On swivel mount 308, serve as the eyes of bionic eye in the present system, be used for obtaining video information, and video data passed to upper
Machine module 600.
Described bionic eye The Cloud Terrace module 300 includes: 301, carrying platform 302 of a The Cloud Terrace base, left sub-The Cloud Terrace
303, right sub-The Cloud Terrace 304, as it is shown on figure 3, wherein, is closed by a horizontal rotation between The Cloud Terrace base 301 and carrying platform 302
Joint is connected, and is imitated the neck rotation of bio-robot by carrying platform horizontally rotating on The Cloud Terrace base 301;Carrying
Being provided with left sub-The Cloud Terrace 303 and right sub-The Cloud Terrace 304 on platform 302, left sub-The Cloud Terrace 303 has a horizontal rotating frame 305 and one
Pitching swivel mount 306, right sub-The Cloud Terrace 304 has a horizontal rotating frame 307 and a pitching swivel mount 308, so, every sub-cloud
Platform all has horizontal rotation and pitching rotates two cradle heads, and two photographic head are separately mounted in the middle of two the sub-The Cloud Terraces in left and right
Pitching swivel mount on, serve as the eyes of bionic eye, rotated by the respective horizontal rotation of sub-The Cloud Terrace of two, left and right and pitching and close
The rotation of joint, can adjust the photographic head being arranged on two the sub-The Cloud Terraces in left and right respectively and point to, imitate the rotation of eyes, so
A total of five freedoms of motion of bionic eye The Cloud Terrace module 300, and at the center position of carrying platform 302, bind one
Cervical region coordinate system O that can horizontally rotate with carrying platform 302 and rotate1, as shown in Figure 6.
Described The Cloud Terrace motion-control module 400 includes: the horizontal rotation being arranged on left sub-The Cloud Terrace 303 drives motor
402 and pitching rotary drive motor 403, the horizontal rotation that is arranged on right sub-The Cloud Terrace 304 drive motor 404 and pitching to rotate to drive
The horizontal rotation of 405, carrying platform 302 of galvanic electricity machine drives motor 401, data transmission module 406, as shown in Figure 4, The Cloud Terrace
Motion-control module 400 controls according to each driving electric machine rotation signal of controller module 500 output received accordingly
The driving motor in joint rotates.
Described controller module 500, on the one hand gathers the output signal of sound collection sensor array 100, operation sound
Source location algorithm, it is thus achieved that speaker's mouth is relative to The Cloud Terrace base coordinate system O0D coordinates value, and by the three of this speaker
Dimension coordinate passes to upper computer module 600, as shown in Figure 1;On the other hand the control instruction sum that upper computer module 600 sends is received
It is believed that breath, control each and drive electric machine rotation respective angles, also return the motion shape of system simultaneously to upper computer module 600
State.
Described upper computer module 600 includes: the eye level corner α of bionic eye The Cloud Terrace1And β1Derivation program,
Carrying platform 302 is horizontally rotated two sub-The Cloud Terrace joint spaces about the derivation program of rotational angle theta, bionic eye The Cloud Terrace module
Derivation program, vision processing algorithm program and the signal procedure of controller 500.
The present invention realizes bionic eye and mainly comprises the steps of the track and localization specific works method of face of speaking:
Step one, system initialization: system electrification, sound collection sensor array 100 enters duty, host computer mould
Block 600 shows the video image that camera module 200 obtains, and controller module 500 controls to be arranged on the water on left sub-The Cloud Terrace 303
Flat rotary drive motor 402 and pitching rotary drive motor 403, the horizontal rotation being arranged on right sub-The Cloud Terrace 304 drive motor
404 and 405, carrying platform 302 of pitching rotary drive motor on horizontal rotation drive motor 401 reset to initial angle
Degree.
Step 2, sound localization: controller module 500 reads the data that sound collection sensor array 100 gathers, and transports
Row sound localization algorithm obtains speaker's mouth relative to The Cloud Terrace base coordinate system O0D coordinates value, this coordinate figure is passed to
Upper computer module 600.
Step 3, eye level corner solve: i.e. turn about 300 two sub-The Cloud Terrace each levels of bionic eye The Cloud Terrace module
Angle α1And β1Solve.
Upper computer module 600 according to speaker's mouth relative to The Cloud Terrace base coordinate system O0D coordinates value, be converted to
Speaker's mouth is relative to cervical region coordinate system O1D coordinates value, under the current non-rotation status of carrying platform 302, solve
When left photographic head 201 and right photographic head 202 turn to speaker's mouth respectively, the horizontal rotation of left sub-The Cloud Terrace 303 drives the corner of motor
Degree α1With the gyration β that the horizontal rotation of right sub-The Cloud Terrace 304 drives motor1If wherein the level angle of any one exceedes ± 30 °
(set point), at this moment needs to carry out carrying platform 302 cervical region rotating to adjust, then turns to step 4;If the left side solving out
The horizontal rotation of sub-The Cloud Terrace 303 drives the gyration α of motor1With the gyration β that the horizontal rotation of right sub-The Cloud Terrace 304 drives motor1The most not
Exceed ± 30 ° (set point), at this moment need not carry out carrying platform 302 rotating adjust, then turn to step 5.
Step 4, neck joint is carried out rotate adjust: i.e. carrying platform 302 is horizontally rotated rotational angle theta and solves.
Upper computer module 600 first according to speaker's mouth relative to cervical region coordinate system O1D coordinates value, with meet take
Carrying platform 302 turn to honest towards speaker's mouth be condition, the anti-feathering angle θ solved needed for carrying platform 302 cervical region.
Feathering angle θ needed for carrying platform 302 cervical region is issued controller module 500, by controller module by upper computer module 600
500 horizontal rotations controlling carrying platform 302 drive motor 401 to rotate respective angles θ, it is achieved carrying platform 302 turns to honest
In the face of speaker's mouth direction, so, cervical region coordinate system O1X-axis also turn to honest in the face of speaker's mouth direction, then turn
To step 5, more relatively current cervical region coordinate system O1Carry out the joint space of two sub-The Cloud Terraces about bionic eye The Cloud Terrace module to ask
Solve computing, so that left photographic head 201 and right photographic head 202 turn to speaker's mouth direction respectively
Step 5, about 300 two sub-The Cloud Terrace joint spaces of bionic eye The Cloud Terrace module solve: upper computer module 600 basis
Speaker's mouth relatively current cervical region coordinate system O1Relation carry out the joint of two sub-The Cloud Terraces about bionic eye The Cloud Terrace module
Space derivation, the most anti-horizontal rotation solving left sub-The Cloud Terrace 303 drives motor 402 and pitching rotary drive motor 403
Anglec of rotation α1、α2, the horizontal rotation of right sub-The Cloud Terrace 304 drives motor 404 and the anglec of rotation of pitching rotary drive motor 405
Degree β1、β2, being then issued to controller module 500, controller module 500 horizontal rotation controlling left sub-The Cloud Terrace 303 drives motor
402 and the horizontal rotation of pitching rotary drive motor 403, right sub-The Cloud Terrace 304 drive motor 404 and pitching rotary drive motor
405 rotate respective angles, preliminary to the mouth of speaker with the left photographic head 201 and right photographic head 202 realizing bionic eye.
So far, complete the process of two photographic head (eyes) tentatively alignment speaker's mouth.
Step 6, eyes adjustment in direction: i.e. to two each rotary joint corners of sub-The Cloud Terrace about bionic eye The Cloud Terrace module
Correction.
Upper computer module 600 runs face recognition algorithms, obtains speaker face rectangular peripheral profile and asks for wherein
Heart position is in video image coordinate system O of left photographic head 201l-XlYlVideo image coordinate system O with right photographic head 202r-XrYr
In two-dimensional coordinate value (Xl, Yl) and (Xr, Yr), as it is shown in figure 5, set picture traverse as W, height is H, according to this two-dimensional coordinate
Value and the side-play amount of video image center (W/2, H/2), obtain the horizontal rotation driving motor of left sub-The Cloud Terrace 303 respectively
402 anglecs of rotation α1Correction value Δ α1With pitching rotary drive motor 403 anglec of rotation α2Correction value Δ α2, and right son
The horizontal rotation of The Cloud Terrace 304 drives motor 404 anglec of rotation β1Correction value Δ β1With pitching rotary drive motor 405 anglec of rotation
Degree β2Correction value Δ β2, then above-mentioned anglec of rotation correction value is issued controller module 500 again, by controller module 500
The horizontal rotation controlling left sub-The Cloud Terrace 303 drives motor 402 and pitching rotary drive motor 403, the level rotation of right sub-The Cloud Terrace 304
Turn driving motor 404 and pitching rotary drive motor 405 finely tunes respective angles, to realize left photographic head 201 and the right side of bionic eye
Photographic head 201 is the most precisely towards speaker face rectangular profile center position so that in speaker face rectangular peripheral profile
The heart occurs in the video image center of photographic head identification.
If step 7 controller module 500 detects when new speaker occurs, then circulation performs step 2, step
Three, step 4, step 5, step 6, carries out real-time tracking to emerging face of speaking;If not having new speaker to occur,
Then circulation performs step 6 so that the corresponding degree of freedom of eyes adjusts according to the movement of face location of currently speaking.
Step 8, detection whole system the need of working on, the most then turn to step 7, it may be judged whether have new
Speaker occurs, if it is not, shutdown of then deactivating.
In method of work of the present invention: specific algorithm that the eye level corner mentioned in step 3 solves, step 4 carry
To cervical region is carried out rotate the computational methods adjusted, two sub-The Cloud Terraces close about the bionic eye The Cloud Terrace module mentioned in step 5
Joint space method for solving is technology commonly used in the art.Understand for the ease of auditor, describe in detail separately below.
The present invention describes involved all angular units and is all taken as: degree, all length unit is all taken as: millimeter (mm).
The specific algorithm that the eye level corner mentioned in above-mentioned steps three solves comprises the steps of:
Step 3.1, as shown in Figure 6, sets up fixing The Cloud Terrace base coordinate system in bionic eye The Cloud Terrace module 300 respectively
O0-X0Y0Z0, cervical region coordinate system O that horizontally rotates1-X1Y1Z1, left sub-The Cloud Terrace 303 base coordinate system O2-X2Y2Z2, left sub-The Cloud Terrace
303 horizontal rotation coordinate systems O3-X3Y3Z3, 303 left sub-The Cloud Terrace pitching rotating coordinate system O4-X4Y4Z4, right sub-The Cloud Terrace 304 basis is sat
Mark system O5-X5Y5Z5Coordinate system, right sub-The Cloud Terrace 304 horizontally rotates coordinate system O6-X6Y6Z6, right sub-The Cloud Terrace 304 pitching rotating coordinate system
O7-X7Y7Z7, it is briefly referred to as O0、O1、O2、O4、O5、O6、O7Coordinate system, wherein:
O0Coordinate system build 301 dot center of bionic eye The Cloud Terrace base in, obtains speaker's mouth through sound localization algorithm
D coordinates value is the statement under this coordinate system, for the base coordinate system of whole bionic eye The Cloud Terrace;
O1Coordinate system build carrying platform 302 center above in, bind with carrying platform 302 together with, be equivalent to cervical region
Coordinate system, O1Coordinate system follows horizontally rotating and rotating of carrying platform 302, and its corner θ represents;
O2In the bottom center plane of the horizontal rotation joints axes that coordinate system build left sub-The Cloud Terrace 303 in, with carrying platform
302 bindings together, for left sub-The Cloud Terrace 303 base coordinate system;
O3Its initial point of coordinate system just build the surface of the horizontal rotation joints axes of left sub-The Cloud Terrace 303, and and pitching in
Joints axes is contour, O3Coordinate system is bundled on left sub-The Cloud Terrace 303 horizontal rotating frame 305 rightly, along with left sub-The Cloud Terrace 303
The horizontally rotating and rotate of horizontal rotating frame 305, its corner α1Represent;
O4Coordinate origin and O3Coordinate origin overlaps, O4Coordinate system is bundled in left sub-The Cloud Terrace 303 pitching rightly and rotates
On frame 306, rotate along with the pitch rotation of left sub-The Cloud Terrace 303 pitching swivel mount 306, because O4The X of coordinate system4Axle positive direction
Just with the optical axis coincidence of the left photographic head 201 being fixed on pitching swivel mount 306, as it is shown in figure 5, therefore, O4Coordinate system is also referred to as
For left photographic head 201 coordinate system, left photographic head 201 will be with left sub-The Cloud Terrace 303 horizontal rotating frame 305 and pitching swivel mount
The rotation of 306 and rotate, O4Coordinate system is with the corner α of left sub-The Cloud Terrace 303 pitching swivel mount 306 pitching2Represent;
O5In the bottom center plane of the horizontal rotation joints axes that coordinate system build right sub-The Cloud Terrace 304 in, with carrying platform
302 bindings together, for right sub-The Cloud Terrace 304 base coordinate system;
O6Its initial point of coordinate system just build the surface of the horizontal rotation joints axes of right sub-The Cloud Terrace 304, and and pitching in
Joints axes is contour, O6Coordinate system is bundled on right sub-The Cloud Terrace horizontal rotating frame 307 rightly, along with right sub-The Cloud Terrace 304 level
The horizontally rotating and rotate of swivel mount 307, its corner β1Represent;
O7Coordinate origin and O6Coordinate origin overlaps, O7Coordinate system is bundled in right sub-The Cloud Terrace 304 pitching rightly and rotates
On frame 308, rotate along with the pitch rotation of right sub-The Cloud Terrace 304 pitching swivel mount 308, because O7The X of coordinate system7Axle positive direction
Just with the optical axis coincidence of the right photographic head 201 being fixed on pitching swivel mount 308, therefore, O7Coordinate system is also referred to as right photographic head
Coordinate system, right shooting 201 will be with right sub-The Cloud Terrace 304 horizontal rotating frame 307 and the rotation of pitching swivel mount 308 and turns
Dynamic, O7Coordinate system is with the corner β of right sub-The Cloud Terrace 304 pitching swivel mount 308 pitching2Represent;
Step 3.2, spatial correlation describe:
As shown in Figure 6, by cervical region coordinate system O1Relative to The Cloud Terrace base coordinate system O0Translation and horizontal rotation relation square
Battle array A0Describe, by O2Coordinate system is relative to O1The translation relation matrix A of coordinate system1Describe, by O3Coordinate system is relative to O2Coordinate
The translation of system and horizontal rotation relation matrix A2Describe, by O4Coordinate system is relative to O3The pitching rotation relationship square of coordinate system
Battle array A3Describe, by O5Coordinate system is relative to O1The translation relation matrix A of coordinate system4Describe, by O6Coordinate system is relative to O5Coordinate
The translation of system and horizontal rotation relation matrix A5Describe, by O7Coordinate system is relative to O6The pitching rotation relationship square of coordinate system
Battle array A6Describe, by speaker's mouth relative to The Cloud Terrace base coordinate system O0Vector correlation with matrix P describe, by speaker's mouth
Relative to left photographic head coordinate system O4Vector correlation matrix PLDescribe, by the mouth of speaker relative to right photographic head coordinate
It is O7Vector correlation matrix PRDescribe;
Spatial relation description according to above-mentioned coordinate system can be expressed as following series matrix:
PL=[XL YL ZL 1]T (9)PR=[XR YR ZR 1]T (10)
In matrix: Pz1For O1Coordinate origin is relative to O0The height of coordinate origin, for known;PyFor cervical region coordinate system
O1Distance between initial point and two, left and right sub-The Cloud Terrace base coordinate system initial point, for known;Pz2For O3Coordinate origin relative to
Coordinate origin and O6Coordinate origin is relative to O5The height of coordinate origin, for known;
θ is O1Coordinate system is relative to O0The horizontal rotation angle of coordinate system, is also the horizontal rotary drive motor of cervical region 401
Gyration, seen from above, is just counterclockwise, is negative clockwise, can read at any time, for known quantity;
α1For O3Coordinate system is relative to O2The horizontal rotation angle of coordinate system, is also that the horizontal rotation of left sub-The Cloud Terrace drives motor
The gyration of 402, this corner serves as horizontally rotating of left eye, seen from above, is just counterclockwise, is negative clockwise, is demand solution
Amount;
α2For O4Coordinate system is relative to O3The pitching anglec of rotation of coordinate system, is also left sub-The Cloud Terrace pitching rotary drive motor
The gyration of 403, this corner serves as the pitch rotation of left eye, and from the side, nutation is just, faces upward as negative, is demand solution amount;
β1For O6Coordinate system is relative to O5The horizontal rotation angle of coordinate system, is also that the horizontal rotation of right sub-The Cloud Terrace drives motor
The gyration of 404, this corner serves as horizontally rotating of right eye, seen from above, is just counterclockwise, is negative clockwise, is demand solution
Amount;
β2For O7Coordinate system is relative to O6The pitching anglec of rotation of coordinate system, is also right sub-The Cloud Terrace pitching rotary drive motor
The gyration of 405, this corner serves as the pitch rotation of right eye, and from the side, nutation is just, faces upward as negative, is demand solution amount;
X, Y, Z are the D coordinates value of speaker's mouth, and via controller module 500 is run sound localization algorithm and obtained in real time
, as known quantity when calculating herein;
When the sub-The Cloud Terrace in two, left and right is towards dead ahead, the feathering angle of two the sub-The Cloud Terraces in left and right is respectively as follows: α1=0 °,
β1=0 °;
When left photographic head 201 is just to acoustic target, must there is YL=0, ZL=0, XLIt it is demand solution amount;
When right photographic head 202 is just to acoustic target, must there is YR=0, ZR=0, XRIt it is demand solution amount;
Step 3.3, in the case of cervical region does not rotates, carry out bionic eye The Cloud Terrace left sub-The Cloud Terrace 303 horizontal rotation drive electricity
The gyration α of machine1With the gyration β that the horizontal rotation of right sub-The Cloud Terrace 304 drives motor1Derivation:
According to the spatial relationship between each coordinate system above-mentioned, can be by speaker's mouth relative to The Cloud Terrace base coordinate system
The vector of D coordinates value describe P and left photographic head 201, right photographic head 202 coordinate system X-axis honest towards speaker's mouth time
Vector P is describedL、PRSet up following relational expression:
Now, the horizontal rotation angle α needed for two the sub-The Cloud Terraces in left and right need to only be calculated1With β1, by aforementioned each formula and
Know that condition can be derived acquisition:
So, in the case of neck joint does not rotates (θ=0 °), solve and obtained eyes and see to speaker's mouth position
Time, the horizontal rotation angle α needed for the sub-The Cloud Terrace in two, left and right1With β1, i.e. obtained the horizontal rotation of left sub-The Cloud Terrace 303 and driven motor
Gyration α1With the gyration β that the horizontal rotation of right sub-The Cloud Terrace 304 drives motor1。
The computational methods that cervical region carries out rotating adjustment mentioned in above-mentioned steps four are:
If neck joint being rotated after adjusting, cervical region coordinate system O1Should be honest towards speaker's mouth, if after Tiao Zhenging
The mouth of speaker is relative to cervical region coordinate system O1Vector correlation matrix P1Describe, if O after Tiao Zhenging1Cervical region coordinate system relative to
The Cloud Terrace base coordinate system O0Translation and horizontal rotation relation B describe, then they can be write as lower column matrix:
P1=[X1 Y1 Z1 1]T (13)In matrix:
(X1, Y1, Z1) it is that speaker's mouth is in cervical region coordinate system O1In coordinate statement, when cervical region coordinate system O1X1Axle is honest towards saying
Y during words people's mouth1=0, it is definite value, X1、Z1It it is demand solution amount;θ is the anglec of rotation of the horizontal rotary drive motor of cervical region 401, from
Top is seen, is just counterclockwise, is negative clockwise, is demand solution amount;Pz1For coordinate system relative to O0The height of coordinate system, can measure
Arrive, for known quantity;
According to the spatial relationship between each coordinate system above-mentioned, can be by speaker's mouth relative to The Cloud Terrace base coordinate system O0
The vector of D coordinates value describe P with relative to cervical region coordinate system O after adjusting1Vector P is described1Set up following relational expression:
P=B P1 (15)
(15) formula that (8) formula and known conditions substituted into can be derived acquisition:
Above-mentioned required bionic eye neck joint horizontal rotation angle θ, is, if making carrying platform 302 just to speaking
The mouth direction of people, carrying platform 302 is required obtains the anglec of rotation, is also the rotation needed for the horizontal rotary drive motor of cervical region 401
Angle, θ.
Mentioning in above-mentioned steps five, about bionic eye The Cloud Terrace module, two sub-The Cloud Terrace joint space method for solving are:
Now, neck joint adjusts, according to the relationship description formula between each coordinate system of aforementioned foundation without making to rotate
(1), when two, left and right photographic head is all directed at the mouth position of same speaker, should there be formula (11)-(10) equally.
Can be derived acquisition by aforementioned each formula and known conditions:
Above-mentioned anglec of rotation α solving the left and right corresponding degree of freedom of sub-The Cloud Terrace1、α2、β1、β2Process, be and anti-solve left sub-cloud
The horizontal rotation of platform 303 drives motor 402 anglec of rotation α1With pitching rotary drive motor 403 anglec of rotation α2, right sub-The Cloud Terrace
The horizontal rotation of 304 drives anglec of rotation β of motor 4041Anglec of rotation β with pitching rotary drive motor 4052Calculating
Journey, wherein, XLIt is that the photocentre of left photographic head 201 is to the distance between speaker's mouth, XRIt is that right photographic head photocentre is to mouth of speaking
Distance between portion.
The computational methods step of the eyes adjustment in direction value that step 6 of the present invention is carried is as follows: following all angular units are equal
It is taken as: degree (°), all Parameter units relevant to video image are all taken as: pixel;
Step 6.1, upper computer module 600 read and show what the left photographic head 201 of bionic eye and right photographic head 202 obtained
Video image, if picture traverse is W, height is H;
Step 6.2, upper computer module 600 run face recognition algorithms, know at left photographic head 201, right photographic head 202 respectively
In other video image, calculate the face rectangular peripheral profile center video image coordinate at left photographic head 201 of speaking
It is Ol-XlYlVideo image coordinate system O with right photographic head 201r-XrYrIn two-dimensional coordinate value: (Xl, Yl)、(Xr, Yr), such as figure
Shown in 5.
Step 6.3, take video image coordinate system O of left photographic head 201l-XlYlSit with the video image of right photographic head 201
Mark system Or-XrYrCenter position coordinates value be: (W/2, H/2), then left photographic head 201 identify video image in, speak
Two-dimensional coordinate value (the X of face centerl, Yl) and two-dimensional coordinate value (W/2, the H/2) side-play amount of video image center
For:
In the video image that right photographic head 201 identifies, the two-dimensional coordinate value (X of speaker face centerr, Yr) with
Two-dimensional coordinate value (W/2, the H/2) side-play amount of video image center is:
Step 6.4, ask for eyes corresponding freedom degree rotating angle correction according to side-play amount:
Left eye horizontal rotation freedom degree rotating angle correction value: Δ α1=arctan (Δ Xl/length); (20)
Left eye pitching rotary freedom anglec of rotation correction value: Δ α2=arctan (Δ Yl/length); (21)
Right eye horizontal rotation freedom degree rotating angle correction value: Δ β1=arctan (Δ Xr/length); (22)
Right eye pitching rotary freedom anglec of rotation correction value: Δ β2=arctan (Δ Yr/length); (23)
Wherein, length is the focal length parameter of photographic head, sets according to the actual parameter of photographic head;Anglec of rotation correction value
Δα1、Δα2、Δβ1、Δβ2In all comprise positive and negative number two kinds of situations.
After respective drive motor rotates above-mentioned correction angle, speaker face rectangular peripheral profile can be adjusted in image
Heart position.
Two embodiments are set forth below and further illustrate present invention application in practice
Embodiment:
Embodiment 1: bionic eye cradle head structure design parameter is: Pz1=150mm, Pz2=120mm, Py1=120mm, it is assumed that
Run sound localization algorithm by controller module 500 and obtain speaker's mouth relative to The Cloud Terrace base coordinate system O0Three-dimensional coordinate
Value P=[6,000 2,000 3,000 1]T;First, obtain according to formula (12) formula and known conditions in step 3 of the present invention:
α1=17.398 °, β1=19.461 °
Two angle values, all within ± 30 °, at this moment need not carry out carrying platform 302 cervical region rotating adjust;
Then, according to step 5 of the present invention, known conditions is substituted into formula (17) and obtains:
α1=17.398 °, α2=-23.472 °, XL=6854.7mm, β1=19.461 °, β2=-23.2214 °, XR=
6924.4mm;
If above-mentioned result of calculation and known conditions being taken back formula (11) verify, it can be deduced that speaker's mouth phase
For The Cloud Terrace base coordinate system O0D coordinates value:
' P=[6000.1 1999.7 2999.8 1]T
Understand ' P ≈ P, and error is in allowed band, demonstrates the correctness of angle method for solving of the present invention.
Finally, according to step 6 of the present invention, face rectangular peripheral profile of speaking center is at the video of left photographic head 201
Image coordinate system Ol-XlYlVideo image coordinate system O with right photographic head 201r-XrYrIn two-dimensional coordinate value be respectively as follows: (520,
100), (130,110), be respectively as follows: with two-dimensional coordinate value (320, the 240) side-play amount of video image center
Photographic head focal length parameter is taken as: length=3000, above-mentioned condition is brought into formula (20)-(23) and show that each revolves
Corner correction value:
Δα1=3.81 °, Δ α2=-2.61 °, Δ β1=-3.62 °, Δ β2=-2.48 °
After respective drive motor rotates above-mentioned correction angle, speaker face rectangular peripheral profile can be adjusted in image
Heart position.
Embodiment 2: bionic eye cradle head structure design parameter is: Pz1=150mm, Pz2=120mm, Py1=120mm, it is assumed that
Run sound localization algorithm by controller module 500 and obtain speaker's mouth relative to The Cloud Terrace base coordinate system O0Three-dimensional coordinate
Value P=[3,000 3,000 3,000 1]T;Obtain according to (12) formula and known conditions in step 3 of the present invention:
α1=43.834 °, β1=46.1267 °,
Visible, they are all beyond ± 30 ° of scopes, according to step 4 of the present invention, first have to enter carrying platform 302 cervical region
Row rotates and adjusts, and known conditions substitution (16) formula can be tried to achieve: θ=45.0033 °, and control carrying platform 302 anglec of rotation θ=
After 45.0033 °, make carrying platform 302 just to the mouth direction of speaker;Then, relatively current cervical region coordinate system O1Enter
Row step 5 computing, by P=[3,000 3,000 3,000 1]T, θ=45.0033 ° and other known conditions substitution formula (17),
: α1=-1.6204 °, α2=-32.7519 °, XL=5046.5mm, β1=1.6204 °, β2=-32.7519 °, XR=
5046.5mm, now, α1=-1.6204 °, β1=1.6204 °, all within ± 30 °;If by above-mentioned result of calculation and known
Condition is taken back formula (11) and is verified, it can be deduced that speaker's mouth is relative to The Cloud Terrace base coordinate system O0D coordinates value
For:
' P=[3000.1 2999.8 2999.9 1]T
Understand ' P ≈ P, and error is in allowed band, demonstrates the correctness of angle method for solving of the present invention.
Finally, according to step 6 of the present invention, face rectangular peripheral profile of speaking center is at the video of left photographic head 201
Image coordinate system Ol-XlYlVideo image coordinate system O with right photographic head 201r-XrYrIn two-dimensional coordinate value be respectively as follows:
(400,340), (240,350), with two-dimensional coordinate value (320,240) the side-play amount difference of video image center
For:
Photographic head focal length parameter is taken as: length=3000, above-mentioned condition is brought into formula (20)-(23) and show that each revolves
Corner correction value:
Δα1=1.53 °, Δ α2=1.90 °, Δ β1=-1.53 °, Δ β2=2.09 °
After respective drive motor rotates above-mentioned correction angle, speaker face rectangular peripheral profile can be adjusted in image
Heart position.
The technology such as sound localization, recognition of face and visual processes and algorithm are known technology, are not described in detail at this, this
Invention combines above-mentioned technology and the design of bionic eye cradle head structure and joint space derivation algorithm, it is achieved that bionic eye is to face of speaking
Being accurately positioned and following the trail of of portion's rectangular peripheral profile.
Claims (3)
1. a bionic eye location tracking system, it is characterised in that it is mainly made up of following six parts:
Bionic eye The Cloud Terrace module, sound collection sensor array, camera module, The Cloud Terrace motion-control module, controller module
And upper computer module;Wherein:
Described bionic eye The Cloud Terrace module includes: a The Cloud Terrace base, the sub-The Cloud Terrace of a carrying platform and two, left and right, wherein,
It is connected, by carrying platform level on The Cloud Terrace base by a horizontal rotation joint between The Cloud Terrace base and carrying platform
Rotate the neck rotation imitating bio-robot;Carrying platform is provided with two the sub-The Cloud Terraces in left and right, the sub-cloud in two, left and right
Platform is respectively arranged with a horizontal rotating frame and a pitching swivel mount respectively, and so, every sub-The Cloud Terrace all has horizontal rotation and pitching
Rotating two cradle heads, two photographic head are separately mounted on the pitching swivel mount in the middle of the sub-The Cloud Terrace in two, left and right, serve as imitative
The eyes of raw eye, by the respective horizontal rotation of sub-The Cloud Terrace of two, left and right and the rotation of elevation rotary joint, can adjust respectively
The photographic head being arranged on two the sub-The Cloud Terraces in left and right points to, and imitates the rotation of eyes;
Described sound collection sensor array serves as bionic ear in the present system, by mike identical for four characteristics, with imitative
Centered by raw eye The Cloud Terrace base fulcrum, it is arranged on around The Cloud Terrace base center by the space geometry relation of positive tetrahedron, and by cloud
Platform base props up dot center's initial point as bionic eye The Cloud Terrace base coordinate system, so, by the acoustical signal collected and sound source
Location algorithm, obtains speaker's mouth D coordinates value relative to base coordinate system;
Described camera module is made up of two independent photographic head, is separately mounted to the pitching in the middle of the sub-The Cloud Terrace in two, left and right
On swivel mount, serve as the eyes of bionic eye in the present system, be used for obtaining video information, and video data is passed to host computer mould
Block;
Described The Cloud Terrace motion-control module includes: be separately mounted to left and right two sub-The Cloud Terraces on horizontal rotation drive motor and
Pitching rotary drive motor, the horizontal rotation of a carrying platform drive motor and data transmission module;The Cloud Terrace motor control mould
Tuber enters to the driving motor controlling corresponding joint according to each driving electric machine rotation signal of the controller module output received
Row rotates;
Described controller module, on the one hand gathers the output signal of sound collection sensor array, runs sound localization algorithm,
Obtain speaker's mouth D coordinates value relative to The Cloud Terrace base coordinate system, and the three-dimensional coordinate of this speaker's mouth is passed to
Upper computer module;On the other hand receive control instruction and data message that upper computer module sends, control each and drive motor
Rotate respective angles, the most also return the kinestate of system to upper computer module;
Described upper computer module includes: the eye level corner α of bionic eye The Cloud Terrace1And β1Derivation program, flat to carrying
The derivation of two sub-The Cloud Terrace joint spaces about platform horizontal rotation the derivation program of rotational angle theta, bionic eye The Cloud Terrace module
Program, vision processing algorithm program and the signal procedure with controller.
2. the method for work of a bionic eye location tracking system as claimed in claim 1, it is characterised in that mainly comprise as
Lower step:
Step one, system initialization: system electrification, sound collection sensor array enters duty, and upper computer module shows
The video image that camera module obtains, controller module controls all driving reset motors to initial angle;
Step 2, sound localization: controller module reads the voice data that sound collection sensor array gathers, and runs sound source
Location algorithm obtains speaker's mouth D coordinates value relative to The Cloud Terrace base coordinate system, and this coordinate figure is passed to host computer mould
Block;
Step 3, eye level corner solve: i.e. seek two sub-The Cloud Terrace each level angles about bionic eye The Cloud Terrace module
Solve;
Upper computer module relative to the D coordinates value of The Cloud Terrace base coordinate system, is converted to speaker's mouth according to speaker's mouth
Relative to the D coordinates value of cervical region coordinate system, when, under the non-rotation status of forward neck portion, solving two eyes and turn to respectively
Level angle required during words people's mouth;If wherein the level angle of any one exceedes ± and 30 degree, at this moment need cervical region is entered
Row rotates and adjusts, then turn to step 4;If the feathering angle of solve out two eyes is all not less than ± 30 degree, the most not
Need to carry out cervical region rotating to adjust, then turn to step 5;
Step 4, neck joint is carried out rotate adjust: i.e. to carrying platform horizontal rotation corner solve;
Upper computer module first according to speaker's mouth relative to the D coordinates value of cervical region coordinate system, turn to meeting carrying platform
Honest is condition towards speaker's mouth, and the anti-feathering angle solved needed for cervical region, the level of cervical region is revolved by upper computer module
Corner issues controller module, controller module control the horizontal rotary drive motor of cervical region and rotate respective angles, it is achieved carry
Platform turns to honest in the face of speaker's mouth direction, and so, cervical region coordinate system also turns to honest in the face of speaker's mouth direction, so
Rear steering step 5, more relatively current cervical region coordinate system carries out the joint space of two sub-The Cloud Terraces about bionic eye The Cloud Terrace module
Derivation, so that two eyes turn to speaker's mouth direction respectively;
About step 5, bionic eye The Cloud Terrace module, two sub-The Cloud Terrace joint spaces solve:
Upper computer module is carried out about bionic eye The Cloud Terrace module according to the relation of speaker's mouth relatively current cervical region coordinate system
Two sub-The Cloud Terrace joint space derivations, the most anti-pitching rotary drive motor solving two the sub-The Cloud Terraces in left and right and level rotation
Turn the angle information that driving motor should rotate, be then issued to controller module, by controller module control about two sub-clouds
Platform respective pitching rotary drive motor and horizontal rotation drive electric machine rotation respective angles, preliminary to realize the eyes of bionic eye
To the mouth of speaker;
So far, complete two photographic head are tentatively directed at the process of speaker's mouth;
Step 6, eyes adjustment in direction: i.e. two each rotary joint corners of sub-The Cloud Terrace about bionic eye The Cloud Terrace module are repaiied
Just;
Upper computer module runs face recognition algorithms, obtains speaker face rectangular peripheral profile and asks for its center and exist
Two-dimensional coordinate value in video image coordinate system, according to the side-play amount of this two-dimensional coordinate value Yu video image center, respectively
Obtain pitching rotary drive motor and the correction value of the horizontal rotation driving motor anglec of rotation of two the sub-The Cloud Terraces in left and right, then will
Above-mentioned anglec of rotation correction value issues controller module again, by controller module control about two respective pitching of sub-The Cloud Terrace
Rotary drive motor and horizontal rotation drive motor fine setting respective angles, with realize to two eyes of bionic eye the most precisely towards
The face mask center position of speaker so that speaker face rectangular profile center occurs in the video image of photographic head identification
Center;
If step 7 controller module detects when new speaker occurs, then circulation perform step 2, step 3, step 4,
Step 5 and step 6, carry out real-time tracking to emerging face of speaking;If not having new speaker to occur, then circulate execution
Step 6 so that the corresponding degree of freedom of eyes adjusts according to the movement of face location of currently speaking;
Step 8, detection whole system the need of working on, the most then turn to step 7, it may be judged whether have new speaking
People occurs, if it is not, shutdown of then deactivating.
3. method of work as claimed in claim 2, it is characterised in that the calculating side of the eyes adjustment in direction value that step 6 is carried
Method step is as follows:
The most all angular units are all taken as: degree (°), all Parameter units relevant to video image are all taken as: pixel;
Step 6.1, upper computer module (600) read and show that the left photographic head (201) of bionic eye and right photographic head (202) obtain
Video image, if picture traverse is W, height be H;
Step 6.2, upper computer module (600) run face recognition algorithms, respectively at left photographic head (201), right photographic head (202)
In the video image identified, calculate the face rectangular peripheral profile center video image in left photographic head (201) of speaking
Coordinate system Ol-XlYlVideo image coordinate system O with right photographic head (201)r-XrYrIn two-dimensional coordinate value: (Xl, Yl)、(Xr,
Yr);
Step 6.3, take video image coordinate system O of left photographic head (201)l-XlYlSit with the video image of right photographic head (201)
Mark system Or-XrYrCenter position coordinates value be: (W/2, H/2), then in the video image that left photographic head (201) identifies, say
Two-dimensional coordinate value (the X of words face centerl, Yl) and two-dimensional coordinate value (W/2, the H/2) skew of video image center
Amount is:
In the video image that right photographic head (201) identifies, the two-dimensional coordinate value (X of speaker face centerr, Yr) with regard
Frequently two-dimensional coordinate value (W/2, the H/2) side-play amount of image center location is:
Step 6.4, ask for eyes corresponding freedom degree rotating angle correction according to side-play amount:
Left eye horizontal rotation freedom degree rotating angle correction value: Δ α1=arctan (Δ Xl/length);
Left eye pitching rotary freedom anglec of rotation correction value: Δ α2=arctan (Δ Yl/length);
Right eye horizontal rotation freedom degree rotating angle correction value: Δ β1=arctan (Δ Xr/length);
Right eye pitching rotary freedom anglec of rotation correction value: Δ β2=arctan (Δ Yr/length);
Wherein, length is the focal length parameter of photographic head, sets according to the actual parameter of photographic head;Anglec of rotation correction value Δ α1、
Δα2、Δβ1、Δβ2In all comprise positive and negative number two kinds of situations;
After respective drive motor rotates above-mentioned correction angle, thus speaker face rectangular peripheral profile is adjusted to the center of image
Position.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610379179.5A CN106042005B (en) | 2016-06-01 | 2016-06-01 | The method of work of bionic eye location tracking system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610379179.5A CN106042005B (en) | 2016-06-01 | 2016-06-01 | The method of work of bionic eye location tracking system |
Publications (2)
Publication Number | Publication Date |
---|---|
CN106042005A true CN106042005A (en) | 2016-10-26 |
CN106042005B CN106042005B (en) | 2018-02-06 |
Family
ID=57171669
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201610379179.5A Expired - Fee Related CN106042005B (en) | 2016-06-01 | 2016-06-01 | The method of work of bionic eye location tracking system |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN106042005B (en) |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106838563A (en) * | 2017-01-20 | 2017-06-13 | 上海大学 | A kind of robot bionic machinery head |
WO2018121730A1 (en) * | 2016-12-31 | 2018-07-05 | 深圳市优必选科技有限公司 | Video monitoring and facial recognition method, device and system |
WO2018145261A1 (en) * | 2017-02-08 | 2018-08-16 | 深圳市大疆创新科技有限公司 | Multifunctional camera and control method therefor, wearable device, pan-tilt, and aerial vehicle |
CN108702458A (en) * | 2017-11-30 | 2018-10-23 | 深圳市大疆创新科技有限公司 | Image pickup method and device |
CN109249386A (en) * | 2017-04-27 | 2019-01-22 | 丰田自动车株式会社 | Voice dialogue robot and speech dialogue system |
CN109382849A (en) * | 2017-08-07 | 2019-02-26 | 刘海云 | Using the robot eyes of cone coordinate system vibration zoom |
CN109605397A (en) * | 2019-01-11 | 2019-04-12 | 山东元脉电子技术股份有限公司 | Artificial intelligence educational robot and its control method with automatic tracking function |
CN109693235A (en) * | 2017-10-23 | 2019-04-30 | 中国科学院沈阳自动化研究所 | A kind of Prosthetic Hand vision tracking device and its control method |
CN111136650A (en) * | 2018-11-02 | 2020-05-12 | 深圳市优必选科技有限公司 | Robot and visual tracking method thereof and computer readable storage medium |
WO2020114154A1 (en) * | 2018-12-05 | 2020-06-11 | 腾讯科技(深圳)有限公司 | Animation control method and device, storage medium, and electronic device |
CN112418296A (en) * | 2020-11-18 | 2021-02-26 | 中国科学院上海微***与信息技术研究所 | Bionic binocular target recognition and tracking method based on human eye visual attention mechanism |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6826284B1 (en) * | 2000-02-04 | 2004-11-30 | Agere Systems Inc. | Method and apparatus for passive acoustic source localization for video camera steering applications |
CN202607678U (en) * | 2012-04-19 | 2012-12-19 | 上海大学 | Eyeball movement mechanism for bionic-robot |
CN103072140A (en) * | 2012-12-18 | 2013-05-01 | 北京理工大学 | Binocular three-dimensional bionic eye mechanism with panoramic capture and positioning functions |
CN103841357A (en) * | 2012-11-21 | 2014-06-04 | 中兴通讯股份有限公司 | Microphone array sound source positioning method, device and system based on video tracking |
CN104469154A (en) * | 2014-12-05 | 2015-03-25 | 合肥国科声拓信息技术有限公司 | Camera guiding device and method based on microphone array |
CN105184214A (en) * | 2015-07-20 | 2015-12-23 | 北京进化者机器人科技有限公司 | Sound source positioning and human face detection-based human body positioning method and system |
-
2016
- 2016-06-01 CN CN201610379179.5A patent/CN106042005B/en not_active Expired - Fee Related
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6826284B1 (en) * | 2000-02-04 | 2004-11-30 | Agere Systems Inc. | Method and apparatus for passive acoustic source localization for video camera steering applications |
CN202607678U (en) * | 2012-04-19 | 2012-12-19 | 上海大学 | Eyeball movement mechanism for bionic-robot |
CN103841357A (en) * | 2012-11-21 | 2014-06-04 | 中兴通讯股份有限公司 | Microphone array sound source positioning method, device and system based on video tracking |
CN103072140A (en) * | 2012-12-18 | 2013-05-01 | 北京理工大学 | Binocular three-dimensional bionic eye mechanism with panoramic capture and positioning functions |
CN104469154A (en) * | 2014-12-05 | 2015-03-25 | 合肥国科声拓信息技术有限公司 | Camera guiding device and method based on microphone array |
CN105184214A (en) * | 2015-07-20 | 2015-12-23 | 北京进化者机器人科技有限公司 | Sound source positioning and human face detection-based human body positioning method and system |
Cited By (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2018121730A1 (en) * | 2016-12-31 | 2018-07-05 | 深圳市优必选科技有限公司 | Video monitoring and facial recognition method, device and system |
CN106838563A (en) * | 2017-01-20 | 2017-06-13 | 上海大学 | A kind of robot bionic machinery head |
WO2018145261A1 (en) * | 2017-02-08 | 2018-08-16 | 深圳市大疆创新科技有限公司 | Multifunctional camera and control method therefor, wearable device, pan-tilt, and aerial vehicle |
CN109249386B (en) * | 2017-04-27 | 2022-01-11 | 丰田自动车株式会社 | Voice conversation robot and voice conversation system |
CN109249386A (en) * | 2017-04-27 | 2019-01-22 | 丰田自动车株式会社 | Voice dialogue robot and speech dialogue system |
CN109382849A (en) * | 2017-08-07 | 2019-02-26 | 刘海云 | Using the robot eyes of cone coordinate system vibration zoom |
CN109693235A (en) * | 2017-10-23 | 2019-04-30 | 中国科学院沈阳自动化研究所 | A kind of Prosthetic Hand vision tracking device and its control method |
CN108702458B (en) * | 2017-11-30 | 2021-07-30 | 深圳市大疆创新科技有限公司 | Shooting method and device |
CN108702458A (en) * | 2017-11-30 | 2018-10-23 | 深圳市大疆创新科技有限公司 | Image pickup method and device |
US11388333B2 (en) | 2017-11-30 | 2022-07-12 | SZ DJI Technology Co., Ltd. | Audio guided image capture method and device |
CN111136650A (en) * | 2018-11-02 | 2020-05-12 | 深圳市优必选科技有限公司 | Robot and visual tracking method thereof and computer readable storage medium |
WO2020114154A1 (en) * | 2018-12-05 | 2020-06-11 | 腾讯科技(深圳)有限公司 | Animation control method and device, storage medium, and electronic device |
US11783523B2 (en) | 2018-12-05 | 2023-10-10 | Tencent Technology (Shenzhen) Company Limited | Animation control method and apparatus, storage medium, and electronic device |
CN109605397A (en) * | 2019-01-11 | 2019-04-12 | 山东元脉电子技术股份有限公司 | Artificial intelligence educational robot and its control method with automatic tracking function |
CN112418296A (en) * | 2020-11-18 | 2021-02-26 | 中国科学院上海微***与信息技术研究所 | Bionic binocular target recognition and tracking method based on human eye visual attention mechanism |
CN112418296B (en) * | 2020-11-18 | 2024-04-02 | 中国科学院上海微***与信息技术研究所 | Bionic binocular target identification and tracking method based on human eye visual attention mechanism |
Also Published As
Publication number | Publication date |
---|---|
CN106042005B (en) | 2018-02-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106042005B (en) | The method of work of bionic eye location tracking system | |
CN109753076B (en) | Unmanned aerial vehicle visual tracking implementation method | |
CN106529495B (en) | Obstacle detection method and device for aircraft | |
CN102509261B (en) | Distortion correction method for fisheye lens | |
US7272306B2 (en) | Bionic automatic vision and line of sight control system and method | |
CN101435704B (en) | Star tracking method of star sensor under high dynamic state | |
Kuniyoshi et al. | Active stereo vision system with foveated wide angle lenses | |
CN106327571A (en) | Three-dimensional face modeling method and three-dimensional face modeling device | |
CN109933096B (en) | Cloud deck servo control method and system | |
CN105835036B (en) | A kind of parallel connected bionic eye device and its control method | |
CN210201937U (en) | Image acquisition device | |
CN105222758A (en) | Based on multiple mobile object search and locating device and the method for birds visual signature | |
CN103929635B (en) | Binocular vision image compensation method when a kind of UUV shakes in length and breadth | |
WO2020063058A1 (en) | Calibration method for multi-degree-of-freedom movable vision system | |
CN106799732A (en) | For the control system and its localization method of the motion of binocular head eye coordination | |
CN105631859A (en) | Three-degree of freedom bionic stereo vision system | |
CN106780621A (en) | The method that axicon lens face catadioptric video camera minute surface parameter is solved using straight line | |
CN115291536A (en) | Vision-based verification method for ground target tracking semi-physical simulation platform of unmanned aerial vehicle | |
CN112288801A (en) | Four-in-one self-adaptive tracking shooting method and device applied to inspection robot | |
JP2008233081A (en) | Method of recognizing and tracking spatial point | |
AU2020103932A4 (en) | Virtual-coaxial portable three-dimensional camera with small volume | |
WO2021093804A1 (en) | Omnidirectional stereo vision camera configuration system and camera configuration method | |
WO2022041129A1 (en) | Three-dimensional capturing apparatus, method and system for ethology recording, and application of system | |
CN206743361U (en) | A kind of VR pan-shots machine | |
CN115503024B (en) | Binocular robot with 720-degree visual angle and working method thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180206 Termination date: 20210601 |
|
CF01 | Termination of patent right due to non-payment of annual fee |