WO2012019322A1 - 利用单目摄像机标定技术的视觉指向型鼠标输入方法、输入***和输入装置 - Google Patents
利用单目摄像机标定技术的视觉指向型鼠标输入方法、输入***和输入装置 Download PDFInfo
- Publication number
- WO2012019322A1 WO2012019322A1 PCT/CN2010/001229 CN2010001229W WO2012019322A1 WO 2012019322 A1 WO2012019322 A1 WO 2012019322A1 CN 2010001229 W CN2010001229 W CN 2010001229W WO 2012019322 A1 WO2012019322 A1 WO 2012019322A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- target
- image
- image sensor
- computer
- pointing
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
- G06F3/0317—Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
- G06F3/0321—Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet
Definitions
- the invention relates to a computer peripheral input technology and device, and is a method and device for realizing precise positioning of a camera vision and driving a mouse cursor or other display target by using a monocular camera calibration technology, in particular, a monocular image imaging device can be realized.
- a directional pointing mouse device that points to the input device. Background technique
- the mouse device used to drive cursor movement in the graphical user interface is a mechanical mouse, typically having a trackball or direction key component that controls the movement of the cursor using the motion of the trackball relative to the mouse pad or by pressing the direction keys.
- the optical mouse gradually replaced the mechanical mouse.
- it moved on the working plane it continuously acquired the image reflected by the plane, processed the sequence image by image processing technology, and extracted the moving direction and displacement to drive the cursor motion.
- Mechanical mice and optical mice generally need to move on a certain working plane, which limits the convenience of their use.
- the touch screen can use the touch method to locate the input position and drive the cursor, thereby realizing the accurate directional input of the graphical user interface.
- the use of the touch screen requires direct contact of the pointer with the screen, which imposes certain constraints on its application, and is costly to manufacture, which is difficult to manufacture on a large-area screen.
- an image capturing device such as a digital camera is generally placed and calibrated in advance, and the field of view includes a computer display screen and an indicator bar.
- a pointing device such as a finger, so that the display coordinates of the mouse cursor can be determined according to the relative position of the pointing point in the display screen.
- This method requires a certain amount of space, and often requires the use of auxiliary devices such as laser pointers, selective reflection films, filter polarizers, etc., which makes the entire system extremely complicated.
- a visually directional input technique is to bind the digital camera to the pointing end of the finger or the pointing stick.
- the camera captures the localized display image content and then inputs it into the host computer. Scan the area of the screen being displayed area by area, find the position of the pointing area in the display screen; or control the movement of the cursor by the relative displacement of the captured image content caused by the movement of the indicating end.
- this patent proposes a simple and easy way to use the monocular camera calibration technology in machine vision technology to accurately extract the cursor pointing point coordinates of the visual pointing mouse input method, which can be used to manufacture accurate non-contact vision pointing mouse devices.
- the patent proposes a visual pointing mouse input method using a monocular camera calibration technique for controlling a graphical target such as a mouse cursor to accurately follow the pointing movement of the virtual pointing axis of the image sensor, the method comprising the following steps:
- the sensor is directed to a planar target, the target has a defined target coordinate system, wherein a plurality of characteristic targets are set, the image sensor is activated, and the image sensor is connected to the computer by wire or wirelessly;
- the image sensor collects the feature content included in the target pointing area Image, extracting the image coordinates of the feature target in the image; using the acquired target image coordinates and the coordinates of the target in the target coordinate system, and calculating the imaging parameters of the monocular camera according to the monocular camera calibration technique;
- the image coordinates of a fixed image point on the imaging surface of the sensor, using the calculated monocular camera imaging parameters calculate the coordinates of the object point corresponding to the image point in the target coordinate system, that is, the indication point is in the target coordinate system.
- the patent proposes a visual pointing type mouse input method using a monocular camera calibration technique for implementing a visual input method of a spatial motion gesture, the method comprising the steps of: pointing a monocular image sensor to a planar target, The target has a certain target coordinate system, wherein a plurality of characteristic targets are set, the image sensor is activated, the image sensor is connected to the computer by wire or wirelessly, and the image sensor is activated; the image sensor collects an image of the feature content included in the target pointing region.
- the spatial orientation coordinates include the three-axis rotation angle ( ⁇ , ⁇ , ⁇ ) and the origin coordinates (X, Y, Z).
- ⁇ , ⁇ , ⁇ the three-axis rotation angle
- X, Y, Z the origin coordinates
- This method is suitable for the target can be set on the dynamic display device, for example.
- the computer first determines that the image sensor is pointing to the approximate area of the display screen, and then the computer dynamically determines a target in the display pointing area of the image sensor.
- the present patent proposes two methods of determining the position of the image sensor pointing to the display screen.
- the first method comprises the following steps: collecting an image sensor to a target area of the display screen, the device is connected to the computer host by wired or wireless communication, and the display screen is connected with the computer host; the image sensor is activated to notify the computer host Outputs a number of different colors or graphics on the display in a very short time
- the coding pattern consisting of the feature block arrangement of the content, each color or graphic content is coded into different numbers, and the code composed of all the feature blocks in a certain range near each feature block is unique in the entire coded picture, and the entire coded picture
- the area codes of all the feature blocks constitute a positioning lookup table.
- a coding pattern composed of a plurality of rectangular feature block arrangements of different colors or graphic contents may be output on the display screen, and the nX n range near each rectangular feature block
- the coding of the feature block is unique in the coded picture; the image sensor collects the coded image of the pointed area, extracts the coding of the local pattern from it, compares it with the spatial position lookup table of the coded picture, determines the image sensor pointing The approximate location of the display.
- the second method for determining the position of the image sensor pointing to the display screen proposed by the patent includes the following steps: collecting an image sensor to a target area of the display screen, and the device is connected to the host computer by wired or wireless communication, and the display screen is The computer host is connected; the image sensor is activated to work, firstly outputting a coarse resolution coding pattern composed of a plurality of different color or graphic content feature block arrangements on the display screen, each color or graphic content is encoded into a different number, image sensor acquisition Pointing to the area image to determine the position of the feature block to which it is located; then, the host computer outputs the same coding pattern again within the determined large feature block, the size of which is the size of the feature block, and the image sensor collects the image of the pointing area, further Judging the position of the small feature block to which it is located; thus performing this operation in a fast loop of large and small, finally determining the position of the display screen pointed by the image sensor.
- the directional pointing mouse input method proposed by the patent wherein the planar target is specifically characterized as follows: The size of the target area is determined, and the area includes a plurality of characteristic target points, and the target has a specific color, shape and the like, which is convenient for The image is extracted and the coordinates of the target in the target coordinate system of the target region are known.
- planar target may be a fixed planar target
- the frame of the display screen may be selected as a target area
- the characteristic target point is set on the frame, and the distance between each point
- a size-determined plane around the display screen can be selected as the target area
- the feature target points are set in the area
- the distance between the points is known;
- a certain fixed area can be selected by the computer as a target on the display screen.
- the size of the area is determined by the computer display coordinates of the area, and a number of feature points are determined by the computer within the display target area as feature targets.
- planar target may be a dynamic planar target
- the target may be dynamically generated by a computer on a display screen
- the target generating position always follows the direction of the image sensor
- the target area is The range can be adjusted according to the imaging distance of the image sensor, the size of which is determined by the computer display coordinates of the area, and the computer determines a number of feature points as feature points in the display target area.
- the visual pointing type mouse input method proposed by the patent wherein the computer determines the feature target point in the target area of the display screen, the computer refers to: the computer processes the display content in a certain range including the target area on the display screen, and uses the color, Features such as edge, corner, azimuth, and surrounding environment information, select several feature targets from the display content of the computer, define the target region range from these feature targets, and record the feature information.
- the visual pointing type mouse input method proposed by the patent wherein the determining, by the computer, the feature target point in the target area of the display screen comprises the following steps: the computer counts the color of the display content in a certain range of the target area on the display screen, and selects Displaying a color that is not in the content and has a large difference from the existing color as the color of the generated feature target; by the computer in the target area on the display screen, a certain display content is additionally generated by the selected color, and the generated display is generated.
- the content includes features such as intersections, corners, and center points. Based on these features, several feature targets can be selected, and the target regions are defined by the feature targets.
- the visual pointing type mouse input method proposed by the patent refers to: a visual pointing type mouse input method using an image sensor that does not previously calibrate internal physical parameters such as a focal length and a pixel interval, and needs to be displayed At least 4 feature targets are determined on the screen.
- the visual pointing type mouse input method proposed by the patent refers to: a visual pointing type mouse input method using an image sensor that has previously calibrated internal physical parameters such as a focal length and a pixel interval, and needs to be displayed At least 3 feature targets are determined on the screen.
- the solving monocular camera imaging parameter refers to: using the acquired target image coordinate and the target coordinate coordinate of the target, according to the monocular camera calibration technology, the solution is calculated.
- Monocular camera external imaging parameters of the image sensor
- the solving monocular camera imaging parameter refers to: using the acquired target image coordinate and the target coordinate coordinate of the target, according to the monocular camera calibration technology, the solution is calculated.
- the fixed image point on the imaging surface of the image sensor means that the image point can be any image point on the imaging surface, and the image point and the imaging lens optical center
- the connection constitutes a virtual indication axis, and the object point corresponding to the image point is the pointing point of the virtual indication axis.
- the fixed image point on the imaging surface of the image sensor means that the image point can be a central image point on the imaging surface, and the image point and the optical center of the imaging lens
- the connection that is, the optical axis of the imaging system, constitutes a virtual indication axis, and the object point corresponding to the central image point is the pointing point of the virtual axis.
- the visual pointing type mouse input method proposed by the patent calculates the display coordinates of the mouse cursor by the coordinates of the indication point in the target coordinate system, and refers to: the unit length of the target coordinate system and the pixel interval size of the computer display screen.
- the coordinates of the target that have been calculated in the target coordinate system are the display coordinates.
- the unit length of the target coordinate system is different from the pixel interval size of the computer display screen, the calculated target point needs to be
- the coordinates in the target coordinate system are multiplied by a scaling factor to obtain display coordinates obtained by dividing the cell spacing size of the display screen by the unit length of the target coordinate system.
- the patent proposes a vision-oriented mouse input system using monocular camera calibration technology, which comprises: a computer mainframe, and a display screen connected thereto; a planar target including a plurality of characteristic targets, the distance between each target point has been It is known that each target has a certain feature to facilitate the extraction of the target from the image; the monocular image sensor is connected to the host computer through the processing circuit, and the monocular image sensor is used as a pointing device to point to the target area of the display screen.
- the image is a virtual indication axis with a fixed point on the imaging surface of the image sensor and the center of the lens; the control function is arranged with various control function buttons for generating system trigger, left button, right button, page turning, moving, etc.
- the control function is connected to the host computer through the processing circuit; the system processing circuit is connected to the image sensor and the control function component, and is connected to the host computer by wire or wirelessly; the visual pointing type mouse information receiving and processing device is inserted in the computer host , with computer operating system and processing Road communication interaction.
- the functions of the processing circuit of the system include: processing the acquired image, completing the pointing orientation of the image sensor, extracting the feature target, calculating the parameter of the monocular camera imaging model, and calculating the coordinates of the pointing point of the cursor; generating a system trigger, Control function signals such as left button, right button, page turning, and movement; communicate with the host computer through wired or wireless communication, and transmit information such as images, feature information, operation results, and control signals.
- the functions of the information receiving and processing device of the system include: receiving information such as images and operation results transmitted by the processing circuit; processing the acquired images to complete pointing positioning of the image sensor, extracting characteristic points, and calculating parameter of the monocular camera imaging model The function of calculating the coordinates of the cursor pointing to the point; receiving the control function signals generated by the processing circuit, such as system trigger, left button, right button, page turning, and movement; transmitting the feature information of the target to the visual pointing mouse input system processing circuit, Coordinate information; output the calculated cursor coordinate information to the computer operating system.
- the patent proposes a vision-oriented mouse input system using monocular camera calibration technology, which comprises: a computer mainframe, and a display screen connected thereto; a planar target including a plurality of characteristic targets, the distance between each target point has been It is known that each target has a certain feature to facilitate extraction of a target from an image; a monocular image sensor is connected to a host computer through an information receiving processing device, and a monocular image sensor is used as a pointing device to point to a target of the display screen.
- the area captures the image, and the connection line of a certain fixed point and the lens center on the image sensor image surface is used as a virtual indication axis;
- the control function component is arranged with various control function buttons for generating system trigger, left button, right button, page turning,
- a control function such as movement is connected to the host computer through the information receiving and processing device;
- the visual pointing type mouse information receiving and processing device is inserted in the computer host, and communicates with the computer operating system and the image sensor and the control function, and the functions thereof include: receiving Image sent by the monocular image sensor Receiving control function signals generated by the control function, such as system trigger, left button, right button, page turning, movement, etc.; processing the acquired image to complete the orientation of the image sensor, feature target extraction, monocular camera imaging model The parameter calculation, the display coordinate calculation of the cursor pointing point, etc.; output the calculated cursor coordinate information to the computer operating system.
- the patent proposes a glove-type directional pointing type mouse input device, which comprises: a computer host, and a display screen connected thereto; a plane target including a plurality of characteristic targets, a distance between each target point is known, each target point It has a certain feature to facilitate the extraction of the target from the image; a pointing finger sleeve with a monocular image sensor is used to collect the image from the target area of the display screen, and is connected to the host computer through the processing circuit to image the image sensor The connection between a fixed point and the center of the lens is used as a virtual indicator axis; the control function key finger sleeve includes a plurality of buttons, touch keys or pressure switches, which are controlled by a thumb for generating system trigger, left button, right button, page turning, and movement The control signal is connected to the host computer through the processing circuit; the auxiliary function key finger sleeve may include a function control button such as turning the page, and the finger is triggered by the finger itself by bending the finger, and is
- the functions of the processing circuit of the visual pointing type mouse input device include: processing the acquired image, completing the pointing orientation of the image sensor, extracting the feature target point, calculating the parameter of the monocular camera imaging model, and performing the coordinate calculation of the pointing point of the cursor.
- Generate control signals such as system trigger, left button, right button, page turning, and movement; communicate with the host computer through wired or wireless communication, and transfer images, feature information, operation results, control signals, and other information.
- the function of the information receiving processing device of the visual pointing type mouse input device comprises: receiving information such as an image transmitted by the processing circuit, an operation result, etc.; processing the acquired image to complete pointing positioning of the image sensor, extracting feature points, and monocular camera Function calculation of imaging model, display coordinate calculation of cursor pointing point; receiving control function signals generated by processing circuit, such as system trigger, left button, right button, page turning, movement; sending target to visual pointing mouse input system processing circuit Feature information and coordinate information of the point; output the calculated cursor coordinate information to the computer operating system.
- the patent proposes a glove-type directional pointing type mouse input device, which comprises: a computer host, and a display screen connected thereto; a plane target including a plurality of characteristic targets, a distance between each target point is known, each target point Determining features to facilitate the extraction of targets from images; pointing finger sleeves with monocular image sensors for capturing images in the target area of the display, connected to the host computer, to a fixed point on the imaging surface of the image sensor The connection to the center of the lens serves as a virtual indicator axis; the control function finger cot includes a number of buttons, touch keys or pressure switches, controlled by the thumb, for generating control signals such as system trigger, left button, right button, page turning, movement, etc.
- auxiliary function key finger sleeve can include function control keys such as page turning, by bending the finger, the finger itself triggers the switch control, and is connected with the computer host, the auxiliary function key finger sleeve is selected according to the use condition;
- Type mouse information receiving and processing device installed in the host computer
- the communication interaction with the computer operating system and the image sensor and the control function includes: receiving image information sent by the image sensor; receiving system trigger, left button, right button, page turning, moving, etc. generated by the control function component Function signal; processing the acquired image to complete the orientation of the image sensor, feature target extraction, parameter calculation of the monocular camera imaging model, display coordinate calculation of the cursor pointing point, etc.; output the calculated cursor to the computer operating system Coordinate information.
- the patent proposes a finger-type directional pointing type mouse input device, which comprises: a computer host, and a display screen connected thereto; a plane target including a plurality of characteristic targets, a distance between each target point is known, each target The point has a certain feature to facilitate the extraction of the target from the image; the visual pointing mouse input finger sleeve integrates the monocular image sensor, the processing circuit, and the control function key on one finger sleeve, and is worn on the index finger or other fingers.
- the thumb realizes function control, and the image sensor located at the front end of the finger sleeve is pointed to the target area of the display screen to collect images, and the image sensor is connected to the computer host through a processing circuit in a wired or wireless manner, and the image sensor is fixed on the imaging surface of the image sensor.
- the connection between the point and the center of the lens serves as a virtual indicator axis.
- the control function keys of the finger set include a number of buttons, touch keys or pressure switches, which are controlled by the thumb for generating system trigger, left button, right button, page turning, movement, etc.
- Function signal connected to the host computer through the processing circuit; information reception Processing means placed in the host computer, the processing circuit communication device interacting with a computer operating system and mouse input.
- the processing circuit in the mouse input finger sleeve is connected with the image sensor and the control function component, and is connected to the host computer by wire or wirelessly, and the functions thereof include: processing the image of the image set, completing the pointing position of the image sensor, and defining the target Point extraction, parameter calculation of the monocular camera imaging model, display coordinate calculation of the cursor pointing point, etc.; generating control function signals such as system trigger, left button, right button, page turning, and movement; and wired or wireless with the host computer Communication interaction, passing information such as images, feature information, operation results, control signals, and so on.
- the function of the information receiving processing device of the mouse input device includes: receiving information such as an image transmitted by a processing circuit of the mouse input device, an operation result, and the like; receiving a system trigger, a left button, a right button, a page turning, a moving, and the like generated by the processing circuit Signal; processing the acquired image to complete the orientation of the image sensor, feature target extraction, parameter calculation of the monocular camera imaging model, display coordinate calculation of the cursor pointing point, etc.; sending characteristic information of the target to the processing circuit, Coordinate information; output the calculated cursor coordinate information to the computer operating system.
- the patent proposes a finger-type directional pointing type mouse input device, which comprises: a computer host, and a display screen connected thereto; a plane target including a plurality of characteristic targets, a distance between each target point is known, each target The point has a certain feature to facilitate the extraction of the target from the image; the visual pointing type mouse input finger sleeve, the monocular image sensor and the control function key are integrated on one finger sleeve, worn on the index finger or other fingers, and the function is realized by the thumb Control, when using, the image sensor located at the front end of the finger cot is directed to the target area of the display screen to collect images, and the image sensor is connected to the host computer by wire or wirelessly, with a fixed point of the image sensor image surface and the lens center connection line as
- the virtual indicator axis, the control function keys of the finger sleeve include a thousand buttons, a touch button or a pressure switch, which are controlled by a thumb, and are used to generate control signals such as system trigger, left
- the method is connected to the host computer; the information receiving and processing device is inserted in the host computer Communicating with computer operating system and image sensor and control function, the functions include: receiving image information sent by the image sensor, receiving system trigger, left button, right button, page turning, moving, etc. generated by the control function Signal, processing the acquired image, completing the pointing function of the image sensor, extracting the feature target, calculating the parameter of the monocular camera imaging model, and calculating the coordinates of the cursor pointing point, and outputting the calculated cursor coordinates to the computer operating system information.
- the patent proposes a visual pointing mouse application, which resides in a computer host and communicates with a computer operating system and a visually pointing mouse input system, and includes the following functions: a control function program, receiving by a visual pointing type The control function signal generated by the control function of the mouse input system, such as system trigger, left button, right button, page turning, movement, etc.; image receiving processing program, receiving image information sent by the image sensor of the vision pointing type mouse input system; image sensor positioning The program determines the position of the image sensor pointing to the display screen: after receiving the work trigger signal of the visual pointing type mouse input system, notifying the computer to output a positioning code pattern for determining the pointing position of the image sensor on the display screen in a very short time, Processing the image of the pointing area collected by the image sensor, extracting the included pattern code from the image of the local area, comparing it with the spatial position lookup table of the positioning code pattern, and determining the approximate position of the image sensor pointing to the display screen; Feature target generation program, determining
- the image coordinates of the fixed point that is, the image coordinates of the intersection of the virtual indication axis and the imaging surface, are used to calculate the corresponding intersection display on the computer display screen using the calculated monocular camera imaging parameters. Coordinate;
- the cursor displays the program, informing the computer to display the mouse cursor or other image target at the pointing point of the virtual pointing axis on the display screen, and re-determining the display pointing area centering on the display cursor, and re-determining the feature target in the pointing area point.
- FIG. 1 is a schematic view of a first embodiment of a directional pointing mouse using a monocular camera calibration technique
- FIG. 2A is a schematic view of a pinhole model of a monocular camera imaging
- FIG. 2B is a schematic diagram of calibrating the plane of the display screen by using a monocular camera calibration technique
- Figure 3A is a schematic diagram of coded positioning composed of different colors
- FIG. 3B is a schematic diagram showing pointing positioning according to local area coding of the display screen
- Figure 3C is a schematic diagram of coarse resolution coding positioning composed of different colors
- Figure 3D is a schematic view showing the pointing position of the image sensor for the first time
- Figure 3E is a schematic view showing the pointing position of the second positioning image sensor
- FIG. 4 is a schematic diagram of selecting a feature target by using features such as color, edge, corner, and the like in the display content
- FIG. 5A is a schematic diagram showing a color distribution of a display point and a color selection region of a target point in a target point setting area in a color space
- Figure 5B is a schematic diagram showing the generation of four cross targets in a square arrangement
- Figure 6 is a workflow diagram of a first embodiment of a directional pointing mouse utilizing monocular camera calibration technology
- FIG. 7 is a basic system block diagram of a visual pointing type mouse input system using monocular camera calibration technology
- FIG. 8A to FIG. 8C are schematic diagrams showing a wearing method and a usage method of a glove type visual pointing input device
- Figure 8E is a schematic diagram of the main control function
- Figure 8F is a schematic diagram of the auxiliary control function
- Figure 9A is a schematic view of a finger-type directional pointing input device that integrates all components into one body
- Figure 9B is a schematic view of a pen-type visual pointing input device
- Figure 10 is a flow chart showing a second embodiment of a directional pointing mouse utilizing monocular camera calibration technology
- FIGS. 11A to 11C are diagrams showing a third embodiment of a directional pointing type mouse using a monocular camera calibration technique
- Figure 12 is a workflow diagram of a third embodiment of a directional pointing mouse utilizing monocular camera calibration techniques.
- Figure 13 is a schematic illustration of a fourth embodiment of a directional pointing mouse utilizing monocular camera calibration techniques.
- Figure 14 is a workflow diagram showing a fourth embodiment of a directional pointing mouse utilizing monocular camera calibration techniques. detailed description
- the first embodiment relates to a visual pointing input method for controlling a graphical target such as a mouse cursor to be accurately displayed at an intersection of a virtual pointing axis of an image sensor pointing to a computer display screen.
- a dynamic display device such as a display screen is used as The setting area of the target is firstly determined by the computer to locate the position of the image sensor pointing to the display screen, and then the computer sets the feature target on the display screen to define the target area, and the target area can be pointed along with the image sensor. Movement, always keeping the target area in the imaged area of the image sensor.
- FIG. 1 is a schematic view of a first embodiment in which the visual pointing input device 10 is worn on the hand in the form of a glove, and the core working component is a small monocular image sensor 100 worn on the index finger for pointing
- the target area collects an image.
- the target area is set by a computer on the display screen, and the image sensor can automatically zoom according to the imaging distance to obtain a clear image (when the imaging distance is much larger than the focal length of the image sensor, the fixed focus image is used) The sensor can always get a clearer image);
- the input device also includes a control function 102 worn on the middle finger or other fingers, on which the control function keys are arranged, through the thumb control, for system triggering, left button, right button and page Scrolling and the like; and processing circuit 104, which is placed on the back of the hand in FIG.
- image sensor 100 or control function 102 which is coupled to image sensor 100 and control function 102, and is wired Or wirelessly connected to the host computer.
- the functions of the processing circuit 104 mainly include: processing the acquired image, completing the pointing orientation of the image sensor, extracting the feature target, calculating the parameter of the imaging model, and calculating the coordinate of the pointing point of the cursor; generating a system trigger, a left button, Control function signals such as right button and page scroll; communicate with the host computer through wired or wireless communication, and transmit information such as images, feature information, operation results, and control signals.
- the specific structure of the visual pointing input device 10 will be further explained later.
- the computer when the image sensor 100 worn on the index finger is photographed in a certain area of the display screen 18, the computer first uses some positioning technique to determine the approximate position of the display screen corresponding to the image sensor. The introduction of the technology will be explained later. Then, the computer selects several feature targets in the region according to the color, edge, corner point and the like of the display content, or generates a plurality of feature targets 12 in the region, the feature targets constitute the target region, and the desired target The number is related to the specific calibration technique used. At the same time, the computer transmits the characteristic information of the targets to the processing circuit 104 by wired or wireless communication, so that the input device can extract the special image from the acquired image by using these features. Target.
- the imaging parameters of the imaging model are calculated by the monocular camera calibration technique, and then the imaging parameters are used to calculate the intersection of the virtual indication axis 14 and the display screen 16 The relative coordinates of the target.
- the actual display coordinates of the intersection 16 on the display screen are calculated, so that the mouse cursor or other graphic object can be accurately displayed.
- the cursor can accurately follow the direction of the image sensor and move. For example, for an image sensor with a frame rate of 15 S, the time interval between the two images is about 67 ms.
- Figure 1 shows the computer using a pointing input device for processing, which also satisfies the situation where multiple similar pointing input devices are used simultaneously.
- 2A is a pinhole model of monocular camera imaging, showing a schematic view of imaging any object point in space.
- Object point 20a is projected through image center 22 on image plane 24 to become image point 20b.
- the imaging device may be CCD devices, CMOS devices, or other digital imaging devices.
- the world coordinate system O w -X w Y w Z w where the object point is located, the camera coordinate system O c -X c Y c Z c and the image coordinate system U-0- where the image plane is located V three coordinate systems, respectively, are 200, 202, 204.
- the entire imaging model relationship that is, the expression relationship between the coordinates (X w , Y w , Z w ) of the object point in the world coordinate system and the image coordinate system coordinate V of the image point on the image plane, can be determined by the world coordinates.
- Six external physical parameters, and three internal physical parameters such as the horizontal pixel interval d x and the vertical pixel interval d y of the camera focal length imaging device, a parameter matrix determined by a total of nine physical parameters are described:
- the external parameter matrix, M 2 is the internal parameter matrix determined by the camera's internal physical parameters: focal length f, lateral pixel spacing d x , longitudinal pixel spacing d y , ( uo , vo ) is the camera optical axis and imaging plane The coordinates of the intersection point in the image coordinate system.
- the coordinates of the world coordinate system of six spatial points and the coordinates of its image coordinate system are known, all parameters of the parameter matrices Mi, M 2 can be solved, and thus can be determined by any object point.
- the coordinates of the world coordinate system (X w , Y w , Z w ) calculate its coordinates (u, v) in the image coordinate system, and can also be inversely calculated from the arbitrary coordinate coordinates (u, V) in the world coordinate system.
- This patent is applied to the visual directivity input of the computer display screen. All the target points are located on the plane of the display screen. Therefore, the display plane can be used as the X w O w Y w plane of the world coordinate system, and the pixel interval of the display screen is taken. As the coordinate unit length of the world coordinate system X w O w Y w plane, that is, the X w O w Y w plane coordinate is consistent with the computer display coordinate unit, and the direction perpendicular to the plane is the Z w axis, as shown in FIG. 2B.
- 18 is a display screen located on the X w O w Y w plane
- the camera calibration theory even in the case of the internal physical parameters of the unknown image sensor, only the object space display by the four targets on the display plane is solved.
- the set of imaging equations determined by the coordinates and image space image coordinates can be used to solve the external parameter matrix that satisfies the requirements.
- the display coordinates of the intersection of the virtual pointing axis determined by a certain fixed point on the imaging surface and the lens center line and the computer display screen can be obtained, thereby accurately positioning the mouse cursor or other display targets.
- the optical axis of the image sensor can be used as an indicator axis for convenience.
- 12a is a feature target set in the display screen
- 12b is its image point in the imaging plane
- the required imaging parameters can be calibrated, and thus can be
- the image coordinates of the intersection 16b of the optical axis and the imaging surface calculate the coordinates of the intersection 16a in the corresponding display screen with respect to the target point, and finally the accurate computer display coordinates can be obtained.
- the application method described above can be directly used in the case of the internal physical parameters of the unknown image sensor. Furthermore, if the internal physical parameters such as the focal length and pixel spacing of the image sensor have been previously calibrated before use, according to the camera calibration theory, only the object space display coordinates and image space image coordinates of the three targets are determined.
- the imaging equations can be used to solve the external parameter matrix that meets the requirements.
- the external physical parameters determining these external parameters that is, the three-axis rotation angle ( ⁇ , ⁇ , ⁇ ) and the origin coordinates of the world coordinate system in the camera coordinate system can be further obtained. . . . , ⁇ .), thereby also determining the spatial attitude of the image sensor relative to the display screen, such an application would be used in the fourth embodiment that follows.
- an image sensor positioning technique was developed to determine the approximate location of the display screen to which the image sensor is directed.
- the function of this positioning technology is that when the visual pointing input device works, if the camera is close to the display screen, only the local area can be imaged, and the local area of the display screen corresponding to the camera needs to be determined first, and then the computer can be notified.
- the processor generates a feature target in this area.
- This patent proposes two methods for locating the location of the coarsely determined image sensor, which are described separately below.
- 3A and 3B illustrate a first positioning method. When the input device 10 is activated, a synchronization signal is sent to the computer, and the computer outputs a specially designed color coding pattern to the display screen 18.
- the color squares are arranged in a square map, and the color coding of the square area around the square is unique.
- the coded figure 30 in the figure shown in FIG. 3, 30 is a frame of 300, 302, 304, 306.
- the four color arrays are arranged in a grid, each color is given a different code, and the color coding of the 3x3 square area around each square in 30 is unique, so that the entire coding pattern corresponds to a coded lookup table, and thus,
- the approximate position of the display screen to which the camera is directed can be determined based on the color coding of the area extracted from the captured image. As shown in FIG.
- the image sensor 100 extracts the code 022302010 from the acquired local area image 32, so that the approximate position corresponding to the code can be found from the lookup table, so that the feature target can be further generated in the area.
- the coding pattern can take many forms, in addition to being color coded, it can also be encoded by the geometric content of the block; of course, it can also be designed as other geometric coded pictures, such as a ring coded picture.
- the color coding pattern can be translucently blended on the original display content, or it can be completely displayed in a very short time. After being captured by the image sensor, the original content is displayed.
- FIG. 3C Another method for determining the position of the image sensor is as shown in FIG. 3C, FIG. 3D, and FIG. 3E, which is different from the previous one-time output coding pattern, which can continuously output the sequence coding pattern in a very short time.
- the image sensor position is gradually determined coarsely and finely.
- 34 shown in FIG. 3C is a coarse resolution coding pattern of four colors of 300, 302, 304, and 306.
- the computer outputs a coarse resolution coding pattern 34 on the full screen of the display to roughly determine the image sensor.
- the area is as shown in FIG. 3D; then, the same coding pattern 34 is outputted in the determined local area to further determine the position of the image sensor, as shown in FIG. 3E; so that the image can be accurately determined by repeating the loop several times.
- the location of the sensor is not limited to determine the image sensor.
- the computer After the above image sensor positioning method determines the region in which it is located, the computer needs to set a number of targets in the region for image sensor calibration. For this reason, two methods are proposed to set the feature target.
- the first method is shown in FIG. 4.
- the computer processes the display content in the determined image acquisition area, and selects several features from the computer display content by using features such as color, edge, corner point, orientation, and surrounding environment information. Target, and record its characteristic information.
- the second feature target setting method can be used to work, that is, in the region
- the computer dynamically generates several feature targets. The method is illustrated in FIG. 5A and FIG. 5B.
- the dynamically generated target color can select a color that is not in the image area.
- the display points in the area are distributed in the RGB color coordinate system according to the color coordinate, and the 50 area in the figure is the image color.
- the blank area away from the existing color point in the coordinate body, such as the color of the 52 area is used as the color of the target point, so that it can be collected from the color image sensor.
- the feature target is easily extracted from the image; secondly, the shape of the target can be selected as a positive cross or oblique ten Intersections, or any other convenient form of identification, FIG. 5B generated a regular tetragonal arrangement of four crosshead target 12, of course, any shape may be generated to meet the needs of the target pattern.
- FIG. 5B generated a regular tetragonal arrangement of four crosshead target 12, of course, any shape may be generated to meet the needs of the target pattern.
- the first embodiment of the directional pointing mouse using the monocular camera calibration technology comprises the following steps: pointing the monocular image sensor to any position on the display screen, starting the image sensor operation, and the image sensor is wired or wireless.
- the computer determines the position of the image sensor pointing to the display screen:
- the computer outputs the positioning code pattern determining the pointing position of the image sensor on the display screen in a very short time, and the image sensor collects the image of the pointed area from the local area Extracting the included pattern code into the image, comparing it with the spatial position lookup table of the positioning code pattern, determining the approximate position of the image sensor pointing to the display screen; determining the feature target on the positioned display pointing area: by the computer Selecting a plurality of feature targets distributed in the image acquisition area of the image sensor according to the display content, or generating a plurality of feature targets distributed in the image collection area of the image sensor on the displayed display pointing area;
- the sensor collects the image of the pointing area containing the characteristic target point, extracts
- Camera imaging parameters a line connecting a certain fixed point on the imaging surface of the image sensor and the center of the lens as a virtual indication axis, and the image coordinates of the fixed point, that is, the image coordinates of the intersection of the virtual indication axis and the imaging surface, are calculated by using the solution
- the monocular camera imaging parameters calculate the corresponding display coordinates of the intersection point on the computer display screen; the mouse cursor or other image target is displayed by the computer at the pointing point of the virtual indication axis on the display screen, and the computer displays the cursor as The center re-determines the display pointing area, and re-determines the feature target in the pointing area; in a very short time interval, repeating the above steps with the pointing movement of the image sensor, so that the mouse cursor or other image target follows the virtual pointing axis Point to point motion on the display.
- the workflow of the first embodiment is designed as shown in FIG. 6 , and includes the following steps: after starting to point to the input device, in step 600, notifying the computer central processor to send a positioning code pattern to the screen;
- the image sensor of the input device collects an image, and the image processing circuit extracts the image code therefrom; in step 604, it is determined whether the image code matches the code map lookup table, finds the approximate position corresponding to the image sensor, and then proceeds to the next step, otherwise continues to repeat 600
- Step 602 In step 606a, the computer is notified to find or generate a feature target in the area; in step 608, an image is acquired by the image sensor, and each target coordinate is extracted from the acquired image according to the target feature; , judging whether the coordinates of each target point are correctly extracted, otherwise repeating step 608; in step 612, calculating the external parameters of the imaging model by using the display coordinates of the target point and the extracted image coordinates; in step 614a, using the calculated external Parameter, the coordinates of the center
- FIG. 7 is a basic system block diagram of a visual pointing type mouse input system using monocular camera calibration technology proposed in this patent, and 10, 30, and 18 are respectively three main devices of the system: a visual pointing input device, a computer host, and The functional block diagram of the display screen is not shown in the detailed details.
- the complete visual pointing input device includes an external input device 10 that is operated by the operator and an information receiving processing device 702 that is placed in the host computer.
- the external input device 10 for visual pointing includes three main functional components of the image sensor 100, the control function component 102 and the processing circuit 104, and the processing circuit 104 further includes a communication function module 704, an image processing module 706, and a control function module 708.
- the function module; the information receiving processing device 702 is responsible for communication interaction with the input device 10 and the computer operating system 700, and is embedded in the computer host in the form of software or hardware. As shown in FIG. 7, after the input device 10 starts working, the communication receiving module 702 activates the information receiving processing device 702, 702 to interact with the host operating system program 700, informing the operating system program 700 where to display the image content, and where The method generates a feature target, and simultaneously informs the input device of the feature information and the coordinate information of the target.
- the image processing module 706 of the input device extracts a target from the image acquired by the image sensor 100, performs an arithmetic operation, and transmits the resolved coordinates of the pointing point to the information receiving processing device 702 by the communication module 704, thereby notifying the computer operating system 700 , Display the image target such as the mouse cursor at the position of the pointing point.
- the control signal generated by control module 102 drive control module circuit 708 is also communicated by communication module 704 to information receiving processing device 702, which instructs the computer operating system to generate corresponding control operations.
- arithmetic functions such as image information processing may be flexibly arranged in the processing circuit 104, or may be disposed in the information receiving processing device 702 inserted in the computer.
- the glove type directional pointing type input device mainly includes an image sensor indicating part 100, a main control function part 102, a processing circuit 104, and an auxiliary control function part 106. These parts are fixed on gloves made of flexible materials such as nylon or plastic. The joints of the gloves can be properly opened.
- the parts are placed at appropriate positions on the fingers, which can not affect the bending action of the fingers, so that the typing can still be guaranteed. Waiting for the convenience of hand operation.
- the image sensor indicating unit 100 is used to point to a target such as a display screen.
- a target such as a display screen.
- the main control function component 102 has a function control key on the side, which is convenient for realizing mouse activation, left key, right key, page turning, etc., and its working schematic diagram is shown in FIG. 8B.
- the function control key can have various forms, which can be The push button 82a, which may also be the touch button 82b, as shown in Fig. 8E, may also be a combination of the two, for example, when the thumb touch function key activates the input device to operate, and when the thumb is pressed down, the left button function is issued.
- the auxiliary control function component 106 has a squeeze switch or a touch switch 84. As shown in FIG. 8F, the work is triggered by bending the little finger.
- the working schematic diagram is shown in FIG. 8C, and the right key function and the page turning function can be placed on the main control unit. It can also be placed on the auxiliary control unit.
- the processing circuit 104 can be flexibly arranged on the back of the hand or other positions, wherein an information processing module for performing image processing and data decoding using a digital processing chip such as a DSP or an FPGA, a wired or wireless communication module for communicating with the computer, and data Memory, etc. Further, an arithmetic function such as image information processing may be disposed in the information receiving processing device inserted in the computer.
- the above components can be flexibly designed for appearance and increase or decrease as needed, and can be operated in various possible ways.
- the above describes a glove type directional pointing input device.
- the letter processing circuit can be integrated into other components.
- it can also be well designed to integrate all the components into one, forming a finger cot and working on the index finger, as shown in Figure 9A. It can also be designed in various shapes such as a pen shape or a gun shape depending on the platform used.
- Figure 9B shows a pen-type visual input device in which all components are assembled in a pen-type housing.
- Figure 10 depicts a second embodiment of a directional pointing mouse that utilizes monocular camera calibration techniques.
- the second embodiment is similar to the first embodiment, and also relates to a visual pointing input method for controlling a graphical object such as a mouse cursor to be accurately displayed at an intersection of a virtual indicator axis of an image sensor pointing to a computer display screen.
- the dynamic display device such as a display screen is still used as the target setting area, but the second method does not locate the pointing position of the image sensor, but the computer directly sets the characteristic target map in a certain area of the display screen to determine the target.
- the operator actively points an image sensor such as a camera to the area, acquires a target image, and positions the mouse cursor.
- the target region is still moved following the orientation of the image sensor, keeping the target region at the imaged area of the image sensor.
- the second embodiment only lacks the link of image sensor positioning. Therefore, the second embodiment includes the following steps: starting the image sensor operation, and connecting the image sensor to the computer by wire or wirelessly; Determining the feature target on the display screen: The computer selects a plurality of feature targets distributed in the image acquisition area of the image sensor according to the display content, or generates thousands of distributions in the image acquisition area of the image sensor.
- Characteristic target point the monocular image sensor to the feature target area on the display screen, and collect the feature target Pointing to the area image, extracting the image coordinates of the target image according to the target feature; using the image coordinates of the acquired target point and its computer display coordinates, and calculating the imaging parameters of the monocular camera according to the monocular camera calibration technique;
- the line connecting a fixed point on the imaging surface and the center of the lens is used as a virtual indication axis.
- the image coordinates of the fixed point that is, the image coordinates of the intersection of the virtual indication axis and the imaging surface, are calculated by using the calculated monocular camera imaging parameters.
- the workflow of designing the second embodiment is shown in FIG. 10, and includes the following steps: After starting to point to the input device, directly enter step 606b, and notify the computer to find or generate a feature target in this area; The image is acquired by the image sensor, and the coordinates of each target point are extracted from the acquired image. In step 610, it is determined whether the coordinates of each target point are correctly extracted, otherwise the step 608 is repeated; in step 612, the display coordinates and extraction of the target point are utilized.
- the image coordinates of the imaging model are calculated by the coordinates of the image; in step 614a, the calculated imaging parameters are used, and the coordinates of the intersection of the corresponding optical axis and the display screen are calculated from the coordinates of the central image point; Display the mouse cursor or other target at the position of the optical axis pointing point; in step 618, reset the feature target and display it in the area centered on the optical axis; in step 620, determine if the input device is still in Working status, continue to shoot continuous images if still working Kind of operation, otherwise terminate the system work.
- a third embodiment relates to a visual pointing input method for controlling the movement of a graphical target such as a mouse cursor to accurately follow the motion of a virtual indicator axis of an image sensor in a target region.
- a graphical target such as a mouse cursor
- the size of the target region is determined, wherein a planar target coordinate system is set, and the target region contains a plurality of characteristic targets, and the target has a specific color, shape and the like, and is convenient for the image. Extracted, and the coordinates of the target in the target coordinate system are known.
- the image sensor is pointed to the area to obtain the target image.
- the monocular camera calibration technique is used to locate the coordinates of the virtual indicator axis of the image sensor in the target area in the target coordinate system, and then according to the target coordinate system and The unit scale relationship of the display coordinate system is used to find the corresponding mouse cursor display coordinates.
- the main features of the third embodiment are: pre-set a number of characteristic targets in a fixed target area, the coordinates of the target in the area are known, as shown in FIG. 11A to FIG. 11C, the area may be on a computer display screen.
- a certain fixed area (Fig. 11A) may also be the border of the display screen (Fig. 11B) or the target surface around the display screen (Fig. 11C).
- There are various options for the setting of the target which can be a display point with a specific color and shape generated by a computer on the display screen, or a physical point device such as an LED light source, which is disposed on a physical object such as a display frame. Colored reflective patches, etc.
- the third embodiment comprises the following steps: selecting a fixed target area as a pointing area of the collected image, the size of the area is determined, a plurality of characteristic target points are set in the area, and the coordinates of the target point in the area are known;
- the monocular image sensor is directed to the target area, and the image sensor is connected to the host computer by wire or wirelessly to activate the image sensor; the image of the pointing area containing the feature target is acquired, and the image coordinates of the target are extracted in the image according to the target feature;
- the monocular camera imaging parameters are calculated; the connection between a certain fixed point and the lens center on the imaging surface of the image sensor is used.
- Virtual indicator axis From the image coordinates of the fixed point, that is, the image coordinates of the intersection of the virtual indication axis and the imaging surface, using the calculated monocular camera imaging parameters, the coordinates of the corresponding intersection points in the target area are calculated; the calculated indication is The coordinate of the axis intersection point in the target coordinate system is multiplied by a proportional coefficient, which is obtained by dividing the actual size of the display screen by the actual size of the corresponding target area, and determining the display coordinates of the cursor on the display screen; The mouse cursor or other image target is displayed on the display according to the calculated display coordinates; in a very short time interval, with the pointing movement of the image sensor, repeat the above steps to make the mouse cursor or other image target follow the virtual indication The axis moves in the direction of the target area.
- FIG. 12 is a flow chart of the third embodiment, including the following steps: after starting to point to the input device, in step 606c, aligning the image sensor with a fixed target area set in advance; in step 608, collecting by the image sensor Image, extracting the coordinates of each target from the acquired image; in step 610, determining whether the coordinates of each target are correctly extracted, otherwise repeating step 608; in step 612, using the display coordinates of the target and the extracted image coordinates Outputting the imaging parameters of the imaging model; in step 614b, using the calculated imaging parameters, calculating the coordinates of the intersection of the corresponding indication axis and the target region from the coordinates of the central image point; in step 622, the calculated intersection coordinates Multiply by a scale factor to obtain the display coordinates of the cursor on the display screen; in step 616, the mouse cursor or other target is displayed by the computer at the position of the optical axis pointing point; in step 620, it is judged whether the input device is still in the working state. If you are
- Figure 13 depicts a fourth embodiment of a directional pointing mouse utilizing monocular camera calibration techniques.
- the fourth embodiment is a spatial motion attitude visual input method realized by the monocular camera calibration technology, which utilizes an image sensor that pre-calibrates internal physical parameters such as focal length and lateral and longitudinal pixel intervals, on the display screen or other
- the fixed feature target of the known mutual distance in the target region is subjected to continuous image acquisition, and the image coordinates of the target point are extracted, and the external imaging parameters are obtained according to the monocular camera calibration theory, and then the external parameters are determined.
- the external physical parameter that is, the three-axis rotation angle ( ⁇ , ⁇ , ⁇ ) and the origin coordinates ( ⁇ , ⁇ , ⁇ 0 ) of the world coordinate system in which the display screen is located in the camera coordinate system of the image sensor, thereby determining the image sensor
- the spatial orientation coordinates relative to the display In this way, by processing the sequence image acquired by the image sensor, the spatial attitude and the motion trajectory of the image sensor relative to the display screen can be obtained, and the specific spatial operation can be used to achieve certain specific operations.
- multiple visual input devices can be simultaneously used for multiple moving objects, and the spatial orientation of each imaging device can be calculated at the same time, so that an overall motion including multi-part motion can be formed.
- a small camera can be placed on each finger to align with the same set of targets on the screen for movement, and the active content of the hand can be identified according to the calculated spatial posture and motion trajectory of each imaging device, such as grasping, Rotate, pan, etc., to perform the corresponding control operations.
- the input device is initially located at the 90 position.
- an image of the feature target 12 in a certain fixed area of the display screen is acquired, and the external image is calculated from the image coordinates of the target point, the display coordinates, and the known internal parameters.
- the value of the imaging parameter, and then the orientation relationship of the input device relative to the target point that is, the parameters such as ( ⁇ , ⁇ , ⁇ ; ⁇ , ⁇ , ⁇ ::); when the input device moves to the position 92, Perform the same operation to calculate the azimuth parameter of the input device relative to the target point. According to the difference of the orientation relationship between the two positions, the spatial attitude and motion trajectory of the input device can be obtained.
- the fourth embodiment comprises the following steps: selecting a fixed target area as a pointing area of the acquired image, setting a plurality of characteristic target points in the area, and knowing the coordinates of the target point in the area; the focal length, image will be calibrated Meta interval
- the monocular image sensor of the internal physical parameter points to the target area, and the image sensor is connected to the computer by wire or wirelessly, starts the image sensor work, collects the image of the pointing area containing the characteristic target point, and extracts the target point in the image according to the target feature.
- Image coordinates using the image coordinates of the acquired target and its coordinates in the fixed area, according to the monocular camera calibration technique, the imaging parameters of the monocular camera are calculated; and then the imaging parameters of the calculated imaging model are used to obtain an image
- the spatial orientation coordinates in the system are connected to the spatial orientation coordinates of a series of image forming devices, and the spatial motion posture of the image sensor relative to the display screen is obtained.
- Figure 14 is a flow chart of the fourth embodiment, comprising the following steps: after starting the pointing input device, in step 606c, aligning the image sensor with a fixed target area set in advance; in step 608, by the image sensor The image is acquired, and the coordinates of each target point are extracted from the acquired image; in step 610, it is determined whether the coordinates of each target point are correctly extracted, otherwise the step 608 is repeated; in step 612, the display coordinates of the target point and the extracted image coordinates are used.
- step 624 Calculating the imaging parameters of the imaging model; in step 624, using the calculated imaging parameters, further obtaining the spatial orientation coordinates of the image sensor; in step 620, determining whether the input device is still in the working state, and continuing to shoot if still working The continuous image is processed in the same way, otherwise the system is terminated.
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Position Input By Displaying (AREA)
Description
说 明 书
利用单目摄像机标定技术的视觉指向型鼠标输入方法、 输入***和输入装置 发明领域
本发明有关计算机***输入技术与设备, 是一种利用单目摄像机标定技术, 实现摄像机 视觉精确定位并驱动鼠标光标或其他显示目标的方法和装置, 特别可以实现一种以单目图像 成像器件为指向输入装置的视觉指向型鼠标设备。 背景技术
伴随着计算机技术的发展, 图形用户界面的输入技术与设备经历了不断的发展。 最初, 用于驱动图形用户界面中的光标运动的鼠标装置是机械鼠标,一般具有轨迹球或方向键部件, 利用轨迹球相对于鼠标垫的运动或按压方向键来控制光标的移动。 随后, 光电鼠标逐渐取代 了机械鼠标, 当它在工作平面上移动时, 不断采集平面反射的图像, 利用图像处理技术对序 列图像进行处理, 提取运动方向与位移量, 从而驱动光标运动。 机械鼠标和光电鼠标一般都 需要在一定的工作平面上移动, 限制了其使用的方便性。
随着电子技术的发展, 触摸屏的使用已经越来越广泛, 触摸屏可以利用触摸方式来定位 输入位置, 驱动光标, 从而实现了图形用户界面的准确指向式输入。 然而, 触摸屏的使用需 要指示物与屏幕的直接接触, 使其应用受到一定制约, 并且造价成本较高, 在大面积屏幕的 制造上具有难度。
目前, 利用机器视觉技术来实现人机交互输入的技术越来越引起人们关注, 已经有很多 利用手势、 眼球、 头部等身体部位或其他控制物的空间运动来产生计算机输入命令的技术与 设备。 这种技术一般利用设置好的数码相机釆集目标的连续视频图像, 利用图像处理技术提 取出位置、 方向、 位移等参量, 再利用这些控制参量来驱动鼠标光标等显示目标, 或执行某 些特定控制动作。 利用这种技术可以实现指向型输入驱动鼠标光标, 如专利 CN2609054Y、 CN2602419Y、 CN101236468A所述,一般要事先放置并标定好若干数码摄像头等图像采集装 置, 其采集视场同时包括计算机显示屏和指示棒或手指等指示装置, 这样就能根据指示点在 显示屏中的相对位置来确定出鼠标光标的显示坐标。 这种方法需要占据一定空间, 并且经常 需要使用一些辅助装置, 如激光指示器、 选择性反射膜、 滤光偏振片等等, 导致整个***十 分复杂。
另外, 值得注意的一种视觉指向型输入技术是将数码摄像头绑定在手指或指示棒的指示 端, 当其指向显示屏时, 摄像头采集局部范围的显示屏图像内容, 然后将其输入计算机主机, 与正在显示的屏幕内容逐区扫描, 找到指向区域在显示屏中的位置; 或者利用指示端的移动 引起的采集图像内容的相对位移来控制光标的运动。这种方法虽然有很大的便利性和可行性, 但存在以下不足: 首先, 如果显示屏显示内容为同一种颜色, 如空白, 则导致无法进行图像 匹配或位移提取; 其次, 由于摄像头的旋转、 偏移、 距离变化, 引起所成图像的仿射投影变 形, 从而导致匹配困难; 另外, 由于摄像头旋转, 导致摄像头图像坐标系与显示屏的显示坐 标系方向不一致, 使得图像相对位移的方向与指示平台的运动方向不一致, 产生错误的鼠标 光标输入方向, 需要加入方向传感器来矫正; 此外, 由于摄像头距显示屏的成像距离变化, 导致图像大小与实际屏幕图像尺寸成比例变化, 因此在不同距离上的相同空间位移量产生的
图像位移量并不相同, 也需要加以校正。 因此, 仅靠这种技术并不能精确驱动鼠标光标, 往 往还需要加入方向传感器和距离传感器的配合, 导致结构复杂。
综上所述, 以上技术在制造结构简单、 简便易行、 造价便宜的精确指向型鼠标输入设备 上存在不足。 为此, 本专利提出了一种简便易行的利用机器视觉技术中的单目摄像机标定技 术来精确提取光标指向点坐标的视觉指向型鼠标输入方法, 可用于制造精确非接触视觉指向 型鼠标设备。 发明内容
本专利提出一种利用单目摄像机标定技术的视觉指向型鼠标输入方法, 用于控制鼠标光 标等图形目标准确跟随图像传感器的虚拟指示轴的指向运动, 所述方法包括如下步骤: 将单 目图像传感器指向一平面靶标, 该靶标内有确定的靶标坐标系, 其中设定若干特征靶点, 启 动图像传感器工作, 图像传感器通过有线或无线方式与计算机相连; 图像传感器采集靶标指 向区域包含的特征内容的图像, 在图像中提取出特征靶点的图像坐标; 利用采集的靶点图像 坐标和靶点在靶标坐标系中的坐标,根据单目摄像机标定技术,解算出单目摄像机成像参数; 由图像传感器的成像面上的一固定像点的图像坐标, 利用解算出的单目摄像机成像参数, 计 算出该像点对应的物点在靶标坐标系内的坐标, 即指示点在靶标坐标系内的坐标, 此时固定 像点、 镜头光心和指示物点构成一个虚拟指示轴; 由指示点在靶标坐标系中的坐标计算出鼠 标光标或其他图像目标在显示屏上的显示坐标, 并由计算机将光标显示在显示屏上。 在极短 的时间间隔内, 重复上述过程, 使鼠标光标或其他图像目标跟随图像传感器的指向而运动。
本专利提出一种利用单目摄像机标定技术的视觉指向型鼠标输入方法, 用于实现一种空 间运动姿态的视觉输入方法, 所述方法包括如下步骤: 将单目图像传感器指向一平面靶标, 该靶标内有确定的靶标坐标系, 其中设定若干特征靶点, 启动图像传感器工作, 图像传感器 通过有线或无线方式与计算机相连, 启动图像传感器工作; 图像传感器采集靶标指向区域包 含的特征内容的图像, 在图像中提取出特征靶点的图像坐标; 利用采集的靶点图像坐标和靶 点在靶标坐标系中的坐标, 根据单目摄像机标定技术, 解算出单目摄像机成像参数; 进而利 用计算出的摄像机成像参数, 求出图像传感器在靶标坐标系中的空间方位坐标, 空间方位坐 标包括三轴旋转角 (α,β,γ)和原点坐标 (X。, Y。, Z。)。 在极短的时间间隔内, 随着图像传感器的 指向运动, 重复上述过程, 求出图像传感器在不同位置时的空间方位坐标, 将一系列图像成 像器件的空间方位坐标连接起来, 就得到了图像传感器相对于显示屏的空间运动姿态。
本专利提出的视觉指向型鼠标输入方法, 其中所述将图像传感器指向一平面靶标, 具体 有不同方式: 可以将图像传感器直接指向一个事先己经设定好的靶标, 也就是说, 已经存在 一个靶标, 然后再将图像传感器指向这个靶标; 也可以先确定图像传感器指向的位置, 然后 在其指向区域再设置相应靶标, 这种方式对于靶标可以设定在动态显示设备上的情况较为适 用, 例如, 当图像传感器指向计算机显示屏时, 首先由计算机判定图像传感器指向显示屏的 大致区域, 然后再由计算机在图像传感器的显示屏指向区域动态确定一个靶标。
当需要将靶标设置在显示屏等动态显示设备上时, 本专利提出两种确定图像传感器指向 显示屏的位置的方法。 第一种方法包括如下步骤: 将图像传感器指向显示屏的目标区域采集 图像, 该装置通过有线或无线通讯方式与计算机主机相连, 显示屏与计算机主机相连接; 启 动图像传感器工作, 通知计算机主机在极短的时间内在显示屏上输出由若干不同颜色或图形
内容的特征块排列组成的编码图案, 每种颜色或图形内容被编成不同号码, 每个特征块附近 的一定范围内的所有特征块组成的编码在整个编码图中是唯一的, 整个编码图的所有特征块 的区域编码构成一个定位査找表, 特别的, 可以在在显示屏上输出由若干不同颜色或图形内 容的矩形特征块排列组成的编码图案,每个矩形特征块附近的 nX n范围内的特征块组成的编 码在编码图中是唯一的; 图像传感器采集所指向区域的编码图像,从中提取局部图案的编码, 将其与编码图的空间位置査找表进行比对, 确定图像传感器指向显示屏的大致位置。
本专利提出的第二种确定图像传感器指向显示屏的位置的方法, 包括如下步骤: 将图像 传感器指向显示屏的目标区域采集图像,该装置通过有线或无线通讯方式与计算机主机相连, 显示屏与计算机主机相连接; 启动图像传感器工作, 首先在显示屏上输出由若干不同颜色或 图形内容的特征块排列组成的粗分辨率编码图案, 每种颜色或图形内容被编成不同号码, 图 像传感器采集指向区域图像, 判断自身所对的特征块位置; 然后, 计算机主机在已确定的大 特征块范围内再次输出同样的编码图案, 其大小为该特征块的大小, 图像传感器采集指向区 域图像, 进一步判断自身所对的小特征块的位置; 如此由大及小地快速循环执行此操作, 最 终确定出图像传感器所指向的显示屏的位置。
本专利提出的视觉指向型鼠标输入方法, 其中所述的平面靶标, 具体如下特征: 靶标区 域的大小确定, 该区域内包含若干特征靶点, 靶点具有特定的颜色、 形状等特征, 便于从图 像中提取, 并且已知靶点在该靶标区域的靶标坐标系中的坐标。
本专利提出的视觉指向型鼠标输入方法, 其中所述的平面靶标, 可以是一个固定的平面 靶标, 可以选取显示屏的边框作为靶标区域, 特征靶点设置在边框上, 各点之间的距离已知; 可以选取显示屏周边的一个大小确定的平面作为靶标区域, 特征靶点设置在该区域内, 各点 之间的距离己知; 可以由计算机在显示屏上选取某局部固定区域作为靶标区域, 其大小由该 区域的计算机显示坐标确定, 并且由计算机在显示屏靶标区域内确定若干特征点作为特征靶 占。
本专利提出的视觉指向型鼠标输入方法, 其中所述的平面靶标, 可以是一个动态平面靶 标, 该靶标可由计算机在显示屏上动态生成, 靶标的生成位置始终追随图像传感器的指向, 靶标区域的范围可根据图像传感器的成像距离远近来调整, 其大小由该区域的计算机显示坐 标确定, 并且由计算机在显示屏靶标区域内确定若干特征点作为特征靶点。
本专利提出的视觉指向型鼠标输入方法, 所述的由计算机在显示屏靶标区域确定特征靶 点, 是指: 计算机对显示屏上包含靶标区域的一定范围内的显示内容进行处理, 利用颜色、 边缘、 角点、 方位、 周边环境信息等特征, 从计算机的显示内容中选择出若干特征靶点, 由 这些特征靶点界定靶标区域范围, 并记录其特征信息。
本专利提出的视觉指向型鼠标输入方法, 所述的由计算机在显示屏靶标区域确定特征靶 点,包括如下步骤:计算机对显示屏上包含靶标区域的一定范围内的显示内容颜色进行统计, 选择显示内容中没有的、 且与存在的颜色差异较大的颜色作为生成的特征靶点的颜色; 由计 算机在显示屏上的靶标区域内, 以选择的颜色额外生成一定的显示内容, 生成的显示内容中 包含交叉点、 角点、 中心点等特征, 可以根据这些特征选择出若干特征靶点, 由这些特征靶 点界定靶标区域范围。
本专利提出的视觉指向型鼠标输入方法, 所述若干特征靶点, 是指: 对于使用事先没有 标定出焦距、 像元间隔等内部物理参量的图像传感器的视觉指向型鼠标输入方法, 需要在显
示屏上确定至少 4个特征靶点。
本专利提出的视觉指向型鼠标输入方法, 所述若干特征靶点, 是指: 对于使用事先已经 标定出焦距、 像元间隔等内部物理参量的图像传感器的视觉指向型鼠标输入方法, 需要在显 示屏上确定至少 3个特征靶点。
本专利提出的视觉指向型鼠标输入方法, 所述的解算单目摄像机成像参数, 是指: 利用 采集的靶点图像坐标和靶点的靶标坐标系坐标, 根据单目摄像机标定技术, 解算出图像传感 器的单目摄像机外部成像参数。
本专利提出的视觉指向型鼠标输入方法, 所述的解算单目摄像机成像参数, 是指: 利用 采集的靶点图像坐标和靶点的靶标坐标系坐标, 根据单目摄像机标定技术, 解算出图像传感 器的单目摄像机内部成像参数和外部成像参数。
本专利提出的视觉指向型鼠标输入方法, 所述的图像传感器成像面上的一固定像点, 是 指- 该像点可以是成像面上的任意一像点, 该像点与成像镜头光心的连线构成一个虚拟指示 轴, 该像点对应的物点即为虚拟指示轴的指向点。
本专利提出的视觉指向型鼠标输入方法, 所述的图像传感器成像面上的一固定像点, 是 指- 该像点可以是成像面上的中心像点, 该像点与成像镜头光心的连线, 即成像***的光轴, 构成一个虚拟指示轴, 该中心像点对应的物点即为虚拟 示轴的指向点。
本专利提出的视觉指向型鼠标输入方法, 所述的由指示点在靶标坐标系中的坐标计算鼠 标光标的显示坐标,是指: 当靶标坐标系的单位长度与计算机显示屏的像元间隔尺寸相同时, 则已经计算出的靶点在靶标坐标系中的坐标就是显示坐标; 当靶标坐标系的单位长度与计算 机显示屏的像元间隔尺寸不同时, 则需要将已计算出的靶点在靶标坐标系中的坐标乘以一个 比例系数, 得到显示坐标, 该比例系数由显示屏的像元间隔尺寸除以靶标坐标系的单位长度 获得。
本专利提出一种利用单目摄像机标定技术的视觉指向型鼠标输入***, 其包括: 计算机 主机, 和与之相连的显示屏; 包含若干特征靶点的平面靶标, 各靶点之间的距离已知, 各靶 点具有确定的特征, 便于从图像中提取靶点; 单目图像传感器, 通过处理电路与计算机主机 相连, 单目图像传感器被用作指向装置, 将其指向显示屏的目标区域采集图像, 以图像传感 器成像面上的某固定点和镜头中心的连线作为虚拟指示轴; 控制功能部件, 布置各种控制功 能按键, 用于产生***触发、 左键、 右键、 翻页、 移动等控制功能, 通过处理电路与计算机 主机相连; ***处理电路, 与图像传感器和控制功能部件相连, 同时通过有线或无线的方式 与计算机主机相连; 视觉指向型鼠标信息接收处理装置, 安插在计算机主机中, 与计算机操 作***和处理电路通讯交互。 ***的处理电路的功能包括: 对采集的图像进行处理, 完成图 像传感器的指向定位、 特征靶点提取、 单目摄像机成像模型的参数计算、 光标指向点的显示 坐标计算等功能; 产生***触发、 左键、 右键、 翻页、 移动等控制功能信号; 与计算机主机 通过有线或无线的方式进行通讯交互, 传递图像、 特征信息、 运算结果、 控制信号等信息。 ***的信息接收处理装置的功能包括: 接收由处理电路发送的图像、 运算结果等信息; 对采 集的图像进行处理, 完成图像传感器的指向定位、 特征靶点提取、 单目摄像机成像模型的参 数计算、光标指向点的显示坐标计算等功能; 接收由处理电路产生的***触发、左键、右键、 翻页、 移动等控制功能信号; 向视觉指向型鼠标输入***处理电路发送靶点的特征信息、 坐 标信息; 向计算机操作***输出计算得到的光标坐标信息。
本专利提出一种利用单目摄像机标定技术的视觉指向型鼠标输入***, 其包括: 计算机 主机, 和与之相连的显示屏; 包含若干特征靶点的平面靶标, 各靶点之间的距离已知, 各靶 点具有确定的特征, 便于从图像中提取靶点; 单目图像传感器, 通过信息接收处理装置与计 算机主机相连, 单目图像传感器被用作指向装置, 将其指向显示屏的目标区域采集图像, 以 图像传感器成像面上的某固定点和镜头中心的连线作为虚拟指示轴; 控制功能部件, 布置各 种控制功能按键, 用于产生***触发、 左键、 右键、 翻页、 移动等控制功能, 通过信息接收 处理装置与计算机主机相连; 视觉指向型鼠标信息接收处理装置, 安插在计算机主机中, 与 计算机操作***和图像传感器及控制功能部件进行通讯交互, 其功能包括: 接收单目图像传 感器发送的图像信息; 接收由控制功能部件产生的***触发、 左键、 右键、 翻页、 移动等控 制功能信号; 对采集的图像进行处理, 完成图像传感器的指向定位、 特征靶点提取、 单目摄 像机成像模型的参数计算、 光标指向点的显示坐标计算等功能; 向计算机操作***输出计算 得到的光标坐标信息。
本专利提出一种手套式视觉指向型鼠标输入装置, 其包括: 计算机主机, 和与之相连的 显示屏; 包含若干特征靶点的平面靶标, 各靶点之间的距离已知, 各靶点具有确定的特征, 便于从图像中提取靶点; 安装有单目图像传感器的指向指套, 用于指向显示屏的目标区域采 集图像, 通过处理电路与计算机主机相连, 以图像传感器成像面上的某固定点和镜头中心的 连线作为虚拟指示轴; 控制功能键指套, 包括若干按键、触摸键或压力开关, 由拇指来控制, 用于产生***触发、左键、右键、翻页、移动等控制信号, 通过处理电路与计算机主机相连; 辅助功能键指套, 可以包含翻页等功能控制键, 通过弯曲手指, 由手指自身触发开关控制, 通过处理电路与计算机主机相连, 该辅助功能键指套根据使用情况选用; 处理电路, 与图像 传感器和控制功能部件相连, 同时通过有线或无线的方式与计算机主机相连; 视觉指向型鼠 标信息接收处理装置, 安插在计算机主机中, 与计算机操作***和处理电路通讯交互。 视觉 指向型鼠标输入装置的处理电路的功能包括: 对采集的图像进行处理, 完成图像传感器的指 向定位、 特征靶点提取、 单目摄像机成像模型的参数计算、 光标指向点的显示坐标计算等功 能; 产生***触发、 左键、 右键、 翻页、 移动等控制功能信号; 与计算机主机通过有线或无 线的方式进行通讯交互, 传递图像、 特征信息、 运算结果、 控制信号等信息。 视觉指向型鼠 标输入装置的信息接收处理装置的功能包括:接收由处理电路发送的图像、运算结果等信息; 对采集的图像进行处理, 完成图像传感器的指向定位、 特征靶点提取、 单目摄像机成像模型 的参数计算、 光标指向点的显示坐标计算等功能; 接收由处理电路产生的***触发、 左键、 右键、 翻页、 移动等控制功能信号; 向视觉指向型鼠标输入***处理电路发送靶点的特征信 息、 坐标信息; 向计算机操作***输出计算得到的光标坐标信息。
本专利提出一种手套式视觉指向型鼠标输入装置, 其包括: 计算机主机, 和与之相连的 显示屏; 包含若干特征靶点的平面靶标, 各靶点之间的距离已知, 各靶点具有确定的特征, 便于从图像中提取靶点; 安装有单目图像传感器的指向指套, 用于指向显示屏的目标区域采 集图像, 与计算机主机相连, 以图像传感器成像面上的某固定点和镜头中心的连线作为虚拟 指示轴; 控制功能键指套, 包括若干按键、 触摸键或压力开关, 由拇指来控制, 用于产生系 统触发、 左键、 右键、 翻页、 移动等控制信号, 与计算机主机相连; 辅助功能键指套, 可以 包含翻页等功能控制键, 通过弯曲手指, 由手指自身触发开关控制, 与计算机主机相连, 该 辅助功能键指套根据使用情况选用; 视觉指向型鼠标信息接收处理装置, 安插在计算机主机
中, 与计算机操作***和图像传感器及控制功能部件进行通讯交互, 其功能包括: 接收图像 传感器发送的图像信息; 接收由控制功能部件产生的***触发、 左键、 右键、 翻页、 移动等 控制功能信号; 对采集的图像进行处理, 完成图像传感器的指向定位、 特征靶点提取、 单目 摄像机成像模型的参数计算、 光标指向点的显示坐标计算等功能; 向计算机操作***输出计 算得到的光标坐标信息。
本专利提出一种指套式视觉指向型鼠标输入装置, 其包括: 计算机主机, 和与之相连的 显示屏; 包含若干特征靶点的平面靶标, 各靶点之间的距离已知, 各靶点具有确定的特征, 便于从图像中提取靶点; 视觉指向型鼠标输入指套, 将单目图像传感器、 处理电路、 控制功 能键集成在一个指套上, 戴于食指或其他手指上, 由拇指实现功能控制, 使用时将位于指套 前端的图像传感器指向显示屏的目标区域采集图像, 图像传感器通过处理电路以有线或无线 方式与计算机主机相连, 图像传感器以图像传感器成像面上的某固定点和镜头中心的连线作 为虚拟指示轴, 指套的控制功能键包括若干按键、 触摸键或压力开关, 由拇指来控制, 用于 产生***触发、左键、 右键、翻页、移动等控制功能信号, 通过处理电路与计算机主机相连; 信息接收处理装置, 安插在计算机主机中, 与计算机操作***和鼠标输入装置的处理电路通 讯交互。 鼠标输入指套中的处理电路与图像传感器和控制功能部件相连, 同时通过有线或无 线的方式与计算机主机相连, 其功能包括: 对釆集的图像进行处理, 完成图像传感器的指向 定位、特征靶点提取、单目摄像机成像模型的参数计算、光标指向点的显示坐标计算等功能; 产生***触发、 左键、 右键、 翻页、 移动等控制功能信号; 与计算机主机通过有线或无线的 方式进行通讯交互, 传递图像、 特征信息、 运算结果、 控制信号等信息。 鼠标输入装置的信 息接收处理装置的功能包括:接收由鼠标输入装置的处理电路发送的图像、运算结果等信息; 接收由处理电路产生的***触发、 左键、 右键、 翻页、 移动等控制功能信号; 对采集的图像 进行处理, 完成图像传感器的指向定位、 特征靶点提取、 单目摄像机成像模型的参数计算、 光标指向点的显示坐标计算等功能; 向处理电路发送靶点的特征信息、 坐标信息; 向计算机 操作***输出计算得到的光标坐标信息。
本专利提出一种指套式视觉指向型鼠标输入装置, 其包括: 计算机主机, 和与之相连的 显示屏; 包含若干特征靶点的平面靶标, 各靶点之间的距离已知, 各靶点具有确定的特征, 便于从图像中提取靶点; 视觉指向型鼠标输入指套, 将单目图像传感器、 控制功能键集成在 一个指套上, 戴于食指或其他手指上, 由拇指实现功能控制, 使用时将位于指套前端的图像 传感器指向显示屏的目标区域采集图像, 图像传感器以有线或无线方式与计算机主机相连, 以图像传感器成像面上的某固定点和镜头中心的连线作为虚拟指示轴, 指套的控制功能键包 括若千按键、 触摸键或压力开关, 由拇指来控制, 用于产生***触发、 左键、 右键、 翻页、 移动等控制功能信号, 以有线或无线方式与计算机主机相连; 信息接收处理装置, 安插在计 算机主机中, 与计算机操作***和图像传感器及控制功能部件进行通讯交互, 其功能包括- 接收图像传感器发送的图像信息, 接收由控制功能部件产生的***触发、左键、右键、翻页、 移动等控制功能信号, 对采集的图像进行处理, 完成图像传感器的指向定位、特征靶点提取、 单目摄像机成像模型的参数计算、 光标指向点的显示坐标计算等功能, 向计算机操作***输 出计算得到的光标坐标信息。
本专利提出一种视觉指向型鼠标应用程序, 驻留在计算机主机中, 与计算机操作***和 视觉指向型鼠标输入***进行通讯交互, 包括如下功能: 控制功能程序, 接收由视觉指向型
鼠标输入***的控制功能部件产生的***触发、 左键、 右键、 翻页、 移动等控制功能信号; 图像接收处理程序, 接收由视觉指向型鼠标输入***的图像传感器发送的图像信息; 图像传 感器定位程序, 确定图像传感器指向显示屏的位置: 在接到视觉指向型鼠标输入***的工作 触发信号后, 通知计算机在极短的时间内, 在显示屏上输出确定图像传感器指向位置的定位 编码图案, 对图像传感器采集的指向区域的图像进行处理, 从局部区域的图像中提取出包含 的图案编码, 将其与定位编码图案的空间位置查找表进行比对, 确定图像传感器指向显示屏 的大致位置; 特征靶点产生程序, 在显示屏上确定特征靶点: 在显示屏上的靶点显示区域, 根据显示内容选择若干分布在图像传感器的图像采集区域内的特征靶点, 或者生成若干分布 在图像传感器的图像采集区域内的特征靶点; 靶点提取程序, 对图像传感器采集的含有特征 靶点的指向区域图像进行处理, 根据靶点特征在图像中提取靶点的图像坐标; 摄像机成像参 数计算程序, 利用采集的靶点的图像坐标和其计算机显示坐标, 根据单目摄像机标定技术, 解算出单目摄像机成像参数; 显示坐标计算程序, 以图像传感器成像面上的某固定点和镜头 中心的连线作为虚拟指示轴, 由该固定点的图像坐标, 即虚拟指示轴与成像面的交点的图像 坐标, 利用解算出的单目摄像机成像参数, 计算出其对应的在计算机显示屏上的交点的显示 坐标; 光标显示程序, 通知计算机将鼠标光标或其他图像目标显示在虚拟指示轴在显示屏上 的指向点处, 并以显示光标为中心重新确定显示屏指向区域, 在指向区域内重新确定特征靶 点。 附图说明
图 1所示是利用单目摄像机标定技术的视觉指向型鼠标的第一种实施方式的概视图; 图 2A所示是单目摄像机成像的针孔模型示意图;
图 2B所示是利用单目摄像机标定技术对显示屏平面进行标定的示意图;
图 3A所示是由不同颜色构成的编码定位示意图;
图 3B所示是根据显示屏局部区域编码来进行指向定位的示意图;
图 3C所示是由不同颜色构成的粗分辨率编码定位示意图;
图 3D所示是第一次定位图像传感器指向位置的示意图;
图 3E所示是第二次定位图像传感器指向位置的示意图;
图 4所示是利用显示内容中的颜色、 边缘、 角点等特征选择特征靶点的示意图; 图 5A所示是色彩空间中靶点设置区域中的显示点颜色分布和靶点颜色选择区域示意图; 图 5B所示是生成正四方排列的四个十字靶点的示意图;
图 6所示是利用单目摄像机标定技术的视觉指向型鼠标的第一种实施方式的工作流程 图;
图 7所示是利用单目摄像机标定技术的视觉指向型鼠标输入***的基本***原理框图; 图 8A—图 8C所示是手套式视觉指向输入装置的穿戴方法和使用方法示意图; 图 8D所示是单目图像传感器指示部件示意图;
图 8E所示是主控制功能部件示意图;
图 8F所示是辅助控制功能部件示意图;
图 9A所示是将所有部件整合于一体的指套式视觉指向输入装置示意图;
图 9B所示是笔式视觉指向输入装置示意图;
图 10所示是利用单目摄像机标定技术的视觉指向型鼠标的第二种实施方式的工作流程 图;
图 11A至图 11C所示是利用单目摄像机标定技术的视觉指向型鼠标的第三种实施方式示 意图;
图 12所示是利用单目摄像机标定技术的视觉指向型鼠标的第三种实施方式的工作流程 图。
图 13所示是利用单目摄像机标定技术的视觉指向型鼠标的第四种实施方式示意图。
图 14所示是利用单目摄像机标定技术的视觉指向型鼠标的第四种实施方式的工作流程 图。 具体实施方式
本发明将结合附图, 对利用单目摄像机标定技术的视觉指向型鼠标实施方式进行详细描 述, 各附图中的相同部件使用相同标号。
[实施方式 1]
图 1至图 9B描述的是利用单目摄像机标定技术的视觉指向型鼠标的第一种实施方式。 第一种实施方式涉及一种控制鼠标光标等图形目标准确显示在一个图像传感器的虚拟指示轴 指向计算机显示屏的交点处的视觉指向输入方法, 在这种方法中, 显示屏等动态显示设备作 为靶标的设置区域, 首先由计算机采用特定方法定位图像传感器指向显示屏的位置, 然后再 由计算机将特征靶点设置在显示屏上, 从而界定靶标区域, 并且靶标区域可以随着图像传感 器的指向而运动, 始终保持靶标区域在图像传感器的指向成像区域。
图 1所示是第一种实施方式的概视图,图中的视觉指向输入装置 10以手套形式穿戴在手 上, 其核心工作部件是穿在食指上的小型单目图像传感器 100, 用于指向靶标区域采集图像, 在第一种实施方式中靶标区域由计算机设置在显示屏上, 图像传感器可以根据成像距离自动 变焦, 以得到清晰图像(当成像距离远大于图像传感器焦距时, 使用定焦图像传感器就可以 始终得到较清晰的图像); 输入装置还包括穿戴在中指或其他手指上的控制功能部件 102, 上 面布置控制功能键, 通过拇指控制, 用来实现***触发、 左键、 右键和页面滚动等功能; 以 及处理电路 104, 在图 1中其被放置于手背上, 也可以整合进图像传感器 100或控制功能部 件 102中, 处理电路与图像传感器 100和控制功能部件 102相连, 并且通过有线或无线的方 式与计算机主机相连。 处理电路 104的功能主要包括: 对采集的图像进行处理, 完成图像传 感器的指向定位、特征靶点提取、成像模型的参数计算、光标指向点的显示坐标计算等功能; 产生***触发、 左键、 右键、 页面滚动等控制功能信号; 与计算机主机通过有线或无线的方 式进行通讯交互, 传递图像、 特征信息、 运算结果、 控制信号等信息。 该视觉指向输入装置 10的具体结构还将在后面进一步阐述。
如图 1中所示, 当穿在食指上的图像传感器 100对准显示屏 18的某一区域拍摄时,计算 机首先运用某种定位技术确定图像传感器所对应显示屏的大致位置, 关于这种定位技术的介 绍将在后面进行阐述。 然后, 计算机在该区域根据显示内容的颜色、 边缘、 角点等特征, 在 其中选择若干特征靶点, 或者在该区域生成若干特征靶点 12, 这些特征靶点构成靶标区域, 所需靶点的数目与采用的具体标定技术有关。 同时, 计算机将这些靶点的特征信息通过有线 或无线通讯方式传送给处理电路 104, 使输入装置可以凭这些特征从采集的图像中提取出特
征靶点。 进而可以根据这些特征靶点的计算机显示坐标和其图像坐标, 利用单目摄像机标定 技术计算出成像模型的有关成像参数,然后再利用这些成像参数计算出虚拟指示轴 14与显示 屏的交点 16相对于靶点的相对坐标。最后, 根据特征靶点在显示屏上的显示坐标, 计算出交 点 16在显示屏上的实际显示坐标,从而能够精确显示鼠标光标或其他图形目标。在极短的时 间内, 不断重复以上过程, 就可以实现光标精确跟随图像传感器的指向而运动, 例如, 对于 帧频为 15 S的图像传感器, 两幅图像之间的时间间隔约为 67ms, 而对于帧频为 30 S的图 像传感器, 两幅图像之间的时间间隔约为 33ms, 帧频越高则时间间隔越短。 图 1中显示了计 算机对使用一个指向输入装置进行处理的情况, 它还可以满足同时使用多个类似指向输入装 置的情况。
为了更好地理解本专利提出的单目指向型视觉鼠标输入技术如何精确获取虚拟指示轴指 向显示屏的指示点的显示坐标, 下面对所用到的单目摄像机标定技术进行图解说明。
图 2A所示是单目摄像机成像的针孔模型, 显示了对空间任意一个物点成像的示意图, 物点 20a经过成像透镜 22的中心投影在像平面 24上成为像点 20b, 成像器件可以是 CCD器 件、 CMOS器件, 或其他数字成像器件。 在成像过程的描述中, 涉及到物点所在的世界坐标 系 Ow-XwYwZw、摄像机坐标系 Oc-XcYcZc和像平面所在的图像坐标系 U-0-V三个坐标系, 图 中分别为 200、 202、 204。 整个成像模型关系, 即物点在世界坐标系中的坐标 (Xw, Yw, Zw)与 其在像平面上的像点的图像坐标系坐标 V)之间的表达关系,可以由世界坐标系 Ow-XwYwZw 在摄像机坐标系 Oe-XcYcZc中的三轴旋转角 (α, β, γ)和其原点在摄像机坐标系中的坐标 (Χο, Υ。, Ζ。)等 6个外部物理参量, 以及摄像机焦距 成像器件横向像元间隔 dx、 纵向像元间隔 dy等 3个内部物理参量, 共 9个物理参量决定的参数矩阵来描述:
f 0 "o 0
0 f
v0 0
式中 是由世界坐标系在摄像机坐标系中的三轴旋转角 (a, β, γ)和原点坐标 (Χο, Υ。, Ζ0
的外部参数矩阵, M2是由摄像机的内部物理参量: 焦距 f、 横向像元间隔 dx、 纵向像元间隔 dy决定的内部参数矩阵, (uo, vo)是摄像机光轴与成像面的交点在图像坐标系中的坐标。 将 Zc 消去, 可以得到两个只包含世界坐标系中的点坐标 (XW,YW, ZW)和像平面坐标 (u, v)的方程。根 据摄像机标定理论, 在一般情况下, 如果已知 6个空间点的世界坐标系坐标和其图像坐标系 坐标, 就可以解算出参数矩阵 Mi、 M2的所有参数, 从而可以由任意物点的世界坐标系坐标 (Xw, Yw, Zw)计算出其在图像坐标系中的坐标 (u, v),也可以由任意像点坐标 (u, V)反算出其在世 界坐标系中的坐标 (Xw, Yw, Zw)。
本专利应用于计算机显示屏的视觉指向性输入, 所有目标点都位于一个显示屏平面上, 因此可以用显示屏平面作为世界坐标系的 XwOwYw平面, 取显示屏的像元间隔作为世界坐标 系 XwOwYw平面的坐标单位长度, 即 XwOwYw平面坐标与计算机显示坐标单位一致, 同时取 与平面垂直的方向为 Zw轴, 如图 2B所示, 图中 18为位于 XwOwYw平面上的显示屏, 16a、 16b 即为摄像机光轴与显示屏的交点和对应的成像面上的中心像点。 由于用显示屏平面作为 XwOwYw平面, 因此显示屏上所有点的∑ =0, 此时计算工作得到简化。 并且由于已知特征靶 点的显示屏显示坐标, 因此只需求出指示轴指示点相对于靶点的相对坐标, 就可以根据靶点 的显示坐标进而求出指示点在显示屏中的显示坐标。 针对以上特定的应用条件, 使得所需的 标定工作相应简化, 根据摄像机标定理论, 即使在未知图像传感器的内部物理参数情况下, 只需解算由显示平面上的四个靶点的物空间显示坐标和像空间图像坐标确定的成像方程组, 就可以解算出满足要求的外部参数矩阵。 进而由标定得到的所需成像参数, 可以求出由成像 面上的某固定点与透镜中心连线决定的虚拟指示轴与计算机显示屏交点的显示坐标, 从而精 确定位鼠标光标或其他显示目标, 在实际应用中, 为了方便起见可以把图像传感器光轴作为 指示轴。 在图 2B中, 12a为在显示屏中设定的特征靶点, 12b为其在成像面中的像点, 根据 它们的显示坐标和图像坐标, 可以标定出所需的成像参数, 进而可以由光轴与成像面的交点 16b的图像坐标计算出其对应的显示屏中的交点 16a相对于靶点的坐标, 最终可以得到其准 确的计算机显示坐标。
以上所述的应用方式, 可以在未知图像传感器的内部物理参数情况下直接使用。 更进一 步, 如果在使用前己经事先标定出图像传感器的焦距、 像元间距等内部物理参数, 根据摄像 机标定理论, 只需解算由三个靶点的物空间显示坐标和像空间图像坐标确定的成像方程组, 就可以解算出满足要求的外部参数矩阵。另外,在求出外部参数矩阵 后,还可以进一步求 出决定这些外部参数的外部物理参量, 即世界坐标系在摄像机坐标系中的三轴旋转角 (α, β, γ) 和原点坐标 (Χ。, Υ。, Ζ。), 从而还可以确定图像传感器相对于显示屏的空间姿态, 这种应用在 后面的第四种实施方式中会用到。
如前所述, 在专利的第一种实施方式中, 开发了图像传感器定位技术来确定图像传感器 所指向显示屏的大致位置。 这种定位技术的作用在于, 当启动视觉指向输入装置工作时, 如 果摄像机距离显示屏比较近, 只能对局部区域成像, 需要先判断出摄像机所对应的显示屏局 部区域位置, 然后才能通知计算机处理器在该区域产生特征靶点。 本专利提出了两种粗判定 图像传感器位置的定位方法, 下面分别进行介绍。 图 3Α、 图 3Β描述的是第一种定位方法, 当启动输入装置 10工作时, 向计算机发出同步信号, 计算机向显示屏 18输出一幅经过特殊 设计的彩色编码图案, 这是一幅由若干颜色方块排列成的方格图, 每个方块周围 ηχη的方块 区的颜色编码是唯一的, 例如图 3Α所示中的编码图 30, 30是一幅由 300、 302、 304、 306
四种颜色排列成的方格图, 每种颜色被赋予不同编码, 30中每个方块周围 3x3的方块区的颜 色编码是唯一的, 这样整幅编码图案就对应一个编码査找表, 于是, 就可以根据从采集图像 中提取的区域色彩编码判断出摄像机所对的显示屏大致位置。如图 3B所示, 图像传感器 100 从采集的局部区域图像 32中提取出编码 023302010, 这样就可以从査找表中找到该编码对应 的大致位置, 从而进一步可以在该区域生成特征靶点。 编码图案可以采用多种形式, 除了可 以按颜色编码外, 还可以由方块中的几何内容来编码; 当然也可以设计成其他几何形状编码 图, 如环形编码图。 彩色编码图案可以半透明融合在原来的显示内容上, 也可以在极短时间 内完全显示, 由图像传感器采集后, 再显示原来内容。
另外一种判定图像传感器位置的方法如图 3C、 图 3D、 图 3E所示, 不同于前一种一次性 输出编码图的方式, 这种方法可以在极短时间内连续输出序列编码图案, 由粗及精地逐步确 定图像传感器位置。 举例来说, 图 3C所示的 34为 300、 302、 304、 306四种颜色的粗分辨 率编码图案, 首先, 计算机在显示器全屏输出一幅粗分辨率的编码图案 34, 大致确定出图像 传感器所在区域, 如图 3D所示; 然后, 在确定的局部区域再输出同样编码图案 34, 进一步 确定图像传感器所在位置,如图 3E所示; 如此快速循环几次, 就可以比较精确地确定出图像 传感器所在位置。
由以上图像传感器定位方法确定了其所处区域后,就需要计算机在该区域设定若干靶点, 供图像传感器釆集标定使用, 为此本专利提出了两种方法来设定特征靶点。 第一种方法如图 4所示, 计算机对已确定的图像采集区域中的显示内容进行处理, 利用颜色、 边缘、 角点、 方位、周边环境信息等特征, 从计算机显示内容中选择出若干特征靶点, 并记录其特征信息。 当在该区域难以或不能选择出适当的特征靶点,如对于空白区域或同色区域,如图 1中所示, 则可以采用第二种特征靶点设定方法来工作, 即在该区域由计算机动态生成若干特征靶点, 图 5A、 图 5B对该方法进行说明, 为了便于识别提取靶点, 可以在其颜色和形状上赋予一定 特征, 本专利提出如下技术生成便于识别提取的靶点: 首先, 动态生成的靶点颜色可以选择 该图像区域中没有的颜色, 如图 5A所示, 把该区域中所有的显示点按色彩坐标分布在 RGB 色彩坐标系中, 图中 50区域是图像色点集中的区域, 当选择靶点颜色时, 为了便于区分, 要 选择坐标体中远离已有色彩点的空白区域,如 52区域的色彩作为标靶点的色彩,这样就可以 从彩色图像传感器采集的图像中很容易地提取出特征靶点; 其次, 靶点的形状可以选为正十 字交叉点或斜十字交叉点, 或者其他便于识别的形式, 图 5B 中生成了正四方排列的四个十 字靶点 12, 当然也可以生成满足需要的任何形状的靶点图样。 此外, 还可以采取在短时间内 连续变化生成的靶点颜色的方法, 进一步提高靶点提取的速度和准确性。 在有些场合下, 可 能还需要赋予特征靶点不同的颜色或形状特征, 便于判定靶点的方向顺序。
综上所述,利用单目摄像机标定技术的视觉指向型鼠标的第一种实施方式包括如下步骤: 将单目图像传感器指向显示屏上任意位置, 启动图像传感器工作, 图像传感器通过有线或无 线方式与计算机相连; 确定图像传感器指向显示屏的位置: 由计算机在极短的时间内, 在显 示屏上输出确定图像传感器指向位置的定位编码图案, 图像传感器采集所指向区域的图像, 从局部区域的图像中提取出包含的图案编码, 将其与定位编码图案的空间位置査找表进行比 对, 确定图像传感器指向显示屏的大致位置; 在已定位的显示屏指向区域上确定特征靶点: 由计算机在已定位的显示屏指向区域上, 根据显示内容选择若干分布在图像传感器的图像采 集区域内的特征靶点, 或者生成若干分布在图像传感器的图像釆集区域内的特征靶点; 图像
传感器采集含有特征靶点的指向区域图像, 根据靶点特征在图像中提取靶点的图像坐标; 利 用采集的靶点的图像坐标和其计算机显示坐标, 根据单目摄像机标定技术, 解算出单目摄像 机成像参数; 以图像传感器成像面上的某固定点和镜头中心的连线作为虚拟指示轴, 由该固 定点的图像坐标, 即虚拟指示轴与成像面的交点的图像坐标, 利用解算出的单目摄像机成像 参数, 计算出其对应的在计算机显示屏上的交点的显示坐标; 由计算机将鼠标光标或其他图 像目标显示在虚拟指示轴在显示屏上的指向点处, 计算机以显示光标为中心重新确定显示屏 指向区域, 在指向区域内重新确定特征靶点; 在极短的时间间隔内, 随着图像传感器的指向 运动,重复上述步骤,使鼠标光标或其他图像目标跟随虚拟指示轴在显示屏上的指向点运动。
根据上述步骤, 设计第一种实施方式的工作流程如图 6所示, 包括如下环节: 启动指向 输入装置后, 在 600步骤, 通知计算机中央处理器向屏幕发送定位编码图案; 在 602步骤, 由输入装置的图像传感器采集图像, 由图像处理电路从中提取图像编码; 在 604步骤, 判断 图像编码是否与编码图査找表匹配, 找到图像传感器对应的大概位置, 然后转入下一步, 否 则继续重复 600、 602步骤; 在 606a步骤,通知计算机在此区域找出或生成特征靶点; 在 608 步骤, 由图像传感器采集图像, 根据靶点特征从采集的图像中提取出各个靶点坐标; 在 610 步骤, 判断是否正确提取出各靶点坐标, 否则重复 608步骤; 在 612步骤, 利用靶点的显示 坐标和提取出的图像坐标计算出成像模型的外部参数; 在 614a步骤, 再利用计算出的外部参 数, 由中心像点的坐标计算出对应的光轴与显示屏交点的显示坐标; 在 616步骤, 由计算机 将鼠标光标或其他目标显示在光轴指向点坐标位置上; 在 618步骤, 重新设置特征靶点, 将 其显示在以光轴指向点为中心的区域; 在 620步骤, 判断输入装置是否仍然处于工作状态, 如果仍在工作则继续对拍摄的连续图像进行同样的操作处理, 否则终止***工作。 通过以上 工作流程, 就可以保证光标准确跟随图像传感器光轴指向点运动。
图 7所示是本专利提出的利用单目摄像机标定技术的视觉指向型鼠标输入***的基本系 统原理框图, 10、 30、 18分别为***的三个主要设备: 视觉指向输入装置、 计算机主机和显 示屏的功能结构框图, 其具体细节图中并未体现。 完整的视觉指向输入装置包括由操作者指 向操作的外部输入装置 10和安插在计算机主机中的信息接收处理装置 702。用于视觉指向的 外部输入装置 10又包括图像传感器 100、控制功能部件 102和处理电路 104三个主要功能部 件, 其处理电路 104又包含通讯功能模块 704、 图像处理模块 706和控制功能模块 708等功 能模块;信息接收处理装置 702负责和输入装置 10及计算机操作*** 700进行通讯交互, 以 软件或硬件的形式安插在计算机主机中。如图 7中所示, 当输入装置 10启动工作后, 由通讯 模块 704激活信息接收处理装置 702, 702与主机操作***程序 700交互, 告知操作***程序 700在显示图像内容的何处、 以何种方式产生特征靶点, 同时将靶点的特征信息和坐标信息 告知输入装置。 输入装置的图像处理模块 706从图像传感器 100采集的图像中提取靶点, 进 行运算操作, 并将指向点的解算坐标由通讯模块 704传递给信息接收处理装置 702, 由其通 知计算机操作*** 700, 在指向点的位置显示鼠标光标等图像目标。 控制功能部件 102驱动 控制模块电路 708产生的控制信号, 也由通讯模块 704传递给信息接收处理装置 702, 由其 通知计算机操作***, 产生相应的控制操作。 在实际应用中, 图像信息处理等运算功能可以 灵活配置在处理电路 104中, 也可以配置在安插于计算机中的信息接收处理装置 702里面。
如图 1中所示, 根据本专利提出的利用单目摄像机标定技术的视觉指向输入技术, 设计 了一种手套式视觉指向型输入装置, 图 8A至图 8C显示了各部分的穿戴方法和使用方法, 图
8D至图 8F显示了各部分的基本机构, 下面结合示意图进一步详细介绍。 如图 8A所示, 该 手套式视觉指向型输入装置主要包括图像传感器指示部件 100、主控制功能部件 102、处理电 路 104和辅助控制功能部件 106。 这些部件固定在用尼龙或塑料等柔性材质制作的手套上, 在手套的关节弯曲部位可以适当开孔, 各部件布置在手指上的适当位置, 可以不影响手指的 弯曲动作, 从而仍然可以保证打字等手部操作的便利性。 图像传感器指示部件 100用于指向 显示屏等目标,对于利用单目摄像机标定技术的视觉指向输入技术,只使用一个图像传感器, 其基本结构如图 8D中所示,图像传感器 80包装在机械外套之中,可以夹持或绑定在手指上。 主控制功能部件 102—侧有功能控制键, 方便用拇指实现鼠标的激活、 左键、 右键、 翻页等 功能, 其工作示意图见图 8B所示, 功能控制键可以有多种形式, 可以是按压键 82a, 也可以 是触摸键 82b, 如图 8E中所示, 也可以是二者的组合, 例如当拇指接触功能键激发输入装置 工作, 当拇指下按时发出左键功能。 辅助控制功能部件 106上有挤压开关或触摸开关 84, 如 图 8F所示, 通过弯曲小指来触发工作, 其工作示意图参见图 8C, 可以将右键功能和翻页功 能放置在主控部件上, 也可以放置在辅助控制部件上。 处理电路 104可以灵活布置在手背或 其他位置, 其中有利用 DSP、 FPGA等数字处理芯片实现的进行图像处理和数据解算的信息 处理模块, 负责同计算机进行通讯的有线或无线通讯模块, 以及数据存储器等。 此外, 也可 以把图像信息处理等运算功能配置在安插于计算机中的信息接收处理装置里面。 上述各部件 可以根据需要灵活地设计外观和进行增减, 也可以用各种可能的方式进行操作。
以上介绍了一种手套式视觉指向输入装置, 实际上, 利用本专利提供的视觉指向输入技 术, 可以根据使用需求, 设计出多种多样的应用装置。 例如, 对上述手套式视觉指向输入装 置, 可以将信处理电路集成到其他部件中。 当然, 也可以经过良好设计, 将所有部件整合于 一体, 构成一个指套, 穿戴在食指上工作, 如图 9A所示。 也可以根据使用平台不同, 设计 成笔形或枪形等各种结构样式, 图 9B所示就是一个笔式视觉输入装置, 所有部件都组装在 一个笔式外壳之中。 另外, 还可以把这种视觉指向型输入装置固定在头部, 通过头部的运动 来驱动光标运动。 总之, 在保留主要功能部件的前提下, 可以根据需要设计出各种各样结构 的视觉指向输入装置, 以及各种不同的使用方式。
[实施方式 2]
图 10描述的是利用单目摄像机标定技术的视觉指向型鼠标的第二种实施方式。第二种实 施方式与第一种实施方式类似, 同样涉及一种控制鼠标光标等图形目标准确显示在一个图像 传感器的虚拟指示轴指向计算机显示屏的交点处的视觉指向输入方法, 在这种方法中, 仍然 以显示屏等动态显示设备作为靶标的设置区域, 但第二种方法并不定位图像传感器的指向位 置, 而是由计算机在显示屏的某区域直接设定特征靶点图, 确定靶点区域, 由操作者主动将 摄像头等图像传感器指向该区域, 获取靶点图像, 定位鼠标光标。 在第二种方法中, 仍然使 靶标区域跟随图像传感器的指向而运动, 始终保持靶标区域在图像传感器的指向成像区域。
第二种实施方式与第一种实施方式相比, 只是缺少了图像传感器定位的环节, 因此, 第 二种实施方式包括如下步骤: 启动图像传感器工作, 图像传感器通过有线或无线方式与计算 机相连; 在显示屏上确定特征靶点: 由计算机在显示屏上某区域, 根据显示内容选择若干分 布在图像传感器的图像采集区域内的特征靶点, 或者生成若千分布在图像传感器的图像采集 区域内的特征靶点; 将单目图像传感器指向显示屏上的特征靶点区域, 采集含有特征靶点的
指向区域图像, 根据靶点特征在图像中提取靶点的图像坐标; 利用采集的靶点的图像坐标和 其计算机显示坐标, 根据单目摄像机标定技术, 解算出单目摄像机成像参数; 以图像传感器 成像面上的某固定点和镜头中心的连线作为虚拟指示轴, 由该固定点的图像坐标, 即虚拟指 示轴与成像面的交点的图像坐标, 利用解算出的单目摄像机成像参数, 计算出其对应的在计 算机显示屏上的交点的显示坐标; 由计算机将鼠标光标或其他图像目标显示在虚拟指示轴在 显示屏上的指向点处, 计算机以显示光标为中心重新确定显示屏指向区域, 在指向区域内重 新确定特征靶点; 在极短的时间间隔内, 随着图像传感器的指向运动, 重复上述步骤, 使鼠 标光标或其他图像目标跟随虚拟指示轴在显示屏上的指向点运动。
根据上述步骤, 设计第二种实施方式的工作流程见图 10所示, 包括如下环节: 启动指向 输入装置后, 直接进入 606b步骤, 通知计算机在此区域找出或生成特征靶点; 在 608步骤, 由图像传感器采集图像, 从采集的图像中提取出各个靶点坐标; 在 610步骤, 判断是否正确 提取出各靶点坐标, 否则重复 608步骤; 在 612步骤, 利用靶点的显示坐标和提取出的图像 坐标计算出成像模型的成像参数; 在 614a步骤, 再利用计算出的成像参数, 由中心像点的坐 标计算出对应的光轴与显示屏交点的显示坐标; 在 616步骤, 由计算机将鼠标光标或其他目 标显示在光轴指向点坐标位置上; 在 618步骤, 重新设置特征靶点, 将其显示在以光轴指向 点为中心的区域; 在 620步骤, 判断输入装置是否仍然处于工作状态, 如果仍在工作则继续 对拍摄的连续图像进行同样的操作处理, 否则终止***工作。 通过以上工作流程, 就可以保 证光标准确跟随图像传感器光轴指向点运动。
[实施方式 3]
图 11A至图 11C描述的是利用单目摄像机标定技术的视觉指向型鼠标的第三种实施方 式。 第三种实施方式有关一种控制鼠标光标等图形目标准确跟随一个图像传感器的虚拟指示 轴在靶标区域的运动的视觉指向输入方法。 在这种方法中, 使用一个固定的平面靶标, 靶标 区域的大小确定, 其中设置一个平面靶标坐标系, 靶标区域内包含若干特征靶点, 靶点具有 特定的颜色、 形状等特征, 便于从图像中提取, 并且已知靶点在靶标坐标系中的坐标。 操作 时, 将图像传感器指向该区域, 获取靶点图像, 首先利用单目摄像机标定技术定位图像传感 器的虚拟指示轴在靶标区域的指向点在靶标坐标系内的坐标, 然后再根据靶标坐标系和显示 屏坐标系的单位比例关系求出对应的鼠标光标显示坐标。
第三种实施方式的主要特征是: 预先在某固定靶标区域设置若干特征靶点, 靶点在该区 域内的坐标已知, 如图 11A至图 11C所示, 该区域可以是计算机显示屏上的某局部固定区域 (图 11A), 也可以是显示屏的边框(图 11B), 或者是显示屏周边的靶面(图 11C)。 靶点的 设置可以有各种选择, 既可以是由计算机在显示屏上生成的具有特定颜色和形状的显示点; 也可以是设置在显示屏边框等实物上的实物点装置, 如 LED光源、 有色反光贴片等。
第三种实施方式包括如下步骤: 选取某固定靶标区域作为采集图像的指向区域, 该区域 的大小确定, 在该区域内设置若干特征靶点, 并且己知靶点在该区域中的坐标; 将单目图像 传感器指向该靶标区域, 图像传感器通过有线或无线方式与计算机主机相连, 启动图像传感 器工作; 采集含有特征靶点的指向区域图像, 根据靶点特征在图像中提取靶点的图像坐标; 利用采集的靶点的图像坐标和其在该固定区域中的坐标, 根据单目摄像机标定技术, 解算出 单目摄像机成像参数;以图像传感器成像面上的某固定点和镜头中心的连线作为虚拟指示轴,
由该固定点的图像坐标, 即虚拟指示轴与成像面的交点的图像坐标, 利用解算出的单目摄像 机成像参数, 计算出其对应的在靶标区域内的交点的坐标; 将计算出的指示轴交点在靶标坐 标系中的坐标乘以一个比例系数, 该比例系数由显示屏的实际尺寸除以对应的靶标区域的实 际尺寸而得到, 求出光标在显示屏上的显示坐标; 由计算机将鼠标光标或其他图像目标根据 已计算出的显示坐标, 显示在显示屏上; 在极短的时间间隔内, 随着图像传感器的指向运动, 重复上述步骤, 使鼠标光标或其他图像目标跟随虚拟指示轴在靶标区域的指向而运动。
图 12所示是第三种实施方式的工作流程图, 包括如下环节: 启动指向输入装置后, 在 606c步骤, 将图像传感器对准事先设定的固定靶标区域; 在 608步骤, 由图像传感器采集图 像, 从采集的图像中提取出各个靶点坐标; 在 610步骤, 判断是否正确提取出各靶点坐标, 否则重复 608步骤; 在 612步骤, 利用靶点的显示坐标和提取出的图像坐标计算出成像模型 的成像参数; 在 614b步骤, 再利用计算出的成像参数, 由中心像点的坐标计算出对应的指示 轴与靶标区域交点在其中的坐标; 在 622步骤, 将计算出的交点坐标乘以一个比例系数, 得 到光标在显示屏上的显示坐标; 在 616步骤, 由计算机将鼠标光标或其他目标显示在光轴指 向点坐标位置上; 在 620步骤, 判断输入装置是否仍然处于工作状态, 如果仍在工作则继续 对拍摄的连续图像进行同样的操作处理, 否则终止***工作。 通过以上工作流程, 就可以保 证光标准确跟随图像传感器光轴指向点运动。
[实施方式 4]
图 13描述的是利用单目摄像机标定技术的视觉指向型鼠标的第四种实施方式。第四种实 施方式是一种利用单目摄像机标定技术实现的空间运动姿态视觉输入方法, 它利用预先标定 出焦距及横向、 纵向像元间隔等内部物理参量的图像传感器, 对显示屏上或其他预设靶标区 域中的已知相互间距离的固定特征靶点进行连续图像采集, 提取出靶点的图像坐标, 根据单 目摄像机标定理论求出外部成像参数后, 进一步求出决定这些外部参数的外部物理参量, 即 显示屏所在的世界坐标系在图像传感器的摄像机坐标系中的三轴旋转角 (α, β, γ)和原点坐标 (Χο, Υ。, Ζ0), 从而可以确定图像传感器相对于显示屏的空间方位坐标。 这样, 对图像传感器 采集的序列图像进行处理, 就可以得到图像传感器相对于显示屏的空间姿态和运动轨迹, 利 用这种提取的空间运动可以实现某些特定操作。
在这种应用模式下, 还可以对多个运动物体同时采用多个视觉输入装置, 同时解算出每 个成像器件的空间方位, 从而可以组成一个包含多部分运动的整体运动。 例如, 可以在每个 手指上都安置一个小型摄像头, 对准屏幕上同一组靶点进行运动, 可以根据解算出的各个成 像器件的空间姿态和运动轨迹, 辨识出手的活动内容, 如抓握、 旋转、 平移等, 从而进行相 应的控制操作。
如图 13所示, 输入装置最初位于 90位置处, 此时采集显示屏中某固定区域内的特征靶 点 12的图像,由靶点的图像坐标、显示坐标和己知的内部参数,计算出外部成像参数的数值, 进而求出输入装置在该处相对于靶点的方位关系, 即 (α, β, γ; Χο, Υ。, Ζ。:)等参量; 当输入装置 移至 92位置处, 进行同样操作, 计算出输入装置在此处相对于靶点的方位参量, 根据两个位 置的方位关系的差值, 即可得到输入装置的空间姿态和运动轨迹。
第四种实施方式包括如下步骤: 选取某固定靶标区域作为采集图像的指向区域, 在该区 域内设置若干特征靶点, 并且已知靶点在该区域中的坐标; 将已经标定出焦距、 像元间隔等
内部物理参量的单目图像传感器指向靶点区域, 图像传感器通过有线或无线方式与计算机相 连, 启动图像传感器工作, 采集含有特征靶点的指向区域图像, 根据靶点特征在图像中提取 靶点的图像坐标; 利用采集的靶点的图像坐标和其在该固定区域中的坐标, 根据单目摄像机 标定技术, 解算出单目摄像机成像参数; 进而利用计算出的成像模型的成像参数, 求出图像 传感器在特征靶点所在的坐标系中的空间方位坐标; 在极短的时间间隔内, 随着图像传感器 的指向运动, 重复上述步骤, 求出图像传感器在不同位置时处于特征靶点所在的坐标系中的 空间方位坐标, 将一系列图像成像器件的空间方位坐标连接起来, 就得到了图像传感器相对 于显示屏的空间运动姿态。
图 14所示是第四种实施方式的工作流程图, 包括如下环节: 启动指向输入装置后, 在 606c步骤, 将图像传感器对准事先设定的固定靶点区域; 在 608步骤, 由图像传感器采集图 像, 从采集的图像中提取出各个靶点坐标; 在 610步骤, 判断是否正确提取出各靶点坐标, 否则重复 608步骤; 在 612步骤, 利用靶点的显示坐标和提取出的图像坐标计算出成像模型 的成像参数; 在 624步骤, 利用计算出的成像参数, 进一步求出图像传感器的空间方位坐标; 在 620步骤, 判断输入装置是否仍然处于工作状态, 如果仍在工作则继续对拍摄的连续图像 进行同样的操作处理, 否则终止***工作。 通过以上工作流程, 就可以保证光标准确跟随图 像传感器光轴指向点运动。
以上具体实施方式仅用于说明本专利, 而非用于限定本专利, 根据本发明的技术方案和 技术构思做出的其他各种改变和变形都应属于本专利权利要求的保护范围。
Claims
1. 一种利用单目摄像机标定技术的视觉指向型鼠标输入方法, 所述方法包括如下步骤: ^
( i)将单目图像传感器指向一靶标, 启动图像传感器工作;
( ii)图像传感器采集靶标指向区域包含的特征内容的图像, 在图像中提取出特征靶点的 图像坐标;
(m)利用采集的靶点图像坐标和靶点在靶标坐标系中的坐标,根据单目摄像机标定技术, 解算出单目摄像机成像参数;
( iv)由图像传感器的成像面上的一固定像点的图像坐标,利用解算出的单目摄像机成像 参数, 计算出该像点对应的物点在靶标坐标系中的坐标, 即指示点在靶标坐标系中的坐标;
(v)由指示点在靶标坐标系中的坐标计算出鼠标光标或其他图像目标在显示屏上的显示 坐标, 并由计算机将光标显示在显示屏上;
在极短的时间间隔内, 重复 ii到 V步骤, 使鼠标光标或其他图像目标跟随图像传感器的 指向而运动。
2. 一种利用单目摄像机标定技术的视觉指向型鼠标输入方法, 所述方法包括如下步骤:
( i)将单目图像传感器指向一靶标, 启动图像传感器工作;
(ii )图像传感器釆集靶标指向区域包含的特征内容的图像, 在图像中提取出特征靶点的 图像坐标;
( iii )利用采集的靶点图像坐标和靶点在靶标坐标系中的坐标,根据单目摄像机标定技术, 解算出单目摄像机成像参数;
(iv)进而利用计算出的摄像机成像参数, 求出图像传感器在靶标坐标系中的空间方位 坐标;
在极短的时间间隔内, 随着图像传感器的指向运动, 重复 ii到 iv步骤, 求出图像传感器 在不同位置时的空间方位坐标, 将一系列图像成像器件的空间方位坐标连接起来, 就得到了 图像传感器相对于显示屏的空间运动姿态。
3. 如权利要求 1、 2所述的方法, 其特征在于, 所述将图像传感器指向一靶标, 是指: 将图 像传感器指向一个事先己经设定好的靶标。
4. 如权利要求 1、 2所述的方法, 其特征在于, 所述将图像传感器指向一靶标, 是指: 当图 像传感器指向计算机显示屏时, 首先由计算机判定图像传感器指向显示屏的大致区域, 然后 在图像传感器的显示屏指向区域动态确定一个靶标。
5. 如权利要求 4所述的方法, 其特征在于, 所述的判定图像传感器指向显示屏的大致区域, 包括如下步骤:
( 1 )将图像传感器指向显示屏的某一区域采集图像, 图像传感器通过有线或无线通讯方 式与计算机相连, 显示屏与计算机相连接;
(2)启动图像传感器工作,通知计算机在极短的时间内在显示屏上输出由若干不同颜色 或图形内容的特征块排列组成的编码图案, 每种颜色或图形内容被编成不同号码, 每个特征
' 7 块附近的一定范围内的特征块组成的编码在整个编码图中是唯一的, 整个编码图的所有特征 块的区域编码构成一个定位査找表;
(3 )图像传感器采集所指向区域的编码图像, 从中提取局部图案的编码, 将其与编码图 的空间位置査找表进行比对, 确定图像传感器指向显示屏的大致位置。
6. 如权利要求 5所述的方法, 其特征在于, 所述特征块编码图案, 是指: 在显示屏上输出由 若干不同颜色或图形内容的矩形特征块排列组成的编码图案, 每种颜色或图形内容被编成不 同号码,每个矩形特征块附近的 nXn范围内的所有特征块组成的编码在整个编码图中是唯一 的, 整个编码图的所有特征块的区域编码构成一个定位查找表。
7. 如权利要求 4所述的方法, 其特征在于, 所述的判定图像传感器指向显示屏的大致区域, 包括如下步骤:
( 1 )将图像传感器指向显示屏的某一区域采集图像, 图像传感器通过有线或无线通讯方 式与计算机相连, 显示屏与计算机相连接;
(2)启动图像传感器工作,首先在显示屏上输出由若干不同颜色或图形内容的特征块排 列组成的粗分辨率编码图案, 每种颜色或图形内容被编成不同号码, 图像传感器采集指向区 域图像, 判断自身所指向的特征块位置;
(3 )然后, 计算机在已确定的大特征块范围内再次输出编码图案, 其范围为该特征块的 大小, 图像传感器采集指向区域图像, 进一步判断自身所对的小特征块的位置;
(4) 由大及小地快速循环执行此操作, 最终确定出图像传感器所指向的显示屏的位置。
8. 如权利要求 1、 2所述的方法, 其特征在于, 所述靶标, 是指: 一个固定平面靶标, 靶标 区域的大小确定, 该区域内包含若干特征靶点, 靶点具有特定的颜色、 形状等特征, 便于从 图像中提取, 并且已知靶点在该区域的靶标坐标系中的坐标。
9. 如权利要求 8所述的方法, 其特征在于, 所述的固定平面靶标, 是指: 选取显示屏的边框 作为靶标区域, 特征靶点设置在边框上, 各点之间的距离已知。
10. 如权利要求 8所述的方法, 其特征在于, 所述的固定平面靶标, 是指: 选取显示屏周边 的一个大小确定的平面作为靶标区域, 特征靶点设置在该区域内, 各点之间的距离已知。
11. 如权利要求 8所述的方法, 其特征在于, 所述的固定平面靶标, 是指: 由计算机在显示 屏上选取某局部固定区域作为靶标区域, 其大小由该区域的计算机显示坐标确定, 并且由计 算机在显示屏靶标区域内确定若干特征点作为特征靶点。
12. 如权利要求 1、 2所述的方法, 其特征在于, 所述靶标, 是指: 一个动态平面靶标, 该靶 标可由计算机在显示屏上动态生成, 靶标的生成位置始终追随图像传感器的指向, 靶标区域 的范围可根据图像传感器的成像距离调整, 其大小由该区域的计算机显示坐标确定, 并且由 计算机在显示屏靶标区域内确定若干特征点作为特征靶点。
13. 如权利要求 11、 12所述的方法, 其特征在于, 所述的由计算机在显示屏靶标区域确定特 征靶点, 是指: 计算机对显示屏上包含靶标区域的一定范围内的显示内容进行处理, 利用颜 色、边缘、 角点、 方位、 周边环境信息等特征, 从计算机的显示内容中选择出若干特征靶点, 由这些特征靶点界定靶标区域范围, 并记录其特征信息。
14. 如权利要求 11、 12所述的方法, 其特征在于, 所述的由计算机在显示屏靶标区域确定特 征靶点, 包括如下步骤:
( 1 )计算机对显示屏上包含靶标区域的一定范围内的显示内容颜色进行统计,选择显示 内容中没有的、 且与存在的颜色差异较大的颜色作为生成的特征靶点的颜色;
(2)由计算机在显示屏上的靶标区域内, 以选择的颜色额外生成一定的显示内容, 生成 的显示内容中包含交叉点、 角点、 中心点等特征, 可以根据这些特征选择出若干特征靶点, 由这些特征靶点界定靶标区域范围。
15. 如权利要求 8所述的方法, 其特征在于, 所述若干特征靶点, 是指: 对于使用事先没有 标定出焦距、 像元间隔等内部物理参量的图像传感器的方法, 需要在显示屏上确定至少 4个 特征靶点。
16. 如权利要求 8所述的方法, 其特征在于, 所述若干特征靶点, 是指: 对于使用事先已经 标定出焦距、 像元间隔等内部物理参量的图像传感器的方法, 需要在显示屏上确定至少 3个 特征靶点。
17. 如权利要求 1、 2所述的方法, 其特征在于, 所述的解算单目摄像机成像参数, 是指: 利 用采集的靶点图像坐标和靶点的靶标坐标系坐标, 根据单目摄像机标定技术, 解算出图像传 感器的单目摄像机外部成像参数。
18. 如权利要求 1、 2所述的方法, 其特征在于, 所述的解算单目摄像机成像参数, 是指: 利 用采集的靶点图像坐标和靶点的靶标坐标系坐标, 根据单目摄像机标定技术, 解算出图像传 感器的单目摄像机内部成像参数和外部成像参数。
19. 如权利要求 1所述的方法, 其特征在于, 所述的图像传感器成像面上的一固定像点, 是 指: 该像点可以是成像面上的任意一像点, 该像点与成像镜头光心的连线构成一个虚拟指示 轴, 该像点对应的物点即为虚拟指示轴的指向点。
20. 如权利要求. 1所述的方法, 其特征在于, 所述的图像传感器成像面上的一固定像点, 是 指: 该像点可以是成像面上的中心像点, 该像点与成像镜头光心的连线, 即成像***的光轴, 构成一个虚拟指示轴, 该中心像点对应的物点即为虚拟指示轴的指向点。
21. 如权利要求 1所述的方法, 其特征在于, 所述的由指示点在靶标坐标系中的坐标计算鼠
( 标光标的显示坐标,是指: 当靶标坐标系的单位长度与计算机显示屏的像元间隔尺寸相同时, 则已经计算出的靶点在靶标坐标系中的坐标就是显示坐标; 当靶标坐标系的单位长度与计算 机显示屏的像元间隔尺寸不同时, 则需要将已计算出的靶点在靶标坐标系中的坐标乘以一个 比例系数, 得到显示坐标, 该比例系数由显示屏的像元间隔尺寸除以靶标坐标系的单位长度 获得。
22. 如权利要求 2所述的方法, 其特征在于, 所述的图像传感器在靶标坐标系中的空间方位 坐标, 是指: 空间方位坐标包括三轴旋转角((!^^)和原点坐标(^,¥。,2。)。
23. 一种利用单目摄像机标定技术的视觉指向型鼠标输入***, 其包括:
(i)计算机主机, 和与之相连的显示屏;
(ii)靶标, 用于标定图像传感器的摄像机成像参数;
(Hi) 单目图像传感器, 用于采集特征靶标区域的图像, 通过处理电路与计算机主机相 连;
(iv)控制功能部件, 产生所需控制功能信号, 通过处理电路与计算机主机相连;
(V)处理电路, 实现图像信息处理和通讯功能, 与图像传感器和控制功能部件相连, 同 时通过有线或无线的方式与计算机主机相连;
(vi)信息接收处理装置, 安插在计算机主机中, 与处理电路通过有线或无线方式相连, 同时与计算机操作***通讯交互。
24. 如权利要求 23所述的***, 其特征在于, 所述处理电路, 包括图像采集模块、 数据处理 模块、 控制信号模块、 数据接口模块, 具有如下功能: 接收图像传感器传递的图像信息和控 制功能部件产生的控制功能信号; 对采集的图像进行处理, 完成图像传感器的指向定位、 特 征靶点提取、单目摄像机成像参数计算、光标指向点的显示坐标计算等功能; 产生***触发、 左键、 右键、 翻页、 滚动等控制功能信号; 与信息接收处理装置通过有线或无线的方式进行 通讯交互, 传递图像、 特征信息、 运算结果、 控制功能信号等信息。
25. 如权利要求 23所述的***, 其特征在于, 所述信息接收处理装置, 包括数据接口模块、 数据处理模块、 数据通讯模块, 具有如下功能: 接收由处理电路发送的图像、 运算结果等信 息; 接收由处理电路产生的***触发、 左键、 右键、 翻页、 滚动等控制功能信号; 向处理电 路发送靶点的特征信息、 坐标信息; 向计算机操作***输出计算得到的光标坐标信息。
26. 一种利用单目摄像机标定技术的视觉指向型鼠标输入***, 其包括-
(i)计算机主机, 和与之相连的显示屏;
(ii)靶标, 用于标定图像传感器的摄像机成像参数;
(iii) 单目图像传感器, 用于采集特征靶标区域的图像, 通过信息接收处理装置与计算 机主机相连;
(iv)控制功能部件, 产生所需控制功能信号, 通过信息接收处理装置与计算机主机相 连;
0 (v)信息接收处理装置, 安插在计算机主机中, 实现图像信息处理和通讯功能, 并且与 计算机操作***通讯交互。
27. 如权利要求 23、 26所述的***, 其特征在于, 所述控制功能部件: 布置若干控制功能按 键, 用于产生***触发、 左键、 右键、 翻页、 滚动等控制功能信号。
28. 如权利要求 26所述的***, 其特征在于, 所述信息接收处理装置, 包括图像采集模块、 数据处理模块、 控制信号模块、 数据接口模块、 数据通讯模块, 具有如下功能: 接收图像传 感器传递的图像信息和控制功能部件产生的控制功能信号; 对采集的图像进行处理, 完成图 像传感器的指向定位、 特征靶点提取、 单目摄像机成像模型的参数计算、 光标指向点的显示 坐标计算等功能; 产生***触发、 左键、 右键、 翻页、 滚动等控制功能信号; 通知计算机操 作***将鼠标光标或其他图像目标显示在显示屏上的显示坐标处。
29. 一种手套式视觉指向型鼠标输入装置, 其包括:
(i)计算机主机, 和与之相连的显示屏;
(ii)靶标, 用于标定图像传感器的摄像机成像参数;
(iii) 安装有单目图像传感器的指向指套, 用于指向靶标区域采集图像, 通过处理电路 与计算机主机相连;
(iv)控制功能键指套, 包括若干按键、 触摸键或压力开关, 用于产生所需控制功能信 号, 通过处理电路与计算机主机相连;
(V)辅助功能键指套, 通过弯曲手指, 由手指自身触发开关控制, 通过处理电路与计算 机主机相连, 该辅助功能键指套根据使用情况选用;
(vi) 处理电路, 实现图像信息处理和通讯功能, 与图像传感器和控制功能部件相连, 同时通过有线或无线的方式与计算机主机相连;
(vii)信息接收处理装置, 安插在计算机主机中, 与处理电路通过有线或无线方式相连, 同时与计算机操作***通讯交互。
30. 一种手套式视觉指向型鼠标输入装置, 其包括:
(i)计算机主机, 和与之相连的显示屏;
(ii)靶标, 用于标定图像传感器的摄像机成像参数;
(iii) 安装有单目图像传感器的指向指套, 用于指向靶标区域采集图像, 通过处理电路 与计算机主机相连;
(iv)控制功能键指套, 包括若干按键、 触摸键或压力开关, 用于产生所需控制功能信 号, 通过处理电路与计算机主机相连;
(v) 辅助功能键指套, 通过弯曲手指, 由手指自身触发开关控制, 与计算机主机相连, 该辅助功能键指套根据使用情况选用;
(vi)信息接收处理装置, 安插在计算机主机中, 实现图像信息处理和通讯功能, 并且 与计算机操作***通讯交互。
31 . 一种指套式视觉指向型鼠标输入装置, 其包括:
( i ) 计算机主机, 和与之相连的显示屏;
(ii )靶标, 用于标定图像传感器的摄像机成像参数;
( iii )视觉指向型鼠标输入指套, 将单目图像传感器、 处理电路、 控制功能键集成在一 个指套上, 可穿戴于一个手指上, 以有线或无线方式与计算机主机相连;
( iv)信息接收处理装置, 安插在计算机主机中, 实现图像信息处理和通讯功能, 并且 与计算机操作***通讯交互。
32. 一种指套式视觉指向型鼠标输入装置, 其包括:
( i)计算机主机, 和与之相连的显示屏;
(ii )靶标, 用于标定图像传感器的摄像机成像参数;
( iii)视觉指向型鼠标输入指套, 将单目图像传感器、 控制功能键集成在一个指套上, 可穿戴于一个手指上, 以有线或无线方式与计算机主机相连;
( iv) 信息接收处理装置, 安插在计算机主机中, 实现图像信息处理和通讯功能, 并且 与计算机操作***通讯交互。
33. 一种视觉指向型鼠标应用程序, 驻留在计算机主机中, 与计算机操作***和视觉指向型 鼠标输入***进行通讯交互, 包括如下内容-
(i )图像接收处理程序,接收由视觉指向型鼠标输入***的图像传感器发送的图像信息;
( ii) 图像传感器定位程序, 确定图像传感器指向显示屏的位置;
(iii)特征靶点产生程序, 在显示屏上确定所需特征靶点;
(iv)靶点提取程序, 根据靶点特征在图像中提取靶点的图像坐标;
(v)成像参数计算程序, 根据单目摄像机标定技术, 解算出单目摄像机成像参数;
(vi )显示坐标计算程序, 计算出指示轴指向点对应的光标显示坐标;
( vii)光标显示程序, 通知计算机将鼠标光标或其他图像目标显示在显示屏上, 并以显 示光标为中心重新确定显示屏指向区域, 在指向区域内重新确定特征靶点;
(viii )控制功能程序, 产生***触发、 左键、 右键、 翻页、 移动等控制功能信号。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2010/001229 WO2012019322A1 (zh) | 2010-08-13 | 2010-08-13 | 利用单目摄像机标定技术的视觉指向型鼠标输入方法、输入***和输入装置 |
CN201080068268.9A CN103124949B (zh) | 2010-08-13 | 2010-08-13 | 利用单目摄像机标定技术的视觉指向型鼠标输入方法、输入***和输入装置 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2010/001229 WO2012019322A1 (zh) | 2010-08-13 | 2010-08-13 | 利用单目摄像机标定技术的视觉指向型鼠标输入方法、输入***和输入装置 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012019322A1 true WO2012019322A1 (zh) | 2012-02-16 |
Family
ID=45567259
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2010/001229 WO2012019322A1 (zh) | 2010-08-13 | 2010-08-13 | 利用单目摄像机标定技术的视觉指向型鼠标输入方法、输入***和输入装置 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN103124949B (zh) |
WO (1) | WO2012019322A1 (zh) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104281301A (zh) * | 2013-07-05 | 2015-01-14 | 联想(北京)有限公司 | 一种输入方法及电子设备 |
CN104536556A (zh) * | 2014-09-15 | 2015-04-22 | 联想(北京)有限公司 | 一种信息处理方法及电子设备 |
CN104683067A (zh) * | 2015-02-10 | 2015-06-03 | 厦门精图信息技术股份有限公司 | 一种用于非接触式数据传递的网格编码方法 |
CN105371784A (zh) * | 2015-12-24 | 2016-03-02 | 吉林大学 | 基于机器视觉的汽车检测全息人机交互*** |
CN105446460A (zh) * | 2014-06-13 | 2016-03-30 | 联想(北京)有限公司 | 一种信息处理方法、交互类附件及电子设备 |
CN105678709A (zh) * | 2016-01-12 | 2016-06-15 | 西安交通大学 | 一种led手持靶标光心偏差校正算法 |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104951080A (zh) | 2015-07-06 | 2015-09-30 | 联想(北京)有限公司 | 一种控制方法以及装置 |
CN107291272A (zh) * | 2017-06-20 | 2017-10-24 | 广州视源电子科技股份有限公司 | 光标位置确定及图像捕获的方法、装置和相关设备 |
CN109907932B (zh) * | 2019-04-08 | 2023-12-26 | 中国航天员科研训练中心 | 一种前庭刺激影响人体上肢运动控制研究实验*** |
CN110286825B (zh) * | 2019-06-26 | 2020-09-08 | 西南民族大学 | 一种基于机器视觉的机械式全自动鼠标操作装置 |
CN112753047B (zh) * | 2020-12-30 | 2022-08-26 | 华为技术有限公司 | 摄像头的硬件在环标定、靶点设置方法、***及相关设备 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0871252A (ja) * | 1994-09-05 | 1996-03-19 | Taito Corp | 射撃ゲーム装置におけるビデオ画面とガンの相対位置を検出する方法及び装置 |
CN1923326A (zh) * | 2005-09-01 | 2007-03-07 | 任天堂株式会社 | 信息处理***和程序 |
CN101282411A (zh) * | 2008-03-14 | 2008-10-08 | 青岛海信电器股份有限公司 | 控制装置、包括所述控制装置的视频装置及控制方法 |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN200950250Y (zh) * | 2006-09-19 | 2007-09-19 | 李大为 | 指套式计算机位置输入装置 |
-
2010
- 2010-08-13 WO PCT/CN2010/001229 patent/WO2012019322A1/zh active Application Filing
- 2010-08-13 CN CN201080068268.9A patent/CN103124949B/zh not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0871252A (ja) * | 1994-09-05 | 1996-03-19 | Taito Corp | 射撃ゲーム装置におけるビデオ画面とガンの相対位置を検出する方法及び装置 |
CN1923326A (zh) * | 2005-09-01 | 2007-03-07 | 任天堂株式会社 | 信息处理***和程序 |
CN101282411A (zh) * | 2008-03-14 | 2008-10-08 | 青岛海信电器股份有限公司 | 控制装置、包括所述控制装置的视频装置及控制方法 |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104281301A (zh) * | 2013-07-05 | 2015-01-14 | 联想(北京)有限公司 | 一种输入方法及电子设备 |
CN104281301B (zh) * | 2013-07-05 | 2017-12-29 | 联想(北京)有限公司 | 一种输入方法及电子设备 |
CN105446460A (zh) * | 2014-06-13 | 2016-03-30 | 联想(北京)有限公司 | 一种信息处理方法、交互类附件及电子设备 |
CN104536556A (zh) * | 2014-09-15 | 2015-04-22 | 联想(北京)有限公司 | 一种信息处理方法及电子设备 |
CN104536556B (zh) * | 2014-09-15 | 2021-01-15 | 联想(北京)有限公司 | 一种信息处理方法及电子设备 |
CN104683067A (zh) * | 2015-02-10 | 2015-06-03 | 厦门精图信息技术股份有限公司 | 一种用于非接触式数据传递的网格编码方法 |
CN105371784A (zh) * | 2015-12-24 | 2016-03-02 | 吉林大学 | 基于机器视觉的汽车检测全息人机交互*** |
CN105678709A (zh) * | 2016-01-12 | 2016-06-15 | 西安交通大学 | 一种led手持靶标光心偏差校正算法 |
Also Published As
Publication number | Publication date |
---|---|
CN103124949A (zh) | 2013-05-29 |
CN103124949B (zh) | 2016-08-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2012019322A1 (zh) | 利用单目摄像机标定技术的视觉指向型鼠标输入方法、输入***和输入装置 | |
CN110476168B (zh) | 用于手部跟踪的方法和*** | |
CN107615214B (zh) | 界面控制***、界面控制装置、界面控制方法及程序 | |
CN108027656B (zh) | 输入设备、输入方法和程序 | |
KR101809636B1 (ko) | 컴퓨터 장치의 원격 제어 | |
TWI450132B (zh) | A portrait recognition device, an operation judgment method, and a computer program | |
CN108027654B (zh) | 输入设备、输入方法和程序 | |
US20090278915A1 (en) | Gesture-Based Control System For Vehicle Interfaces | |
EP2790089A1 (en) | Portable device and method for providing non-contact interface | |
NO339941B1 (no) | System og fremgangsmåte for et gestbasert styringssystem | |
US20110304548A1 (en) | Mouse provided with a dot pattern reading function | |
KR101343748B1 (ko) | 포인터를 표시하지 않는 투명 디스플레이 가상 터치 장치 | |
US20150009119A1 (en) | Built-in design of camera system for imaging and gesture processing applications | |
KR20120126508A (ko) | 포인터를 사용하지 않는 가상 터치 장치에서의 터치 인식 방법 | |
CN111596757A (zh) | 一种基于指尖交互的手势控制方法和装置 | |
JP2016103137A (ja) | ユーザインタフェース装置、画像処理装置及び制御用プログラム | |
JP6746419B2 (ja) | 情報処理装置、及びその制御方法ならびにコンピュータプログラム | |
JP2017117373A (ja) | 操作装置とその制御方法、及びプログラム | |
CN117472189B (zh) | 具有实物感的打字或触控的实现方法 | |
JP2003280813A (ja) | ポインティングデバイス、ポインタ制御装置、ポインタ制御方法及びその方法を記録した記録媒体 | |
EP2879029B1 (en) | Coordinate detection system, information processing apparatus, and recording medium | |
JP6618301B2 (ja) | 情報処理装置、その制御方法、プログラム、及び記憶媒体 | |
JP2003076488A (ja) | 指示位置判断装置及び方法 | |
Haubner et al. | Recognition of dynamic hand gestures with time-of-flight cameras | |
EP3059664A1 (en) | A method for controlling a device by gestures and a system for controlling a device by gestures |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201080068268.9 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 10855743 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 10855743 Country of ref document: EP Kind code of ref document: A1 |