CN107693117A

CN107693117A - Augmented reality aided surgery system and the method that 3D models and patient with operation are subjected to automatic reclosing matching

Info

Publication number: CN107693117A
Application number: CN201710910238.1A
Authority: CN
Inventors: 谢锋
Original assignee: Heilongjiang Lan Zhi Technology Co Ltd
Current assignee: Wuxi lanruan Intelligent Medical Technology Co., Ltd
Priority date: 2017-09-29
Filing date: 2017-09-29
Publication date: 2018-02-16
Anticipated expiration: 2037-09-29
Also published as: CN107693117B

Abstract

Augmented reality aided surgery system and the method that 3D models and patient with operation are carried out into automatic reclosing matching, are related to a kind of aided surgery system and method.In order to solve the problems, such as that the sight that existing surgical procedure has doctor toggles between the operative site and display of reality, and cause the problem of less efficient.The system of the present invention includes：For load 3D model files 3D model files load-on module, based on AR glasses by AR technologies present 3D models 3D model files present module, for provide 3D models and patient with operation to overlaps automatic reclosing matching and overlap manually matching model selection gear selection module, for 3D models and patient with operation to be carried out to the Auto-matching of automatic reclosing matching overlaps module and for overlapping the manual matching matched coincidence module by 3D models and patient with operation progress are manual.The auxiliary that the present invention is applied to operation is implemented.

Description

Augmented reality aided surgery system and 3D models and patient with operation are subjected to automatic reclosing The method of matching

Technical field

The present invention relates to a kind of aided surgery system and method.

Background technology

Before surgical treatment at present, first need to carry out the inspection such as CT, nuclear-magnetism (Multi Slice Mode) by patient, and generate phase The 3D model files at position are answered, and it is included on two dimensional display；Then focus or ill portion of the doctor for patient The 3D model files of position are understood and are familiar with, and formulate corresponding operation plan (order for including operation).In the hand of reality , it is necessary to which camera is imaged to operative site during art, then doctor has a look at the operative site feelings presented on display Condition or 3D model files, then sight and notice are placed on to actual operative site again, according to doctor to 3D model files Memory is performed the operation to focus or disease sites.Such case is directed to, the sight of doctor not only needs the operation in reality Toggled between position and display, also to rely on after sight leaves display and model memory is performed the operation, not only imitate Rate is low, but also can expend the substantial amounts of energy of doctor, causes the fatigue of doctor, so doctor one performs the operation, often It is exhausted, or even certain risk in surgical procedure is present.

The content of the invention

The present invention in order to solve existing surgical procedure exist the sight of doctor reality operative site and display it Between the problem of toggling, and cause the problem of less efficient.

1st, augmented reality aided surgery system, including：

3D model file load-on modules, for loading 3D model files；

Module is presented in 3D model files, and 3D models are presented by AR technologies based on AR glasses；Also carried in described glasses Camera；

Gear selection module, for providing 3D models and the matching of patient with operation automatic reclosing and the manual pattern for overlapping matching Selection；And the neutral gear provided under the pattern for overlapping matching manually selects shelves, modal position movement shelves, model rotation shelves, model The selection of big minor adjustment shelves, subassembly selection shelves, transparency adjustment shelves, model color adjustment shelves, and switched according to the selection of user To corresponding module；

Auto-matching overlaps module, under automatic reclosing match pattern, 3D models and patient with operation being carried out automatic Overlap matching；

Matching overlaps module manually, in the case where overlapping match pattern manually, 3D models and patient with operation being carried out manual Overlap matching；Matching overlaps module and included manually：

Neutral gear selects submodule, in addition to switching gear order, no longer responding any other order, 3D models being consolidated It is fixed motionless；

Modal position moves submodule, by identifying that human hand posture realizes that 3D models are mobile based on the three-dimensional under AR；

Model rotates submodule, by identifying that human hand posture realizes 3D models based on the three-dimensional rotation under AR；

Model size adjusts submodule, by identifying that human hand posture realizes that 3D models are based on size under AR and adjusted；

3D model assemblies select submodule, by identifying that human hand posture realizes the selection of each building block in 3D models；

Transparency adjusts submodule, by identifying that human hand posture realizes the adjustment of 3D model transparencies；

Model color adjusts submodule, by identifying that human hand posture realizes the adjustment of 3D model colors.

Further, the model rotation shelves in the gear selection module include X-axis rotation shelves, Y-axis rotation shelves, Z axis rotation Turn shelves；

Model rotation submodule includes：

X-axis rotary unit, by identifying that human hand posture realizes that 3D models are rotated round the X-axis of local Coordinate System；

Y-axis rotary unit, by identifying that human hand posture realizes that 3D models are rotated round the Y-axis of local Coordinate System；

Z axis rotary unit, by identifying that human hand posture realizes that 3D models are rotated round the Z axis of local Coordinate System.

Further, it is dense to include red-color concentration adjustment shelves, green for the model color adjustment shelves in the gear selection module Degree adjustment shelves, blue intensity adjustment shelves；

Model color adjustment submodule includes：

Red-color concentration unit, adjusted by the way that system is adjusted into red-color concentration under shelves, pass through thumb and forefinger under the gear Posture is pinched, two dimension moves up and down in space, controls the red colour system concentration in 3D model self color RGBA values, and gesture moves up For increase red colour system concentration, move down to reduce red colour system concentration, support intermodulation up and down；

Green concentration cell, adjusted by the way that system is adjusted into green concentration under shelves, pass through thumb and forefinger under the gear Posture is pinched, two dimension moves up and down in space, and it is concentration to control the green in 3D model self color RGBA values, and gesture moves up For increase green be concentration, move down for reduce green be concentration, support up and down intermodulation；

Blue intensity unit, adjusted by the way that system is adjusted into blue intensity under shelves, pass through thumb and forefinger under the gear Posture is pinched, two dimension moves up and down in space, controls the blue series concentration in 3D model self color RGBA values, and gesture moves up For increase blue series concentration, move down to reduce blue series concentration, support intermodulation up and down.

Further, the Auto-matching overlaps the process that module carries out 3D models and patient with operation automatic reclosing matching Comprise the following steps：

(A) situation at thoracic cavity position is directed to, i.e., two nipple marker points are set on patient with operation, and draw a black Straight line is the situation for drawing midsternal line：

Black line positions：

A1, the continuous two-dimension picture read in AR glasses camera seizure video, by spiral since two-dimension picture midpoint Shape circulates each pixel, the pixel until finding first black, is designated as point A,

B1, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, searching radius is r The point color of individual pixel is the pixel of black, is circulated and looked for down with this, until the length that all black pixel points connect into Degree is determined as that straight line determines successfully more than l with this；

Described straight length l is obtained after carrying out standard conversion according to the resolution ratio of camera shooting photo；

If circulation is broken during finding black pixel point, taken out most from the storehouse point of last spiral search record The coordinate of the latter point, continue spiral search path, the point A of black pixel point is continually looked for, until straight line determines successfully；

Locating mark points：

The straight line that c1, basis have been found, cut-off line both ends, carry out semicircle search nipple marker to rectilinear direction at twice Point, the center of circle until finding two black circular marks, i.e. nipple marker point；

D1, using a mark point in nipple marker point and straight line as feature, by with it is corresponding on 3D models Theligonum and breastbone are matched, and realize that 3D models carry out automatic reclosing matching with patient with operation.

(B) situation at abdominal cavity position is directed to, i.e., a navel marker point is set on patient with operation, and draws a black Straight line is the situation for drawing midsternal line：

Black line positions：

A2, the continuous two-dimension picture read in AR glasses camera seizure video, by spiral since two-dimension picture midpoint Shape circulates each pixel, the pixel until finding first black, is designated as point A,

B2, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, searching radius is r The point color of individual pixel is the pixel of black, is circulated and looked for down with this, until the length that all black pixel points connect into Degree is determined as that straight line determines successfully more than l with this；

Locating mark points：

C2, the mark point according to corresponding to the straight line found to both ends extension search navel, after finding stain, find half Footpath is that the point color of r pixel is the point of black, until find a point around it point of r pixel be determined as black entirely Point, it is considered as navel marker dot center point；

D2, using the navel marker point of determination and straight line as feature, pass through navel corresponding with 3D models and chest Bone is matched, and realizes that 3D models carry out automatic reclosing matching with patient with operation.

(C) for thoracic cavity, the situation at abdominal cavity position, i.e., a navel marker point and two nipples are set on patient with operation Mark point, and draw the situation that a black line draws midsternal line：

Black line positions：

A3, the continuous two-dimension picture read in AR glasses camera seizure video, by spiral since two-dimension picture midpoint Shape circulates each pixel, the pixel until finding first black, is designated as point A,

B3, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, searching radius is r The point color of individual pixel is the pixel of black, is circulated and looked for down with this, until the length that all black pixel points connect into Degree is determined as that straight line determines successfully more than l with this；

Locating mark points：

C3, the mark point according to corresponding to the straight line found to both ends extension search navel, after finding stain, find half Footpath is that the point color of r pixel is the point of black, until find a point around it point of r pixel be determined as black entirely Point, it is considered as navel marker dot center point；

D3, the mark point according to the straight line that has found and navel, on cut-off line, with navel to straight line end points to straight Semicircle search is done in line direction, the center of circle until finding two or any one black circular marks, i.e. nipple marker point；

If not finding navel marker point in step c3, directly at straight line both ends, semicircle search breast is carried out at twice Labeling head point, the center of circle until finding two black circular marks, i.e. nipple marker point；

E3, the navel marker point of determination, nipple marker point are amounted to a mark point in three mark points, and one Straight line is matched by corresponding theligonum or navel point of the mark point on 3D models, and breastbone as feature, realizes 3D Model carries out automatic reclosing matching with patient with operation.

(D) situation on head is directed to, i.e., a face mark point and two eye mark points are set on patient with operation, and Draw the situation that a black line draws nasal bone center line：

Black line positions：

A4, the continuous two-dimension picture read in AR glasses camera seizure video, by spiral since two-dimension picture midpoint Shape circulates each pixel, the pixel until finding first black, is designated as point A,

B4, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, searching radius is r The point color of individual pixel is the pixel of black, is circulated and looked for down with this, until the length that all black pixel points connect into Degree is determined as that straight line determines successfully more than l with this；

Locating mark points：

C4, the mark point according to corresponding to the straight line found to both ends extension search face, after finding stain, find half Footpath is that the point color of r pixel is the point of black, until find a point around it point of r pixel be determined as black entirely Point, it is considered as face mark point central point；

D4, the mark point according to the straight line that has found and face, on cut-off line, with face to straight line end points to straight Semicircle search is done in line direction, the center of circle until finding two or any one black circular marks, i.e. eye mark point；

If not finding face mark point in step c4, directly at straight line both ends, semicircle search eye is carried out at twice Eyeball mark point, the center of circle until finding two black circular marks, i.e. eye mark point；

E4, the face mark point of determination, eye mark point are amounted to a mark point in three mark points, and one Straight line is matched by corresponding eyes point or face point of the mark point on 3D models, and nasal bone as feature, realizes 3D Model carries out automatic reclosing matching with patient with operation.

Further, radius is that the r described in r pixel takes 1 to 3.

A kind of method that 3D models and patient with operation are carried out into automatic reclosing matching, comprises the following steps：

Step 1, patient skin is set with physical toughness model, to pass through space length sensor and identifying patient place Skin surface above table top and patient；

Step 2, the position for obtaining and identifying sight alignment actual on patient with operation body；

Described sight alignment is bilateral nipple marker point, navel marker point, breastbone of the operation consent to the patient indicia that performs the operation Center line；

Midsternal line therein is that operation consent doctor with length is the black of l with black mark stroke one on patient's breastbone Line；Bilateral nipple marker point, navel marker point are with the circular patch of black adhesive sticker viscous bilateral nipple and navel；

Step 3, by three step positioning modes carry out coincidence matching：

Step 3.1, doctor adjust AR glasses viewing area by AR glasses, and operation is presented in AR glasses viewing area and suffers from The position of person's sight alignment；It is and on two-dimensional space that 3D models and the sight alignment of patient is tentatively close；

Step 3.2, pass through the logical progress Auto-matching of augmented reality aided surgery system：

Black line positions：

Step 3.2a3, the two-dimension picture in AR glasses camera seizure video is continuously read, since two-dimension picture midpoint Each pixel is circulated by spiral shape, the pixel until finding first black, is designated as point A；

Step 3.2b3, based on the point A found, point A coordinate records to storehouse array then using point A as the center of circle, are found Radius is that the point color of r pixel is the pixel of black, is circulated and looked for down with this, connected into until all black pixel points Straight length is determined as that straight line determines successfully more than l with this；

Locating mark points：

Step 3.2c3, extend mark point corresponding to searching for navel to both ends according to the straight line found, find stain Afterwards, it is that the point color of r pixel is the point of black to find radius, until find a point around it point of r pixel sentence entirely It is set to the point of black, is considered as navel marker dot center point；

Step 3.2d3, according to the mark point of the straight line that has found and navel, on cut-off line, with navel to straight line End points does semicircle search to rectilinear direction, the center of circle until finding two or any one black circular marks, i.e. nipple Mark point；

If not finding navel marker point in step 3.2c3, directly at straight line both ends, semicircle search is carried out at twice Nipple marker point, the center of circle until finding two black circular marks, i.e. nipple marker point；

Step 3.2E3, augmented reality aided surgery system automatically moves model to patient's epithelial surface, utilizes rigid not phase Hold characteristic, model is moved in patient surface；

Model moves in patient surface, and first to making 3D model center lines parallel with midsternal line by rotation, then translation makes Center line overlaps, and then single shaft movement makes a little to overlap in three characteristic points up and down, completes matching.

Step 2, the position for obtaining and identifying sight alignment actual on patient with operation head；

Described sight alignment is bilateral eye mark point, face mark point, nasal bone of the operation consent to the patient indicia that performs the operation Center line；

Nasal bone center line therein is that operation consent doctor with length is the black of l with black mark stroke one on patient's nasal bone Line；Bilateral eye mark point, face mark point are with the circular patch of black adhesive sticker viscous bilateral eyes and face；

Step 3, by three step positioning modes carry out coincidence matching：

Step 3.2a4, the two-dimension picture in AR glasses camera seizure video is continuously read, since two-dimension picture midpoint Each pixel is circulated by spiral shape, the pixel until finding first black, is designated as point A,

Step 3.2b4, based on the point A found, point A coordinate records to storehouse array then using point A as the center of circle, are found Radius is that the point color of r pixel is the pixel of black, is circulated and looked for down with this, connected into until all black pixel points Straight length is determined as that straight line determines successfully more than l with this；

Locating mark points：

Step 3.2c4, extend mark point corresponding to searching for face to both ends according to the straight line found, find stain Afterwards, it is that the point color of r pixel is the point of black to find radius, until find a point around it point of r pixel sentence entirely It is set to the point of black, is considered as face mark point central point；

Step 3.2d4, according to the mark point of the straight line that has found and face, on cut-off line, with face to straight line End points does semicircle search to rectilinear direction, the center of circle until finding two or any one black circular marks, i.e. eyes Mark point

If not finding face mark point in step 3.2c4, directly at straight line both ends, semicircle search is carried out at twice Eye mark point, the center of circle until finding two black circular marks, i.e. eye mark point；

Step 3.2e4, augmented reality aided surgery system automatically moves model to patient's epithelial surface, utilizes rigid not phase Hold characteristic, model is moved in patient surface；

Model moves in patient surface, and first to making 3D models center line parallel with nasal bone center line by rotation, then translation makes Center line overlaps, and then single shaft movement makes a little to overlap in three characteristic points up and down, completes matching.

The invention has the advantages that：

By the augmented reality aided surgery system of the present invention, can make doctor by AR glasses etc. by the focus of patient or The 3D model file files of person's disease sites are carried in actual operative site, and doctor not only need not be by sight in actual operation Toggled between position and display, and also ensure that the notice of doctor all concentrates on operative site, while this hair Bright system can also drop 3D model files and actual operative site to combination, doctor is become more apparent upon the situation of operative site, It is also easier to plan operation plan and order before the surgery.Also really avoided sight just because of doctor in reality Toggled between operative site and display, it is possible to increase the procedure efficiency of doctor, general doctor is by sight from operative site Moving on to display needs to expend 0.25-03s time, if average stop 2-3s over the display, then moves sight Returning operative site needs to expend 0.25s, once sees that the time of screen has at least saved 2.5s time so saving.According to operation Be difficult to degree and duration limit, for when several hours, hand can largely be saved by the present invention The art time, improve procedure efficiency.Fatigue caused by the excessive frequent switching of doctor is realized, further increase are it also avoid simultaneously Procedure efficiency, and improve operation safety coefficient.

The system of the present invention can realize that 3D models quickly are carried in into operation suffers from by adjust automatically first simultaneously It with person, while can also provide manually operated, using grade mode (including neutral lock function), reduce in surgical procedure Misoperation probability, and stepping operation is relatively stable, under some gear operation be more prone to realize, it is and relatively stable, no It is easily caused an operation to have influence on overall 3D models and be loaded into patient with operation, and influences efficiency.

Brief description of the drawings

Fig. 1 is the identification schematic diagram that midsternal line and three characteristic points are identified in embodiment four；

Fig. 2 is to click identification operation chart in embodiment one；

Fig. 3 is that the middle finger of embodiment one and forefinger mutually pinch moving operation schematic diagram.

Embodiment

Embodiment one：

Augmented reality aided surgery system, including：

3D model file load-on modules, the 3D model files for loading the foundation of CT, nuclear-magnetism etc. (Multi Slice Mode) equipment are (more The 3D model files of layer scanned photograph synthesis)；

Module is presented in 3D model files, and 3D models are presented by AR technologies based on AR glasses；Can be that doctor wears AR eyes 3D models acquiescence is initially presented in front of human eye at 0.2m by mirror, AR glasses；Camera is also carried in described glasses；

Gear selection module, for providing 3D models and the matching of patient with operation automatic reclosing and the manual pattern for overlapping matching Selection；And the neutral gear provided under the pattern for overlapping matching manually selects shelves, modal position movement shelves, model rotation shelves, model The selection of big minor adjustment shelves, subassembly selection shelves, transparency adjustment shelves, model color adjustment shelves, and switched according to the selection of user To corresponding module；Select gear not only can double-click switching by thumb and forefinger, can also support what is spoken by user Voice command controls；In handoff procedure corresponding to gear selection, viewing area provides text prompt in AR glasses, including but not It is limited to position：Upper left, in upper, upper right, it is left, right in, lower-left, under, bottom right etc.；After gear switches successfully, sound equipment language is used Sound is prompted；Gear confirms to complete by single-click operation.Illustrate single-click operation with reference to Fig. 2, a-b-c is performed integrally one in Fig. 2 Secondary single-click operation, i.e., operation technique is gathered by camera in the front of AR glasses, thumb and forefinger separate (corresponding a), then Forefinger is drawn close to thumb, and thumb and forefinger closure (corresponding b), then forefinger and thumb are remote, and thumb and forefinger are separately (right herein Answer c), entirely form a single-click operation；Double-clicking can be that continuous single-click operation is formed twice.

Model rotation shelves in the gear selection module include X-axis rotation shelves, Y-axis rotation shelves, Z axis rotation shelves；

Model color adjustment shelves in the gear selection module include red-color concentration adjustment shelves, green concentration adjusts shelves, Blue intensity adjusts shelves；

Neutral gear selection submodule (N), in addition to switching gear order, no longer responding any other order, by 3D moulds Type is fixed；When 3D models overlap with patient with operation the match is successful when, in order to avoid movement causes to have overlapped matching again Situation change, when 3D models overlap with patient with operation matching, system modulation makes neutral gear selection sub to neutral gear selection shelves Module ensures that 3D models are fixed；

Modal position movement submodule (P), by identifying that human hand posture realizes that 3D models are mobile based on the three-dimensional under AR；Such as The posture that pinches that fruit is mutually close to using thumb and forefinger is instructed as identification, and posture state is pinched what thumb and forefinger were mutually close to Lower movement (as shown in Figure 3), correspondingly control three-dimensional of the 3D models in space mobile by the locus of mobile thumb and forefinger； It is directed to for AR glasses, it is only necessary to identify that human hand posture is corresponding by AR glasses and start, the calculation of the movement of corresponding implementation model Method is voluntarily realized by the algorithm inside existing AR glasses；In program development human hand is identified merely by image processing techniques Posture, human hand posture then will be corresponded to as enabled instruction, in the mould that data open interface startup AR glasses corresponding to AR carry Type movement is handled；

Model rotates submodule, by identifying that human hand posture realizes 3D models based on the three-dimensional rotation under AR；It can pass through System is adjusted under model rotation shelves, posture is pinched by thumb and forefinger under the gear, two dimension moves up and down in space, Control the three-dimensional rotation of 3D models；

Described model rotation submodule includes：

X-axis rotary unit (RX), by identifying that human hand posture realizes that 3D models are revolved round the X-axis of local Coordinate System Turn；It can be rotated by the way that system is adjusted into X-axis under shelves, posture is pinched by thumb and forefinger under the gear, in space two Dimension is moved up and down, and control 3D models are rotated round the X-axis of local Coordinate System, and gesture is moved up to rotate forward, moving down to invert, can Support intermodulation up and down；

Y-axis rotary unit (RY), by identifying that human hand posture realizes that 3D models are revolved round the Y-axis of local Coordinate System Turn；It can be rotated by the way that system is adjusted into Y-axis under shelves, posture is pinched by thumb and forefinger under the gear, in space two Dimension is moved up and down, and control 3D models are rotated round the Y-axis of local Coordinate System, and gesture is moved up to rotate forward, moving down to invert, can Support intermodulation up and down；

Z axis rotary unit (RZ), by identifying that human hand posture realizes that 3D models are revolved round the Z axis of local Coordinate System Turn；It can be rotated by the way that system is adjusted into Z axis under shelves, posture is pinched by thumb and forefinger under the gear, in space two Dimension is moved up and down, and control 3D models are rotated round the Z axis of local Coordinate System, and gesture is moved up to rotate forward, moving down to invert, can Support intermodulation up and down.

Why 3D models are individually rotated in three axles of local Coordinate System respectively, be because if by three axles Integrated straight taps into row rotation and tends not to enough to make 3D models coincide with the human body of patient with operation to match under whole coordinate system, single Axially mutually it can realize fine setting, it is easier to 3D models is coincided with the human body of patient with operation and match.

Model size regulation submodule (S), by identifying that human hand posture realizes that 3D models are based on size under AR and adjusted；Can be with By the way that system is adjusted under the big minor adjustment shelves of model, posture is pinched by thumb and forefinger under the gear, it is two-dimentional in space Move up and down, control the amplification and diminution of 3D models itself, gesture is moved up to amplify, moving down to reduce, and can support intermodulation up and down；

3D model assemblies selection submodule (C), by identifying that human hand posture realizes that each building block is (corresponding in 3D models In a certain position of patient with operation) selection；Thumb can be passed through under the gear by the way that system is adjusted under subassembly selection shelves Posture is pinched with forefinger, two dimension moves up and down in space, selects each building block of 3D models, selected portion in order Part, which is given, is highlighted prompting, gesture move up for sequential selection, move down for inverted sequence select, can support up and down intermodulation, select one wheel The operation for once selecting all parts entirely is provided afterwards；

Transparency adjustment submodule (A), by identifying that human hand posture realizes the adjustment of 3D model transparencies；Can pass through by System is adjusted under transparency adjustment shelves, and posture is pinched by thumb and forefinger under the gear, and two dimension moves up and down in space, Control 3D model self color RGBA values in transparency, gesture move up for reduce transparency up to complete opaque state, Move down and disappeared for increase transparency up to fully transparent, intermodulation up and down can be supported；

Model color adjusts submodule, by identifying that human hand posture realizes the adjustment of 3D model colors；Can be by that will be System is adjusted under model color adjustment shelves, and posture is pinched by thumb and forefinger under the gear, and two dimension moves up and down in space, The red, green, blue colour system concentration in 3D model self color RGBA values is controlled, gesture is moved up as increase colour system concentration, moved down to subtract Few colour system concentration, can support intermodulation up and down.

Described model color adjustment submodule includes：

Red-color concentration unit (R), adjusted by the way that system is adjusted into red-color concentration under shelves, pass through thumb and food under the gear Finger pinches posture, and two dimension moves up and down in space, the red colour system concentration in control 3D model self color RGBA values, in gesture Move as increase red colour system concentration, move down to reduce red colour system concentration, intermodulation up and down can be supported；

Green concentration cell (G), adjusted by the way that system is adjusted into green concentration under shelves, pass through thumb and food under the gear Finger pinches posture, and two dimension moves up and down in space, and it is concentration to control the green in 3D model self color RGBA values, in gesture Move for increase green be concentration, move down for reduce green be concentration, can support up and down intermodulation；

Blue intensity unit (B), adjusted by the way that system is adjusted into blue intensity under shelves, pass through thumb and food under the gear Finger pinches posture, and two dimension moves up and down in space, the blue series concentration in control 3D model self color RGBA values, in gesture Move as increase blue series concentration, move down to reduce blue series concentration, intermodulation up and down can be supported.

Why model color is adjusted into submodule to adjust according to tri- colors of RGB respectively, be not only able to realize panchromatic system Adjustment, and the color need corresponding with patient with operation of 3D models when 3D models overlap with patient with operation matching, can also be controlled The color at the position to be performed the operation produces certain discrimination, to carry out actual operation with reference to 3D models.

Embodiment two：

Auto-matching described in present embodiment overlaps the mistake that module carries out 3D models and patient with operation automatic reclosing matching Journey comprises the following steps：

For the situation at thoracic cavity position, i.e., two nipple marker points are set on patient with operation, and it is straight to draw a black Line is the situation for drawing midsternal line：

Black line positions：

A1, the continuous two-dimension picture read in AR glasses camera seizure video, by spiral since two-dimension picture midpoint Shape circulates each pixel, until find first black pixel (color depth error can according to reality situation Choose, color depth error can be judged as black within 20% in present embodiment), point A is designated as,

B1, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, find radius 1-3 For the pixel of black, (color depth error can be chosen the point color of individual pixel according to the situation of reality, in present embodiment Color depth error can be judged as black within 20%), circulated and looked for down with this, until all black pixel points connect into Straight length more than l, described l be 10cm (doctor can adjust the length parameter according to practical operation), be determined as directly with this Line determines successfully；In the search procedure of the black pixel point of reality, it may appear that the situation of straight line bifurcated, then select length most long The straight line that is formed as final black pixel point of straight line；

Locating mark points：

Other modules and parameter are identical with embodiment one.

Embodiment three：

For the situation at abdominal cavity position, i.e., a navel marker point is set on patient with operation, and it is straight to draw a black Line is the situation for drawing midsternal line：

Black line positions：

A2, the continuous two-dimension picture read in AR glasses camera seizure video, by spiral since two-dimension picture midpoint Shape circulates each pixel, until find first black pixel (color depth error can according to reality situation Choose, color depth error can be judged as black within 20% in present embodiment), point A is designated as,

B2, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, find radius 1-3 For the pixel of black, (color depth error can be chosen the point color of individual pixel according to the situation of reality, in present embodiment Color depth error can be judged as black within 20%), circulated and looked for down with this, until all black pixel points connect into Straight length more than l, described l be 10cm (doctor can adjust the length parameter according to practical operation), be determined as directly with this Line determines successfully；In the search procedure of the black pixel point of reality, it may appear that the situation of straight line bifurcated, then select length most long The straight line that is formed as final black pixel point of straight line；

Locating mark points：

C2, the mark point according to corresponding to the straight line found to both ends extension search navel, after finding stain, find half The point color of 1-3, footpath pixel is the point of black, until find a point around it point of 1-3 pixel be to be determined as black entirely Point, be considered as navel marker dot center point；

Other modules and parameter are identical with embodiment one.

Embodiment four：

For the situation in thoracic cavity, abdominal cavity position, i.e., a navel marker point and two nipple marks are set on patient with operation Remember point, and draw the situation that a black line draws midsternal line：

Black line positions：

A3, the continuous two-dimension picture read in AR glasses camera seizure video, by spiral since two-dimension picture midpoint Shape circulates each pixel, until find first black pixel (color depth error can according to reality situation Choose, color depth error can be judged as black within 20% in present embodiment), point A is designated as,

B3, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, find radius 1-3 For the pixel of black, (color depth error can be chosen the point color of individual pixel according to the situation of reality, in present embodiment Color depth error can be judged as black within 20%), circulated and looked for down with this, until all black pixel points connect into Straight length more than l, described l be 10cm (doctor can adjust the length parameter according to practical operation), be determined as directly with this Line determines successfully；In the search procedure of the black pixel point of reality, it may appear that the situation of straight line bifurcated, then select length most long The straight line that is formed as final black pixel point of straight line；

Locating mark points：

C3, the mark point according to corresponding to the straight line found to both ends extension search navel, after finding stain, find half The point color of 1-3, footpath pixel is the point of black, until find a point around it point of 1-3 pixel be to be determined as black entirely Point, be considered as navel marker dot center point；

D3, the mark point according to the straight line that has found and navel, on cut-off line, with navel to straight line end points to straight Semicircle search is done in line direction, until finding two or the center of circle of any one black circular marks according to upper one step process, i.e., Nipple marker point；The error in the center of circle determines that the error in the center of circle can be in 5mm in present embodiment according to actual requirement；

Above procedure as shown in figure 1, each square represents a pixel in Fig. 1, the sequence number in square frame represent find it is suitable Sequence, spiral is searched since 1, and black represents nipple marker point, navel marker point and the black line finally found in Fig. 1, Actual to occur two black lines in Fig. 1, one short broken string is noise (may be due to caused by former scar), and a long line is The straight line of actual breastbone line correspondences, when the length according to doctor's setting-out, choose the straight length that black pixel point connects into and surpass Cross l then can cancelling noise, leave the straight lines of breastbone line correspondences.

Other modules and parameter are identical with embodiment one.

Embodiment five：

For the situation on head, i.e., a face mark point and two eye mark points are set on patient with operation, and draw Go out the situation that a black line draws nasal bone center line：

Black line positions：

A4, the continuous two-dimension picture read in AR glasses camera seizure video, by spiral since two-dimension picture midpoint Shape circulates each pixel, until find first black pixel (color depth error can according to reality situation Choose, color depth error can be judged as black within 20% in present embodiment), point A is designated as,

B4, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, find radius 1-3 For the pixel of black, (color depth error can be chosen the point color of individual pixel according to the situation of reality, in present embodiment Color depth error can be judged as black within 20%), circulated and looked for down with this, until all black pixel points connect into Straight length more than l, doctor can adjust l according to practical operation, be determined as that straight line determines successfully with this；In the black picture of reality In the search procedure of vegetarian refreshments, it may appear that the situation of straight line bifurcated, then select the most long straight line of length as final black picture element The straight line that point is formed；

Locating mark points：

C4, the mark point according to corresponding to the straight line found to both ends extension search face, after finding stain, find half The point color of 1-3, footpath pixel is the point of black, until find a point around it point of 1-3 pixel be to be determined as black entirely Point, be considered as face mark point central point；

D4, the mark point according to the straight line that has found and face, on cut-off line, with face to straight line end points to straight Semicircle search is done in line direction, until finding two or the center of circle of any one black circular marks according to upper one step process, i.e., Eye mark point；The error in the center of circle determines that the error in the center of circle can be in 5mm in present embodiment according to actual requirement；

Other modules and parameter are identical with embodiment one.

Embodiment six：

Step 1, patient skin is set for (other positions are arranged to non-rigid dummy model with physical toughness model Body), skin surface above patient's place table top and patient is identified by space length sensor；

Midsternal line therein is that operation consent doctor with length is the black of l with black mark stroke one on patient's breastbone Line, l at least 10cm；Bilateral nipple marker point, navel marker point are the viscous bilateral breast of the circular patch of black adhesive sticker with a diameter of 3cm On head and navel；

Step 3, by three step positioning modes carry out coincidence matching：

Step 3.1, doctor adjust AR glasses viewing area by AR glasses, and operation is presented in AR glasses viewing area and suffers from The position of person's sight alignment；And it is on two-dimensional space that 3D models and the sight alignment of patient is tentatively close, error range exists In 10cm；

Black line positions：

Step 3.2a3, the two-dimension picture in AR glasses camera seizure video is continuously read, since two-dimension picture midpoint Each pixel is circulated by spiral shape, (color depth error can be according to reality until the pixel that finds first black Situation choose, color depth error can be judged as black within 20% in present embodiment), be designated as point A,

Step 3.2b3, based on the point A found, point A coordinate records to storehouse array then using point A as the center of circle, are found The point color of 1-3 pixel of radius for black pixel (color depth error can according to reality situation choose, this implementation Color depth error can be judged as black within 20% in mode), circulated and looked for down with this, until all black pixel points The straight length connected into is 10cm (doctor can adjust the length parameter according to practical operation) more than l, described l, is sentenced with this It is set to straight line to determine successfully；In the search procedure of the black pixel point of reality, it may appear that the situation of straight line bifurcated, then selection are grown The straight line that the most long straight line of degree is formed as final black pixel point；

Locating mark points：

Step 3.2c3, extend mark point corresponding to searching for navel to both ends according to the straight line found, find stain Afterwards, the point color for finding 1-3 pixel of radius is the point of black, until find a point around it point of 1-3 pixel be entirely It is determined as the point of black, is considered as navel marker dot center point；

Step 3.2d3, according to the mark point of the straight line that has found and navel, on cut-off line, with navel to straight line End points does semicircle search to rectilinear direction, until finding two or any one black circular marks according to upper one step process The center of circle, i.e. nipple marker point；The error in the center of circle determines that the error in the center of circle can be in 5mm in present embodiment according to actual requirement It is interior；

Model moves in patient surface, and first to making 3D model center lines parallel with midsternal line by rotation, then translation makes Center line overlaps, and then single shaft movement makes a little to overlap in three characteristic points up and down, completes matching (as long as bright spot weight in actual use Conjunction, error range is in 5mm).

Embodiment seven：

Step 3, by three step positioning modes carry out coincidence matching：

Step 3.2a4, the two-dimension picture in AR glasses camera seizure video is continuously read, since two-dimension picture midpoint Each pixel is circulated by spiral shape, (color depth error can be according to reality until the pixel that finds first black Situation choose, color depth error can be judged as black within 20% in present embodiment), be designated as point A,

Step 3.2b4, based on the point A found, point A coordinate records to storehouse array then using point A as the center of circle, are found The point color of 1-3 pixel of radius for black pixel (color depth error can according to reality situation choose, this implementation Color depth error can be judged as black within 20% in mode), circulated and looked for down with this, until all black pixel points The straight length connected into is 10cm more than l, described l, is determined as that straight line determines successfully with this；In the black pixel point of reality Search procedure in, it may appear that the situation of straight line bifurcated, then select the most long straight line of length as final black pixel point structure Into straight line；

Locating mark points：

Step 3.2c4, extend mark point corresponding to searching for face to both ends according to the straight line found, find stain Afterwards, the point color for finding 1-3 pixel of radius is the point of black, until find a point around it point of 1-3 pixel be entirely It is determined as the point of black, is considered as face mark point central point；

Step 3.2d4, according to the mark point of the straight line that has found and face, on cut-off line, with face to straight line End points does semicircle search to rectilinear direction, until finding two or any one black circular marks according to upper one step process The center of circle, i.e. eye mark point；The error in the center of circle determines that the error in the center of circle can be in 5mm in present embodiment according to actual requirement It is interior；

Claims

1. augmented reality aided surgery system, it is characterised in that including：

3D model file load-on modules, for loading 3D model files；

Module is presented in 3D model files, and 3D models are presented by AR technologies based on AR glasses；Also with shooting in described glasses Head；

Gear selection module, selected for providing 3D models with the matching of patient with operation automatic reclosing and the manual pattern for overlapping matching Select；And neutral gear selection shelves, modal position movement shelves, model rotation shelves, the model provided under the pattern for overlapping matching manually is big The selection of minor adjustment shelves, subassembly selection shelves, transparency adjustment shelves, model color adjustment shelves, and be switched to according to the selection of user Corresponding module；

Auto-matching overlaps module, under automatic reclosing match pattern, 3D models and patient with operation to be carried out into automatic reclosing Matching；

Matching overlaps module manually, in the case where overlapping match pattern manually, 3D models to be overlapped manually with patient with operation Matching；Matching overlaps module and included manually：

Neutral gear selects submodule, and in addition to switching gear order, no longer responding any other order, 3D models are fixed not It is dynamic；

2. augmented reality aided surgery system according to claim 1, it is characterised in that in the gear selection module Model rotation shelves include X-axis rotation shelves, Y-axis rotation shelves, Z axis rotation shelves；

Model rotation submodule includes：

3. augmented reality aided surgery system according to claim 1, it is characterised in that in the gear selection module Model color adjustment shelves include red-color concentration adjustment shelves, green concentration adjustment shelves, blue intensity adjustment shelves；

Model color adjustment submodule includes：

Red-color concentration unit, adjusted under shelves by the way that system is adjusted into red-color concentration, pinched under the gear by thumb and forefinger Posture, in space two dimension move up and down, control 3D model self color RGBA values in red colour system concentration, gesture move up for increasing Add red colour system concentration, move down to reduce red colour system concentration, support intermodulation up and down；

Green concentration cell, adjusted under shelves by the way that system is adjusted into green concentration, pinched under the gear by thumb and forefinger Posture, in space two dimension move up and down, it is concentration to control the green in 3D model self color RGBA values, and gesture is moved up as increasing It is concentration to add green to be concentration, move down to reduce green, supports intermodulation up and down；

Blue intensity unit, adjusted under shelves by the way that system is adjusted into blue intensity, pinched under the gear by thumb and forefinger Posture, in space two dimension move up and down, control 3D model self color RGBA values in blue series concentration, gesture move up for increasing Add blue series concentration, move down to reduce blue series concentration, support intermodulation up and down.

4. according to the augmented reality aided surgery system described in claim 1,2 or 3, it is characterised in that the Auto-matching weight The process that matched moulds block carries out 3D models with patient with operation automatic reclosing matching comprises the following steps：

For the situation at thoracic cavity position, i.e., two nipple marker points are set on patient with operation, and draw a black line i.e. Draw the situation of midsternal line：

Black line positions：

A1, the continuous two-dimension picture read in AR glasses camera seizure video, are followed since two-dimension picture midpoint by spiral shape Each pixel of ring, the pixel until finding first black, point A is designated as,

B1, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, searching radius is r picture The point color of element is the pixel of black, is circulated and looked for down with this, until the straight length that all black pixel points connect into surpasses L is crossed, is determined as that straight line determines successfully with this；

If circulation is broken during finding black pixel point, last is taken out from the storehouse point of last spiral search record The coordinate of individual point, continue spiral search path, the point A of black pixel point is continually looked for, until straight line determines successfully；

Locating mark points：

The straight line that c1, basis have been found, cut-off line both ends, semicircle search nipple marker point is carried out to rectilinear direction at twice, The center of circle until finding two black circular marks, i.e. nipple marker point；

D1, using a mark point in nipple marker point and straight line as feature, by with the corresponding nipple on 3D models Point and breastbone are matched, and realize that 3D models carry out automatic reclosing matching with patient with operation.

5. according to the augmented reality aided surgery system described in claim 1,2 or 3, it is characterised in that the Auto-matching weight The process that matched moulds block carries out 3D models with patient with operation automatic reclosing matching comprises the following steps：

For the situation at abdominal cavity position, i.e., a navel marker point is set on patient with operation, and draws a black line i.e. Draw the situation of midsternal line：

Black line positions：

A2, the continuous two-dimension picture read in AR glasses camera seizure video, are followed since two-dimension picture midpoint by spiral shape Each pixel of ring, the pixel until finding first black, point A is designated as,

B2, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, searching radius is r picture The point color of element is the pixel of black, is circulated and looked for down with this, until the straight length that all black pixel points connect into surpasses L is crossed, is determined as that straight line determines successfully with this；

Locating mark points：

C2, the mark point according to corresponding to the straight line found to both ends extension search navel, after finding stain, finding radius is The point color of r pixel is the point of black, until find a point around it point of r pixel be the point that is determined as black entirely, It is considered as navel marker dot center point；

D2, using the navel marker point of determination and straight line as feature, entered by navel corresponding with 3D models and breastbone Row matching, realize that 3D models carry out automatic reclosing matching with patient with operation.

6. according to the augmented reality aided surgery system described in claim 1,2 or 3, it is characterised in that the Auto-matching weight The process that matched moulds block carries out 3D models with patient with operation automatic reclosing matching comprises the following steps：

For the situation in thoracic cavity, abdominal cavity position, i.e., a navel marker point and two nipple marker points are set on patient with operation, And draw the situation that a black line draws midsternal line：

Black line positions：

A3, the continuous two-dimension picture read in AR glasses camera seizure video, are followed since two-dimension picture midpoint by spiral shape Each pixel of ring, the pixel until finding first black, point A is designated as,

B3, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, searching radius is r picture The point color of element is the pixel of black, is circulated and looked for down with this, until the straight length that all black pixel points connect into surpasses L is crossed, is determined as that straight line determines successfully with this；

Locating mark points：

C3, the mark point according to corresponding to the straight line found to both ends extension search navel, after finding stain, finding radius is The point color of r pixel is the point of black, until find a point around it point of r pixel be the point that is determined as black entirely, It is considered as navel marker dot center point；

D3, the mark point according to the straight line that has found and navel, on cut-off line, with navel to straight line end points to straight line side To semicircle search is done, the center of circle until finding two or any one black circular marks, i.e. nipple marker point；

If not finding navel marker point in step c3, directly at straight line both ends, semicircle search nipple mark is carried out at twice Remember point, the center of circle until finding two black circular marks, i.e. nipple marker point；

E3, the navel marker point of determination, nipple marker point are amounted to a mark point in three mark points, and straight line As feature, matched by corresponding theligonum or navel point of the mark point on 3D models, and breastbone, realize 3D models Automatic reclosing matching is carried out with patient with operation.

7. according to the augmented reality aided surgery system described in claim 1,2 or 3, it is characterised in that the Auto-matching weight The process that matched moulds block carries out 3D models with patient with operation automatic reclosing matching comprises the following steps：

For the situation on head, i.e., a face mark point and two eye mark points are set on patient with operation, and draw one Bar black line is to draw the situation of nasal bone center line：

Black line positions：

A4, the continuous two-dimension picture read in AR glasses camera seizure video, are followed since two-dimension picture midpoint by spiral shape Each pixel of ring, the pixel until finding first black, point A is designated as,

B4, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, searching radius is r picture The point color of element is the pixel of black, is circulated and looked for down with this, until the straight length that all black pixel points connect into surpasses L is crossed, is determined as that straight line determines successfully with this；

Locating mark points：

C4, the mark point according to corresponding to the straight line found to both ends extension search face, after finding stain, finding radius is The point color of r pixel is the point of black, until find a point around it point of r pixel be the point that is determined as black entirely, It is considered as face mark point central point；

D4, the mark point according to the straight line that has found and face, on cut-off line, with face to straight line end points to straight line side To semicircle search is done, the center of circle until finding two or any one black circular marks, i.e. eye mark point；

If not finding face mark point in step c4, directly at straight line both ends, semicircle search eyes mark is carried out at twice Remember point, the center of circle until finding two black circular marks, i.e. eye mark point；

E4, the face mark point of determination, eye mark point are amounted to a mark point in three mark points, and straight line As feature, matched by corresponding eyes point or face point of the mark point on 3D models, and nasal bone, realize 3D models Automatic reclosing matching is carried out with patient with operation.

8. augmented reality aided surgery system according to claim 7, it is characterised in that radius is the r described in r pixel Take 1 to 3.

A kind of 9. method that 3D models and patient with operation are subjected to automatic reclosing matching, it is characterised in that comprise the following steps：

Step 1, patient skin is set to be the table top where identifying patient by space length sensor with physical toughness model With skin surface above patient；

Described sight alignment is bilateral nipple marker point, navel marker point, midsternal line of the operation consent to the patient indicia that performs the operation；

Midsternal line therein is the black line that operation consent doctor is l with length with black mark stroke one on patient's breastbone；It is double Side nipple marker point, navel marker point are with the circular patch of black adhesive sticker viscous bilateral nipple and navel；

Step 3, by three step positioning modes carry out coincidence matching：

Step 3.1, doctor adjust AR glasses viewing area by AR glasses, and patient with operation three is presented in AR glasses viewing area The position of one line of point；It is and on two-dimensional space that 3D models and the sight alignment of patient is tentatively close；

Black line positions：

Step 3.2a3, the two-dimension picture in AR glasses camera seizure video is continuously read, spiral shell is pressed since two-dimension picture midpoint Rotation shape circulates each pixel, the pixel until finding first black, is designated as point A；

Step 3.2b3, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, radius is found Point color for r pixel is the pixel of black, is circulated and looked for down with this, until the straight line that all black pixel points connect into Length is determined as that straight line determines successfully more than l with this；

Locating mark points：

Step 3.2c3, extend mark point corresponding to searching for navel to both ends according to the straight line found, after finding stain, seek The point color for looking for radius to be r pixel is the point of black, until find a point around it point of r pixel be entirely be determined as it is black The point of color, it is considered as navel marker dot center point；

Step 3.2d3, according to the mark point of the straight line that has found and navel, on cut-off line, with navel to straight line end points Semicircle search is done to rectilinear direction, the center of circle until finding two or any one black circular marks, i.e. nipple marker Point；

If not finding navel marker point in step 3.2c3, directly at straight line both ends, semicircle search nipple is carried out at twice Mark point, the center of circle until finding two black circular marks, i.e. nipple marker point；

Step 3.2E3, augmented reality aided surgery system automatically moves model to patient's epithelial surface, utilizes the incompatible spy of rigidity Property, model is moved in patient surface；

Model moves in patient surface, and first to making 3D model center lines parallel with midsternal line by rotation, then translation makes center line Overlap, then single shaft movement makes a little to overlap in three characteristic points up and down, completes matching.

A kind of 10. method that 3D models and patient with operation are subjected to automatic reclosing matching, it is characterised in that comprise the following steps：

Described sight alignment is bilateral eye mark point, face mark point, nasal bone center line of the operation consent to the patient indicia that performs the operation；

Nasal bone center line therein is the black line that operation consent doctor is l with length with black mark stroke one on patient's nasal bone；It is double Branch hole eyeball mark point, face mark point are with the circular patch of black adhesive sticker viscous bilateral eyes and face；

Step 3, by three step positioning modes carry out coincidence matching：

Step 3.2a4, the two-dimension picture in AR glasses camera seizure video is continuously read, spiral shell is pressed since two-dimension picture midpoint Rotation shape circulates each pixel, the pixel until finding first black, is designated as point A,

Step 3.2b4, based on the point A found, by point A coordinate records to storehouse array, then using point A as the center of circle, radius is found Point color for r pixel is the pixel of black, is circulated and looked for down with this, until the straight line that all black pixel points connect into Length is determined as that straight line determines successfully more than l with this；

Locating mark points：

Step 3.2c4, extend mark point corresponding to searching for face to both ends according to the straight line found, after finding stain, seek The point color for looking for radius to be r pixel is the point of black, until find a point around it point of r pixel be entirely be determined as it is black The point of color, it is considered as face mark point central point；

Step 3.2d4, according to the mark point of the straight line that has found and face, on cut-off line, with face to straight line end points Semicircle search is done to rectilinear direction, the center of circle until finding two or any one black circular marks, i.e. eye mark Point

If not finding face mark point in step 3.2c4, directly at straight line both ends, semicircle search eyes are carried out at twice Mark point, the center of circle until finding two black circular marks, i.e. eye mark point；

Step 3.2e4, augmented reality aided surgery system automatically moves model to patient's epithelial surface, utilizes the incompatible spy of rigidity Property, model is moved in patient surface；

Model moves in patient surface, and first to making 3D models center line parallel with nasal bone center line by rotation, then translation makes center line Overlap, then single shaft movement makes a little to overlap in three characteristic points up and down, completes matching.