WO2020172779A1 - Camera system for transcranial magnetic stimulation therapy - Google Patents

Camera system for transcranial magnetic stimulation therapy Download PDF

Info

Publication number
WO2020172779A1
WO2020172779A1 PCT/CN2019/076100 CN2019076100W WO2020172779A1 WO 2020172779 A1 WO2020172779 A1 WO 2020172779A1 CN 2019076100 W CN2019076100 W CN 2019076100W WO 2020172779 A1 WO2020172779 A1 WO 2020172779A1
Authority
WO
WIPO (PCT)
Prior art keywords
head
patient
camera
magnetic stimulation
treatment
Prior art date
Application number
PCT/CN2019/076100
Other languages
French (fr)
Chinese (zh)
Inventor
孙聪
王波
蔡胜安
Original Assignee
武汉资联虹康科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 武汉资联虹康科技股份有限公司 filed Critical 武汉资联虹康科技股份有限公司
Priority to CN201980001093.0A priority Critical patent/CN110300993B/en
Priority to PCT/CN2019/076100 priority patent/WO2020172779A1/en
Publication of WO2020172779A1 publication Critical patent/WO2020172779A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
    • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/30—Assessment of water resources

Definitions

  • the invention relates to the technical field of medical equipment, in particular to a camera system used for transcranial magnetic stimulation diagnosis and treatment.
  • the total number of patients with mental illness in my country has exceeded 100 million, but the public awareness rate of mental illness is less than 50%, and the rate of consultation is even lower. At present, about 20% of these mental patients receive timely treatment, and 80% of the mental patients are not treated in time, or even the most basic treatment.
  • the number of patients with mental illness is as high as 16 million.
  • global mental illness medications have exceeded 36 billion US dollars, accounting for 5% of total drug sales.
  • domestically the current mental illness drug market is still relatively small, accounting for about 1.5% of the total hospital sales.
  • Transcranial Magnetic Stimulation is a technique that uses a pulsed magnetic field to generate an electric current in the local cerebral cortex to temporarily activate or inhibit the cortex.
  • Transcranial magnetic stimulation technology has been widely used in epilepsy, neuropsychology, rehabilitation, pediatrics and other fields.
  • medical staff need to aim the treatment equipment at the patient’s head to be treated. Discomfort caused by the patient’s long-term maintenance of the same posture will cause the patient to change the head posture.
  • Medical staff need to always pay attention to the treatment instrument and Whether the patient’s head is aligned or not, this method is costly, time-consuming, and large in error. Therefore, medical staff urgently need a fast, high-precision head posture tracking system that can truly feedback the patient’s head posture to the mechanical control equipment .
  • the traditional TMS treatment method usually includes the following steps: the operator finds the treatment magnetic stimulation point area on the patient's head (location based on experience); places the TMS coil on this area, ensuring that the center of the coil is directly opposite the magnetic stimulation point, and the coil plane is with the scalp Tangent; Use mechanical devices to clamp and fix the TMS coil; start TMS and start treatment. Because TMS treatment usually takes a long time, according to the course of treatment, the magnetic stimulation points that are located purely by experience and memory cannot guarantee the accuracy of each treatment.
  • the problem of TMS accurate repeat positioning has been studied worldwide, but most of the other methods currently are based on the infrared positioning method of the light guide ball.
  • a special light guide ball must be worn for the patient and the TMS coil every time it is positioned, and the positioning is performed according to the relative position of the light guide ball in the infrared camera. Because the position of wearing is not fixed each time, it is necessary to additionally use a special positioning pen to determine the relative position of the light guide ball and the human head through complex positioning. Therefore, the operation is quite inconvenient and inefficient.
  • the time for wearing and positioning the light guide ball is about 30 minutes per person. Since a TMS treatment only takes 10-20 minutes, the current positioning method is not suitable for TMS Repeat positioning efficiently.
  • a more accurate positioning method can be a transcranial magnetic stimulation navigation system.
  • a machine vision positioning navigation system such as the Chinese patent "Transcranial Magnetic Stimulation Navigation System and Transcranial Magnetic Stimulation Coil Positioning Method” 201210281507.X , 201210281472.X; the other is an optical positioning navigation system, such as the Chinese patent "a navigational transcranial magnetic stimulation treatment system” 201010235826.8, "a calibration device and method for repetitive transcranial magnetic stimulation optical positioning navigation system” 201010235828.7.
  • the machine vision positioning and navigation system requires a positioning cap.
  • the thickness of the positioning cap increases the distance between the coil and the cerebral cortex, increases the stimulation intensity, and causes a waste of energy, especially when transcranial magnetic stimulation and EEG recording are required at the same time.
  • Positioning caps and EEG electrode caps will increase the distance between the coil and the cerebral cortex.
  • RGBD camera is one of the cameras that can obtain image information and distance information at the same time. Therefore, the present invention combines the advantages that the RGBD camera can obtain image information and distance information at the same time.
  • the RGBD camera is used to photograph the patient's posture, and then the photographed image is processed to locate the therapeutic magnet. Stimulation point, without the need to wear a light guide ball to ensure the precise positioning of the magnetic stimulation point during each treatment.
  • the purpose of the present invention is to solve the problems existing in the prior art and provide a camera system for transcranial magnetic stimulation diagnosis and treatment.
  • the patient's head posture can be obtained quickly and accurately, and adjusted according to the head posture
  • Treatment of magnetic stimulation points improves the accuracy of magnetic stimulation point positioning and repeated positioning in TMS treatment.
  • a camera system for transcranial magnetic stimulation diagnosis and treatment comprising a first photographing subsystem for head modeling, a second photographing subsystem for diagnosis and treatment, and an intelligent terminal.
  • the first photographing subsystem includes 3D scanning
  • the second photographing subsystem includes a 3D camera, a lying bed, and a headrest; the first photographing subsystem sends the photographed head data to the intelligent terminal for head modeling, and obtains A 3D head model that identifies facial feature points; the second photographing subsystem sends the face data captured in real time to a smart terminal for face detection, and the smart terminal compares the detected face image with the 3D head model The matching is performed to obtain the location information of the magnetic stimulation point for transcranial magnetic stimulation diagnosis and treatment.
  • the 3D scanner includes a 3D camera and a rotating bracket, the 3D camera is mounted on the rotating bracket, the rotating bracket is driven to rotate by a motor, and the motor is electrically connected to the smart terminal;
  • the intelligent terminal controls the motor to drive the rotating bracket to rotate at a constant speed, so that the 3D camera moves in a circular motion around the patient's head at a constant speed, and collects 3D image data of the patient's head from all directions.
  • the 3D scanner includes a plurality of 3D cameras and a fixed bracket, and the plurality of 3D cameras are all installed on the fixed bracket; when collecting 3D image data of the patient's head, the plurality of 3D cameras are controlled by an intelligent terminal The camera simultaneously collects 3D image data of the patient's head from different directions.
  • the image data captured by the 3D camera includes color images, depth images, and 3D point cloud images.
  • the 3D camera in the second photographing sub-system is arranged above the patient's face and can fully incorporate his face into the photographing range.
  • the positioning cap is a white hood made of elastic material, which is used to cover the patient's hair; the positioning cap is provided with a number of Mark points to facilitate the 3D camera to collect image data.
  • a camera-based transcranial magnetic stimulation diagnosis and treatment modeling system adopts the camera system, and the modeling method of the modeling system includes the following steps:
  • S1 Start the smart terminal, collect 3D image data of the patient's head wearing the positioning cap from various directions through the 3D scanner, and send the collected 3D image data to the smart terminal;
  • step S2 the method for integrating the 3D image data collected from various directions is: calculating the matching relationship between the images by identifying the feature points in the images collected in each direction, and then using 3D
  • the point cloud ICP algorithm obtains the spatial position relationship between the point cloud images collected in various directions, and finally, according to the matching relationship and the spatial position relationship, all the point cloud image data are rotated and translated to obtain a complete head of the patient 3D point cloud image.
  • a camera-based transcranial magnetic stimulation diagnosis and treatment detection system adopts the camera system, and the detection method of the detection system includes the following steps:
  • the 3D camera is used to take image data of the patient's head, and the intelligent terminal is used to model the head to establish a 3D head model of the patient's head;
  • the 3D camera is used to take a real-time facial image of the patient
  • the smart terminal is used to perform pose matching
  • the real-time facial image is matched with the established 3D head model for position matching, which further includes:
  • the facial feature points used for matching are marked in the 3D head model;
  • the facial feature points of the patient's real-time facial image are automatically recognized by the 3D camera;
  • the conversion matrix is obtained by affine transformation through feature point matching, and the patient's real-time facial image is calculated
  • the conversion relationship with the established 3D head mold calculate the position of the 3D head mold in space; calculate the position coordinates of the magnetic stimulation points on the 3D head mold in space.
  • a camera-based transcranial magnetic stimulation diagnosis and treatment navigation system using the camera system, further comprising a manipulator and a TMS coil, the manipulator and the TMS coil are respectively electrically connected to an intelligent terminal, and the navigation method of the navigation system includes the following steps :
  • S3 Match the position of the patient's head model with the actual position of the patient's head through the 3D camera and the smart terminal, and determine the spatial position of the magnetic stimulation point on the patient's head model to be magnetically stimulated;
  • step S5 Put the device model built in step S4 and the patient head model built in step S2 in the same spatial coordinate system; and then calculate through the smart terminal that the TMS coil model arrives on the head model for magnetic stimulation
  • the smart terminal automatically navigates the movement of the manipulator according to the optimal path, and finally moves the TMS coil to the magnetic stimulation point on the patient's head for treatment.
  • step S3 the method of matching the position of the patient's head mold with the actual position of the patient's head includes the following steps:
  • step S33 Perform a matching calculation between the facial feature points marked in step S31 and the facial feature points identified in step S32 to obtain the rotation and translation relationship between the patient's head and the patient's head model;
  • S34 Perform rotation and translation operations on the patient's head mold according to the rotation and translation relationship, so that the position of the patient's head mold matches the actual position of the patient's head.
  • step S4 after modeling the manipulator, TMS coil, and 3D camera, the spatial positions of the manipulator model, TMS coil model, and 3D camera model need to be compared with the actual spatial positions of the manipulator, TMS coil, and 3D camera.
  • Matching the specific matching method is:
  • step S43 Perform matching calculation on the feature points marked in step S41 and the feature points identified in step S42 to obtain the rotation and translation relationship between the manipulator model and the manipulator;
  • step S45 Perform rotation and translation operations on the manipulator model, TMS coil model, and 3D camera model according to the rotation and translation relationship in step S43 and step S44, so that the manipulator model, TMS coil model, and 3D camera model are positioned in space They are matched with the actual spatial positions of the manipulator, TMS coil and 3D camera.
  • the detection method further includes: during the magnetic stimulation treatment of the patient's head, the smart terminal also uses a 3D camera to follow the positioning of the patient's head; during the treatment, it will record each time the positioning is completed.
  • the position information of the magnetic stimulation point on the patient's head If the distance between the current and the previous magnetic stimulation point is more than 5mm due to the patient's head movement in the next moment, follow-up positioning will be started; if it is not more than 5mm, follow-up positioning will not be started .
  • the navigation method further includes a following positioning step, and the following positioning step includes: adjusting the spatial pose of the patient's head model through the smart terminal so that the spatial pose of the patient's head model is consistent with that of the patient's head.
  • the current actual spatial pose is matched, and then the latest magnetic stimulation point is repositioned on the head model, and finally the moving path of the manipulator is re-planned, and the TMS coil is moved to the latest magnetic stimulation point for treatment.
  • the present invention uses a camera to capture video image data of the patient’s head, models the patient’s head, and based on the modeling data and real-time facial video captured The image detects the patient’s face posture, obtains the patient’s face posture data, and then performs manipulator navigation based on the face posture data to locate the TMS treatment magnetic stimulation point.
  • This method has high positioning accuracy and good real-time performance, which solves the existing problem.
  • the problem of precise positioning and repeated positioning in TMS diagnosis and treatment 2)
  • the present invention uses the camera to directly obtain the face image, performs 3D modeling, and combines the face detection and recognition algorithm to estimate the accurate face pose.
  • the pose positioning is accurate and real-time It has strong performance and does not need to wear any auxiliary tools on the patient's head;
  • the present invention uses a manipulator to carry out TMS coil navigation and positioning, with rapid positioning and high repeat accuracy.
  • Fig. 1 is a schematic diagram of a camera system according to an embodiment of the present invention
  • Fig. 2 is a schematic diagram of a first photographing subsystem according to an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of a first photographing subsystem according to another embodiment of the present invention.
  • FIG. 4 is a schematic diagram of the distribution of cameras in the first photographing subsystem according to another embodiment of the present invention.
  • FIG. 5 is a schematic flowchart of a modeling system according to another embodiment of the present invention.
  • Fig. 6 is a schematic structural diagram of a detection system according to another embodiment of the present invention.
  • FIG. 7 is a schematic flowchart of a detection system according to another embodiment of the present invention.
  • FIG. 8 is a schematic flowchart of a navigation system according to another embodiment of the present invention.
  • connection should be understood in a broad sense.
  • it can be a mechanical connection or an electrical connection, or the internal communication between two components, which can be directly connected, or through an intermediate connection.
  • the media is indirectly connected, and those of ordinary skill in the art can understand the specific meanings of the above-mentioned terms according to specific circumstances.
  • a camera system for transcranial magnetic stimulation diagnosis and treatment includes a first photographing subsystem for head modeling and a second photographing sub-system for diagnosis and treatment.
  • the first photographing subsystem includes a 3D scanner and a positioning cap
  • the second photographing subsystem includes a 3D camera, a lying bed and a headrest; the first photographing subsystem will photograph the head
  • the part data is sent to the smart terminal for head modeling to obtain a 3D head model with identified facial feature points
  • the second photographing subsystem sends the real-time photographed face data to the smart terminal for face detection
  • the smart terminal matches the detected face image with the 3D head model to obtain magnetic stimulation point location information for transcranial magnetic stimulation diagnosis and treatment.
  • the 3D scanner includes a 3D camera and a rotating bracket, the 3D camera is mounted on the rotating bracket, the rotating bracket is driven to rotate by a motor, and the motor is electrically connected to the smart terminal. Connection; when collecting 3D image data of the patient's head, the intelligent terminal controls the motor to drive the rotating bracket to rotate at a uniform speed, so that the 3D camera moves in a circular motion around the patient's head at a uniform speed, and collects 3D image data of the patient's head from all directions. The collected 3D image data is sent to an intelligent terminal, and the intelligent terminal performs head modeling to obtain a 3D model of the patient's head.
  • the 3D camera in the second photographing subsystem is set above the patient’s face and can fully incorporate the patient’s face into the position of the photographing range, and is used to photograph the patient’s face and facial data in real time during the TMS diagnosis and treatment process, and to photograph
  • the face and face data of is sent to the smart terminal, and the smart terminal performs face detection, and performs manipulator navigation according to the face detection result, and controls the TMS coil to move to the therapeutic magnetic stimulation point.
  • the image data captured by the 3D camera includes color images, depth images, and 3D point cloud images.
  • the positioning cap is a white hood made of elastic material, which is used to cover the patient's hair; because the 3D scanner cannot scan black hair without heat, the white hood needs to be used to cover the hair and expose the patient
  • the facial features and forehead are marked with characteristic points (brow center, nose tip, etc.);
  • the positioning cap is flexible, suitable for a wide range of people, and convenient to wear;
  • the positioning cap is provided with a number of Mark points to facilitate the 3D camera to collect image data.
  • the 3D scanner in the first photographing subsystem includes three 3D cameras and a fixed bracket.
  • the fixed bracket is provided with three camera installation positions, the angle between two adjacent camera installation positions is 120 degrees, and the three 3D cameras are respectively installed on the three camera installation positions ;
  • the three 3D cameras are controlled by an intelligent terminal to simultaneously collect 3D image data of the patient's head from three directions.
  • the 3D image data of the patient's head is simultaneously collected through three 3D cameras, and the collected data is sent to the intelligent terminal for head modeling, which has good real-time performance.
  • a camera-based transcranial magnetic stimulation diagnosis and treatment head model modeling system which includes a 3D scanner, a positioning cap, a seat, and a smart terminal.
  • the 3D scanner and the smart terminal are electrically connected ;
  • the smart terminal may be a computer.
  • the modeling method of the head model modeling system includes the following steps:
  • S1 The patient sits on the seat and wears the positioning cap, starts the smart terminal, collects 3D image data of the patient's head from various directions through the 3D scanner, and then integrates the collected 3D image data Sent to the smart terminal;
  • the positioning cap is a white hood made of elastic material, which is used to cover the patient’s hair; because the 3D scanner cannot scan black hair without heat, it needs to be covered by the white hood Hair, exposing the facial features and forehead of the patient, and marking the characteristic points (brow center, nose tip, etc.); the positioning cap is flexible, suitable for a wide range of people, and convenient to wear; the positioning cap is provided with several Mark points, which is convenient for 3D camera to collect images data.
  • step S2 the method for integrating the 3D image data collected from various directions is: calculating the matching relationship between the images by identifying the feature points in the images collected in each direction, and then using 3D
  • the point cloud ICP algorithm obtains the spatial position relationship between the point cloud images collected in various directions, and finally, according to the matching relationship and the spatial position relationship, all the point cloud image data are rotated and translated to obtain a complete head of the patient 3D point cloud image.
  • 3D scan data of the patient's head it is necessary to collect 3D scan data of the patient's head through a 3D camera.
  • a 3D camera takes a photo
  • a color map, a depth map, and a 3D point cloud map are generated.
  • These three images are generated at the same time, so each The points on the image have a fixed correspondence. This correspondence is known and is obtained through the calibration of the camera; 3D scanning is to take a series of images around the patient’s head, and then stitch these images into a complete image.
  • the image stitching needs to find the same parts in the two images and match them; in the 3D camera, the 3D point cloud cannot be obtained for the hair, and the 3D data of the skull is needed in the medical treatment of the head model (no hair), so the patient is in the head model scan You need to wear a specific positioning cap. In order to make the matching more accurate, some mark points are usually set on the cap; 3D scanning finally needs to stitch the 3D point cloud, and the rotation and translation relationship between the point clouds of each image is required for stitching.
  • the splicing of point clouds mainly relies on the ICP algorithm, which sometimes fails, so rough matching is required first.
  • the mapping method is to compare the four points NZ, CZ, AL, and AR on the patient's head with the four points on the skull model to obtain the skull model conversion matrix, and then transfer the data in the MNI space
  • the point of is multiplied by the transformation matrix to obtain the coordinate points of the patient’s head model; where NZ represents the root of the nose, AL represents the left ear, AR represents the right ear, and CZ represents the line connecting the root of the nose and the occipital protuberance to the left and right ears. The intersection of the lines.
  • a camera-based transcranial magnetic stimulation diagnosis and treatment system includes a lying bed, a headrest 2, a 3D camera, a manipulator, a TMS coil, and a smart terminal; the 3D camera, manipulator, and TMS coil are connected to the smart terminal respectively; the smart terminal can choose a computer or a notebook , Tablet, etc.
  • the lying bed is a horizontal translation platform, which can move back and forth, and is used to adjust the relative position of the patient's head and the camera.
  • the headrest 2 mainly functions as a bracket, and the supporting site is the skull and also includes the neck.
  • the function is to limit the movement of the patient without causing discomfort to the patient, and cannot hinder the magnetic stimulation of the head.
  • the 3D camera is used to obtain the patient's head posture data and real-time facial posture data. Before the treatment, the 3D camera is used to obtain the patient's head posture data, and the intelligent terminal is used for head 3D modeling; after the treatment is started , Use a 3D camera to obtain the patient's real-time facial data, combine the intelligent terminal to process the real-time facial data, and match the modeled 3D head model with the real-time facial image.
  • the 3D camera is also used to obtain the spatial pose of the manipulator and the TMS coil, so that the manipulator is used for navigation and clamps the TMS coil to the position of the magnetic stimulation point.
  • the manipulator is also used to clamp the TMS coil to stimulate the magnetic stimulation points on the patient's head for magnetic stimulation treatment.
  • the detection method of the detection system includes the following steps:
  • the patient lies flat on the reclining bed, and adjusting the front and back positions of the reclining bed so that the reclining bed reaches the treatment position;
  • the 3D camera is used to capture the image data of the patient's head, and the intelligent terminal is used for modeling to establish a 3D head model of the patient's head;
  • the 3D camera is used to take a real-time facial image of the patient
  • the smart terminal is used to perform pose matching
  • the real-time facial image is matched with the established 3D head model for position matching, which further includes:
  • the facial feature points for matching are marked in the 3D head model, and the facial feature points are automatically recognized by the camera during the modeling process;
  • the facial feature points of the patient’s real-time facial image are automatically recognized by the 3D camera;
  • the feature points are passed Matching and performing affine transformation to obtain a conversion matrix, calculate the conversion relationship between the patient’s real-time facial image and the established 3D head model; calculate the position of the 3D head model in the camera coordinate system; calculate the magnetic field on the 3D head model The position coordinates of the stimulus point in space.
  • modeling the patient's head includes the following steps:
  • S22 The smart terminal integrates the 3D image data to obtain a complete 3D point cloud image of the patient's head, and then obtains the complete 3D head model data of the patient's head through sampling, smoothing, and plane fitting processing;
  • the method of integrating the 3D image data is: calculating the matching relationship between the images by identifying the feature points in the images collected in each direction, and then obtaining the point clouds collected in each direction through the ICP algorithm of the 3D point cloud The spatial position relationship between the images, and finally all the point cloud image data are rotated and translated according to the matching relationship and the spatial position relationship to obtain a complete 3D point cloud image of the patient's head;
  • the 3D image captured by the 3D camera in real time only has facial information of the patient, and no head information. Therefore, the head model built in S2 must be registered with the facial data captured in real time. Due to the ICP algorithm The amount of calculation is too large to meet the requirements of real-time detection.
  • the position registration method is to first mark the facial feature points (corners of the eyes, nose tip, etc.) for registration in the head model, and then automatically identify the facial feature points in the real-time image. Through feature point matching, the conversion relationship between the real-time image and the head mold is calculated, and the position of the head mold in space is calculated, and then the position coordinates of the magnetic stimulation points on the head mold in space are calculated.
  • the conversion relationship includes the rotation and translation relationship between the patient's real-time facial image and the 3D head model in the camera coordinate system.
  • the 3D head model is rotated and translated according to the rotation and translation relationship, and the 3D The head model is matched to the patient's real-time facial image.
  • a camera-based transcranial magnetic stimulation diagnosis and treatment navigation system includes a lying bed, a headrest 2, a 3D camera, a manipulator, a TMS coil, and a smart terminal; the 3D camera, the manipulator, and the TMS coil are respectively electrically connected to the smart terminal; the smart terminal may be a computer;
  • the lying bed is a horizontal translation platform, which can move back and forth, and is used to adjust the relative position of the patient's head and the camera;
  • the headrest 2 mainly functions as a bracket, and the supporting site is the skull and also includes the neck.
  • the function is to limit the movement of the patient without causing discomfort to the patient, and cannot hinder the magnetic stimulation of the back of the head;
  • the 3D camera is used to obtain the spatial pose of the patient's head and the manipulator, so as to navigate the manipulator;
  • the manipulator is used to clamp the TMS coil to stimulate the magnetic stimulation points on the patient's head for magnetic stimulation treatment;
  • the navigation method of the navigation system includes the following steps:
  • S3 Match the position of the patient's head model with the actual position of the patient's head through the 3D camera and the smart terminal, and determine the spatial position of the magnetic stimulation point on the patient's head model to be magnetically stimulated;
  • step S5 Put the device model built in step S4 and the patient head model built in step S2 in the same spatial coordinate system; and then calculate through the smart terminal that the TMS coil model arrives on the head model for magnetic stimulation
  • the optimal path of the point the shortest moving distance and no collision with other devices during the movement
  • the smart terminal then automatically navigates the movement of the manipulator according to the optimal path, and finally moves the TMS coil to the patient
  • the head is to be treated by magnetic stimulation at the magnetic stimulation point.
  • modeling the patient's head includes the following steps:
  • S22 Integrate the 3D image data through the smart terminal to obtain a complete 3D point cloud image of the patient's head, and then obtain complete 3D head model data of the patient's head after sampling, smoothing, and plane fitting;
  • the method of integrating the 3D image data is: calculating the matching relationship between the images by identifying the feature points in the images collected in each direction, and then obtaining the point clouds collected in each direction through the ICP algorithm of the 3D point cloud The spatial position relationship between the images, and finally all the point cloud image data are rotated and translated according to the matching relationship and the spatial position relationship to obtain a complete 3D point cloud image of the patient's head;
  • step S3 the 3D image captured by the 3D camera in real time has only the patient’s facial information and no head information. Therefore, the head model built in S2 must be registered with the real-time facial data. Due to the ICP algorithm The amount of calculation is too large to meet the requirements of real-time detection.
  • the position registration method is to first mark the facial feature points (brow center, earlobe, corner of the eye, nose tip, corner of the mouth and chin) for registration in the head model, and then in the real-time image Automatically recognize facial feature points, calculate the conversion relationship between the real-time image and the head model through feature point matching, and calculate the position of the head model in space, and then calculate the position coordinates of the magnetic stimulation points on the head model in space; the specific steps are as follows :
  • step S33 Perform a matching calculation between the facial feature points marked in step S31 and the facial feature points identified in step S32 to obtain the rotation and translation relationship between the patient's head and the patient's head model;
  • S34 Perform rotation and translation operations on the patient's head mold according to the rotation and translation relationship, so that the position of the patient's head mold matches the actual position of the patient's head.
  • step S4 SolidWorks software can be used to model the manipulator, TMS coil and 3D camera.
  • the spatial positions of the manipulator model, TMS coil model, and 3D camera model need to be compared with the manipulator, The actual spatial position of the TMS coil and the 3D camera are matched; the specific matching method is:
  • step S43 Perform matching calculation between the feature points marked in step S41 and the feature points identified in step S42 to obtain the rotation and translation relationship between the manipulator model and the manipulator;
  • step S45 Perform rotation and translation operations on the manipulator model, TMS coil model, and 3D camera model according to the rotation and translation relationship in step S43 and step S44, so that the manipulator model, TMS coil model, and 3D camera model are positioned in space They are matched with the actual spatial positions of the manipulator, TMS coil and 3D camera.
  • step S5 the general movement path planning algorithm of the manipulator is relatively complicated. Since the model, obstacles, and path in this embodiment are all known, the method of manually planning the path is adopted, and the distance from the head mold is far (greater than Use a straight path at the position of 30mm), and use a circular path near the head mold (less than/equal to 30mm) to move the TMS coil around the head to the next magnetic stimulation point; since the 3D data of the head mold is known, Therefore, the head mold data can be enlarged to leave a safe distance for operation, and the shortest arc path between two points on the head mold can be calculated.
  • the present invention avoids the influence of human factors on the treatment effect, and at the same time improves the patient's sense of experience.
  • the present invention also has a following positioning function. In the process of navigating the manipulator, or in the process of face detection, even if the posture of the patient's head changes, the posture of the patient's head can be tracked in real time through the 3D camera to ensure the accuracy of treatment and improve Treatment effect and patient experience.
  • the smart terminal also uses a 3D camera to follow the positioning of the patient's head; during the treatment, the position information of the patient's head is recorded every time the positioning is completed, If the distance between the current and the previous magnetic stimulation point is more than 5mm due to the movement of the patient’s head at the next moment, follow-up positioning will be activated; if it does not exceed 5mm, follow-up positioning will not be activated; if the patient’s head turns more often, Then the 3D camera and the manipulator will be suspended, and the magnetic stimulation of the TMS coil will be suspended at the same time; if the patient is not within the adjustable range of the 3D camera or left, the magnetic stimulation action of the manipulator and the coil will be stopped.
  • the step of following positioning is: adjusting the spatial pose of the patient's head model through the smart terminal, so that the spatial pose of the patient's head model is matched with the current actual spatial pose of the patient's head, Then reposition the latest magnetic stimulation site on the head model, and finally re-plan the moving path of the manipulator, and move the TMS coil to the latest magnetic stimulation site for treatment.
  • the invention uses a camera to capture video image data of the patient's head, models the patient's head, and detects and estimates the patient's facial posture based on the modeling data and the captured facial video image to obtain the patient's facial posture According to the data, the manipulator navigation is carried out according to the face posture data, and the TMS treatment magnetic stimulation point is adjusted. It ensures the accurate positioning of the magnetic stimulation point during each treatment without the need to wear a light guide ball, which solves the problem of TMS positioning and repeated positioning .

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Medical Informatics (AREA)
  • Primary Health Care (AREA)
  • Public Health (AREA)
  • Biomedical Technology (AREA)
  • Epidemiology (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • General Business, Economics & Management (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)
  • Magnetic Treatment Devices (AREA)
  • Image Analysis (AREA)

Abstract

Disclosed is a camera system for transcranial magnetic stimulation therapy. The camera system comprises a first photographing sub-system for head modeling, a second photographing sub-system for therapy, and an intelligent terminal (9), wherein the first photographing sub-system comprises a 3D scanner (8) and a positioning cap, and the second photographing sub-system comprises a 3D camera (3), a reclining-type bed (1) and a headrest (2). The first photographing sub-system sends photographed head data to the intelligent terminal (9) for head modeling to obtain a 3D head model with marked facial feature points; the second photographing sub-system sends facial data photographed in real time to the intelligent terminal (9) for facial detection; and the intelligent terminal (9) matches a detected facial image with the 3D head model to obtain magnetic stimulation point positioning information for transcranial magnetic stimulation therapy. The head posture of a patient is rapidly and precisely acquired by means of the 3D camera (3), and therapeutic magnetic stimulation points are adjusted based on the head posture, thereby improving the accuracy of magnetic stimulation point positioning and repositioning during TMS therapy.

Description

一种用于经颅磁刺激诊疗的摄像***A camera system for transcranial magnetic stimulation diagnosis and treatment 技术领域Technical field
本发明涉及医疗设备技术领域，具体涉及一种用于经颅磁刺激诊疗的摄像***。The invention relates to the technical field of medical equipment, in particular to a camera system used for transcranial magnetic stimulation diagnosis and treatment.
背景技术Background technique
据中国疾病预防控制中心精神卫生中心统计，目前我国精神疾病患者总数已超过1亿，但公众对精神疾病的知晓率不足5成，就诊率更低。目前这些精神病人得到及时救治的约20％，有80％的精神病人得不到及时救治，甚至得不到最基本的救治，症精神疾病患者人数更是高达1600万人。根据IMS health的最新统计数据，全球精神疾病用药已经超过360亿美元，占药品销售总额的5％。不过，就国内而言，目前的精神疾病用药市场规模仍相对较小，大约占医院销售总额的1.5％左右。我国精神病专科医院已经超过600家，但与日益增长的精神病发病率相比，在数量和质量上与精神病患者需求之间还存在较大差距，仍有为数众多的精神病患者不能得到专业、***、有效的治疗。According to the statistics of the Mental Health Center of the Chinese Center for Disease Control and Prevention, the total number of patients with mental illness in my country has exceeded 100 million, but the public awareness rate of mental illness is less than 50%, and the rate of consultation is even lower. At present, about 20% of these mental patients receive timely treatment, and 80% of the mental patients are not treated in time, or even the most basic treatment. The number of patients with mental illness is as high as 16 million. According to the latest statistics from IMS health, global mental illness medications have exceeded 36 billion US dollars, accounting for 5% of total drug sales. However, domestically, the current mental illness drug market is still relatively small, accounting for about 1.5% of the total hospital sales. There are already more than 600 psychiatric hospitals in my country. However, compared with the increasing incidence of mental illness, there is still a large gap between the quantity and quality and the needs of mental patients. There are still a large number of mental patients who cannot get professional, systematic, and Effective treatment.
经颅磁刺激(Transcranial Magnetic Stimulation，TMS)，是一种通过脉冲磁场在局部大脑皮层中产生电流以暂时激活或抑制该皮层的技术。在癫痫病、神经心理科、康复科、儿科等领域经颅磁刺激技术都得到了广泛的使用。现阶段在进行经颅磁治疗时需要医护人员将治疗设备对准患者头部的待治疗区域，由于患者长期保持同一姿势造成的不适会让患者改变头部姿势，医护人员需要时刻关注治疗仪和患者头部是否对齐，该方法成本高、耗时长、误差大，因此医护人员急需一种速度快、精度高的人头姿态跟踪***，该***能够将患者的头部姿态真实的反馈给机械控制设备。Transcranial Magnetic Stimulation (TMS) is a technique that uses a pulsed magnetic field to generate an electric current in the local cerebral cortex to temporarily activate or inhibit the cortex. Transcranial magnetic stimulation technology has been widely used in epilepsy, neuropsychology, rehabilitation, pediatrics and other fields. At this stage, during transcranial magnetic therapy, medical staff need to aim the treatment equipment at the patient’s head to be treated. Discomfort caused by the patient’s long-term maintenance of the same posture will cause the patient to change the head posture. Medical staff need to always pay attention to the treatment instrument and Whether the patient’s head is aligned or not, this method is costly, time-consuming, and large in error. Therefore, medical staff urgently need a fast, high-precision head posture tracking system that can truly feedback the patient’s head posture to the mechanical control equipment .
传统的TMS治疗方式通常包括以下步骤：操作者在患者头部找到治疗磁刺激点区域(凭经验定位)；将TMS线圈置于该区域上，保证线圈中心正对磁刺激点，线圈平面与头皮相切；使用机械装置将TMS线圈夹持固定；启动TMS，开始治疗。由于TMS治疗通常需要长时间，按疗程治疗，纯粹靠经验和记忆的方式来定位的磁刺激点无法保证每次治疗的精准性。患者每次治疗的磁刺激点位置都不一样是普遍情况，并且由于磁场的衰落特性以及有 效磁刺激点区域较小的原因，如果不能保证治疗的磁刺激点精准性，TMS治疗的疗效就会大打折扣。因而找到一种方法来保证磁刺激点的精准是很有必要的。The traditional TMS treatment method usually includes the following steps: the operator finds the treatment magnetic stimulation point area on the patient's head (location based on experience); places the TMS coil on this area, ensuring that the center of the coil is directly opposite the magnetic stimulation point, and the coil plane is with the scalp Tangent; Use mechanical devices to clamp and fix the TMS coil; start TMS and start treatment. Because TMS treatment usually takes a long time, according to the course of treatment, the magnetic stimulation points that are located purely by experience and memory cannot guarantee the accuracy of each treatment. It is common for patients to have different magnetic stimulation points for each treatment, and due to the fading characteristics of the magnetic field and the small area of the effective magnetic stimulation points, if the accuracy of the treatment magnetic stimulation points cannot be guaranteed, the curative effect of TMS treatment will be Greatly discounted. Therefore, it is necessary to find a way to ensure the accuracy of the magnetic stimulation points.
TMS精准重复定位的问题在世界范围内有一些研究，但是目前其他方法大多是基于导光球的红外定位法。该方法每次定位时都必须给患者和TMS线圈佩戴专用导光球，根据导光球在红外拍摄装置中的相对位置来进行定位。因为每次佩戴的位置都不固定，所以还需要额外使用一种专用的定位笔通过复杂的定位来确定导光球和人头的相对位置，因此操作起来颇为不便，且效率低下，目前使用基于导光球的红外定位法中，用于佩戴和定位导光球的时间大约为30分钟/人，由于一次TMS治疗也仅需10-20分钟，因此目前的这种定位方法不适用于TMS的高效重复定位。The problem of TMS accurate repeat positioning has been studied worldwide, but most of the other methods currently are based on the infrared positioning method of the light guide ball. In this method, a special light guide ball must be worn for the patient and the TMS coil every time it is positioned, and the positioning is performed according to the relative position of the light guide ball in the infrared camera. Because the position of wearing is not fixed each time, it is necessary to additionally use a special positioning pen to determine the relative position of the light guide ball and the human head through complex positioning. Therefore, the operation is quite inconvenient and inefficient. In the infrared positioning method of the light guide ball, the time for wearing and positioning the light guide ball is about 30 minutes per person. Since a TMS treatment only takes 10-20 minutes, the current positioning method is not suitable for TMS Repeat positioning efficiently.
较精确的定位方法可采用经颅磁刺激导航***，目前有两种，一种是机器视觉定位导航***，如中国专利“经颅磁刺激导航***及经颅磁刺激线圈定位方法”201210281507.X，201210281472.X；另一种是光学定位导航***，如中国专利“一种导航经颅磁刺激治疗***”201010235826.8，“一种用于重复经颅磁刺激光学定位导航***的校准装置和方法”201010235828.7。机器视觉定位导航***需要定位帽，定位帽的厚度增加了线圈与大脑皮层的距离，增加了刺激强度，造成能量的浪费，特别是在需要经颅磁刺激和脑电记录同时进行时，同时佩戴定位帽和脑电电极帽会增加更多的线圈与大脑皮层距离。在光学定位导航***中，由于物体对光的阻碍作用，对光学传感器和光源的位置提出了很高的要求，也在一定程度上限制了光学定位***的使用。A more accurate positioning method can be a transcranial magnetic stimulation navigation system. There are currently two, one is a machine vision positioning navigation system, such as the Chinese patent "Transcranial Magnetic Stimulation Navigation System and Transcranial Magnetic Stimulation Coil Positioning Method" 201210281507.X , 201210281472.X; the other is an optical positioning navigation system, such as the Chinese patent "a navigational transcranial magnetic stimulation treatment system" 201010235826.8, "a calibration device and method for repetitive transcranial magnetic stimulation optical positioning navigation system" 201010235828.7. The machine vision positioning and navigation system requires a positioning cap. The thickness of the positioning cap increases the distance between the coil and the cerebral cortex, increases the stimulation intensity, and causes a waste of energy, especially when transcranial magnetic stimulation and EEG recording are required at the same time. Positioning caps and EEG electrode caps will increase the distance between the coil and the cerebral cortex. In the optical positioning navigation system, due to the obstructive effect of the object on the light, high requirements are put forward on the position of the optical sensor and the light source, which also limits the use of the optical positioning system to a certain extent.
近年来，将相机与深度传感器结合起来在三维场景重建、室内定位和移动机械手等领域都有重要的应用，RGBD相机就是其中一款能够同时获取图像信息和距离信息的相机。因此，本发明结合RGBD相机能够同时获取图像信息和距离信息的优势，在经颅磁刺激诊疗的TMS线圈定位中，使用RGBD相机进行患者位姿拍摄，然后通过对拍摄图像的处理来定位治疗磁刺激点，在无需佩戴导光球定位的情况下保证每次治疗时磁刺激点定位的精准。In recent years, the combination of cameras and depth sensors has important applications in three-dimensional scene reconstruction, indoor positioning, and mobile manipulators. RGBD camera is one of the cameras that can obtain image information and distance information at the same time. Therefore, the present invention combines the advantages that the RGBD camera can obtain image information and distance information at the same time. In the TMS coil positioning of transcranial magnetic stimulation diagnosis and treatment, the RGBD camera is used to photograph the patient's posture, and then the photographed image is processed to locate the therapeutic magnet. Stimulation point, without the need to wear a light guide ball to ensure the precise positioning of the magnetic stimulation point during each treatment.
发明内容Summary of the invention
本发明的目的是针对现有技术存在的问题，提供一种用于经颅磁刺激诊疗的摄像系 统，在TMS诊疗过程中，快速、精确的获取患者的头部姿态，并根据头部姿态调整治疗磁刺激点，提高TMS治疗中的磁刺激点定位及重复定位的准确性。The purpose of the present invention is to solve the problems existing in the prior art and provide a camera system for transcranial magnetic stimulation diagnosis and treatment. During the TMS diagnosis and treatment, the patient's head posture can be obtained quickly and accurately, and adjusted according to the head posture Treatment of magnetic stimulation points improves the accuracy of magnetic stimulation point positioning and repeated positioning in TMS treatment.
为实现上述目的，本发明采用的技术方案是：In order to achieve the above objective, the technical solution adopted by the present invention is:
一种用于经颅磁刺激诊疗的摄像***，包括用于头部建模的第一拍摄子***、用于诊疗的第二拍摄子***和智能终端，所述第一拍摄子***包括3D扫描仪和定位帽，所述第二拍摄子***包括3D摄像头、躺式床和头枕；所述第一拍摄子***将拍摄到的头部数据发送至智能终端进行头部建模，得到具有已标识面部特征点的3D头模；所述第二拍摄子***将实时拍摄到的人脸数据发送至智能终端进行人脸检测，所述智能终端将检测出的人脸图像与所述3D头模进行匹配，得到用于经颅磁刺激诊疗的磁刺激点定位信息。A camera system for transcranial magnetic stimulation diagnosis and treatment, comprising a first photographing subsystem for head modeling, a second photographing subsystem for diagnosis and treatment, and an intelligent terminal. The first photographing subsystem includes 3D scanning The second photographing subsystem includes a 3D camera, a lying bed, and a headrest; the first photographing subsystem sends the photographed head data to the intelligent terminal for head modeling, and obtains A 3D head model that identifies facial feature points; the second photographing subsystem sends the face data captured in real time to a smart terminal for face detection, and the smart terminal compares the detected face image with the 3D head model The matching is performed to obtain the location information of the magnetic stimulation point for transcranial magnetic stimulation diagnosis and treatment.
优选地，所述3D扫描仪包括一个3D摄像头和一个旋转支架，所述3D摄像头安装在旋转支架上，所述旋转支架由电机驱动旋转，所述电机与智能终端电连接；采集患者头部的3D图像数据时，通过智能终端控制电机驱动旋转支架匀速转动，从而使3D摄像头匀速绕患者头部做圆周运动，从各个方向采集患者头部的3D图像数据。Preferably, the 3D scanner includes a 3D camera and a rotating bracket, the 3D camera is mounted on the rotating bracket, the rotating bracket is driven to rotate by a motor, and the motor is electrically connected to the smart terminal; In the case of 3D image data, the intelligent terminal controls the motor to drive the rotating bracket to rotate at a constant speed, so that the 3D camera moves in a circular motion around the patient's head at a constant speed, and collects 3D image data of the patient's head from all directions.
优选地，所述3D扫描仪包括若干个3D摄像头和一个固定支架，所述若干个3D摄像头均安装在固定支架上；采集患者头部的3D图像数据时，通过智能终端控制所述若干个3D摄像头同时从不同方向采集患者头部的3D图像数据。Preferably, the 3D scanner includes a plurality of 3D cameras and a fixed bracket, and the plurality of 3D cameras are all installed on the fixed bracket; when collecting 3D image data of the patient's head, the plurality of 3D cameras are controlled by an intelligent terminal The camera simultaneously collects 3D image data of the patient's head from different directions.
优选地，所述3D摄像头拍摄到的图像数据包括彩色图像、深度图像和3D点云图像。所述第二拍摄子***中的3D摄像头设置在患者面部上方且能够将其面部完全纳入拍摄范围的位置。Preferably, the image data captured by the 3D camera includes color images, depth images, and 3D point cloud images. The 3D camera in the second photographing sub-system is arranged above the patient's face and can fully incorporate his face into the photographing range.
优选地，所述定位帽为弹性材质做成的白色头罩，用于遮住患者头发；所述定位帽上设有若干Mark点，便于3D摄像头采集图像数据。Preferably, the positioning cap is a white hood made of elastic material, which is used to cover the patient's hair; the positioning cap is provided with a number of Mark points to facilitate the 3D camera to collect image data.
一种基于摄像头的经颅磁刺激诊疗建模***，采用所述的摄像***，所述建模***的建模方法包括以下步骤：A camera-based transcranial magnetic stimulation diagnosis and treatment modeling system adopts the camera system, and the modeling method of the modeling system includes the following steps:
S1，启动所述智能终端，通过所述3D扫描仪从各个方向采集佩戴有定位帽的患者头部的3D图像数据，并将采集到的3D图像数据发送给所述智能终端；S1. Start the smart terminal, collect 3D image data of the patient's head wearing the positioning cap from various directions through the 3D scanner, and send the collected 3D image data to the smart terminal;
S2，通过所述智能终端将3D扫描仪从各个方向采集到的3D图像数据进行整合，得到患者头部完整的3D点云图像，再通过抽样、平滑、平面拟合处理后得到患者头部完整的3D头模数据；S2: Integrate the 3D image data collected by the 3D scanner from various directions through the smart terminal to obtain a complete 3D point cloud image of the patient's head, and then obtain the complete patient's head after sampling, smoothing, and plane fitting. 3D head model data;
S3，利用所述3D头模数据，结合MNI脑空间坐标，将MNI空间的颅骨3D数据映射到患者的3D头模数据上，得到患者的3D头模，然后在患者的3D头模上建立磁刺激点模型。S3. Using the 3D head model data, combined with the MNI brain space coordinates, map the skull 3D data in the MNI space to the patient's 3D head model data to obtain the patient's 3D head model, and then establish a magnetic field on the patient's 3D head model Stimulus point model.
进一步地，步骤S2中，对所述从各个方向采集到的3D图像数据进行整合的方法为：通过识别各个方向采集到的图像中的特征点计算出各个图像之间的匹配关系，再通过3D点云的ICP算法得到各个方向采集到的点云图像之间的空间位置关系，最后根据所述匹配关系和空间位置关系将所有的点云图像数据进行旋转和平移操作，得到患者头部完整的3D点云图像。Further, in step S2, the method for integrating the 3D image data collected from various directions is: calculating the matching relationship between the images by identifying the feature points in the images collected in each direction, and then using 3D The point cloud ICP algorithm obtains the spatial position relationship between the point cloud images collected in various directions, and finally, according to the matching relationship and the spatial position relationship, all the point cloud image data are rotated and translated to obtain a complete head of the patient 3D point cloud image.
一种基于摄像头的经颅磁刺激诊疗检测***，采用所述的摄像***，所述检测***的检测方法包括以下步骤：A camera-based transcranial magnetic stimulation diagnosis and treatment detection system adopts the camera system, and the detection method of the detection system includes the following steps:
S1，通过调节所述躺式床的前后位置，使载有患者的躺式床到达治疗位置；S1, by adjusting the front and rear positions of the lying bed, the lying bed carrying the patient reaches the treatment position;
S2，治疗开始前，采用所述3D摄像头拍摄患者头部的图像数据，采用所述智能终端进行头部建模，建立患者头部的3D头模；S2, before the treatment starts, the 3D camera is used to take image data of the patient's head, and the intelligent terminal is used to model the head to establish a 3D head model of the patient's head;
S3，治疗开始，采用所述3D摄像头拍摄患者的实时面部图像，采用所述智能终端进行位姿匹配，将所述实时面部图像与已建立的3D头模进行位置匹配，进一步包括：在所述3D头模中标出用于匹配的面部特征点；通过所述3D摄像头自动识别出患者的实时面部图像的面部特征点；通过特征点匹配进行仿射变换得到转换矩阵，计算出患者的实时面部图像与已建立的3D头模的转换关系；计算所述3D头模在空间中的位置；计算所述3D头模上的磁刺激点在空间中的位置坐标。S3. At the beginning of treatment, the 3D camera is used to take a real-time facial image of the patient, the smart terminal is used to perform pose matching, and the real-time facial image is matched with the established 3D head model for position matching, which further includes: The facial feature points used for matching are marked in the 3D head model; the facial feature points of the patient's real-time facial image are automatically recognized by the 3D camera; the conversion matrix is obtained by affine transformation through feature point matching, and the patient's real-time facial image is calculated The conversion relationship with the established 3D head mold; calculate the position of the 3D head mold in space; calculate the position coordinates of the magnetic stimulation points on the 3D head mold in space.
一种基于摄像头的经颅磁刺激诊疗导航***，采用所述的摄像***，还包括机械手和TMS线圈，所述机械手、TMS线圈分别与智能终端电连接，所述导航***的导航方法包括以下步骤：A camera-based transcranial magnetic stimulation diagnosis and treatment navigation system, using the camera system, further comprising a manipulator and a TMS coil, the manipulator and the TMS coil are respectively electrically connected to an intelligent terminal, and the navigation method of the navigation system includes the following steps :
S1，通过调节所述躺式床的前后位置，使载有患者的躺式床到达治疗位置；S1, by adjusting the front and rear positions of the lying bed, the lying bed carrying the patient reaches the treatment position;
S2，采用所述3D摄像头和智能终端对患者头部进行建模；S2, using the 3D camera and smart terminal to model the patient's head;
S3，通过所述3D摄像头和智能终端将患者的头模的位置与患者头部的实际位置进行匹配，确定患者头模上待磁刺激磁刺激点的空间位置；S3: Match the position of the patient's head model with the actual position of the patient's head through the 3D camera and the smart terminal, and determine the spatial position of the magnetic stimulation point on the patient's head model to be magnetically stimulated;
S4，通过所述智能终端对机械手、TMS线圈和3D摄像头进行建模；S4, modeling the manipulator, TMS coil, and 3D camera through the smart terminal;
S5，将步骤S4中建好的设备模型与步骤S2中建好的患者头模放在同一个空间坐标系中；再通过所述智能终端计算出TMS线圈模型到达头模上待磁刺激磁刺激点的最佳路径，所述智能终端再根据该最佳路径对机械手的移动进行自动导航，最终将所述TMS线圈移动至患者头部待磁刺激磁刺激点进行治疗。S5: Put the device model built in step S4 and the patient head model built in step S2 in the same spatial coordinate system; and then calculate through the smart terminal that the TMS coil model arrives on the head model for magnetic stimulation The smart terminal automatically navigates the movement of the manipulator according to the optimal path, and finally moves the TMS coil to the magnetic stimulation point on the patient's head for treatment.
进一步地，步骤S3中，将患者的头模的位置与患者头部的实际位置进行匹配的方法包括以下步骤：Further, in step S3, the method of matching the position of the patient's head mold with the actual position of the patient's head includes the following steps:
S31，在患者头模上标出用于配准的面部特征点；S31, mark the facial feature points for registration on the patient's head model;
S32，通过3D摄像头识别出患者面部的特征点；S32: Recognizing the feature points of the patient's face through the 3D camera;
S33，将步骤S31中标出的面部特征点与步骤S32中识别出的面部特征点进行匹配计算，得到患者头部与患者头模的旋转、平移关系；S33: Perform a matching calculation between the facial feature points marked in step S31 and the facial feature points identified in step S32 to obtain the rotation and translation relationship between the patient's head and the patient's head model;
S34，根据所述旋转、平移关系对患者头模进行旋转、平移操作，使患者头模的位置与患者头部的实际位置匹配上。S34: Perform rotation and translation operations on the patient's head mold according to the rotation and translation relationship, so that the position of the patient's head mold matches the actual position of the patient's head.
进一步地，步骤S4中，对所述机械手、TMS线圈和3D摄像头建模后，需要将机械手模型、TMS线圈模型、3D摄像头模型的空间位置分别与机械手、TMS线圈、3D摄像头的实际空间位置进行匹配；具体匹配方法为：Further, in step S4, after modeling the manipulator, TMS coil, and 3D camera, the spatial positions of the manipulator model, TMS coil model, and 3D camera model need to be compared with the actual spatial positions of the manipulator, TMS coil, and 3D camera. Matching; the specific matching method is:
S41,在机械手模型上标出用于配准的特征点；S41, mark the feature points for registration on the manipulator model;
S42，通过3D摄像头识别出机械手处于初始位置时的特征点；S42: Recognizing the characteristic points when the manipulator is in the initial position through the 3D camera;
S43，将步骤S41中标出的特征点与步骤S42中识别出的特征点进行匹配计算，得到机械手模型与机械手的旋转、平移关系；S43: Perform matching calculation on the feature points marked in step S41 and the feature points identified in step S42 to obtain the rotation and translation relationship between the manipulator model and the manipulator;
S44，根据机械手处于初始位置时所述3D摄像头、TMS线圈与机械手的相对位置是固定的原理，得到3D摄像头模型、TMS线圈模型分别与3D摄像头、TMS线圈的旋转、平移 关系；S44: According to the principle that the relative positions of the 3D camera, TMS coil and the manipulator are fixed when the manipulator is in the initial position, obtain the rotation and translation relationships of the 3D camera model and the TMS coil model with the 3D camera and the TMS coil respectively;
S45，根据步骤S43和步骤S44中的旋转、平移关系，对所述机械手模型、TMS线圈模型、3D摄像头模型进行旋转、平移操作，使所述机械手模型、TMS线圈模型、3D摄像头模型的空间位置分别与机械手、TMS线圈、3D摄像头的实际空间位置匹配上。S45: Perform rotation and translation operations on the manipulator model, TMS coil model, and 3D camera model according to the rotation and translation relationship in step S43 and step S44, so that the manipulator model, TMS coil model, and 3D camera model are positioned in space They are matched with the actual spatial positions of the manipulator, TMS coil and 3D camera.
优选地，所述检测方法还包括：在对患者头部进行磁刺激治疗的过程中，所述智能终端还通过3D摄像头对患者头部进行跟随定位；在治疗过程中会记录每次定位完成时患者头部磁刺激点的位置信息，若下一时刻由于患者头部运动造成当前时刻与上一时刻的磁刺激点位置距离超过5mm，则启动跟随定位；若不超过5mm，则不启动跟随定位。Preferably, the detection method further includes: during the magnetic stimulation treatment of the patient's head, the smart terminal also uses a 3D camera to follow the positioning of the patient's head; during the treatment, it will record each time the positioning is completed. The position information of the magnetic stimulation point on the patient's head. If the distance between the current and the previous magnetic stimulation point is more than 5mm due to the patient's head movement in the next moment, follow-up positioning will be started; if it is not more than 5mm, follow-up positioning will not be started .
优选地，所述导航方法还包括跟随定位步骤，所述跟随定位步骤包括：通过所述智能终端对患者头部模型的空间位姿进行调整，使患者头部模型的空间位姿与患者头部当前的实际空间位姿进行匹配，然后在头部模型上重新定位最新的磁刺激点，最后重新规划机械手的移动路径，将TMS线圈移动到最新的磁刺激点进行治疗。Preferably, the navigation method further includes a following positioning step, and the following positioning step includes: adjusting the spatial pose of the patient's head model through the smart terminal so that the spatial pose of the patient's head model is consistent with that of the patient's head. The current actual spatial pose is matched, and then the latest magnetic stimulation point is repositioned on the head model, and finally the moving path of the manipulator is re-planned, and the TMS coil is moved to the latest magnetic stimulation point for treatment.
与现有技术相比，本发明的有益效果是：1)本发明通过摄像头拍摄患者头部的视频图像数据，对患者的头部进行建模，并根据建模数据以及实时拍摄的人脸视频图像对患者的人脸姿态进行检测，得到患者的人脸姿态数据，然后根据人脸姿态数据进行机械手导航，定位TMS治疗磁刺激点，该方式的定位精度高、实时性好，解决了现有TMS诊疗中的精确定位及重复定位的问题；2)本发明采用摄像头直接获取人脸图像，进行3D建模，并结合人脸检测识别算法估计准确的人脸位姿，位姿定位准确，实时性强，不需要在患者头部佩戴任何辅助工具；3)本发明利用机械手进行TMS线圈导航定位，定位迅速，重复精度高。Compared with the prior art, the present invention has the following beneficial effects: 1) The present invention uses a camera to capture video image data of the patient’s head, models the patient’s head, and based on the modeling data and real-time facial video captured The image detects the patient’s face posture, obtains the patient’s face posture data, and then performs manipulator navigation based on the face posture data to locate the TMS treatment magnetic stimulation point. This method has high positioning accuracy and good real-time performance, which solves the existing problem. The problem of precise positioning and repeated positioning in TMS diagnosis and treatment; 2) The present invention uses the camera to directly obtain the face image, performs 3D modeling, and combines the face detection and recognition algorithm to estimate the accurate face pose. The pose positioning is accurate and real-time It has strong performance and does not need to wear any auxiliary tools on the patient's head; 3) The present invention uses a manipulator to carry out TMS coil navigation and positioning, with rapid positioning and high repeat accuracy.
附图说明Description of the drawings
图1为根据本发明一实施方式的摄像***的示意图；Fig. 1 is a schematic diagram of a camera system according to an embodiment of the present invention;
图2为根据本发明一实施方式的第一拍摄子***的示意图；Fig. 2 is a schematic diagram of a first photographing subsystem according to an embodiment of the present invention;
图3为根据本发明另一实施方式的第一拍摄子***的示意图；3 is a schematic diagram of a first photographing subsystem according to another embodiment of the present invention;
图4为根据本发明另一实施方式的第一拍摄子***中摄像头的分布示意图；4 is a schematic diagram of the distribution of cameras in the first photographing subsystem according to another embodiment of the present invention;
图5为根据本发明又一实施方式的建模***的流程示意图；FIG. 5 is a schematic flowchart of a modeling system according to another embodiment of the present invention;
图6为根据本发明又一实施方式的检测***的结构示意图；Fig. 6 is a schematic structural diagram of a detection system according to another embodiment of the present invention;
图7为根据本发明又一实施方式的检测***的流程示意图；FIG. 7 is a schematic flowchart of a detection system according to another embodiment of the present invention;
图8为根据本发明又一实施方式的导航***的流程示意图。FIG. 8 is a schematic flowchart of a navigation system according to another embodiment of the present invention.
图中：1、躺式床；2、头枕；3、3D摄像头；4、机械手；5、TMS线圈；6、旋转支架；7、电机；8、3D扫描仪；9、智能终端；10、座椅；11、摄像头安装位；12、固定支架。In the picture: 1. Recumbent bed; 2. Headrest; 3. 3D camera; 4. Manipulator; 5. TMS coil; 6. Rotating bracket; 7. Motor; 8. 3D scanner; 9. Smart terminal; 10. Seat; 11. Camera installation position; 12. Fixed bracket.
具体实施方式detailed description
下面将结合本发明中的附图，对本发明的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动条件下所获得的所有其它实施例，都属于本发明保护的范围。The technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.
本发明中，术语“安装、”“相连”、“连接”应做广义理解，例如，可以是机械连接或电连接，也可以是两个元件内部的连通，可以是直接相连，也可以通过中间媒介间接相连，对于本领域的普通技术人员而言，可以根据具体情况理解上述术语的具体含义。In the present invention, the terms "installation," "connection", and "connection" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or the internal communication between two components, which can be directly connected, or through an intermediate connection. The media is indirectly connected, and those of ordinary skill in the art can understand the specific meanings of the above-mentioned terms according to specific circumstances.
作为本发明的一实施方式，提供一种用于经颅磁刺激诊疗的摄像***，如图1所示，包括用于头部建模的第一拍摄子***、用于诊疗的第二拍摄子***和智能终端，所述第一拍摄子***包括3D扫描仪和定位帽，所述第二拍摄子***包括3D摄像头、躺式床和头枕；所述第一拍摄子***将拍摄到的头部数据发送至智能终端进行头部建模，得到具有已标识面部特征点的3D头模；所述第二拍摄子***将实时拍摄到的人脸数据发送至智能终端进行人脸检测，所述智能终端将检测出的人脸图像与所述3D头模进行匹配，得到用于经颅磁刺激诊疗的磁刺激点定位信息。As an embodiment of the present invention, a camera system for transcranial magnetic stimulation diagnosis and treatment is provided. As shown in FIG. 1, it includes a first photographing subsystem for head modeling and a second photographing sub-system for diagnosis and treatment. System and smart terminal, the first photographing subsystem includes a 3D scanner and a positioning cap, the second photographing subsystem includes a 3D camera, a lying bed and a headrest; the first photographing subsystem will photograph the head The part data is sent to the smart terminal for head modeling to obtain a 3D head model with identified facial feature points; the second photographing subsystem sends the real-time photographed face data to the smart terminal for face detection, The smart terminal matches the detected face image with the 3D head model to obtain magnetic stimulation point location information for transcranial magnetic stimulation diagnosis and treatment.
如图2所示，本实施中，所述3D扫描仪包括一个3D摄像头和一个旋转支架，所述3D摄像头安装在旋转支架上，所述旋转支架由电机驱动旋转，所述电机与智能终端电连接；采集患者头部的3D图像数据时，通过智能终端控制电机驱动旋转支架匀速转动，从而使3D摄像头匀速绕患者头部做圆周运动，从各个方向采集患者头部的3D图像数据。采集的3D图像数据发送至智能终端，由所述智能终端进行头部建模，得到患者头部的3D模型。As shown in Figure 2, in this implementation, the 3D scanner includes a 3D camera and a rotating bracket, the 3D camera is mounted on the rotating bracket, the rotating bracket is driven to rotate by a motor, and the motor is electrically connected to the smart terminal. Connection; when collecting 3D image data of the patient's head, the intelligent terminal controls the motor to drive the rotating bracket to rotate at a uniform speed, so that the 3D camera moves in a circular motion around the patient's head at a uniform speed, and collects 3D image data of the patient's head from all directions. The collected 3D image data is sent to an intelligent terminal, and the intelligent terminal performs head modeling to obtain a 3D model of the patient's head.
所述第二拍摄子***中的3D摄像头设置在患者面部上方且能够将其面部完全纳入拍摄范围的位置，用于在进行TMS诊疗过程中，实时拍摄患者的人脸面部数据，并将拍摄到的人脸面部数据发送至智能终端，由所述智能终端进行人脸检测，并根据人脸检测结果进行机械手导航，控制TMS线圈移动至治疗磁刺激点。The 3D camera in the second photographing subsystem is set above the patient’s face and can fully incorporate the patient’s face into the position of the photographing range, and is used to photograph the patient’s face and facial data in real time during the TMS diagnosis and treatment process, and to photograph The face and face data of is sent to the smart terminal, and the smart terminal performs face detection, and performs manipulator navigation according to the face detection result, and controls the TMS coil to move to the therapeutic magnetic stimulation point.
具体地，所述3D摄像头拍摄到的图像数据包括彩色图像、深度图像和3D点云图像。Specifically, the image data captured by the 3D camera includes color images, depth images, and 3D point cloud images.
具体地，所述定位帽为弹性材质做成的白色头罩，用于遮住患者头发；因为3D扫描仪无法扫描黑色无热量的头发，故需要通过所述白色头罩遮住毛发，露出患者五官及额头，并对特征点(眉心、鼻尖等)做标记；该定位帽具有弹性，适用人群广，佩戴方便；所述定位帽上设有若干Mark点，便于3D摄像头采集图像数据。Specifically, the positioning cap is a white hood made of elastic material, which is used to cover the patient's hair; because the 3D scanner cannot scan black hair without heat, the white hood needs to be used to cover the hair and expose the patient The facial features and forehead are marked with characteristic points (brow center, nose tip, etc.); the positioning cap is flexible, suitable for a wide range of people, and convenient to wear; the positioning cap is provided with a number of Mark points to facilitate the 3D camera to collect image data.
作为本发明的另一实施方式，如图3、图4所示，所述第一拍摄子***中的3D扫描仪包括3个3D摄像头和一个固定支架。As another embodiment of the present invention, as shown in Figs. 3 and 4, the 3D scanner in the first photographing subsystem includes three 3D cameras and a fixed bracket.
具体地，所述固定支架上设有3个摄像头安装位，相邻两个摄像头安装位之间的夹角均为120度，所述3个3D摄像头分别安装在所述3个摄像头安装位上；Specifically, the fixed bracket is provided with three camera installation positions, the angle between two adjacent camera installation positions is 120 degrees, and the three 3D cameras are respectively installed on the three camera installation positions ；
采集患者头部的3D图像数据时，通过智能终端控制所述3个3D摄像头同时从三个方向采集患者头部的3D图像数据。When collecting 3D image data of the patient's head, the three 3D cameras are controlled by an intelligent terminal to simultaneously collect 3D image data of the patient's head from three directions.
本实施中，通过3个3D摄像头同时采集患者头部的3D图像数据，并将采集到的数据发送至智能终端进行头部建模，实时性较好。In this implementation, the 3D image data of the patient's head is simultaneously collected through three 3D cameras, and the collected data is sent to the intelligent terminal for head modeling, which has good real-time performance.
作为本发明的又一实施方式，提供一种基于摄像头的经颅磁刺激诊疗头模建模***，包括3D扫描仪、定位帽、座椅和智能终端，所述3D扫描仪和智能终端电连接；所述智能终端可以为计算机。As another embodiment of the present invention, a camera-based transcranial magnetic stimulation diagnosis and treatment head model modeling system is provided, which includes a 3D scanner, a positioning cap, a seat, and a smart terminal. The 3D scanner and the smart terminal are electrically connected ; The smart terminal may be a computer.
如图5所示，所述头模建模***的建模方法包括以下步骤：As shown in Figure 5, the modeling method of the head model modeling system includes the following steps:
S1，患者坐在所述座椅上并佩戴好所述定位帽，启动所述智能终端，通过所述3D扫描仪从各个方向采集患者头部的3D图像数据，并将采集到的3D图像数据发送给所述智能终端；S1: The patient sits on the seat and wears the positioning cap, starts the smart terminal, collects 3D image data of the patient's head from various directions through the 3D scanner, and then integrates the collected 3D image data Sent to the smart terminal;
S2，通过所述智能终端将3D扫描仪从各个方向采集到的3D图像数据进行整合，得到 患者头部完整的3D点云图像，再通过抽样、平滑、平面拟合处理后得到患者头部完整的3D头模数据；S2: Integrate the 3D image data collected by the 3D scanner from various directions through the smart terminal to obtain a complete 3D point cloud image of the patient's head, and then obtain the complete patient's head after sampling, smoothing, and plane fitting. 3D head model data;
S3，利用所述3D头模数据，结合医学常用的MNI脑空间坐标，将MNI空间的颅骨3D数据映射到患者的3D头模数据上，得到患者的3D头模，然后在患者的3D头模上建立磁刺激点模型。S3, using the 3D head model data, combined with the MNI brain space coordinates commonly used in medicine, map the skull 3D data in the MNI space to the patient's 3D head model data to obtain the patient's 3D head model, and then display it on the patient's 3D head model Establish a magnetic stimulation point model on the
具体地，步骤S1中，所述定位帽为弹性材质做成的白色头罩，用于遮住患者头发；因为3D扫描仪无法扫描黑色无热量的头发，故需要通过所述白色头罩遮住毛发，露出患者五官及额头，并对特征点(眉心、鼻尖等)做标记；该定位帽具有弹性，适用人群广，佩戴方便；所述定位帽上设有若干Mark点，便于3D摄像头采集图像数据。Specifically, in step S1, the positioning cap is a white hood made of elastic material, which is used to cover the patient’s hair; because the 3D scanner cannot scan black hair without heat, it needs to be covered by the white hood Hair, exposing the facial features and forehead of the patient, and marking the characteristic points (brow center, nose tip, etc.); the positioning cap is flexible, suitable for a wide range of people, and convenient to wear; the positioning cap is provided with several Mark points, which is convenient for 3D camera to collect images data.
具体地，步骤S2中，对所述从各个方向采集到的3D图像数据进行整合的方法为：通过识别各个方向采集到的图像中的特征点计算出各个图像之间的匹配关系，再通过3D点云的ICP算法得到各个方向采集到的点云图像之间的空间位置关系，最后根据所述匹配关系和空间位置关系将所有的点云图像数据进行旋转和平移操作，得到患者头部完整的3D点云图像。Specifically, in step S2, the method for integrating the 3D image data collected from various directions is: calculating the matching relationship between the images by identifying the feature points in the images collected in each direction, and then using 3D The point cloud ICP algorithm obtains the spatial position relationship between the point cloud images collected in various directions, and finally, according to the matching relationship and the spatial position relationship, all the point cloud image data are rotated and translated to obtain a complete head of the patient 3D point cloud image.
进一步地，对头部进行建模需要通过3D摄像头采集患者头部的3D扫描数据，3D摄像头每拍照一次会产生彩色图、深度图和3D点云图，这3幅图是同时产生的，所以各个图像上的点有固定的对应关系，这个对应关系是已知的，是通过摄像头的标定得到的；3D扫描就是围绕患者头部拍摄一系列的图像，然后将这些图像拼接成一个完整的图像，而图像拼接要找到两幅图像中相同的部分并进行匹配；在3D摄像头中头发无法得到3D点云，而在医学治疗头模需要颅骨的3D数据(不要头发)，故在头模扫描时患者需要戴上特定的定位帽，为了使匹配更准确，帽子上一般会设一些mark点；3D扫描最终需要对3D点云进行拼接，拼接时需要每一幅图像点云之间的旋转平移关系，点云的拼接主要依靠ICP算法，ICP算法有时会失败，故需要先做粗匹配。Furthermore, to model the head, it is necessary to collect 3D scan data of the patient's head through a 3D camera. Each time the 3D camera takes a photo, a color map, a depth map, and a 3D point cloud map are generated. These three images are generated at the same time, so each The points on the image have a fixed correspondence. This correspondence is known and is obtained through the calibration of the camera; 3D scanning is to take a series of images around the patient’s head, and then stitch these images into a complete image. The image stitching needs to find the same parts in the two images and match them; in the 3D camera, the 3D point cloud cannot be obtained for the hair, and the 3D data of the skull is needed in the medical treatment of the head model (no hair), so the patient is in the head model scan You need to wear a specific positioning cap. In order to make the matching more accurate, some mark points are usually set on the cap; 3D scanning finally needs to stitch the 3D point cloud, and the rotation and translation relationship between the point clouds of each image is required for stitching. The splicing of point clouds mainly relies on the ICP algorithm, which sometimes fails, so rough matching is required first.
进一步地，点云的拼接步骤如下：Further, the splicing steps of point clouds are as follows:
S21，先在彩色图中通过OpenCV中cv::FeatureDetector和cv::DescriptorExtractor 计算“关键点”，并对这些关键点周围的像素计算其“描述子”，再通过cv::DMatch对上述的描述子进行匹配，然后调用OpenCV里的SolvePnPRansac函数求解PnP得到两幅图像的位移和旋转的信息；S21, first calculate the "key points" in the color image through cv::FeatureDetector and cv::DescriptorExtractor in OpenCV, and calculate the "descriptors" for the pixels around these key points, and then use cv::DMatch to describe the above Then, call the SolvePnPRansac function in OpenCV to solve PnP to obtain the displacement and rotation information of the two images;
S22，使用上面计算得到的位移和旋转的信息作为ICP算法的初始粗匹配的结果对两幅点云数据进行计算得到更加精准的位移和旋转数据；S22: Use the displacement and rotation information calculated above as the result of the initial coarse matching of the ICP algorithm to calculate the two point cloud data to obtain more accurate displacement and rotation data;
S23，使用上述位移和旋转数据得到位移和旋转矩阵，并对上一幅点云图中的点全部进行旋转和平移，将计算的到的新点云添加到当前点云图中，得到一个更大的点云，完成两幅点云的整合；S23. Use the above displacement and rotation data to obtain the displacement and rotation matrix, rotate and translate all the points in the previous point cloud image, and add the calculated new point cloud to the current point cloud image to obtain a larger Point cloud, complete the integration of two point clouds;
S24，重复步骤S21至S23，将所有点云图整合成一个更大的点云图，再对此点云图进行滤波平滑处理，抽样减少点的数量，拟合得到3D曲面数据；即得到患者头部完整的3D数据。S24, repeat steps S21 to S23 to integrate all the point cloud images into a larger point cloud image, and then perform filtering and smoothing processing on the point cloud image, sampling to reduce the number of points, and fitting to obtain 3D surface data; that is, the patient's head is complete 3D data.
具体地，步骤S3中，所述映射方法为，通过选取患者头部NZ、CZ、AL、AR四点与颅骨模型上这四个点进行比对，得到颅骨模型转换矩阵，再将MNI空间中的点乘以该转换矩阵得到患者头模坐标点；其中，NZ表示鼻根，AL表示左耳，AR表示右耳，CZ表示鼻根与枕骨隆突的连线相交于左耳与右耳连线的交点。Specifically, in step S3, the mapping method is to compare the four points NZ, CZ, AL, and AR on the patient's head with the four points on the skull model to obtain the skull model conversion matrix, and then transfer the data in the MNI space The point of is multiplied by the transformation matrix to obtain the coordinate points of the patient’s head model; where NZ represents the root of the nose, AL represents the left ear, AR represents the right ear, and CZ represents the line connecting the root of the nose and the occipital protuberance to the left and right ears. The intersection of the lines.
作为本发明的又一实施方式，提供一种基于摄像头的经颅磁刺激诊疗检测***。如图6所示，包括躺式床、头枕2、3D摄像头、机械手、TMS线圈和智能终端；所述3D摄像头、机械手、TMS线圈分别与智能终端连接；所述智能终端可选择计算机、笔记本、平板电脑等。As another embodiment of the present invention, a camera-based transcranial magnetic stimulation diagnosis and treatment system is provided. As shown in Figure 6, it includes a lying bed, a headrest 2, a 3D camera, a manipulator, a TMS coil, and a smart terminal; the 3D camera, manipulator, and TMS coil are connected to the smart terminal respectively; the smart terminal can choose a computer or a notebook , Tablet, etc.
所述躺式床为卧式平移平台，可以前后移动，用于调整患者头部与摄像头的相对位置。The lying bed is a horizontal translation platform, which can move back and forth, and is used to adjust the relative position of the patient's head and the camera.
所述头枕2主要起到托架作用，支撑位点为头骨，还包括颈部，起到的作用是限制病人的移动，且不会造成病人的不适，并且不能阻碍头部的磁刺激。The headrest 2 mainly functions as a bracket, and the supporting site is the skull and also includes the neck. The function is to limit the movement of the patient without causing discomfort to the patient, and cannot hinder the magnetic stimulation of the head.
所述3D摄像头用于获取患者的头部姿态数据及实时的面部姿态数据，在进行治疗前，采用3D摄像头获取患者的头部姿态数据，结合智能终端进行头部3D建模；在开始治疗后，采用3D摄像头获取患者的实时面部数据，结合智能终端对实时面部数据进行处理，将已 建模的3D头模与实时面部图像进行匹配。The 3D camera is used to obtain the patient's head posture data and real-time facial posture data. Before the treatment, the 3D camera is used to obtain the patient's head posture data, and the intelligent terminal is used for head 3D modeling; after the treatment is started , Use a 3D camera to obtain the patient's real-time facial data, combine the intelligent terminal to process the real-time facial data, and match the modeled 3D head model with the real-time facial image.
所述3D摄像头还用于获取机械手和TMS线圈的的空间位姿，从而利用机械手进行导航，将TMS线圈夹持到磁刺激点位置。The 3D camera is also used to obtain the spatial pose of the manipulator and the TMS coil, so that the manipulator is used for navigation and clamps the TMS coil to the position of the magnetic stimulation point.
所述机械手还用于夹持TMS线圈对患者头部刺激磁刺激点进行磁刺激治疗。The manipulator is also used to clamp the TMS coil to stimulate the magnetic stimulation points on the patient's head for magnetic stimulation treatment.
如图7所示，所述检测***的检测方法包括以下步骤：As shown in Figure 7, the detection method of the detection system includes the following steps:
S1，患者平躺在所述躺式床上，调节所述躺式床的前后位置，使所述躺式床到达治疗位置；S1, the patient lies flat on the reclining bed, and adjusting the front and back positions of the reclining bed so that the reclining bed reaches the treatment position;
S2，治疗开始前，采用所述3D摄像头拍摄患者头部的图像数据，采用所述智能终端进行建模，建立患者头部的3D头模；S2: Before the treatment starts, the 3D camera is used to capture the image data of the patient's head, and the intelligent terminal is used for modeling to establish a 3D head model of the patient's head;
S3，治疗开始，采用所述3D摄像头拍摄患者的实时面部图像，采用所述智能终端进行位姿匹配，将所述实时面部图像与已建立的3D头模进行位置匹配，进一步包括：在所述3D头模中标出用于匹配的面部特征点，该面部特征点是在建模过程中由摄像头自动识别的；通过所述3D摄像头自动识别出患者的实时面部图像的面部特征点；通过特征点匹配进行仿射变换得到转换矩阵，计算出患者的实时面部图像与已建立的3D头模的转换关系；计算所述3D头模在摄像头坐标系下的位置；计算所述3D头模上的磁刺激点在空间中的位置坐标。S3. At the beginning of treatment, the 3D camera is used to take a real-time facial image of the patient, the smart terminal is used to perform pose matching, and the real-time facial image is matched with the established 3D head model for position matching, which further includes: The facial feature points for matching are marked in the 3D head model, and the facial feature points are automatically recognized by the camera during the modeling process; the facial feature points of the patient’s real-time facial image are automatically recognized by the 3D camera; the feature points are passed Matching and performing affine transformation to obtain a conversion matrix, calculate the conversion relationship between the patient’s real-time facial image and the established 3D head model; calculate the position of the 3D head model in the camera coordinate system; calculate the magnetic field on the 3D head model The position coordinates of the stimulus point in space.
具体地，S2中，对患者头部进行建模包括以下步骤：Specifically, in S2, modeling the patient's head includes the following steps:
S21，通过所述3D摄像头从各个方向采集患者头部的3D图像数据，并将采集到的3D图像数据发送给所述智能终端；S21, collecting 3D image data of the patient's head from various directions through the 3D camera, and sending the collected 3D image data to the smart terminal;
S22，所述智能终端将所述3D图像数据进行整合，得到患者头部完整的3D点云图像，再通过抽样、平滑、平面拟合处理后得到患者头部完整的3D头模数据；S22: The smart terminal integrates the 3D image data to obtain a complete 3D point cloud image of the patient's head, and then obtains the complete 3D head model data of the patient's head through sampling, smoothing, and plane fitting processing;
对所述3D图像数据进行整合的方法为：通过识别各个方向采集到的图像中的特征点计算出各个图像之间的匹配关系，再通过3D点云的ICP算法得到各个方向采集到的点云图像之间的空间位置关系，最后根据所述匹配关系和空间位置关系将所有的点云图像数据进行旋转和平移操作，得到患者头部完整的3D点云图像；The method of integrating the 3D image data is: calculating the matching relationship between the images by identifying the feature points in the images collected in each direction, and then obtaining the point clouds collected in each direction through the ICP algorithm of the 3D point cloud The spatial position relationship between the images, and finally all the point cloud image data are rotated and translated according to the matching relationship and the spatial position relationship to obtain a complete 3D point cloud image of the patient's head;
S23，利用所述3D头模数据，结合MNI脑空间坐标，将MNI空间的颅骨3D数据映射到患者的3D头模数据上，得到患者的3D头模，然后在患者的3D头模上建立磁刺激点模型。S23. Using the 3D head model data, combined with the MNI brain space coordinates, map the skull 3D data in the MNI space to the patient's 3D head model data to obtain the patient's 3D head model, and then establish a magnetic field on the patient's 3D head model Stimulus point model.
具体地，步骤S3中，3D摄像头实时拍摄的3D图像只有患者的面部信息，没有头部信息，所以要将S2中建好的头模与实时拍摄的面部数据进行位置的配准，由于ICP算法计算量较大，无法满足实时检测的要求，位置配准方法是先在头模中标出用于配准的面部特征点(眼角、鼻尖等)，再在实时图像中自动识别出面部特征点，通过特征点匹配计算出实时画面与头模的转换关系，并计算出头模在空间的位置，再计算出头模上的磁刺激点在空间中的位置坐标。Specifically, in step S3, the 3D image captured by the 3D camera in real time only has facial information of the patient, and no head information. Therefore, the head model built in S2 must be registered with the facial data captured in real time. Due to the ICP algorithm The amount of calculation is too large to meet the requirements of real-time detection. The position registration method is to first mark the facial feature points (corners of the eyes, nose tip, etc.) for registration in the head model, and then automatically identify the facial feature points in the real-time image. Through feature point matching, the conversion relationship between the real-time image and the head mold is calculated, and the position of the head mold in space is calculated, and then the position coordinates of the magnetic stimulation points on the head mold in space are calculated.
所述转换关系包括患者的实时面部图像与所述3D头模在摄像头坐标系下的旋转、平移关系，根据所述旋转、平移关系对所述3D头模进行旋转、平移操作，将所述3D头模匹配到患者的实时面部图像上。The conversion relationship includes the rotation and translation relationship between the patient's real-time facial image and the 3D head model in the camera coordinate system. The 3D head model is rotated and translated according to the rotation and translation relationship, and the 3D The head model is matched to the patient's real-time facial image.
作为本发明的又一实施方式，提供一种基于摄像头的经颅磁刺激诊疗导航***。包括躺式床、头枕2、3D摄像头、机械手、TMS线圈和智能终端；所述3D摄像头、机械手、TMS线圈分别与智能终端电连接；所述智能终端可以为计算机；As another embodiment of the present invention, a camera-based transcranial magnetic stimulation diagnosis and treatment navigation system is provided. It includes a lying bed, a headrest 2, a 3D camera, a manipulator, a TMS coil, and a smart terminal; the 3D camera, the manipulator, and the TMS coil are respectively electrically connected to the smart terminal; the smart terminal may be a computer;
所述躺式床为卧式平移平台，可以前后移动，用于调整患者头部与摄像头的相对位置；The lying bed is a horizontal translation platform, which can move back and forth, and is used to adjust the relative position of the patient's head and the camera;
所述头枕2主要起到托架作用，支撑位点为头骨，还包括颈部，起到的作用是限制病人的移动，且不会造成病人的不适，并且不能阻碍头背部的磁刺激；The headrest 2 mainly functions as a bracket, and the supporting site is the skull and also includes the neck. The function is to limit the movement of the patient without causing discomfort to the patient, and cannot hinder the magnetic stimulation of the back of the head;
所述3D摄像头用于获取患者头部和机械手的空间位姿，从而对机械手进行导航；The 3D camera is used to obtain the spatial pose of the patient's head and the manipulator, so as to navigate the manipulator;
所述机械手用于夹持TMS线圈对患者头部刺激磁刺激点进行磁刺激治疗；The manipulator is used to clamp the TMS coil to stimulate the magnetic stimulation points on the patient's head for magnetic stimulation treatment;
如图8所示，所述导航***的导航方法包括以下步骤：As shown in Figure 8, the navigation method of the navigation system includes the following steps:
S1，患者平躺在所述躺式床上，开启所述智能终端，通过智能终端调节所述躺式床的前后位置，使所述躺式床到达治疗位置；S1: The patient lies flat on the reclining bed, turning on the smart terminal, and adjusting the front and back positions of the reclining bed through the smart terminal, so that the reclining bed reaches the treatment position;
S2，采用所述3D摄像头和智能终端对患者头部进行建模；S2, using the 3D camera and smart terminal to model the patient's head;
S3，通过所述3D摄像头和智能终端将患者的头模的位置与患者头部的实际位置进行 匹配，确定患者头模上待磁刺激磁刺激点的空间位置；S3: Match the position of the patient's head model with the actual position of the patient's head through the 3D camera and the smart terminal, and determine the spatial position of the magnetic stimulation point on the patient's head model to be magnetically stimulated;
S4，通过所述智能终端对机械手、TMS线圈和3D摄像头进行建模；S4, modeling the manipulator, TMS coil, and 3D camera through the smart terminal;
S5，将步骤S4中建好的设备模型与步骤S2中建好的患者头模放在同一个空间坐标系中；再通过所述智能终端计算出TMS线圈模型到达头模上待磁刺激磁刺激点的最佳路径(移动距离最短，且移动过程中不会与其它设备发送碰撞)，所述智能终端再根据该最佳路径对机械手的移动进行自动导航，最终将所述TMS线圈移动至患者头部待磁刺激磁刺激点进行治疗。S5: Put the device model built in step S4 and the patient head model built in step S2 in the same spatial coordinate system; and then calculate through the smart terminal that the TMS coil model arrives on the head model for magnetic stimulation The optimal path of the point (the shortest moving distance and no collision with other devices during the movement), the smart terminal then automatically navigates the movement of the manipulator according to the optimal path, and finally moves the TMS coil to the patient The head is to be treated by magnetic stimulation at the magnetic stimulation point.
具体地，步骤S2中，对患者头部进行建模包括以下步骤：Specifically, in step S2, modeling the patient's head includes the following steps:
S21，通过所述3D摄像头从各个方向采集患者头部的3D图像数据，并将采集到的3D图像数据发送给所述智能终端；S21, collecting 3D image data of the patient's head from various directions through the 3D camera, and sending the collected 3D image data to the smart terminal;
S22，通过所述智能终端将所述3D图像数据进行整合，得到患者头部完整的3D点云图像，再通过抽样、平滑、平面拟合处理后得到患者头部完整的3D头模数据；S22: Integrate the 3D image data through the smart terminal to obtain a complete 3D point cloud image of the patient's head, and then obtain complete 3D head model data of the patient's head after sampling, smoothing, and plane fitting;
对所述3D图像数据进行整合的方法为：通过识别各个方向采集到的图像中的特征点计算出各个图像之间的匹配关系，再通过3D点云的ICP算法得到各个方向采集到的点云图像之间的空间位置关系，最后根据所述匹配关系和空间位置关系将所有的点云图像数据进行旋转和平移操作，得到患者头部完整的3D点云图像；The method of integrating the 3D image data is: calculating the matching relationship between the images by identifying the feature points in the images collected in each direction, and then obtaining the point clouds collected in each direction through the ICP algorithm of the 3D point cloud The spatial position relationship between the images, and finally all the point cloud image data are rotated and translated according to the matching relationship and the spatial position relationship to obtain a complete 3D point cloud image of the patient's head;
S23，利用所述3D头模数据，结合MNI脑空间坐标，将MNI空间的颅骨3D数据映射到患者的3D头模数据上，得到患者的3D头模，然后在患者的3D头模上建立磁刺激点模型。S23. Using the 3D head model data, combined with the MNI brain space coordinates, map the skull 3D data in the MNI space to the patient's 3D head model data to obtain the patient's 3D head model, and then establish a magnetic field on the patient's 3D head model Stimulus point model.
具体地，步骤S3中，3D摄像头实时拍摄的3D图像只有患者的面部信息，没有头部信息，所以要将S2中建好的头模与实时拍摄的面部数据进行位置的配准，由于ICP算法计算量较大，无法满足实时检测的要求，位置配准方法是先在头模中标出用于配准的面部特征点(眉心、耳垂、眼角、鼻尖、嘴角和下巴)，再在实时图像中自动识别出面部特征点，通过特征点匹配计算出实时画面与头模的转换关系，并计算出头模在空间的位置，再计算出头模上的磁刺激点在空间中的位置坐标；具体步骤如下：Specifically, in step S3, the 3D image captured by the 3D camera in real time has only the patient’s facial information and no head information. Therefore, the head model built in S2 must be registered with the real-time facial data. Due to the ICP algorithm The amount of calculation is too large to meet the requirements of real-time detection. The position registration method is to first mark the facial feature points (brow center, earlobe, corner of the eye, nose tip, corner of the mouth and chin) for registration in the head model, and then in the real-time image Automatically recognize facial feature points, calculate the conversion relationship between the real-time image and the head model through feature point matching, and calculate the position of the head model in space, and then calculate the position coordinates of the magnetic stimulation points on the head model in space; the specific steps are as follows :
S31，在患者头模上标出用于配准的面部特征点；S31, mark the facial feature points for registration on the patient's head model;
S32，通过3D摄像头识别出患者面部的特征点；S32: Recognizing the feature points of the patient's face through the 3D camera;
S33，将步骤S31中标出的面部特征点与步骤S32中识别出的面部特征点进行匹配计算，得到患者头部与患者头模的旋转、平移关系；S33: Perform a matching calculation between the facial feature points marked in step S31 and the facial feature points identified in step S32 to obtain the rotation and translation relationship between the patient's head and the patient's head model;
S34，根据所述旋转、平移关系对患者头模进行旋转、平移操作，使患者头模的位置与患者头部的实际位置匹配上。S34: Perform rotation and translation operations on the patient's head mold according to the rotation and translation relationship, so that the position of the patient's head mold matches the actual position of the patient's head.
具体地，步骤S4中，对所述机械手、TMS线圈和3D摄像头建模可采用SolidWorks软件进行建模，建模完成后需要将机械手模型、TMS线圈模型、3D摄像头模型的空间位置分别与机械手、TMS线圈、3D摄像头的实际空间位置进行匹配；具体匹配方法为：Specifically, in step S4, SolidWorks software can be used to model the manipulator, TMS coil and 3D camera. After the modeling is completed, the spatial positions of the manipulator model, TMS coil model, and 3D camera model need to be compared with the manipulator, The actual spatial position of the TMS coil and the 3D camera are matched; the specific matching method is:
S41,在机械手模型上标出用于配准的特征点；S41, mark the feature points for registration on the manipulator model;
S42，通过3D摄像头识别出机械手处于初始位置时的特征点；S42: Recognizing the characteristic points when the manipulator is in the initial position through the 3D camera;
S43，将步骤S41中标出的特征点与步骤S42中识别出的特征点进行匹配计算，得到机械手模型与机械手的旋转、平移关系；S43: Perform matching calculation between the feature points marked in step S41 and the feature points identified in step S42 to obtain the rotation and translation relationship between the manipulator model and the manipulator;
S44，根据机械手处于初始位置时所述3D摄像头、TMS线圈与机械手的相对位置是固定的原理，得到3D摄像头模型、TMS线圈模型分别与3D摄像头、TMS线圈的旋转、平移关系；S44: According to the principle that the relative positions of the 3D camera, the TMS coil and the manipulator are fixed when the manipulator is in the initial position, obtain the rotation and translation relationships of the 3D camera model and the TMS coil model with the 3D camera and the TMS coil respectively;
S45，根据步骤S43和步骤S44中的旋转、平移关系，对所述机械手模型、TMS线圈模型、3D摄像头模型进行旋转、平移操作，使所述机械手模型、TMS线圈模型、3D摄像头模型的空间位置分别与机械手、TMS线圈、3D摄像头的实际空间位置匹配上。S45: Perform rotation and translation operations on the manipulator model, TMS coil model, and 3D camera model according to the rotation and translation relationship in step S43 and step S44, so that the manipulator model, TMS coil model, and 3D camera model are positioned in space They are matched with the actual spatial positions of the manipulator, TMS coil and 3D camera.
具体地，步骤S5中，一般机械手的移动路径规划算法比较复杂，由于本实施例中的模型、障碍和路径都是已知的，故采用手动规划路径的方法，在距离头模较远(大于30mm)的位置使用直线路径，在头模附近(小于/等于30mm)使用圆弧路径，使TMS线圈围绕头部运动到下一个磁刺激磁刺激点；由于头模的3D数据是已知的，故可将头模数据放大从而留出运行的安全距离，计算出头模上两点的最短弧线路径。Specifically, in step S5, the general movement path planning algorithm of the manipulator is relatively complicated. Since the model, obstacles, and path in this embodiment are all known, the method of manually planning the path is adopted, and the distance from the head mold is far (greater than Use a straight path at the position of 30mm), and use a circular path near the head mold (less than/equal to 30mm) to move the TMS coil around the head to the next magnetic stimulation point; since the 3D data of the head mold is known, Therefore, the head mold data can be enlarged to leave a safe distance for operation, and the shortest arc path between two points on the head mold can be calculated.
本发明通过自动对机械手的移动进行导航，避免了人为因素对治疗效果造成的影响， 同时提升了患者的体验感。By automatically navigating the movement of the manipulator, the present invention avoids the influence of human factors on the treatment effect, and at the same time improves the patient's sense of experience.
作为本发明的又一实施方式，本发明还具有跟随定位功能。在对机械手进行导航的过程中，或者在进行人脸检测的过程中，即使患者头部姿态发生变化，也能通过3D摄像头对患者头部的姿态进行实时跟随定位，保证治疗的精准性，提高治疗效果和患者的体验感。As another embodiment of the present invention, the present invention also has a following positioning function. In the process of navigating the manipulator, or in the process of face detection, even if the posture of the patient's head changes, the posture of the patient's head can be tracked in real time through the 3D camera to ensure the accuracy of treatment and improve Treatment effect and patient experience.
具体地，在对患者头部进行磁刺激治疗的过程中，所述智能终端还通过3D摄像头对患者头部进行跟随定位；在治疗过程中会记录每次定位完成时患者头部的位置信息，若下一时刻由于患者头部运动造成当前时刻与上一时刻的磁刺激点位置距离超过5mm，则启动跟随定位；若不超过5mm，则不启动跟随定位；若患者头部转动次数较多，则暂停3D摄像头和机械手的跟随，并同时暂停TMS线圈的磁刺激；若患者不在3D摄像头的可调整范围内或离开，则停止机械手和线圈的磁刺激动作。Specifically, in the process of performing magnetic stimulation treatment on the patient's head, the smart terminal also uses a 3D camera to follow the positioning of the patient's head; during the treatment, the position information of the patient's head is recorded every time the positioning is completed, If the distance between the current and the previous magnetic stimulation point is more than 5mm due to the movement of the patient’s head at the next moment, follow-up positioning will be activated; if it does not exceed 5mm, follow-up positioning will not be activated; if the patient’s head turns more often, Then the 3D camera and the manipulator will be suspended, and the magnetic stimulation of the TMS coil will be suspended at the same time; if the patient is not within the adjustable range of the 3D camera or left, the magnetic stimulation action of the manipulator and the coil will be stopped.
进一步地，所述跟随定位的步骤为：通过所述智能终端对患者头部模型的空间位姿进行调整，使患者头部模型的空间位姿与患者头部当前的实际空间位姿进行匹配，然后在头部模型上重新定位最新的磁刺激位点，最后重新规划机械手的移动路径，将TMS线圈移动到最新的磁刺激位点进行治疗。Further, the step of following positioning is: adjusting the spatial pose of the patient's head model through the smart terminal, so that the spatial pose of the patient's head model is matched with the current actual spatial pose of the patient's head, Then reposition the latest magnetic stimulation site on the head model, and finally re-plan the moving path of the manipulator, and move the TMS coil to the latest magnetic stimulation site for treatment.
本发明通过摄像头拍摄患者头部的视频图像数据，对患者的头部进行建模，并根据建模数据以及拍摄的人脸视频图像对患者的人脸姿态进行检测估计，得到患者的人脸姿态数据，然后根据人脸姿态数据进行机械手导航，调整TMS治疗磁刺激点，在无需佩戴导光球定位的情况下保证每次治疗时磁刺激点定位的精准，解决了TMS定位及重复定位的问题。The invention uses a camera to capture video image data of the patient's head, models the patient's head, and detects and estimates the patient's facial posture based on the modeling data and the captured facial video image to obtain the patient's facial posture According to the data, the manipulator navigation is carried out according to the face posture data, and the TMS treatment magnetic stimulation point is adjusted. It ensures the accurate positioning of the magnetic stimulation point during each treatment without the need to wear a light guide ball, which solves the problem of TMS positioning and repeated positioning .
尽管已经示出和描述了本发明的实施例，对于本领域的普通技术人员而言，可以理解在不脱离本发明的原理和精神的情况下可以对这些实施例进行多种变化、修改、替换和变型，本发明的范围由所附权利要求及其等同物限定。Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. And variations, the scope of the present invention is defined by the appended claims and their equivalents.

Claims (10)

  1. 一种用于经颅磁刺激诊疗的摄像***，其特征在于，包括用于头部建模的第一拍摄子***、用于诊疗的第二拍摄子***和智能终端，所述第一拍摄子***包括3D扫描仪和定位帽，所述第二拍摄子***包括3D摄像头、躺式床和头枕；所述第一拍摄子***将拍摄到的头部数据发送至智能终端进行头部建模，得到具有已标识面部特征点的3D头模；所述第二拍摄子***将实时拍摄到的人脸数据发送至智能终端进行人脸检测，所述智能终端将检测出的人脸图像与所述3D头模进行匹配，得到用于经颅磁刺激诊疗的磁刺激点定位信息。A camera system for transcranial magnetic stimulation diagnosis and treatment, which is characterized in that it comprises a first photographing subsystem for head modeling, a second photographing subsystem for diagnosis and treatment, and an intelligent terminal. The system includes a 3D scanner and a positioning cap. The second photographing subsystem includes a 3D camera, a lying bed and a headrest; the first photographing subsystem sends the photographed head data to the intelligent terminal for head modeling , Obtain a 3D head model with identified facial feature points; the second photographing subsystem sends the real-time photographed face data to the smart terminal for face detection, and the smart terminal compares the detected face image with all The 3D head model is matched to obtain the location information of the magnetic stimulation point for transcranial magnetic stimulation diagnosis and treatment.
  2. 根据权利要求1所述的一种用于经颅磁刺激诊疗的摄像***，其特征在于，所述3D扫描仪包括一个3D摄像头和一个旋转支架，所述3D摄像头安装在旋转支架上，所述旋转支架由电机驱动旋转，所述电机与智能终端电连接；采集患者头部的3D图像数据时，通过智能终端控制电机驱动旋转支架匀速转动，从而使3D摄像头匀速绕患者头部做圆周运动，从各个方向采集患者头部的3D图像数据。The camera system for diagnosis and treatment of transcranial magnetic stimulation according to claim 1, wherein the 3D scanner comprises a 3D camera and a rotating bracket, the 3D camera is mounted on the rotating bracket, and The rotating bracket is driven to rotate by a motor, which is electrically connected to the smart terminal; when collecting 3D image data of the patient's head, the smart terminal controls the motor to drive the rotating bracket to rotate at a uniform speed, so that the 3D camera moves in a circular motion around the patient's head at a uniform speed. Collect 3D image data of the patient's head from all directions.
  3. 根据权利要求1所述的一种用于经颅磁刺激诊疗的摄像***，其特征在于，所述3D扫描仪包括若干个3D摄像头和一个固定支架，所述若干个3D摄像头均安装在固定支架上；采集患者头部的3D图像数据时，通过智能终端控制所述若干个3D摄像头同时从不同方向采集患者头部的3D图像数据。The camera system for diagnosis and treatment of transcranial magnetic stimulation according to claim 1, wherein the 3D scanner includes a plurality of 3D cameras and a fixed bracket, and the plurality of 3D cameras are all mounted on the fixed bracket Above; when collecting 3D image data of the patient's head, the smart terminal controls the several 3D cameras to simultaneously collect 3D image data of the patient's head from different directions.
  4. 根据权利要求2或3中任一项所述的一种用于经颅磁刺激诊疗的摄像***，其特征在于，所述3D摄像头拍摄到的图像数据包括彩色图像、深度图像和3D点云图像。The camera system for transcranial magnetic stimulation diagnosis and treatment according to any one of claims 2 or 3, wherein the image data captured by the 3D camera includes color images, depth images and 3D point cloud images .
  5. 根据权利要求1所述的一种用于经颅磁刺激诊疗的摄像***，其特征在于，所述定位帽为弹性材质做成的白色头罩，用于遮住患者头发；所述定位帽上设有若干Mark点，便于3D摄像头采集图像数据。The camera system for diagnosis and treatment of transcranial magnetic stimulation according to claim 1, wherein the positioning cap is a white hood made of elastic material for covering the patient's hair; There are several Mark points to facilitate the 3D camera to collect image data.
  6. 一种基于摄像头的经颅磁刺激诊疗建模***，采用权利要求1中所述的摄像***，其特征在于，所述建模***的建模方法包括以下步骤：A camera-based transcranial magnetic stimulation diagnosis and treatment modeling system adopts the camera system described in claim 1, wherein the modeling method of the modeling system includes the following steps:
    S1，启动所述智能终端，通过所述3D扫描仪从各个方向采集佩戴有定位帽的患者头部的3D图像数据，并将采集到的3D图像数据发送给所述智能终端；S1. Start the smart terminal, collect 3D image data of the patient's head wearing the positioning cap from various directions through the 3D scanner, and send the collected 3D image data to the smart terminal;
    S2，通过所述智能终端将3D扫描仪从各个方向采集到的3D图像数据进行整合，得到患者头部完整的3D点云图像，再通过抽样、平滑、平面拟合处理后得到患者头部完整的3D头模数据；S2: Integrate the 3D image data collected by the 3D scanner from various directions through the smart terminal to obtain a complete 3D point cloud image of the patient's head, and then obtain the complete patient's head after sampling, smoothing, and plane fitting. 3D head model data;
    S3，利用所述3D头模数据，结合MNI脑空间坐标，将MNI空间的颅骨3D数据映射到患者的3D头模数据上，得到患者的3D头模，然后在患者的3D头模上建立磁刺激点模型。S3. Using the 3D head model data, combined with the MNI brain space coordinates, map the skull 3D data in the MNI space to the patient's 3D head model data to obtain the patient's 3D head model, and then establish a magnetic field on the patient's 3D head model Stimulus point model.
  7. 一种基于摄像头的经颅磁刺激诊疗检测***，采用权利要求1中所述的摄像***，其特征在于，所述检测***的检测方法包括以下步骤：A camera-based transcranial magnetic stimulation diagnosis and treatment detection system adopts the camera system described in claim 1, wherein the detection method of the detection system includes the following steps:
    S1，通过调节所述躺式床的前后位置，使载有患者的躺式床到达治疗位置；S1, by adjusting the front and rear positions of the lying bed, the lying bed carrying the patient reaches the treatment position;
    S2，治疗开始前，采用所述3D摄像头拍摄患者头部的图像数据，采用所述智能终端进行头部建模，建立患者头部的3D头模；S2, before the treatment starts, the 3D camera is used to take image data of the patient's head, and the intelligent terminal is used to model the head to establish a 3D head model of the patient's head;
    S3，治疗开始，采用所述3D摄像头拍摄患者的实时面部图像，采用所述智能终端进行位姿匹配，将所述实时面部图像与已建立的3D头模进行位置匹配，进一步包括：在所述3D头模中标出用于匹配的面部特征点；通过所述3D摄像头自动识别出患者的实时面部图像的面部特征点；通过特征点匹配进行仿射变换得到转换矩阵，计算出患者的实时面部图像与已建立的3D头模的转换关系；计算所述3D头模在空间中的位置；计算所述3D头模上的磁刺激点在空间中的位置坐标。S3. At the beginning of treatment, the 3D camera is used to take a real-time facial image of the patient, the smart terminal is used to perform pose matching, and the real-time facial image is matched with the established 3D head model for position matching, which further includes: The facial feature points used for matching are marked in the 3D head model; the facial feature points of the patient's real-time facial image are automatically recognized by the 3D camera; the conversion matrix is obtained by affine transformation through feature point matching, and the patient's real-time facial image is calculated The conversion relationship with the established 3D head mold; calculate the position of the 3D head mold in space; calculate the position coordinates of the magnetic stimulation points on the 3D head mold in space.
  8. 一种基于摄像头的经颅磁刺激诊疗导航***，采用权利要求1中所述的摄像***，其特征在于，还包括机械手和TMS线圈，所述机械手、TMS线圈分别与智能终端电连接，所述导航***的导航方法包括以下步骤：A camera-based transcranial magnetic stimulation diagnosis and treatment navigation system, which adopts the camera system described in claim 1, is characterized in that it further comprises a manipulator and a TMS coil, the manipulator and the TMS coil are electrically connected to the smart terminal, and the The navigation method of the navigation system includes the following steps:
    S1，通过调节所述躺式床的前后位置，使载有患者的躺式床到达治疗位置；S1, by adjusting the front and rear positions of the lying bed, the lying bed carrying the patient reaches the treatment position;
    S2，采用所述3D摄像头和智能终端对患者头部进行建模；S2, using the 3D camera and smart terminal to model the patient's head;
    S3，通过所述3D摄像头和智能终端将患者的头模的位置与患者头部的实际位置进行匹配，确定患者头模上待磁刺激磁刺激点的空间位置；S3: Match the position of the patient's head model with the actual position of the patient's head through the 3D camera and the smart terminal, and determine the spatial position of the magnetic stimulation point on the patient's head model to be magnetically stimulated;
    S4，通过所述智能终端对机械手、TMS线圈和3D摄像头进行建模；S4, modeling the manipulator, TMS coil, and 3D camera through the smart terminal;
    S5，将步骤S4中建好的设备模型与步骤S2中建好的患者头模放在同一个空间坐标系 中；再通过所述智能终端计算出TMS线圈模型到达头模上待磁刺激磁刺激点的最佳路径，所述智能终端再根据该最佳路径对机械手的移动进行自动导航，最终将所述TMS线圈移动至患者头部待磁刺激磁刺激点进行治疗。S5: Put the device model built in step S4 and the patient head model built in step S2 in the same spatial coordinate system; and then calculate through the smart terminal that the TMS coil model arrives on the head model for magnetic stimulation The smart terminal automatically navigates the movement of the manipulator according to the optimal path, and finally moves the TMS coil to the magnetic stimulation point on the patient's head for treatment.
  9. 根据权利要求7所述的一种基于摄像头的经颅磁刺激诊疗检测***，其特征在于，所述检测方法还包括：在对患者头部进行磁刺激治疗的过程中，所述智能终端还通过3D摄像头对患者头部进行跟随定位；在治疗过程中会记录每次定位完成时患者头部磁刺激点的位置信息，若下一时刻由于患者头部运动造成当前时刻与上一时刻的磁刺激点位置距离超过5mm，则启动跟随定位；若不超过5mm，则不启动跟随定位。The camera-based transcranial magnetic stimulation diagnosis and treatment system according to claim 7, wherein the detection method further comprises: in the process of performing magnetic stimulation treatment on the patient's head, the smart terminal also passes The 3D camera follows the positioning of the patient's head; during the treatment, the position information of the magnetic stimulation point on the patient's head is recorded each time the positioning is completed. If the next moment is caused by the patient's head movement, the current and previous magnetic stimulation If the distance of the point position exceeds 5mm, the following positioning will be started; if it is less than 5mm, the following positioning will not be started.
  10. 根据权利要求8所述的一种基于摄像头的经颅磁刺激诊疗导航***，其特征在于，所述导航方法还包括跟随定位步骤，所述跟随定位步骤包括：通过所述智能终端对患者头部模型的空间位姿进行调整，使患者头部模型的空间位姿与患者头部当前的实际空间位姿进行匹配，然后在头部模型上重新定位最新的磁刺激点，最后重新规划机械手的移动路径，将TMS线圈移动到最新的磁刺激点进行治疗。The camera-based transcranial magnetic stimulation diagnosis and treatment navigation system according to claim 8, wherein the navigation method further comprises a following positioning step, and the following positioning step comprises: using the smart terminal to control the patient's head Adjust the spatial pose of the model to match the spatial pose of the patient's head model with the current actual spatial pose of the patient's head, then reposition the latest magnetic stimulation point on the head model, and finally re-plan the movement of the manipulator Path, move the TMS coil to the latest magnetic stimulation point for treatment.
PCT/CN2019/076100 2019-02-26 2019-02-26 Camera system for transcranial magnetic stimulation therapy WO2020172779A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201980001093.0A CN110300993B (en) 2019-02-26 2019-02-26 Imaging system for transcranial magnetic stimulation diagnosis and treatment
PCT/CN2019/076100 WO2020172779A1 (en) 2019-02-26 2019-02-26 Camera system for transcranial magnetic stimulation therapy

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/076100 WO2020172779A1 (en) 2019-02-26 2019-02-26 Camera system for transcranial magnetic stimulation therapy

Publications (1)

Publication Number Publication Date
WO2020172779A1 true WO2020172779A1 (en) 2020-09-03

Family

ID=68033040

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/076100 WO2020172779A1 (en) 2019-02-26 2019-02-26 Camera system for transcranial magnetic stimulation therapy

Country Status (2)

Country Link
CN (1) CN110300993B (en)
WO (1) WO2020172779A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112634336A (en) * 2020-12-31 2021-04-09 华科精准（北京）医疗科技有限公司 Registration method and system

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111216109A (en) * 2019-10-22 2020-06-02 东北大学 Visual following device and method for clinical treatment and detection
CN111249622B (en) * 2020-01-17 2024-04-09 上海市精神卫生中心（上海市心理咨询培训中心） Accurate transcranial magnetic stimulation on-line navigation method based on augmented reality
CN111729200B (en) * 2020-07-27 2022-06-17 浙江大学 Transcranial magnetic stimulation automatic navigation system and method based on depth camera and magnetic resonance
CN113041500A (en) * 2021-03-31 2021-06-29 南京伟思医疗科技股份有限公司 Memorable transcranial magnetic stimulation navigation positioning device and positioning method
CN115008753A (en) * 2022-06-28 2022-09-06 德清县第三人民医院 3D printing accurate positioning navigation transcranial magnetic stimulation head model and manufacturing method

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060161039A1 (en) * 2005-01-20 2006-07-20 Neuronetics, Inc. Articulating arm
CN102814002A (en) * 2012-08-08 2012-12-12 深圳先进技术研究院 Cerebral magnetic stimulation navigation system and cerebral magnetic stimulation coil positioning method
US20130345491A1 (en) * 2011-03-09 2013-12-26 A School Corporation Kansai University Image data processing device and transcranial magnetic stimulation apparatus
CN107497049A (en) * 2017-09-30 2017-12-22 武汉资联虹康科技股份有限公司 A kind of electromagnetic location air navigation aid and device for transcranial magnetic stimulation device
US20170372006A1 (en) * 2014-02-10 2017-12-28 Neuronetics, Inc. Head modeling for a therapeutic or diagnostic procedure
CN109173063A (en) * 2018-09-21 2019-01-11 武汉资联虹康科技股份有限公司 A kind of the transcranial magnetic stimulation system and transcranial magnetic stimulation method of 3D scanning
CN109260593A (en) * 2018-09-27 2019-01-25 武汉资联虹康科技股份有限公司 A kind of transcranial magnetic stimulation treatment method and apparatus

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7651459B2 (en) * 2004-01-06 2010-01-26 Neuronetics, Inc. Method and apparatus for coil positioning for TMS studies
FR2970638B1 (en) * 2011-01-26 2014-03-07 Inst Nat Rech Inf Automat METHOD AND SYSTEM FOR ASSISTING THE POSITIONING OF A MEDICAL TOOL ON THE HEAD OF A SUBJECT
CN104883974B (en) * 2012-10-26 2019-03-19 瓦里安医疗***公司 The targeting of NIR image guidance
US10980273B2 (en) * 2013-11-12 2021-04-20 VMR Products, LLC Vaporizer, charger and methods of use
CN106955122A (en) * 2017-03-20 2017-07-18 苏州网颢信息科技有限公司 Head 3 D detection method based on single camera module in radiotherapy

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060161039A1 (en) * 2005-01-20 2006-07-20 Neuronetics, Inc. Articulating arm
US20130345491A1 (en) * 2011-03-09 2013-12-26 A School Corporation Kansai University Image data processing device and transcranial magnetic stimulation apparatus
CN102814002A (en) * 2012-08-08 2012-12-12 深圳先进技术研究院 Cerebral magnetic stimulation navigation system and cerebral magnetic stimulation coil positioning method
US20170372006A1 (en) * 2014-02-10 2017-12-28 Neuronetics, Inc. Head modeling for a therapeutic or diagnostic procedure
CN107497049A (en) * 2017-09-30 2017-12-22 武汉资联虹康科技股份有限公司 A kind of electromagnetic location air navigation aid and device for transcranial magnetic stimulation device
CN109173063A (en) * 2018-09-21 2019-01-11 武汉资联虹康科技股份有限公司 A kind of the transcranial magnetic stimulation system and transcranial magnetic stimulation method of 3D scanning
CN109260593A (en) * 2018-09-27 2019-01-25 武汉资联虹康科技股份有限公司 A kind of transcranial magnetic stimulation treatment method and apparatus

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112634336A (en) * 2020-12-31 2021-04-09 华科精准（北京）医疗科技有限公司 Registration method and system

Also Published As

Publication number Publication date
CN110300993B (en) 2023-10-27
CN110300993A (en) 2019-10-01

Similar Documents

Publication Publication Date Title
EP3858433B1 (en) Tms positioning navigation apparatus for transcranial magnetic stimulation treatment
WO2020172779A1 (en) Camera system for transcranial magnetic stimulation therapy
WO2020172781A1 (en) Camera-based transcranial magnetic stimulation diagnosis and treatment navigation system
CN110896609B (en) TMS positioning navigation method for transcranial magnetic stimulation treatment
WO2020172782A1 (en) Transcranial magnetic stimulation diagnosis and treatment detection system based on camera
TWI678181B (en) Surgical guidance system
CN109173062B (en) Efficient TMS (TMS) repeated positioning method
CN108888341A (en) A kind of scaling method of augmented reality Helmet Mounted Display position real-time tracking
CN111729200B (en) Transcranial magnetic stimulation automatic navigation system and method based on depth camera and magnetic resonance
WO2015098155A1 (en) Operation teaching apparatus and transcranial magnetic stimulation apparatus
CN106139423A (en) A kind of image based on photographic head guides seeds implanted system
US11948247B2 (en) Camera-based transcranial magnetic stimulation diagnosis and treatment head modeling system
CN116630382B (en) Nerve regulation and control image monitoring registration system and control method
CN115006737A (en) Radiotherapy body position monitoring system based on depth camera
CN110896610A (en) Transcranial magnetic stimulation diagnosis and treatment equipment
US20240123248A1 (en) Transcranial magnetic stimulation system and method
US20240122537A1 (en) Transcranial magnetic stimulation system and method
CN112752537A (en) Method for determining at least one geometric form parameter of a subject in a natural posture for determining a vision correction equipment
US20240041415A1 (en) Chest x-ray system and method
WO2024069042A1 (en) A dental x-ray imaging system and a method for dental x-ray imaging of a patient
TW201211937A (en) Human face matching system and method thereof
Yasumuro et al. Coil positioning system for repetitive transcranial magnetic stimulation treatment by ToF camera ego-motion
CN113081013A (en) Method, device and system for scanning positioning sheet
CN115008753A (en) 3D printing accurate positioning navigation transcranial magnetic stimulation head model and manufacturing method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19917179

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19917179

Country of ref document: EP

Kind code of ref document: A1