CN113143185B - Gastroscope system, image display method and device and readable storage medium - Google Patents
Gastroscope system, image display method and device and readable storage medium Download PDFInfo
- Publication number
- CN113143185B CN113143185B CN202110441044.8A CN202110441044A CN113143185B CN 113143185 B CN113143185 B CN 113143185B CN 202110441044 A CN202110441044 A CN 202110441044A CN 113143185 B CN113143185 B CN 113143185B
- Authority
- CN
- China
- Prior art keywords
- information
- dimensional display
- image
- stomach
- display device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 210000002784 stomach Anatomy 0.000 claims abstract description 148
- 239000002775 capsule Substances 0.000 claims abstract description 97
- 238000005259 measurement Methods 0.000 claims abstract description 27
- 238000012545 processing Methods 0.000 claims description 17
- 230000006641 stabilisation Effects 0.000 claims description 13
- 238000011105 stabilization Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 5
- 238000012549 training Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000013527 convolutional neural network Methods 0.000 description 6
- 238000001914 filtration Methods 0.000 description 6
- 230000000087 stabilizing effect Effects 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000004297 night vision Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000010794 food waste Substances 0.000 description 1
- 210000001035 gastrointestinal tract Anatomy 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 230000001404 mediated effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000008855 peristalsis Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00043—Operational features of endoscopes provided with output arrangements
- A61B1/00045—Display arrangement
- A61B1/00048—Constructional features of the display
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00043—Operational features of endoscopes provided with output arrangements
- A61B1/00055—Operational features of endoscopes provided with output arrangements for alerting the user
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00147—Holding or positioning arrangements
- A61B1/00158—Holding or positioning arrangements using magnetic field
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
- A61B1/041—Capsule endoscopes for imaging
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/273—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for the upper alimentary canal, e.g. oesophagoscopes, gastroscopes
- A61B1/2736—Gastroscopes
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Biophysics (AREA)
- Physics & Mathematics (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Optics & Photonics (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Gastroenterology & Hepatology (AREA)
- Signal Processing (AREA)
- Endoscopes (AREA)
Abstract
The application discloses gastroscope system, method, device and non-volatile storage medium, this system includes capsule gastroscope, three-dimensional display device and waist ring, and the capsule gastroscope includes camera lens, geomagnetic sensor, inertial measurement sensor, and the waist ring includes a plurality of electromagnetic equipment, wherein: the capsule gastroscope acquires azimuth information based on electromagnetic equipment by using a geomagnetic sensor, and acquires attitude information by using an inertial measurement sensor; the capsule gastroscope acquires a stomach image by using a lens, and generates information to be transmitted by using the stomach image, azimuth information and attitude information; transmitting the information to be transmitted to the three-dimensional display equipment by utilizing the information to be transmitted; the three-dimensional display device performs three-dimensional display on the stomach image based on the azimuth information and the gesture information; the azimuth information represents the position and the orientation of the capsule gastroscope, the gesture information represents the gesture of the capsule gastroscope in the stomach, the position and the gesture information are matched with each other to determine the gesture of the capsule gastroscope in a three-dimensional space, the three-dimensional display is carried out on the stomach image, and the careful observation is allowed to be carried out on any part of the stomach.
Description
Technical Field
The present disclosure relates to the field of gastroscopes, and in particular, to a gastroscope system, an image display method, an image display device, and a nonvolatile storage medium.
Background
Gastroscope is a medical examination instrument which can directly observe the real condition of the examined part and is the preferred examination method of the upper digestive tract lesions. In the related art, a gastroscope generally includes a slim and flexible tube which is inserted into the stomach and connected to peripheral devices, and then images are captured through a camera lens installed at the front end of the tube, and then the images are processed through a video processor and displayed on a screen. However, the related art can only see a two-dimensional image, and the movement of the tube is limited to a certain extent, so that detailed observation for a specific portion is not possible.
Therefore, the problem of the related art that detailed observation cannot be performed for a specific portion is a technical problem that a person skilled in the art needs to solve.
Disclosure of Invention
In view of the above, an object of the present application is to provide a gastroscope system, an image display method, an image display device, and a nonvolatile storage medium, which can carefully observe an arbitrary portion of the stomach with a gastroscope.
To solve the above technical problem, the present application provides a gastroscope system, including:
capsule gastroscope, three-dimensional display device and waist ring, capsule gastroscope includes camera lens, geomagnetic sensor, inertial measurement sensor, waist ring includes a plurality of electromagnetic equipment, wherein:
the capsule gastroscope acquires azimuth information based on the electromagnetic equipment by using the geomagnetic sensor, and acquires attitude information by using the inertial measurement sensor;
the capsule gastroscope acquires a stomach image by using the lens, and generates information to be transmitted by using the stomach image, the azimuth information and the attitude information;
transmitting the information to be transmitted to the three-dimensional display equipment;
the three-dimensional display device performs three-dimensional display on the stomach image based on the azimuth information and the gesture information.
Optionally, the capsule gastroscope further comprises a magnet, the waist ring acquires control information, and the electromagnetic device is adjusted according to the control information; the electromagnetic device acts on the magnet to adjust the spatial position of the capsule gastroscope.
Optionally, the three-dimensional display device acquires motion information;
and generating the control information by utilizing the motion information, and sending the control information to the waist ring.
Optionally, the three-dimensional display device performs part recognition processing on the stomach image to obtain a recognition result;
and carrying out three-dimensional display on the stomach image and the recognition result based on the azimuth information and the gesture information.
Optionally, the three-dimensional display device performs mechanical image stabilization processing on the stomach image by using the gesture information to obtain a first stomach image;
performing digital image stabilization processing on the first stomach image to obtain a second stomach image;
and carrying out three-dimensional display on the second stomach image based on the azimuth information and the gesture information.
Optionally, the waist ring further comprises an identification signal transmitter corresponding to the electromagnetic device, and the three-dimensional display device comprises an identification signal receiver corresponding to the identification signal transmitter;
the three-dimensional display device acquires the identification signal sent by the identification signal transmitter by using the identification signal receiver;
and determining the position of the electromagnetic device on the waist ring by using the identification signal.
Optionally, the capsule gastroscope further comprises an exposure light.
The application also provides an image display method, which comprises the following steps:
the capsule gastroscope acquires azimuth information based on electromagnetic equipment in the waist ring by using a geomagnetic sensor, and acquires attitude information by using an inertial measurement sensor;
obtaining a stomach image, and generating information to be transmitted by using the stomach image, the azimuth information and the attitude information;
and sending the information to be transmitted to a three-dimensional display device so that the three-dimensional display device can perform three-dimensional display on the stomach image based on the azimuth information and the gesture information.
The application also provides an image display device, comprising:
the acquisition module is used for acquiring azimuth information based on electromagnetic equipment in the waist ring by using a geomagnetic sensor and acquiring attitude information by using an inertial measurement sensor;
the generation module is used for acquiring a stomach image and generating information to be transmitted by utilizing the stomach image, the azimuth information and the gesture information;
and the display module is used for sending the information to be transmitted to a three-dimensional display device so that the three-dimensional display device can display the stomach image in a three-dimensional mode based on the azimuth information and the gesture information.
The application also provides a non-volatile storage medium for storing a computer program, wherein the computer program realizes the image display method when being executed by a processor.
The utility model provides a gastroscope system, including capsule gastroscope, three-dimensional display device and waist ring, capsule gastroscope includes camera lens, geomagnetic sensor, inertial measurement sensor, and the waist ring includes a plurality of electromagnetic equipment, wherein: the capsule gastroscope acquires azimuth information based on electromagnetic equipment by using a geomagnetic sensor, and acquires attitude information by using an inertial measurement sensor; the capsule gastroscope acquires a stomach image by using a lens, and generates information to be transmitted by using the stomach image, azimuth information and attitude information; transmitting the information to be transmitted to the three-dimensional display equipment by utilizing the information to be transmitted; three-dimensional display device for three-dimensional display of stomach images based on azimuth information and attitude information
It can be seen that the system adopts the capsule gastroscope to replace the tube provided with the gastroscope, so that the system can flexibly move, and the problem that the tube movement is limited, so that the specific part cannot be carefully observed is avoided. The geomagnetic sensor in the capsule gastroscope can act with electromagnetic equipment in the waist ring to obtain azimuth information, and the azimuth information can represent the specific position of the capsule gastroscope in the plane of the waist ring and can also represent the orientation of the capsule gastroscope. The inertial measurement sensor can acquire attitude information, and the attitude information can represent the attitude of the capsule gastroscope in the stomach. Meanwhile, a stomach image is acquired by using a lens, and information to be transmitted is generated by using the stomach image, the gesture information and the azimuth information. After the three-dimensional display device acquires the information to be transmitted, the gesture information and the azimuth information can be used for determining the gesture of the capsule gastroscope in the three-dimensional space, so that the stomach image can be displayed in three dimensions based on the gesture information and the azimuth information. The three-dimensional display has a space sense compared with a two-dimensional image, can flexibly move in cooperation with the capsule gastroscope, can acquire and display the image of any stomach part, allows a user to carefully observe any stomach part, and solves the problem that the detailed observation cannot be performed on a specific part in the related technology.
In addition, the application also provides an image display method, an image display device and a nonvolatile storage medium, and the image display method, the image display device and the nonvolatile storage medium have the same beneficial effects.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings that are required to be used in the embodiments or the related technical descriptions will be briefly described below, and it is apparent that the drawings in the following description are only embodiments of the present application, and other drawings may be obtained according to the provided drawings without inventive effort to those of ordinary skill in the art.
Fig. 1 is a schematic structural diagram of a gastroscope system according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of a structure of a gastroscope system according to an embodiment of the present disclosure;
fig. 3 is a schematic distribution diagram of a specific electromagnetic device on a waist ring according to an embodiment of the present application;
fig. 4 is a flowchart of an image display method according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an image display device according to an embodiment of the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a gastroscope system according to an embodiment of the present application. Comprises a capsule gastroscope 11, a three-dimensional display device 12 and a waist ring 10, wherein the capsule gastroscope 11 comprises a lens 111, a geomagnetic sensor 112 and an inertial measurement sensor 113, and the waist ring 10 comprises an electromagnetic device 101. Specifically, the waist ring 10 is located outside the human body, and the capsule gastroscope 11 is located in the stomach of the human body. The electromagnetic devices 101 in the waist ring 10 are located at different positions of the waist ring 10, and the number of the electromagnetic devices is a plurality of electromagnetic devices, which are used for positioning and direction judgment of the capsule gastroscope 11. The electromagnetic device 101 may in particular be a solenoid and corresponding control components, or may be other electromagnetic components or a combination of electromagnetic components.
Wherein the capsule gastroscope 11 acquires azimuth information based on the electromagnetic device 101 using the geomagnetic sensor 112, and acquires attitude information using the inertial measurement sensor 113. The geomagnetic sensor may be also referred to as a geomagnetic chip, and the specific model is not limited. The geomagnetic sensor can receive electromagnetic signals formed by electromagnetic equipment and determine the azimuth information of the capsule gastroscope by using the electromagnetic signals. Because the capsule gastroscope is provided with the lens, the orientation of the lens is the orientation of the capsule gastroscope, and the orientation of the capsule gastroscope can reflect which part of the stomach corresponds to the stomach image acquired by the lens. Therefore, in this embodiment, the azimuth information is direction and position information, which includes not only the specific position of the capsule gastroscope on the lumbar ring plane, but also the direction information of the capsule gastroscope. The plane of the waist ring may be any plane parallel to the plane formed by the waist ring. Inertial measurement sensors, also known as inertial measurement units (Inertial Measurement Unit, IMUs), are not limited in their specific model numbers, and typically include gyroscopes and accelerometers, and some models of inertial measurement sensors may also include magnetometers. The inertial measurement sensor can detect the posture of the capsule gastroscope in the stomach to obtain posture information. The specific content of the posture information is not limited, and may be, for example, a distance of movement of the capsule gastroscope in up-down, left-right, front-back directions, or an angle of the capsule gastroscope in a vertical space at a certain time.
The capsule gastroscope 11 acquires a stomach image using the lens 111, and generates information to be transmitted using the stomach image, azimuth information, and posture information. It should be noted that the lens 111, the geomagnetic sensor 112, and the inertial measurement sensor 113 may work independently, that is, the three may work in parallel, for example, the capsule gastroscope may acquire the orientation information by using the geomagnetic sensor while acquiring the stomach image by using the lens, and acquire the posture information by using the inertial sensor. The lens, the geomagnetic sensor and the inertial measurement sensor can work in a mutual correlation mode, namely the lens, the geomagnetic sensor and the inertial measurement sensor can work in series, for example, a stomach image can be acquired by the aid of the lens, azimuth information is acquired by the aid of the geomagnetic sensor after the stomach image is acquired, and attitude information is acquired by the aid of the inertial sensor after the azimuth information is acquired. Since the stomach space is a dark space, in order to obtain valuable stomach images, in one embodiment, the lens may be a night vision lens with night vision; in another embodiment, the capsule gastroscope may further include an exposure light for supplementing the lens when taking the stomach image. After the stomach image, the azimuth information and the attitude information are acquired, the information to be transmitted can be generated by using the stomach image, the azimuth information and the attitude information. The information to be transmitted can be generated according to a preset format, and the specific content of the preset format is not limited, for example, the information can be MAC (Media Access Control) address-IP (Internet Protocol) address-UCD (Upstream Channel Descriptor Message, uplink channel description message) message header-timestamp-azimuth information-attitude information-stomach image.
After the information to be transmitted is generated, the information to be transmitted is sent to the three-dimensional display device 12. The three-dimensional display device 12 presents the stomach image in three dimensions based on the orientation information and the posture information. The azimuth information can indicate the plane direction and the position of the capsule gastroscope relative to the waist ring, the gesture information can indicate the space gesture of the capsule gastroscope, and the two can accurately indicate the position, the direction and the gesture in the three-dimensional space of the capsule gastroscope in a human body by combining the two, so that the three-dimensional reality equipment can carry out three-dimensional display on stomach images. The particular manner of three-dimensional presentation is related to the particular type of three-dimensional presentation device, which may be a VR (Virtual Reality) device, AR (Augmented Reality) technology, MR (Mediated Reality) device, or the like.
The gastroscope system provided by the embodiment of the application adopts the capsule gastroscope to replace the tube provided with the gastroscope, so that the motion can be flexible, and the problem that the tube cannot be carefully observed at a specific position due to the limited motion is avoided. The geomagnetic sensor in the capsule gastroscope can act with electromagnetic equipment in the waist ring to obtain azimuth information, and the azimuth information can represent the specific position of the capsule gastroscope in the plane of the waist ring and can also represent the orientation of the capsule gastroscope. The inertial measurement sensor can acquire attitude information, and the attitude information can represent the attitude of the capsule gastroscope in the stomach. Meanwhile, a stomach image is acquired by using a lens, and information to be transmitted is generated by using the stomach image, the gesture information and the azimuth information. After the three-dimensional display device acquires the information to be transmitted, the gesture information and the azimuth information can be used for determining the gesture of the capsule gastroscope in the three-dimensional space, so that the stomach image can be displayed in three dimensions based on the gesture information and the azimuth information. The three-dimensional display has a space sense compared with a two-dimensional image, can flexibly move in cooperation with the capsule gastroscope, can acquire and display the image of any stomach part, allows a user to carefully observe any stomach part, and solves the problem that the detailed observation cannot be performed on a specific part in the related technology.
Based on the above embodiments, the present embodiment will be further described with respect to a gastroscope system. In one embodiment, in order to conveniently and accurately adjust the position, posture, etc. of the capsule gastroscope, the capsule gastroscope may further include a magnet, which may be a permanent magnet or an electromagnet. The waist ring can acquire control information, the control information is used for adjusting electromagnetic equipment, the electromagnetic equipment acts on the magnet, and then the space position of the capsule gastroscope is adjusted, and the space position comprises the position and the gesture of the capsule gastroscope. The specific adjusting process of the electromagnetic device is not limited, for example, when the electromagnetic device is an electromagnetic coil, the current size, direction and the like of the electromagnetic device can be adjusted; when the electromagnetic equipment is an electromagnetic coil and a corresponding control component, the control signal sent to the control component can be adjusted, and the adjustment of the electromagnetic equipment is completed. The present embodiment is not limited to the transmitting end of the control information, and may be, for example, a three-dimensional display device, or may be a dedicated control device.
Further, it will be appreciated that since the orientation information is used to represent the positional relationship between the capsule gastroscope and the electromagnetic device, if it is desired to determine the specific spatial position of the capsule gastroscope in the human body (i.e. the stomach of the human body), it is first necessary to determine the position of the electromagnetic device on the waist ring, and thus the position of the electromagnetic device with respect to the human body. In one embodiment, physical markers may be placed on the lumbar ring and the position of the electromagnetic device relative to the human body determined from the markers. In this case, the waist ring needs to be worn in accordance with the mark, and if the waist ring is not worn in accordance with the mark, there is a problem in that the stomach region to be observed and the actual stomach image are not matched. For example, if one wants to observe the stomach right in front of it, if the waist ring is worn askew, the resulting stomach image may be an image of the stomach right in front of it. To solve this problem, the waist ring may further include an identification signal transmitter corresponding to the electromagnetic device, and the three-dimensional display device includes an identification signal receiver corresponding to the identification signal transmitter. The three-dimensional display device obtains the identification signal sent by the identification signal transmitter by using the identification signal receiver, and determines the first position of the electromagnetic device on the waist ring by using the identification signal. The identification signal transmitter corresponds to the identification signal receiver, and the specific type thereof is not limited, and for example, an infrared transmitter and an infrared receiver may be used. After the first position is obtained, a basic coordinate system can be updated based on the first position, namely, the coordinate relation between the electromagnetic device and the human body is updated, and accordingly, when the three-dimensional display device performs three-dimensional display on the stomach image based on the azimuth information and the gesture information, the stomach image can be subjected to three-dimensional display based on the azimuth information and the gesture information under the basic coordinate system, so that the stomach image corresponds to the stomach part to be observed. Correspondingly, after updating the basic coordinate system, the control information can also be generated according to the updated basic coordinate system so as to accurately adjust the moving direction of the capsule gastroscope in the stomach.
Still further, in an embodiment, when the three-dimensional display device is a wearable three-dimensional display device such as a VR headset, the electromagnetic device may be adjusted along with movement of the three-dimensional display device, so as to obtain a stomach image matched with the movement of the three-dimensional display device, so that the user can adjust the stomach image more conveniently, and observe the stomach. Specifically, the three-dimensional display device acquires motion information, generates control information using the motion information, and transmits the control information to the lumbar ring. The embodiment is not limited to a specific generation manner and process of the control information, and may refer to the related art.
Based on any of the above embodiments, because the movement mode and the falling position of the capsule gastroscope after being swallowed into the stomach are unexpected, the positions of the stomach of different people in the human body are different, and the problems of the environment in the stomach (such as poor light, food residues and the like) are difficult to avoid. Thus, when the stomach image is acquired, the user may not be able to directly determine the specific position of the capsule gastroscope in the stomach from the stomach image, and it takes much time. To solve the above problem, in one possible embodiment, the process of three-dimensionally displaying the stomach image by the three-dimensional display device may include:
the three-dimensional display equipment performs part recognition processing on the stomach image to obtain a recognition result; and carrying out three-dimensional display on the stomach image and the recognition result based on the azimuth information and the gesture information. That is, in this embodiment, the three-dimensional display device may recognize what portion of the stomach corresponds to the stomach, and display the result of the recognition together with the stomach image, so that the user may directly determine the position of the capsule gastroscope. The present embodiment is not limited to the processing method of performing the part recognition processing on the stomach image, and for example, the part recognition processing may be performed by using a convolutional neural network. Specifically, the convolutional neural network may be trained based on training data, where the training data may be a training image having corresponding marker information, and the training image may be a directly acquired stomach image. Or in order to improve the recognition capability of the convolutional neural network, the training data can be stomach images acquired under different angles, different light conditions and different shielding conditions, the stomach images acquired under the conditions can generate local feature indexes, the convolutional neural network can master the local feature indexes by training the convolutional neural network, and the training process is the process that the convolutional neural network master the local feature indexes step by step. When the training is finished and the stomach is identified, the stomach image can be matched in the local feature index to obtain a corresponding identification result. The specific training process is not limited, for example, the training image may be initialized, the feature extracted, the loss calculated, etc., and the specific parameter correlation technique is not described herein.
When the stomach image display is carried out, the conditions such as human body shaking, stomach peristalsis or capsule gastroscope shaking can appear, and the conditions can lead to capsule gastroscope shaking, and then cause the video that comprises the stomach image to shake in succession, be unfavorable for observing the stomach. In order to solve the above problems, in the present application, a three-dimensional display device performs mechanical image stabilization processing on a stomach image using pose information to obtain a first stomach image. The gesture information is acquired by the IMU sensor, so that the gesture information can be calibrated with a camera coordinate system, the motion condition of the capsule gastroscope is determined through gesture data, and further the camera coordinate is compensated to finish image stabilization, so that a first stomach image is obtained. In order to improve the image stabilizing effect, after the mechanical image stabilizing treatment is performed, the digital image stabilizing treatment can be performed on the first stomach image to obtain a second stomach image. The digital image stabilization process may include a motion estimation process, a motion filtering process, a motion compensation process, and the like, wherein the basic idea of the motion estimation process is to divide each frame of an image sequence into a plurality of macro blocks which are not overlapped with each other, consider that the displacement amounts of all pixels in the macro blocks are the same, and then find out a block which is most similar to a current block, i.e. a matching block, according to a certain matching criterion within a certain given specific search range from each macro block to a reference frame, and the relative displacement between the matching block and the current block is a motion vector. After the motion estimation obtains the estimation result, the motion estimation can be performed on the motion estimation result, and the motion filtering can be specifically mean filtering, motion filtering based on beta-spline, kalman filtering and the like. After motion filtering, reconstructing the first stomach image by using the estimation result to obtain a second image, wherein the process is motion compensation. And after the second image is obtained, carrying out three-dimensional display on the second image based on the azimuth information and the attitude information. Through the processing, the video stream formed by the second image has no shaking problem, so that stomach observation is facilitated.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a specific gastroscope system according to an embodiment of the present application. In this embodiment, the VR HMD (Head Mount Display, head mounted display) is a three-dimensional display device, where bluetooth is used to interact with bluetooth in a capsule (i.e. a gastroscope capsule), and transmit images and relative position information, i.e. information to be transmitted. The infrared camera on the VR HMD is used for capturing the infrared LED on the waist ring, and then captures the waist ring position. The wireless transceiver module is used for interacting with the wireless transceiver module on the waist ring and transmitting a control command, namely control information. The MCU (Micro Control Unit ) on the waist ring controls other parts or modules, and the electromagnetic coil on the waist ring acts on the magnet to control the movement of the capsule. And simultaneously, is used for providing azimuth information for a geomagnetic sensor (Magnetic) in the capsule. The capsule also comprises a camera and an LED lamp for obtaining stomach images. The IMU in the capsule is configured to obtain pose information, so as to form relative position information with the azimuth information, and then send the relative position information to the VR HMD. Referring to fig. 3, fig. 3 is a schematic distribution diagram of a specific electromagnetic device on a waist ring according to an embodiment of the present application. The big circle represents the waist ring, the three small circles indicated by the arrows are electromagnetic devices, and the three electromagnetic devices are uniformly distributed on the waist ring.
The following describes an image display method provided in the embodiments of the present application, and the image display method described below and the gastroscope system described above may be referred to correspondingly.
Referring to fig. 4, fig. 4 is a flowchart of an image display method provided in an embodiment of the present application, including:
s101: the capsule gastroscope acquires azimuth information based on electromagnetic equipment in the lumbar ring by using a geomagnetic sensor, and acquires attitude information by using an inertial measurement sensor.
S102: and acquiring a stomach image, and generating information to be transmitted by using the stomach image, the azimuth information and the gesture information.
S103: and sending the information to be transmitted to the three-dimensional display equipment so that the three-dimensional display equipment can perform three-dimensional display on the stomach image based on the azimuth information and the gesture information.
By applying the image display method provided by the embodiment of the application, the capsule gastroscope is adopted to replace the tube provided with the gastroscope, so that the tube can flexibly move, and the problem that the specific part cannot be carefully observed due to the limitation of the tube movement is avoided. The geomagnetic sensor in the capsule gastroscope can act with electromagnetic equipment in the waist ring to obtain azimuth information, and the azimuth information can represent the specific position of the capsule gastroscope in the plane of the waist ring and can also represent the orientation of the capsule gastroscope. The inertial measurement sensor can acquire attitude information, and the attitude information can represent the attitude of the capsule gastroscope in the stomach. Meanwhile, a stomach image is acquired by using a lens, and information to be transmitted is generated by using the stomach image, the gesture information and the azimuth information. After the three-dimensional display device acquires the information to be transmitted, the gesture information and the azimuth information can be used for determining the gesture of the capsule gastroscope in the three-dimensional space, so that the stomach image can be displayed in three dimensions based on the gesture information and the azimuth information. The three-dimensional display has a space sense compared with a two-dimensional image, can flexibly move in cooperation with the capsule gastroscope, can acquire and display the image of any stomach part, allows a user to carefully observe any stomach part, and solves the problem that the detailed observation cannot be performed on a specific part in the related technology.
Optionally, the method further comprises:
the waist ring of the gastroscope system which is the same as the capsule gastroscope acquires control information, and electromagnetic equipment on the waist ring is regulated according to the control information; electromagnetic device acts on magnet in capsule gastroscope to adjust the spatial position of capsule gastroscope
Optionally, the method further comprises:
the three-dimensional display equipment which is positioned in the same gastroscope system as the capsule gastroscope acquires motion information;
the three-dimensional display device generates control information using the motion information and transmits the control information to the lumbar ring.
Optionally, the method further comprises:
the three-dimensional display equipment performs part recognition processing on the stomach image to obtain a recognition result;
and carrying out three-dimensional display on the stomach image and the recognition result based on the azimuth information and the gesture information.
Optionally, the method further comprises:
the three-dimensional display device performs mechanical image stabilization processing on the stomach image by using the gesture information to obtain a first stomach image;
performing digital image stabilization processing on the first stomach image to obtain a second stomach image;
and carrying out three-dimensional display on the second stomach image based on the azimuth information and the gesture information.
Optionally, the method further comprises:
the three-dimensional display equipment acquires an identification signal sent by an identification signal transmitter on the waist ring by using an identification signal receiver;
determining a first position of the electromagnetic equipment on the waist ring by using the identification signal, and setting a basic coordinate system by using the first position;
correspondingly, the three-dimensional display device performs three-dimensional display on the stomach image based on the azimuth information and the gesture information, and comprises:
and the three-dimensional display device performs three-dimensional display on the stomach image based on the azimuth information and the gesture information under the basic coordinate system.
The image display device provided in the embodiments of the present application is described below, and the image display device described below and the image display method described above may be referred to correspondingly.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an image display device according to an embodiment of the present application, including:
the acquisition module 210 is used for acquiring azimuth information based on electromagnetic equipment in the waist ring by using a geomagnetic sensor and acquiring attitude information by using an inertial measurement sensor by using a capsule gastroscope;
the generating module 220 is configured to acquire a stomach image, and generate information to be transmitted by using the stomach image, the azimuth information and the gesture information;
the display module 230 is configured to send the information to be transmitted to the three-dimensional display device, so that the three-dimensional display device performs three-dimensional display on the stomach image based on the azimuth information and the gesture information.
The following describes a nonvolatile storage medium provided in an embodiment of the present application, and the nonvolatile storage medium described below and the image display method described above may be referred to correspondingly.
The present application also provides a non-volatile storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the image display method described above.
The nonvolatile storage medium may include: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may implement the described functionality using different approaches for each particular application, but such implementation should not be considered to be beyond the scope of this application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it is further noted that, in this document, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms include, comprise, or any other variation is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
The principles and embodiments of the present application are described herein with specific examples, the above examples being provided only to assist in understanding the methods of the present application and their core ideas; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (8)
1. The utility model provides a gastroscope system, its characterized in that includes capsule gastroscope, three-dimensional display device and waist ring, capsule gastroscope includes camera lens, geomagnetic sensor, inertial measurement sensor, the waist ring includes a plurality of electromagnetic equipment, each electromagnetic equipment is located the different positions of waist ring, be provided with physical mark on the waist ring, wherein:
the capsule gastroscope acquires azimuth information based on the electromagnetic equipment by using the geomagnetic sensor, and acquires attitude information by using the inertial measurement sensor;
the capsule gastroscope acquires a stomach image by using the lens, and generates information to be transmitted by using the stomach image, the azimuth information and the attitude information;
transmitting the information to be transmitted to the three-dimensional display equipment;
the three-dimensional display device performs three-dimensional display on the stomach image based on the azimuth information and the gesture information;
the capsule gastroscope also comprises a magnet, wherein the waist ring acquires control information and adjusts the electromagnetic equipment according to the control information; the electromagnetic device acts on the magnet to adjust the spatial position of the capsule gastroscope; wherein:
the three-dimensional display device performs mechanical image stabilization processing on the stomach image by using the gesture information to obtain a first stomach image;
performing digital image stabilization processing on the first stomach image to obtain a second stomach image;
and carrying out three-dimensional display on the second stomach image based on the azimuth information and the gesture information.
2. The gastroscope system of claim 1, wherein the three-dimensional display device acquires motion information;
and generating the control information by utilizing the motion information, and sending the control information to the waist ring.
3. The gastroscope system of claim 1, wherein the three-dimensional display device performs a portion recognition process on the stomach image to obtain a recognition result;
and carrying out three-dimensional display on the stomach image and the recognition result based on the azimuth information and the gesture information.
4. The gastroscope system of claim 1, wherein the waist ring further comprises an identification signal transmitter corresponding to the electromagnetic device, the three-dimensional display device comprising an identification signal receiver corresponding to the identification signal transmitter;
the three-dimensional display device acquires the identification signal sent by the identification signal transmitter by using the identification signal receiver;
determining a first position of the electromagnetic equipment on the waist ring by using the identification signal, and setting a basic coordinate system by using the first position;
correspondingly, the three-dimensional display device performs three-dimensional display on the stomach image based on the azimuth information and the gesture information, and the three-dimensional display device comprises:
and the three-dimensional display device performs three-dimensional display on the stomach image based on the azimuth information and the gesture information under the basic coordinate system.
5. The gastroscope system of claim 1, wherein the capsule gastroscope further comprises an exposure light.
6. An image display method, comprising:
the capsule gastroscope acquires azimuth information based on electromagnetic equipment in the waist ring by using a geomagnetic sensor, and acquires attitude information by using an inertial measurement sensor; the waist ring comprises a plurality of electromagnetic devices, wherein each electromagnetic device is positioned at different positions of the waist ring, and a physical mark is arranged on the waist ring;
obtaining a stomach image, and generating information to be transmitted by using the stomach image, the azimuth information and the attitude information;
transmitting the information to be transmitted to a three-dimensional display device so that the three-dimensional display device can perform three-dimensional display on the stomach image based on the azimuth information and the gesture information;
the capsule gastroscope also comprises a magnet, wherein the waist ring acquires control information and adjusts the electromagnetic equipment according to the control information; the electromagnetic device acts on the magnet to adjust the spatial position of the capsule gastroscope; wherein:
the three-dimensional display device performs mechanical image stabilization processing on the stomach image by using the gesture information to obtain a first stomach image;
performing digital image stabilization processing on the first stomach image to obtain a second stomach image;
and carrying out three-dimensional display on the second stomach image based on the azimuth information and the gesture information.
7. An image display device, comprising:
the acquisition module is used for acquiring azimuth information based on electromagnetic equipment in the waist ring by using a geomagnetic sensor and acquiring attitude information by using an inertial measurement sensor; the waist ring comprises a plurality of electromagnetic devices, wherein each electromagnetic device is positioned at different positions of the waist ring, and a physical mark is arranged on the waist ring;
the generation module is used for acquiring a stomach image and generating information to be transmitted by utilizing the stomach image, the azimuth information and the gesture information;
the display module is used for sending the information to be transmitted to a three-dimensional display device so that the three-dimensional display device can display the stomach image in a three-dimensional mode based on the azimuth information and the gesture information;
the capsule gastroscope also comprises a magnet, wherein the waist ring acquires control information and adjusts the electromagnetic equipment according to the control information; the electromagnetic device acts on the magnet to adjust the spatial position of the capsule gastroscope; wherein:
the three-dimensional display device performs mechanical image stabilization processing on the stomach image by using the gesture information to obtain a first stomach image;
performing digital image stabilization processing on the first stomach image to obtain a second stomach image;
and carrying out three-dimensional display on the second stomach image based on the azimuth information and the gesture information.
8. A non-volatile storage medium for storing a computer program, wherein the computer program when executed by a processor implements the image presentation method of claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110441044.8A CN113143185B (en) | 2021-04-23 | 2021-04-23 | Gastroscope system, image display method and device and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110441044.8A CN113143185B (en) | 2021-04-23 | 2021-04-23 | Gastroscope system, image display method and device and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113143185A CN113143185A (en) | 2021-07-23 |
| CN113143185B true CN113143185B (en) | 2023-05-02 |
Family
ID=76869902
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110441044.8A Active CN113143185B (en) | 2021-04-23 | 2021-04-23 | Gastroscope system, image display method and device and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113143185B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114637871A (en) * | 2022-03-23 | 2022-06-17 | 安翰科技(武汉)股份有限公司 | Method and device for establishing digestive tract database and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111374630A (en) * | 2018-12-28 | 2020-07-07 | 王子华 | Magnetic superfine swallowable endoscope |
| CN111543924A (en) * | 2020-05-28 | 2020-08-18 | 江苏数字鹰科技股份有限公司 | Ultra-miniature peep-in robot |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3974769B2 (en) * | 2001-11-06 | 2007-09-12 | オリンパス株式会社 | Capsule medical device |
| TWI285542B (en) * | 2005-05-06 | 2007-08-21 | Neovictory Technology Company | Active type capsule endoscope capable of being operated three-dimensionally |
| JP5652157B2 (en) * | 2010-11-25 | 2015-01-14 | ソニー株式会社 | Imaging apparatus, image processing method, and computer program |
| TWI463965B (en) * | 2012-03-05 | 2014-12-11 | Gi Shih Lien | Magnetic maneuvering system of capsule endoscope |
| CN103142205A (en) * | 2013-02-16 | 2013-06-12 | 深圳市资福技术有限公司 | Image acquisition method of capsule gastroscope |
| TWI540995B (en) * | 2014-02-19 | 2016-07-11 | 群曜醫電股份有限公司 | Alimentary tract survey system and control method for the same |
| CN107072475A (en) * | 2014-12-08 | 2017-08-18 | 奥林巴斯株式会社 | Capsule-type endoscope system |
| CN105361841B (en) * | 2015-11-30 | 2018-10-16 | 青岛大学附属医院 | Wireless capsule endoscope system for gastrointestinal tract diagnosis and treatment |
| CN205758500U (en) * | 2016-06-01 | 2016-12-07 | 安翰光电技术（武汉）有限公司 | Capsule endoscope system |
-
2021
- 2021-04-23 CN CN202110441044.8A patent/CN113143185B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111374630A (en) * | 2018-12-28 | 2020-07-07 | 王子华 | Magnetic superfine swallowable endoscope |
| CN111543924A (en) * | 2020-05-28 | 2020-08-18 | 江苏数字鹰科技股份有限公司 | Ultra-miniature peep-in robot |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113143185A (en) | 2021-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11796309B2 (en) | Information processing apparatus, information processing method, and recording medium | |
| US9636188B2 (en) | System and method for 3-D tracking of surgical instrument in relation to patient body | |
| US10389937B2 (en) | Information processing device, information processing method, and program | |
| US20090207246A1 (en) | Interactive image acquisition device | |
| CN109069103B (en) | Ultrasound imaging probe positioning | |
| US8167789B2 (en) | Image processing system and method for body-insertable apparatus | |
| CN109288591A (en) | Surgical robot system | |
| US20160037998A1 (en) | Endoscopic Operating System and Endoscopic Operation Program | |
| CN107238396A (en) | The pose recovery of ultrasonic transducer | |
| JP2013258614A (en) | Image generation device and image generation method | |
| WO2015046152A1 (en) | Endoscopy system | |
| CN109166181A (en) | A kind of mixing motion capture system based on deep learning | |
| CN113143185B (en) | Gastroscope system, image display method and device and readable storage medium | |
| CN109844600B (en) | Information processing apparatus, information processing method, and program | |
| JP7247371B2 (en) | Image processing device, system, image processing method, and image processing program | |
| JP2015043920A (en) | Imaging support device, imaging system, and program | |
| JP6929674B2 (en) | Environmental image display system and environmental image display method | |
| JP2020039052A (en) | Image processing system and HMD | |
| JP6890524B2 (en) | Attitude control system | |
| KR101876985B1 (en) | Medical device using augmented reality and short distance location tracking | |
| KR101734516B1 (en) | Flexible large intestine endoscope for shape recognition inside large intestine using inertia sensor, method for shape recognition inside large intestine thereby | |
| KR20220056702A (en) | Augmented reality apparatus and method for operating augmented reality apparatus | |
| JP2020177066A (en) | Observation state display system, observation state display method and program | |
| WO2016079471A1 (en) | System and method for position tracking in a head mounted display |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20221123 Address after: No. 500 Songling Road, Laoshan District, Qingdao City, Shandong Province, 266100 Applicant after: GOERTEK TECHNOLOGY Co.,Ltd. Address before: 261031 workshop 1, phase III, Geer Photoelectric Industrial Park, 3999 Huixian Road, Yongchun community, Qingchi street, Weifang High tech Zone, Weifang City, Shandong Province Applicant before: GoerTek Optical Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |