CN111421537A - Remote control system - Google Patents
Remote control system Download PDFInfo
- Publication number
- CN111421537A CN111421537A CN202010196799.1A CN202010196799A CN111421537A CN 111421537 A CN111421537 A CN 111421537A CN 202010196799 A CN202010196799 A CN 202010196799A CN 111421537 A CN111421537 A CN 111421537A
- Authority
- CN
- China
- Prior art keywords
- remote
- local
- positioning device
- information
- equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1679—Programme controls characterised by the tasks executed
- B25J9/1689—Teleoperation
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- H04L67/125—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks involving control of end-device applications over a network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q9/00—Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q2209/00—Arrangements in telecontrol or telemetry systems
- H04Q2209/40—Arrangements in telecontrol or telemetry systems using a wireless architecture
Abstract
The present application relates to a remote control system comprising: the local positioning equipment is used for acquiring multimedia information in the operating room and sending the multimedia information to the main remote positioning equipment; the main remote positioning equipment is used for receiving the multimedia information and sending a remote operation instruction to the local positioning equipment according to the multimedia information; each auxiliary remote positioning device is used for connecting and watching the operation in the operation room through the 5G communication technology between the auxiliary remote positioning device and the local positioning device; the operation equipment is used for executing the remote operation instruction received by the local positioning equipment and feeding back the operation information acquired in the operation process to the main remote positioning equipment through the local positioning equipment; the operation information includes video information, audio information and local operation information during operation. The system can improve the positioning and adjusting accuracy of the operation positioning equipment of a local hospital in a remote operation on the operation point.
Description
Technical Field
The application relates to the technical field of medical treatment, in particular to a remote control system.
Background
With the development of communication technology, telesurgery has become an achievable surgical approach, and the telesurgery can solve the problem of uneven medical level to a certain extent, so that patients and doctors are no longer long-distance running for one surgery.
Among the conventional art, remote operation mainly makes the operation plan and sends the operation positioning device to local hospital through remote expert, and the audio and video signal that the operation positioning device of local hospital sent is received, and remote expert watches the operation process of the operation positioning device of local hospital in long-range real time to send speech information to the operation positioning device of local hospital, the operation positioning device of supplementary local hospital carries out the positioning adjustment to the operation point, thereby accomplishes the operation.
However, the traditional method has the problem of inaccurate positioning adjustment.
Disclosure of Invention
In view of the above, there is a need to provide a remote control system capable of improving the accuracy of positioning and adjusting the surgical site by the surgical positioning device of the local hospital in the remote surgery.
A remote control system, the system comprising: the system comprises a local positioning device, a main remote positioning device, a plurality of auxiliary remote positioning devices and an operation device; the local positioning equipment is connected with the main remote positioning equipment and each auxiliary remote positioning equipment by adopting a 5G communication technology; the main remote positioning equipment is connected with each auxiliary remote positioning equipment by adopting the 5G communication technology; the local positioning equipment is in communication connection with the operating equipment;
the local positioning equipment is used for acquiring multimedia information in an operating room and sending the multimedia information to the main remote positioning equipment; the multimedia information comprises environmental information in an operating room and local operation information in the positioning operation process;
the main remote positioning equipment is used for receiving the multimedia information and sending a remote operation instruction to the local positioning equipment according to the multimedia information;
each auxiliary remote positioning device is used for watching the operation in the operation room through the 5G communication technology connection with the local positioning device;
the operating device is used for executing the remote operating instruction received by the local positioning device and feeding back the operating information acquired in the operating process to the main remote positioning device through the local positioning device; the operation information comprises video information, audio information and the local operation information in the operation process.
In one embodiment, the operating device is integrated with a first camera and a microphone.
In one embodiment, the master remote location device includes a first rights management module, a first communication interlock module, a mode selection module, and a control module;
the first authority management module is used for authenticating the login authority of a user and confirming the operation authority of the user when the user sends the remote operation instruction to the local positioning equipment;
the first communication interlocking module is used for judging whether the communication bandwidth of the current main remote positioning equipment and the current local positioning equipment meets the preset bandwidth requirement or not; if not, sending prompt information to the user and forbidding the user to carry out remote mechanical arm control operation;
the mode selection module is used for enabling the user to select an operation mode of the local positioning equipment; the modes of operation include a full mode of operation and an access only mode;
the control module is used for controlling the user to send the local positioning equipment operation instruction; the operation instruction is used for instructing the operation equipment to control the mechanical arm of the operation equipment.
In one embodiment, the control module controls the scram distance of the mechanical arm to be smaller than a preset distance threshold when the mechanical arm is controlled to scram.
In one embodiment, the local positioning device comprises a second authority management module, a second communication interlocking module, a mode confirmation module, a resolution confirmation module and an operation module;
the second authority management module is used for managing the access authority of the main remote positioning equipment accessed to the local positioning equipment so as to ensure that the main remote positioning equipment with authorized authority is accessed;
the second communication interlocking module is used for judging whether the communication bandwidth of the current local positioning equipment and the main remote positioning equipment meets the preset bandwidth requirement or not; if not, sending prompt information to a display interface of the local positioning equipment;
the mode confirmation module is used for confirming the operation mode initiated by the main remote positioning equipment; the modes of operation include a full mode of operation and an access only mode;
the analysis confirmation module is used for analyzing and confirming the operation instruction sent by the main remote positioning equipment;
the operation module is used for sending an emergency stop instruction to the operation equipment so as to control the mechanical arm of the operation equipment to stop moving.
In one embodiment, the multimedia information includes video information and audio information, and the local positioning device further includes a processing module configured to perform synchronization processing on the video information and the audio information to obtain synchronized video information and synchronized audio information.
In one embodiment, the local positioning device further comprises: and the acquisition module is used for acquiring the electrophysiological signals of the operated object.
In one embodiment, the system further comprises a second camera; the second camera is in communication connection with the local positioning equipment; the second camera is used for acquiring the environmental information in the operating room and sending the environmental information in the operating room to the local positioning equipment.
In one embodiment, the second camera is a camera having a video encoding and decoding function based on h.264/h.265 encoding and decoding rules.
In one embodiment, the master remote positioning device further comprises an adjustment module for sending a zoom operation to the local positioning device.
The remote control system comprises a local positioning device, a main remote positioning device, a plurality of auxiliary remote positioning devices and an operating device, wherein the local positioning device can collect multimedia information in an operating room, and sends the multimedia information to the main remote positioning device, so that the main remote positioning device can send a remote operation instruction to the local positioning device according to the multimedia information sent by the local positioning device, so that the operation equipment executes the remote operation instruction received by the local positioning equipment, the operation equipment feeds back the operation information collected in the operation process to the main remote positioning equipment through the local positioning equipment, therefore, the main remote positioning equipment can send a remote operation instruction to the local positioning equipment in time according to the operation information collected in the operation process, control the operation equipment to complete the remote operation instruction received by the local positioning equipment, and accurately realize positioning; in addition, a plurality of auxiliary remote positioning devices included in the system can watch the operation in the operating room in real time, the plurality of auxiliary remote positioning devices can perform information interaction with the main remote positioning device in time, and the auxiliary main remote positioning device formulates a remote operation instruction, so that the accuracy of the remote operation instruction sent by the main remote positioning device to the local peripheral device is improved.
Drawings
FIG. 1 is a schematic diagram of a remote control system according to an embodiment;
FIG. 1a provides a remote control system interworking workflow, according to one embodiment;
FIG. 2 is a schematic diagram of a remote control system according to another embodiment;
FIG. 2a is a schematic diagram of an operation interface of a master remote positioning device and a local positioning device in remote control operation according to an embodiment;
FIG. 3 is a schematic diagram of a remote control system according to another embodiment;
FIG. 4 is a diagram illustrating an apparatus deployment for remote control operations, according to an embodiment;
FIG. 5 is a diagram of data flow interactions of a local positioning device and a master remote positioning device, provided by one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In the traditional technology, the remote operation is mainly to make an operation plan by a remote expert and send the operation plan to operation positioning equipment in a local hospital, and receives the audio and video signals sent by the operation positioning equipment of the local hospital, the remote expert remotely watches the operation process of the operation positioning equipment of the local hospital in real time, and sends voice information to the operation positioning equipment of the local hospital to assist the operation positioning equipment of the local hospital to position and adjust the operation point, in some complex application scenarios or difficult cases, however, the path planned by the surgical plan does not meet the actual requirement, and the positioning site must be adjusted during the operation, however, doctors in local hospitals cannot independently complete positioning adjustment due to insufficient experience and other reasons, so that the remote operation method in the traditional technology cannot accurately position an operation point, and the problem of inaccurate positioning adjustment exists. Therefore, the embodiment of the present application provides a remote control system, which aims to solve the above technical problems of the conventional technology.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 1 is a schematic diagram of a remote control system according to an embodiment. FIG. 1a provides a remote control system interworking workflow, according to one embodiment. As shown in fig. 1, the remote control system includes: a local positioning device 100, a primary remote positioning device 200, a plurality of secondary remote positioning devices 300 and an operating device 400; the local positioning device 100 is connected with the main remote positioning device 200 and each auxiliary remote positioning device 300 by adopting a 5G communication technology; the main remote positioning device 200 is connected with each auxiliary remote positioning device 300 by adopting a 5G communication technology; the local positioning device 100 is in communication connection with the operating device 400; the local positioning device 100 is used for collecting multimedia information in the operating room and sending the multimedia information to the main remote positioning device 200; the multimedia information comprises environmental information in an operating room and local operation information in the positioning operation process; a main remote positioning device 200 for receiving the multimedia information and sending a remote operation instruction to the local positioning device 100 according to the multimedia information; each auxiliary remote positioning device 300 is used for connecting and watching the operation in the operation room through the 5G communication technology with the local positioning device; the operation device 400 is configured to execute a remote operation instruction received by the local positioning device 100, and feed back operation information acquired in an operation process to the master remote positioning device 200 through the local positioning device 100; the operation information includes video information, audio information and local operation information during operation.
Specifically, referring to fig. 1 and fig. 1a, the local positioning device 100 is connected to the main remote positioning device 200 by using a 5G communication technology, and the local positioning device 100 is configured to collect multimedia information in an operating room and send the multimedia information to the main remote positioning device 200; the multimedia information sent to the master remote positioning device 200 includes environmental information in the operating room and local operation information during the positioning operation. Alternatively, the local positioning device may collect multimedia information in the operating room through a camera and a microphone installed in the operating room.
The main remote positioning device 200 is connected to the local positioning device 100 by using a 5G communication technology, the main remote positioning device 200 is connected to each of the auxiliary remote positioning devices 300 by using a 5G communication technology, and the main remote positioning device 200 is configured to receive multimedia information sent by the local positioning device 100 and send a remote operation instruction to the local positioning device 100 according to the received multimedia information. It can be understood that the protocol delay of the 5G communication technology transmission between the main remote positioning device 200 and the local positioning device 100 can be controlled within 10ms, which can meet the delay requirement of remote operation (the communication delay parameter of the 4G technology and the delay parameter of the wired private network can generally only meet the requirement of remote viewing); in addition, the 5G communication technology is compatible with the advantages of low delay and high reliability of network transmission, and can ensure the continuity and stability of the remote operation process.
Each auxiliary remote positioning device 300 is connected to the main remote positioning device 200 by a 5G communication technology, each auxiliary remote positioning device 300 is connected to the local positioning device 100 by a 5G communication technology, and each auxiliary remote positioning device 300 is used for viewing the operation in the operating room by the 5G communication technology connection with the local positioning device 100. Optionally, each auxiliary remote positioning device 300 may perform information interaction with the main remote positioning device 200 through a 5G communication technology during an operation process in the viewing operation room, and send interaction information to the main remote positioning device 200 in real time.
The operation device 400 is communicatively connected to the local positioning device 100, and is configured to execute a remote operation instruction received by the local positioning device, and feed back operation information collected during an operation process to the master remote positioning device 200 through the local positioning device 100. The operation information collected by the operation device 400 includes video information, audio information, and local operation information during the operation. Alternatively, the operation device 400 may collect video information, audio information, and local operation information during the operation process through a camera and a microphone integrated with the operation device.
Table 1 shows an information data flow of the local positioning device provided in this embodiment in the remote positioning process, as shown in table 1, the multimedia information in the operating room acquired by the local positioning device 100 includes environment information in the operating room and local operation information in the positioning operation process, and the environment information in the operating room includes a plurality of panoramic view fields in the operating room, display view fields of the electrophysiological device, field of the system camera device, and field of other active cameras; the local operation information in the positioning operation process comprises an operation part local video and a preoperative image view field of the patient. Wherein, the electrophysiological equipment display field of view is used for displaying the electrophysiological real-time monitoring result of the patient during the positioning process.
TABLE 1
In this embodiment, the remote control system comprises a local positioning device, a main remote positioning device, a plurality of auxiliary remote positioning devices and an operating device, the local positioning device being capable of collecting multimedia information in the operating room, and sends the multimedia information to the main remote positioning device, so that the main remote positioning device can send a remote operation instruction to the local positioning device according to the multimedia information sent by the local positioning device, so that the operation equipment executes the remote operation instruction received by the local positioning equipment, the operation equipment feeds back the operation information collected in the operation process to the main remote positioning equipment through the local positioning equipment, therefore, the main remote positioning equipment can send a remote operation instruction to the local positioning equipment in time according to the operation information collected in the operation process, control the operation equipment to complete the remote operation instruction received by the local positioning equipment, and accurately realize positioning; in addition, a plurality of auxiliary remote positioning devices included in the system can watch the operation in the operating room in real time, the plurality of auxiliary remote positioning devices can perform information interaction with the main remote positioning device in time, and the auxiliary main remote positioning device formulates a remote operation instruction, so that the accuracy of the remote operation instruction sent by the main remote positioning device to the local peripheral device is improved.
Fig. 2 is a schematic diagram of a remote control system according to another embodiment. On the basis of the above embodiment, as an alternative implementation, as shown in fig. 2, the operation device 400 is integrated with a first camera 401 and a microphone 402. Optionally, the first camera may include a panoramic camera and a local operation field camera, the panoramic camera is used for collecting the operation scenes of each operator in the operating room in the remote positioning process in an omnidirectional manner, and the local operation field camera is used for collecting the high-definition images of the operating equipment in the local operation process in the remote positioning process. Alternatively, the microphone may be a microphone array. In this embodiment, the first camera and the microphone integrated with the operation device can acquire the operation scenes of each operator in the operation room and the high-definition image and audio information of the operation device in the local operation process in real time in the remote positioning process, so that the timeliness of the operation device for feeding back the operation information acquired in the operation process to the main remote positioning device through the local positioning device is improved, and the main remote positioning device can make a remote operation instruction in time according to the received operation information.
Fig. 2a is a schematic view of an operation interface of a master remote positioning device and a local positioning device in remote control operation according to an embodiment. With continued reference to fig. 2, the master remote positioning device 200 includes a first rights management module 201, a first communication interlock module 202, a mode selection module 203, and a control module 204; the first authority management module 201 is configured to authenticate login authority of a user, and confirm operation authority of the user when the user sends a remote operation instruction to the local positioning device 100; a first communication interlock module 202, configured to determine whether a communication bandwidth between the current master remote positioning device 200 and the local positioning device 100 meets a preset bandwidth requirement; if not, sending prompt information to the user and forbidding the user to carry out remote mechanical arm control operation; a mode selection module 203 for enabling a user to select an operation mode for the local positioning apparatus 100; the operation modes include a full operation mode and an access only mode; a control module 204, configured to control sending of an operation instruction to the local positioning apparatus 100 by a user; the operating instructions are used for instructing the operating equipment to control the mechanical arm of the operating equipment.
Specifically, the first authority management module 201 is configured to authenticate login authority of a user, and confirm operation authority of the user when the user sends a remote operation instruction to the local positioning device. Referring to fig. 2a, optionally, the first rights management module 201 has functions of rights entry and rights confirmation; the authority entry means that the main remote positioning equipment can only be logged in or used by a specified pre-entry user; the permission confirmation means that permission needs to be confirmed when a pre-recorded user connects with the local positioning equipment and sends a command to the local positioning equipment, and the permission confirmation can be realized by face recognition, voice recognition, fingerprint recognition and other online recognition modes and is used for improving the safety of system operation.
A first communication interlock module 202, configured to determine whether a communication bandwidth between the current master remote positioning device 200 and the local positioning device 100 meets a preset bandwidth requirement; and if the communication bandwidth of the current main remote positioning equipment 200 and the local positioning equipment 100 does not meet the preset bandwidth requirement, sending prompt information to the user and prohibiting the user from performing remote mechanical arm control operation. Optionally, if the communication bandwidth of the current master remote positioning device 200 and the local positioning device 100 does not meet the preset bandwidth requirement, the master remote positioning device 200 and the local positioning device 100 may still perform the operation of remote information interaction.
A mode selection module 203 for enabling a user to select a mode of operation for the local positioning apparatus 100. The manner in which the user of the master remote positioning device 200 operates the local positioning device 100 includes, among other things, a full mode of operation and an access only mode. It is understood that the full operation mode means that the master remote positioning device 200 can send a remote operation instruction to the local positioning device 100, causing the operation device 400 to execute the remote operation instruction received by the local positioning device 100; the access only mode means that the master remote positioning device 200 can only interact with the local positioning device 100 for information and cannot send remote operation instructions to the local positioning device 100.
A control module 204 for controlling the sending of operating instructions to the local positioning apparatus 100 by a user of the master remote positioning apparatus 200, the operating instructions being for instructing the operating apparatus to control its robotic arm. Optionally, when the mechanical arm of the control operation device is suddenly stopped, the control module 204 controls the sudden stop distance of the mechanical arm to be smaller than a preset distance threshold, for example, the preset distance threshold may be 0.1 mm.
In this embodiment, the master remote positioning device includes a first permission management module, a first communication interlock module, a mode selection module, and a control module, which are capable of authenticating login permission of a user and confirming operation permission of the user when the user sends a remote operation instruction to the local positioning device, so as to ensure that the logged-in user is a user with permission and the user sending the remote operation instruction is a user with operation permission, and in addition, it can be determined whether a communication bandwidth between the master remote positioning device and the local positioning device meets a preset bandwidth requirement, and the user is controlled to perform a remote robot arm control operation only when the preset bandwidth requirement is met, thereby further ensuring the safety of user operation.
Fig. 3 is a schematic diagram of a remote control system according to another embodiment. As shown in fig. 3, the local positioning device 100 includes a second rights management module 101, a second communication interlock module 102, a mode confirmation module 103, a resolution confirmation module 104, and an operation module 105; a second right management module 101, configured to manage access rights of the master remote positioning device 200 accessing the local positioning device 100, so as to ensure that the master remote positioning device 200 having authorized rights accesses; the second communication interlocking module 102 is configured to determine whether a communication bandwidth between the current local positioning device 100 and the master remote positioning device 200 meets a preset bandwidth requirement; if not, sending prompt information to a display interface of the local positioning equipment 100; a mode confirmation module 103 for confirming the mode of operation initiated by the master remote positioning device 200; the operation modes include a full operation mode and an access only mode; an analysis confirmation module 104, configured to perform analysis confirmation on the operation instruction sent by the master remote positioning device 200; the operation module 105 is configured to send an emergency stop command to the operation device 400 to control the robot arm of the operation device 400 to stop moving.
In particular, the second rights management module 101 is configured to manage access rights of the master remote positioning device 200 accessing the local positioning device 101, so as to ensure that the master remote positioning device 200 having authorized rights accesses. With continued reference to fig. 2a, optionally, the second rights management module 101 may verify the ID of the accessing master remote positioning device 200, thereby ensuring that the master remote positioning device 200 accessing the local positioning device 101 has the authorization rights.
The second communication interlocking module 102 is configured to determine whether a communication bandwidth between the current local positioning device 100 and the master remote positioning device 200 meets a preset bandwidth requirement; if the communication bandwidth between the current local positioning device and the main remote positioning device does not meet the preset bandwidth requirement, the second communication interlock module 102 sends a prompt message to the display interface of the local positioning device 100. Optionally, the prompt information sent by the second communication interlock module 102 to the display interface of the local positioning device 100 may be text information or a prompt pop frame.
A mode confirmation module 103 for confirming the mode of operation initiated by the master remote positioning device 200; the modes of operation include a full mode of operation and an access only mode. It is understood that the full operation mode refers to the local positioning device 100 allowing the master remote positioning device 200 to transmit a remote operation instruction to the local positioning device 100, causing the operation device 400 to execute the remote operation instruction received by the local positioning device 100; the access only mode means that the local positioning apparatus 100 allows only the master remote positioning apparatus 200 and the local positioning apparatus 100 to perform information interaction, and cannot transmit a remote operation instruction to the local positioning apparatus 100.
And the analysis confirmation module 104 is configured to perform analysis confirmation on the operation instruction sent by the master remote positioning device 200. Optionally, the parsing confirmation module 104 may confirm whether the operation instruction sent by the master remote positioning device is safe, so as to ensure the safety of the transferred operation instruction.
The operation module 105 is configured to send an emergency stop command to the operation device 400 to control the robot arm of the operation device 400 to stop moving. It is understood that the emergency stop instruction is an instruction issued by the operation module 105 to the operation device 400 to control the operation device 400 to stop emergently in a case of a relatively emergency.
In this embodiment, the local positioning device includes a second permission management module, a second communication interlock module, a mode confirmation module, an analysis confirmation module, and an operation module, which can manage the access permission of the master remote positioning device accessing the local positioning device, and ensure the access of the master remote positioning device having the authorization permission; in addition, whether the communication bandwidth of the current local positioning equipment and the main remote positioning equipment meets the preset bandwidth requirement can be judged, if not, prompt information is sent to a display interface of the local positioning equipment, and the operation mode initiated by the main remote positioning equipment is confirmed, so that the main remote positioning equipment can accurately operate the local positioning equipment; furthermore, the operation module of the local positioning device can send an emergency stop instruction to the operation device, so that the mechanical arm of the operation device can be timely controlled to stop moving under the emergency condition, and the safety in the operation process is ensured.
In the scenario described above where the local positioning device sends multimedia information to the master remote positioning device, the sent multimedia information includes video information and audio information. Based on the foregoing embodiment, as an optional implementation manner, please continue to refer to fig. 3, the local positioning apparatus 100 further includes a processing module 106, configured to perform synchronization processing on the video information and the audio information, so as to obtain synchronized video information and synchronized audio information.
Specifically, the local positioning apparatus 100 further includes a processing module 106, configured to perform synchronization processing on the video information and the audio information, so as to obtain synchronized video information and synchronized audio information. Optionally, the processing module 106 may process the video frame of the video information according to the audio information to obtain the synchronized video information and audio information. Optionally, the processing module 106 may further implement synchronization between the video information and the audio information by encoding and decoding the acquired video information, and may compensate the network delay test in the synchronization parameter in advance, so as to finally implement "absolute synchronization", that is, zero delay, between the video information and the audio information. In this embodiment, the processing module 106 performs synchronization processing on the video information and the audio information to obtain synchronized video information and synchronized audio information, and sends the synchronized video information and synchronized audio information to the master remote positioning device, so that the master remote positioning device can accurately obtain multimedia information in the operating room, and the accuracy of the multimedia information received by the master remote positioning device is improved.
On the basis of the above embodiment, as an optional implementation manner, please refer to fig. 3, the local positioning apparatus 100 further includes an acquisition module 107 for acquiring the electrophysiological signals of the operated object.
In particular, the local positioning device 100 comprises an acquisition module 107 for acquiring electrophysiological signals of the object to be operated. In this embodiment, the acquisition module is configured to acquire an electrophysiological signal of the operated object, and may monitor the physiological signal of the operated object in real time, so as to improve the acquisition efficiency of the physiological signal of the operated object.
With continued reference to fig. 3, on the basis of the foregoing embodiment, as an optional implementation manner, the remote control system further includes a second camera 500; the second camera 500 is in communication connection with the local positioning device 100; the second camera 500 is configured to acquire environmental information in the operating room and send the environmental information in the operating room to the local positioning apparatus 100.
Specifically, the remote control system further includes a second camera 500; the second camera 500 is in communication connection with the local positioning device 100; the second camera 500 is configured to acquire environmental information in the operating room and send the environmental information in the operating room to the local positioning apparatus 100. Optionally, the second camera 500 is a camera having a video encoding and decoding function based on the h.264/h.265 encoding and decoding rule, and it should be noted that, for a camera having a video encoding and decoding function based on the h.264/h.265 encoding and decoding rule, the delay of encoding and decoding an image with a resolution of 4K may be controlled to be about 175ms, and the delay of encoding and decoding a video may be implemented by the second camera. Optionally, the second camera may be a plurality of panoramic cameras respectively installed in the operating room, or may be a panoramic camera array. Optionally, the master remote positioning device 200 further comprises an adjustment module 205 for sending a zoom operation to the local positioning device 100. Optionally, the zoom operation sent by the adjusting module 205 to the local positioning apparatus 100 may be used to adjust the focal length of the second camera 500, which can meet the requirement of the main remote positioning apparatus 200 for focusing the second camera.
In this embodiment, the second camera included in the remote control system can send the acquired environmental information in the operating room to the local positioning device through the communication connection with the local positioning device, so that the local positioning device can send the acquired environmental information in the operating room to the main remote positioning device in time, thereby improving the efficiency of sending the environmental information in the operating room to the main remote positioning device by the local positioning device.
FIG. 4 is a diagram illustrating an apparatus deployment for remote control operation, according to an embodiment. FIG. 5 is a diagram of data flow interactions of a local positioning device and a master remote positioning device, provided by one embodiment. Referring to fig. 4 and 5, for the local positioning device 100, Console is a host of the local positioning device 100 and software on the host, and is a main interface of an interactive system between the local positioning device 100 and the main remote positioning device 200, and Console obtains audio and video information of the main remote positioning device 200 through an ethernet switch, decodes the audio and video information, displays the decoded audio and video information on a large screen of an operating room, and provides a configuration function for the audio and video devices of the local positioning device 100. Meanwhile, the information of the main control display of the local positioning device 100 can be transmitted to the Console host of the main remote positioning device 200 through the switch, the user of the main remote positioning device 200 can watch the flow of the remote operation instruction executed by the operating device of the local positioning device 100 in real time, and the panoramic camera and the local operation information local operation field camera for acquiring the environmental information in the operating room can be accessed by the Console of the main remote positioning device 200 through the local ethernet switch in real time and can also be accessed by the Console of the local positioning device 100 in real time. The audio device of fig. 4 includes a microphone for voice pickup and a speaker for real-time voice playback, both external microphone and speaker capable of supporting wireless and wired connections. The external audio device supports analog audio input and output, i.e. interconnection with the 3.5 mm audio input and output interface provided by the system, and also supports digital audio input and output, i.e. interconnection with the USB interface provided by the system, and fig. 4 is an example of a wireless external microphone. Wireless microphone access systems require a 3.5 millimeter audio input interface or USB interface to connect the wireless receiver to the system. The pickup end of the wireless microphone is connected with the wireless transmitter in a wired mode, the transmitter is generally worn at the waist of a user, the external speaker shown in fig. 4 is in a wired mode, and when the speaker is used, the general position of the speaker does not need to be changed along with the position change of the user, namely the external audio input and output configuration scheme of fig. 4 not only considers the application convenience, but also considers the reliability of the quality of audio input and output signals. In addition, the local positioning device is also compatible with integrated audio input and output devices, namely, the microphone array and the loudspeaker are integrated into the operation device, in this case, a user does not need to wear any voice pickup device, and the convenience of operation is reflected. In order to ensure the compatibility of the scheme and the usability of the user when using the system, when the external audio device and the integrated device are used simultaneously, the system can only obtain the signal of one voice device, so that a channel switching function for the voice input and output device is added in the system in fig. 4, the external input and output audio device can be switched by the user, and the hardware implementation of the switching function is completed by a system interface board card (SIB) in the operation device. In addition, the audio input devices (i.e., the external microphone and the integrated microphone) on the local side in fig. 4 can also be connected to the audio input interface of the panoramic camera on the local side. In the remote positioning process, the electrophysiological equipment is required to be used for acquiring the electroencephalogram signals of the patient, so in the remote positioning scene, the signal transmission or mapping of the electrophysiological equipment to the display of the main remote positioning equipment 200 needs to be considered, so that the user of the main remote positioning equipment 200 can perform real-time diagnosis operation. When the electrophysiological signals are used in the operating room, a dedicated notebook computer is usually used to display the electrophysiological signals acquired by the electrophysiological equipment in real time. Fig. 4 provides a scheme compatible with three remote mapping functions related to electrophysiological signals. The first scheme is as follows: the screen of the notebook computer used by the local positioning device 100 for electrophysiological signal display is sent to the main remote positioning device 200 via a video encoder supporting HDMI or VGA interface, which supports HDMI interface input of multiple screens and can send encoded data to the network via the network interface for access by the main remote positioning device. The second scheme is as follows: the notebook computer screen used by the local positioning equipment for displaying the electrophysiological signals is placed in a high-definition panoramic camera view field, and a doctor of the main remote positioning equipment can enlarge or reduce the screen by adjusting the optical focal length of the panoramic camera, so as to acquire the electrophysiological signals and guide the operation of the local positioning equipment in real time. In the third scheme: the signals acquired by the electrophysiological equipment are directly transmitted to the operating equipment 400, the electrophysiological signals are processed by an electrophysiological equipment signal processing board card in the system and are displayed on the local positioning equipment 100 and the main remote positioning equipment 200, and local amplification or reduction operations for the electrophysiological signals can be synchronously operated on the local positioning equipment 100 and the main remote positioning equipment 200. The local positioning device 100 provides only one external network interface that can be connected to the operating room's network switch through which it can communicate with the master remote positioning device 200 over a private network, which can be either a wired or wireless network. The example of fig. 4 mainly considers real-time network transmission based on 5G, and can be realized by an external 5G router of a local positioning device and a remote positioning device. Compared with the traditional 4G network and the wired network, the 5G network has the advantages that the protocol delay of communication transmission can be controlled within 10ms, the delay requirement of remote operation can be met (the delay parameters of the 4G delay and the wired private network can only meet the remote diagnosis requirement generally), meanwhile, the 5G network is compatible with the advantages of low delay and high reliability of network transmission, and the continuity and stability of the remote operation process can be ensured. In addition, the device deployment diagram of the remote control operation in fig. 4 also considers the display design scheme of the compatible tablet, that is, in the network switch connected to the local positioning device 100 through the tablet, the video information of the network camera of the main remote positioning device 200 can be acquired, and at the same time, the video information can be used for voice pickup of the local positioning device 100 and voice playing of the main remote positioning device 200. The embodiment variant increases a configuration interface for a camera of the local positioning equipment, and can realize the isolation function of the remote control function module and the audio and video function module in Console software to a certain extent.
Referring to fig. 4 and 5, for the master remote positioning apparatus 200, the master remote positioning apparatus 200 also deploys a Console host, which is referred to as a remote Console host. The Console of the remote end will be installed with remote end software, which can display various video information of the local end (in general, the video information of multiple local ends can be expanded and displayed on the large screen of the remote end) on the remote end and display of the main interface of the control software of the local positioning device 100. The video of the master remote positioning device 200 is directly accessible by the Console host or tablet of the local positioning device 100 via a webcam with IP access. While the master remote location device 200Console provides initial interface parameter configuration functions for the camera. The audio input and output of the main remote positioning device 200 can be realized by an external gooseneck microphone and a loudspeaker, the gooseneck microphone and the loudspeaker are directly connected with an audio input and output interface of a Console host of the main remote positioning device, the audio input and output signals are encoded by the main remote positioning device Console and then output to a network, and the audio input and output signals are interacted with audio and video of a local end through the network. In addition, the network interface of the main remote positioning device 200 in fig. 4 also realizes local 5G communication through an external 5G route, and the audio and video devices (an external gooseneck microphone, an external speaker, an IP camera, etc.) of the main remote positioning device 200 can be replaced by a tablet or a notebook computer with a camera function, and only the tablet or the notebook computer needs to be connected to the network switch of the main remote positioning device. In addition, when performing remote control, it is necessary to consider the performance of voice and speech synchronization, audio and video synchronization, video transmission delay, audio transmission delay, etc., when a 5G network is used for transmission, the transmission delay of the protocol can be controlled within 10ms, and other delays mainly consist of audio and video codec delays, wherein the bidirectional video codec delay is realized by the camera hardware device itself, the camera with the video codec function based on the h.264/h.265 codec rule adopted in the scheme of fig. 4 can be controlled to about 175ms for the image codec delay of 4K resolution, but if the local positioning device 100 and the main remote positioning device 200 monitor the real-time video stream of the panoramic camera at the same time, the display delay experienced by the user only consists of the network transmission delay (the local positioning device can also realize synchronization by codec of the acquired video), and the network delay can be tested in advance and then compensated into the synchronization parameter, and finally, the absolute synchronization, namely zero delay, is realized). If the local positioning device does not display the real-time video stream of the panoramic camera, and only displays the video of the panoramic camera at the main remote positioning device, the video picture seen by the user of the main remote positioning device is actually a scene about 180ms ahead of the local end. In addition, the time of encoding and decoding in the audio transmission process can be controlled to be about 50ms, and the transmission delay of the audio data can be controlled to be about 60ms in consideration of the protocol transmission delay of the 5G network. In order to consider audio and video synchronization, active delay operation may be performed on the transmitted audio data in the Console software of the local positioning device 100 and the remote positioning device 200, that is, delay operation of about 120ms is performed in the Console software, and finally, the total transmission delay of the audio data is also controlled to be about 180ms, so that audio and video synchronization of the local positioning device 100 and the main remote positioning device 200 is realized.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A remote control system, the system comprising: the system comprises a local positioning device, a main remote positioning device, a plurality of auxiliary remote positioning devices and an operation device; the local positioning equipment is connected with the main remote positioning equipment and each auxiliary remote positioning equipment by adopting a 5G communication technology; the main remote positioning equipment is connected with each auxiliary remote positioning equipment by adopting the 5G communication technology; the local positioning equipment is in communication connection with the operating equipment;
the local positioning equipment is used for acquiring multimedia information in an operating room and sending the multimedia information to the main remote positioning equipment; the multimedia information comprises environmental information in an operating room and local operation information in the positioning operation process;
the main remote positioning equipment is used for receiving the multimedia information and sending a remote operation instruction to the local positioning equipment according to the multimedia information;
each auxiliary remote positioning device is used for watching the operation in the operation room through the 5G communication technology connection with the local positioning device;
the operating device is used for executing the remote operating instruction received by the local positioning device and feeding back the operating information acquired in the operating process to the main remote positioning device through the local positioning device; the operation information comprises video information, audio information and the local operation information in the operation process.
2. The system of claim 1, wherein the operating device is integrated with a first camera and a microphone.
3. The system of claim 1 or 2, wherein the master remote location device comprises a first rights management module, a first communication interlock module, a mode selection module, and a control module;
the first authority management module is used for authenticating the login authority of a user and confirming the operation authority of the user when the user sends the remote operation instruction to the local positioning equipment;
the first communication interlocking module is used for judging whether the communication bandwidth of the current main remote positioning equipment and the current local positioning equipment meets the preset bandwidth requirement or not; if not, sending prompt information to the user and forbidding the user to carry out remote mechanical arm control operation;
the mode selection module is used for enabling the user to select an operation mode of the local positioning equipment; the modes of operation include a full mode of operation and an access only mode;
the control module is used for controlling the user to send the local positioning equipment operation instruction; the operation instruction is used for instructing the operation equipment to control the mechanical arm of the operation equipment.
4. The system of claim 3, wherein the control module controls the robotic arm to have a hard stop distance less than a preset distance threshold when controlling the robotic arm to hard stop.
5. The system of claim 1, wherein the local positioning device comprises a second rights management module, a second communication interlock module, a mode confirmation module, a resolution confirmation module, and an operation module;
the second authority management module is used for managing the access authority of the main remote positioning equipment accessed to the local positioning equipment so as to ensure that the main remote positioning equipment with authorized authority is accessed;
the second communication interlocking module is used for judging whether the communication bandwidth of the current local positioning equipment and the main remote positioning equipment meets the preset bandwidth requirement or not; if not, sending prompt information to a display interface of the local positioning equipment;
the mode confirmation module is used for confirming the operation mode initiated by the main remote positioning equipment; the modes of operation include a full mode of operation and an access only mode;
the analysis confirmation module is used for analyzing and confirming the operation instruction sent by the main remote positioning equipment;
the operation module is used for sending an emergency stop instruction to the operation equipment so as to control the mechanical arm of the operation equipment to stop moving.
6. The system of claim 5, wherein the multimedia information comprises video information and audio information, and the local positioning device further comprises a processing module for synchronizing the video information and the audio information to obtain synchronized video information and synchronized audio information.
7. The system of claim 6, wherein the local positioning device further comprises: and the acquisition module is used for acquiring the electrophysiological signals of the operated object.
8. The system of claim 1, further comprising a second camera; the second camera is in communication connection with the local positioning equipment; the second camera is used for acquiring the environmental information in the operating room and sending the environmental information in the operating room to the local positioning equipment.
9. The system of claim 8, wherein the second camera is a camera having a video codec function based on h.264/h.265 codec rules.
10. The system of claim 9, wherein the master remote positioning device further comprises an adjustment module to send a zoom operation to the local positioning device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010196799.1A CN111421537A (en) | 2020-03-19 | 2020-03-19 | Remote control system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010196799.1A CN111421537A (en) | 2020-03-19 | 2020-03-19 | Remote control system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111421537A true CN111421537A (en) | 2020-07-17 |
Family
ID=71548239
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010196799.1A Pending CN111421537A (en) | 2020-03-19 | 2020-03-19 | Remote control system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111421537A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114770493A (en) * | 2022-01-24 | 2022-07-22 | 四川广目科技有限公司 | ROS network-based visual remote control system for service mechanical arm |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1426564A (en) * | 2000-12-28 | 2003-06-25 | 高桥淳 | Remote internet technical guidance/education distribution system using practitioner's vision, and guidance system using communication network |
| CN102089751A (en) * | 2008-07-11 | 2011-06-08 | 英塔茨科技公司 | Tele-presence robot system with multi-cast features |
| US9123155B2 (en) * | 2011-08-09 | 2015-09-01 | Covidien Lp | Apparatus and method for using augmented reality vision system in surgical procedures |
| CN106375421A (en) * | 2016-08-30 | 2017-02-01 | 上海交通大学 | Remote control-based robot-assisted intelligent maintenance system |
| CN106426169A (en) * | 2016-10-21 | 2017-02-22 | 中国科学院力学研究所 | Method and device for remotely controlling one operand through multiple operation terminals |
| CN107661111A (en) * | 2016-07-27 | 2018-02-06 | 上海联影医疗科技有限公司 | Medical apparatus system, the tele-control system of Medical Devices and method |
| WO2018089334A1 (en) * | 2016-11-10 | 2018-05-17 | Think Surgical, Inc. | Remote mentoring station |
| CN108537418A (en) * | 2018-03-22 | 2018-09-14 | 中国科学院力学研究所 | Method and system for assessing multioperation end remote control multioperation object |
| CN110215287A (en) * | 2019-03-29 | 2019-09-10 | 上海联影医疗科技有限公司 | A kind of master-slave mode medical treatment control system |
| CN110418215A (en) * | 2019-08-23 | 2019-11-05 | 山东临工工程机械有限公司 | A kind of excavator tele-control system and excavator |
-
2020
- 2020-03-19 CN CN202010196799.1A patent/CN111421537A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1426564A (en) * | 2000-12-28 | 2003-06-25 | 高桥淳 | Remote internet technical guidance/education distribution system using practitioner's vision, and guidance system using communication network |
| CN102089751A (en) * | 2008-07-11 | 2011-06-08 | 英塔茨科技公司 | Tele-presence robot system with multi-cast features |
| US9123155B2 (en) * | 2011-08-09 | 2015-09-01 | Covidien Lp | Apparatus and method for using augmented reality vision system in surgical procedures |
| CN107661111A (en) * | 2016-07-27 | 2018-02-06 | 上海联影医疗科技有限公司 | Medical apparatus system, the tele-control system of Medical Devices and method |
| CN106375421A (en) * | 2016-08-30 | 2017-02-01 | 上海交通大学 | Remote control-based robot-assisted intelligent maintenance system |
| CN106426169A (en) * | 2016-10-21 | 2017-02-22 | 中国科学院力学研究所 | Method and device for remotely controlling one operand through multiple operation terminals |
| WO2018089334A1 (en) * | 2016-11-10 | 2018-05-17 | Think Surgical, Inc. | Remote mentoring station |
| CN108537418A (en) * | 2018-03-22 | 2018-09-14 | 中国科学院力学研究所 | Method and system for assessing multioperation end remote control multioperation object |
| CN110215287A (en) * | 2019-03-29 | 2019-09-10 | 上海联影医疗科技有限公司 | A kind of master-slave mode medical treatment control system |
| CN110418215A (en) * | 2019-08-23 | 2019-11-05 | 山东临工工程机械有限公司 | A kind of excavator tele-control system and excavator |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114770493A (en) * | 2022-01-24 | 2022-07-22 | 四川广目科技有限公司 | ROS network-based visual remote control system for service mechanical arm |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11787060B2 (en) | Remote presence system mounted to operating room hardware | |
| US11157842B2 (en) | System, equipment and method for performing and documenting in real-time a remotely assisted professional procedure | |
| US8924234B2 (en) | Streaming video network system | |
| EP2973105B1 (en) | Surgical imaging system and method for processing surgical images | |
| CN107809609B (en) | Video monitoring conference system based on touch equipment | |
| US11910997B2 (en) | Apparatus, systems, and methods for intraoperative visualization | |
| US20170336635A1 (en) | Imaging data capture and video streaming system | |
| US20160331217A1 (en) | Endoscopic imaging system and method for adapting to remote stimulus | |
| WO2018193799A1 (en) | Information processing device, information processing method, and program | |
| US7890887B1 (en) | System and method for the operation of diagnostic medical equipment | |
| CN111421537A (en) | Remote control system | |
| WO2017187676A1 (en) | Control device, control method, program, and sound output system | |
| US8219859B2 (en) | Medical support control system | |
| US20190394454A1 (en) | Medical-image display control device, medical-image display device, medical-information processing system, and medical-image display control method | |
| US11671872B2 (en) | Communication system and transmission apparatus | |
| Schneider et al. | An evaluation of a surgical telepresence system for an intrahospital local area network | |
| KR102168948B1 (en) | Mobile video control system and and operation method thereof | |
| US7871403B2 (en) | Medical support control system | |
| WO2020054595A1 (en) | Surgery assistance system, display control device, and display control method | |
| CN113066588A (en) | Endoscopic surgery consultation platform and method | |
| CN113965682B (en) | Remote catheter room control system and remote catheter room control method | |
| WO2023145447A1 (en) | Information processing method, information processing system, and program | |
| EP4316410A1 (en) | Surgical operation room system, image recording method, program, and medical information processing system | |
| CN114974549A (en) | Ultrasonic diagnostic method, apparatus and storage medium | |
| BR122023011961B1 (en) | SYSTEM, EQUIPMENT AND PROCESS FOR PERFORMING AND DOCUMENTING A PROFESSIONAL DISTANCE ASSISTED PROCEDURE IN REAL TIME |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |