CN111660294B - Augmented reality control system of hydraulic heavy-duty mechanical arm - Google Patents

Abstract

The invention provides an augmented reality control system of a hydraulic heavy-duty mechanical arm, which comprises a field image information acquisition module, a real-time monitoring module and a real-time monitoring module, wherein the field image information acquisition module is used for acquiring image information of field operation and spatial position information of a target object; the manual interactive operation module is used for an operator to intervene in the operation process; the MySQL database module is used for storing user information, image information and path planning information; the cloud server module is used for generating a target model at the cloud end and calculating a path according to the operation task and the path planning information; the data visualization module is used for displaying a data analysis result on the operation interface; the anti-collision detection module is used for verifying whether the calculated path can be safely and reliably executed on the model data level; the data integration module is used for writing the verified reliable data into a data frame according to the protocol requirement; and the emergency stop module is used for emergently disconnecting the control on the mechanical arm in emergency. The invention can improve the intelligence, the operation friendliness and the reliability of the mechanical arm operation.

Description

Augmented reality control system of hydraulic heavy-duty mechanical arm

Technical Field

The invention relates to the technical field of mechanical arm control, in particular to an augmented reality control system for a hydraulic heavy-duty mechanical arm.

Background

Since the birth of computers and the Internet, the computers and the Internet affect our lives all the time, and bring more unexpected changes to the world. From home to large-scale industrial sites of common people, people can see the figure of various robots, and the life of people is changed and influenced. Nowadays, Artificial Intelligence (AI) is well developed, our lives have already stepped into the intelligent era, and industrial robots are also continuously developing. Since the 21 st century, with the continuous improvement of labor cost and the continuous progress of technology, the manufacturing industry is continuously transformed and upgraded in various countries, and the hot tide of replacing people with robots appears. In the future, robots will become new battlefields for industrial robots in the general manufacturing field.

The multiple versions of the Unity3D have strong adaptability, can be installed and used on various computer operating systems, including common Windows, macOS and Android, and can be used on various mainstream platforms such as Web, and the like, and can meet the requirements of different users in different scenes. Meanwhile, the product is also helped to show own advantages in the market, and a market gap is opened, so that the factors of the cost, the field and the equipment also have smaller influence on the operation process. Meanwhile, the operating system platform provided by the Unity3D can also utilize teaching and training processes, and can better meet user experience. From the safety perspective, related operations are performed through the operating system platform, so that an operator of the mechanical arm can be located in an operating site far away from the site environment, data transmission is performed through network communication modes such as an Ethernet cable, unnecessary danger problems in various complex site industrial environments are avoided, and the personal safety problem of the operator is further guaranteed. When the training device is put into the market for use, the learning cost and the training expense brought by training operators can be greatly reduced by a simple, understandable and intuitive operation mode.

The mechanical arm put into the market at the present stage has insufficient autonomous ability and certain defects, the human-computer interaction ability needs to be improved, and the insufficient good human-computer interaction experience can restrict the further development of the robot industry in China.

Disclosure of Invention

The invention aims to provide an augmented reality control system of a hydraulic heavy-duty mechanical arm, which aims to solve the problems of inconvenience in operation, lack of good mobility and multi-platform compatibility in the prior art.

To solve the above technical problem, an embodiment of the present invention provides the following solutions:

an augmented reality control system of hydraulic heavy-duty mechanical arm, comprising:

the field image information acquisition module is used for acquiring the image information of field operation and the spatial position information of the target object;

the manual interactive operation module is used for an operator to intervene in the operation process, issue operation tasks and control the mechanical arm to move through the controller;

the MySQL database module is used for storing user information, collected image information, space position information and path planning information;

the cloud server module is used for generating a target model and a spatial position thereof according to the acquired image information and the spatial position information of the target object at the cloud end, and calculating a path according to the operation task and the path planning information;

the data visualization module is used for displaying the analysis result of the data in the MySQL database module on an operation interface;

the anti-collision detection module is used for verifying whether the calculated path can be safely and reliably executed on the model data level;

the data integration module is used for sorting related calculation data and writing the reliable data verified by the anti-collision detection module into a data frame according to a protocol requirement;

the emergency stop module is used for emergently disconnecting the control on the mechanical arm and locking each movable joint when an emergency happens;

and the 5G communication module is used for providing wireless communication support when information transmission and data exchange are carried out between each platform system and the cloud server module and the MySQL database module.

Preferably, the field image information acquisition module comprises an RGB camera and a structured light depth camera; the RGB camera is used for acquiring image information of field operation and transmitting the image information back in real time, and the acquired image information realizes image recognition and detection of a target object model through an improved DPM algorithm; the coding mode of the structured light depth camera adopts self-adaptive multi-dimensional parameter coding, and an image formed by coded main structured light reflects depth change details of a target object and is used for acquiring three-dimensional space position information of the target object.

Preferably, the manual interactive operation module adopts a self-adaptive master-slave operation strategy based on a large-scale electro-hydraulic drive robot, and adopts a potential difference bidirectional feedback method to realize accurate control between the master robot and the slave robot;

the manual interaction operation module comprises a controller, and an operator controls the tail end of the mechanical arm to move by operating the controller and records the relevant moving position information for use.

Preferably, the user information stored in the MySQL database module includes identity information of enterprise employees and responsible persons who have authority to use the system, specifically including names, personal photos, job numbers and use authority, and the MySQL database module is used for login verification operation before starting to use the system;

the image information and the spatial position information stored by the MySQL database module are used for backup and used as a training data set for learning the target object so as to optimize the recognition rate of the target object;

and the path planning information stored in the MySQL database module is used for backup and subsequent data integration, and corresponding statistical data is calculated to be used as effective auxiliary data for optimizing a path planning algorithm.

Preferably, the data visualization module performs abnormal value detection based on an isolated forest algorithm, specifically analyzes data by using regression analysis and topological data analysis, draws an analysis result in a graph or a histogram mode, and displays the analysis result on an operation interface; the analysis result comprises single planning time, single running time, target identification rate and job completion rate.

Preferably, the Cloud server module adopts a multi-layer hierarchical network structure, and completes data processing and operation at the Cloud end by using a Cloud computing framework combining a Robot-Cloud platform and an SOA framework.

Preferably, the collision avoidance detection module comprises a deep learning based collision monitoring framework that learns robot collision signals and identifies any collision occurrences based on a designed deep neural network model;

the anti-collision detection module is also used for performing rapid simulation operation of single motion planning on a model level by utilizing the system characteristics of Unity 3D; and when the simulation operation result accords with the normal operation, the specific data is transmitted to the mechanical arm joint driver according to the protocol format.

Preferably, the data integration module is specifically configured to issue the reliable data tested by the anti-collision detection module to the mechanical arm according to the standard format of EtherCAT protocol, the frame header, the header, and the format of the message, and according to the joint number of the mechanical arm, so that each joint driver of the mechanical arm reads corresponding data in the data frame to complete work.

The scheme of the invention at least comprises the following beneficial effects:

in the scheme, the real-time image and image augmented reality function is realized on a system operation interface through a field image information acquisition module; the system can check whether the user has the right of login or not if the login system needs to obtain the corresponding right; the user information with login authority is recorded in the MySQL database module in advance; the system stores various data obtained by a field image information acquisition module and a manual interaction operation module through a MySQL database module, and stores the various data in different linked lists according to different classifications; the cloud server module receives data acquired by the field image information acquisition module and generated by the manual interaction operation module, completes an augmented reality task, and performs calculation by combining a path planning algorithm which is prepared in advance and meets the current actual mechanical arm requirement; the obtained data is sent to an anti-collision detection module, and the data is uploaded to a MySQL database module for backup; after the anti-collision detection module receives the data, a collision monitoring framework (Robot CollisionNet) is utilized to perform rapid simulation operation of the motion planning in Unity3D so as to detect the correctness and reliability of the calculation result; when the model simulation operation is successful, the calculation result is correct and reliable, and then the data are further sent to the data integration module, otherwise, the system carries out calculation again; the data frame format is arranged by the data integration module according to the requirements of an EtherCAT protocol and the requirements of actual mechanical arm control, and then data is issued; the system is also provided with an emergency stop module, and after the emergency stop button is pressed down, the system uploads and issues emergency stop data and simultaneously cuts off the power supply of the mechanical arm driver so as to ensure the operation safety; therefore, the intelligent operation, the operation friendliness and the reliability of the mechanical arm can be effectively improved through the coordinated operation of all parts.

Drawings

Fig. 1 is a schematic structural diagram of an augmented reality control system of a hydraulic heavy-duty mechanical arm according to an embodiment of the present invention;

fig. 2 is a schematic workflow diagram of an augmented reality control system of a hydraulic heavy-duty mechanical arm according to an embodiment of the present invention;

fig. 3 is a schematic diagram illustrating an operation principle of a field image information collecting module according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of the operation principle of the MySQL database module provided by the embodiment of the invention;

fig. 5 is a schematic diagram illustrating an operating principle of the cloud server module according to an embodiment of the present invention;

fig. 6 is a schematic diagram illustrating an operation principle of a collision avoidance detection module according to an embodiment of the present invention;

fig. 7 is a schematic diagram of an EtherCAT protocol frame structure according to an embodiment of the present invention.

Description of reference numerals: 101-a field image information acquisition module; 102-a manual interoperation module; 103-MySQL database module; 104-cloud server module; 105-a data visualization module; 106-collision avoidance detection module; 107-a data integration module; 108-emergency stop module; 109-5G communication module; 110-a robotic arm.

Detailed Description

In order to make the technical problems, technical solutions and advantages of the present invention more apparent, the following detailed description is given with reference to the accompanying drawings and specific embodiments.

Aiming at the problem that an intelligent, mobile and visual real-time operation monitoring system for the mechanical arm is lacked in the prior art, the invention provides an augmented reality control system for the hydraulic heavy-duty mechanical arm, so as to solve the demand. As shown in fig. 1, the augmented reality control system for a hydraulic heavy-duty mechanical arm provided by the invention comprises:

the field image information acquisition module 101 is used for acquiring image information of field operation and spatial position information of a target object;

the manual interactive operation module 102 is used for an operator to intervene in an operation process, issue an operation task and control the movement of the mechanical arm through the controller;

the MySQL database module 103 is used for storing user information, acquired image information, spatial position information and path planning information;

the cloud server module 104 is used for generating a target model and a spatial position thereof at the cloud according to the acquired image information and the spatial position information of the target object, completing an augmented reality task, and calculating a path according to the operation task and the path planning information;

the data visualization module 105 is used for displaying the analysis result of the data in the MySQL database module 103 on an operation interface; for example, the data stored in the MySQL database module 103 is analyzed by categories using Anaconda, respectively, and the analysis result is: data such as single planning time, single running time, target recognition rate, operation completion rate and the like are drawn in an easily understood and intuitive mode such as a graph or a histogram and the like and are displayed on an operation interface;

the collision avoidance detection module 106 is used for verifying whether the calculated path can be safely and reliably executed in the model data level; the invention provides a Collision monitoring frame Robot Collision Net based on a deep learning method, a deep neural network model is designed to learn Robot Collision signals and identify any Collision, and rapid simulation operation of the motion planning is carried out on a model level by utilizing the system characteristics of Unity3D, so that whether a path planning algorithm of calculation can be reliably and safely completed or not is verified on a model data level, and unnecessary personnel and property loss is avoided;

the data integration module 107 is used for sorting the related calculation data and writing the reliable data verified by the anti-collision detection module 106 into a data frame according to the protocol requirement;

an emergency stop module 108, configured to, when an emergency occurs, urgently disconnect control over the mechanical arm 110 and lock each movable joint;

the 5G communication module 109 is used for providing wireless communication support when information transmission and data exchange are performed between each platform system and the cloud server module 104 and the MySQL database module 103; for example, the system is used for providing wireless communication support when a real-time image feedback of a camera of a PC-side version system and the field image information acquisition module 101 and data exchange is performed between different versions of each platform system and the cloud server module 104 and the MySQL database module 103.

Fig. 2 is a schematic diagram of a workflow of an augmented reality control system of a hydraulic heavy-duty mechanical arm according to an embodiment of the present invention, in which a real-time image and an image augmented reality function are implemented on an operation interface of the system through a field image information acquisition module; the system can check whether the user has the right of login or not if the login system needs to obtain the corresponding right; the user information with login authority is recorded in the MySQL database module in advance; the system stores various data obtained by a field image information acquisition module and a manual interaction operation module through a MySQL database module, and stores the various data in different linked lists according to different classifications; the cloud server module receives data acquired by the field image information acquisition module and generated by the manual interaction operation module, completes an augmented reality task, and performs calculation by combining a path planning algorithm which is prepared in advance and meets the current actual mechanical arm requirement; the obtained data is sent to an anti-collision detection module, and the data is uploaded to a MySQL database module for backup; after the anti-collision detection module receives the data, a collision monitoring framework (Robot CollisionNet) is utilized to perform rapid simulation operation of the motion planning in Unity3D so as to detect the correctness and reliability of the calculation result; when the model simulation operation is successful, the calculation result is correct and reliable, and then the data are further sent to the data integration module, otherwise, the system carries out calculation again; the data frame format is arranged by the data integration module according to the requirements of an EtherCAT protocol and the requirements of actual mechanical arm control, and then data is issued; the system is also provided with an emergency stop module, and after the emergency stop button is pressed down, the system uploads and issues emergency stop data and simultaneously cuts off the power supply of the mechanical arm driver so as to ensure the operation safety; therefore, the intelligent operation, the operation friendliness and the reliability of the mechanical arm can be effectively improved through the coordinated operation of all parts.

Furthermore, the model of the mechanical arm itself and the field environment model required to be obtained in the embodiment of the invention are drawn by drawing software such as 3DMax and the like on a computer and then exported into files in formats such as fbx or obj, the model is put under an Assets folder of a project, and then the model can be imported into Unity3D engine software, and the used mechanical arm model can also be seen in a Scene view; and C # language is used for writing corresponding basic programs according to different specifications and models of the mechanical arm.

Further, the field image information acquisition module 101 includes an RGB camera and a structured light depth camera; the RGB camera is used for acquiring and transmitting back image information of field operation in real time, and the acquired image information realizes image identification and detection of a target object model through an improved DPM (deformable Parts model) algorithm; the coding mode of the structured light depth camera adopts self-adaptive multi-dimensional parameter coding, and an image formed by coded main structured light reflects depth change details of a target object and is used for acquiring three-dimensional space position information of the target object.

The image recognition target detection work adopts the DPM technology to perform a target detection function. The method comprises the steps of training by utilizing an image recognition technology and a large number of samples of a pre-collected target image to obtain a training model, wherein the DPM is a detection algorithm based on components and combined with an SVM (support Vector machine). Therefore, target selection and target detection and identification are carried out through the camera module, and after a target object is detected, the camera module does not stop working and still returns a real-time picture to the interactive operation interface. The use of DPM techniques involves several steps: preprocessing an image; gamma correction; calculating gradient values of Cell and Block; integrating the Block gradient direction histogram into a feature vector; normalizing the feature vector; and finally, carrying out classification treatment. Then, classification training is carried out by using an SVM, and model training is carried out through training samples with labels printed manually.

The structured light depth camera is used for emitting an infrared dot matrix to a target object, the structured light camera encodes a projection light source, and therefore, an image which is projected to the object by the encoded light source and is subjected to depth modulation on the surface of the object is shot. And when the matching results are consistent, the target identification and detection are finished. And then, calculating the acquired information by using a convolutional neural network to obtain the spatial position information of the target object.

Fig. 3 is a schematic diagram illustrating an operation principle of a field image information collecting module according to an embodiment of the present invention. In this embodiment, the field image information collecting module 101 includes a common RGB camera module and a structured light depth camera module.

In this embodiment, the common RGB camera module may select a sony IMX686 sensor including 1/1.33 ″, a 6400 ten thousand pixel, f/1.69 aperture camera module, and may provide a picture with 1080p resolution and 60fps frames, which is mainly used to collect site environment information and provide the information to an operator to visualize a real site interface. The structured light depth camera module mainly can select a Microsoft Kinectv1 structured light depth camera, the depth camera can realize the resolution of a color picture 640 x 480, the frame number can reach 30fps, the resolution of a depth picture 320 x 240, the frame number is also 30fps, the detection range is 0.8-4.0 m, the acquisition range of 57 degrees in the horizontal direction is reached, and the acquisition range of 43 degrees in the vertical direction is reached.

The common RGB camera module is mainly used for collecting the image information of the site environment and transmitting the site pictures to the interactive operation interface in real time, and meanwhile, the camera module is also used for the target recognition function when the target object is not found at the tail end of the mechanical arm, and the target detection work is finished by adopting the DPM technology. And after the target object is detected, switching to a structured light depth camera module, wherein the structured light depth camera module comprises an infrared camera, a common camera and an infrared dot matrix projector. When the structured light depth camera module begins to operate, the infrared IR emitting end of Kinectv1 projects onto the object pseudo-random speckle infrared spots invisible to the human eye, each pseudo-random speckle spot and each position in the spatial distribution of the set of points in its surrounding window are unique and known. This is because all the data has been pre-stored in the memory of Kinect 1. The size and shape of the speckle projections on the object being viewed will vary depending on the distance and orientation of the object and the camera.

Further, the manual interaction operation module 102 adopts a self-adaptive master-slave operation strategy based on a large-scale electro-hydraulic drive robot, and adopts a potential difference bidirectional feedback method to realize accurate control between the master robot and the slave robot;

the human interaction module 102 includes a controller by which an operator controls the movement of the end of the robot arm and records the position information associated with the movement for use.

Further, the user information stored in the MySQL database module 103 includes identity information of enterprise employees and responsible persons who have authority to use the system, specifically including names, personal photos, job numbers and usage authority, and the MySQL database module 103 is used for starting login verification operation before using the system;

the image information and the spatial position information stored by the MySQL database module 103 are used for backup and used as a training data set for learning the target object so as to optimize the recognition rate of the target object;

the path planning information stored in the MySQL database module 103 is used for backup and subsequent data integration, and the corresponding statistical data is calculated to be used as effective auxiliary data for optimizing the path planning algorithm.

Fig. 4 is a schematic diagram of the operating principle of the MySQL database module according to the embodiment of the present invention. The MySQL database module may employ a MySQL 8.0.19 version database for which use should be made by employees familiar with the operational process with certain operational rights, taking into account that the system may be related to job security issues. Therefore, the module stores the identity information of enterprise employees and responsible persons who have the authority to use the system for login verification. The image information acquired by the field image information acquisition module through the camera is stored in the database, on one hand, the image information is used as a backup, and on the other hand, the image information can be used as a learning training data set for the information of the target object in the future, so that the recognition rate of the target object is optimized. Meanwhile, the path planning data calculated by the cloud server module is also stored in the MySQL database module to serve as a backup, so that the calculated data can be conveniently integrated in the future, and statistical data such as corresponding accuracy and the like are calculated to serve as effective auxiliary data for optimizing the path planning algorithm.

Further, the data visualization module 105 performs outlier detection based on the isolated forest algorithm, specifically analyzes the data by using regression analysis and topological data analysis, and draws the analysis result in a graph or a histogram and the like, and displays the analysis result on the operation interface; the analysis result comprises single planning time, single running time, target recognition rate, job completion rate and the like.

Further, the Cloud server module 104 adopts a multi-layer hierarchical network structure, and a Cloud computing framework combining a Robot-Cloud platform and an SOA (service oriented architecture) is used to complete data processing and operation at the Cloud, so that the computing pressure of local equipment is reduced, the computing convenience is improved, and favorable conditions are provided for a user on system upgrading, file verification, log checking and the like. The invention can reduce the computational burden of field equipment, increase the use friendliness of the system, and is beneficial to popularizing the system, so that the system can be used on more common equipment and mobile equipment, such as: notebook computer, cell-phone, panel computer etc. no longer receive the restriction of distance, increase the portability of operating device.

Fig. 5 is a schematic diagram of an operating principle of the cloud server module according to an embodiment of the present invention. In this embodiment, the cloud server module processes data required by the system to execute a task through an arry cloud, an Tencent cloud, or an AmazonAWS service. With the popularization of the 5G technology, more and more products and devices can be connected together through the 5G technology. The information collected by the field image information collection module, the tasks for completing augmented reality and the path planning algorithm information are uploaded to the cloud terminal through a 5G communication technology, and the cloud terminal resources are utilized for calculation. When a Robot or robotic arm connected to the system sends a request to the Robot-Cloud, the controller will look for the services required by the request, provide the services when they are available, or wait in line. The SOA interface layer is responsible for providing configuration and management interfaces of the service, and the service situation information is monitored and recorded by the SOA interface layer. The resource layer of the module belongs to an IaaS layer and integrates hardware resources; the cloud platform belongs to a PaaS layer and provides computing services.

Further, the collision avoidance detection module 106 includes a deep learning based collision monitoring framework Robot CollisionNet that learns Robot collision signals based on a designed deep neural network model and identifies any collision occurrences;

the collision avoidance detection module 106 is further configured to perform a fast simulation operation of single motion planning at a model level by using the system characteristics of Unity 3D; and when the simulation operation result accords with the normal operation, the specific data is transmitted to the mechanical arm joint driver according to the protocol format.

Fig. 6 is a schematic view of an operation principle of the collision avoidance detection module according to the embodiment of the present invention. In the present embodiment, the one-dimensional CNN is adopted to process the obtained data. This data-driven approach unifies the ability to extract features from high-dimensional signals and decision-making processes. The network has no pooling layers and utilizes efficient convolution filling to reduce the length of the feature map. Robot Collision Net eliminates heuristic and tedious properties of a traditional decision making process, and displays high detection performance and generalization capability in real time. By using the system characteristics of Unity3D, after the data are uploaded to a cloud server for path planning calculation, a calculation result is finally obtained and returned to the local, but before the data are sent to the mechanical arm in an EtherCAT bus communication mode, a delay is set for eliminating possible calculation errors or planning errors of a path planning algorithm, the data obtained by calculation are sent to Unity3D, the calculation data are used for an existing model file, and the rapid simulation operation of the motion planning is performed on a model level. And if the simulation operation result accords with the normal operation, the specific data is transmitted to the mechanical arm joint driver according to the protocol format.

Further, the data integration module 107 is specifically configured to issue the reliable data tested by the anti-collision detection module 106 to the mechanical arm according to the EtherCAT protocol standard format, the frame header, the header, and the format of the message, and according to the joint number of the mechanical arm, so that each joint driver of the mechanical arm reads corresponding data in the data frame to complete work. The standard format of EtherCAT protocol, the frame header, the header and the message format are shown in fig. 7.

The invention utilizes the characteristic that the Unity3D can be used for manufacturing multi-platform multi-version applications, in order to maximize the mobility and portability functions of the system platform and consider the operation limitation of the mobile terminal equipment, part of the functions of the system are manufactured into the mobile terminal version (such as a mobile phone and a tablet computer terminal) and are released. Wherein, the functions of the 5G communication module 109 include:

returning the image information acquired on site in real time to display the returned picture on an operation interface; various data required by the calculation of the mechanical arm path planning are uploaded to the cloud end through a 5G network technology, and calculation is carried out at the cloud end; after the computation is completed, the obtained data is sent back to the local by the 5G network for later operation.

In summary, in the augmented reality control system for the hydraulic heavy-duty mechanical arm according to the embodiment of the present invention, the real-time image and the image augmented reality function are realized on the system operation interface through the field image information acquisition module; the system can check whether the user has the right of login or not if the login system needs to obtain the corresponding right; the user information with login authority is recorded in the MySQL database module in advance; the system stores various data obtained by a field image information acquisition module and a manual interaction operation module through a MySQL database module, and stores the various data in different linked lists according to different classifications; the cloud server module receives data acquired by the field image information acquisition module and generated by the manual interaction operation module, completes an augmented reality task, and performs calculation by combining a path planning algorithm which is prepared in advance and meets the current actual mechanical arm requirement; the obtained data is sent to an anti-collision detection module, and the data is uploaded to a MySQL database module for backup; after the anti-collision detection module receives the data, a collision monitoring framework (Robot CollisionNet) is utilized to perform rapid simulation operation of the motion planning in Unity3D so as to detect the correctness and reliability of the calculation result; when the model simulation operation is successful, the calculation result is correct and reliable, and then the data are further sent to the data integration module, otherwise, the system carries out calculation again; the data frame format is arranged by the data integration module according to the requirements of an EtherCAT protocol and the requirements of actual mechanical arm control, and then data is issued; the system is also provided with an emergency stop module, and after the emergency stop button is pressed down, the system uploads and issues emergency stop data and simultaneously cuts off the power supply of the mechanical arm driver so as to ensure the operation safety.

In the embodiment of the invention, the hydraulic heavy-duty mechanical arm augmented reality control system is adopted to carry out autonomous or man-machine interaction operation, so that the intelligent and interactive friendly operation of the hydraulic heavy-duty mechanical arm augmented reality control system is realized.

While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims (3)

1. The utility model provides a hydraulic pressure heavily carries arm augmented reality control system which characterized in that includes:

the field image information acquisition module is used for acquiring the image information of field operation and the spatial position information of the target object;

the field image information acquisition module comprises an RGB camera and a structured light depth camera; the RGB camera is used for acquiring image information of field operation and transmitting the image information back in real time, and the acquired image information realizes image recognition and detection of a target object model through an improved DPM algorithm; the coding mode of the structured light depth camera adopts self-adaptive multi-dimensional parameter coding, and an image formed by coded main structured light reflects depth change details of a target object and is used for acquiring three-dimensional space position information of the target object;

the manual interactive operation module is used for an operator to intervene in the operation process, issue operation tasks and control the mechanical arm to move through the controller;

the manual interactive operation module adopts a self-adaptive master-slave operation strategy based on a large-scale electro-hydraulic drive robot and adopts a potential difference bidirectional feedback method to realize accurate control between the master robot and the slave robot;

the manual interaction operation module comprises a controller, and an operator controls the tail end of the mechanical arm to move by operating the controller and records the relevant moving position information for use;

the MySQL database module is used for storing user information, collected image information, space position information and path planning information;

the user information stored by the MySQL database module comprises identity information of enterprise employees and responsible persons who have authority to use the system, specifically comprises names, personal photos, job numbers and use authority, and the MySQL database module is used for login verification operation before the system is used;

the image information and the spatial position information stored by the MySQL database module are used for backup and used as a training data set for learning the target object so as to optimize the recognition rate of the target object;

the path planning information stored in the MySQL database module is used for backup and subsequent data integration, and corresponding statistical data are calculated to serve as effective auxiliary data for optimizing a path planning algorithm;

the cloud server module is used for generating a target model and a spatial position thereof according to the acquired image information and the spatial position information of the target object at the cloud end, and calculating a path according to the operation task and the path planning information;

the Cloud server module adopts a multi-layer hierarchical network structure, and completes data processing and operation at the Cloud end by using a Cloud computing framework combining a Robot-Cloud platform and an SOA framework;

the data visualization module is used for displaying the analysis result of the data in the MySQL database module on an operation interface;

the anti-collision detection module is used for verifying whether the calculated path can be safely and reliably executed on the model data level;

the data integration module is used for sorting related calculation data and writing the reliable data verified by the anti-collision detection module into a data frame according to a protocol requirement;

the data integration module is specifically used for issuing the reliable data tested by the anti-collision detection module to the mechanical arm according to the standard format of an EtherCAT protocol, the frame header, the header and the format of a message and according to the joint number of the mechanical arm so that each joint driver of the mechanical arm reads corresponding data in the data frame to complete work;

the emergency stop module is used for emergently disconnecting the control on the mechanical arm and locking each movable joint when an emergency happens;

and the 5G communication module is used for providing wireless communication support when information transmission and data exchange are carried out between each platform system and the cloud server module and the MySQL database module.

2. The hydraulic heavy-duty mechanical arm augmented reality control system as claimed in claim 1, wherein the data visualization module performs outlier detection based on an isolated forest algorithm, performs specific analysis on the data by using regression analysis and topological data analysis, and plots the analysis result in a graph or a histogram manner and displays the result on an operation interface; the analysis result comprises single planning time, single running time, target identification rate and job completion rate.

3. The hydraulic heavy-duty robotic arm augmented reality handling system of claim 1, wherein the collision avoidance detection module comprises a deep learning based collision monitoring framework that learns robot collision signals and identifies any collision occurrences based on a designed deep neural network model;

the anti-collision detection module is also used for performing rapid simulation operation of single motion planning on a model level by utilizing the system characteristics of Unity 3D; and when the simulation operation result accords with the normal operation, the specific data is transmitted to the mechanical arm joint driver according to the protocol format.

Images