CN113157953B - Cross-terminal picture transmission method and system - Google Patents
Cross-terminal picture transmission method and system Download PDFInfo
- Publication number
- CN113157953B CN113157953B CN202110205500.9A CN202110205500A CN113157953B CN 113157953 B CN113157953 B CN 113157953B CN 202110205500 A CN202110205500 A CN 202110205500A CN 113157953 B CN113157953 B CN 113157953B
- Authority
- CN
- China
- Prior art keywords
- picture
- deep learning
- pictures
- exchange space
- learning program
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/535—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
Abstract
The utility model provides a cross-terminal picture transmission method and system, which are applied to a first terminal and a second terminal, wherein the first terminal is provided with a robot operating system, and the second terminal is provided with a preset deep learning network model, and the method comprises the following steps: the data forwarding node on the second terminal stores pictures from the data stream of the robot operating system to the exchange space in real time according to the picture dislocation storage mode, and a deep learning program of a preset deep learning network model reads and processes the pictures from the exchange space according to the picture dislocation reading mode; the invention provides a picture misplacing storage mode and a picture misplacing reading mode, which can avoid file reading and writing conflict in the high-speed data transmission process, and enable the reading and writing of files to run in parallel in different processes, thereby ensuring the final low-delay transmission.
Description
Technical Field
The present disclosure relates to the field of data transmission technologies, and in particular, to a method and a system for transmitting a cross-terminal picture.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
ROS (Robot Operating System) has been widely used in Robot development to provide a communication framework and a dependency package for various programs of a Robot. By utilizing the ROS framework, various parts of the robot can be controlled efficiently, and various robot algorithms can be operated. The ROS supports multiple programming languages, efficient communication among scripts of different programming languages is achieved, the best two programming languages are Python and C + +, and programs serve as ROS nodes to run in the robot. ROS will still be the dominant framework for robot development for the next several years. Deep learning is a new method widely applied to many computer vision problems such as article recognition, face recognition, image segmentation, human body tracking and the like. Most deep learning programs require high computational power, are usually run on a server with a high-performance graphics card, and cannot be run on the robot body.
The application of deep learning programs such as article identification, face identification, image segmentation and human body tracking to the robot based on the ROS becomes an important development requirement, and to achieve the purpose, the deep learning program on the server needs to be ensured to obtain real-time image data from the ROS data stream of the robot body in a low-delay manner.
The inventor finds that the ROS supports cross-terminal data transmission among nodes, but many deep learning programs based on Python3 are poor in compatibility with the ROS, and the deep learning program of Python3 is difficult to directly acquire real-time pictures from ROS data streams through ROS communication. This is mainly due to several reasons: (1) the most popular ROS version at present is the Python2 version supported by the native, but many packages of Python3 cannot be perfectly adapted to ROS, or even cannot be directly operated; (2) deep learning based on Python3 is generally complex and highly dependent, and is difficult to reconstruct into a Python2 program; (3) each Python node in the ROS frame shares the same operating environment, so that the environmental protection is difficult to realize; the deep learning program usually needs to be operated in various ways and in different versions, and environment protection needs to be performed on the deep learning program in different depths. The three points bring great difficulty to the application of the deep learning program of the server end to the robot based on the ROS.
Disclosure of Invention
In order to solve the defects of the prior art, the disclosure provides a cross-terminal picture transmission method and a system, provides a picture misplaced storage mode and a picture misplaced reading mode, can avoid file read-write conflict in the high-speed data transmission process, and enables the read and write of a file to run in parallel in different processes, thereby ensuring the final low-delay transmission.
In order to achieve the purpose, the following technical scheme is adopted in the disclosure:
the disclosure provides a cross-terminal picture transmission method.
A cross-terminal picture transmission method is applied to a first terminal and a second terminal, wherein the first terminal is provided with a robot operating system, and the second terminal is provided with a preset deep learning network model, and comprises the following steps:
and the data forwarding node on the second terminal stores the pictures from the data stream of the robot operating system to the exchange space in real time according to the picture dislocation storage mode, and a deep learning program of the preset deep learning network model reads and processes the pictures from the exchange space according to the picture dislocation reading mode.
As a further limitation, a picture misplaced storage mode and a picture misplaced reading mode dynamically maintain two pictures in the exchange space and ensure the real-time performance of the pictures, if only one picture currently exists in the exchange space, the data forwarding node captures and stores the two pictures from the real-time data stream of the robot operating system according to the picture misplaced storage mode into another missing picture to realize picture updating, if two pictures currently exist in the exchange space, the deep learning program reads and processes one of the pictures according to the picture misplaced reading mode, and circularly reads and processes different pictures for two adjacent times, and the deep learning network model deletes the picture after processing one picture to prepare for receiving a new picture.
As a further limitation, in the staggered storage mode, only one picture in the first picture and the second picture is stored at the same time, and the other picture is not processed by the data forwarding node; in the staggered reading mode, only one picture in the first picture and the second picture at the same time is read and processed, and the other picture cannot be processed by the deep learning program; and the data forwarding node and the deep learning program can not process the same picture at the same time.
As a further limitation, in the operation process of the data forwarding node, firstly, the number and name of the pictures in the exchange space at the current moment are retrieved;
if the first picture and the second picture exist in the exchange space at the same time, the data forwarding node does not process the first picture and continues to circularly execute the first step; otherwise, executing downwards;
if no picture exists in the exchange space, the data forwarding node continuously stores the pictures in the two real-time data streams in the exchange space, stores the pictures as a first picture and a second picture in sequence, and continues to circularly execute the first step; otherwise, executing downwards;
if the number of the pictures is 1 and the current picture in the exchange space is the first picture, reading one picture in a real-time data stream of the robot operating system by the data forwarding node at the moment, and storing the picture as a second picture in the exchange space;
if the number of the pictures is 1 and the current picture in the exchange space is the second picture, the data forwarding node saves the picture as the first picture, and then the data forwarding node continues to circularly execute the first step, namely, the number of the pictures and the names of the pictures in the exchange space at the current moment are retrieved.
As a further limitation, in the running process of the deep learning program, firstly, the number and the name of the pictures in the exchange space at the current moment are retrieved;
if the first picture and the second picture exist in the exchange space at the same time, the deep learning program executes the next step; otherwise, not processing, and continuously and circularly executing the first step;
if the picture is circularly read by running for the first time, deleting the first picture and the second picture in the exchange space by the deep learning program, and continuously and circularly executing the first step; otherwise, executing the next step downwards;
if the picture processed by the deep learning program last time is the first picture, reading a second picture in the exchange space; if the picture processed by the deep learning program last time is the second picture or the picture processed for the first time at the moment, reading the first picture in the exchange space, and then executing the next step downwards;
the deep learning model processes the read pictures, and then the next step is executed downwards;
and deleting the processed pictures in the exchange space by the deep learning program, and continuously and circularly executing the first step, namely retrieving the number and the names of the pictures in the exchange space at the current moment.
As a further limitation, when the data forwarding node and the deep learning program are started, the number of pictures in the exchange space is zero, at this time, the deep learning program does not read the pictures according to the staggered reading mode, and the data forwarding node reads the data stream of the real-time robot operating system according to the staggered storage mode and sequentially writes the data stream into the first picture and the second picture;
after the previous step is completed, a first picture and a second picture exist in the exchange space at the same time, at the moment, because the picture writing process is that an empty file is created first and then the content is filled, the deep learning program detects that the second picture is not necessarily completely written, and the deep learning program reads the first picture according to a staggered reading mode and deletes the first picture after the processing is completed; if the data forwarding node does not finish writing the second picture, continuing writing, and after the writing is finished, waiting for the deep network model to finish processing the first picture;
at the moment, only a second picture exists in the exchange space, and at the moment, the deep learning program does not read the picture according to a dislocation reading mode; the data forwarding node reads a data stream of the real-time robot operating system according to the staggered storage mode and starts to write a first picture;
after the previous step is completed, a first picture and a second picture exist in the exchange space at the same time, at the moment, because the picture writing process is that an empty file is created first and then the content is filled, the deep learning program detects that the first picture is not necessarily completely written, and the deep learning program reads the second picture according to a staggered reading mode and deletes the second picture after the processing is completed; if the data forwarding node does not finish the writing of the first picture, continuing the writing, and after the writing is finished, waiting for the completion of the processing of the second picture by the deep network model;
after the previous step, only the first picture exists in the exchange space, and at the moment, the deep learning program does not read the picture according to the staggered reading mode; the data forwarding node reads the data stream of the real-time robot operating system according to the staggered storage mode and starts to write a second picture;
and returning to the step when the first picture and the second picture exist in the exchange space at the same time, and executing in a circulating way.
As a further limitation, the data forwarding node and the deep learning program run independently and in parallel, the data forwarding node and the deep learning program are not related in two levels of running dependence and code, and the deep learning program can configure a corresponding Python environment at will.
As a further limitation, the data forwarding node is based on the Python2 environment in the robot operating system, all Python nodes in the robot operating system share the Python2 environment, the deep learning program is based on the Python3 environment, and the two environments are isolated from each other and do not affect each other.
A second aspect of the present disclosure provides a cross-terminal picture transmission system.
A cross-terminal picture transfer system comprising: the robot comprises a first terminal and a second terminal, wherein the first terminal is provided with a robot operating system, and the second terminal is provided with a preset deep learning network model;
and the data forwarding node on the second terminal stores the pictures from the data stream of the robot operating system to the exchange space in real time according to the picture dislocation storage mode, and a deep learning program of the preset deep learning network model reads and processes the pictures from the exchange space according to the picture dislocation reading mode.
Compared with the prior art, the beneficial effect of this disclosure is:
1. the method or the system can realize that the real-time pictures in the ROS data stream are transmitted to the deep learning program with low delay, and the picture staggered storage mode and the picture staggered reading mode can avoid file read-write conflict in the high-speed data transmission process, and enable the reading and writing of the files to run in parallel in different processes, thereby ensuring the final low-delay transmission.
2. The method or the system has high universality, can be applied to various deep learning programs with different operating environments, and the deep learning Python3 environment does not conflict with the Python2 environment of the ROS; the environment of different depth learning network models can be isolated by combining Anaconda to ensure that environment dependence does not have any conflict when different depth learning programs and ROS forwarding nodes are simultaneously started.
3. The method or the system has strong usability, does not need to perform complex processes such as code reconstruction or package-dependent recompilation, can package the original deep learning program only by adding extra file reading and writing codes to the deep learning program and realizing the staggered reading rule of the picture, and has small change on the original deep learning program and easy deployment; in addition, the method and the device are easy to use, and the starting sequence of the ROS forwarding node and the packaged deep learning program is not required to be concerned.
Advantages of additional aspects of the disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the disclosure.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure and are not to limit the disclosure.
Fig. 1 is a schematic flow chart of a method for transmitting pictures of ROS data flow to a Python 3-based deep learning process according to embodiment 1 of the present disclosure.
Fig. 2 is a schematic flowchart of a conventional deep learning procedure provided in embodiment 1 of the present disclosure.
Fig. 3 is a schematic flow chart illustrating a rule for storing a picture in a misaligned state according to embodiment 1 of the present disclosure.
Fig. 4 is a schematic flow chart illustrating a rule for reading a misaligned picture according to embodiment 1 of the present disclosure.
Fig. 5 is a schematic diagram of a state change during runtime provided in embodiment 1 of the present disclosure.
Detailed Description
The present disclosure is further described with reference to the following drawings and examples.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.
Example 1:
as shown in fig. 1-5, embodiment 1 of the present disclosure provides a method for transmitting pictures at high speed across terminals by ROS data streaming to a Python 3-based deep learning procedure. The method can realize the cross-terminal transmission of the ROS image data stream on the robot body computer and the deep learning program on the server with low delay; the Python3 environment for protecting deep learning can not conflict with the public Python2 environment of ROS, and the method can be applied to various deep learning programs with different running environments and has high universality; the deployment of the method can be completed under the condition of less change to the deep learning program, code reconstruction is not needed, the starting sequence is not needed to be concerned, and the deployment and the use are easy.
The method of the embodiment adopts a hard disk-based picture storage and reading mode to solve the problem of the data stream in the ROS and the picture transmission of the deep learning program without using the ROS and the problem of the conflict between the common Python2 running environment of the ROS and the Python3 environment of the deep learning. In addition, the strict time sequence of the storage and reading of the picture based on the hard disk is ensured by the proposed picture misplaced storage rule and the proposed picture misplaced reading rule, so that the problems of conflict during high-speed file reading and writing and difficulty in controlling the reading and writing time interval are solved, and finally, low-delay high-speed transmission of picture data is realized.
The general flow chart of this embodiment is shown in fig. 1. In this embodiment, a new empty folder is first established as an exchange space for reading and writing pictures. On the basis, the embodiment establishes an ROS forwarding node and encapsulates deep learning codes.
As shown in the left side of fig. 1, the ROS forwarding node first initializes the ROS data stream so as to obtain real-time image data in the robot body at any time, and then continuously stores images from the real-time ROS data stream into the swap space according to the image-misplacing storage rule proposed in this embodiment.
In addition, as shown in the right side of fig. 1, the method encapsulates the deep learning procedure that needs to invoke the ROS picture data stream. Fig. 2 shows a flow of a conventional deep learning procedure, and a code for calling a deep learning model according to a picture misalignment reading rule is added to the method according to the embodiment of the present invention on the basis of a flow of a complete deep learning procedure. The codes added in the deep learning program in the method only relate to file reading and writing and judgment, and the non-functional codes do not influence the original codes or dependence.
As can be seen from the general flowchart of fig. 1, the core of the method described in this embodiment is a picture misalignment storage rule and a picture misalignment reading rule, which are respectively deployed in the ROS forwarding node and the encapsulated deep learning program to implement ordered reading and writing of the swap space through mutual cooperation, so that the pictures are transmitted from the ROS forwarding node to the deep learning program with low delay. The dislocation storage rule and the dislocation reading rule have double functions of an exchange space, namely a data transmission channel in the form of a hard disk file is provided, and mark signals of respective read-write start-stop of an ROS forwarding node and a packaged deep learning program are provided.
Therefore, the following detailed description is directed to three aspects of the picture offset storage rule, the picture offset reading rule, and the starting method and principle analysis of the picture transmission method.
S1: rule of storing pictures in dislocation
The flow of the picture misplacement storage rule is shown in fig. 3, in order to enable the deep learning procedure to call the pictures in the ROS data stream at a high speed, the picture misplacement storage rule is responsible for the write-in logic of the picture forwarding, and the rule can ensure that the real-time performance of two pictures of 1.jpg and 2.jpg and the pictures is dynamically maintained in the swap space. If only one picture (e.g., 1.jpg) currently exists in the swap space, the rule captures and saves another missing picture (e.g., 2.jpg) from the ROS real-time data stream in time.
The specific process is illustrated as follows:
(1) and retrieving the number and name of the pictures in the exchange space at the current moment.
(2) If two pictures of 1.jpg and 2.jpg exist in the exchange space at the same time, the step (1) is continuously and circularly executed without processing; otherwise, executing (3) downwards.
(3) If no picture exists in the exchange space (the situation occurs when the ROS forwarding node is just started or the deep learning program is just started), continuously storing the pictures in two real-time data streams in the exchange space into 1.jpg and 2.jpg in sequence, and continuously and circularly executing the step (1); otherwise, the step (4) is executed downwards.
(4) If the number of the pictures is 1 and the number of the pictures currently existing in the exchange space is 1.jpg, reading one picture in the ROS real-time data stream at the moment, and storing the picture in the exchange space as 2. jpg; if the number of pictures is 1 and the current picture in the swap space is 2.jpg, then the picture is saved as 1. jpg. And then continuing to circularly execute the step (1).
The staggered storage rule ensures that only one picture is stored in the 1.jpg and the 2.jpg at the same time, and the other picture cannot be processed by the ROS forwarding node, so that the possibility of reading the other picture in parallel by other programs is provided. In addition, the real-time picture retrieval ensures that a certain picture is lost in the exchange space and then can be supplemented in time.
S2: rule of reading pictures in dislocation
The flow of the picture misplaced reading rule is shown in fig. 4, and the picture misplaced reading rule is matched with the picture misplaced storage rule, so that the pictures stored in the exchange space are read into the deep learning program according to strict logic to be used as model input. Under the rule, only when two pictures of 1.jpg and 2.jpg exist in the swap space at the same time, the pictures are read, and two adjacent times process different pictures circularly. After each picture is processed by the deep learning, the picture is deleted.
The specific process is illustrated as follows:
(1) and retrieving the number and name of the pictures in the exchange space at the current moment.
(2) If two pictures of 1.jpg and 2.jpg exist in the swap space at the same time, then execute (3) downwards; otherwise, not processing, and continuing to circularly execute the step (1).
(3) If the pictures are circularly read by running for the first time, deleting two pictures of 1.jpg and 2.jpg in the exchange space, and continuously and circularly executing the step (1); otherwise, execute (4) downward.
(4) If the name of the last deep learning processing picture is 1.jpg, reading a 2.jpg file in the exchange space; if the picture name of the last deep learning processing is 2.jpg or the picture is processed for the first time at this time, the 1.jpg file in the swap space is read, and next (5) is executed downward.
(5) The deep learning model processes the read pictures, which is then performed downwards (6).
(6) And (4) deleting the processed pictures in the exchange space, and continuing to circularly execute the step (1).
In the above rule, the key to avoid read-write collision in the high-speed read-write state is that two adjacent loops process different pictures.
Taking one of the cases as an example, the process of avoiding read-write collision is analyzed as follows:
for example, when the deep learning function finishes processing 1.jpg, the ROS forwarding node immediately deletes 1.jpg, and then the ROS forwarding node immediately rewrites the real-time image of the ROS data stream into the deleted 1. jpg. The process of writing the picture is divided into two steps of writing an empty file and filling the content, and the fact that the file name exists under the condition of high-speed reading and writing does not mean that the writing is finished. In order to prevent the ROS forwarding node from not writing 1.jpg completely, the picture misplaced reading rule reads and processes 2.jpg at the next processing, and enough writing time is left for 1. jpg. Since the time required for calling the deep learning model is far longer than the time required for reading and writing the file, when the deep learning processing 2.jpg is called, 1.jpg can be completely written and completed by using the time.
In addition, the step of deleting the processed file in the picture misplaced reading rule plays two roles of giving a mark signal for the ROS forwarding node to start writing and enabling the deep learning program to stop reading the picture by controlling the number of the pictures. When the picture is read in a first running cycle, deleting 1.jpg and 2.jpg is to prevent the delay caused by too long time interval from the time when the ROS forwarding node reads the picture for the first time to the time when the deep learning program is started, so that self-adjustment during starting is realized, and a user does not need to care about the starting sequence of the ROS forwarding node and the deep learning program.
The strict logic provided by the picture dislocation reading rule can effectively avoid the read-write conflict of the two programs in the high-speed read-write process, thereby improving the read-write speed and reducing the delay.
S3: starting mode and principle analysis of picture transmission method
Only two steps are needed by using the method of the embodiment, and in the first step, all pictures in the swap space are emptied; and the second step starts the ROS forwarding node and the packaged deep learning program (without concerning the starting sequence).
After the start, the ROS forwarding node and the deep learning program run independently and in parallel, according to the picture storage rule and the picture reading rule proposed in this embodiment, the state change time sequence of the ROS forwarding node, the deep learning program, and the swap space during the running process is shown in fig. 5 (the state when both the ROS forwarding node and the deep learning program are completely started is time sequence 1). As can be seen from fig. 5, after the method of this embodiment is started, under the control of strict picture storing and reading rules, there are five execution states in total, and the execution states are switched cyclically, and this embodiment describes the five execution states separately:
(1) when the ROS forwarding node and the deep learning program are started, the number of the exchange space pictures is 0, and at the moment, the deep learning program does not read the pictures according to the dislocation reading rule; the ROS forwarding node reads a real-time ROS data stream according to a dislocation storage rule and sequentially writes 1.jpg and 2. jpg;
(2) after time sequence 1, 1.jpg and 2.jpg exist in the exchange space at the same time, at this time, because the picture writing process is to create an empty file and then fill in the content, the deep learning program detects that 2.jpg but the complete writing is not necessarily finished, the deep learning program processes 1.jpg according to the dislocation reading rule and deletes the 1.jpg after the completion of the processing; if the ROS forwarding node does not finish the 2.jpg writing, continuing to write, and waiting after the writing is finished (because the deep learning calculation amount is large, the sum of the image reading time and the deep learning model processing time is certainly larger than the image writing time, the 2.jpg can be completely written before the 1.jpg processing is finished);
(3) through the time sequence 2, only 2.jpg exists in the exchange space, and at the moment, the deep learning program does not read the picture according to the dislocation reading rule; the ROS forwarding node reads the real-time ROS data stream according to the dislocation storage rule and starts to write in 1. jpg;
(4) after a time sequence 3, 1.jpg and 2.jpg exist in the exchange space at the same time, at this time, because the picture writing process is to create an empty file and then fill in the content, the deep learning program detects that 1.jpg but the complete writing is not necessarily finished, the deep learning program processes 2.jpg according to the dislocation rule and deletes the 2.jpg after the processing is finished; if the ROS forwarding node does not finish 1.jpg writing, continuing writing, and waiting after writing is finished;
(5) through a time sequence 4, only 1.jpg exists in the exchange space, and at the moment, the deep learning program does not read the picture according to the dislocation reading rule; the ROS forwarding node reads the real-time ROS data stream according to the dislocation storage rule and starts to write in 2. jpg;
(6) jump to sequence 2 and execute in a loop.
Combining the transformation timing of the execution state and the picture dislocation reading and storing rule, it can be seen that, in this embodiment, the ROS forwarding node maximally reduces the delay in the picture transmission process when transmitting the picture to the deep learning procedure, and finally reduces the transmission delay to a time close to the time necessary for the deep learning model processing by the following aspects: (1) read-write conflicts can be effectively avoided through the dislocation rules, and strict logics are stipulated; (2) the reading and storing operations of the pictures are completely carried out in parallel, and when one picture is read, the other picture is written in, so that the time consumption of reading and writing switching is reduced; (3) the method of the embodiment fully utilizes the dislocation rule and the execution time of the deep learning model to ensure the complete writing of the picture.
In this embodiment, the ROS forwarding node and the deep learning program run independently and in parallel, and both the ROS forwarding node and the deep learning program have no association in both the running dependency level and the code level, so that the deep learning program can configure a corresponding Python environment at will. The ROS forwarding node is based on the Python2 environment in the ROS, and all Python nodes in the ROS share the Python2 environment. While deep learning programs are mostly Python3 environments and different deep learning programs may depend on different packages and versions.
Therefore, the method of the embodiment ensures that the deep learning program is suitable for different operating environments, so that the method has high universality and usability by the following aspects: (1) the environment dependence is free of any conflict when the learning programs with different depths and the ROS forwarding node are simultaneously started by combining the environment of isolating the learning programs with different depths of Anaconda, the method can be applied to the problem that the dependence on various learning programs with different depths (2) is not required to face the problem that the compatibility of Python3 and the ROS is poor (3) the complicated processes of code reconstruction or the recompilation of dependence packages and the like are not required, only extra codes of file reading and writing and the image dislocation reading rule are required to be added into the deep learning programs, the original deep learning programs are slightly changed, and the method is easy to deploy.
The present section further describes the application and implementation process of the present embodiment by taking the background of article identification as an example.
Suppose the article identification model uses an M2Det deep learning article identification model written in Python3, which runs on a server and depends on packages such as pytorch0.4.1 and opencv-Python. Suppose that the ROS node of the robot body control camera publishes a real-time captured picture to the/camera _ color _ raw topic.
The Python environment in the ROS frame is Python2, and the M2Det item recognition program cannot be directly added to the ROS frame, which means that the item recognition program cannot directly use the real-time picture on the topic of camera _ color _ raw in the robot body. In this case, by using the method of this embodiment, the M2Det article identification program can invoke the real-time picture data on the topic of/camera _ color _ raw in the robot body at high speed and with low delay, and the dependency of the M2Det article identification program can be configured arbitrarily, so that the operating environment is isolated from the Python2 environment of the ROS itself.
According to the main flow diagram of fig. 1, an empty folder is first created as swap space.
The ROS forwarding node is written according to the flow on the left side of fig. 1. The ROS forwarding node is written based on the ROS, and writes the real-time pictures on the/camera _ color _ raw topic into the exchange space according to the picture dislocation storage rule. The ROS forwarding node is written in Python 2. The ROS forwarding node is written, firstly, an ROS data stream is initialized by using an ROS framework, the initialization mainly comprises the processes of node registration, topic subscription and the like, and the process is consistent with the initialization process of a conventional ROS subscriber. After the ROS data stream is initialized, the picture can be stored to the exchange space circularly according to the picture dislocation storage rule. The picture misplacement storage rule is the core of the ROS forwarding node, and provides strict writing logic for the whole picture transmission process. The specific contents of the picture misplacement storage rule are as follows:
(1) retrieving the number and name of the pictures in the exchange space at the current moment;
(2) if two pictures of 1.jpg and 2.jpg exist in the exchange space at the same time, the step (1) is continuously and circularly executed without processing; otherwise, executing (3) downwards;
(3) if no picture exists in the exchange space (the situation occurs when the ROS forwarding node is just started or the deep learning program is just started), continuously storing the pictures in two real-time data streams in the exchange space into 1.jpg and 2.jpg in sequence, and continuously and circularly executing the step (1); otherwise, executing (4) downwards;
(4) if the number of the pictures is 1 and the number of the pictures currently existing in the exchange space is 1.jpg, reading one picture in the ROS real-time data stream at the moment, and storing the picture in the exchange space as 2. jpg; if the number of pictures is 1 and the current picture in the swap space is 2.jpg, then the picture is saved as 1. jpg. Continuing to circularly execute the step (1);
and compiling the picture dislocation storage rule into a specific code form, and circularly executing in the ROS forwarding node to finish the compiling of the ROS forwarding node.
The M2Det item identification procedure is encapsulated according to the flow on the right side of fig. 1. Most of code flows of the deep learning program are shown in fig. 2, and the method of this embodiment uses a picture misplaced reading rule to circularly call the deep learning model on the premise of ensuring that the original flow of the deep learning program is not changed, so as to implement packaging of the deep learning program. Therefore, the picture dislocation reading rule is the core of the packaging deep learning program, and strict reading logic is provided for ROS data flow to the deep learning program to transmit pictures across terminals at high speed.
The specific content of the picture dislocation reading rule is as follows:
(1) and retrieving the number and name of the pictures in the exchange space at the current moment.
(2) If two pictures of 1.jpg and 2.jpg exist in the swap space at the same time, then execute (3) downwards; otherwise, not processing, and continuing to circularly execute the step (1).
(3) If the pictures are circularly read by running for the first time, deleting two pictures of 1.jpg and 2.jpg in the exchange space, and continuously and circularly executing the step (1); otherwise, execute (4) downward.
(4) If the name of the last deep learning processing picture is 1.jpg, reading a 2.jpg file in the exchange space; if the picture name of the last deep learning processing is 2.jpg or the picture is processed for the first time at this time, the 1.jpg file in the swap space is read, and next (5) is executed downward.
(5) And transmitting the read pictures to a deep learning model for processing. Next (6) is performed downwards.
(6) And (4) deleting the processed pictures in the exchange space, and continuing to circularly execute the step (1).
And adding the picture dislocation reading rule into the deep learning program in a specific code form, so that the deep learning program can be packaged. Since the packaged deep learning program is completely independent, the Python environment dependency of deep learning can be configured using Anaconda.
When the method of the embodiment is used, only the ROS forwarding node and the packaged deep learning program need to be started respectively, and the starting sequence does not need to be concerned.
Example 2:
an embodiment 2 of the present disclosure provides a cross-terminal image transmission system, including: the robot comprises a first terminal and a second terminal, wherein the first terminal is provided with a robot operating system, and the second terminal is provided with a preset deep learning network model;
and the data forwarding node on the second terminal stores the pictures from the data stream of the robot operating system to the exchange space in real time according to the picture dislocation storage mode, and a deep learning program of the preset deep learning network model reads and processes the pictures from the exchange space according to the picture dislocation reading mode.
The working method of the system is the same as that provided in embodiment 1, and is not described herein again.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure, and various modifications and changes may be made to the present disclosure by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.
Claims (6)
1. A cross-terminal picture transmission method is characterized in that: the method is applied to a first terminal and a second terminal, the first terminal is provided with a robot operating system, the second terminal is provided with a preset deep learning network model, and the method comprises the following steps:
the data forwarding node on the second terminal stores pictures from the data stream of the robot operating system to the exchange space in real time according to a picture dislocation storage mode, a deep learning program of a preset deep learning network model reads and processes the pictures from the exchange space according to a picture dislocation reading mode, and the picture dislocation storage mode and the picture dislocation reading mode dynamically maintain two pictures in the exchange space;
if only one picture currently exists in the exchange space, the data forwarding node captures and stores the picture into another missing picture from the real-time data stream of the robot operating system according to the picture dislocation storage mode, so as to realize picture updating;
in a staggered storage mode, only one picture in the first picture and the second picture is stored at the same time, and the other picture cannot be processed by the data forwarding node; the dislocation storage mode comprises the following specific steps:
in the operation process of the data forwarding node, firstly, retrieving the number and name of pictures in the exchange space at the current moment;
if the first picture and the second picture exist in the exchange space at the same time, the data forwarding node does not process the first picture and continues to circularly execute the first step; otherwise, executing downwards;
if no picture exists in the exchange space, the data forwarding node continuously stores the pictures in the two real-time data streams in the exchange space, stores the pictures as a first picture and a second picture in sequence, and continues to circularly execute the first step; otherwise, executing downwards;
if the number of the pictures is 1 and the current picture in the exchange space is the first picture, reading one picture in a real-time data stream of the robot operating system by the data forwarding node at the moment, and storing the picture as a second picture in the exchange space;
if the number of the pictures is 1 and the current picture in the exchange space is the second picture, the data forwarding node saves the picture as the first picture, and then the data forwarding node continues to circularly execute the first step, namely, the number of the pictures and the names of the pictures in the exchange space at the current moment are retrieved;
if two pictures currently exist in the exchange space, the deep learning program reads and processes one of the pictures according to the picture dislocation reading mode, different pictures are read and processed in two adjacent cycles, and the pictures are deleted after the deep learning network model processes one picture, so that preparation is made for receiving new pictures;
in the staggered reading mode, only one picture in the first picture and the second picture at the same time is read and processed, and the other picture cannot be processed by the deep learning program; and the data forwarding node and the deep learning program at the same moment are ensured not to process the same picture; the dislocation reading mode comprises the following specific steps:
in the running process of the deep learning program, firstly, retrieving the number and name of pictures in an exchange space at the current moment;
if the first picture and the second picture exist in the exchange space at the same time, the deep learning program executes the next step; otherwise, not processing, and continuously and circularly executing the first step;
if the picture is circularly read by running for the first time, deleting the first picture and the second picture in the exchange space by the deep learning program, and continuously and circularly executing the first step; otherwise, executing the next step downwards;
if the picture processed by the deep learning program last time is the first picture, reading a second picture in the exchange space; if the picture processed by the deep learning program last time is the second picture or the picture processed for the first time at the moment, reading the first picture in the exchange space, and then executing the next step downwards;
the deep learning model processes the read pictures, and then the next step is executed downwards;
and deleting the processed pictures in the exchange space by the deep learning program, and continuously and circularly executing the first step, namely retrieving the number and the names of the pictures in the exchange space at the current moment.
2. The cross-terminal picture transmission method according to claim 1, wherein:
when the data forwarding node and the deep learning program are started, the number of pictures in an exchange space is zero, the deep learning program does not read the pictures according to a staggered reading mode, the data forwarding node reads a real-time robot operating system data stream according to a staggered storage mode and sequentially writes a first picture and a second picture;
after the previous step is finished, a first picture and a second picture exist in the exchange space at the same time, the deep learning program detects the second picture but the second picture is not necessarily completely written, and the deep learning program reads the first picture according to a staggered reading mode and deletes the first picture after the processing is finished; if the data forwarding node does not finish writing the second picture, continuing writing, and after the writing is finished, waiting for the deep network model to finish processing the first picture;
at the moment, only a second picture exists in the exchange space, and the deep learning program does not read the picture according to a staggered reading mode; the data forwarding node reads a data stream of the real-time robot operating system according to the staggered storage mode and starts to write a first picture;
after the previous step is finished, a first picture and a second picture exist in the exchange space at the same time, at the moment, the deep learning program detects that the first picture is not necessarily completely written, and the deep learning program reads the second picture according to a staggered reading mode and deletes the second picture after the processing is finished; if the data forwarding node does not finish the writing of the first picture, continuing the writing, and after the writing is finished, waiting for the completion of the processing of the second picture by the deep network model;
after the previous step, only the first picture exists in the exchange space, and at the moment, the deep learning program does not read the picture according to the staggered reading mode; the data forwarding node reads the data stream of the real-time robot operating system according to the staggered storage mode and starts to write a second picture;
and returning to the step when the first picture and the second picture exist in the exchange space at the same time, and executing in a circulating way.
3. The cross-terminal picture transmission method according to claim 2, wherein:
the data forwarding node and the deep learning program independently run in parallel, the data forwarding node and the deep learning program are not related in two levels of running dependence and codes, and the deep learning program is configured with a corresponding Python environment at will.
4. The cross-terminal picture transmission method according to claim 2, wherein:
the data forwarding node is based on a Python2 environment in the robot operating system, all Python nodes in the robot operating system share the Python2 environment, the deep learning program is based on the Python3 environment, and the two environments are isolated from each other and do not affect each other.
5. A cross-terminal picture transmission system is characterized in that:
the method comprises the following steps: the robot comprises a first terminal and a second terminal, wherein the first terminal is provided with a robot operating system, and the second terminal is provided with a preset deep learning network model;
the data forwarding node on the second terminal stores pictures from the data stream of the robot operating system to the exchange space in real time according to a picture dislocation storage mode, a deep learning program of a preset deep learning network model reads and processes the pictures from the exchange space according to a picture dislocation reading mode, and the picture dislocation storage mode and the picture dislocation reading mode dynamically maintain two pictures in the exchange space;
if only one picture currently exists in the exchange space, the data forwarding node captures and stores the picture into another missing picture from the real-time data stream of the robot operating system according to the picture dislocation storage mode, so as to realize picture updating;
in a staggered storage mode, only one picture in the first picture and the second picture is stored at the same time, and the other picture cannot be processed by the data forwarding node; the dislocation storage mode comprises the following specific steps:
in the operation process of the data forwarding node, firstly, retrieving the number and name of pictures in the exchange space at the current moment;
if the first picture and the second picture exist in the exchange space at the same time, the data forwarding node does not process the first picture and continues to circularly execute the first step; otherwise, executing downwards;
if no picture exists in the exchange space, the data forwarding node continuously stores the pictures in the two real-time data streams in the exchange space, stores the pictures as a first picture and a second picture in sequence, and continues to circularly execute the first step; otherwise, executing downwards;
if the number of the pictures is 1 and the current picture in the exchange space is the first picture, reading one picture in a real-time data stream of the robot operating system by the data forwarding node at the moment, and storing the picture as a second picture in the exchange space;
if the number of the pictures is 1 and the current picture in the exchange space is the second picture, the data forwarding node saves the picture as the first picture, and then the data forwarding node continues to circularly execute the first step, namely, the number of the pictures and the names of the pictures in the exchange space at the current moment are retrieved;
if two pictures currently exist in the exchange space, the deep learning program reads and processes one of the pictures according to the picture dislocation reading mode, different pictures are read and processed in two adjacent cycles, and the pictures are deleted after the deep learning network model processes one picture, so that preparation is made for receiving new pictures;
in the staggered reading mode, only one picture in the first picture and the second picture at the same time is read and processed, and the other picture cannot be processed by the deep learning program; and the data forwarding node and the deep learning program at the same moment are ensured not to process the same picture; the dislocation reading mode comprises the following specific steps:
in the running process of the deep learning program, firstly, retrieving the number and name of pictures in an exchange space at the current moment;
if the first picture and the second picture exist in the exchange space at the same time, the deep learning program executes the next step; otherwise, not processing, and continuously and circularly executing the first step;
if the picture is circularly read by running for the first time, deleting the first picture and the second picture in the exchange space by the deep learning program, and continuously and circularly executing the first step; otherwise, executing the next step downwards;
if the picture processed by the deep learning program last time is the first picture, reading a second picture in the exchange space; if the picture processed by the deep learning program last time is the second picture or the picture processed for the first time at the moment, reading the first picture in the exchange space, and then executing the next step downwards;
the deep learning model processes the read pictures, and then the next step is executed downwards;
and deleting the processed pictures in the exchange space by the deep learning program, and continuously and circularly executing the first step, namely retrieving the number and the names of the pictures in the exchange space at the current moment.
6. The cross-terminal picture transfer system of claim 5, wherein:
when the data forwarding node and the deep learning program are started, the number of pictures in an exchange space is zero, the deep learning program does not read the pictures according to a staggered reading mode, the data forwarding node reads a real-time robot operating system data stream according to a staggered storage mode and sequentially writes a first picture and a second picture;
after the previous step is finished, a first picture and a second picture exist in the exchange space at the same time, the deep learning program detects the second picture but the second picture is not necessarily completely written, and the deep learning program reads the first picture according to a staggered reading mode and deletes the first picture after the processing is finished; if the data forwarding node does not finish writing the second picture, continuing writing, and after the writing is finished, waiting for the deep network model to finish processing the first picture;
at the moment, only a second picture exists in the exchange space, and the deep learning program does not read the picture according to a staggered reading mode; the data forwarding node reads a data stream of the real-time robot operating system according to the staggered storage mode and starts to write a first picture;
after the previous step is finished, a first picture and a second picture exist in the exchange space at the same time, at the moment, the deep learning program detects that the first picture is not necessarily completely written, and the deep learning program reads the second picture according to a staggered reading mode and deletes the second picture after the processing is finished; if the data forwarding node does not finish the writing of the first picture, continuing the writing, and after the writing is finished, waiting for the completion of the processing of the second picture by the deep network model;
after the previous step, only the first picture exists in the exchange space, and at the moment, the deep learning program does not read the picture according to the staggered reading mode; the data forwarding node reads the data stream of the real-time robot operating system according to the staggered storage mode and starts to write a second picture;
and returning to the step when the first picture and the second picture exist in the exchange space at the same time, and executing in a circulating way.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110205500.9A CN113157953B (en) | 2021-02-24 | 2021-02-24 | Cross-terminal picture transmission method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110205500.9A CN113157953B (en) | 2021-02-24 | 2021-02-24 | Cross-terminal picture transmission method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113157953A CN113157953A (en) | 2021-07-23 |
| CN113157953B true CN113157953B (en) | 2022-04-29 |
Family
ID=76883271
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110205500.9A Active CN113157953B (en) | 2021-02-24 | 2021-02-24 | Cross-terminal picture transmission method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113157953B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103176750A (en) * | 2013-02-27 | 2013-06-26 | 武汉虹旭信息技术有限责任公司 | Mobile internet data storage system based on interlaced time partition and method thereof |
| CN105282124A (en) * | 2014-07-24 | 2016-01-27 | 上海未来宽带技术股份有限公司 | Transmission method and presentation method of progressive picture based on XMPP |
| CN108710863A (en) * | 2018-05-24 | 2018-10-26 | 东北大学 | Unmanned plane Scene Semantics dividing method based on deep learning and system |
| CN108805798A (en) * | 2017-05-05 | 2018-11-13 | 英特尔公司 | Fine granularity for deep learning frame calculates communication and executes |
| CN109669640A (en) * | 2018-12-24 | 2019-04-23 | 浙江大华技术股份有限公司 | A kind of date storage method, device, electronic equipment and medium |
| CN110728368A (en) * | 2019-10-25 | 2020-01-24 | 中国人民解放军国防科技大学 | Acceleration method for deep reinforcement learning of simulation robot |
| CN111737255A (en) * | 2020-06-02 | 2020-10-02 | 通号城市轨道交通技术有限公司 | Method and system for storing interlocking monitoring data |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130262787A1 (en) * | 2012-03-28 | 2013-10-03 | Venugopal Santhanam | Scalable memory architecture for turbo encoding |
| US11354601B2 (en) * | 2018-03-16 | 2022-06-07 | Ricoh Company, Ltd. | Learning classification device and learning classification method |
-
2021
- 2021-02-24 CN CN202110205500.9A patent/CN113157953B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103176750A (en) * | 2013-02-27 | 2013-06-26 | 武汉虹旭信息技术有限责任公司 | Mobile internet data storage system based on interlaced time partition and method thereof |
| CN105282124A (en) * | 2014-07-24 | 2016-01-27 | 上海未来宽带技术股份有限公司 | Transmission method and presentation method of progressive picture based on XMPP |
| CN108805798A (en) * | 2017-05-05 | 2018-11-13 | 英特尔公司 | Fine granularity for deep learning frame calculates communication and executes |
| CN108710863A (en) * | 2018-05-24 | 2018-10-26 | 东北大学 | Unmanned plane Scene Semantics dividing method based on deep learning and system |
| CN109669640A (en) * | 2018-12-24 | 2019-04-23 | 浙江大华技术股份有限公司 | A kind of date storage method, device, electronic equipment and medium |
| CN110728368A (en) * | 2019-10-25 | 2020-01-24 | 中国人民解放军国防科技大学 | Acceleration method for deep reinforcement learning of simulation robot |
| CN111737255A (en) * | 2020-06-02 | 2020-10-02 | 通号城市轨道交通技术有限公司 | Method and system for storing interlocking monitoring data |
Non-Patent Citations (1)
| Title |
|---|
| Online Collision Avoidance for Human-Robot Collaborative Interaction Concerning Safety and Efficiency;Guoliang Liu等;《IEEE》;20200805;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113157953A (en) | 2021-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7583268B2 (en) | Graphics pipeline precise interrupt method and apparatus | |
| CN106790599B (en) | A kind of symbiosis virtual machine communication method based on multicore without lock buffer circle | |
| US7545381B2 (en) | Interruptible GPU and method for context saving and restoring | |
| CN101882092B (en) | Management method and device for application program | |
| US7580040B2 (en) | Interruptible GPU and method for processing multiple contexts and runlists | |
| CN108563517A (en) | The call method and device of system interface | |
| CN111274019B (en) | Data processing method, device and computer readable storage medium | |
| CN109819317A (en) | A kind of method for processing video frequency, device, terminal and storage medium | |
| CN113034629B (en) | Image processing method, image processing device, computer equipment and storage medium | |
| CN109032680A (en) | A kind of real time operating system quick start method and real time operating system | |
| TWI441091B (en) | Method for performing image signal processing with aid of a graphics processing unit and apparatus for performing image signal processing | |
| CN114500138A (en) | Linkage control configuration method, linkage control method and linkage control system | |
| CN111522620A (en) | Double-process efficient HMI design scheme of full liquid crystal instrument | |
| WO2024074012A1 (en) | Video transmission control method, apparatus and device, and nonvolatile readable storage medium | |
| CN111831353A (en) | OpenXR standard-based runtime library, data interaction method, device and medium | |
| CN113157953B (en) | Cross-terminal picture transmission method and system | |
| CN107122176B (en) | Graph drawing method and device | |
| US20150254116A1 (en) | Data processing apparatus for pipeline execution acceleration and method thereof | |
| CN113296979B (en) | Data communication method for unreal engine and external program | |
| CN114570020A (en) | Data processing method and system | |
| CN111310638B (en) | Data processing method, device and computer readable storage medium | |
| CN111198843B (en) | File system writing acceleration method based on bus control on application processor chip | |
| US8896610B2 (en) | Error recovery operations for a hardware accelerator | |
| WO2023124428A1 (en) | Chip, accelerator card, electronic device and data processing method | |
| WO2023124361A1 (en) | Chip, acceleration card, electronic device and data processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |