CN113808061A - Image processing method and device - Google Patents
Image processing method and device Download PDFInfo
- Publication number
- CN113808061A CN113808061A CN202111111394.4A CN202111111394A CN113808061A CN 113808061 A CN113808061 A CN 113808061A CN 202111111394 A CN202111111394 A CN 202111111394A CN 113808061 A CN113808061 A CN 113808061A
- Authority
- CN
- China
- Prior art keywords
- image
- color
- depth
- loss parameter
- sample image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 288
- 238000012549 training Methods 0.000 claims abstract description 104
- 238000000034 method Methods 0.000 claims abstract description 53
- 238000004590 computer program Methods 0.000 claims description 17
- 230000015654 memory Effects 0.000 claims description 16
- 230000000694 effects Effects 0.000 description 26
- 230000006870 function Effects 0.000 description 19
- 238000010586 diagram Methods 0.000 description 14
- 210000002569 neuron Anatomy 0.000 description 12
- 239000013598 vector Substances 0.000 description 7
- 239000012636 effector Substances 0.000 description 6
- 230000003631 expected effect Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000010606 normalization Methods 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 230000002787 reinforcement Effects 0.000 description 3
- 230000001186 cumulative effect Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000006386 memory function Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the application discloses an image processing method and device, wherein the method comprises the following steps: acquiring a training sample set, wherein the training sample set comprises at least one color sample image and at least one depth sample image corresponding to the color sample image; inputting the training sample set into an image processing model for processing; inputting the color sample image into a first coding module of a color image processing model, and outputting a coded first color sample image; inputting the depth sample image into a second coding module of a depth image processing model, and outputting a coded first depth sample image; calculating to obtain a second loss parameter by using the encoded first color sample image and the encoded first depth sample image; taking the second loss parameter as a first loss parameter; and adjusting the color image processing model based on the first loss parameter to obtain a trained color image processing model.
Description
The application is a divisional application with the application date of Chinese patent application No. 201910351289.4 of 2019, 04 and 28 months, and the invention name of the method and the device for processing the image.
Technical Field
The present application relates to robotics, and in particular, to an image processing method and apparatus.
Background
The research of robot learning mainly comprises two directions of supervised learning and reinforcement learning, wherein the supervised learning also comprises subdivision directions of imitation learning, self-supervised learning and the like. The simulation learning method trains the network model by collecting teaching information of experts, and has the disadvantages that a large amount of expert teaching information is needed, the assumed environment is known and limited, and the effect on an open and complex scene in reality is poor; the self-supervision learning collects the labeled data through a robot trial-and-error experiment, the success rate of the trial-and-error experiment is low, the collected data is very inefficient, and the trial-and-error in a real environment can cause safety risks. The reinforcement learning method learns to complete a specified task through a great amount of exploration of the robot, and has the defect that the number of required exploration times is very large and the reinforcement learning method is almost impossible to complete in reality.
In the existing robot learning process, a color image is directly used as a network model input, an encoder capable of well encoding an input RGB image is learned by generating a countermeasure network (GAN) model, and then the output of the encoder is input into a Long Short-Term Memory (LSTM) model, so that the network learning can complete a sequence of actions. The disadvantage of this method is that the amount of data required is still large, amounting to about 2500 expert teaching paradigms. The method is time-consuming in data acquisition, physical loss of the robot can occur, scenes and actions taught by experts are limited, and the robot cannot make proper prediction on unfamiliar scenes. Therefore, how to efficiently and quickly acquire data and how to train a network model with a good effect through a small amount of data is a technical problem to be solved urgently by the robot technology.
Disclosure of Invention
In order to solve the foregoing technical problems, embodiments of the present application are intended to provide an image processing method and apparatus.
The technical scheme of the application is realized as follows:
in a first aspect, an image processing method is provided, which includes: acquiring a training sample set, wherein the training sample set comprises at least one color sample image and at least one depth sample image corresponding to the color sample image;
inputting the training sample set into an image processing model for processing, wherein the image processing model comprises a color image processing model and a depth image processing model, the color image processing model is used for processing the color sample image to obtain a color reconstructed image, and the depth image processing model is used for processing the depth image to obtain a depth reconstructed image;
determining a first loss parameter of the image processing model based on the color sample image, the color reconstructed image, the depth sample image and the depth reconstructed image, and adjusting the color image processing model based on the first loss parameter to obtain a trained color image processing model.
Therefore, the color image and the depth image are simultaneously used as training samples to have the information complementation effect, the color image processing model with good effect can be trained by using fewer training samples, the training difficulty can be reduced by adopting a step-by-step training mode, and the training efficiency is improved.
In the above scheme, the method further comprises: and adjusting the depth image processing model based on the first loss parameter to obtain a trained depth image processing model.
Therefore, in the process of training the image processing model, the color image processing model and the depth image processing model can be trained simultaneously, the trained depth image processing model is used for monitoring and training the color image processing model, and the training effects of the color image processing model and the depth image processing model can be improved simultaneously.
In the foregoing solution, the determining a first loss parameter of the image processing model based on the color sample image, the color reconstructed image, the depth sample image and the depth reconstructed image includes: inputting the color sample image into a first coding module of the color image processing model, and outputting a coded first color sample image; inputting the depth sample image into a second coding module of the color image processing model, and outputting a coded first depth sample image; obtaining a first loss parameter based on the encoded first color sample image and the encoded first depth sample image.
In the foregoing solution, the color image processing model further includes a first decoding module, the depth image processing model further includes a second decoding module, and the obtaining a first loss parameter based on the encoded first color sample image and the encoded first depth sample image includes: inputting the encoded first color sample image to the first decoding module, and outputting the color reconstructed image; inputting the coded first depth sample image to the second decoding module, and outputting the depth reconstruction image; calculating to obtain a second loss parameter by using the encoded first color sample image and the encoded first depth sample image; determining a third loss parameter based on the encoded first color sample image and the color reconstructed image; determining a fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image; determining the first loss parameter based on the second loss parameter, the third loss parameter, and the fourth loss parameter.
In this way, the second loss function obtained by the encoded first color sample image and the encoded first depth sample image is used as a loss parameter of the image processing model, so that the encoded color image and the encoded depth image obtained in the image processing model can be ensured to be similar as much as possible, and the training effect is improved.
In the above scheme, the training sample set further includes: at least one status label of the robot corresponding to the color sample image;
the method further comprises the following steps: inputting the color sample image to a first coding module of the trained color image processing model, and outputting a coded second color sample image; determining a fifth loss parameter of a control model based on the encoded second color sample image and a status label of the robot; adjusting a control model based on the fifth loss parameter to obtain a trained control model;
alternatively, the method further comprises: inputting the color sample image to a first coding module of the trained color image processing model, and outputting a coded second color sample image; inputting the depth sample image to a second coding module of the trained depth image processing model, and outputting a coded second depth sample image; determining a fifth loss parameter of a control model based on the encoded second color sample image and the encoded second depth sample image, and a state label of the robot; and adjusting the control model based on the fifth loss parameter to obtain the trained control model.
Therefore, after the training control model obtains the predicted state information, whether the predicted state information is accurate is checked by using the state label, and the training effect of the control model can be improved by enabling the predicted state information to be the same as or similar to the state label.
In the above scheme, the status label at least includes one of the following: joint angle, joint linear velocity, and joint angular velocity.
In the above scheme, the method further comprises: when a robot is controlled to execute a target task on a target object, a target color image containing the target object is obtained; inputting the target color image to a first coding module of the trained color image processing model, and outputting a coded target color image; and inputting the coded target color image into the trained control model to obtain result state information of the robot, and controlling the robot based on the result state information.
Therefore, the scene information in the color image can be recognized by utilizing the image recognition capability of the trained color image processing model, the control operation which can be executed aiming at the current scene information is determined by the control capability of the trained control model, and the control difficulty is reduced by step-by-step operation.
In the above scheme, the method further comprises: when a robot is controlled to execute a target task on a target object, a target color image and a target depth image which contain the target object are obtained; inputting the target color image to a first coding module of the trained color image processing model, and outputting a coded target color image; inputting the target depth image into a second coding module of the trained depth image processing model, and outputting a coded target depth image; and inputting the coded target color image and the coded target depth image into the trained control model to obtain result state information of the robot, and controlling the robot based on the result state information.
Therefore, more comprehensive scene information can be obtained by utilizing the image recognition capability of the trained color image processing model and the image recognition capability of the trained depth image processing model, the trained control model can obtain more accurate control information by utilizing the information, and the control precision is improved.
In the foregoing scheme, the obtaining of the training sample set includes: and performing data enhancement processing on the collected color sample image and the collected depth sample image to increase the number of samples to obtain the training sample set.
Therefore, the data volume can be increased through data enhancement processing, network overfitting can be relieved, and model training efficiency is improved.
In a second aspect, an embodiment of the present application further provides an image processing apparatus, including:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a training sample set, and the training sample set comprises at least one color sample image and at least one depth sample image corresponding to the color sample image;
the processing unit is used for inputting the training sample set into an image processing model for processing, the image processing model comprises a color image processing model and a depth image processing model, the color image processing model is used for processing the color sample image to obtain a color reconstructed image, and the depth image processing model is used for processing the depth image to obtain a depth reconstructed image;
the processing unit is further configured to determine a first loss parameter of the image processing model based on the color sample image, the color reconstructed image, the depth sample image, and the depth reconstructed image, and adjust the color image processing model based on the first loss parameter to obtain a trained color image processing model.
Therefore, the color image and the depth image are simultaneously used as training samples to have the information complementation effect, the color image processing model with good effect can be trained by using fewer training samples, the training difficulty can be reduced by adopting a step-by-step training mode, and the training efficiency is improved.
In the foregoing scheme, the processing unit is further configured to adjust the depth image processing model based on the first loss parameter, so as to obtain a trained depth image processing model.
Therefore, in the process of training the image processing model, the color image processing model and the depth image processing model can be trained simultaneously, the trained depth image processing model is used for monitoring and training the color image processing model, and the training effects of the color image processing model and the depth image processing model can be improved simultaneously.
In the foregoing solution, the color image processing model includes a first encoding module, the depth image processing model includes a second encoding module, and the processing unit is specifically configured to input the color sample image into the first encoding module of the color image processing model, and output an encoded first color sample image; inputting the depth sample image into a second coding module of the color image processing model, and outputting a coded first depth sample image; obtaining a first loss parameter based on the encoded first color sample image and the encoded first depth sample image.
In the foregoing solution, the color image processing model further includes a first decoding module, the depth image processing model further includes a second decoding module, and the processing unit is specifically configured to input the encoded first color sample image to the first decoding module and output the color reconstructed image; inputting the coded first depth sample image to the second decoding module, and outputting the depth reconstruction image; calculating to obtain a second loss parameter by using the encoded first color sample image and the encoded first depth sample image; determining a third loss parameter based on the encoded first color sample image and the color reconstructed image; determining a fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image; determining the first loss parameter based on the second loss parameter, the third loss parameter, and the fourth loss parameter.
In this way, the second loss function obtained by the encoded first color sample image and the encoded first depth sample image is used as a loss parameter of the image processing model, so that the encoded color image and the encoded depth image obtained in the image processing model can be ensured to be similar as much as possible, and the training effect is improved.
In the above scheme, the training sample set further includes: at least one status label of the robot corresponding to the color sample image;
the processing unit is further configured to input the color sample image to a first encoding module of the trained color image processing model, and output an encoded second color sample image; determining a fifth loss parameter of a control model based on the encoded second color sample image and a status label of the robot; adjusting a control model based on the fifth loss parameter to obtain a trained control model;
or, the processing unit is further configured to input the color sample image to a first encoding module of the trained color image processing model, and output an encoded second color sample image; inputting the depth sample image to a second coding module of the trained depth image processing model, and outputting a coded second depth sample image; determining a fifth loss parameter of a control model based on the encoded second color sample image and the encoded second depth sample image, and a state label of the robot; and adjusting the control model based on the fifth loss parameter to obtain the trained control model.
Therefore, after the training control model obtains the predicted state information, whether the predicted state information is accurate is checked by using the state label, and the training effect of the control model can be improved by enabling the predicted state information to be the same as or similar to the state label.
In the above scheme, the status label at least includes one of the following: joint angle, joint linear velocity, and joint angular velocity.
In the above scheme, the obtaining unit is further configured to obtain a target color image including a target object when the robot executes a target task on the target object; the processing unit is further configured to input the target color image to a first encoding module of the trained color image processing model, and output an encoded target color image; and inputting the coded target color image into the trained control model to obtain result state information of the robot, and controlling the robot based on the result state information.
Therefore, the scene information in the color image can be recognized by utilizing the image recognition capability of the trained color image processing model, the control operation which can be executed aiming at the current scene information is determined by the control capability of the trained control model, and the control difficulty is reduced by step-by-step operation.
In the above scheme, the obtaining unit is further configured to obtain a target color image and a target depth image that include a target object when the robot executes a target task on the target object;
the processing unit is further configured to input the target color image to a first encoding module of the trained color image processing model, and output an encoded target color image; inputting the target depth image into a second coding module of the trained depth image processing model, and outputting a coded target depth image; and inputting the coded target color image and the coded target depth image into the trained control model to obtain result state information of the robot, and controlling the robot based on the result state information.
Therefore, more comprehensive scene information can be obtained by utilizing the image recognition capability of the trained color image processing model and the image recognition capability of the trained depth image processing model, the trained control model can obtain more accurate control information by utilizing the information, and the control precision is improved.
In the above scheme, the obtaining unit is specifically configured to perform data enhancement processing on the collected color sample image and the collected depth sample image to increase the number of samples, so as to obtain the training sample set.
Therefore, the data volume can be increased through data enhancement processing, network overfitting can be relieved, and model training efficiency is improved.
In a third aspect, this application provides another image processing apparatus, including a processor and a memory configured to store a computer program that is executable on the processor, where the processor is configured to execute the steps of any one of the foregoing methods when executing the computer program.
In a fourth aspect, the present application further provides a computer storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the steps of the foregoing method.
By adopting the technical scheme, the color sample image and the depth sample image are input into the image processing model, and the color image processing model is adjusted by utilizing the first loss parameter of the image processing model so as to obtain the trained color image processing model. Therefore, the color sample image and the depth sample image are simultaneously used as training samples to have the information complementation effect, so that a color image processing model with a good effect can be trained by using fewer training samples, the training difficulty is reduced, and the training efficiency is improved.
Drawings
FIG. 1 is a schematic flow chart diagram 1 of an image processing method in an embodiment of the present application;
FIG. 2 is a schematic diagram of a component structure of a data acquisition system according to an embodiment of the present application;
FIG. 3 is a schematic diagram illustrating a structure of an image processing model according to an embodiment of the present disclosure;
FIG. 4 is a schematic flowchart of an image processing method according to an embodiment of the present application, FIG. 2;
FIG. 5 is a flowchart illustrating an image processing method according to an embodiment of the present application, schematically illustrated in FIG. 3;
FIG. 6 is a flowchart illustrating an image processing method according to an embodiment of the present application in FIG. 4;
FIG. 7 is a schematic diagram of a component structure of a robot control system according to an embodiment of the present disclosure;
FIG. 8 is a schematic diagram of the structure of an image processing apparatus in the embodiment of the present application, in FIG. 1;
fig. 9 is a schematic diagram of a composition structure of an image processing apparatus according to an embodiment of the present application 2.
Detailed Description
So that the manner in which the features and elements of the present embodiments can be understood in detail, a more particular description of the embodiments, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings.
Example one
As shown in fig. 1, the image processing method may specifically include the following:
step 101: acquiring a training sample set, wherein the training sample set comprises at least one color sample image and at least one depth sample image corresponding to the color sample image;
step 102: inputting the training sample set into an image processing model for processing, wherein the image processing model comprises a color image processing model and a depth image processing model, the color image processing model is used for processing the color sample image to obtain a color reconstructed image, and the depth image processing model is used for processing the depth image to obtain a depth reconstructed image;
step 103: determining a first loss parameter of the image processing model based on the color sample image, the color reconstructed image, the depth sample image and the depth reconstructed image, and adjusting the color image processing model based on the first loss parameter to obtain a trained color image processing model.
Here, the execution subject of steps 101 to 103 may be a processor of the image processing apparatus.
In practical application, when the robot is controlled to learn, at least one network model of the robot needs to be trained by using sample data, and at the moment, a trainer controls the robot to execute at least one historical target task aiming at a target object and collects the sample data. Here, the sample data includes a color sample image and a depth sample image corresponding thereto acquired by the image acquisition device, and a training sample set is created from these color sample image and depth sample image.
For example, the image capturing device may include a camera for capturing a color image and a camera for capturing a depth image. For example, color and depth images of the target object are acquired using the Kinect. Here, the color image may include information of color, texture, and the like of the target object, and the pixel value of each pixel point in the depth image represents the distance from the sensor to the physical object.
As shown in fig. 2, the trainer controls the robot to complete a certain task through the controller. In the process, the data acquisition System records the color image and the depth image of the camera and the state information of the Robot at each moment through a Robot Operating System (ROS). The information that the data acquisition system can acquire includes: color images and depth images of cameras (such as Kinect V2) of different models are adjustable in image resolution; status information of different model robots, such as UR robots. Here, the state information may include joint angles, joint linear velocities, joint angular velocities, robot loads, end effector positions and poses, and the like.
In practical applications, the acquiring the training sample set includes: and performing data enhancement processing on the collected color sample image and the collected depth sample image to increase the number of samples to obtain the training sample set.
In order to increase the data volume and relieve the network overfitting, the acquired color sample images and depth sample images are enhanced, more training samples can be obtained, a plurality of images can be obtained on the basis of one image, the number of samples is increased, and the model training efficiency can be improved. The enhancement treatment comprises the following steps: and carrying out preprocessing such as random clipping, horizontal turning, size scaling, tone adjustment, brightness adjustment, saturation adjustment and the like on the given sample image.
In practical applications, the color image processing model includes a first encoding module, the depth image processing model includes a second encoding module, and the determining a first loss parameter of the image processing model based on the color sample image, the color reconstructed image, the depth sample image and the depth reconstructed image includes: inputting the color sample image into a first coding module of the color image processing model, and outputting a coded first color sample image; inputting the depth sample image into a second coding module of the color image processing model, and outputting a coded first depth sample image; obtaining a first loss parameter based on the encoded first color sample image and the encoded first depth sample image.
Here, the first loss parameter is a basis for measuring the training effect of the image processing model. After the training sample set is input into the image processing model, the coded color image and the coded depth image are obtained, and the coded color image and the coded depth image are similar as much as possible, so that the color image processing model with good effect can be obtained.
Here, the color sample image input to the color image processing model and the depth sample image input to the depth image processing model are image information acquired for the same scene at the same time. The color sample image and the depth sample image at the same moment carry more comprehensive scene information, the color sample image and the depth sample image have an information complementary effect, the feature map obtained after the color sample image is coded is similar to the feature map obtained after the depth sample image is coded as much as possible, the scene information can be better learned, and an image processing model with a good effect is trained by less data. According to the embodiment of the application, the depth image processing model can be used for monitoring and training the color image processing model in the training process, so that the training efficiency and the training effect of the color image processing model are improved.
In some embodiments, the color image processing model further comprises a first decoding module, the depth image processing model further comprises a second decoding module, and the deriving the first loss parameter based on the encoded first color sample image and the encoded first depth sample image comprises: inputting the encoded first color sample image to the first decoding module, and outputting the color reconstructed image; inputting the coded first depth sample image to the second decoding module, and outputting the depth reconstruction image; calculating to obtain a second loss parameter by using the encoded first color sample image and the encoded first depth sample image; determining a third loss parameter based on the encoded first color sample image and the color reconstructed image; determining a fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image; determining the first loss parameter based on the second loss parameter, the third loss parameter, and the fourth loss parameter.
Illustratively, a cumulative sum of the second loss parameter, the third loss parameter, and the fourth loss parameter is calculated, and the cumulative sum result is taken as the first loss parameter.
As shown in fig. 3, the image processing model includes a color image processing model 31 and a depth image processing model 32, wherein the color image processing model 31 includes a first encoding module 311 and a first decoding module 312, and the depth image processing model 32 includes a second encoding module 321 and a second decoding module 322; the color sample image sequentially passes through the first coding module and the first decoding module to obtain a color reconstruction image, the depth sample image sequentially passes through the second coding module and the second decoding module to obtain a depth reconstruction image, the coded color image output by the first coding module 311 and the coded depth image output by the second coding module 321 are compared to obtain a second loss parameter (loss2), the color sample image and the color reconstruction image are compared to obtain a third loss parameter (loss3), the depth sample image and the depth reconstruction image are compared to obtain a fourth loss parameter (loss4), and the loss2, the loss3 and the loss4 are summed to obtain a first loss parameter (loss 1).
Here, parameters of the color image processing model and the depth image processing model in the image processing model are initialized at random, and the initialized image processing model is trained.
In still other embodiments, the deriving a first loss parameter based on the encoded first color sample image and the encoded first depth sample image includes: calculating to obtain a second loss parameter by using the encoded first color sample image and the encoded first depth sample image; taking the second loss parameter as the first loss parameter;
or, determining the first loss parameter based on the second loss parameter and the third loss parameter;
alternatively, the first loss parameter is determined based on the second loss parameter and the fourth loss parameter.
In practical application, the adjusting the color image processing model based on the first loss parameter to obtain a trained color image processing model includes: and under the condition that the first loss parameter is larger than a first target loss parameter, adjusting the color image processing model to obtain a trained color image processing model.
That is to say, when the color image processing model is trained, the first target loss parameter is used as a basis for completing the color image processing model training, when the first loss parameter is greater than the first target loss parameter, it is indicated that the current color image processing model does not achieve the expected effect, it is determined that the training fails, the model parameter needs to be continuously adjusted, and a new first loss parameter is obtained after the model parameter is adjusted; and when the first loss parameter is less than or equal to the first target loss parameter, determining that the model training is finished, and fixing the model parameter.
In some embodiments, the method further comprises: and adjusting the depth image processing model based on the first loss parameter to obtain a trained depth image processing model.
That is, the color image processing model and the depth image processing model are adjusted simultaneously based on the first loss parameter. Similarly, the first target loss parameter is used as a basis for completing the training of the depth image processing model, when the first loss parameter is greater than the first target loss parameter, the fact that the current depth image processing model does not achieve the expected effect is indicated, the training failure is determined, the model parameter needs to be continuously adjusted, and a new first loss parameter is obtained after the model parameter is adjusted; and when the first loss parameter is less than or equal to the first target loss parameter, determining that the model training is finished, and fixing the model parameter.
Here, the method of adjusting the color image processing model and the depth image processing model may be gradient descent, random gradient descent, mini-batch.
In practical application, after the trained color image processing model is obtained, the method further comprises a training control model, the process of training the control model can be regarded as a robot testing process, whether the trained image processing model obtained in the learning process can enable the robot to execute correct control operation or not is detected, and the robot can be put into production and life only when the learning and testing achieve the expected effect.
In some embodiments, the training sample set further comprises: at least one status label of the robot corresponding to the color sample image; the method for training the control model comprises the following steps: inputting the color sample image to a first coding module of the trained color image processing model, and outputting a coded second color sample image; determining a fifth loss parameter of a control model based on the encoded second color sample image and a status label of the robot; and adjusting the control model based on the fifth loss parameter to obtain the trained control model.
Here, the fifth loss parameter is a basis for characterizing the control model training effect. After the trained color image processing model is obtained, the robot needs to be tested, the control model obtains predicted state information according to the coded second color sample image, and whether the preset state information is accurate or not is judged according to the state label corresponding to the color sample image. The control model with good effect can be obtained only by making the predicted state information the same as or similar to the state label.
Alternatively, the fifth loss parameter is used to characterize an error between the predicted state information and the state label, and the error between the predicted state information and the state label may be: joint angle errors, robot velocity errors, end effector position errors, or other state information errors. The second target loss parameter may be an error threshold, which is specifically set for different status information, such as: the corresponding error threshold value of the joint angle error is plus or minus 2 degrees, the corresponding error threshold value of the robot speed error is 5 meters/minute, and the corresponding error threshold value of the end effector position error is 2 millimeters.
The error may specifically be a mean square error between the predicted state information and the state label, and a calculation formula of the mean square error is used as a loss function of the control model.
Further, the determining a fifth loss parameter of a control model based on the encoded second color sample image and the status label of the robot comprises: inputting the encoded second color sample image to the control model to obtain the predicted state information of the robot; determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.
In other embodiments, the method of training a control model comprises: inputting the color sample image to a first coding module of the trained color image processing model, and outputting a coded second color sample image; inputting the depth sample image to a second coding module of the trained depth image processing model, and outputting a coded second depth sample image; determining a fifth loss parameter of a control model based on the encoded second color sample image and the encoded second depth sample image, and a state label of the robot; and adjusting the control model based on the fifth loss parameter to obtain the trained control model.
Further, the determining a fifth loss parameter of a control model based on the encoded second color sample image and the encoded second depth sample image, and the state label of the robot includes: inputting the coded second color sample image and the coded second depth sample image into the control model to obtain the predicted state information of the robot; determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.
In other embodiments, the method of training a control model comprises: inputting the depth sample image to a second coding module of the trained depth image processing model, and outputting a coded second depth sample image; determining a fifth loss parameter of a control model based on the encoded second depth sample image and a state label of the robot; and adjusting the control model based on the fifth loss parameter to obtain the trained control model.
Further, the determining a fifth loss parameter of a control model based on the encoded second depth sample image and the state label of the robot comprises: inputting the coded second depth sample image into the control model to obtain the predicted state information of the robot; determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.
That is, when training the control model, the control model may be trained using only the encoded second color sample image, or may be trained using the encoded second color sample image and the encoded depth sample image, or may be trained using only the encoded second depth sample image.
Based on fifth loss parameter adjustment control model obtains the control model after the training, include: and under the condition that the fifth loss parameter is greater than a second target loss parameter, adjusting the control model to enable the fifth loss parameter corresponding to the trained control model not to be greater than the second target loss parameter.
That is to say, when the control model is trained, the second target loss parameter is used as a basis for completing the training of the control model, and when the fifth loss parameter is greater than the second target loss parameter, it is indicated that the current control model does not achieve the expected effect, it is determined that the training fails, the model parameter needs to be continuously adjusted, and a new fifth loss parameter is obtained after the model parameter is adjusted; and when the fifth loss parameter is less than or equal to the second target loss parameter, determining that the model training is finished, and fixing the model parameter.
In practical applications, the status label includes at least one of the following: joint angle, joint linear velocity, and joint angular velocity.
Accordingly, the predicted state information includes at least one of: joint angle, joint linear velocity, and joint angular velocity. The resulting status information includes at least one of: joint angle, joint linear velocity, and joint angular velocity.
Here, the control model may be adjusted by a gradient descent, a random gradient descent, a mini-batch.
By adopting the technical scheme, the color sample image and the depth sample image are input into the image processing model, and the color image processing model is adjusted by utilizing the first loss parameter of the image processing model so as to obtain the trained color image processing model. Therefore, the color sample image and the depth sample image are simultaneously used as training samples to have the information complementation effect, so that a color image processing model with a good effect can be trained by using fewer training samples, the training difficulty is reduced, and the training efficiency is improved.
Example two
To further illustrate the object of the present application, based on the first embodiment of the present application, as shown in fig. 4, after obtaining the trained color image processing model, the image processing method further includes:
step 401: inputting the color sample image to a first coding module of the trained color image processing model, and outputting a coded second color sample image;
here, the trained color image processing model also includes: the device comprises a first encoding module and a first decoding module.
Step 402: determining a fifth loss parameter of a control model based on the encoded second color sample image and a status label of the robot;
in practical applications, the training sample set further includes: at least one status label of the robot corresponding to the color sample image.
Specifically, the image acquisition device acquires the historical state information of the robot while acquiring a color sample image and a depth sample image containing the target object, and the historical state information is used as a state label.
The method for determining the fifth loss parameter may specifically include: inputting the encoded second color sample image to the control model to obtain the predicted state information of the robot; determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.
In some embodiments, when the control model includes only the first control model, the encoded second color sample image is used as input information of the control model, and the predicted state information of the robot is output. The predicted state information may include joint angles, joint linear velocities, joint angular velocities, robot loads, end effector positions and poses, and the like.
In some embodiments, when the control model includes at least two control models, such as a first control model and a second control model; and taking the coded second color sample image as input information of the second control model, taking output information of the second control model as input information of the first control model, and outputting prediction state information. For example, the second control model is an LSTM model with a memory function, and the first control model is a Mixed Density Network (MDN) model.
Step 403: adjusting a control model based on the fifth loss parameter to obtain a trained control model;
specifically, under the condition that the fifth loss parameter is greater than the second target loss parameter, the control model is adjusted to make the fifth loss parameter corresponding to the trained control model not greater than the second target loss parameter.
Step 404: when a robot is controlled to execute a target task on a target object, a target color image containing the target object is obtained;
specifically, when a robot test is performed, the robot is controlled to perform a target task on a target object (for example, the robot arm is controlled to perform a grabbing operation), a target color image is collected by the color camera, and then, result state information for the target object is determined according to target object information carried in the target color image.
Step 405: inputting the target color image to a first coding module of the trained color image processing model, and outputting a coded target color image;
step 406: and inputting the coded target color image into the trained control model to obtain result state information of the robot, and controlling the robot based on the result state information.
That is, when the robot is controlled to perform an action, the result state information of the robot is determined using the trained color image processing model and the control model. For example, the resulting state information may include angle information of 6 joints of the robot, and the operation of the robot is controlled based on the state control information.
EXAMPLE III
To further illustrate the object of the present application, based on the first embodiment of the present application, as shown in fig. 5, after obtaining the trained color image processing model and the trained control model, the image processing method further includes:
step 501: inputting the color sample image to a first coding module of the trained color image processing model, and outputting a coded second color sample image;
here, the trained color image processing model also includes: the device comprises a first encoding module and a first decoding module.
Step 502: inputting the depth sample image to a second coding module of the trained depth image processing model, and outputting a coded second depth sample image;
here, the trained depth image processing model also includes: a second encoding module and a second decoding module.
In practical applications, the first encoding module and the second encoding module may be encoders, and the first decoding module and the second decoding module may be decoders.
Step 503: determining a fifth loss parameter of a control model based on the encoded second color sample image and the encoded second depth sample image, and a state label of the robot;
in practical applications, the training sample set further includes: at least one status label of the robot corresponding to the color sample image.
Specifically, the image acquisition device acquires the historical state information of the robot while acquiring a color sample image and a depth sample image containing the target object, and the historical state information is used as a state label.
The method for determining the fifth loss parameter may specifically include: inputting the coded second color sample image and the coded second depth sample image into the control model to obtain the predicted state information of the robot; determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.
In some embodiments, when the control model includes only the first control model, the encoded second color sample image is used as input information of the control model, and the predicted state information of the robot is output. The predicted state information may include joint angles, joint linear velocities, joint angular velocities, robot loads, end effector positions and poses, and the like.
In some embodiments, when the control model includes at least two control models, such as a first control model and a second control model; and taking the coded second color sample image as input information of the second control model, taking output information of the second control model as input information of the first control model, and outputting prediction state information. For example, the second control model is an LSTM model with a memory function, and the first control model is an MDN model.
Step 504: adjusting a control model based on the fifth loss parameter to obtain a trained control model;
specifically, under the condition that the fifth loss parameter is greater than the second target loss parameter, the control model is adjusted to make the fifth loss parameter corresponding to the trained control model not greater than the second target loss parameter.
Step 505: when a robot is controlled to execute a target task on a target object, a target color image and a target depth image which contain the target object are obtained;
specifically, when a robot test is performed, the robot is controlled to perform a target task on a target object (for example, the robot arm is controlled to perform a grabbing operation), a color camera acquires a target color image, a depth camera acquires a target depth image, and then, result state information for the target object is determined according to target object information carried in the color image and the depth image.
Step 506: inputting the target color image to a first coding module of the trained color image processing model, and outputting a coded target color image; inputting the target depth image into a second coding module of the trained depth image processing model, and outputting a coded target depth image;
step 507: and inputting the coded target color image and the coded target depth image into the trained control model to obtain result state information of the robot, and controlling the robot based on the result state information.
That is, when the robot is controlled to perform an action, the result state information of the robot may be determined using the trained color image processing model, depth image processing model, and control model. For example, the resulting state information may include angle information of 6 joints of the robot, and the operation of the robot is controlled based on the state control information.
In other embodiments, the trained depth image processing model and control model may also be used to determine the resulting state information of the robot.
Based on the above embodiment, an application scenario is given, in which an image processing model is a Variational Auto Encoder (VAE) model, a first control model is an MDN model, and a second control model is an LSTM model; as shown in fig. 6, the image processing method includes:
step 601: collecting data;
as shown in fig. 2, the human teach pendant controls the robot to perform a task through the controller. In the process, the data acquisition System records image information of the camera and state information of the Robot at each moment through a Robot Operating System (ROS). The information that the data acquisition system can acquire includes: color images and depth images acquired by cameras (such as Kinect V2) of different models are adjustable in image resolution; status information of different models of robots (such as UR robots), which may include joint angles, joint linear velocities, joint angular velocities, robot loads, end effector positions and poses, etc.
Step 602: preprocessing data;
in order to increase the data volume and relieve the network overfitting, the acquired image data is subjected to enhancement processing to increase the number and diversity of samples. The enhancement treatment comprises the following steps: and carrying out preprocessing such as random clipping, horizontal turning, size scaling, tone adjustment, brightness adjustment, saturation adjustment and the like on the given sample image. For example, the original color image size is 540 × 540, and the original color image size can be randomly cut into 480 × 480 sizes, randomly rotated by 1 to 3 °, and randomly inverted in the left-right direction. This method of data enhancement can increase the amount of data by tens of times.
Step 603: training an image processing model;
in the embodiment of the application, a double-stream (two-stream) network structure for cross-modal information fusion is adopted, information of multiple modalities such as color images, depth images and robot state information is fused, a feature map obtained by color image coding is similar to a feature map obtained by depth image coding in the double-stream network, scene information can be better learned, and a depth neural network with a good effect is trained by less data.
As shown in fig. 7, the robot control system consists of a two-stream network comprising a first tributary with color images as input and a second tributary with depth images as input. The first tributary comprises an encoder 1 and a decoder 1 and the second tributary comprises an encoder 2 and a decoder 2. The encoder can encode the image, compressing the image into a shorter vector that can implicitly represent important information in the image, such as the position and pose of the mechanical arm, the color, shape and position of the object, etc. The vector is sampled and used as the input of a decoder, the decoder decodes the vector into a reconstructed image, and the reconstructed image is compared with the sample image, so that the effect of VAE network training can be observed visually.
Illustratively, encoder 1 and encoder 2 each consist of 4 convolutional layers, each followed by Batch Normalization. The convolutional layer uses 4 x 4 convolutional kernels with a step size of 2. Decoder 1 and decoder 2 consist of 4 deconvolution layers, each followed by Batch Normalization. The deconvolution layer used 4 x 4 convolution kernels with a step size of 2. And the last deconvolution layer uses tanh as an activation function, the rest layers use a leakage _ relu function as the activation function, and a decoder performs deconvolution processing on the image and then performs image reconstruction to obtain a reconstructed image. The activation function may also be ReLU, Sigmoids, softmax, etc. Optionally, before the image data is input to the VAE network, normalization processing is performed on the image data, so that image pixels are reduced in an equal proportion, the difficulty in processing computer data is reduced, and the data processing speed is increased.
The embodiment of the present application uses two encoders and two decoders, wherein the encoder 1 and the decoder 1 are used for encoding and decoding color images, and the encoder 2 and the decoder 2 are used for decoding and encoding depth images. The encoder 1 encodes the color image and outputs the encoded color image (i.e., the first characteristic diagram) denoted as F1(ii) a The encoder 2 encodes the depth image and outputs the encoded depth image (i.e., the second feature map) as F2. Because the input color image and the input depth image correspond to the same scene, the characteristic images output by the two encoders are similar as much as possible, and information of different modes can be mutually supervised to realize better learning of the scene, so that a good network is trained by using less data.
The mean square error between the two profiles will be the first loss function of the VAE network:
wherein the first loss function is used to calculate a second loss parameter, L1Is the mean square error (i.e., the second loss parameter) of the first profile and the second profile, Fi 1Is the ith vector, F, in the first feature mapi 2Is the ith vector in the second feature map, and N is the total number of vectors in the feature map. In fig. 7, mse loss2 is used to calculate the second loss parameter, and the mse loss2 is input into the first characteristic diagram and the second characteristic diagram, and the output is the second loss parameter.
The first reconstructed image output by the decoder 1 is compared with the color sample image, and the mean square error between the two images is taken as the second loss function of the VAE network:
wherein the second loss function is used to calculate a third loss parameter, L2Is a color sample imageAnd a first reconstructed imageThe mean square error of (i.e. the third loss parameter),is the data of the ith row and jth column pixel point of the color sample image,the data of the ith row and the jth column pixel point of the first reconstructed image is obtained, H is the row number of the image pixel point, and W is the column number of the image pixel point. In fig. 7, mse loss3 is used to calculate the third loss parameter, and mse loss3 is input as the color sample image and the first reconstructed image, and output as the third loss parameter.
The second reconstructed image output by the decoder 2 is compared with the depth sample image of the depth image, and the mean square error between the two images is taken as a third loss function of the VAE network:
wherein the third loss function is used to calculate a fourth loss parameter, L3Is a depth sample imageAnd a second reconstructed imageThe mean square error of (i.e. the fourth loss parameter),the data of the ith row and jth column pixel point of the original image,the data of the ith row and the jth column pixel point of the second reconstructed image is obtained, H is the row number of the image pixel point, and W is the column number of the image pixel point. In fig. 7, mse loss4 is used to calculate the fourth loss parameter, and mse loss4 is input as the depth sample image and the second reconstructed image, and output as the fourth loss parameter.
The first loss parameter L may be calculated as:
L=L1+L2+L3
in fig. 7, VAE loss1 is used to calculate the first loss parameter, and VAE loss1 is input as the second loss parameter, the third loss parameter, and the fourth loss parameter, and output is the first loss parameter.
Step 604: training a control model;
then, the vector z output by the encoder 1 is sampled and input into an LSTM model of a control network, so that the robot learns how to complete a sequence of actions. The application uses a layer of LSTM, which consists of 100 LSTM neurons, each LSTM neuron comprising a forgetting gate, an input gate, and an output gate.
And finally, inputting the output information of the LSTM network into the MDN model, and outputting the predicted joint angle of each joint of the robot after sampling. The MDN model is suitable for solving multi-solution problems, which are widely existing in the field of robots. The method uses 360 neurons to form an MDN layer to represent angle information of 6 joints of the robot, and every 60 neurons represent related information of one joint angle: 20 neurons represent the mean μ of the joint, 20 neurons represent the variance σ of the joint, and 20 neurons represent the mixing coefficient α of the joint.
The loss function of an MDN network is defined as:
wherein x is a predicted joint angle (i.e., joint output) output by the MDN, y is a historical joint angle (i.e., joint label), m is the number of neurons, and c is the number of degrees of freedom of the mechanical arm. For example, every 60 neurons represent information about a joint angle: 20 neurons represent the mean of the joint, 20 neurons represent the variance of the joint, 20 neurons represent the mixing coefficient of the joint, m is 20, in UR5 manipulator, c is 6.
When in training setting, firstly, the historical image information set is used for training the VAE model, after the VAE model is trained, the parameters of the VAE model are fixed, and the historical color image and the robot state information are used for training the LSTM model and the MDN model. The three models were all gradient-descent using an Adam optimizer, the learning rate was set to 10e-4, the mini-batch size was 64, and all parameters were initialized randomly.
Step 605: a control operation is performed.
Here, the trained image processing model and the control model are used to perform control operations, that is, image information including the target object is collected, the image information may be a color image or a depth image, the image information encoded by the image processing model is used as input information of the control model, the control model outputs state control information for at least one time of the target object, and the operation is performed on the target object based on the state control information for at least one time. For example, when the target object is an object to be grasped by the robot, since the grasping operation is a series of operations, it is necessary to specify state information of the robot at a plurality of times to realize a continuous control operation.
Here, the state information of at least one time may be state information of the robot at the final time or state information corresponding to the predetermined moving path at the equal interval time.
Example four
Based on the same inventive concept, an embodiment of the present application further provides an image processing apparatus, as shown in fig. 8, the apparatus including:
an obtaining unit 801, configured to obtain a training sample set, where the training sample set includes at least one color sample image and at least one depth sample image corresponding to the color sample image;
a processing unit 802, configured to input the training sample set into an image processing model for processing, where the image processing model includes a color image processing model and a depth image processing model, the color image processing model is used to process the color sample image to obtain a color reconstructed image, and the depth image processing model is used to process the depth image to obtain a depth reconstructed image;
the processing unit 802 is further configured to determine a first loss parameter of the image processing model based on the color sample image, the color reconstructed image, the depth sample image, and the depth reconstructed image, and adjust the color image processing model based on the first loss parameter to obtain a trained color image processing model.
In some embodiments, the processing unit is further configured to adjust the depth image processing model based on the first loss parameter, so as to obtain a trained depth image processing model.
In some embodiments, the color image processing model comprises a first encoding module, the depth image processing model comprises a second encoding module, and the processing unit, in particular, is configured to input the color sample image into the first encoding module of the color image processing model, and output an encoded first color sample image; inputting the depth sample image into a second coding module of the color image processing model, and outputting a coded first depth sample image; obtaining a first loss parameter based on the encoded first color sample image and the encoded first depth sample image.
In some embodiments, the color image processing model further comprises a first decoding module, the depth image processing model further comprises a second decoding module,
the processing unit is specifically configured to input the encoded first color sample image to the first decoding module, and output the color reconstructed image;
inputting the coded first depth sample image to the second decoding module, and outputting the depth reconstruction image;
calculating to obtain a second loss parameter by using the encoded first color sample image and the encoded first depth sample image;
determining a third loss parameter based on the encoded first color sample image and the color reconstructed image;
determining a fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image;
determining the first loss parameter based on the second loss parameter, the third loss parameter, and the fourth loss parameter.
In some embodiments, the training sample set further comprises: at least one status label of the robot corresponding to the color sample image;
the processing unit is further configured to input the color sample image to a first encoding module of the trained color image processing model, and output an encoded second color sample image; determining a fifth loss parameter of a control model based on the encoded second color sample image and a status label of the robot; adjusting a control model based on the fifth loss parameter to obtain a trained control model;
or, the processing unit is further configured to input the color sample image to a first encoding module of the trained color image processing model, and output an encoded second color sample image; inputting the depth sample image to a second coding module of the trained depth image processing model, and outputting a coded second depth sample image; determining a fifth loss parameter of a control model based on the encoded second color sample image and the encoded second depth sample image, and a state label of the robot; and adjusting the control model based on the fifth loss parameter to obtain the trained control model.
In some embodiments, the status tag includes at least one of: joint angle, joint linear velocity, and joint angular velocity.
In some embodiments, the acquiring unit is further configured to acquire a target color image including a target object when the robot is controlled to perform a target task on the target object;
the processing unit is further configured to input the target color image to a first encoding module of the trained color image processing model, and output an encoded target color image; and inputting the coded target color image into the trained control model to obtain result state information of the robot, and controlling the robot based on the result state information.
In some embodiments, the acquiring unit is further configured to acquire a target color image and a target depth image including a target object when the robot is controlled to perform a target task on the target object;
the processing unit is further configured to input the target color image to a first encoding module of the trained color image processing model, and output an encoded target color image; inputting the target depth image into a second coding module of the trained depth image processing model, and outputting a coded target depth image; and inputting the coded target color image and the coded target depth image into the trained control model to obtain result state information of the robot, and controlling the robot based on the result state information.
In some embodiments, the obtaining unit is specifically configured to perform data enhancement processing on the acquired color sample image and the acquired depth sample image to increase the number of samples, so as to obtain the training sample set.
By adopting the technical scheme, the color sample image and the depth sample image are input into the image processing model, and the color image processing model is adjusted by utilizing the first loss parameter of the image processing model so as to obtain the trained color image processing model. Therefore, the color sample image and the depth sample image are simultaneously used as training samples to have the information complementation effect, so that a color image processing model with a good effect can be trained by using fewer training samples, the training difficulty is reduced, and the training efficiency is improved.
An embodiment of the present application further provides another image processing apparatus, as shown in fig. 9, the apparatus includes: a processor 901 and a memory 902 configured to store a computer program capable of running on the processor;
wherein the processor 901 is configured to execute the method steps in the previous embodiments when running the computer program.
Of course, in actual use, as shown in fig. 9, the respective components in the image processing apparatus are coupled together by a bus system 903. It is understood that the bus system 903 is used to enable communications among the components. The bus system 903 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as the bus system 903 in FIG. 9.
In practical applications, the processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, and a microprocessor. It is understood that the electronic devices for implementing the above processor functions may be other devices, and the embodiments of the present application are not limited in particular.
The Memory may be a volatile Memory (volatile Memory), such as a Random-Access Memory (RAM); or a non-volatile Memory (non-volatile Memory), such as a Read-Only Memory (ROM), a flash Memory (flash Memory), a Hard Disk (HDD), or a Solid-State Drive (SSD); or a combination of the above types of memories and provides instructions and data to the processor.
In an exemplary embodiment, the present application further provides a computer readable storage medium, such as a memory 902, comprising a computer program, which is executable by a processor 901 of an image processing apparatus to perform the foregoing method steps.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart block or blocks and/or flowchart block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks and/or flowchart block or blocks.
The above description is only a preferred embodiment of the present application, and is not intended to limit the scope of the present application.
Claims (12)
1. An image processing method, characterized in that the method comprises:
acquiring a training sample set, wherein the training sample set comprises at least one color sample image and at least one depth sample image corresponding to the color sample image;
inputting the training sample set into an image processing model for processing, wherein the image processing model comprises a color image processing model and a depth image processing model, the color image processing model is used for processing the color sample image to obtain a color reconstructed image, and the depth image processing model is used for processing the depth image to obtain a depth reconstructed image;
inputting the color sample image into a first coding module of the color image processing model, and outputting a coded first color sample image;
inputting the depth sample image into a second coding module of the depth image processing model, and outputting a coded first depth sample image;
calculating to obtain a second loss parameter by using the encoded first color sample image and the encoded first depth sample image;
taking the second loss parameter as a first loss parameter; or,
determining a third loss parameter based on the encoded first color sample image and the color reconstructed image; determining the first loss parameter based on the second loss parameter and the third loss parameter; in the alternative to this, either,
determining a third loss parameter based on the encoded first color sample image and the color reconstructed image; determining a fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image; determining the first loss parameter based on the second loss parameter and the fourth loss parameter;
and adjusting the color image processing model based on the first loss parameter to obtain a trained color image processing model.
2. The method of claim 1, further comprising: and adjusting the depth image processing model based on the first loss parameter to obtain a trained depth image processing model.
3. The method of claim 1, wherein the training sample set further comprises: at least one status label of the robot corresponding to the color sample image, the method further comprising:
inputting the color sample image to a first coding module of the trained color image processing model, and outputting a coded second color sample image;
determining a fifth loss parameter of a control model based on the encoded second color sample image and a status label of the robot;
adjusting a control model based on the fifth loss parameter to obtain a trained control model;
alternatively, the method further comprises:
inputting the color sample image to a first coding module of the trained color image processing model, and outputting a coded second color sample image;
inputting the depth sample image to a second coding module of the trained depth image processing model, and outputting a coded second depth sample image;
determining a fifth loss parameter of a control model based on the encoded second color sample image and the encoded second depth sample image, and a state label of the robot;
and adjusting the control model based on the fifth loss parameter to obtain the trained control model.
4. The method of claim 3, wherein determining a fifth loss parameter of a control model based on the encoded second color sample image and a state label of the robot comprises:
inputting the encoded second color sample image to the control model to obtain the predicted state information of the robot;
determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.
5. The method of claim 4, wherein the control model comprises a first control model and a second control model, and wherein inputting the encoded second color sample image to the control model results in predicted state information of the robot, comprises:
and outputting the prediction state information by using the encoded second color sample image as input information of the second control model and using output information of the second control model as input information of the first control model.
6. The method of claim 3, wherein determining fifth loss parameters for a control model based on the encoded second color sample image and the encoded second depth sample image, and the state label of the robot comprises:
inputting the coded second color sample image and the coded second depth sample image into the control model to obtain the predicted state information of the robot;
determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.
7. The method of claim 3, further comprising:
when a robot is controlled to execute a target task on a target object, a target color image containing the target object is obtained;
inputting the target color image to a first coding module of the trained color image processing model, and outputting a coded target color image;
and inputting the coded target color image into the trained control model to obtain result state information of the robot, and controlling the robot based on the result state information.
8. The method of claim 3, further comprising:
when a robot is controlled to execute a target task on a target object, a target color image and a target depth image which contain the target object are obtained;
inputting the target color image to a first coding module of the trained color image processing model, and outputting a coded target color image;
inputting the target depth image into a second coding module of the trained depth image processing model, and outputting a coded target depth image;
and inputting the coded target color image and the coded target depth image into the trained control model to obtain result state information of the robot, and controlling the robot based on the result state information.
9. The method of claim 3, wherein the obtaining a training sample set comprises:
and performing data enhancement processing on the collected color sample image and the collected depth sample image to increase the number of samples to obtain the training sample set.
10. An image processing apparatus, characterized in that the apparatus comprises:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a training sample set, and the training sample set comprises at least one color sample image and at least one depth sample image corresponding to the color sample image;
the processing unit is used for inputting the training sample set into an image processing model for processing, the image processing model comprises a color image processing model and a depth image processing model, the color image processing model is used for processing the color sample image to obtain a color reconstructed image, and the depth image processing model is used for processing the depth image to obtain a depth reconstructed image;
the processing unit is further configured to input the color sample image into a first encoding module of the color image processing model, and output an encoded first color sample image; inputting the depth sample image into a second coding module of the depth image processing model, and outputting a coded first depth sample image; calculating to obtain a second loss parameter by using the encoded first color sample image and the encoded first depth sample image; taking the second loss parameter as a first loss parameter; or, determining the first loss parameter based on the second loss parameter and the third loss parameter; or, determining the first loss parameter based on the second loss parameter and the fourth loss parameter; and adjusting the color image processing model based on the first loss parameter to obtain a trained color image processing model.
11. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 9.
12. An image processing apparatus, characterized in that the apparatus comprises a processor and a memory configured to store a computer program capable of running on the processor,
wherein the processor is configured to perform the steps of the method of any one of claims 1 to 9 when running the computer program.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111111394.4A CN113808061A (en) | 2019-04-28 | 2019-04-28 | Image processing method and device |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111111394.4A CN113808061A (en) | 2019-04-28 | 2019-04-28 | Image processing method and device |
CN201910351289.4A CN110111289B (en) | 2019-04-28 | 2019-04-28 | Image processing method and device |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910351289.4A Division CN110111289B (en) | 2019-04-28 | 2019-04-28 | Image processing method and device |
Publications (1)
Publication Number | Publication Date |
---|---|
CN113808061A true CN113808061A (en) | 2021-12-17 |
Family
ID=67487112
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202111112440.2A Pending CN113808062A (en) | 2019-04-28 | 2019-04-28 | Image processing method and device |
CN201910351289.4A Active CN110111289B (en) | 2019-04-28 | 2019-04-28 | Image processing method and device |
CN202111111394.4A Pending CN113808061A (en) | 2019-04-28 | 2019-04-28 | Image processing method and device |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202111112440.2A Pending CN113808062A (en) | 2019-04-28 | 2019-04-28 | Image processing method and device |
CN201910351289.4A Active CN110111289B (en) | 2019-04-28 | 2019-04-28 | Image processing method and device |
Country Status (1)
Country | Link |
---|---|
CN (3) | CN113808062A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113808062A (en) * | 2019-04-28 | 2021-12-17 | 深圳市商汤科技有限公司 | Image processing method and device |
CN114155366A (en) * | 2022-02-07 | 2022-03-08 | 北京每日优鲜电子商务有限公司 | Dynamic cabinet image recognition model training method and device, electronic equipment and medium |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110795522B (en) * | 2019-11-06 | 2023-01-06 | 中国人民解放军战略支援部队信息工程大学 | Method and device for predicting track position of mobile user |
CN111669587B (en) * | 2020-04-17 | 2021-07-20 | 北京大学 | Mimic compression method and device of video image, storage medium and terminal |
CN111931770B (en) * | 2020-09-16 | 2021-02-12 | 腾讯科技(深圳)有限公司 | Image processing method, device, equipment and storage medium |
CN112991493B (en) * | 2021-04-09 | 2023-07-18 | 华南理工大学 | Gray image coloring method based on VAE-GAN and mixed density network |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105139385A (en) * | 2015-08-12 | 2015-12-09 | 西安电子科技大学 | Image visual saliency region detection method based on deep automatic encoder reconfiguration |
CN108520535A (en) * | 2018-03-26 | 2018-09-11 | 天津大学 | Object classification method based on depth recovery information |
CN108520202A (en) * | 2018-03-15 | 2018-09-11 | 华南理工大学 | Confrontation robustness image characteristic extracting method based on variation spherical projection |
US20180315215A1 (en) * | 2015-10-30 | 2018-11-01 | Agfa Healthcare | Compressing and uncompressing method for high bit-depth medical gray scale images |
WO2018209932A1 (en) * | 2017-05-17 | 2018-11-22 | 清华大学 | Multi-quantization depth binary feature learning method and device |
CN108876745A (en) * | 2018-06-27 | 2018-11-23 | 厦门美图之家科技有限公司 | Image processing method and device |
CN108961327A (en) * | 2018-05-22 | 2018-12-07 | 深圳市商汤科技有限公司 | A kind of monocular depth estimation method and its device, equipment and storage medium |
CN109409435A (en) * | 2018-11-01 | 2019-03-01 | 上海大学 | A kind of depth perception conspicuousness detection method based on convolutional neural networks |
CN110111289A (en) * | 2019-04-28 | 2019-08-09 | 深圳市商汤科技有限公司 | A kind of image processing method and device |
Family Cites Families (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20130122419A (en) * | 2012-04-30 | 2013-11-07 | 삼성전자주식회사 | Apparatus and method for image synthesis using depth edge |
KR102277259B1 (en) * | 2014-11-26 | 2021-07-14 | 엘지전자 주식회사 | Device control system, digital device and method of controlling the same |
JP6754619B2 (en) * | 2015-06-24 | 2020-09-16 | 三星電子株式会社Samsung Electronics Co.,Ltd. | Face recognition method and device |
CN105163121B (en) * | 2015-08-24 | 2018-04-17 | 西安电子科技大学 | Big compression ratio satellite remote sensing images compression method based on depth autoencoder network |
CN105472393B (en) * | 2015-11-24 | 2018-04-17 | 宁波大学 | A kind of colour and depth image post-processing approach based on joint dictionary |
CN106408524B (en) * | 2016-08-17 | 2020-04-28 | 南京理工大学 | Depth image enhancement method based on two-dimensional image assistance |
CN107871100B (en) * | 2016-09-23 | 2021-07-06 | 北京眼神科技有限公司 | Training method and device of face model, and face authentication method and device |
CN106503106B (en) * | 2016-10-17 | 2019-10-18 | 北京工业大学 | A kind of image hash index construction method based on deep learning |
US10733744B2 (en) * | 2017-05-11 | 2020-08-04 | Kla-Tencor Corp. | Learning based approach for aligning images acquired with different modalities |
CN109427052B (en) * | 2017-08-29 | 2020-10-02 | ***通信有限公司研究院 | Related method and device for processing fundus images based on deep learning |
CN107767413B (en) * | 2017-09-20 | 2020-02-18 | 华南理工大学 | Image depth estimation method based on convolutional neural network |
WO2019075276A1 (en) * | 2017-10-11 | 2019-04-18 | Aquifi, Inc. | Systems and methods for object identification |
CN107833183B (en) * | 2017-11-29 | 2021-05-25 | 安徽工业大学 | Method for simultaneously super-resolving and coloring satellite image based on multitask deep neural network |
CN108491848B (en) * | 2018-03-09 | 2022-02-25 | 北京大学深圳研究生院 | Image saliency detection method and device based on depth information |
CN108596865B (en) * | 2018-03-13 | 2021-10-26 | 中山大学 | Feature map enhancement system and method for convolutional neural network |
CN108416803B (en) * | 2018-03-14 | 2020-01-24 | 大连理工大学 | Scene depth recovery method based on multi-information fusion of deep neural network |
CN109145939B (en) * | 2018-07-02 | 2021-11-02 | 南京师范大学 | Semantic segmentation method for small-target sensitive dual-channel convolutional neural network |
CN109034017B (en) * | 2018-07-12 | 2021-05-14 | 北京华捷艾米科技有限公司 | Head pose estimation method and machine readable storage medium |
CN108921846A (en) * | 2018-07-17 | 2018-11-30 | 北京航空航天大学 | A kind of rail tread defect identification method combined based on gray level image and depth image |
CN109087256A (en) * | 2018-07-19 | 2018-12-25 | 北京飞搜科技有限公司 | A kind of image deblurring method and system based on deep learning |
CN109086821A (en) * | 2018-07-31 | 2018-12-25 | 太原理工大学 | The grasping body method of discrimination of robot |
CN109271990A (en) * | 2018-09-03 | 2019-01-25 | 北京邮电大学 | A kind of semantic segmentation method and device for RGB-D image |
CN109461184B (en) * | 2018-10-31 | 2020-11-10 | 聚时科技(上海)有限公司 | Automatic positioning method for grabbing point for grabbing object by robot mechanical arm |
CN109460734B (en) * | 2018-11-08 | 2020-07-31 | 山东大学 | Video behavior identification method and system based on hierarchical dynamic depth projection difference image representation |
CN109544656B (en) * | 2018-11-23 | 2023-05-23 | 南京信息工程大学 | Compressed sensing image reconstruction method and system based on generation countermeasure network |
-
2019
- 2019-04-28 CN CN202111112440.2A patent/CN113808062A/en active Pending
- 2019-04-28 CN CN201910351289.4A patent/CN110111289B/en active Active
- 2019-04-28 CN CN202111111394.4A patent/CN113808061A/en active Pending
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105139385A (en) * | 2015-08-12 | 2015-12-09 | 西安电子科技大学 | Image visual saliency region detection method based on deep automatic encoder reconfiguration |
US20180315215A1 (en) * | 2015-10-30 | 2018-11-01 | Agfa Healthcare | Compressing and uncompressing method for high bit-depth medical gray scale images |
WO2018209932A1 (en) * | 2017-05-17 | 2018-11-22 | 清华大学 | Multi-quantization depth binary feature learning method and device |
CN108520202A (en) * | 2018-03-15 | 2018-09-11 | 华南理工大学 | Confrontation robustness image characteristic extracting method based on variation spherical projection |
CN108520535A (en) * | 2018-03-26 | 2018-09-11 | 天津大学 | Object classification method based on depth recovery information |
CN108961327A (en) * | 2018-05-22 | 2018-12-07 | 深圳市商汤科技有限公司 | A kind of monocular depth estimation method and its device, equipment and storage medium |
CN108876745A (en) * | 2018-06-27 | 2018-11-23 | 厦门美图之家科技有限公司 | Image processing method and device |
CN109409435A (en) * | 2018-11-01 | 2019-03-01 | 上海大学 | A kind of depth perception conspicuousness detection method based on convolutional neural networks |
CN110111289A (en) * | 2019-04-28 | 2019-08-09 | 深圳市商汤科技有限公司 | A kind of image processing method and device |
Non-Patent Citations (2)
Title |
---|
刘云鹏;STEPHAN BECK;王仁芳;王阳;郑文;BERND FROEHLICH;: "面向3D呈现的有损和无损混合深度视频编码", 光电子・激光, no. 02, 15 February 2017 (2017-02-15), pages 93 - 98 * |
刘晓春;陈炼;: "基于深度学习的手写体图像分类识别研究", 江西通信科技, no. 04, 15 December 2016 (2016-12-15), pages 38 - 42 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113808062A (en) * | 2019-04-28 | 2021-12-17 | 深圳市商汤科技有限公司 | Image processing method and device |
CN114155366A (en) * | 2022-02-07 | 2022-03-08 | 北京每日优鲜电子商务有限公司 | Dynamic cabinet image recognition model training method and device, electronic equipment and medium |
CN114155366B (en) * | 2022-02-07 | 2022-05-20 | 北京每日优鲜电子商务有限公司 | Dynamic cabinet image recognition model training method and device, electronic equipment and medium |
Also Published As
Publication number | Publication date |
---|---|
CN110111289B (en) | 2021-09-28 |
CN113808062A (en) | 2021-12-17 |
CN110111289A (en) | 2019-08-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110111289B (en) | Image processing method and device | |
CN111515961B (en) | Reinforcement learning reward method suitable for mobile mechanical arm | |
Dasari et al. | Robonet: Large-scale multi-robot learning | |
US20180157973A1 (en) | Method and device for a computerized mechanical device | |
CN111890365B (en) | Target tracking method and device, computer equipment and storage medium | |
CN109086869B (en) | Human body action prediction method based on attention mechanism | |
CN111445476A (en) | Monocular depth estimation method based on multi-mode unsupervised image content decoupling | |
Lonini et al. | Robust active binocular vision through intrinsically motivated learning | |
Teulière et al. | Self-calibrating smooth pursuit through active efficient coding | |
Passalis et al. | Deep reinforcement learning for controlling frontal person close-up shooting | |
CN115990875B (en) | Flexible cable state prediction and control system based on hidden space interpolation | |
CN116524299A (en) | Image sample generation method, device, equipment and storage medium | |
US20210341884A1 (en) | Generation of a control system for a target system | |
Chandrasekaran et al. | A combinatorial approach to testing deep neural network-based autonomous driving systems | |
Inoue et al. | Transfer learning from synthetic to real images using variational autoencoders for robotic applications | |
Pérez-Dattari et al. | Interactive learning of temporal features for control: Shaping policies and state representations from human feedback | |
CN115205586A (en) | Knowledge distillation-based multi-self-supervision task fusion method and device and storage medium | |
CN113534678B (en) | Migration method from simulation of operation question-answering task to physical system | |
CN117150881A (en) | Reinforced learning training loop method and device for bionic robot fish navigation task | |
CN116227582A (en) | Knowledge distillation method, device, equipment and storage medium of mask self-encoder | |
Monfort et al. | Asynchronous data aggregation for training end to end visual control networks | |
Pretorius et al. | A neural network-based kinematic and light-perception simulator for simple robotic evolution | |
CN111259762B (en) | Pantograph abnormity detection method | |
Limoyo et al. | Heteroscedastic uncertainty for robust generative latent dynamics | |
CN113156961B (en) | Driving control model training method, driving control method and related device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |