TWI777185B - Robot image enhancement method, processor, electronic equipment, computer readable storage medium - Google Patents
Robot image enhancement method, processor, electronic equipment, computer readable storage medium Download PDFInfo
- Publication number
- TWI777185B TWI777185B TW109122654A TW109122654A TWI777185B TW I777185 B TWI777185 B TW I777185B TW 109122654 A TW109122654 A TW 109122654A TW 109122654 A TW109122654 A TW 109122654A TW I777185 B TWI777185 B TW I777185B
- Authority
- TW
- Taiwan
- Prior art keywords
- image
- feature data
- processed
- processing
- robot
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 90
- 238000012545 processing Methods 0.000 claims description 150
- 238000000605 extraction Methods 0.000 claims description 49
- 238000013507 mapping Methods 0.000 claims description 39
- 230000015654 memory Effects 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 13
- 230000033001 locomotion Effects 0.000 claims description 12
- 230000009466 transformation Effects 0.000 claims description 12
- 238000007499 fusion processing Methods 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 description 27
- 230000008569 process Effects 0.000 description 25
- 238000012549 training Methods 0.000 description 17
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 15
- 230000000694 effects Effects 0.000 description 7
- 239000010813 municipal solid waste Substances 0.000 description 7
- 239000013598 vector Substances 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 239000000284 extract Substances 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012512 characterization method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000013138 pruning Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000013140 knowledge distillation Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Signal Processing (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Image Analysis (AREA)
- Manipulator (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Numerical Control (AREA)
Abstract
Description
本申請實施例關於圖像處理技術領域，關於但不限於一種機器人圖像增強方法、處理器、電子設備、電腦可讀儲存介質。The embodiments of the present application relate to the technical field of image processing, and relate to, but are not limited to, a robot image enhancement method, a processor, an electronic device, and a computer-readable storage medium.
相關技術中，機器人在自主控制方面的應用越來越廣，基於機器人的攝影頭拍攝的圖像可實現對機器人的自主控制，但在拍攝的圖像的品質較低的情況下，對機器人的控制精度較低，通過更換拍攝效果更好的攝影頭可提高拍攝的圖像的品質。In related technologies, the application of robots in autonomous control is becoming more and more extensive. The images captured by the camera based on the robot can realize the autonomous control of the robot, but when the quality of the captured images is low, the robot's The control is less precise, and the quality of the captured image can be improved by replacing the camera with a better capture.
本申請實施例提供一種機器人圖像增強方法、處理器、電子設備、電腦可讀儲存介質。Embodiments of the present application provide a robot image enhancement method, a processor, an electronic device, and a computer-readable storage medium.
本申請實施例提供了一種機器人圖像增強方法，包括：通過機器人的攝影頭獲取待處理圖像；對所述待處理圖像進行特徵提取處理，得到第一特徵資料；對所述第一特徵資料進行解碼處理，得到增強後的圖像。An embodiment of the present application provides a robot image enhancement method, including: acquiring an image to be processed through a camera of a robot; performing feature extraction processing on the to-be-processed image to obtain first feature data; The data is decoded to obtain an enhanced image.
可以看出，本申請實施例能夠通過機器人的攝影頭獲取待處理圖像，並對待處理圖像進行特徵提取處理，得到第一特徵資料，通過對第一特徵資料進行解碼處理，得到增強後的圖像，可提高攝影頭拍攝的圖像的品質。It can be seen that the embodiment of the present application can obtain the image to be processed through the camera of the robot, and perform feature extraction processing on the image to be processed to obtain the first feature data, and by decoding the first feature data, the enhanced image is obtained. image to improve the quality of images captured by the camera.
在本申請一些實施例中，所述增強後的圖像包括預設路線的路線標識；所述方法還包括：對所述增強後的圖像進行特徵提取處理，得到第二特徵資料；所述第二特徵資料包括所述預設路線的路線標識的特徵；根據所述第二特徵資料和第一映射關係，得到控制資訊；所述第一映射關係為特徵資料與控制資訊之間的映射關係；所述控制資訊包括速度和/或轉向角。In some embodiments of the present application, the enhanced image includes a route identification of a preset route; the method further includes: performing feature extraction processing on the enhanced image to obtain second feature data; the The second feature data includes the feature of the route identifier of the preset route; control information is obtained according to the second feature data and the first mapping relationship; the first mapping relationship is the mapping relationship between the feature data and the control information ; the control information includes speed and/or steering angle.
通過這種方式，可根據預設路線的路線標識的特徵與控制資訊之間的映射關係，得到機器人的速度和/或轉向角；進而，能夠依據速度和/或轉向角精確地實現對機器人的控制。In this way, the speed and/or the steering angle of the robot can be obtained according to the mapping relationship between the characteristics of the route identification of the preset route and the control information; furthermore, the robot can be accurately controlled according to the speed and/or the steering angle. control.
在本申請一些實施例中，所述增強後的圖像包括球；所述方法還包括：根據所述增強後的圖像，得到所述球的運動軌跡；根據所述運動軌跡，得到控制資訊，所述控制資訊包括速度和/或轉向角。In some embodiments of the present application, the enhanced image includes a ball; the method further includes: obtaining a movement trajectory of the ball according to the enhanced image; obtaining control information according to the movement trajectory , the control information includes speed and/or steering angle.
通過這種方式，可根據球的運動軌跡，得到機器人的速度和/或轉向角；進而，能夠依據速度和/或轉向角精確地實現對機器人的控制。In this way, the speed and/or the steering angle of the robot can be obtained according to the motion trajectory of the ball; further, the robot can be precisely controlled according to the speed and/or the steering angle.
在本申請一些實施例中，所述增強後的圖像包括手、人臉或目標物體中的任意一種；所述方法還包括：對所述增強後的圖像進行以下至少一種識別：人臉識別、手勢識別、目標物識別；根據識別結果，得到控制資訊，所述控制資訊包括速度和/或轉向角。In some embodiments of the present application, the enhanced image includes any one of a hand, a human face or a target object; the method further includes: performing at least one of the following recognition on the enhanced image: a human face Recognition, gesture recognition, target recognition; according to the recognition result, control information is obtained, and the control information includes speed and/or steering angle.
在一種實施方式中，機器人內部的深度神經網路對增強後的圖像進行手勢、人臉或目標物體的特徵提取，根據提取到的手勢、人臉或目標物體特徵，得到機器人的速度和/或轉向角；進而，能夠依據速度和/或轉向角精確地實現對機器人的控制。In one embodiment, the deep neural network inside the robot extracts the features of gestures, faces or target objects from the enhanced images, and obtains the speed and/or speed of the robot according to the extracted features of gestures, faces or target objects. or steering angle; further, the control of the robot can be achieved precisely depending on the speed and/or the steering angle.
在本申請一些實施例中，所述對所述待處理圖像進行特徵提取處理，得到第一特徵資料，包括：對所述待處理圖像進行卷積處理，得到第三特徵資料；將所述待處理圖像與所述第三特徵資料進行融合處理，得到所述第一特徵資料。In some embodiments of the present application, performing feature extraction processing on the to-be-processed image to obtain first feature data includes: performing convolution processing on the to-be-processed image to obtain third feature data; The to-be-processed image and the third feature data are fused to obtain the first feature data.
可以看出，機器人通過對待處理圖像進行卷積處理， 得到待處理圖像的特徵資料，將待處理圖像與待處理圖像的特徵資料進行融合處理，可以將圖像的尺寸縮小，減小計算量，提高運算速度。It can be seen that the robot obtains the characteristic data of the to-be-processed image by performing convolution processing on the to-be-processed image, and fuses the to-be-processed image with the characteristic data of the to-be-processed image, which can reduce the size of the image and reduce the size of the image. Small amount of calculation, improve operation speed.
在本申請一些實施例中，所述對所述第一特徵資料進行解碼處理，得到增強後的圖像，包括：獲取所述待處理圖像的前一幀圖像；對所述前一幀圖像進行卷積處理，得到第四特徵資料；將所述第一特徵資料與所述第四特徵資料進行合併處理，得到第五特徵資料；對所述第五特徵資料進行解碼處理，得到所述增強後的圖像。In some embodiments of the present application, performing decoding processing on the first feature data to obtain an enhanced image includes: acquiring a previous frame of the image to be processed; Perform convolution processing on the image to obtain fourth feature data; combine the first feature data and the fourth feature data to obtain fifth feature data; perform decoding processing on the fifth feature data to obtain the the enhanced image.
可以看出，通過將前一幀的圖像的特徵資料與待處理圖像的特徵資料進行合併，可豐富圖像特徵資訊，更有利於圖像增強，提高增強後的圖像的品質。It can be seen that by combining the feature data of the image of the previous frame with the feature data of the image to be processed, the feature information of the image can be enriched, which is more conducive to image enhancement and improves the quality of the enhanced image.
在本申請一些實施例中，所述對所述第五特徵資料進行解碼處理，得到所述增強後的圖像，包括：對所述第五特徵資料進行反卷積處理，得到所述增強後的圖像。In some embodiments of the present application, performing decoding processing on the fifth feature data to obtain the enhanced image includes: performing deconvolution processing on the fifth feature data to obtain the enhanced image Image.
通過這種方式，可實現對第五特徵資料的解碼，並得到增強後的圖像；由於得到增強後的圖像的整個過程可在機器人上運行，提高處理速度，並且整個過程均是在獲取到待處理圖像後即時完成的，因此，可進一步提高後續基於增強後的圖像進行相應處理的速度。In this way, the fifth feature data can be decoded, and an enhanced image can be obtained; because the whole process of obtaining the enhanced image can be run on the robot, the processing speed can be improved, and the whole process is obtained during the acquisition. It is completed immediately after the image to be processed is reached, so the speed of subsequent corresponding processing based on the enhanced image can be further improved.
在本申請一些實施例中，所述對所述待處理圖像進行特徵提取處理，得到第一特徵資料之前，所述方法還包括：對所述待處理圖像依次進行卷積處理、歸一化處理、線性變換、非線性變換，確定所述待處理圖像中的對象所屬類別；根據所述類別和第二映射關係，確定閾值；所述第二映射關係為類別與解析度閾值之間的映射關係；在所述待處理圖像的解析度小於或等於所述閾值的情況下，執行所述對所述待處理圖像進行特徵提取處理，得到第一特徵資料的步驟。In some embodiments of the present application, before the feature extraction process is performed on the to-be-processed image to obtain the first feature data, the method further includes: sequentially performing convolution processing and normalization on the to-be-processed image. process, linear transformation, and nonlinear transformation to determine the category of the object in the image to be processed; determine the threshold according to the category and the second mapping relationship; the second mapping relationship is between the category and the resolution threshold If the resolution of the to-be-processed image is less than or equal to the threshold, perform the feature extraction process on the to-be-processed image to obtain the first feature data.
可以看出，通過待處理圖像中的對象所屬類別，確定對應的解析度閾值，能夠更加針對性的對待處理圖像進行處理，提高處理效率。It can be seen that, by determining the corresponding resolution threshold value according to the category of the object in the image to be processed, the image to be processed can be processed more pertinently, and the processing efficiency can be improved.
本申請實施例提供了一種機器人圖像增強裝置，包括：圖像採集部分，配置為通過機器人的攝影頭獲取待處理圖像；第一特徵提取部分，配置為對所述待處理圖像進行特徵提取處理，得到第一特徵資料；解碼處理部分，配置為對所述第一特徵資料進行解碼處理，得到增強後的圖像。An embodiment of the present application provides a robot image enhancement device, including: an image acquisition part, configured to acquire an image to be processed through a camera of a robot; a first feature extraction part, configured to perform feature extraction on the to-be-processed image Extraction processing to obtain first feature data; and a decoding processing part configured to perform decoding processing on the first feature data to obtain an enhanced image.
可以看出，本申請實施例能夠通過機器人的攝影頭獲取待處理圖像，並對待處理圖像進行特徵提取處理，得到第一特徵資料，通過對第一特徵資料進行解碼處理，得到增強後的圖像，可提高攝影頭拍攝的圖像的品質。It can be seen that the embodiment of the present application can obtain the image to be processed through the camera of the robot, and perform feature extraction processing on the image to be processed to obtain the first feature data, and by decoding the first feature data, the enhanced image is obtained. image to improve the quality of images captured by the camera.
在本申請一些實施例中，所述增強後的圖像包括預設路線的路線標識；所述機器人圖像增強裝置還包括：第二特徵提取部分，配置為對所述增強後的圖像進行特徵提取處理，得到第二特徵資料；所述第二特徵資料包括所述預設路線的路線標識的特徵；第一處理部分，配置為根據所述第二特徵資料和第一映射關係，得到控制資訊；所述第一映射關係為特徵資料與控制資訊之間的映射關係；所述控制資訊包括速度和/或轉向角。In some embodiments of the present application, the enhanced image includes a route identification of a preset route; the robot image enhancement device further includes: a second feature extraction part configured to perform an image processing operation on the enhanced image. Feature extraction processing to obtain second feature data; the second feature data includes the features of the route identification of the preset route; the first processing part is configured to obtain control according to the second feature data and the first mapping relationship information; the first mapping relationship is a mapping relationship between feature data and control information; the control information includes speed and/or steering angle.
通過這種方式，可根據預設路線的路線標識的特徵與控制資訊之間的映射關係，得到機器人的速度和/或轉向角；進而，能夠依據速度和/或轉向角精確地實現對機器人的控制。In this way, the speed and/or the steering angle of the robot can be obtained according to the mapping relationship between the characteristics of the route identification of the preset route and the control information; furthermore, the robot can be accurately controlled according to the speed and/or the steering angle. control.
在本申請一些實施例中，所述增強後的圖像包括球；所述機器人圖像增強裝置還包括：第二處理部分，配置為根據所述增強後的圖像，得到所述球的運動軌跡；第三處理部分，配置為根據所述運動軌跡，得到控制資訊，所述控制資訊包括速度和/或轉向角。In some embodiments of the present application, the enhanced image includes a ball; the robot image enhancement device further includes: a second processing part configured to obtain the motion of the ball according to the enhanced image a trajectory; a third processing part configured to obtain control information according to the motion trajectory, where the control information includes speed and/or steering angle.
通過這種方式，可根據球的運動軌跡，得到機器人的速度和/或轉向角；進而，能夠依據速度和/或轉向角精確地實現對機器人的控制。In this way, the speed and/or the steering angle of the robot can be obtained according to the motion trajectory of the ball; further, the robot can be precisely controlled according to the speed and/or the steering angle.
在本申請一些實施例中，所述增強後的圖像包括手、人臉或目標物體中的任意一種；所述機器人圖像增強裝置還包括：識別部分，配置為對所述增強後的圖像進行以下至少一種識別：人臉識別、手勢識別、目標物識別；第四處理部分，配置為根據識別結果，得到控制資訊，所述控制資訊包括速度和/或轉向角。In some embodiments of the present application, the enhanced image includes any one of a hand, a human face, or a target object; the robot image enhancement device further includes: a recognition part configured to analyze the enhanced image The image performs at least one of the following recognitions: face recognition, gesture recognition, and target recognition; the fourth processing part is configured to obtain control information according to the recognition result, and the control information includes speed and/or steering angle.
在一種實施方式中，機器人內部的深度神經網路對增強後的圖像進行手勢、人臉或目標物體的特徵提取，根據提取到的手勢、人臉或目標物體特徵，得到機器人的速度和/或轉向角；進而，能夠依據速度和/或轉向角精確地實現對機器人的控制。In one embodiment, the deep neural network inside the robot extracts the features of gestures, faces or target objects from the enhanced images, and obtains the speed and/or speed of the robot according to the extracted features of gestures, faces or target objects. or steering angle; further, the control of the robot can be achieved precisely depending on the speed and/or the steering angle.
在本申請一些實施例中，所述第一特徵提取部分包括：第一卷積處理子部分，配置為對所述待處理圖像進行卷積處理，得到第三特徵資料；融合處理子部分，配置為將所述待處理圖像與所述第三特徵資料進行融合處理，得到所述第一特徵資料。In some embodiments of the present application, the first feature extraction section includes: a first convolution processing subsection, configured to perform convolution processing on the to-be-processed image to obtain third feature data; a fusion processing subsection, It is configured to perform fusion processing on the to-be-processed image and the third feature data to obtain the first feature data.
可以看出，機器人通過對待處理圖像進行卷積處理， 得到待處理圖像的特徵資料，將待處理圖像與待處理圖像的特徵資料進行融合處理，可以將圖像的尺寸縮小，減小計算量，提高運算速度。It can be seen that the robot obtains the characteristic data of the to-be-processed image by performing convolution processing on the to-be-processed image, and fuses the to-be-processed image with the characteristic data of the to-be-processed image, which can reduce the size of the image and reduce the size of the image. Small amount of calculation, improve operation speed.
在本申請一些實施例中，所述解碼處理部分包括：獲取子部分，配置為獲取所述待處理圖像的前一幀圖像；第二卷積處理子部分，配置為對所述前一幀圖像進行卷積處理，得到第四特徵資料；合併處理子部分，配置為將所述第一特徵資料與所述第四特徵資料進行合併處理，得到第五特徵資料；解碼處理子部分，配置為對所述第五特徵資料進行解碼處理，得到所述增強後的圖像。In some embodiments of the present application, the decoding processing part includes: an acquisition subsection, configured to acquire a previous frame of the image to be processed; a second convolution processing subsection, configured to acquire the previous frame of the image to be processed; The frame image is subjected to convolution processing to obtain fourth feature data; the merging processing sub-section is configured to perform merging processing on the first feature data and the fourth feature data to obtain fifth feature data; the decoding processing sub-section, It is configured to perform decoding processing on the fifth feature data to obtain the enhanced image.
可以看出，通過將前一幀的圖像的特徵資料與待處理圖像的特徵資料進行合併，可豐富圖像特徵資訊，更有利於圖像增強，提高增強後的圖像的品質。It can be seen that by combining the feature data of the image of the previous frame with the feature data of the image to be processed, the feature information of the image can be enriched, which is more conducive to image enhancement and improves the quality of the enhanced image.
在本申請一些實施例中，所述解碼處理子部分配置為：對所述第五特徵資料進行反卷積處理，得到所述增強後的圖像。In some embodiments of the present application, the decoding processing subsection is configured to perform deconvolution processing on the fifth feature data to obtain the enhanced image.
通過這種方式，可實現對第五特徵資料的解碼，並得到增強後的圖像；由於得到增強後的圖像的整個過程可在機器人上運行，提高處理速度，並且整個過程均是在獲取到待處理圖像後即時完成的，因此，可進一步提高後續基於增強後的圖像進行相應處理的速度。In this way, the fifth feature data can be decoded, and an enhanced image can be obtained; because the whole process of obtaining the enhanced image can be run on the robot, the processing speed can be improved, and the whole process is obtained during the acquisition. It is completed immediately after the image to be processed is reached, so the speed of subsequent corresponding processing based on the enhanced image can be further improved.
在本申請一些實施例中，所述機器人圖像增強裝置還包括：第五處理部分，配置為對所述待處理圖像依次進行卷積處理、歸一化處理、線性變換、非線性變換，確定所述待處理圖像中的對象所屬類別；第六處理部分，配置為根據所述類別和第二映射關係，確定閾值；所述第二映射關係為類別與解析度閾值之間的映射關係；所述第一特徵提取部分，還配置為在所述待處理圖像的解析度小於或等於所述閾值的情況下，執行所述對所述待處理圖像進行特徵提取處理，得到第一特徵資料的步驟。In some embodiments of the present application, the robot image enhancement device further includes: a fifth processing part configured to sequentially perform convolution processing, normalization processing, linear transformation, and nonlinear transformation on the to-be-processed image, Determine the category to which the object in the image to be processed belongs; the sixth processing part is configured to determine a threshold according to the category and a second mapping relationship; the second mapping relationship is a mapping relationship between the category and the resolution threshold ; The first feature extraction part is further configured to perform the feature extraction process on the to-be-processed image when the resolution of the to-be-processed image is less than or equal to the threshold to obtain the first Steps for characterization data.
可以看出，通過待處理圖像中的對象所屬類別，確定對應的解析度閾值，能夠更加針對性的對待處理圖像進行處理，提高處理效率。It can be seen that, by determining the corresponding resolution threshold value according to the category of the object in the image to be processed, the image to be processed can be processed more pertinently, and the processing efficiency can be improved.
本申請實施例提供了一種處理器，所述處理器執行上述任意一項所述的機器人圖像增強方法。An embodiment of the present application provides a processor, where the processor executes any one of the robot image enhancement methods described above.
本申請實施例提供了一種電子設備，包括：處理器、輸入裝置、輸出裝置和記憶體，所述處理器、輸入裝置、輸出裝置和記憶體相互連接，所述記憶體中儲存有程式指令；所述程式指令被所述處理器執行時，使所述處理器執行上述任意一項所述的機器人圖像增強方法。An embodiment of the present application provides an electronic device, including: a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are connected to each other, and program instructions are stored in the memory; When the program instructions are executed by the processor, the processor is caused to execute the robot image enhancement method described in any one of the above.
本申請實施例提供了一種電腦可讀儲存介質，所述電腦可讀儲存介質中儲存有電腦程式，所述電腦程式包括程式指令，所述程式指令當被電子設備的處理器執行時，使所述處理器執行上述任意一項所述的機器人圖像增強方法。An embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program includes program instructions, and when executed by a processor of an electronic device, the program instructions cause all The processor executes the robot image enhancement method described in any one of the above.
本申請實施例還提供了一種電腦程式，包括電腦可讀代碼，當所述電腦可讀代碼在電子設備中運行時，所述電子設備中的處理器執行用於實現上述任意一種機器人圖像增強方法。Embodiments of the present application further provide a computer program, including computer-readable codes. When the computer-readable codes are run in an electronic device, a processor in the electronic device executes any one of the above-mentioned robot image enhancements. method.
本申請實施例提出一種機器人圖像增強方法及裝置、處理器、電子設備、電腦可讀儲存介質和電腦程式；能夠通過機器人的攝影頭獲取待處理圖像，並對待處理圖像進行特徵提取處理，得到第一特徵資料，通過對第一特徵資料進行解碼處理，得到增強後的圖像，可提高攝影頭拍攝的圖像的品質。The embodiments of the present application provide a method and device, processor, electronic device, computer-readable storage medium, and computer program for enhancing image of a robot; the image to be processed can be acquired through the camera of the robot, and the feature extraction processing of the image to be processed can be performed. to obtain the first feature data, and by decoding the first feature data, an enhanced image is obtained, which can improve the quality of the image captured by the camera.
應當理解的是，以上的一般描述和後文的細節描述僅是示例性和解釋性的，而非限制本申請實施例。It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, rather than limiting, of the embodiments of the present application.
為了使本技術領域的人員更好地理解本申請方案，下面將結合本申請實施例中的附圖，對本申請實施例中的技術方案進行清楚、完整地描述，顯然，所描述的實施例僅僅是本申請一部分實施例，而不是全部的實施例。基於本申請中的實施例，本領域普通技術人員在沒有做出創造性勞動前提下所獲得的所有其他實施例，都屬於本申請保護的範圍。In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.
本申請的說明書和申請專利範圍及上述附圖中的術語的說明書和申請專利範圍及用於區別不同對象，而不是用於描述特定順序。此外，術語“包括”和“具有”以及它們任何變形，意圖在於覆蓋不排他的包含。例如包含了一系列步驟或部分的過程、方法、系統、產品或設備沒有限定於已列出的步驟或部分，而是在本申請的一些實施例中還包括沒有列出的步驟或部分，或在本申請的一些實施例中還包括對於這些過程、方法、產品或設備固有的其他步驟或部分。The description and scope of the present application and the terms in the above-mentioned drawings are used to distinguish between different objects, rather than to describe a particular order. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or parts is not limited to the listed steps or parts, but in some embodiments of the present application also includes unlisted steps or parts, or Other steps or portions inherent to these processes, methods, products or devices are also included in some embodiments of the present application.
在本文中提及“實施例”意味著，結合實施例描述的特定特徵、結構或特性可以包含在本申請的至少一個實施例中。在說明書中的各個位置出現該短語並不一定均是指相同的實施例，也不是與其它實施例互斥的獨立的或備選的實施例。本領域技術人員顯式地和隱式地理解的是，本文所描述的實施例可以與其它實施例相結合。Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
下面結合本申請實施例中的附圖對本申請實施例進行描述。The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.
請參閱圖1，圖1是本申請實施例提供的一種機器人圖像增強方法的流程示意圖，所述機器人圖像增強方法包括如下。Please refer to FIG. 1. FIG. 1 is a schematic flowchart of a robot image enhancement method provided by an embodiment of the present application. The robot image enhancement method includes the following.
步驟101、通過機器人的攝影頭獲取待處理圖像。Step 101: Acquire an image to be processed through a camera of the robot.
在本申請的一些實施例中，機器人為可進行自主控制的機器，上述自主控制可以包括以下至少一項：人物跟蹤、循跡行走、自主避障行走等。本申請實施例對機器人的形狀不做限定，可以是智慧小車，也可以是人形機器人。In some embodiments of the present application, the robot is a machine that can perform autonomous control, and the above-mentioned autonomous control may include at least one of the following: character tracking, walking on a track, and walking on autonomous obstacle avoidance. The embodiment of the present application does not limit the shape of the robot, which may be a smart car or a humanoid robot.
機器人上裝載有攝影頭，機器人通過攝影頭可對機器人周圍的環境進行即時拍攝，其中，拍攝的方式可以為拍攝視頻，也可以為拍攝圖像，本申請實施例對此不做限定。A camera is mounted on the robot, and the robot can shoot the environment around the robot in real time through the camera. The shooting method may be video shooting or image shooting, which is not limited in the embodiments of the present application.
在本申請的一些實施例中，待處理圖像可以是通過攝影頭拍攝得到的圖像，也可以是從拍攝的視頻中截取的圖像，本申請實施例對此不做限定。In some embodiments of the present application, the image to be processed may be an image captured by a camera, or an image captured from a captured video, which is not limited in this embodiment of the present application.
在本申請的一些實施例中，圖像尺寸可以為預定大小，例如：後續處理對圖像的尺寸要求為256*256*3，則可將待處理圖像的尺寸調整為256*256*3，調整方式可以是對待處理圖像進行縮放，也可以是對待處理圖進行裁剪，還可以是對待處理圖像中進行特徵提取，截取包含後續處理所需對象的圖像區域得到預定大小的圖像，本申請實施例對調整待處理圖像的方式不做限定。In some embodiments of the present application, the size of the image may be a predetermined size. For example, if the size of the image required for subsequent processing is 256*256*3, the size of the image to be processed may be adjusted to 256*256*3 , the adjustment method can be zooming the image to be processed, or cropping the image to be processed, or extracting features from the image to be processed, intercepting the image area containing the objects required for subsequent processing to obtain an image of a predetermined size , the embodiment of the present application does not limit the manner of adjusting the image to be processed.
步驟102、對待處理圖像進行特徵提取處理，得到第一特徵資料。Step 102: Perform feature extraction processing on the image to be processed to obtain first feature data.
在本申請的一些實施例中，第一特徵資料可以包括待處理圖像中的對象（包括目標人物或目標物體）的特徵資料，以及待處理圖像中的背景（除目標人物或目標物體之外的圖像內容）的特徵資料。In some embodiments of the present application, the first feature data may include feature data of the object in the image to be processed (including the target person or the target object), and the background in the image to be processed (except the target person or the target object) external image content).
對待處理圖像進行特徵提取處理，得到待處理圖像的特徵資料。其中，特徵提取處理可以為卷積處理。在本申請的一些實施例中，對於待處理圖像中的任意一個像素點，使卷積範本的中心點和該像素點重合，卷積範本上的點與待處理圖像上對應的像素點相乘，最後再將每個像素點的乘積相加，得到該像素點的卷積值，通過對待處理圖像中每個像素點進行上述卷積處理，將待處理圖像縮小，並提取出第一特徵資料。Feature extraction processing is performed on the image to be processed to obtain feature data of the image to be processed. The feature extraction processing may be convolution processing. In some embodiments of the present application, for any pixel point in the image to be processed, the center point of the convolution template is made to coincide with the pixel point, and the point on the convolution template corresponds to the pixel point on the image to be processed. Multiply, and finally add the products of each pixel to obtain the convolution value of the pixel. By performing the above convolution processing on each pixel in the image to be processed, the image to be processed is reduced and extracted. The first characteristic data.
在本申請的一些實施例中，上述特徵提取處理可以由機器人執行。In some embodiments of the present application, the feature extraction process described above may be performed by a robot.
步驟103、對第一特徵資料進行解碼處理，得到增強後的圖像。Step 103: Decode the first feature data to obtain an enhanced image.
在本申請的一些實施例中，增強後的圖像相較於待處理圖像，圖像內容不變，但圖像品質（包括圖像解析度、解析度、清晰度等）更好。In some embodiments of the present application, compared with the to-be-processed image, the image content after the enhancement remains unchanged, but the image quality (including image resolution, resolution, clarity, etc.) is better.
通過對第一特徵資料進行解碼處理，可得到增強後的圖像，解碼處理可以為以下任意一種：反卷積處理、雙線性插值處理、反池化處理。在本申請的一些實施例中，運行於機器人上的圖像增強網路包括卷積層（用於對待處理圖像進行特徵提取處理，得到第一特徵資料），以及反卷積層（用於對第一特徵資料進行解碼處理）。其中，圖像增強網路為預先訓練好的，通過訓練使圖像增強網路學習到特徵資料與增強後的圖像的映射關係，因此，可通過圖像增強網路對第一特徵資料進行解碼處理，得到增強後的圖像。An enhanced image can be obtained by performing decoding processing on the first feature data, and the decoding processing can be any one of the following: deconvolution processing, bilinear interpolation processing, and inverse pooling processing. In some embodiments of the present application, the image enhancement network running on the robot includes a convolution layer (for performing feature extraction processing on the image to be processed to obtain the first feature data), and a deconvolution layer (for a feature data for decoding). The image enhancement network is pre-trained, and the image enhancement network learns the mapping relationship between the feature data and the enhanced image through training. Therefore, the first feature data can be processed by the image enhancement network. Decoding process to get the enhanced image.
本申請實施例通過對待處理圖像進行特徵提取處理，得到第一特徵資料，再通過訓練學習到的特徵資料與增強後的圖像的映射關係對第一特徵資料進行解碼，得到增強後的圖像，可提高攝影頭拍攝的圖像的品質。In the embodiment of the present application, the first feature data is obtained by performing feature extraction processing on the image to be processed, and then the first feature data is decoded through the mapping relationship between the feature data learned through training and the enhanced image, and the enhanced image is obtained. image to improve the quality of the image captured by the camera.
機器人通過對攝影頭採集到的圖像進行處理，可實現包括：人物跟蹤、循跡行走、自主避障行走等至少一種控制，但攝影頭採集的圖像的品質將極大的影響控制效果，低品質圖像（如：圖像雜訊多、圖像解析度低、圖像清晰度低等）甚至無法實現上述控制。因此，本申請實施例提供機器人圖像增強方法，用於對低品質圖像進行增強處理，以提高圖像品質。By processing the images collected by the camera, the robot can achieve at least one control including: character tracking, walking on track, and autonomous obstacle avoidance walking. However, the quality of the images collected by the camera will greatly affect the control effect. High-quality images (such as: high image noise, low image resolution, low image clarity, etc.) cannot even achieve the above control. Therefore, the embodiments of the present application provide a robot image enhancement method for performing enhancement processing on low-quality images to improve image quality.
在本申請的一些實施例中，所述機器人圖像增強方法可以由機器人圖像增強裝置執行，機器人圖像增強裝置可以是使用者設備（User Equipment，UE）、移動設備、使用者終端、終端、蜂窩電話、無線電話、個人數位助理（Personal Digital Assistant，PDA）、手持設備、計算設備、車載設備、可穿戴設備等，所述方法可以通過處理器調用記憶體中儲存的電腦可讀指令的方式來實現。或者，可通過伺服器執行該方法。In some embodiments of the present application, the robot image enhancement method may be performed by a robot image enhancement device, and the robot image enhancement device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal , cellular phones, wireless phones, personal digital assistants (Personal Digital Assistant, PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc., the method can call the computer-readable instructions stored in the memory through the processor. way to achieve. Alternatively, the method can be performed by a server.
下面結合一個應用場景對本申請進行進一步說明。The present application will be further described below with reference to an application scenario.
本申請實施例的機器人圖像增強方法，能夠應用於進行自主控制的機器等應用場景中；圖2為本申請實施例的一個應用場景的示意圖，如圖2所示，機器人200頭頂上的攝影頭20c為上述機器人的攝影頭，人臉圖像201為上述待處理圖像，可以通過機器人200的攝影頭20c獲取人臉圖像201，在機器人圖像增強裝置202中，通過前述實施例記載的機器人圖像增強方法進行處理，可以得到增強後的人臉圖像；其中，增強後的人臉圖像相比於原始人臉圖像具有更好的圖像品質；得到增強後的人臉圖像後，能夠更加精確地實現對機器人的控制，例如，人物跟蹤、循跡行走、自主避障行走等。需要說明的是，圖2所示的場景僅僅是本申請實施例的一個示例性場景，本申請對具體的應用場景不作限制。The robot image enhancement method according to the embodiment of the present application can be applied to application scenarios such as autonomously controlled machines; FIG. 2 is a schematic diagram of an application scenario of the embodiment of the present application. As shown in FIG. 2 , the camera on the top of the
請參閱圖3，圖3是本申請實施例提供的另一種機器人圖像增強方法的流程示意圖，所述機器人圖像增強方法包括如下。Please refer to FIG. 3 . FIG. 3 is a schematic flowchart of another robot image enhancement method provided by an embodiment of the present application. The robot image enhancement method includes the following.
步驟301、通過機器人的攝影頭獲取待處理圖像。Step 301: Acquire an image to be processed through a camera of the robot.
具體可參見步驟101，此處將不再贅述。For details, refer to step 101, which will not be repeated here.
步驟302、對待處理圖像進行卷積處理，得到第一特徵資料。Step 302: Perform convolution processing on the image to be processed to obtain first feature data.
在本申請的一些實施例中，在對待處理圖像進行卷積處理之前，可根據待處理圖像的品質判斷是否需要對待處理圖像進行卷積處理。在一種實施方式中，可以對待處理圖像依次進行卷積處理、歸一化處理、線性變換、非線性變換，確定待處理圖像中的對象所屬類別，根據類別和第二映射關係，確定閾值，其中，第二映射關係為類別與解析度閾值之間的映射關係。在待處理圖像的圖像解析度小於或等於解析度閾值的情況下，對待處理圖像進行卷積處理。舉例來說，第二映射關係可以為預先設定的，可參見表1。
表1、類別與解析度閾值之間的映射關係表
其中，若後續處理需要識別人臉，則需要更多特徵，對圖像品質要求較高，因此，解析度閾值較高；若後續處理需要識別球，則需要的特徵較少，對圖像品質要求較低，因此解析度閾值較低。對待處理圖像依次進行卷積處理、歸一化處理、線性變換、非線性變換，確定待處理圖像中的對象為人臉。通過獲取圖像解析度的演算法確定待處理圖像的解析度為102PPI（Pixels Per Inch），由於待處理圖像的解析度小於解析度閾值，因此，需要對待處理圖像進行卷積處理，這裡，獲取圖像解析度的演算法可以是雙重循環法，也可以是連續掃描法，本申請實施例對此不做限定。Among them, if the subsequent processing needs to recognize faces, more features are needed, and the image quality is required to be higher, so the resolution threshold is higher; The requirements are lower, so the resolution threshold is lower. The image to be processed is sequentially subjected to convolution processing, normalization processing, linear transformation, and nonlinear transformation to determine that the object in the image to be processed is a human face. The algorithm for obtaining the image resolution determines that the resolution of the image to be processed is 102PPI (Pixels Per Inch). Since the resolution of the image to be processed is less than the resolution threshold, it is necessary to perform convolution on the image to be processed. Here, the algorithm for obtaining the image resolution may be a double loop method or a continuous scanning method, which is not limited in this embodiment of the present application.
第二映射關係可以為預先設定的，而第二映射關係中的解析度閾值可替換為像素閾值，即以待處理圖像的像素為依據，判斷待處理圖像是否需要進行卷積處理。第二映射關係中的解析度閾值還可替換為清晰度閾值，即以待處理圖像的清晰度為依據，判斷待處理圖像是否需要進行卷積處理，本申請實施例對此不做限定。The second mapping relationship may be preset, and the resolution threshold in the second mapping relationship may be replaced by a pixel threshold, that is, based on the pixels of the image to be processed, it is determined whether the image to be processed needs to be convoluted. The resolution threshold in the second mapping relationship can also be replaced with a sharpness threshold, that is, based on the sharpness of the image to be processed, it is determined whether the image to be processed needs to undergo convolution processing, which is not limited in this embodiment of the present application .
在本申請的一些實施例中，圖像增強網路可以是預先訓練得到的，在本申請的一些實施例中，訓練所用的圖像集均為與機器人上裝載的攝影頭同一型號的攝影頭採集得到，可以包括：低品質圖像，以及與低品質圖像對應的高品質圖像。其中，高品質圖像可通過對低品質圖像進行圖像增強處理獲得，該圖像增強處理可以包括以下一種或多種：對低品質圖像進行去模糊處理、對低品質圖像進行超解析度處理、對低品質圖像進行補光處理，該圖像增強處理可以通過機器人之外的其他設備（如電腦）完成，對每一張低品質圖像進行該圖像增強處理均可獲得一張對應的高品質圖像。通過將低品質圖像與高品質圖像作為訓練集對圖像增強網路進行訓練，可使圖像增強網路學習到低品質圖像的特徵與高品質圖像的特徵之間的映射關係。通過以特定型號（與機器人上裝載的攝影頭同一型號）採集的圖像為訓練集對圖像增強網路進行訓練，可使訓練後的圖像增強網路更適配於該型號的攝影頭，對應於不同型號的攝影頭，可通過重新訓練，提高圖像增強的效果。In some embodiments of the present application, the image enhancement network may be pre-trained. In some embodiments of the present application, the image sets used for training are all cameras of the same model as the cameras loaded on the robot The acquisition can include: low-quality images, and high-quality images corresponding to the low-quality images. The high-quality image can be obtained by performing image enhancement processing on the low-quality image, and the image enhancement processing can include one or more of the following: deblurring the low-quality image, performing super-resolution on the low-quality image The image enhancement processing can be completed by other equipment (such as a computer) other than the robot, and each low-quality image can be obtained by performing the image enhancement processing on each low-quality image. corresponding high-quality images. By training the image enhancement network with low-quality images and high-quality images as training sets, the image enhancement network can learn the mapping relationship between the features of the low-quality images and the features of the high-quality images . By training the image enhancement network with images collected from a specific model (the same model as the camera mounted on the robot) as the training set, the trained image enhancement network can be more suitable for the camera of this model , corresponding to different types of cameras, the effect of image enhancement can be improved by retraining.
在本申請的一些實施例中，通過與機器人上裝載的攝影頭同一型號採集的圖像為訓練集對圖像增強網路進行訓練，可以得到最佳的圖像增強效果，而在其它解析度近似的其他攝影頭上也有很好的圖像增強效果。In some embodiments of the present application, the image enhancement network can be trained by using images collected by the same model as the camera mounted on the robot as the training set, so that the best image enhancement effect can be obtained. There are also good image enhancements on other similar cameras.
在本申請的一些實施例中，在通過機器人的攝影頭獲取待處理圖像後，可以根據使用者需求確定是否對待處理圖像進行圖像增強處理，可以將未進行圖像增強的資料和進行圖像增強的資料進行儲存，用來進行後續處理。In some embodiments of the present application, after the image to be processed is acquired through the camera of the robot, it can be determined whether to perform image enhancement processing on the image to be processed according to the needs of the user, and the data without image enhancement can be processed and processed. The image-enhanced data is stored for subsequent processing.
神經網路的深度越深，訓練起來的難度就越大，優化神經網路的難度也就越大，如果不能很好的通過訓練學習到合適的權重，深的神經網路的效果反而不如相對較淺的網路，而通過在神經網路中加入殘差塊可解決上述訓練難度大和優化難度大的問題，並提升神經網路的效率。因此，圖像增強網路中包含有一個或多個殘差塊，其中，殘差塊中可以是多層卷積層，也可以是多層全連接層，對此，本申請實施例不做具體限定。The deeper the depth of the neural network, the more difficult it is to train, and the more difficult it is to optimize the neural network. If the appropriate weights cannot be learned through training, the effect of a deep neural network will not be as good as a relative one. For shallower networks, adding residual blocks to the neural network can solve the above-mentioned problems of difficulty in training and optimization, and improve the efficiency of the neural network. Therefore, the image enhancement network includes one or more residual blocks, wherein the residual blocks may be multi-layer convolutional layers or multi-layer fully connected layers, which are not specifically limited in the embodiments of the present application.
在本申請的一些實施例中，圖像增強網路可以包含1個殘差塊，殘差塊包含2層卷積層，且這2層卷積層之間串聯，即上一層卷積層的輸出為下一層卷積層的輸入，這2卷積層依次對待處理圖像進行卷積處理，得到第三特徵資料，再將待處理圖像與第三特徵資料進行融合處理，得到第一特徵資料。在本申請的一些實施例中，上述融合處理可以為特徵資料相加。In some embodiments of the present application, the image enhancement network may include one residual block, and the residual block includes two convolutional layers, and the two convolutional layers are connected in series, that is, the output of the previous convolutional layer is the lower convolutional layer. The input of a layer of convolutional layers, these two convolutional layers perform convolution processing on the image to be processed in turn to obtain the third feature data, and then fuse the to-be-processed image with the third feature data to obtain the first feature data. In some embodiments of the present application, the above-mentioned fusion processing may be feature data addition.
在一種實施方式中，圖像增強網路可以包含34個殘差塊，每個殘差塊包含2層卷積層，所有卷積層之間串聯，即上一層卷積層的輸出為下一層卷積層的輸入，且對每個殘差塊的輸入和輸出進行融合處理，並將融合處理後的特徵資料作為下一個殘差塊的輸入。如：待處理圖像經過第一個殘差塊的處理，得到第六特徵資料，將待處理圖像與第六特徵資料進行融合處理，得到第七特徵資料，第二個殘差塊再對第七特徵資料進行處理，得到第八特徵資料，對第七特徵資料與第八特徵資料進行融合處理，得到第九特徵資料，第三個殘差塊再對第九特徵資料進行處理，以此類推，直到第34個殘差塊輸出第三特徵資料。In one embodiment, the image enhancement network may include 34 residual blocks, each residual block includes 2 convolutional layers, and all convolutional layers are connected in series, that is, the output of the previous convolutional layer is the output of the next convolutional layer. The input and output of each residual block are fused, and the fused feature data is used as the input of the next residual block. For example, the image to be processed is processed by the first residual block to obtain the sixth feature data, the image to be processed and the sixth feature data are fused to obtain the seventh feature data, and the second residual block The seventh feature data is processed to obtain the eighth feature data, the seventh feature data and the eighth feature data are fused to obtain the ninth feature data, and the third residual block processes the ninth feature data, so as to obtain the ninth feature data. And so on, until the 34th residual block outputs the third feature data.
本申請實施例中，待處理圖像可以為向量，上述所有特徵資料也可以為向量，因此，待處理圖像與特徵資料的融合即為向量的融合。In the embodiment of the present application, the image to be processed may be a vector, and all the above-mentioned feature data may also be a vector. Therefore, the fusion of the image to be processed and the feature data is the fusion of vectors.
步驟303、獲取待處理圖像的前一幀圖像。Step 303: Acquire the image of the previous frame of the image to be processed.
在本申請的一些實施例中，當待處理圖像為攝影頭拍攝的圖像時，攝影頭在拍攝待處理圖像之前拍攝得到的圖像即為待處理圖像的前一幀圖像。當待處理圖像為視頻中截取的圖像時，待處理圖像的前一幀圖像即為待處理圖像在視頻中的前一幀圖像。In some embodiments of the present application, when the image to be processed is an image captured by a camera, the image captured by the camera before capturing the image to be processed is the previous frame of the image to be processed. When the image to be processed is an image captured from a video, the image of the previous frame of the image to be processed is the image of the previous frame of the image to be processed in the video.
步驟304、對前一幀圖像進行特徵提取處理，得到第四特徵資料。Step 304: Perform feature extraction processing on the image of the previous frame to obtain fourth feature data.
機器人通過圖像增強網路中的卷積層對前一幀圖像進行特徵提取處理，以從前一幀圖像中提取出第四特徵資料。在一種實施方式中，圖像增強網路可以包括多層卷積層，通過圖像增強網路對前一幀圖像逐層進行卷積處理完成對前一幀圖像的特徵提取處理。其中，每個卷積層提取出的特徵內容及語義資訊均不一樣，可以表現為，特徵提取處理一步步地將圖像的特徵抽象出來，同時也將逐步去除相對次要的特徵，因此，越到後面提取出的特徵尺寸越小，內容及語義資訊就越濃縮。通過多層卷積層逐級對前一幀圖像進行卷積處理，並提取相應的特徵，最終得到固定大小的特徵資料。這樣可在獲得前一幀圖像主要內容資訊（即前一幀圖像的特徵資料）的同時，將圖像尺寸縮小，減小系統的計算量，提高運算速度。在一種實施方式中，通過增強網路中的殘差塊對前一幀圖像進行特徵提取處理，得到第四特徵資料，殘差塊對前一幀圖像的處理過程可參見步驟302，此處將不再贅述。The robot performs feature extraction processing on the previous frame image through the convolution layer in the image enhancement network, so as to extract the fourth feature data from the previous frame image. In one embodiment, the image enhancement network may include multiple convolution layers, and the image enhancement network performs convolution processing on the previous frame of image layer by layer to complete the feature extraction process on the previous frame of image. Among them, the feature content and semantic information extracted by each convolutional layer are different, which can be expressed as the feature extraction process that abstracts the features of the image step by step, and also gradually removes relatively minor features. The smaller the feature size extracted later, the more concentrated the content and semantic information. The image of the previous frame is convolved step by step through the multi-layer convolution layer, and the corresponding features are extracted, and finally the feature data of a fixed size is obtained. In this way, while obtaining the main content information of the previous frame image (ie, the feature data of the previous frame image), the image size can be reduced, the calculation amount of the system can be reduced, and the operation speed can be improved. In one embodiment, the feature extraction process is performed on the image of the previous frame by the residual block in the enhancement network to obtain the fourth feature data. For the process of processing the image of the previous frame by the residual block, please refer to step 302. will not be repeated here.
步驟305、將第一特徵資料與第四特徵資料進行合併處理，得到第五特徵資料。Step 305: Combine the first feature data and the fourth feature data to obtain fifth feature data.
在本申請的一些實施例中，合併處理可以理解為特徵向量的擴充。例如：假設第一特徵資料和第四特徵資料中分別包含7個特徵向量，將第一特徵資料與第四特徵資料進行合併處理，得到的第五特徵資料包含14個特徵向量，且不對向量中的元素做任何處理。In some embodiments of the present application, the merging process can be understood as the expansion of the feature vector. For example: assuming that the first feature data and the fourth feature data contain 7 feature vectors respectively, the first feature data and the fourth feature data are combined, and the obtained fifth feature data contains 14 feature vectors, and no vector elements do any processing.
通過將前一幀的圖像的特徵資料與待處理圖像的特徵資料進行合併，可豐富圖像特徵資訊，更有利於圖像增強，提高增強後的圖像的品質。By merging the feature data of the image of the previous frame with the feature data of the image to be processed, the feature information of the image can be enriched, which is more conducive to image enhancement and improves the quality of the enhanced image.
步驟306、對第五特徵資料進行反卷積處理，得到增強後的圖像。Step 306: Perform deconvolution processing on the fifth feature data to obtain an enhanced image.
卷積層的前向傳播過程相當於反卷積層的反向傳播過程，卷積層的反向傳播過程相當於反卷積層的前向傳播過程，因此可通過對上述第五特徵資料進行反卷積處理，可實現對第五特徵資料的解碼，並得到增強後的圖像。這裡，反卷積層的數量與步驟302中卷積層的數量一致。The forward propagation process of the convolution layer is equivalent to the back propagation process of the deconvolution layer, and the back propagation process of the convolution layer is equivalent to the forward propagation process of the deconvolution layer. , the decoding of the fifth feature data can be realized, and the enhanced image can be obtained. Here, the number of deconvolution layers is the same as the number of convolution layers in
在本申請的一些實施例中，由於機器人的硬體設定有限，因此，可通過對適用於硬體設定較高的平臺（如電腦）的神經網路進行壓縮得到圖像增強網路，壓縮方式可以是知識蒸餾、神經網路剪枝、神經網路量化等等，這樣，圖像增強網路可在機器人上運行，並提高整個本實施例的處理速度。在本申請的一些實施例中，獲取訓練資料和第一神經網路；以所述訓練資料對所述第一神經網路進行訓練，得到第二神經網路；對所述第二神經網路進行剪枝處理，得到第三神經網路；以所述訓練資料對所述第三神經網路進行訓練，得到圖像增強網路。其中，訓練資料包括圖像品質低的圖像和與圖像品質低的圖像對應的圖像品質高的圖像，第一神經網路為壓縮前的神經網路（即適用於硬體設定較高的平臺的神經網路）。In some embodiments of the present application, since the hardware setting of the robot is limited, an image enhancement network can be obtained by compressing a neural network suitable for a platform with a high hardware setting (such as a computer). The compression method It can be knowledge distillation, neural network pruning, neural network quantization, etc. In this way, the image enhancement network can be run on the robot, and the processing speed of the entire present embodiment can be improved. In some embodiments of the present application, training data and a first neural network are obtained; the first neural network is trained with the training data to obtain a second neural network; Perform pruning processing to obtain a third neural network; train the third neural network with the training data to obtain an image enhancement network. The training data includes images with low image quality and images with high image quality corresponding to the images with low image quality, and the first neural network is the neural network before compression (that is, suitable for hardware settings neural network for higher platforms).
應用本實施例，機器人可通過圖像增強網路提高待處理圖像的品質，如：降低圖像的雜訊、提高圖像解析度、提升圖像的清晰度（如因機器人的移動導致拍攝對象不清晰）等等，整個過程均是在獲取到待處理圖像後即時完成的，因此，可提高後續基於增強後的圖像進行相應處理的速度。Using this embodiment, the robot can improve the quality of the image to be processed through the image enhancement network, such as: reducing the noise of the image, improving the resolution of the image, and improving the clarity of the image (such as shooting caused by the movement of the robot). The object is not clear), etc. The whole process is completed immediately after the image to be processed is acquired, so the speed of subsequent corresponding processing based on the enhanced image can be improved.
出於攝影頭成本和資料傳輸成本的考慮，在一些情況下機器人搭載攝影頭的解析度往往不高。在本申請的一些實施例中，可以利用圖像增強網路對低碼率視頻圖像或拍攝品質差的圖像進行增強，可以對光線較暗的畫面中產生的噪點進行消除，另外還可以解決因機器人移動而導致拍攝模糊的問題。Considering the cost of the camera and the cost of data transmission, in some cases, the resolution of the camera mounted on the robot is often not high. In some embodiments of the present application, an image enhancement network can be used to enhance low-bit-rate video images or images with poor shooting quality, to eliminate noise generated in low-light pictures, and to Fixed an issue with blurry shots due to robot movement.
本申請實施例提供了一種應用於機器人上的圖像增強網路，通過圖像增強網路對低品質圖像進行卷積處理，提取出圖像的特徵資料，並根據預先訓練學習到的低品質圖像的特徵與高品質圖像的特徵之間的映射關係，對低品質圖像的特徵資料進行反卷積處理，得到增強後的圖像，整個過程通過圖像增強網路自主、快速實現，無需通過更換攝影頭等硬體設定來提高採集到的待處理圖像的品質，可降低成本。The embodiment of the present application provides an image enhancement network applied to a robot. The image enhancement network performs convolution processing on a low-quality image, extracts the feature data of the image, and obtains the low-quality image according to the pre-training learning. The mapping relationship between the features of high-quality images and the features of high-quality images, the feature data of low-quality images is deconvolved to obtain an enhanced image, and the whole process is autonomous and fast through the image enhancement network. It is realized that the quality of the captured images to be processed does not need to be improved by replacing the hardware settings such as the camera head, and the cost can be reduced.
在本申請的一些實施例中，圖像增強網路可以利用cycleGAN模型，將低品質圖像轉換為與低品質圖像對應的高品質圖像，提高圖像品質；該圖像增強網路可位於機器人終端設備中，能夠增強機器人攝影頭的拍攝效果；此外，由於圖像增強網路可以及時進行反覆運算更新，進而，可以提高演算法的速度和性能。In some embodiments of the present application, the image enhancement network can use the cycleGAN model to convert low-quality images into high-quality images corresponding to the low-quality images to improve image quality; the image enhancement network can It is located in the robot terminal device, which can enhance the shooting effect of the robot camera; in addition, because the image enhancement network can perform repeated calculation updates in time, and further, the speed and performance of the algorithm can be improved.
可以看出，本申請實施例利用深度學習框架進行攝影頭拍攝效果的增強，能夠從運算速度和效果兩方面對待處理圖像的品質進行提升，進一步的，可以根據使用者的不同需求有即時處理和後處理兩種方式可選，保證運算速度的同時得到最佳的品質。It can be seen that the embodiment of the present application uses the deep learning framework to enhance the shooting effect of the camera, which can improve the quality of the image to be processed in terms of operation speed and effect. Further, real-time processing can be performed according to different needs of users. There are two options of post-processing and post-processing, which can ensure the operation speed and get the best quality at the same time.
基於增強後的圖像可對機器人進行後續處理，如：人物跟蹤、循跡行走、自主避障行走。此外，若將本申請實施例應用於教育機器人，還可對機器人的整個行動過程進行儲存，以便複盤。以下實施例為本申請提供的一些實施方式。Based on the enhanced image, the robot can perform subsequent processing, such as: character tracking, tracking walking, and autonomous obstacle avoidance walking. In addition, if the embodiment of the present application is applied to an educational robot, the entire action process of the robot can also be stored for review. The following examples provide some embodiments for this application.
通過機器人內部的深度神經網路（與圖像增強網路不同）基於增強後的圖像進行進一步的控制，提高機器人的控制精度。在本申請的一些實施例中，深度神經網路對增強後的圖像進行人臉特徵提取，根據人臉特徵提取的結果，判斷機器人前方是否有人，如若有人，可以通過機器人內部的語音系統發出如“您好，歡迎光臨”之類的問候語。Through the deep neural network inside the robot (different from the image enhancement network), further control is performed based on the enhanced image to improve the control accuracy of the robot. In some embodiments of the present application, the deep neural network performs facial feature extraction on the enhanced image, and judges whether there is a person in front of the robot according to the result of the facial feature extraction. A greeting such as "Hello and welcome".
在本申請的一些實施例中，機器人可作為足球比賽中的守門員，根據增強後的圖像，得到球的運動軌跡，並根據球的運動軌跡預得到控制資訊。在本申請的一些實施例中，對機器人拍攝的視頻中的每一幀圖像進行圖像增強，得到多幀增強後的圖像，對每一幀增強後的圖像進行特徵提取處理，確定每一幀增強後的圖像中球的位置。再根據每一幀增強後的圖像中球位置，確定球的位置的變化，最終確定球的運動軌跡，並預測球的運動速度，根據球的預測速度和/或球的位置，得到控制資訊，控制機器人做出撲救等動作。在一種實施方式中，對機器人拍攝的視頻中的每一幀圖像進行圖像增強，得到多幀增強後的圖像，根據每一幀增強後的圖像中的灰度值的變化確定球的位置，再根據每一幀增強後的圖像中球位置，確定球的位置的變化，最終確定球的運動軌跡，並預測球的運動速度，根據球的預測速度和/或球的位置，得到控制資訊，控制機器人做出撲救等動作。In some embodiments of the present application, the robot can be used as a goalkeeper in a football game, obtain the trajectory of the ball according to the enhanced image, and pre-obtain control information according to the trajectory of the ball. In some embodiments of the present application, image enhancement is performed on each frame of image in the video shot by the robot to obtain multiple frames of enhanced images, and feature extraction processing is performed on each frame of the enhanced image to determine The position of the ball in the enhanced image for each frame. Then, according to the position of the ball in the enhanced image of each frame, determine the change of the position of the ball, finally determine the trajectory of the ball, and predict the speed of the ball, and obtain the control information according to the predicted speed of the ball and/or the position of the ball , control the robot to make rescue and other actions. In one embodiment, image enhancement is performed on each frame of image in the video shot by the robot to obtain multiple frames of enhanced images, and the ball is determined according to the change of the gray value in each frame of the enhanced image. Then, according to the position of the ball in the enhanced image of each frame, determine the change of the position of the ball, finally determine the trajectory of the ball, and predict the speed of the ball. According to the predicted speed of the ball and/or the position of the ball, Get control information and control the robot to make rescue and other actions.
在一種實施方式中，深度神經網路對增強後的圖像進行手勢特徵提取，所述手勢包括以下至少一種：停止手勢、左轉彎手勢、右轉彎手勢、調頭手勢。根據提取到的手勢特徵，判斷手勢的意義，並根據手勢的意義對機器人進行下一步控制，如：急停、左轉彎、右轉彎、調頭，在本申請的一些實施例中，根據識別結果，得到第一控制資訊，所述第一控制資訊包括速度和/或轉向角，然後根據速度和/或轉向角控制機器人運動。In one embodiment, the deep neural network performs gesture feature extraction on the enhanced image, and the gesture includes at least one of the following: a stop gesture, a left turn gesture, a right turn gesture, and a U-turn gesture. According to the extracted gesture features, determine the meaning of the gesture, and control the robot in the next step according to the meaning of the gesture, such as: emergency stop, left turn, right turn, U-turn, in some embodiments of the present application, according to the recognition result, Obtain first control information, where the first control information includes speed and/or steering angle, and then control the robot to move according to the speed and/or steering angle.
在本申請的一些實施例中，深度神經網路對增強後的圖像進行目標物特徵提取，根據提取到的目標物特徵，判斷目標物所處的位置，並控制機器人完成相應的任務，如：機器人在增強後的圖像中提取出垃圾桶的相關特徵，判定拍攝的圖像中包含目標物：垃圾桶，根據機器人當前位置與垃圾桶之間的距離和/或智慧型機器人與垃圾桶之間的夾角，調整機器人行駛的速度即轉向角，使機器人達到垃圾桶前，並將垃圾倒入垃圾桶。In some embodiments of the present application, the deep neural network performs target feature extraction on the enhanced image, determines the location of the target according to the extracted target features, and controls the robot to complete corresponding tasks, such as : The robot extracts the relevant features of the trash can in the enhanced image, and determines that the captured image contains the target: trash can, according to the distance between the robot's current position and the trash can and/or the intelligent robot and the trash can The angle between the two, adjust the speed of the robot, that is, the steering angle, so that the robot reaches the front of the trash can and dumps the trash into the trash can.
在本申請的一些實施例中，對於用於教學的循跡機器人而言，機器人的行走路線及路線周圍的環境都是確定的，通過對路線和路線周圍的環境進行圖像採集，並將採集到的圖像作為訓練集，以期望的轉向角和/或速度對深度神經網路得到的轉向角和/或速度進行監督對深度神經網路進行訓練，確定深度神經網路的參數，使深度神經網路在預設路線中的每個位置與第二控制資訊之間建立第二映射關係，其中，第二控制資訊包括：速度和/或轉向角。將完成訓練的機器人用於同一個環境（指採集用於訓練的圖像的環境）下進行自主循跡行走時，深度神經網路對即時拍攝到的圖像進行特徵提取，得到特徵圖像，並根據特徵圖像確定智慧型機器人此時所處的位置，並根據得到的位置與第三映射關係得到機器人的速度和/或轉向角，依據速度和/或轉向角完成對機器人的控制。In some embodiments of the present application, for the tracking robot used for teaching, the walking route of the robot and the environment around the route are determined. The obtained images are used as a training set, and the steering angle and/or speed obtained by the deep neural network are supervised at the desired steering angle and/or speed. The deep neural network is trained, and the parameters of the deep neural network are determined to make the depth The neural network establishes a second mapping relationship between each position in the preset route and the second control information, wherein the second control information includes: speed and/or steering angle. When the trained robot is used for autonomous tracking in the same environment (referring to the environment in which images for training are collected), the deep neural network performs feature extraction on the images captured in real time to obtain feature images. The position of the intelligent robot at this time is determined according to the characteristic image, and the speed and/or steering angle of the robot is obtained according to the obtained position and the third mapping relationship, and the robot is controlled according to the speed and/or steering angle.
在本申請的一些實施例中，機器人還可將上述任意一種實現方式中的視頻或圖像進行儲存，如：做守門員的機器人可將正常比賽（小朋友的足球比賽）的視頻儲存下來，後續可通過其他處理平臺（如：電腦）對視頻進行進一步增強處理，以提高視頻的品質。In some embodiments of the present application, the robot may also store videos or images in any of the above implementation manners. For example, a robot serving as a goalkeeper may store a video of a normal game (children's soccer game), which can be used later. The video is further enhanced through other processing platforms (such as computers) to improve the quality of the video.
應用本實施例，基於增強後的圖像對機器人進行後續控制。由於圖像增強網路對待處理圖像的圖像增強處理的速度快，且增強後的圖像品質高，因此，可基於增強後的圖像實現一些回應速度快的控制，如：人物跟蹤、目標物跟蹤、手勢識別等。By applying this embodiment, the robot is subsequently controlled based on the enhanced image. Since the image enhancement processing speed of the image to be processed by the image enhancement network is fast, and the quality of the enhanced image is high, some fast-response controls can be realized based on the enhanced image, such as: person tracking, Target tracking, gesture recognition, etc.
本領域技術人員可以理解，在具體實施方式的上述方法中，各步驟的撰寫順序並不意味著嚴格的執行順序而對實施過程構成任何限定，各步驟的具體執行順序應當以其功能和可能的內在邏輯確定。Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.
可以理解，本申請提及的上述各個方法實施例，在不違背原理邏輯的情況下，均可以彼此相互結合形成結合後的實施例。It can be understood that the above-mentioned method embodiments mentioned in this application can be combined with each other to form a combined embodiment without violating the principle and logic.
此外，本申請實施例還提供了機器人圖像增強裝置、處理器、電子設備、電腦可讀儲存介質和電腦程式，上述均可用來實現本申請實施例提供的任一種機器人圖像增強方法，相應技術方案和描述和參見方法部分的相應記載，不再贅述。In addition, the embodiments of the present application also provide a robot image enhancement device, a processor, an electronic device, a computer-readable storage medium, and a computer program, all of which can be used to implement any one of the robot image enhancement methods provided by the embodiments of the present application. The technical solution and description and the corresponding records in the method section are referred to, and will not be repeated.
上述詳細闡述了本申請實施例的方法，下面提供了本申請實施例的裝置。The methods of the embodiments of the present application are described in detail above, and the apparatuses of the embodiments of the present application are provided below.
請參閱圖4，圖4為本申請實施例提供的一種機器人圖像增強裝置的結構示意圖，該機器人圖像增強裝置1包括：圖像採集部分11、第一特徵提取部分12、解碼處理部分13、第二特徵提取部分14、第一處理部分15、第二處理部分16、第三處理部分17、識別部分18、第四處理部分19、第五處理部分20以及第六處理部分21。其中：
圖像採集部分11，配置為通過機器人的攝影頭獲取待處理圖像；
第一特徵提取部分12，配置為對所述待處理圖像進行特徵提取處理，得到第一特徵資料；
解碼處理部分13，配置為對所述第一特徵資料進行解碼處理，得到增強後的圖像；
第二特徵提取部分14，配置為對所述增強後的圖像進行特徵提取處理，得到第二特徵資料；所述第二特徵資料包括所述預設路線的路線標識的特徵；
第一處理部分15，配置為根據所述第二特徵資料和第一映射關係，得到控制資訊；所述第一映射關係為特徵資料與控制資訊之間的映射關係；所述控制資訊包括速度和/或轉向角；
第二處理部分16，配置為根據所述增強後的圖像，得到所述球的運動軌跡；
第三處理部分17，配置為根據所述運動軌跡，得到控制資訊，所述控制資訊包括速度和/或轉向角；
識別部分18，配置為對所述增強後的圖像進行以下至少一種識別：人臉識別、手勢識別、目標物識別；
第四處理部分19，配置為根據識別結果，得到控制資訊，所述控制資訊包括速度和/或轉向角；
第五處理部分20，配置為對所述待處理圖像依次進行卷積處理、歸一化處理、線性變換、非線性變換，確定所述待處理圖像中的對象所屬類別；
第六處理部分21，配置為根據所述類別和第二映射關係，確定閾值；所述第二映射關係為類別與解析度閾值之間的映射關係；
所述第一特徵提取部分12，還配置為在所述待處理圖像的解析度小於或等於所述閾值的情況下，執行所述對所述待處理圖像進行特徵提取處理，得到第一特徵資料的步驟。Please refer to FIG. 4 . FIG. 4 is a schematic structural diagram of a robot image enhancement device provided by an embodiment of the application. The robot image enhancement device 1 includes: an
進一步地，所述第一特徵提取部分12包括：第一卷積處理子部分121，配置為對所述待處理圖像進行卷積處理，得到第三特徵資料；融合處理子部分122，配置為將所述待處理圖像與所述第三特徵資料進行融合處理，得到所述第一特徵資料。Further, the first feature extraction section 12 includes: a first
進一步地，所述解碼處理部分13包括：獲取子部分131，配置為獲取所述待處理圖像的前一幀圖像；第二卷積處理子部分132，配置為對所述前一幀圖像進行卷積處理，得到第四特徵資料；合併處理子部分133，配置為將所述第一特徵資料與所述第四特徵資料進行合併處理，得到第五特徵資料；解碼處理子部分134，配置為對所述第五特徵資料進行解碼處理，得到所述增強後的圖像。Further, the decoding processing part 13 includes: an
進一步地，所述解碼處理子部分134，配置為：對所述第五特徵資料進行反卷積處理，得到所述增強後的圖像。Further, the
在一些實施例中，本申請實施例提供的裝置具有的功能或包含的模組可以配置為執行上文方法實施例描述的方法，其具體實現可以參照上文方法實施例的描述，為了簡潔，這裡不再贅述。In some embodiments, the functions or modules included in the apparatus provided in the embodiments of the present application may be configured to execute the methods described in the above method embodiments. For specific implementation, reference may be made to the above method embodiments. For brevity, I won't go into details here.
本申請實施例還提出一種電腦可讀儲存介質，其上儲存有電腦程式指令，所述電腦程式指令被處理器執行時實現上述方法。電腦可讀儲存介質可以是非易失性電腦可讀儲存介質。An embodiment of the present application further provides a computer-readable storage medium, which stores computer program instructions, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a non-volatile computer-readable storage medium.
圖5為本申請實施例提供的一種機器人圖像增強的硬體結構示意圖。該圖像增強裝置5包括處理器51，還可以包括輸入裝置52、輸出裝置53和記憶體54。該輸入裝置52、輸出裝置53、記憶體54和處理器51之間通過匯流排相互連接。FIG. 5 is a schematic diagram of a hardware structure of a robot image enhancement provided by an embodiment of the present application. The
記憶體54包括但不限於是隨機儲存記憶體（Random Access Memory，RAM）、唯讀記憶體（Read-Only Memory，ROM）、可擦除可程式設計唯讀記憶體（Erasable Programmable Read Only Memory，EPROM）、或可擕式唯讀記憶體（Compact Disc Read-Only Memory，CD-ROM），該記憶體54配置為相關指令及資料。The
輸入裝置52配置為輸入資料和/或信號，以及輸出裝置53配置為輸出資料和/或信號。輸出裝置53和輸入裝置52可以是獨立的器件，也可以是一個整體的器件。The
處理器51可以包括是一個或多個處理器，例如包括一個或多個中央處理器（Central Processing Unit，CPU），在處理器是一個CPU的情況下，該CPU可以是單核CPU，也可以是多核CPU，對此，本申請實施例不做具體限定。The
記憶體54配置為儲存網路設備的程式碼和資料。The
處理器51配置為調用該記憶體中的程式碼和資料，執行上述方法實施例中的步驟。具體可參見方法實施例中的描述，在此不再贅述。The
圖5僅僅示出了一種機器人圖像增強裝置的簡化設計。在實際應用中，機器人圖像增強裝置還可以分別包含必要的其他元件，包含但不限於任意數量的輸入/輸出裝置、處理器、控制器、記憶體等，而所有可以實現本申請實施例的機器人圖像增強裝置都在本申請的保護範圍之內。Figure 5 shows only a simplified design of a robotic image intensification device. In practical applications, the robot image enhancement device may also include other necessary components, including but not limited to any number of input/output devices, processors, controllers, memories, etc., and all the components that can implement the embodiments of the present application Robotic image enhancement devices are all within the scope of protection of this application.
本領域普通技術人員可以意識到，結合本文中所公開的實施例描述的各示例的部分及演算法步驟，能夠以電子硬體、或者電腦軟體和電子硬體的結合來實現。這些功能究竟以硬體還是軟體方式來執行，取決於技術方案的特定應用和設計約束條件。專業技術人員可以對每個特定的應用來使用不同方法來實現所描述的功能，但是這種實現不應認為超出本申請的範圍。Those of ordinary skill in the art can realize that the parts and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.
所屬領域的技術人員可以清楚地瞭解到，為描述的方便和簡潔，上述描述的系統、裝置和部分的具體工作過程，可以參考前述方法實施例中的對應過程，在此不再贅述。所屬領域的技術人員還可以清楚地瞭解到，本申請各個實施例描述各有側重，為描述的方便和簡潔，相同或類似的部分在不同實施例中可能沒有贅述，因此，在某一實施例未描述或未詳細描述的部分可以參見其他實施例的記載。Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the system, device and part described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here. Those skilled in the art can also clearly understand that the description of each embodiment of the present application has its own emphasis. For the convenience and brevity of the description, the same or similar parts may not be repeated in different embodiments. Therefore, in a certain embodiment For the parts that are not described or not described in detail, reference may be made to the descriptions of other embodiments.
在本申請所提供的幾個實施例中，應該理解到，所揭露的系統、裝置和方法，可以通過其它的方式實現。例如，以上所描述的裝置實施例僅僅是示意性的，例如，所述部分的劃分，僅僅為一種邏輯功能劃分，實際實現時可以有另外的劃分方式，例如多個部分或元件可以結合或者可以集成到另一個系統，或一些特徵可以忽略，或不執行。另一點，所顯示或討論的相互之間的耦合或直接耦合或通信連接可以是通過一些介面，裝置或部分的間接耦合或通信連接，可以是電性，機械或其它的形式。In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the parts is only a logical function division. In actual implementation, there may be other divisions, for example, multiple parts or elements may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or parts, and may be in electrical, mechanical or other forms.
所述作為分離部件說明的部分可以是或者也可以不是物理上分開的，作為部分顯示的部件可以是或者也可以不是物理部分，即可以位於一個地方，或者也可以分佈到多個網路部分上。可以根據實際的需要選擇其中的部分或者全部部分來實現本實施例方案的目的。The parts described as separate parts may or may not be physically separated, and the parts shown as parts may or may not be physical parts, that is, they may be located in one place, or may be distributed over multiple network parts. . Some or all of them may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
另外，在本申請各個實施例中的各功能部分可以集成在一個處理部分中，也可以是各個部分單獨物理存在，也可以兩個或兩個以上部分集成在一個部分中。In addition, each functional part in each embodiment of the present application may be integrated into one processing part, or each part may exist physically alone, or two or more parts may be integrated into one part.
在上述實施例中，可以全部或部分地通過軟體、硬體、固件或者其任意組合來實現。當使用軟體實現時，可以全部或部分地以電腦程式產品的形式實現。所述電腦程式產品包括一個或多個電腦指令。在電腦上載入和執行所述電腦程式指令時，全部或部分地產生按照本申請實施例所述的流程或功能。所述電腦可以是通用電腦、專用電腦、電腦網路、或者其他可程式設計裝置。所述電腦指令可以儲存在電腦可讀儲存介質中，或者通過所述電腦可讀儲存介質進行傳輸。所述電腦指令可以從一個網站網站、電腦、伺服器或資料中心通過有線（例如同軸電纜、光纖、數位用戶線路（Digital Subscriber Line，DSL））或無線（例如紅外、無線、微波等）方式向另一個網站網站、電腦、伺服器或資料中心進行傳輸。所述電腦可讀儲存介質可以是電腦能夠存取的任何可用介質或者是包含一個或多個可用介質集成的伺服器、資料中心等資料存放裝置。所述可用介質可以是磁性介質，(例如，軟碟、硬碟、磁帶)、光介質(例如，數位通用光碟（Digital Versatile Disc，DVD）)、或者半導體介質（例如固態硬碟（Solid State Disk ，SSD））等。In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted over a computer-readable storage medium. The computer instructions can be sent from a web site, computer, server or data center via wired (eg coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (eg infrared, wireless, microwave, etc.) Another website website, computer, server or data center for transmission. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, etc. that includes one or more available mediums integrated. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, Digital Versatile Discs (DVDs)), or semiconductor media (eg, Solid State Disks). , SSD)) etc.
本領域普通技術人員可以理解實現上述實施例方法中的全部或部分流程，該流程可以由電腦程式來指令相關的硬體完成，該程式可儲存於電腦可讀取儲存介質中，該程式在執行時，可包括如上述各方法實施例的流程。而前述的儲存介質包括：ROM或RAM、磁碟或者光碟等各種可儲存程式碼的介質。Those of ordinary skill in the art can understand that to realize all or part of the processes in the methods of the above embodiments, the process can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer-readable storage medium. , the process of each method embodiment described above may be included. The aforementioned storage medium includes: ROM or RAM, magnetic disk or optical disk and other mediums that can store program codes.
工業實用性 本申請實施例提供一種機器人圖像增強方法及裝置、處理器、電子設備和電腦可讀儲存介質，所述方法包括：通過機器人的攝影頭獲取待處理圖像；對所述待處理圖像進行特徵提取處理，得到第一特徵資料；對所述第一特徵資料進行解碼處理，得到增強後的圖像。本申請實施例可提高攝影頭拍攝的圖像的品質。Industrial Applicability Embodiments of the present application provide a robot image enhancement method and device, a processor, an electronic device, and a computer-readable storage medium. The method includes: acquiring an image to be processed through a camera of a robot; The feature extraction process is performed to obtain first feature data; the first feature data is decoded to obtain an enhanced image. The embodiments of the present application can improve the quality of images captured by the camera.
101:步驟
102:步驟
103:步驟
20c:攝影頭
200:機器人
201:人臉圖像
202:機器人圖像增強裝置
301:步驟
302:步驟
303:步驟
304:步驟
305:步驟
306:步驟
1:機器人圖像增強裝置
11:圖像採集部分
12:第一特徵提取部分
121:第一卷積處理子部分
122:融合處理子部分
13:解碼處理部分
131:獲取子部分
132:第二卷積處理子部分
133:合併處理子部分
134:解碼處理子部分
14:第二特徵提取部分
15:第一處理部分
16:第二處理部分
17:第三處理部分
18:識別部分
19:第四處理部分
20:第五處理部分
21:第六處理部分
5:圖像增強裝置
51:處理器
52:輸入裝置
53:輸出裝置
54:記憶體101: Steps
102: Steps
103:
為了更清楚地說明本申請實施例或背景技術中的技術方案，下面將對本申請實施例或背景技術中所需要使用的附圖進行說明。 此處的附圖被併入說明書中並構成本說明書的一部分，這些附圖示出了符合本申請的實施例，並與說明書一起用於說明本申請實施例的技術方案。 圖1為本申請實施例提供的一種機器人圖像增強方法的流程示意圖； 圖2為本申請實施例的一個應用場景的示意圖； 圖3為本申請實施例提供的另一種機器人圖像增強方法的流程示意圖； 圖4為本申請實施例提供的一種機器人圖像增強裝置的結構示意圖； 圖5為本申請實施例提供的一種機器人圖像增強裝置的硬體結構示意圖。In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the accompanying drawings required in the embodiments or the background technology of the present application will be described below. The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, these drawings illustrate the embodiments consistent with the present application, and together with the description, are used to explain the technical solutions of the embodiments of the present application. 1 is a schematic flowchart of a robot image enhancement method provided by an embodiment of the present application; 2 is a schematic diagram of an application scenario of an embodiment of the present application; 3 is a schematic flowchart of another robot image enhancement method provided by an embodiment of the present application; FIG. 4 is a schematic structural diagram of a robot image enhancement device provided by an embodiment of the present application; FIG. 5 is a schematic diagram of a hardware structure of a robot image enhancement device provided by an embodiment of the present application.
101:步驟 101: Steps
102:步驟 102: Steps
103:步驟 103: Steps
Claims (10)
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910645990.7 | 2019-07-17 | ||
| CN201910645990.7A CN110378854B (en) | 2019-07-17 | 2019-07-17 | Robot image enhancement method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| TW202105241A TW202105241A (en) | 2021-02-01 |
| TWI777185B true TWI777185B (en) | 2022-09-11 |
Family
ID=68253674
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| TW109122654A TWI777185B (en) | 2019-07-17 | 2020-07-03 | Robot image enhancement method, processor, electronic equipment, computer readable storage medium |
Country Status (5)
| Country | Link |
|---|---|
| JP (1) | JP2022507399A (en) |
| KR (1) | KR20210079331A (en) |
| CN (1) | CN110378854B (en) |
| TW (1) | TWI777185B (en) |
| WO (1) | WO2021008233A1 (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110378854B (en) * | 2019-07-17 | 2021-10-26 | 上海商汤智能科技有限公司 | Robot image enhancement method and device |
| CN111027403B (en) * | 2019-11-15 | 2023-06-06 | 深圳市瑞立视多媒体科技有限公司 | Gesture estimation method, device, equipment and computer readable storage medium |
| CN110991457B (en) * | 2019-11-26 | 2023-12-08 | 北京达佳互联信息技术有限公司 | Two-dimensional code processing method and device, electronic equipment and storage medium |
| CN111340137A (en) * | 2020-03-26 | 2020-06-26 | 上海眼控科技股份有限公司 | Image recognition method, device and storage medium |
| CN113177486B (en) * | 2021-04-30 | 2022-06-03 | 重庆师范大学 | Dragonfly order insect identification method based on regional suggestion network |
| CN116362976A (en) * | 2021-12-22 | 2023-06-30 | 北京字跳网络技术有限公司 | Fuzzy video restoration method and device |
| CN114463584B (en) * | 2022-01-29 | 2023-03-24 | 北京百度网讯科技有限公司 | Image processing method, model training method, device, apparatus, storage medium, and program |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108737750A (en) * | 2018-06-07 | 2018-11-02 | 北京旷视科技有限公司 | Image processing method, device and electronic equipment |
| CN109685709A (en) * | 2018-12-28 | 2019-04-26 | 深圳市商汤科技有限公司 | A kind of illumination control method and device of intelligent robot |
Family Cites Families (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4235858B2 (en) * | 1999-05-07 | 2009-03-11 | ソニー株式会社 | Robot apparatus and obstacle map generation method for robot apparatus |
| JP3930366B2 (en) * | 2002-04-18 | 2007-06-13 | 株式会社デンソー | White line recognition device |
| JP2005242759A (en) * | 2004-02-27 | 2005-09-08 | National Institute Of Information & Communication Technology | Action/intention presumption system, action/intention presumption method, action/intention pesumption program and computer-readable recording medium with program recorded thereon |
| JP5441626B2 (en) * | 2009-11-06 | 2014-03-12 | 日立オートモティブシステムズ株式会社 | In-vehicle multi-app execution device |
| CN102789234B (en) * | 2012-08-14 | 2015-07-08 | 广东科学中心 | Robot navigation method and robot navigation system based on color coding identifiers |
| JP2015002401A (en) * | 2013-06-14 | 2015-01-05 | オムロン株式会社 | Image processing apparatus, image processing method, and image processing program |
| JP6313646B2 (en) * | 2014-04-24 | 2018-04-18 | 日立オートモティブシステムズ株式会社 | External recognition device |
| WO2016132148A1 (en) * | 2015-02-19 | 2016-08-25 | Magic Pony Technology Limited | Machine learning for visual processing |
| JP6409680B2 (en) * | 2015-05-29 | 2018-10-24 | 株式会社デンソー | Driving support device and driving support method |
| WO2017106998A1 (en) * | 2015-12-21 | 2017-06-29 | Sensetime Group Limited | A method and a system for image processing |
| JP6727642B2 (en) * | 2016-04-28 | 2020-07-22 | 株式会社朋栄 | Focus correction processing method by learning algorithm |
| CN106056562B (en) * | 2016-05-19 | 2019-05-28 | 京东方科技集团股份有限公司 | A kind of face image processing process, device and electronic equipment |
| CN106650690A (en) * | 2016-12-30 | 2017-05-10 | 东华大学 | Night vision image scene identification method based on deep convolution-deconvolution neural network |
| CN107219850A (en) * | 2017-05-25 | 2017-09-29 | 深圳众厉电力科技有限公司 | A kind of automatic Pathfinding system of robot based on machine vision |
| CN108053447A (en) * | 2017-12-18 | 2018-05-18 | 纳恩博（北京）科技有限公司 | Method for relocating, server and storage medium based on image |
| CN108305236B (en) * | 2018-01-16 | 2022-02-22 | 腾讯科技（深圳）有限公司 | Image enhancement processing method and device |
| CN108830816B (en) * | 2018-06-27 | 2020-12-04 | 厦门美图之家科技有限公司 | Image enhancement method and device |
| CN108986050B (en) * | 2018-07-20 | 2020-11-10 | 北京航空航天大学 | Image and video enhancement method based on multi-branch convolutional neural network |
| CN109453500B (en) * | 2018-11-14 | 2020-10-27 | 长春理工大学 | Ball picking robot |
| CN109902723A (en) * | 2019-01-31 | 2019-06-18 | 北京市商汤科技开发有限公司 | Image processing method and device |
| CN109993715A (en) * | 2019-04-11 | 2019-07-09 | 杨勇 | A kind of robot vision image preprocessing system and image processing method |
| CN110378854B (en) * | 2019-07-17 | 2021-10-26 | 上海商汤智能科技有限公司 | Robot image enhancement method and device |
-
2019
- 2019-07-17 CN CN201910645990.7A patent/CN110378854B/en active Active
-
2020
- 2020-05-22 JP JP2021526286A patent/JP2022507399A/en active Pending
- 2020-05-22 WO PCT/CN2020/091905 patent/WO2021008233A1/en active Application Filing
- 2020-05-22 KR KR1020217014996A patent/KR20210079331A/en not_active Application Discontinuation
- 2020-07-03 TW TW109122654A patent/TWI777185B/en active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108737750A (en) * | 2018-06-07 | 2018-11-02 | 北京旷视科技有限公司 | Image processing method, device and electronic equipment |
| CN109685709A (en) * | 2018-12-28 | 2019-04-26 | 深圳市商汤科技有限公司 | A kind of illumination control method and device of intelligent robot |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021008233A1 (en) | 2021-01-21 |
| CN110378854B (en) | 2021-10-26 |
| KR20210079331A (en) | 2021-06-29 |
| TW202105241A (en) | 2021-02-01 |
| CN110378854A (en) | 2019-10-25 |
| JP2022507399A (en) | 2022-01-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI777185B (en) | Robot image enhancement method, processor, electronic equipment, computer readable storage medium | |
| CN107274433B (en) | Target tracking method and device based on deep learning and storage medium | |
| TWI753327B (en) | Image processing method, processor, electronic device and computer-readable storage medium | |
| TWI759668B (en) | Method for video image processing and device thereof | |
| US20220417590A1 (en) | Electronic device, contents searching system and searching method thereof | |
| KR20230013243A (en) | Maintain a fixed size for the target object in the frame | |
| WO2021184972A1 (en) | Image segmentation method and apparatus, electronic device, and storage medium | |
| CN111476709B (en) | Face image processing method and device and electronic equipment | |
| JP7286003B2 (en) | Shuffle, Attend, and Adapt: Video Domain Adaptation with Clip Order Prediction and Clip Attention Adjustment | |
| US20210382542A1 (en) | Screen wakeup method and apparatus | |
| CN104125405B (en) | Interesting image regions extracting method based on eyeball tracking and autofocus system | |
| WO2021098616A1 (en) | Motion posture recognition method, motion posture recognition apparatus, terminal device and medium | |
| JP7100306B2 (en) | Object tracking based on user-specified initialization points | |
| CN113065645A (en) | Twin attention network, image processing method and device | |
| CN111353336B (en) | Image processing method, device and equipment | |
| WO2022083118A1 (en) | Data processing method and related device | |
| CN109035257A (en) | portrait dividing method, device and equipment | |
| CN115035456A (en) | Video denoising method and device, electronic equipment and readable storage medium | |
| Liu et al. | End‐to‐end learning interpolation for object tracking in low frame‐rate video | |
| KR20230093191A (en) | Method for recognizing joint by error type, server | |
| Li et al. | Deep online video stabilization using imu sensors | |
| CN115393963A (en) | Motion action correcting method, system, storage medium, computer equipment and terminal | |
| CN110210306B (en) | Face tracking method and camera | |
| Li et al. | [Retracted] Machine‐Type Video Communication Using Pretrained Network for Internet of Things | |
| Yu et al. | Research on Real-time Video Action Classification Based on Three-Dimensional Convolutional Neural Network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| GD4A | Issue of patent certificate for granted invention patent |