WO2021120455A1 - 图像处理方法、装置、设备和存储介质 - Google Patents
图像处理方法、装置、设备和存储介质 Download PDFInfo
- Publication number
- WO2021120455A1 WO2021120455A1 PCT/CN2020/084018 CN2020084018W WO2021120455A1 WO 2021120455 A1 WO2021120455 A1 WO 2021120455A1 CN 2020084018 W CN2020084018 W CN 2020084018W WO 2021120455 A1 WO2021120455 A1 WO 2021120455A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- processed
- area
- acquisition device
- image processing
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 34
- 238000012545 processing Methods 0.000 claims abstract description 163
- 238000013528 artificial neural network Methods 0.000 claims abstract description 126
- 238000001514 detection method Methods 0.000 claims abstract description 55
- 238000012549 training Methods 0.000 claims abstract description 49
- 230000011218 segmentation Effects 0.000 claims abstract description 32
- 238000000034 method Methods 0.000 claims description 77
- 230000004044 response Effects 0.000 claims description 43
- 239000011159 matrix material Substances 0.000 claims description 32
- 238000004590 computer program Methods 0.000 claims description 17
- 230000015654 memory Effects 0.000 claims description 14
- 238000013507 mapping Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 description 11
- 238000005516 engineering process Methods 0.000 description 5
- 238000002372 labelling Methods 0.000 description 5
- KLDZYURQCUYZBL-UHFFFAOYSA-N 2-[3-[(2-hydroxyphenyl)methylideneamino]propyliminomethyl]phenol Chemical compound OC1=CC=CC=C1C=NCCCN=CC1=CC=CC=C1O KLDZYURQCUYZBL-UHFFFAOYSA-N 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 3
- 201000001098 delayed sleep phase syndrome Diseases 0.000 description 3
- 208000033921 delayed sleep phase type circadian rhythm sleep disease Diseases 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 241001465754 Metazoa Species 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 244000025254 Cannabis sativa Species 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Definitions
- the present disclosure relates to computer vision technology, and in particular to an image processing method, device, electronic equipment, and computer storage medium.
- the embodiments of the present disclosure are expected to provide a technical solution for image processing.
- An embodiment of the present disclosure provides an image processing method, the method including:
- Input the image to be processed into a neural network, which is trained based on sample images in different background environments;
- Image processing is performed on the image to be processed based on the neural network to obtain an image processing result; the image processing result includes a target detection result and/or a semantic segmentation result.
- the image to be processed is acquired by an image acquisition device; the method further includes:
- the method for judging whether there is an obstacle in front of the image acquisition device can determine whether there is an obstacle in front of the image acquisition device, and then take follow-up measures.
- the first preset condition includes at least one of the following:
- the distance between at least one target in the image to be processed and the image acquisition device is less than or equal to a minimum distance safety threshold
- the pixel area value of at least one target in the image to be processed is greater than or equal to the maximum area safety threshold.
- the judgment criterion of the obstacle can be made more in line with actual needs and more in line with user needs.
- the method further includes:
- the image processing result of the image to be processed includes the target detection result
- the relationship between each target in the image to be processed and the image acquisition device is obtained. And/or, according to the target detection result, obtain the pixel area value of each target in the image to be processed; wherein, the homography matrix is used to represent the world coordinate system of each pixel and The position mapping relationship between pixel coordinate systems.
- the target in the image to be processed and the distance between the target and the image acquisition device can be accurately identified on the basis of the target detection result, and each target in the image can be obtained.
- the corresponding pixel area value in pixel coordinates can be obtained.
- the homography matrix is determined according to the internal parameters of the image acquisition device and the known position of the calibration board relative to the image acquisition device.
- the homography matrix can be accurately obtained by determining the homography matrix based on the internal parameters of the image acquisition device and the known position of the calibration board relative to the image acquisition device.
- the image acquisition device is arranged on a mobile carrier, and the method further includes:
- the obstacle avoidance response of the mobile carrier is determined.
- the mobile carrier can perform the corresponding obstacle avoidance response when the mobile carrier encounters an obstacle.
- the determining the obstacle avoidance response of the mobile carrier includes:
- the obstacle avoidance response of the mobile carrier is determined.
- the above method for determining the obstacle avoidance response of the mobile carrier takes into account the types of obstacles. Therefore, the mobile carrier can implement different obstacle avoidance strategies for different obstacles, which is smarter and can better meet the needs of practical applications.
- the background environment includes at least one of the following: lighting conditions and texture background.
- the image to be processed is acquired by an image acquisition device; the method further includes:
- this embodiment can accurately determine whether the image acquisition device has reached the boundary between the workable area and the non-workable area by determining whether the second preset condition is satisfied.
- the second preset condition includes at least one of the following:
- the average pixel height value of the boundary is less than or equal to the boundary pixel height threshold
- the area value of the workable area in the image to be processed is less than or equal to the workable area area threshold
- the area ratio of the workable area in the image to be processed is less than or equal to the workable area area ratio threshold.
- the method further includes:
- the area type of each pixel of the image to be processed is determined according to the result of semantic segmentation, and the workable area is determined according to the determined area type of each pixel And unworkable area; according to the determined workable area and unworkable area, obtain the area value of the workable area in the to-be-processed image, and/or determine the average pixel height value of the boundary.
- this embodiment can obtain the area division of the image to be processed, and more accurately determine the workable area and the non-workable area, and the boundary of the workable area and the non-workable area, so that the area value and the workable area can be obtained later.
- the average pixel height value of the border can be obtained later.
- the image acquisition device is arranged on a mobile carrier, and the method further includes:
- the action response of the moving carrier is determined.
- the action response of the mobile carrier can be determined in time, so as to prevent the mobile carrier from reaching the non-workable area.
- the action response of the moving carrier includes at least one of the following: stopping, turning, and turning around.
- the mobile carrier can perform actions such as stopping, turning, and turning around in a timely manner, which is beneficial to prevent the mobile carrier from moving to an inoperable area.
- the image acquisition device is a monocular image acquisition device.
- the neural network is obtained by training through the following steps:
- the sample image is input into the neural network, and the following steps are performed based on the neural network: image processing is performed on the sample image to obtain an image processing result; the image processing result includes the target detection result and/or the semantic segmentation result; wherein, The sample images are used to represent images in different background environments;
- a neural network that can obtain image processing results is obtained to meet the actual needs for image processing results.
- the training process of the network is realized based on the sample images in different background environments. Therefore, the neural network completed through the training processes the images, and the image processing results obtained are not easily affected by the background environment, and the stability and reliability are high. .
- the method further includes:
- the neural network can perform real-time updates of the neural network according to the tasks of the mobile carrier, thereby being able to adapt to new scenarios and tasks.
- the embodiment of the present disclosure also provides an image processing device, which includes:
- the processing module is configured to input the image to be processed into a neural network, the neural network is trained based on sample images in different background environments; image processing is performed on the image to be processed based on the neural network to obtain an image processing result ;
- the image processing results include target detection results and/or semantic segmentation results.
- the image to be processed is collected by an image acquisition device; the processing module is further configured to determine whether the first preset condition is satisfied according to the image processing result of the image to be processed; In this case, it is determined that there is an obstacle in front of the image acquisition device.
- the method for judging whether there is an obstacle in front of the image acquisition device can determine whether there is an obstacle in front of the image acquisition device, and then take follow-up measures.
- the first preset condition includes at least one of the following:
- the distance between at least one target in the image to be processed and the image acquisition device is less than or equal to a minimum distance safety threshold
- the pixel area value of at least one target in the image to be processed is greater than or equal to the maximum area safety threshold.
- the judgment criterion of the obstacle can be made more in line with actual needs and more in line with user needs.
- the processing module is further configured to obtain the to-be-processed image according to a pre-acquired homography matrix and the target detection result when the image processing result of the image to be processed includes the target detection result.
- the distance value between each target in the image and the image acquisition device; and/or, according to the target detection result, the pixel area value of each target in the image to be processed is obtained respectively; wherein, the homography matrix It is used to represent the position mapping relationship between the world coordinate system of each pixel and the pixel coordinate system.
- the target in the image to be processed and the distance between the target and the image acquisition device can be accurately identified on the basis of the target detection result, and each target in the image can be obtained.
- the corresponding pixel area value in pixel coordinates can be obtained.
- the homography matrix is determined according to the internal parameters of the image acquisition device and the known position of the calibration board relative to the image acquisition device.
- the homography matrix can be accurately obtained by determining the homography matrix through the internal parameters of the image acquisition device and the known position of the calibration board relative to the image acquisition device.
- the image acquisition device is arranged on a mobile carrier, and the processing module is further configured to determine the obstacle avoidance response of the mobile carrier when there is an obstacle in front of the image acquisition device.
- the mobile carrier can perform the corresponding obstacle avoidance response when the mobile carrier encounters an obstacle.
- the processing module is further configured to determine the category of the obstacle according to the result of the image processing; and to determine the movement of the obstacle according to the category of the obstacle.
- the obstacle avoidance response of the carrier is further configured to determine the category of the obstacle according to the result of the image processing; and to determine the movement of the obstacle according to the category of the obstacle.
- the above method for determining the obstacle avoidance response of the mobile carrier takes into account the types of obstacles. Therefore, the mobile carrier can implement different obstacle avoidance strategies for different obstacles, which is smarter and can better meet the needs of practical applications.
- the background environment includes at least one of the following: lighting conditions and texture background.
- the image to be processed is collected by an image acquisition device; the processing module is further configured to determine whether the second preset condition is satisfied according to the image processing result of the image to be processed; Under conditions, it is determined that the image acquisition device reaches the boundary between the workable area and the non-workable area.
- this embodiment can accurately determine whether the image acquisition device has reached the boundary between the workable area and the non-workable area by determining whether the second preset condition is satisfied.
- the second preset condition includes at least one of the following:
- the average pixel height value of the boundary is less than or equal to the boundary pixel height threshold
- the area value of the workable area in the image to be processed is less than or equal to the workable area area threshold
- the area ratio of the workable area in the image to be processed is less than or equal to the workable area area ratio threshold.
- the processing module is further configured to determine the area category of each pixel of the image to be processed according to the semantic segmentation result when the image processing result of the image to be processed includes the semantic segmentation result, and according to the determined
- the area category of each pixel determines the workable area and the non-operable area; according to the determined workable area and non-operable area, the area value of the workable area in the image to be processed is obtained, and/or the boundary is determined
- this embodiment can obtain the area division of the image to be processed, and more accurately determine the workable area and the non-workable area, and the boundary of the workable area and the non-workable area, so that the area value and the workable area can be obtained later.
- the average pixel height value of the border can be obtained later.
- the image acquisition device is arranged on a mobile carrier, and the processing module is further configured to determine an action response of the mobile carrier when the image acquisition device reaches the boundary.
- the action response of the mobile carrier can be determined in time, so as to prevent the mobile carrier from reaching the non-workable area.
- the action response of the moving carrier includes at least one of the following: stopping, turning, and turning around.
- the mobile carrier can perform actions such as stopping, turning, and turning around in a timely manner, which is beneficial to prevent the mobile carrier from moving to an inoperable area.
- the image acquisition device is a monocular image acquisition device.
- the neural network is obtained by training through the following steps:
- the sample image is input into the neural network, and the following steps are performed based on the neural network: image processing is performed on the sample image to obtain an image processing result; the image processing result includes the target detection result and/or the semantic segmentation result; wherein, The sample images are used to represent images in different background environments;
- a neural network that can obtain image processing results is obtained to meet the actual needs for image processing results.
- the training process of the network is realized based on the sample images in different background environments. Therefore, the neural network completed through the training processes the images, and the image processing results obtained are not easily affected by the background environment, and the stability and reliability are high. .
- the processing module is further configured to obtain an annotation of the image to be processed
- the neural network can perform real-time updates of the neural network according to the tasks of the mobile carrier, thereby being able to adapt to new scenarios and tasks.
- the embodiment of the present disclosure also provides an electronic device, including a processor and a memory configured to store a computer program that can run on the processor; wherein,
- any one of the image processing methods described above is executed.
- the embodiments of the present disclosure also provide a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned image processing methods is implemented.
- the embodiments of the present disclosure also provide a computer program, which, when executed by a processor, implements any one of the above-mentioned image processing methods.
- the image processing method in the embodiment of the present disclosure can input the image to be processed into a neural network, which is trained based on sample images in different background environments, and based on the neural network.
- Image processing is performed on the image to be processed to obtain an image processing result; the image processing result includes a target detection result and/or a semantic segmentation result. Since the neural network used in the image processing method is trained based on sample images in different background environments, the image processing results of the image to be processed obtained by the image processing method are not easily affected by the background environment and are stable High sex and credibility.
- FIG. 1 is a flowchart of an image processing method according to an embodiment of the disclosure
- Fig. 2 is a flowchart of a neural network training method according to an embodiment of the disclosure
- FIG. 3 is a schematic diagram of the composition structure of an image processing device according to an embodiment of the disclosure.
- FIG. 4 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.
- the terms "including”, “including” or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or device including a series of elements not only includes the explicitly stated Elements, but also include other elements not explicitly listed, or elements inherent to the implementation of the method or device. Without more restrictions, the element defined by the sentence "including a" does not exclude the existence of other related elements (such as steps or steps in the method) in the method or device that includes the element.
- the unit in the device for example, the unit may be a part of a circuit, a part of a processor, a part of a program or software, etc.).
- the neural network training and image processing methods provided by the embodiments of the present disclosure include a series of steps, but the neural network training and image processing methods provided by the embodiments of the present disclosure are not limited to the recorded steps.
- the embodiments of the present disclosure The provided neural network training and image processing device includes a series of modules, but the device provided by the embodiments of the present disclosure is not limited to include the explicitly recorded modules, and may also include settings required to obtain relevant information or to perform processing based on information. Module.
- the embodiments of the present disclosure can be applied to a computer system composed of hardware or hardware such as terminals and servers, and can be operated with many other general-purpose or special-purpose computing system environments or configurations, or can be implemented by a processor running computer executable code. Disclosure of embodiments.
- the terminal can be a thin client, a thick client, a handheld or laptop device, a microprocessor-based system, a set-top box, a programmable consumer electronic product, a network personal computer, a small computer system, etc.
- the server can be a server computer System small computer system, large computer system and distributed cloud computing technology environment including any of the above systems, etc.
- Electronic devices such as terminals and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system.
- program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types.
- the computer system/server can be implemented in a distributed cloud computing environment. In the distributed cloud computing environment, tasks are executed by remote processing equipment linked through a communication network.
- program modules may be located on a storage medium of a local or remote computing system including a storage device.
- an image processing method is proposed.
- the embodiments of the present disclosure can be applied to any image processing scene, for example, it can be applied to image processing scenes such as outdoor work robots and agricultural robots.
- FIG. 1 is a flowchart of an image processing method according to an embodiment of the disclosure. As shown in FIG. 1, the process may include:
- Step 101 Input the image to be processed into a neural network, which is obtained by training based on sample images in different background environments.
- the image to be processed may be a picture or a video file.
- the processing here may be to determine the file type of the image to be processed, according to the file type of the image to be processed.
- the type of processing file determines the processing operation corresponding to the file type. For example, when the image to be processed is a video file, it is necessary to perform frame selection operations on the image to be processed first to obtain the selected image, and perform image preprocessing operations on the selected image; in the case where the image to be processed is a picture file Next, only perform image preprocessing operations on the image to be processed.
- the image preprocessing operation may be to process the size of the image to be processed, for example, it may be to convert all the images to be processed into images of a fixed size.
- the size of the input image to be processed may be a fixed size.
- the size of the image to be processed The size may be non-fixed.
- the specific size of the image to be processed is not limited, and the size of the image to be processed may be a predetermined fixed size.
- the neural network here refers to a neural network obtained through training, and the training here is implemented based on sample images in different background environments.
- the type of neural network is not limited.
- the neural network can be a single-shot multi-box detector (SSD), you only look once (YOLO), RetinaNet, fast area Convolutional Neural Networks (Faster Region-Convolutional Neural Networks, Faster RCNN) or other neural networks that achieve target detection, can also be Fully Convolutional Networks (Fully Convolutional Networks), U-net, SegNet, DeconvNet, or others to achieve semantic segmentation Neural network.
- the sample images in different background environments can be multiple images acquired in different shooting background environments, and the sample images can be multiple images of the same subject in different background environments, or different subjects in different environments.
- the sample images in different background environments there is no restriction on the subject of the image, as long as the background environment of the sample image is different.
- the embodiment of the present disclosure does not limit the format and source of the sample image.
- the sample image may be a sample image obtained in advance.
- the sample image may be obtained from a local storage area or the network, such as ,
- the sample image can be obtained through the public data set, where the public data set can be VOC data set, COCO data set, etc.; the format of the sample image can be Joint Photographic Experts Group (JPEG) image, bitmap (Bitmap) , BMP), Portable Network Graphics (PNG) or other formats.
- JPEG Joint Photographic Experts Group
- Bitmap Bitmap
- BMP Portable Network Graphics
- the neural network is trained based on sample images in different background environments, and has the ability to obtain the image processing results of the input image. Input the image to be processed into the neural network, and the processed image can be obtained. Image processing result.
- Step 102 Perform image processing on the image to be processed based on the neural network to obtain an image processing result; the image processing result includes a target detection result and/or a semantic segmentation result.
- image processing is performed on the to-be-processed image to obtain the image processing result, which may be a target detection result of the sample image to obtain the target detection result of the image, and/or the semantic segmentation of the sample image to obtain the semantics of the image Segmentation result.
- the target detection result may include a bounding box representing the location and size of the target in the image.
- the bounding box may be a rectangular check box or a check box of other shapes.
- the bounding box is a rectangular check box
- the target detection result may include the pixel coordinate position of the point on the upper left corner of the rectangular detection frame and the length and width of the rectangular detection frame.
- the target detection result may include the point and the upper left corner of the detection frame. Position information such as the pixel coordinate position of the point in the lower right corner.
- the result of semantic segmentation can include the category of each pixel in the image, and different colors can be used to indicate different categories of pixels. For example, all pixels corresponding to roads in the image can be represented by blue. All pixels corresponding to the car can be represented by red, and all pixels corresponding to the lawn in the picture are represented by green. Furthermore, different color areas can be obtained and different objects can be distinguished.
- steps 101 to 102 can be implemented by a processor in an electronic device, and the above-mentioned processor can be an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), Digital signal processing device (Digital Signal Processing Device, DSPD), Programmable Logic Device (Programmable Logic Device, PLD), FPGA, Central Processing Unit (CPU), controller, microcontroller, microprocessor At least one.
- ASIC Application Specific Integrated Circuit
- DSP Digital Signal Processor
- DSPD Digital Signal Processing Device
- PLD Programmable Logic Device
- FPGA Central Processing Unit
- CPU Central Processing Unit
- controller microcontroller
- microprocessor At least one.
- the image to be processed is captured by an image capture device; the image capture device may be a device that can capture images such as a camera.
- the image processing method described above further includes: judging according to the image processing result of the image to be processed Whether the first preset condition is met; if the first preset condition is met, it is determined that there is an obstacle in front of the image capture device; if the first preset condition is not met, it is determined that the image capture device is not in front of the There are obstacles.
- the image to be processed may be an image acquired in real time by the image acquisition device; the image processing result of the image to be processed may refer to the target detection result of the image acquired in real time by the image acquisition device.
- the identified obstacle can be an obstacle in contact with the ground, such as a golf ball, a road cone, or a suspended obstacle, such as a pedestrian lifted. foot.
- the method for judging whether there is an obstacle in front of the image acquisition device can determine whether there is an obstacle in front of the image acquisition device, and then take follow-up measures.
- the first preset condition includes at least one of the following:
- the distance between at least one target in the image to be processed and the image acquisition device is less than or equal to the minimum distance safety threshold
- the pixel area value of at least one target in the image to be processed is greater than or equal to the maximum area safety threshold.
- the distance between at least one target in the image to be processed and the image acquisition device is less than or equal to the minimum distance safety threshold, which may be one or more bounding boxes in all bounding boxes in the image to be processed.
- the distance to the image acquisition device is less than or equal to the minimum distance safety threshold distance.
- the distance between the closest bounding box to the image acquisition device and the acquisition device is less than or equal to the distance safety threshold.
- the distance between the bounding box and the image acquisition device may be the distance between the position in the world coordinate system corresponding to the position point such as the center of the bounding box or the boundary point and the image acquisition device.
- the pixel area value of at least one target in the image to be processed is greater than or equal to the maximum area safety threshold, which may be that the pixel area of one or more bounding boxes in all bounding boxes in the image to be processed is greater than or equal to the maximum
- the safety threshold for example, may be that the pixel area of the bounding box with the largest pixel area is greater than or equal to the maximum safety threshold.
- the minimum distance safety threshold and the maximum area safety threshold are specifically determined according to task requirements and user needs.
- the specific sizes of the minimum distance safety threshold and the maximum area safety threshold are not limited here.
- the judgment criterion of the obstacle can be made more in line with actual needs and more in line with user needs.
- the above-mentioned image processing method further includes: in a case where the image processing result of the image to be processed includes the target detection result, obtaining the image to be processed according to the pre-acquired homography matrix and the target detection result.
- the homography matrix here can be used to represent the relationship between the coordinates of the pixels on the image to be processed and the coordinates of the pixels on the image to be processed mapped on the world coordinate system, that is, through the homography matrix Obtain the coordinates of the pixel with known coordinates in the image to be processed in the world coordinate system, and then obtain the distance between the target corresponding to the pixel with known coordinates and the image acquisition device.
- the homography matrix is determined according to the internal parameters of the image acquisition device and the known position of the calibration board relative to the image acquisition device.
- the homography matrix can be determined in the following manner: First, the image acquisition device collects images when the calibration board is placed in different poses, and the internal parameters of the image acquisition device can be calculated, and then, according to the acquired image acquisition The internal parameters of the device can obtain a homography matrix based on a calibration board (ie an image) placed in a fixed position.
- the internal parameters of the image acquisition device here include at least the focal length and pixel size of the image acquisition device.
- the homography matrix can be accurately obtained by determining the homography matrix through the internal parameters of the image acquisition device and the known position of the calibration board relative to the image acquisition device.
- Using the homography matrix as a solution to obtain the depth matrix of the world coordinate system corresponding to the image pixel position is easy to implement, has a small amount of calculation, and can help the target detection algorithm to quickly obtain the distance of the detected object, which is conducive to obstacle avoidance judgment.
- the target detection result may also include bounding box confidence.
- the bounding box confidence is used to indicate the credibility of the bounding box. The higher the confidence, the higher the credibility of the bounding box.
- the target detection result includes a bounding box with a confidence higher than the first threshold, that is, the target identified by target detection is a target corresponding to the bounding box with a confidence higher than the first threshold.
- the first threshold is not limited
- the first threshold may be 50%, 60%, or 80%.
- the distance value between each target in the image to be processed and the image acquisition device is obtained according to the pre-acquired homography matrix and the target detection result
- the position information of the target in the world coordinate system can be determined according to the above-mentioned bounding box and the position mapping relationship between the world coordinate system and the pixel coordinate system, that is, the position information of the target and the image acquisition device can be determined
- the distance value between is not limited here. Specifically, the distance value between the target and the image acquisition device can be detected by lidar or ultrasound.
- the area of the pixel occupied by all the bounding boxes on the image captured by the image capture device in real time can be obtained separately value.
- the method of obtaining the area value of the pixel occupied by the bounding box on the image is not limited here. Specifically, the area value of the pixel occupied by the bounding box on the image can be determined by the GPS positioning system combined with the homography matrix.
- the image acquisition device is arranged on a mobile carrier, and the mobile carrier may be a smart mobile device such as a mobile robot or a smart lawn mower.
- the method further includes: there is an obstacle in front of the image acquisition device In the case of determining the obstacle avoidance response of the mobile carrier.
- the obstacle avoidance reaction of the mobile carrier may refer to the reaction when the mobile carrier is moving when there is an obstacle in front and cannot proceed in the original direction.
- the obstacle avoidance reaction may be stopping forward and waiting for the obstacle.
- the removal can also be a turn or a U-turn, etc.
- the obstacle avoidance response is not specifically limited here.
- the mobile carrier can perform the corresponding obstacle avoidance response when the mobile carrier encounters an obstacle.
- the determining the obstacle avoidance response of the mobile carrier includes: determining the category of the obstacle according to the image processing result; determining the category of the mobile carrier according to the category of the obstacle Obstacle avoidance response.
- the training content of the target category may be added in the training stage of the neural network, so that the neural network obtained by training can obtain the input image
- the ability of the target category, that is, the image processing result also includes the category of each target in the input image, and furthermore, the category of the obstacle can be determined.
- the types of obstacles may be movable objects such as golf balls, immovable objects such as road cones and sprinklers, and movable characters such as pedestrians.
- the obstacle can be a golf ball. If the mobile carrier is equipped with a robotic arm, the obstacle avoidance response can recover the golf ball; the obstacle is also It can be a road cone or other static objects, and the obstacle avoidance response can also be to keep a safe distance from the static obstacle to bypass; the obstacle can also be a pedestrian, if the pedestrian is moving, the obstacle avoidance response can also be a moving carrier waiting for the person to leave Moving forward, when the moving carrier needs to bypass a stationary pedestrian, the obstacle avoidance response may be that the moving carrier needs to reduce the speed and maintain a larger safety distance to ensure the safety of the pedestrian.
- the above method for determining the obstacle avoidance response of the mobile carrier takes into account the types of obstacles. Therefore, the mobile carrier can implement different obstacle avoidance strategies for different obstacles, which is smarter and can better meet the needs of practical applications.
- the moving carrier may refer to the determination that the moving carrier continues to move in the original direction when the image processing result does not meet the first preset condition, or it may be correct
- the mobile carrier that is on the move does not take any intervention measures.
- the mobile carrier can complete the work or task on time when there are no obstacles in front of the image acquisition device.
- the background environment may include at least one of the following: lighting conditions and texture background.
- the lighting condition may refer to the intensity of lighting or other lighting information
- the texture background may be a linear pattern, a non-linear pattern or other texture background used as a background.
- the image to be processed is captured by an image capture device; the image capture device may be a device that can capture images, such as a camera, a camera, or the like.
- the above image processing method further includes: according to the image of the image to be processed According to the processing result, it is determined whether the second preset condition is satisfied; if the second preset condition is satisfied, it is determined that the image acquisition device has reached the boundary between the workable area and the non-workable area; in the case that the second preset condition is not met Next, it is determined that the image acquisition device has not reached the boundary between the workable area and the non-workable area.
- the image to be processed may be an image acquired in real time by an image acquisition device; the image processing result of the image to be processed may refer to a semantic segmentation result of an image acquired in real time by the image acquisition device.
- this embodiment can accurately determine whether the image acquisition device has reached the boundary between the workable area and the non-workable area by determining whether the second preset condition is satisfied.
- the second preset condition includes at least one of the following: the average pixel height value of the boundary is less than or equal to the boundary pixel height threshold; the area value of the workable area in the image to be processed is less than or equal to the workable area area threshold ; The area ratio of the workable area in the image to be processed is less than or equal to the workable area area ratio threshold.
- the average pixel height value of the boundary may refer to the average value of the distance between the boundary formed by the workable area and the non-workable area and the lower edge of the image. It can be understood that the smaller the average value, the workable area The closer the boundary formed by the unworkable area to the lower edge of the image, that is, the closer the image capture device is to the boundary, at this time, it can be determined that the image capture device has reached the boundary between the workable area and the unworkable area, and if the image capture device moves along the original direction If you move a little before, you may leave the workable area and arrive at the non-workable area.
- the area value of the workable area may refer to the area value of the area occupied by the workable area of the image in the pixel coordinate system.
- the area value of the workable area in the image to be processed is less than or equal to the workable area area threshold, it can be considered that the range of the workable area of the image acquisition device is not large enough.
- the area ratio of the workable area in the image to be processed can refer to the ratio of the area of the workable area to the area of the entire image in the image to be processed, or the ratio of the area of the workable area to the area of the non-workable area. It can also refer to the ratio of the area of the workable area in the image to be processed to the area of the preset total workable area, which is not specifically limited here.
- the area ratio of the workable area in the image to be processed is less than or equal to the workable area area ratio threshold, it indicates that the area of the workable area is relatively small.
- the boundary pixel height threshold, workable area threshold, and workable area area ratio are specifically determined according to task requirements and user needs.
- the boundary pixel height threshold, workable area threshold, and workable area area are not determined here.
- the specific size of the proportion threshold is limited.
- the above-mentioned image processing method further includes: when the image processing result of the image to be processed includes a semantic segmentation result, determining the area category of each pixel of the image to be processed according to the semantic segmentation result, and The determined area category of each pixel determines the workable area and the non-operable area; according to the determined workable area and non-operable area, the area value of the workable area in the image to be processed is obtained, and/or determined The average pixel height value of the boundary.
- the area category of each pixel may refer to whether the specific area to which each pixel belongs is a workable area or a non-workable area.
- the workable area may refer to a space area such as grass that can be cut
- the non-workable area may refer to Areas that cannot be mowed on concrete floors, roads, etc.
- this embodiment can obtain the area division of the image to be processed, and more accurately determine the workable area and the non-workable area, and the boundary of the workable area and the non-workable area, so that the area value and the workable area can be obtained later.
- the average pixel height value of the border can be obtained later.
- the image acquisition device is arranged on a mobile carrier, and the above-mentioned image processing method further includes: determining an action response of the mobile carrier when the image acquisition device reaches the boundary.
- the action response of the mobile carrier can be determined in time, so as to prevent the mobile carrier from reaching the non-workable area.
- the method further includes: determining that the moving carrier continues to move in the original direction when the image acquisition device does not reach the boundary.
- the action response of the moving carrier includes at least one of the following: stopping, turning, and turning around.
- the mobile carrier can perform actions such as stopping, turning, and turning around in a timely manner, which is beneficial to prevent the mobile carrier from moving to an inoperable area.
- the image acquisition device is a monocular image acquisition device.
- a monocular image acquisition device refers to an image acquisition device with a single camera, and, for example, it may be a monocular camera.
- FIG. 2 is a flowchart of a neural network training method according to an embodiment of the present disclosure. As shown in FIG. 2, the above-mentioned neural network is trained through the following steps:
- Step 201 Input a sample image into a neural network, and perform image processing on the sample image based on the neural network to obtain an image processing result;
- the image processing result includes a target detection result and/or a semantic segmentation result;
- the sample images are used to represent images in different background environments;
- Step 202 Adjust network parameter values of the neural network according to the image processing result of the sample image and the label of the sample image;
- Step 203 Determine whether the image processing result obtained by the neural network adjusted based on the network parameter value meets the set condition, if not, re-execute step 201 to step 203; if yes, execute step 204.
- Step 204 Use the neural network with the adjusted network parameter values as the neural network that has been trained.
- the neural network here may be an untrained neural network, or a neural network that has undergone neural network training, but the training does not include the training content of the present disclosure.
- the label of the sample image can be a label box and label information, where the label box is used to frame the target in the sample image, and the label box can also Identify the location of the target. For example, you can mark the target's location such as people, animals, etc. in the sample image through the label box.
- the labeling information is used to label the target category. For example, you can mark whether the target is an object, a person or an animal; Labeling can also be labeling information used to label the category of pixels in the image. Since multiple pixels can be of the same category, it can be labeling information for multiple area categories. For example, it can be used to mark the mowing area and Labeling information for areas where mowing is not allowed.
- the adjustment of the network parameter value of the neural network according to the image processing result of the sample image and the annotation of the sample image may be based on the difference between the image processing result of the sample image and the annotation of the sample image .
- the specific method for determining the loss function value may be determined according to the type of neural network, which is not limited in the embodiment of the present disclosure.
- the setting condition can be that the number of times to adjust the network parameters of the neural network is equal to the set number of iterations, or the loss function of the neural network can reach the convergence condition.
- the setting condition can also be tested on a fixed test set to reach the setting. Set accuracy rate.
- the set number of iterations means the maximum number of times the network parameters of the neural network are adjusted, and the number of iterations is set to an integer greater than 1;
- the convergence condition can be that the value of the loss function of the adjusted neural network is less than the set loss, and the set loss can be Pre-set according to actual application requirements.
- the setting accuracy rate may be a preset percentage value, specifically, the setting The percentage value of can be 50% and a value greater than 50%.
- a neural network that can obtain image processing results is obtained to meet the actual needs for image processing results.
- the training process of the network is realized based on the sample images in different background environments. Therefore, the neural network completed through the training processes the images, and the image processing results obtained are not easily affected by the background environment, and the stability and reliability are high. .
- the image processing method further includes: acquiring the annotation of the image to be processed; according to the image processing result of the image to be processed and the annotation of the image to be processed, On the basis of incremental training.
- incremental training refers to a process of adjusting parameters of the neural network by using newly added data on the basis of the aforementioned neural network.
- the modified loss function can be obtained by adding the loss function of the neural network to the preset regularization term; using the neural network to add data Perform image processing to obtain the image processing result of the sample image; determine the loss of the neural network according to the modified loss function and the label of the newly added data; adjust the network parameters of the neural network according to the loss of the neural network; repeat the above determination of the neural network And the steps of adjusting the network parameters of the neural network until the neural network after the adjustment of the network parameters meets the training end condition, and the trained neural network is obtained.
- the neural network can perform real-time updates of the neural network according to the tasks of the mobile carrier, thereby being able to adapt to new scenarios and tasks.
- an embodiment of the present disclosure proposes an image processing device.
- FIG. 3 is a schematic diagram of the composition structure of an image processing apparatus according to an embodiment of the disclosure.
- the apparatus may include: a processing module 301, wherein:
- the processing module 301 is configured to input the image to be processed into a neural network, the neural network is trained based on sample images in different background environments; image processing is performed on the image to be processed based on the neural network to obtain image processing Results; the image processing results include target detection results and/or semantic segmentation results.
- the processing module 301 is further configured to determine whether a first preset condition is satisfied according to the image processing result of the image to be processed; if the first preset condition is satisfied, determine the image acquisition device There is an obstacle ahead.
- the first preset condition includes at least one of the following:
- the distance between at least one target in the image to be processed and the image acquisition device is less than or equal to a minimum distance safety threshold
- the pixel area value corresponding to at least one target in the image to be processed in the pixel coordinates is greater than or equal to the maximum area safety threshold.
- the processing module 301 is further configured to obtain the to-be-processed image according to a pre-acquired homography matrix and the target detection result when the image processing result of the to-be-processed image includes the target detection result.
- the homography matrix is used to represent the positional mapping relationship between the world coordinate system and the pixel coordinate system.
- the homography matrix is determined according to the internal parameters of the image acquisition device and the known position of the calibration board relative to the image acquisition device.
- the image acquisition device is arranged on a mobile carrier, and the processing module 301 is further configured to determine the obstacle avoidance response of the mobile carrier when there is an obstacle in front of the image acquisition device.
- the processing module 301 is further configured to determine the category of the obstacle according to the image processing result; and determine the obstacle avoidance response of the mobile carrier according to the category of the obstacle.
- the processing module 301 is further configured to determine that the moving carrier continues to move in the original direction when there is no obstacle in front of the image acquisition device.
- the background environment includes at least one of the following: lighting conditions and texture background.
- the image to be processed is collected by an image acquisition device, and the processing module 301 is further configured to determine whether the second preset condition is satisfied according to the image processing result of the image to be processed; Under the condition of setting conditions, it is determined that the image acquisition device reaches the boundary between the workable area and the non-workable area.
- the second preset condition includes at least one of the following:
- the average pixel height value of the boundary is less than or equal to the boundary pixel height threshold
- the area value of the workable area in the image to be processed is less than or equal to the workable area area threshold
- the area ratio of the workable area in the image to be processed is less than or equal to the workable area area ratio threshold.
- the processing module 301 is further configured to determine the area category of each pixel of the image to be processed according to the semantic segmentation result when the image processing result of the image to be processed includes the semantic segmentation result, and according to the result of the semantic segmentation.
- the determined area category of each pixel determines the workable area and the non-operable area; according to the determined workable area and non-operable area, the area value of the workable area in the image to be processed is obtained, and/or the The average pixel height value of the border.
- the image acquisition device is set on a mobile carrier
- the processing module 301 is further configured to determine the action response of the mobile carrier when the image acquisition device reaches the boundary.
- the processing module 301 is further configured to determine that the moving carrier continues to move in the original direction when the image acquisition device does not reach the boundary.
- the action response of the moving carrier includes at least one of the following: stopping, turning, and turning around.
- the image acquisition device is a monocular image acquisition device.
- the neural network is obtained by training through the following steps: inputting a sample image into the neural network, and performing image processing on the sample image based on the neural network to obtain an image processing result;
- the image processing result includes Target detection results and/or semantic segmentation results; wherein the sample images are used to represent images in different background environments;
- the processing module 301 is further configured to obtain the annotation of the image to be processed; according to the image processing result of the image to be processed and the annotation of the image to be processed, the training is completed on the basis of the neural network Incremental training.
- the processing module 301 may be implemented by a processor in an electronic device, and the above-mentioned processor may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.
- the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
- the above-mentioned integrated unit can be realized in the form of hardware or software function module.
- the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer readable storage medium.
- the technical solution of this embodiment is essentially or It is said that the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product.
- the computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can It is a personal computer, a server, or a network device, etc.) or a processor (processor) that executes all or part of the steps of the method described in this embodiment.
- the aforementioned storage media include: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.
- the computer program instructions corresponding to a neural network training method or image processing method in this embodiment can be stored on storage media such as optical disks, hard disks, USB flash drives, etc.
- storage media such as optical disks, hard disks, USB flash drives, etc.
- the embodiment of the present disclosure also provides a computer program, which implements any one of the above-mentioned image processing methods when the computer program is executed by a processor.
- FIG. 4 shows an electronic device 400 provided by an embodiment of the present disclosure, which may include: a memory 401 and a processor 402; wherein,
- the memory 401 is configured to store computer programs and data
- the processor 402 is configured to execute a computer program stored in the memory to implement any one of the image processing methods in the foregoing embodiments.
- the aforementioned memory 401 may be a volatile memory (volatile memory), such as RAM; or a non-volatile memory (non-volatile memory), such as ROM, flash memory, or hard disk (Hard Disk). Drive, HDD) or Solid-State Drive (SSD); or a combination of the foregoing types of memories, and provide instructions and data to the processor 402.
- volatile memory volatile memory
- non-volatile memory non-volatile memory
- ROM read-only memory
- flash memory read-only memory
- HDD hard disk
- SSD Solid-State Drive
- the aforementioned processor 402 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understandable that for different augmented reality cloud platforms, the electronic devices used to implement the above-mentioned processor functions may also be other, and the embodiment of the present disclosure does not specifically limit it.
- the functions or modules contained in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments.
- the functions or modules contained in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments.
- brevity, here No longer refer to the description of the above method embodiments.
- the technical solution of the present disclosure essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes several instructions to make a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present disclosure.
- a terminal which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Geometry (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
- Image Processing (AREA)
Abstract
Description
Claims (35)
- 一种图像处理方法,其中,所述方法包括:将待处理图像输入至神经网络,所述神经网络是基于不同背景环境下的样本图像训练得到的;基于所述神经网络对所述待处理图像进行图像处理,得到图像处理结果;所述图像处理结果包括目标检测结果和/或语义分割结果。
- 根据权利要求1所述的方法,其中,所述待处理图像是由图像采集设备采集的;所述方法还包括:根据所述待处理图像的图像处理结果,判断是否满足第一预设条件;在满足第一预设条件的情况下,确定所述图像采集设备前方存在障碍物。
- 根据权利要求2所述的方法,其中,所述第一预设条件包括以下至少一项:所述待处理图像中的至少一个目标分别与图像采集设备之间的距离值小于或等于最小距离安全阈值;所述待处理图像中的至少一个目标的像素面积值大于或等于最大面积安全阈值。
- 根据权利要求3所述的方法,其中,所述方法还包括:在所述待处理图像的图像处理结果包括目标检测结果的情况下,根据预先获取的单应性矩阵以及所述目标检测结果,分别得到所述待处理图像中的各目标与图像采集设备之间的距离值;和/或,根据所述目标检测结果,分别获得所述待处理图像中的各目标的像素面积值;其中,所述单应性矩阵用于表示各像素点的世界坐标系和像素坐标系之间的位置映射关系。
- 根据权利要求4所述的方法,其中,所述单应性矩阵是根据所述图像采集设备的内部参数以及标定板相对于所述图像采集设备的已知位置确定的。
- 根据权利要求2-5任一项所述的方法,其中,所述图像采集设备设置于移动载体上,所述方法还包括:在所述图像采集设备前方存在障碍物的情况下,确定所述移动载体的避障反应。
- 根据权利要求6所述的方法,其中,所述确定所述移动载体的避障反应,包括:根据所述图像处理结果,确定所述障碍物的类别;根据所述障碍物的类别,确定所述移动载体的避障反应。
- 根据权利要求1-7任一项所述的方法,其中,所述背景环境包括以下至少一项:光照条件、纹理背景。
- 根据权利要求1-7任一项所述的方法,其中,所述待处理图像是由图像采集设备采集的;所述方法还包括:根据所述待处理图像的图像处理结果,判断是否满足第二预设条件;在满足第二预设条件的情况下,确定所述图像采集设备抵达可工作区域与不可工作区域的边界。
- 根据权利要求9所述的方法,其中,所述第二预设条件包括以下至少一项:所述边界的平均像素高度值小于或等于边界像素高度阈值;所述待处理图像中的可工作区域的面积值小于或等于可工作区域面积阈值;所述待处理图像中的可工作区域的面积占比小于或等于可工作区域面积占比阈值。
- 根据权利要求10所述的方法,其中,所述方法还包括:在所述待处理图像的图像处理结果包括语义分割结果的情况下,根据语义分割结果确定所述待处理图像的各像素点的区域类别,根据所确定的各像素点的区域类别确定可作业区域和不可作业区域;根据所确定的可作业区域和不可作业区域,获得所述待处理图像中的可工作区域的面积值,和/或确定所述边界的平均像素高度值。
- 根据权利要求9-11任一项所述的方法,其中,所述图像采集设备设置于移动载体上,所述方法还包括:在所述图像采集设备抵达所述边界的情况下,确定所述移动载体的动作反应。
- 根据权利要求12所述的方法,其中,所述移动载体的动作反应包括以下至少一项:停止、转弯、掉头。
- 根据权利要求2-13任一项所述的方法,其中,所述图像采集设备是单目图像采集设备。
- 根据权利要求1-14任一项所述的方法,其中,所述神经网络是通过以下步骤训练得到的:将样本图像输入至神经网络中,基于所述神经网络执行以下步骤:对所述样本图像进行图像处理,得到图像处理结果;所述图像处理结果包括目标检测结果和/或语义分割结果;其中,所述样本图像用于表示不同背景环境下的图像;根据所述样本图像的图像处理结果以及所述样本图像的标注,调整所述神经网络的网络参数值;重复执行上述步骤,直至网络参数值调整后的神经网络满足设定条件,得到训练完成的神经网络。
- 根据权利要求15所述的方法,其中,所述方法还包括:获取所述待处理图像的标注;根据所述待处理图像的图像处理结果以及所述待处理图像的标注,在所述训练完成的神经网络的基础上进行增量训练。
- 一种图像处理装置,其中,所述装置包括:处理模块,其中,处理模块,配置为将待处理图像输入至神经网络,所述神经网络是基于不同背景环境下的样本图像训练得到的;基于所述神经网络对所述待处理图像进行图像处理,得到图像处理结果;所述图像处理结果包括目标检测结果和/或语义分割结果。
- 根据权利要求17所述的装置,其中,所述待处理图像是由图像采集设备采集的;所述处理模块,还配置为根据所述待处理图像的图像处理结果,判断是否满足第一预设条件;在满足第一预设条件的情况下,确定所述图像采集设备前方存在障碍物。
- 根据权利要求18所述的装置,其中,所述第一预设条件包括以下至少一项:所述待处理图像中的至少一个目标分别与图像采集设备之间的距离值小于或等于最小距离安全阈值;所述待处理图像中的至少一个目标在像素坐标上对应的像素面积值大于或等于最大面积安全阈值。
- 根据权利要求19所述的装置,其中,所述处理模块还配置为在所述待处理图像的图像处理结果包括目标检测结果的情况下,根据预先获取的单应性矩阵以及所述目标检测结果,分别得到所述待处理图像中的各目标与图像采集设备之间的距离 值;和/或根据所述目标检测结果,分别获得所述待处理图像中的各目标在像素坐标上对应的像素面积值;其中,所述单应性矩阵用于表示世界坐标系和像素坐标系之间的位置映射关系。
- 根据权利要求20所述的装置,其中,所述单应性矩阵是根据所述图像采集设备的内部参数以及标定板相对于所述图像采集设备的已知位置确定的。
- 根据权利要求18-21任一项所述的装置,其中,所述图像采集设备设置于移动载体上,所述处理模块还配置为在所述图像采集设备前方存在障碍物的情况下,确定所述移动载体的避障反应。
- 根据权利要求22所述的装置,其中,所述处理模块还配置为根据所述图像处理结果,确定所述障碍物的类别;根据所述障碍物的类别,确定所述移动载体的避障反应。
- 根据权利要求17-23任一项所述的装置,其中,所述背景环境包括以下至少一项:光照条件、纹理背景。
- 根据权利要求17-23任一项所述的装置,其中,所述待处理图像是由图像采集设备采集的,所述处理模块还配置为根据所述待处理图像的图像处理结果,判断是否满足第二预设条件;在满足第二预设条件的情况下,确定所述图像采集设备抵达可工作区域与不可工作区域的边界。
- 根据权利要求25所述的装置,其中,所述第二预设条件包括以下至少一项:所述边界的平均像素高度值小于或等于边界像素高度阈值;所述待处理图像中的可工作区域的面积值小于或等于可工作区域面积阈值;所述待处理图像中的可工作区域的面积占比小于或等于可工作区域面积占比阈值。
- 根据权利要求26所述的装置,其中,所述处理模块还配置为在所述待处理图像的图像处理结果包括语义分割结果的情况下,根据语义分割结果确定所述待处理图像的各像素点的区域类别,根据所确定的各像素点的区域类别确定可作业区域和不可作业区域;根据所确定的可作业区域和不可作业区域,获得所述待处理图像中的可工作区域的面积值,和/或确定所述边界的平均像素高度值。
- 根据权利要求25-27任一项所述的装置,其中,所述图像采集设备设置于移动载体上,所述处理模块还配置为在所述图像采集设备抵达所述边界的情况下,确定所述移动载体的动作反应。
- 根据权利要求28所述的装置,其中,所述移动载体的动作反应包括以下至少一项:停止、转弯、掉头。
- 根据权利要求18至29任一项所述的装置,其中,所述图像采集设备是单目图像采集设备。
- 根据权利要求17-30任一项所述的装置,其中,所述神经网络是通过以下步骤训练得到的:将样本图像输入至神经网络中,基于所述神经网络对所述样本图像进行图像处理,得到图像处理结果;所述图像处理结果包括目标检测结果和/或语义分割结果;其中,所述样本图像用于表示不同背景环境下的图像;根据所述样本图像的图像处理结果以及所述样本图像的标注,调整所述神经网络的网络参数值;重复执行上述步骤,直至网络参数值调整后的神经网络满足设定条件,得到训练完成的神经网络。
- 根据权利要求31所述的装置,其中,所述处理模块还配置为获取所述待处理图像的标注;根据所述待处理图像的图像处理结果以及所述待处理图像的标注,在 所述训练完成的神经网络的基础上进行增量训练。
- 一种电子设备,其中,包括处理器和配置为存储能够在处理器上运行的计算机程序的存储器;其中,所述处理器配置为运行所述计算机程序时,执行权利要求1-16任一项所述的图像处理方法。
- 一种计算机存储介质,其上存储有计算机程序,其中,该计算机程序被处理器执行时实现权利要求1-16任一项所述的图像处理方法。
- 一种计算机程序,包括计算机可读代码,当所述计算机可读代码在电子设备中运行时,所述电子设备中的处理器执行用于实现权利要求1-16中的任一权利要求所述的方法。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020217014432A KR20210081364A (ko) | 2019-12-20 | 2020-04-09 | 이미지 처리 방법, 장치, 기기 및 저장 매체 |
JP2021526557A JP2022518324A (ja) | 2019-12-20 | 2020-04-09 | 画像処理方法、装置、機器及び記憶媒体 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911328268.7A CN111126251B (zh) | 2019-12-20 | 2019-12-20 | 图像处理方法、装置、设备和存储介质 |
CN201911328268.7 | 2019-12-20 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021120455A1 true WO2021120455A1 (zh) | 2021-06-24 |
Family
ID=70500812
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/084018 WO2021120455A1 (zh) | 2019-12-20 | 2020-04-09 | 图像处理方法、装置、设备和存储介质 |
Country Status (5)
Country | Link |
---|---|
JP (1) | JP2022518324A (zh) |
KR (1) | KR20210081364A (zh) |
CN (1) | CN111126251B (zh) |
TW (1) | TW202125316A (zh) |
WO (1) | WO2021120455A1 (zh) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113345034A (zh) * | 2021-07-14 | 2021-09-03 | 山东诺德能源科技有限公司 | 一种利用图像识别方法快速测量钢筋直径的方法 |
CN113611079A (zh) * | 2021-07-30 | 2021-11-05 | 北京市商汤科技开发有限公司 | 提示方法及装置、电子设备和存储介质 |
CN113706561A (zh) * | 2021-10-29 | 2021-11-26 | 华南理工大学 | 一种基于区域分离的图像语义分割方法 |
CN114339046A (zh) * | 2021-12-30 | 2022-04-12 | 中元汇吉生物技术股份有限公司 | 基于自动旋转试管的图像采集方法、装置、设备及介质 |
CN116132645A (zh) * | 2023-01-06 | 2023-05-16 | 清华大学 | 图像处理方法、装置、设备和介质 |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112102378B (zh) * | 2020-08-06 | 2024-06-11 | 深圳市优必选科技股份有限公司 | 图像配准方法、装置、终端设备及计算机可读存储介质 |
CN112183476B (zh) * | 2020-10-28 | 2022-12-23 | 深圳市商汤科技有限公司 | 一种障碍检测方法、装置、电子设备以及存储介质 |
CN112418219A (zh) * | 2020-11-24 | 2021-02-26 | 广东工业大学 | 一种服装面料裁片的颜色和形状识别方法及相关装置 |
CN112465794A (zh) * | 2020-12-10 | 2021-03-09 | 无锡卡尔曼导航技术有限公司 | 一种基于YOLOv4和嵌入式平台的高尔夫球检测方法 |
CN112651931B (zh) * | 2020-12-15 | 2024-04-26 | 浙江大华技术股份有限公司 | 建筑物变形监测方法、装置和计算机设备 |
CN112561897A (zh) * | 2020-12-22 | 2021-03-26 | 电子科技大学中山学院 | 一种基于U-Net的光子晶体光纤端面结构提取方法 |
CN112862776A (zh) * | 2021-02-02 | 2021-05-28 | 中电鸿信信息科技有限公司 | 基于ar和多重语义分割的智能测量方法 |
CN113910225A (zh) * | 2021-10-09 | 2022-01-11 | 邦鼓思电子科技(上海)有限公司 | 一种基于视觉边界检测的机器人控制***及方法 |
TWI823491B (zh) * | 2022-07-22 | 2023-11-21 | 鴻海精密工業股份有限公司 | 深度估計模型的優化方法、裝置、電子設備及存儲介質 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108321722A (zh) * | 2018-03-08 | 2018-07-24 | 贵州电网有限责任公司 | 自动避障的可纵向弯曲树障清理空中机器人和避障方法 |
US20180253866A1 (en) * | 2017-03-03 | 2018-09-06 | General Electric Company | Image analysis neural network systems |
CN109063575A (zh) * | 2018-07-05 | 2018-12-21 | 中国计量大学 | 一种基于单目视觉的智能割草机自主有序割草方法 |
CN110228484A (zh) * | 2019-06-17 | 2019-09-13 | 福州视驰科技有限公司 | 一种具有辅助驾驶功能的低时延智能远程驾驶*** |
CN110502019A (zh) * | 2019-09-06 | 2019-11-26 | 北京云迹科技有限公司 | 一种室内机器人的避障方法及装置 |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2007148835A (ja) * | 2005-11-28 | 2007-06-14 | Fujitsu Ten Ltd | 物体判別装置、報知制御装置、物体判別方法および物体判別プログラム |
JP2016181068A (ja) * | 2015-03-24 | 2016-10-13 | 株式会社明電舎 | 学習サンプル撮影装置 |
CN108303747B (zh) * | 2017-01-12 | 2023-03-07 | 清华大学 | 检查设备和检测***的方法 |
CN107301646B (zh) * | 2017-06-27 | 2019-09-17 | 深圳市云洲创新科技有限公司 | 基于单目视觉的无人船智能避障方法和装置 |
CN107895378A (zh) * | 2017-10-12 | 2018-04-10 | 西安天和防务技术股份有限公司 | 目标检测方法和装置、存储介质、电子设备 |
JP6621220B2 (ja) * | 2018-02-09 | 2019-12-18 | 株式会社ビコー | 情報処理装置、情報処理方法およびプログラム |
CN110245710B (zh) * | 2019-06-18 | 2022-11-29 | 腾讯科技(深圳)有限公司 | 语义分割模型的训练方法、语义分割方法及装置 |
-
2019
- 2019-12-20 CN CN201911328268.7A patent/CN111126251B/zh active Active
-
2020
- 2020-04-09 KR KR1020217014432A patent/KR20210081364A/ko active IP Right Grant
- 2020-04-09 WO PCT/CN2020/084018 patent/WO2021120455A1/zh active Application Filing
- 2020-04-09 JP JP2021526557A patent/JP2022518324A/ja active Pending
- 2020-05-22 TW TW109117087A patent/TW202125316A/zh unknown
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180253866A1 (en) * | 2017-03-03 | 2018-09-06 | General Electric Company | Image analysis neural network systems |
CN108321722A (zh) * | 2018-03-08 | 2018-07-24 | 贵州电网有限责任公司 | 自动避障的可纵向弯曲树障清理空中机器人和避障方法 |
CN109063575A (zh) * | 2018-07-05 | 2018-12-21 | 中国计量大学 | 一种基于单目视觉的智能割草机自主有序割草方法 |
CN110228484A (zh) * | 2019-06-17 | 2019-09-13 | 福州视驰科技有限公司 | 一种具有辅助驾驶功能的低时延智能远程驾驶*** |
CN110502019A (zh) * | 2019-09-06 | 2019-11-26 | 北京云迹科技有限公司 | 一种室内机器人的避障方法及装置 |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113345034A (zh) * | 2021-07-14 | 2021-09-03 | 山东诺德能源科技有限公司 | 一种利用图像识别方法快速测量钢筋直径的方法 |
CN113611079A (zh) * | 2021-07-30 | 2021-11-05 | 北京市商汤科技开发有限公司 | 提示方法及装置、电子设备和存储介质 |
CN113706561A (zh) * | 2021-10-29 | 2021-11-26 | 华南理工大学 | 一种基于区域分离的图像语义分割方法 |
CN114339046A (zh) * | 2021-12-30 | 2022-04-12 | 中元汇吉生物技术股份有限公司 | 基于自动旋转试管的图像采集方法、装置、设备及介质 |
CN114339046B (zh) * | 2021-12-30 | 2023-10-03 | 中元汇吉生物技术股份有限公司 | 基于自动旋转试管的图像采集方法、装置、设备及介质 |
CN116132645A (zh) * | 2023-01-06 | 2023-05-16 | 清华大学 | 图像处理方法、装置、设备和介质 |
Also Published As
Publication number | Publication date |
---|---|
KR20210081364A (ko) | 2021-07-01 |
CN111126251B (zh) | 2022-08-02 |
CN111126251A (zh) | 2020-05-08 |
JP2022518324A (ja) | 2022-03-15 |
TW202125316A (zh) | 2021-07-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021120455A1 (zh) | 图像处理方法、装置、设备和存储介质 | |
US11878433B2 (en) | Method for detecting grasping position of robot in grasping object | |
US20210390329A1 (en) | Image processing method, device, movable platform, unmanned aerial vehicle, and storage medium | |
TWI726278B (zh) | 行車偵測方法、車輛及行車處理裝置 | |
WO2022088611A1 (zh) | 障碍检测方法、装置、电子设备、存储介质及计算机程序 | |
CN106650701A (zh) | 基于双目视觉的室内阴影环境下障碍物检测方法及装置 | |
CN111612823A (zh) | 一种基于视觉的机器人自主跟踪方法 | |
WO2023173950A1 (zh) | 障碍物检测方法、移动机器人及机器可读存储介质 | |
US20210216767A1 (en) | Method and computing system for object recognition or object registration based on image classification | |
CN112605993A (zh) | 基于双目视觉引导的自动抓取档案机器人控制***及方法 | |
JP7153264B2 (ja) | 画像解析システム、画像解析方法及び画像解析プログラム | |
CN109543634B (zh) | 定位过程中的数据处理方法、装置、电子设备和存储介质 | |
CN112784639A (zh) | 路口检测、神经网络训练及智能行驶方法、装置和设备 | |
WO2022061632A1 (zh) | 障碍物检测方法、装置、无人机和存储介质 | |
CN113435367A (zh) | 社交距离评估方法、装置及存储介质 | |
CN116503566B (zh) | 一种三维建模方法、装置、电子设备及存储介质 | |
CN116912517B (zh) | 相机视野边界的检测方法及装置 | |
WO2023216555A1 (zh) | 基于双目视觉的避障方法、装置、机器人及介质 | |
WO2022052853A1 (zh) | 目标跟踪方法、装置、设备及计算机可读存储介质 | |
CN115880643A (zh) | 一种基于目标检测算法的社交距离监测方法和装置 | |
CN115902977A (zh) | 基于视觉和gps的变电站机器人双重定位方法及*** | |
Ferreira et al. | Human detection and tracking using a Kinect camera for an autonomous service robot | |
Velazquez et al. | Using Autonomous Drone Interactions towards MobilePersonal Spaces for Indoor Environments | |
CN114333345B (zh) | 停车位被遮挡时的预警方法、设备、存储介质及程序产品 | |
Tian | Effective image enhancement and fast object detection for improved UAV applications |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
ENP | Entry into the national phase |
Ref document number: 20217014432 Country of ref document: KR Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2021526557 Country of ref document: JP Kind code of ref document: A |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20903948 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 19.10.2022) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20903948 Country of ref document: EP Kind code of ref document: A1 |