WO2017117743A1 - 处理图像的方法和设备 - Google Patents

处理图像的方法和设备 Download PDF

Info

Publication number
WO2017117743A1
WO2017117743A1 PCT/CN2016/070275 CN2016070275W WO2017117743A1 WO 2017117743 A1 WO2017117743 A1 WO 2017117743A1 CN 2016070275 W CN2016070275 W CN 2016070275W WO 2017117743 A1 WO2017117743 A1 WO 2017117743A1
Authority
WO
WIPO (PCT)
Prior art keywords
phase
windows
input image
length
phase pixel
Prior art date
Application number
PCT/CN2016/070275
Other languages
English (en)
French (fr)
Inventor
温华栋
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to US16/068,372 priority Critical patent/US20190012797A1/en
Priority to CN201680009659.0A priority patent/CN107211095B/zh
Priority to PCT/CN2016/070275 priority patent/WO2017117743A1/zh
Priority to EP16882888.7A priority patent/EP3389256B1/en
Publication of WO2017117743A1 publication Critical patent/WO2017117743A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/703—SSIS architectures incorporating pixels for producing signals other than image signals
    • H04N25/704—Pixels specially adapted for focusing, e.g. phase difference pixel sets
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
    • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/672—Focus control based on electronic image sensor signals based on the phase difference signals

Definitions

  • Embodiments of the present invention relate to the field of image processing technologies, and, more particularly, to a method and apparatus for processing an image.
  • the depth image reflects the depth information of the image.
  • the depth information indicates the distance between the object in the image and the camera.
  • the pixels of the depth image can be used to reflect the distance information between the corresponding area and the camera.
  • a common way to obtain a depth image is by taking multiple photos with different positions as the focus.
  • it can be implemented by a dual camera.
  • a dual camera refers to a camera with two separate image sensors. Use each image sensor to take a photo, one with a focus on the foreground and the other with a close-up view. Depth images can be generated from these two photos. But the cost of a dual camera is very high.
  • Another way to obtain depth images is based on a time-of-flight system approach.
  • the light unit is used to illuminate an object that needs to be photographed.
  • Another independent sensor captures the light and calculates the time it takes for the light to reach the target. Depending on the transmission time of the light, the distance of the target object can be calculated and a depth image can be generated.
  • the acquisition method of the depth image is complicated or the equipment for acquiring the depth image is expensive.
  • Embodiments of the present invention provide a method and an apparatus for processing an image, which can provide a simple manner of acquiring a depth image.
  • an embodiment of the present invention provides a method for processing an image, the method comprising: acquiring an input image, where the input image includes a plurality of normal pixels and a plurality of phase pixel pairs, each phase of the plurality of phase pixel pairs
  • the pixel pair includes a first phase pixel which is a phase pixel blocked by the left side, and a second phase pixel which is a phase pixel blocked by the right side;
  • the input image is divided into at least two area windows, each of the at least two area windows including at least two adjacent phase pixel pairs of the plurality of phase pixel pairs; according to the at least two area windows
  • a phase difference corresponding to each of the area windows is determined by at least two phase pixel pairs of each area window; and a depth image corresponding to the input image is determined according to a phase difference corresponding to each of the area windows.
  • the input image is acquired by using an image sensor capable of acquiring phase pixels. Based on the phase pixels, a depth image can be determined.
  • the above technical solution does not need to take multiple input images, and does not need to use other auxiliary equipment to obtain depth images.
  • the first phase pixel and the second phase pixel are respectively located in adjacent pixel rows, and the second phase pixel is located in an adjacent column on the right side of the first phase pixel.
  • the dividing the input image into the at least two area windows comprises: extending the input image by the first direction by using the first length At least a portion of the portion is divided into at least two area windows having the same size, the first direction being the horizontal direction of the input image or the vertical direction of the input image.
  • the dividing the input image into the at least one area window further includes: using the second length as a step And dividing the input image into at least a portion of the second image in the second direction, the second direction being perpendicular to the first direction.
  • the at least two phases according to each of the at least two area windows Determining a phase difference corresponding to each of the area windows, comprising: determining, according to at least two phase pixel pairs of each of the at least two area windows, a first phase pixel and a The cross-correlation of the two phase pixels; determining the phase difference corresponding to each of the region windows according to the cross-correlation of the first phase pixel and the second phase pixel in the window of each region.
  • an embodiment of the present invention provides an apparatus for processing an image, the apparatus comprising: means for performing the method provided by the first aspect.
  • an embodiment of the present invention provides an image processing device, where the device includes an image sensor and a processor.
  • the image sensor and the processor are for performing the method provided by the first aspect.
  • an embodiment of the present invention provides a computer readable storage medium, the program stored in the computer readable storage medium comprising instructions for executing the method provided by the first aspect.
  • an embodiment of the present invention provides an apparatus for processing an image, where the apparatus includes the computer readable storage medium of the third aspect, and a processor.
  • a processor is operative to execute instructions of a program stored in the computer readable storage medium to perform processing of the input image.
  • the first length is greater than or equal to the distance between two adjacent phase pixels extending in the first direction. In this way, it can be ensured that different phase pixel pairs exist in any two adjacent area windows in the first direction. Further, the first length may also be smaller than the length of the first region window in the first direction. In this way, it can be ensured that the same phase pixel pair exists in any two adjacent area windows in the first direction.
  • the second length is greater than or equal to the distance between two adjacent phase pixels extending in the second direction. In this way, it is guaranteed that different phase pixel pairs exist in any two adjacent area windows in the second direction. Further, the second length may also be smaller than the length of the window in each of the regions extending in the second direction. In this way, it can be ensured that the same phase pixel pair exists in any two adjacent area windows in the second direction.
  • Figure 1 is a schematic diagram of an input image.
  • FIG. 2 is a schematic flowchart of a method for processing an image according to an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of dividing the input image into four area windows in a first direction.
  • FIG. 4 is a schematic diagram of dividing the input image into six area windows by extending the first direction and the second direction at the same time.
  • FIG. 5 is a schematic diagram of determining a depth map using a phase difference.
  • FIG. 6 is a structural block diagram of an apparatus for processing an image according to an embodiment of the present invention.
  • FIG. 7 is a structural block diagram of a device according to an embodiment of the present invention.
  • Phase focusing is a method of focusing using a special photosensitive element in an image sensor.
  • FIG. 1 is a schematic illustration of an image sensor capable of phase focusing.
  • a plurality of photosensitive elements are evenly distributed in the image sensor shown in FIG.
  • There is a special photosensitive element in the photosensitive element which is a common photosensitive element that is half blocked.
  • Phase focusing is officially implemented using the signal acquired by this particular photosensitive element to calculate the phase difference.
  • the above-mentioned signal is obtained by a plurality of photosensitive elements. What is obtained by each photosensitive element is only a sample in the signal.
  • the sampling obtained by the special photosensitive element for phase focusing is hereinafter referred to as phase pixel
  • the sample acquired by the ordinary photosensitive element is referred to as a normal pixel
  • phase pixels can be obtained for phase focusing.
  • the image sensor is called an image sensor with phase pixels.
  • the phase pixel may also be divided into a first phase pixel which is obtained by a photosensitive element that is blocked on the left side, and a second phase pixel that is a photosensitive element that is blocked by the right side. acquired.
  • the input image referred to in the present invention is an original signal obtained by an image sensor having phase pixels, and is not a photograph obtained through a series of subsequent processes.
  • One of the image sensors having phase pixels corresponds to one pixel in the input image.
  • the pixels in the input image are phase pixels and normal pixels. Therefore, FIG. 1 can also be regarded as a schematic diagram of an input image.
  • the input image includes a plurality of normal and a plurality of phase pixel pairs, each of the plurality of phase pixel pairs including a first phase pixel and a second phase pixel.
  • the first phase pixel and the second phase pixel of each phase pixel pair in the input image shown in FIG. 1 are respectively located in adjacent pixel rows, and the second phase pixel is located in an adjacent column on the right side of the first phase pixel.
  • the arrangement of the phase pixel pairs shown in FIG. 1 is only one embodiment, and the phase pixel pairs may have other arrangements, which are not limited in the present invention. However, the closer the distance between the two phase pixels constituting one phase pixel pair, the higher the accuracy of calculating the phase difference.
  • FIG. 2 is a schematic flowchart of a method for processing an image according to an embodiment of the present invention.
  • Phase Difference a phase difference corresponding to each of the area windows
  • the input image is acquired using an image sensor capable of acquiring phase pixels. Based on the phase pixels in each area window, the phase difference of the area window can be determined. This will determine the depth image.
  • the above technical solution does not need to take multiple input images, and does not need to use other auxiliary equipment to obtain depth images.
  • dividing the input image into at least two area windows includes: dividing at least a portion of the input image in the first direction into at least two of the same size by using the first length as a step size
  • the area window is the horizontal direction of the input image or the vertical direction of the input image.
  • the specific value of the first length can be determined according to the resolution in the first direction required. The lower the resolution, the larger the first length value, and the higher the resolution, the smaller the first length value. However, the first length is greater than or equal to the distance between two adjacent phase pixels along the first direction. This ensures that there are different phase pixel pairs in any two adjacent area windows in the first direction.
  • the first length may also be smaller than the length of the first area window in the first direction.
  • the first length may be equal to the length of the first direction of each of the area windows. If the first length is greater than or equal to the distance between two adjacent phase pixels along the first direction, and the first length is smaller than the distance of the each region window in the first direction, then in the first direction There are common phase pixel pairs in any two adjacent area windows. In other words, the resulting region windows in the first direction are overlapping. If the first length is equal to the distance of the first direction of each of the area windows, at least a portion of the input image extending in the first direction is equally divided into at least two area windows in which no common phase pixel pairs exist. In other words, the resulting area windows are not overlapping. It is easy to understand that more area windows can be divided when the area windows overlap than when the area windows are not overlapped.
  • the first length can be determined using the following formula:
  • W represents the length of the first direction of the input image
  • ROI w represents the length of the area window in the first direction
  • r h represents the desired resolution in the first direction
  • FIG. 3 is a schematic diagram of dividing the input image into four area windows in a first direction. As shown in FIG. 3, the upper half of the input image is first divided into the first area window and the second area window in the horizontal direction by the first length step. The lower half of the input image is then divided into a third area window and a fourth area window in a horizontal direction in a first length step. It can be seen that the first length is greater than the distance between two adjacent pairs of phase pixels in the horizontal direction, and the first length is less than the horizontal length of the window of the region.
  • phase pixel pairs in the first region window and the second region window ie, the third phase pixel pair and the fourth phase pixel pair in the first row and the second row phase pixel pair
  • the third region window and the fourth region window also have a common phase pixel pair (ie, a third phase pixel pair and a fourth phase pixel pair in the third and fourth row phase pixel pairs).
  • the input image can also be divided into at least two area windows in the vertical direction.
  • the specific process is similar to the process of dividing the input image in the horizontal direction shown in FIG. 3, and need not be described here.
  • dividing the input image into at least two area windows further includes: dividing the input image in at least a portion of the second direction into at least two area windows having the same size by using the second length as a step size, the first The two directions are perpendicular to the first direction.
  • the second length is greater than or equal to the distance between two adjacent phase pixels in the second direction.
  • the second length is less than the distance of the window of each of the regions in the second direction. This ensures that there are different pairs of phase pixels in the two adjacent window windows in the second direction and a common phase pixel pair in the two adjacent window windows in the second direction. That is, any two adjacent area windows in the second direction are overlapped.
  • the specific determination manner of the second length is the same as the first length, and need not be described here.
  • FIG. 4 is a schematic diagram of dividing the input image into six area windows by extending the first direction and the second direction at the same time.
  • the input image is divided into six area windows in steps of a first length and a second length, respectively. It can be seen that the first length is greater than the distance between two adjacent pairs of phase pixels in the horizontal direction, and the first length is less than the horizontal length of the window of the region. The first length is greater than a distance between two adjacent pairs of phase pixels in the vertical direction, and the first length is less than a vertical length of the window of the region.
  • FIG. 4 Similar to Fig. 3, six images are used in the schematic shown in Fig. 4 to show six different area windows, respectively. However, the six images shown in Figure 4 are the same input image. The use of six images to show six different area windows, respectively, is only to more clearly show the location of the different area windows.
  • the number of area windows obtained by using one direction dividing area window is smaller than the number of area windows obtained by dividing the area window by two directions.
  • dividing the input image into at least two area windows comprises: equally dividing the input image into at least two area windows having the same size, wherein adjacent two area windows are There are no identical phase pixel pairs.
  • the phase difference corresponding to each area window can be determined. Obviously, the more the number of area windows, the more the phase difference of the input image. The resolution of the depth image of the input image is better.
  • the resolution of the obtained depth image is:
  • W represents the length of the first direction of the input image
  • ROI w represents the length of the area window in the first direction
  • r h represents the resolution of the first direction
  • H represents the length of the second direction of the input image
  • ROI h represents the length of the area window in the second direction
  • r v represents the resolution of the second direction.
  • the resolution of the depth image of the first image in the first direction is higher than that of the window without the overlapping area. degree.
  • the area window overlaps only in the second direction since the second length is smaller than the length of the area window in the second direction, the depth image of the first image has a higher resolution in the second direction than the non-overlapping area.
  • the resolution of the window It can be understood that if the region window overlaps in the first direction and the second direction at the same time, the resolution of the depth image of the first image in the first direction and the second direction is higher than the resolution of the window without the overlap region.
  • the resolution of the obtained depth image is:
  • W represents the length of the first direction of the input image
  • s w represents the area window at the first length
  • r h represents the resolution of the first direction
  • H represents the length of the second direction of the input image
  • s h represents the second length
  • r v represents the resolution of the second direction.
  • the phase difference of each zone window can be obtained by cross-correlation of each zone window.
  • the cross-correlation of the phase pixel pairs in each of the first directions in each region window can be determined by the following formula:
  • f(x)*k(x) is a cross-correlation of phase pixel pairs in a first direction
  • f(x) represents a second phase pixel signal in a phase pixel pair in a first direction
  • k(x) represents a first party
  • T represents the signal width
  • phase difference in each of the first directions can be determined. Those skilled in the art are aware of the specific process of determining the phase difference based on cross-correlation, and need not be described here. Similarly, the phase difference in each of the first directions in each of the area windows can be determined. After determining the phase difference in each of the first directions, the phase difference of each of the region windows can be determined by the following formula:
  • PD (ROI) represents the phase difference of the area window
  • n represents the phase pixel pair of the first direction in the area window
  • PD(n) represents the phase difference of the phase pixel pair of the nth first direction.
  • FIG. 5 is a schematic diagram of determining a depth map using a phase difference.
  • the lens shown in FIG. 5 includes a lens 501 of the camera and an image sensor 502. The distance between the lens 501 and the image sensor 502 is D1.
  • the schematic shown in FIG. 5 also includes objects to be photographed, including: an object 503, an object 504, and an object 505.
  • the phase difference of the object 503 located at the focus is zero.
  • the distance between the object 504 and the lens 501 is less than D2. Therefore, the phase difference of the object 504 is negative.
  • the distance between the object 505 and the lens 501 is greater than D2. Therefore, the phase difference of the object 505 is positive. Since the phase difference of the object different from the lens distance is different, the phase difference can be used to reflect the distance information between the object and the lens. In other words, the phase difference can reflect the depth information of different objects.
  • the phase difference of the object located at the focus is 0, and the phase difference of the object closer to the lens 501 is smaller, and the phase difference of the object farther from the lens 501 is larger.
  • the phase difference determines the depth image corresponding to the input image.
  • the depth image can be a grayscale image.
  • different phase differences may correspond to different gray values. Since the phase difference obtained according to the method shown in Fig. 2 is the phase difference of a region window, one region window corresponds to one gray value. If the phase differences of the two area windows are different, the gray values of the two area windows are also different. For example, the larger the phase difference is, the larger the gradation value is, and the smaller the phase difference is, the smaller the gradation value is.
  • the depth image can also be a color map. In this case, different phase differences may correspond to different colors.
  • phase difference obtained according to the method shown in FIG. 2 is the phase difference of one area window
  • one area window corresponds to one color. If the phase differences of the two area windows are different, the colors of the two area windows are also different. Therefore, the greater the number of area windows, the more the phase difference of the input image. The resolution of the depth image of the input image is better.
  • the signal of the second phase pixel in the first row phase pixel pair in the first area window and the first line phase pixel pair in the first area window may be used.
  • the signal of the first phase pixel in the first cross correlation is determined. Determining the second mutual based on the signal of the second phase pixel in the second row of phase pixel pairs in the first region window and the signal of the first phase pixel in the second row of phase pixel pairs in the first region window Related. Determining, according to the first cross-correlation, a first phase difference PD 1 ; and determining a second phase difference PD 2 according to the second cross-correlation.
  • phase difference of the first region window is (PD 1 + PD 2 )/2.
  • phase difference of all of the area windows in the input image can be determined.
  • a depth image to the input image can be determined based on the phase difference of the respective area windows.
  • the input images shown in Figures 1, 3 and 4 are only one schematic.
  • the size, first length, and second length of the area window shown in the figure are also only an example.
  • the minimum length of the region window may be 20 ⁇ p s , where p s is the distance between two adjacent phase pixels.
  • the width of the area window may be the length including at least two columns of pixel pairs. If the length of the area window is too small, the accuracy of the calculated phase difference will decrease.
  • FIG. 6 is a structural block diagram of an apparatus for processing an image according to an embodiment of the present invention.
  • the apparatus 600 shown in Figure 6 is capable of performing the various steps of the method as shown in Figure 2.
  • the device 600 shown in FIG. 6 includes an obtaining unit 601 and a determining unit 602.
  • An obtaining unit 601 configured to acquire an input image, where the input image includes multiple common pixels and multiple a phase pixel pair, each phase pixel pair of the plurality of phase pixel pairs comprising a first phase pixel and a second phase pixel, wherein the first phase pixel is a phase pixel blocked by the left side, and the second phase pixel is a right side block Phase pixel.
  • the determining unit 602 is configured to divide the input image into at least two area windows, each of the at least two area windows including at least two adjacent phase pixel pairs of the plurality of phase pixel pairs.
  • the determining unit 602 is further configured to determine a phase difference corresponding to each of the area windows according to at least two phase pixel pairs of each of the at least two area windows.
  • the determining unit 602 is further configured to determine a depth image corresponding to the input image according to the phase difference corresponding to each of the area windows.
  • the apparatus according to Fig. 6 can determine the depth image based on the phase pixels.
  • the device does not require multiple input images and does not require additional aids to capture depth images.
  • the determining unit 602 is specifically configured to divide the input image into at least two area windows having the same size by using at least a part of the first direction in a step of the first length, where the One direction is the horizontal direction of the input image or the vertical direction of the input image.
  • the first length is greater than or equal to the distance between two adjacent phase pixels extending in the first direction. In this way, it can be ensured that different phase pixel pairs exist in any two adjacent area windows in the first direction. Further, the first length may also be smaller than the length of the first region window in the first direction. In this way, it can be ensured that the same phase pixel pair exists in any two adjacent area windows in the first direction.
  • the determining unit 602 is further configured to divide the input image into at least two regions having the same size in at least a portion of the second direction by using the second length as a step, the second direction being perpendicular to the first direction .
  • the second length is greater than or equal to the distance between two adjacent phase pixels extending in the second direction. In this way, it is guaranteed that different phase pixel pairs exist in any two adjacent area windows in the second direction. Further, the second length may also be smaller than the length of the window in each of the regions extending in the second direction. In this way, it can be ensured that the same phase pixel pair exists in any two adjacent area windows in the second direction.
  • a determining unit 602 specifically configured to determine, according to at least two phase pixel pairs of each of the at least two area windows, a cross-correlation between the first phase pixel and the second phase pixel in each of the area windows; The first phase pixel and the second phase pixel in each area window are mutually Off, determine the phase difference corresponding to each of the area windows.
  • FIG. 7 is a structural block diagram of an apparatus for processing an image according to an embodiment of the present invention.
  • the device 700 shown in FIG. 7 includes a processor 701 and a memory 702.
  • bus system 703 which in addition to the data bus includes a power bus, a control bus, and a status signal bus.
  • bus system 703 various buses are labeled as bus system 703 in FIG.
  • Processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the foregoing method may be completed by an integrated logic circuit of hardware in the processor 701 or an instruction in a form of software.
  • the processor 701 may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like. Programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • the general purpose processor may be a microprocessor or the processor or any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present invention may be directly implemented by the hardware decoding processor, or may be performed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read only memory or an electrically erasable programmable memory, a register, etc.
  • RAM random access memory
  • ROM read-only memory
  • programmable read only memory or an electrically erasable programmable memory
  • register etc.
  • processor 701 reads the instructions in memory 702 and, in conjunction with its hardware, performs the steps of the above method.
  • the processor 701 is configured to acquire an input image, where the input image includes a plurality of normal pixels and a plurality of phase pixel pairs, each of the plurality of phase pixel pairs includes a first phase pixel and a second phase pixel, the first One phase pixel is a phase pixel occluded by the left side, and the second phase pixel is a phase pixel occluded by the right side.
  • the device 700 may further include an image sensor 704 for capturing the input image, wherein the image sensor 704 includes a transmission for acquiring a normal pixel, a first phase pixel, and a second phase pixel. Sensing component.
  • the processor 701 is specifically configured to acquire an input image from the image sensor 704.
  • the processor 701 is further configured to divide the input image into at least two area windows, each of the at least two area windows including at least two adjacent ones of the plurality of phase pixel pairs Phase pixel pair.
  • the processor 701 is further configured to determine, according to at least two phase pixel pairs of each of the at least two area windows, a phase difference corresponding to each of the area windows.
  • the processor 701 is further configured to determine a depth image corresponding to the input image according to a phase difference corresponding to each of the area windows.
  • the device according to Fig. 7 can determine the depth image based on the phase pixels.
  • the device does not require multiple input images and does not require additional aids to capture depth images.
  • the processor 701 is specifically configured to divide the input image into at least two area windows having the same size in a first length step, the first One direction is the horizontal direction of the input image or the vertical direction of the input image.
  • the first length is greater than or equal to the distance between two adjacent phase pixels extending in the first direction. In this way, it can be ensured that different phase pixel pairs exist in any two adjacent area windows in the first direction. Further, the first length may also be smaller than the length of the first region window in the first direction. In this way, it can be ensured that the same phase pixel pair exists in any two adjacent area windows in the first direction.
  • the processor 701 is further configured to divide the input image into at least two regions having the same size in at least a portion of the second direction by using the second length as a step, the second direction being perpendicular to the first direction .
  • the second length is greater than or equal to the distance between two adjacent phase pixels extending in the second direction. In this way, it is guaranteed that different phase pixel pairs exist in any two adjacent area windows in the second direction. Further, the second length may also be smaller than the length of the window in each of the regions extending in the second direction. In this way, it can be ensured that the same phase pixel pair exists in any two adjacent area windows in the second direction.
  • the processor 701 is configured to determine, according to at least two phase pixel pairs of each of the at least two area windows, a cross correlation between the first phase pixel and the second phase pixel in each of the area windows; A cross-correlation of the first phase pixel and the second phase pixel in each of the area windows determines a phase difference corresponding to each of the area windows.
  • the disclosed systems, devices, and methods may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the functions may be stored in a computer readable storage medium if implemented in the form of a software functional unit and sold or used as a standalone product.
  • the technical solution of the present invention which is essential or contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product, which is stored in a storage medium, including
  • the instructions are used to cause a computer device (which may be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the methods described in various embodiments of the present invention.
  • the foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Measurement Of Optical Distance (AREA)

Abstract

本发明实施例提供一种处理图像的方法和设备，该方法包括：获取输入图像，该输入图像包括多个普通像素和多个相位像素对，该多个相位像素对中的每个相位像素对包括第一相位像素和第二相位像素；将该输入图像划分为至少两个区域窗口，该至少两个区域窗口中的每个区域窗口包括该多个相位像素对中的至少两个相邻的相位像素对；根据该至少两个区域窗口中的每个区域窗口的至少两个相位像素对，确定该每个区域窗口对应的相位差；根据该每个区域窗口对应的相位差，确定与该输入图像对应的深度图像。上述技术方案无需拍摄多张输入图像，也无需借助其他辅助设备来获取深度图像。

Description

处理图像的方法和设备 技术领域
本发明实施例涉及图像处理技术领域，并且更具体地，涉及处理图像的方法和设备。
背景技术
深度图像反映了图像的深度信息。深度信息表示图像中的物体与相机之间的距离。深度图像的像素可以用于反映对应的区域与相机之间的距离信息。
现有技术中，深度图像的获取方式非常复杂。常见的一种深度图像的获取方式是通过以不同位置为焦点拍摄多张照片来实现。例如，可以通过双照相机实现。具体地，双照相机是指具有两个独立的图像传感器的照相机。使用每个图像传感器各拍摄一张照片，其中一张照片的焦点在远景，另一张照片的焦点在近景。深度图像可以根据这两张照片生成。但是双照相机的成本非常高。再如，还可以通过普通相机以不同焦点拍摄多张照片实现。但是，这种方式在拍摄不同焦点的照片的时间是不同的。因此，不能很好地用于拍摄移动的物体。另一种获取深度图像的方式是基于飞行时间的***方案。该方案需要一个独立的发光单元。该发光单元用于照射需要拍摄的物体。另一个独立的传感器会拍摄下光并计算光到达目标物体需要的时间。根据光的传输时间，可以计算目标物体的距离并生成深度图像。
上述方案中，深度图像的获取方式复杂或者用于获取深度图像的设备成本高昂。
发明内容
本发明实施例提供一种处理图像的方法和设备，能够提供一种简单的获取深度图像的方式。
第一方面，本发明实施例提供一种处理图像的方法，该方法包括：获取输入图像，该输入图像包括多个普通像素和多个相位像素对，该多个相位像素对中的每个相位像素对包括第一相位像素和第二相位像素，该第一相位像素为左侧遮挡的相位像素，该第二相位像素为右侧遮挡的相位像素；将该输 入图像划分为至少两个区域窗口，该至少两个区域窗口中的每个区域窗口包括该多个相位像素对中的至少两个相邻的相位像素对；根据该至少两个区域窗口中的每个区域窗口的至少两个相位像素对，确定该每个区域窗口对应的相位差；根据该每个区域窗口对应的相位差，确定与该输入图像对应的深度图像。上述技术方案中，输入图像是利用能够获取相位像素的图像传感器获取的。根据相位像素，可以确定出深度图像。上述技术方案无需拍摄多张输入图像，也无需借助其他辅助设备来获取深度图像。具体地，该第一相位像素和该第二相位像素分别位于相邻的像素行，并且该第二相位像素位于该第一相位像素的右侧的相邻列。
结合第一方面，在第一方面的第一种可能的实现方式中，该将该输入图像划分为至少两个区域窗口，包括：以第一长度为步长，将该输入图像延第一方向的至少一部分划分为具有相同大小的至少两个区域窗口，该第一方向为该输入图像的水平方向或该输入图像的竖直方向。
结合第一方面的第一种可能的实现方式，在第一方面的第二种可能的实现方式中，该将该输入图像划分为至少连个区域窗口，还包括：以第二长度为步长，将该输入图像沿第二方向的至少一部分划分为具有相同大小的至少两个区域窗口，该第二方向与该第一方向垂直。这样，与仅沿第一方向划分该输入图像的情况相比，沿两个方向划分输入图像可以得到更多的区域窗口。从而可以提高输入图像的深度图的解析度。
结合第一方面或第一方面的上述任一种可能的实现方式，在第一方面的第三种可能的实现方式中，该根据该至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定该每个区域窗口对应的相位差，包括：根据该至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定该每个区域窗口中的第一相位像素和第二相位像素的互相关；根据该每个区域窗口中的第一相位像素和第二相位像素的互相关，确定该每个区域窗口对应的相位差。
第二方面，本发明实施例提供一种处理图像的设备，该设备包括：用于执行该第一方面提供的方法的单元。
第三方面，本发明实施例提供一种图像处理设备，该设备包括图像传感器和处理器。该图像传感器和该处理器用于执行该第一方面提供的方法。
第四方面，本发明实施例提供一种计算机可读存储介质，该计算机可读存储介质存储的程序包括用于执行该第一方面提供的方法的指令。
第五方面，本发明实施例提供一种处理图像的设备，该设备包括第三方面的计算机可读存储介质、处理器。处理器用于执行该计算机可读存储介质中存储的程序的指令，以完成对输入图像的处理。
进一步，该第一长度大于或等于延第一方向的两个相邻的相位像素之间的距离。这样，才能保证在该第一方向上的任意两个相邻区域窗口中都存在不同的相位像素对。进一步，该第一长度还可以小于该每个区域窗口延第一方向的长度。这样，可以保证在第一方向上的任意两个相邻的区域窗口中都存在相同的相位像素对。
进一步，该第二长度大于或等于延第二方向的两个相邻的相位像素之间的距离。这样，才能保证在该第二方向上的任意两个相邻区域窗口中都存在不同的相位像素对。进一步，该第二长度还可以小于该每个区域窗口延第二方向的长度。这样，可以保证在第二方向上的任意两个相邻的区域窗口中都存在相同的相位像素对。
附图说明
为了更清楚地说明本发明实施例的技术方案，下面将对本发明实施例中所需要使用的附图作简单地介绍，显而易见地，下面所描述的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是一个输入图像的示意图。
图2是根据本发明实施例提供的处理图像的方法的示意性流程图。
图3是一种沿第一方向将该输入图像划分为四个区域窗口的示意图。
图4是同时延第一方向和第二方向将该输入图像划分为六个区域窗口的示意图。
图5是利用相位差确定深度图的示意图。
图6是根据本发明实施例提供的一种处理图像的设备的结构框图。
图7是根据本发明实施例提供的设备的结构框图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所述的实施例是本发明的一部分实施例，而不是 全部实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动的前提下所获得的所有其他实施例，都应属于本发明保护的范围。
数字照相机是通过使用图像传感器替代传统胶片获取图像的。图像传感器中均匀分布着感光元件，用于将光学图像转换为电信号最终生成图像。相位对焦是一种利用图像传感器中的特殊感光元件进行对焦的方法。
图1是一个能够实现相位对焦的图像传感器的示意图。如图1所示的图像传感器中均匀分布着多个感光元件。感光元件中存在一种特殊的感光元件，这种特殊的感光元件是一半被遮挡的普通感光元件。相位对焦正式利用这种特殊的感光元件获取到的信号计算相位差实现的。
以上所称的信号是由多个感光元件获得的。每个感光元件获得的仅是信号中的采样(sample)。为了方便描述，以下将用于相位对焦的特殊的感光元件所获得的采样称为相位像素(phase pixel)，将普通的感光元件获取到的采样称为普通像素，将能够获取相位像素实现相位对焦的图像传感器称为具有相位像素的图像传感器。
更具体地，相位像素还可以分为第一相位像素和第二相位像素，其中第一相位像素是由左侧被遮挡的感光元件获得的，第二相位像素是由右侧被遮挡的感光元件获得的。本发明中所称的输入图像由具有相位像素的图像传感器获得的原始信号，而并非是经过一系列后续处理得到的照片。该具有相位像素的图像传感器中的一个感光元件对应于该输入图像中的一个像素。该输入图像中的像素是相位像素和普通像素。因此，图1也可以视为是输入图像的示意图。
从图1中可以看出，该输入图像中包括多个普通和多个相位像素对，该多个相位像素对中的每个相位像素包括第一相位像素和第二相位像素。图1所示的输入图像中每个相位像素对中的第一相位像素和第二相位像素分别位于相邻的像素行，并且第二相位像素位于第一相位像素右侧的相邻列。当然，图1所示的相位像素对的排列仅是一个实施例，相位像素对还可以有其他的排列方式，本发明并不限定。但是，组成一个相位像素对的两个相位像素的距离越近，计算相位差的精度越高。
图2是根据本发明实施例提供的处理图像的方法的示意性流程图。
201，获取输入图像。
202，将该输入图像划分为至少两个区域窗口，该至少两个区域窗口中 的每个区域窗口包括该多个相位像素对中的至少两个相邻的相位像素对。
203，根据该至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定该每个区域窗口对应的相位差(英文：Phase Difference，简称：PD)。
204，根据该每个区域窗口对应的相位差，确定与该输入图像对应的深度图像。
根据图2所示的方法，输入图像是利用能够获取相位像素的图像传感器获取的。根据每个区域窗口中的相位像素，可以确定出该区域窗口的相位差。这样就可以确定出深度图像。上述技术方案无需拍摄多张输入图像，也无需借助其他辅助设备来获取深度图像。
可选的，作为一个实施例，将该输入图像划分为至少两个区域窗口，包括：以第一长度为步长，将该输入图像延第一方向的至少一部分划分为具有相同大小的至少两个区域窗口，该第一方向为该输入图像的水平方向或该输入图像的竖直方向。
该第一长度的具体值可以根据需要的第一方向上的解析度来确定。解析度越低，该第一长度值越大，解析度越高，该第一长度值越小。但是，该第一长度大于或等于沿第一方向的两个相邻的相位像素之间的距离。这样才能保证在第一方向上任意两个相邻的区域窗口中都存在不同的相位像素对。
可选的，作为一个实施例，该第一长度还可以小于该每个区域窗口延第一方向的长度。作为另一个实施例，该第一长度可以等于该每个区域窗口的第一方向的长度。若该第一长度大于或等于沿该第一方向的两个相邻的相位像素之间的距离，且该第一长度小于该每个区域窗口沿第一方向的距离，则在第一方向上的任意两个相邻的区域窗口中存在公共的相位像素对。换句话说，在第一方向上的得到的区域窗口是重叠的。若该第一长度等于该每个区域窗口的第一方向的距离，则是将该输入图像延该第一方向的至少一部分等分为不存在公共的相位像素对的至少两个区域窗口。换句话说，得到的区域窗口是不重叠的。容易理解的是，与区域窗口不重叠相比的情况相比，区域窗口存在重叠时可以划分出更多的区域窗口。
具体地，可以使用以下公式确定该第一长度：
s＝(W–ROIW)/(rh–1)，(公式1.1)
其中，s表示第一长度，W表示输入图像的第一方向的长度，ROIw表示区域窗口在第一方向的长度，rh表示第一方向上期望得到的解析度。
图3是一种沿第一方向将该输入图像划分为四个区域窗口的示意图。如图3所示，首先沿水平方向以第一长度为步长，将输入图像的上半部分划分为第一个区域窗口和第二个区域窗口。然后沿水平方向以第一长度为步长，将该输入图像的下半部分划分为第三个区域窗口和第四个区域窗口。可以看出，该第一长度大于水平方向上两个相邻的相位像素对之间的距离，且该第一长度小于该区域窗口的水平方向长度。
可以理解的是，图3所示的示意图中采用了四个图像分别示出四个不同的区域窗口。但是图3所示的四个图像是同一个输入图像。采用四个图像分别示出四个不同的区域窗口仅是为了更清楚地表示出不同的区域窗口所在的位置。
可以看出，该第一区域窗口和该第二区域窗口中存在公共的相位像素对(即第一行和第二行相位像素对中的第三个相位像素对和第四个相位像素对)；该第三区域窗口和该第四区域窗口也存在公共的相位像素对(即第三行和第四行相位像素对中的第三个相位像素对和第四个相位像素对)。
当然，还可以沿竖直方向将该输入图像划分为至少两个区域窗口。具体过程与图3所示的沿水平方向划分该输入图像的过程类似，在此就不必赘述。
进一步，将该输入图像划分为至少两个区域窗口，还包括：以第二长度为步长，将该输入图像延第二方向的至少一部分划分为具有相同大小的至少两个区域窗口，该第二方向与该第一方向垂直。该第二长度大于或等于沿第二方向的两个相邻的相位像素之间的距离。该第二长度小于该每个区域窗口延第二方向的距离。这样才能保证在第二方向上相邻的两个区域窗口中都存在不同的相位像素对以及第二方向上相邻的两个区域窗口中都存在公共的相位像素对。也就是说，在第二方向上的任意两个相邻的区域窗口是重叠的。该第二长度的具体确定方式与该第一长度相同，在此就不必赘述。
图4是同时延第一方向和第二方向将该输入图像划分为六个区域窗口的示意图。
如图4所示，分别以第一长度和第二长度为步长将该输入图像划分为六个区域窗口。可以看出，该第一长度大于水平方向上两个相邻的相位像素对之间的距离，且该第一长度小于该区域窗口的水平方向长度。该第一长度大于竖直方向上两个相邻的相位像素对之间的距离，且该第一长度小于该区域窗口的竖直方向长度。
与图3类似，图4所示的示意图中采用了六个图像分别示出六个不同的区域窗口。但是图4所示的六个图像是同一个输入图像。采用六个图像分别示出六个不同的区域窗口仅是为了更清楚地表示出不同的区域窗口所在的位置。
可以看出，对于同一个输入图像，采用一个方向划分区域窗口得到的区域窗口数量小于采用两个方向划分区域窗口得到的区域窗口数量。
可选的，作为另一个实施例，将该输入图像划分为至少两个区域窗口，包括：将该输入图像等分为具有相同大小的至少两个区域窗口，其中相邻的两个区域窗口中不存在相同的相位像素对。
在确定了区域窗口后，可以确定出每个区域窗口对应的相位差。显然，区域窗口的数量越多，该输入图像的相位差越多。该输入图像的深度图像的解析度越好。
若采用不重叠的区域窗口，则得到的深度图像的解析度为：
Figure PCTCN2016070275-appb-000001
  (公式1.2)
其中，W表示输入图像的第一方向的长度，ROIw表示区域窗口在第一方向的长度，rh表示第一方向的解析度。H表示输入图像的第二方向的长度，ROIh表示区域窗口在第二方向的长度，rv表示第二方向的解析度。
若区域窗口在仅在第一方向重叠，由于第一长度小于该区域窗口在第一方向上的长度，该第一图像的深度图像在第一方向的解析度高于不采用重叠区域窗口的解析度。类似的，若区域窗口仅在第二方向重叠，由于该第二长度小于该区域窗口在第二方向上的长度，该第一图像的深度图像在第二方向的解析度高于不采用重叠区域窗口的解析度。可以理解，若该区域窗口同时在第一方向和第二方向重叠，该第一图像的深度图像在第一方向和第二方向的解析度均高于不采用重叠区域窗口的解析度。
具体地，若该区域窗口同时在第一方向和第二方向重叠，则得到的深度图像的解析度为：
Figure PCTCN2016070275-appb-000002
  (公式1.3)
其中，W表示输入图像的第一方向的长度，sw表示区域窗口在第一长度，rh表示第一方向的解析度。H表示输入图像的第二方向的长度，sh表示第二 长度，rv表示第二方向的解析度。
每个区域窗口的相位差可以通过每个区域窗口的互相关获得。具体地，可以通过以下公式确定出每个区域窗口中的每个第一方向上相位像素对的互相关：
Figure PCTCN2016070275-appb-000003
  (公式1.4)
其中，f(x)*k(x)第一方向上相位像素对的互相关，f(x)表示第一方向上相位像素对中的第二相位像素信号，k(x)表示第一方向上相位像素对中的第一相位像素信号，T表示信号宽幅。
在确定了互相关后，可以确定每个第一方向上的相位差。本领域技术人员知晓根据互相关确定相位差的具体过程，在此就不必赘述。类似的，可以确定出每个区域窗口中的每个第一方向上的相位差。在确定了每个第一方向上的相位差后，可以采用以下公式确定该每个区域窗口的相位差：
Figure PCTCN2016070275-appb-000004
其中，PD(ROI)表示区域窗口的相位差，n表示该区域窗口中存在几个第一方向的相位像素对，PD(n)表示第n个第一方向的相位像素对的相位差。在确定了每个区域窗口的相位差后，就可以根据该每个区域窗口的相位差确定与该输入图像对应的深度图像。
图5是利用相位差确定深度图的示意图。图5所示的示意图中包括相机的镜头501和图像传感器502。镜头501与图像传感器502之间的距离为D1。图5所示的示意图中还包括被拍摄的物体，包括：物体503、物体504和物体505。
如图5所示，假设焦点与镜头501的距离为D2，那么位于焦点上的物体503的相位差为0。物体504与镜头501的距离小于D2。因此，物体504的相位差为负。物体505与镜头501的距离大于D2。因此，物体505的相位差为正。由于与镜头距离不同的物体的相位差不同，因此可以利用相位差来反映物体到镜头之间的距离信息。换句话说，相位差可以反映不同物体的深度信息。位于焦点的物体的相位差的为0，越接近镜头501的物体的相位差越小，越远离镜头501的物体的相位差越大。
这样，在确定出每个区域窗口的相位差后，就可以利用该每个区域窗口 的相位差，确定出与输入图像对应的深度图像。该深度图像可以是灰度图。在此情况下，不同的相位差可以对应于不同的灰度值。由于根据图2所示的方法得到的相位差是一个区域窗口的相位差，那么一个区域窗口对应一个灰度值。如果两个区域窗口的相位差不同，那么这两个区域窗口的灰度值也不同。例如，相位差越大，灰度值越大，相位差越小，灰度值越小。该深度图像也可以是彩色图。在此情况下，不同的相位差可以对应于不同的颜色。由于根据图2所示的方法得到的相位差是一个区域窗口的相位差，那么一个区域窗口对应的一个颜色。如果两个区域窗口的相位差不同，那么这两个区域窗口的颜色也不同。因此，区域窗口的数量越多，该输入图像的相位差越多。该输入图像的深度图像的解析度越好。
以图3中的第一个区域窗口为例，可以根据第一个区域窗口中的第一行相位像素对中的第二相位像素的信号和第一个区域窗口中的第一行相位像素对中的第一相位像素的信号，确定第一互相关。可以根据第一个区域窗口中的第二行相位像素对中的第二相位像素的信号和第一个区域窗口中的第二行相位像素对中的第一相位像素的信号，确定第二互相关。根据该第一互相关，确定第一相位差PD1；根据该第二互相关，确定第二相位差PD2。然后，可以确定该第一区域窗口相位差为(PD1+PD2)/2。采用类似的方式，可以确定出该输入图像中所有区域窗口的相位差。然后，可以根据各个区域窗口的相位差，确定到该输入图像的深度图像。
本领域技术人员可以理解的是，图1、图3和图4中所示的输入图像仅是一个示意图。图中所示的区域窗口的大小、第一长度和第二长度也仅是一个示例。例如，实际应用中，区域窗口的最小长度可以是20×ps，其中ps为两个相邻的相位像素之间的距离。区域窗口的宽度可以是包括至少两列位像素对的长度。如果区域窗口的长度过小，计算出来的相位差的精确度会降低。
此外，本领域技术人员可以理解，本发明实施例中所称的长度、距离以及解析度的单位均为像素。
图6是根据本发明实施例提供的一种处理图像的设备的结构框图。图6所示的设备600能够执行如图2所示方法的各个步骤。如图6所示的设备600包括获取单元601和确定单元602。
获取单元601，用于获取输入图像，该输入图像包括多个普通像素和多 个相位像素对，该多个相位像素对中每个相位像素对包括第一相位像素和第二相位像素，该第一相位像素为左侧遮挡的相位像素，该第二相位像素为右侧遮挡的相位像素。
确定单元602，用于将该输入图像划分为至少两个区域窗口，该至少两个区域窗口中的每个区域窗口包括该多个相位像素对中的至少两个相邻的相位像素对。
确定单元602，还用于根据该至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定该每个区域窗口对应的相位差。
确定单元602，还用于根据该每个区域窗口对应的相位差，确定与该输入图像对应的深度图像。
根据图6所示的设备可以根据相位像素，可以确定出深度图像。该设备无需多张输入图像，也无需借助其他辅助设备来获取深度图像。
可选的，作为一个实施例，确定单元602，具体用于以第一长度为步长，将该输入图像沿着第一方向的至少一部分划分为具有相同大小的至少两个区域窗口，该第一方向为该输入图像的水平方向或该输入图像的竖直方向。
进一步，该第一长度大于或等于延第一方向的两个相邻的相位像素之间的距离。这样，才能保证在该第一方向上的任意两个相邻区域窗口中都存在不同的相位像素对。进一步，该第一长度还可以小于该每个区域窗口延第一方向的长度。这样，可以保证在第一方向上的任意两个相邻的区域窗口中都存在相同的相位像素对。
进一步，确定单元602，还用于以第二长度为步长，将该输入图像沿第二方向的至少一部分划分为具有相同大小的至少两个区域窗口，该第二方向与该第一方向垂直。
进一步，该第二长度大于或等于延第二方向的两个相邻的相位像素之间的距离。这样，才能保证在该第二方向上的任意两个相邻区域窗口中都存在不同的相位像素对。进一步，该第二长度还可以小于该每个区域窗口延第二方向的长度。这样，可以保证在第二方向上的任意两个相邻的区域窗口中都存在相同的相位像素对。
确定单元602，具体用于根据该至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定该每个区域窗口中的第一相位像素和第二相位像素的互相关；根据该每个区域窗口中的第一相位像素和第二相位像素的互相 关，确定该每个区域窗口对应的相位差。
图7是根据本发明实施例提供的处理图像的设备的结构框图。图7所示的设备700包括：处理器701和存储器702。
设备700中的各个组件通过总线***703耦合在一起，其中总线***703除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图7中将各种总线都标为总线***703。
上述本发明实施例揭示的方法可以应用于处理器701中，或者由处理器701实现。处理器701可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器701中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器701可以是通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本发明实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本发明实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存取存储器(Random Access Memory，RAM)、闪存、只读存储器(Read-Only Memory，ROM)、可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器702，处理器701读取存储器702中的指令，结合其硬件完成上述方法的步骤。
处理器701，用于获取输入图像，该输入图像包括多个普通像素和多个相位像素对，该多个相位像素对中每个相位像素对包括第一相位像素和第二相位像素，该第一相位像素为左侧遮挡的相位像素，该第二相位像素为右侧遮挡的相位像素。
可选的，作为一个实施例，设备700还可以包括图像传感器704，用于拍摄所述输入图像，其中所述图像传感器704包括用于获取普通像素、第一相位像素和第二相位像素的传感元件。在此情况下，处理器701具体用于从图像传感器704获取输入图像。
处理器701，还用于将该输入图像划分为至少两个区域窗口，该至少两个区域窗口中的每个区域窗口包括该多个相位像素对中的至少两个相邻的 相位像素对。
处理器701，还用于根据该至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定该每个区域窗口对应的相位差。
处理器701，还用于根据该每个区域窗口对应的相位差，确定与该输入图像对应的深度图像。
根据图7所示的设备可以根据相位像素，可以确定出深度图像。该设备无需多张输入图像，也无需借助其他辅助设备来获取深度图像。
可选的，作为一个实施例，处理器701，具体用于以第一长度为步长，将该输入图像沿着第一方向的至少一部分划分为具有相同大小的至少两个区域窗口，该第一方向为该输入图像的水平方向或该输入图像的竖直方向。
进一步，该第一长度大于或等于延第一方向的两个相邻的相位像素之间的距离。这样，才能保证在该第一方向上的任意两个相邻区域窗口中都存在不同的相位像素对。进一步，该第一长度还可以小于该每个区域窗口延第一方向的长度。这样，可以保证在第一方向上的任意两个相邻的区域窗口中都存在相同的相位像素对。
进一步，处理器701，还用于以第二长度为步长，将该输入图像沿第二方向的至少一部分划分为具有相同大小的至少两个区域窗口，该第二方向与该第一方向垂直。
进一步，该第二长度大于或等于延第二方向的两个相邻的相位像素之间的距离。这样，才能保证在该第二方向上的任意两个相邻区域窗口中都存在不同的相位像素对。进一步，该第二长度还可以小于该每个区域窗口延第二方向的长度。这样，可以保证在第二方向上的任意两个相邻的区域窗口中都存在相同的相位像素对。
处理器701，具体用于根据该至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定该每个区域窗口中的第一相位像素和第二相位像素的互相关；根据该每个区域窗口中的第一相位像素和第二相位像素的互相关，确定该每个区域窗口对应的相位差。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方 法来实现所描述的功能，但是这种实现不应认为超出本发明的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的***、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的***、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个***，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本发明各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本发明的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，可轻易想到的变化或替换，都应涵盖在本发明的保护范围之内，因此本发明的保护范围应以权利要求的保护范围为准。

Claims (25)

  1. 一种处理图像的方法，其特征在于，所述方法包括：
    获取输入图像，所述输入图像包括多个普通像素和多个相位像素对，所述多个相位像素对中每个相位像素对包括第一相位像素和第二相位像素，所述第一相位像素由左侧遮挡的感光元件获取，所述第二相位像素由右侧遮挡的感光元件获取；
    将所述输入图像划分为至少两个区域窗口，所述至少两个区域窗口中的每个区域窗口包括所述多个相位像素对中的至少两个相邻的相位像素对；
    根据所述至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定所述每个区域窗口对应的相位差；
    根据所述每个区域窗口对应的相位差，确定与所述输入图像对应的深度图像。
  2. 如权利要求1所述的方法，其特征在于，所述将所述输入图像划分为至少两个区域窗口，包括：
    以第一长度为步长，将所述输入图像沿着第一方向的至少一部分划分为具有相同大小的至少两个区域窗口，所述第一方向为所述输入图像的水平方向或所述输入图像的竖直方向。
  3. 如权利要求2所述的方法，其特征在于，所述第一长度大于或等于沿所述第一方向的两个相邻的相位像素对之间的距离。
  4. 如权利要求2或3所述的方法，其特征在于，所述第一长度小于所述每个区域窗口延所述第一方向的长度。
  5. 如权利要求2至4中任一项所述的方法，其特征在于，所述将所述输入图像划分为至少两个区域窗口，还包括：
    以第二长度为步长，将所述输入图像沿第二方向的至少一部分划分为具有相同大小的至少两个区域窗口，所述第二方向与所述第一方向垂直。
  6. 如权利要求5所述的方法，其特征在于，所述第二长度大于或等于沿所述第二方向的两个相邻的相位像素对之间的距离。
  7. 如权利要求5或6所述的方法，其特征在于，所述第二长度小于所述每个区域窗口延所述第二方向的长度。
  8. 如权利要求1至7中任一项所述的方法，其特征在于，所述根据所述至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定所述每个 区域窗口对应的相位差，包括：
    根据所述至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定所述每个区域窗口中的第一相位像素和第二相位像素的互相关；
    根据所述每个区域窗口中的第一相位像素和第二相位像素的互相关，确定所述每个区域窗口对应的相位差。
  9. 一种处理图像的设备，其特征在于，所述设备包括：
    获取单元，所述获取单元包括多个感光单元，所述多个感光单元包括多个普通感光单元和多个相位感光单元对，所述多个普通感光单元用于获取多个普通像素，所述多个相位感光单元对用于获取多个相位像素对，所述多个相位感光单元中的每个相位感光单元包括第一相位感光单元和第二相位感光单元，所述第一相位感光单元为左侧被遮挡的普通感光单元，所述第二相位感光单元为右侧被遮挡的普通感光单元，所述第一相位感光单元用于获取第一相位像素，所述第二相位感光单元用于获取第二相位像素，所述多个普通像素和所述多个相位像素组成输入图像；
    确定单元，用于将所述输入图像划分为至少两个区域窗口，所述至少两个区域窗口中的每个区域窗口包括所述多个相位像素对中的至少两个相邻的相位像素对；
    所述确定单元，还用于根据所述至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定所述每个区域窗口对应的相位差；
    所述确定单元，还用于根据所述每个区域窗口对应的相位差，确定与所述输入图像对应的深度图像。
  10. 如权利要求9所述的设备，其特征在于，所述确定单元，具体用于以第一长度为步长，将所述输入图像沿着第一方向的至少一部分划分为具有相同大小的至少两个区域窗口，所述第一方向为所述输入图像的水平方向或所述输入图像的竖直方向。
  11. 如权利要求10所述的设备，其特征在于，所述第一长度大于或等于沿所述第一方向的两个相邻的相位像素对之间的距离。
  12. 如权利要求10或11所述的设备，其特征在于，所述第一长度小于所述每个区域窗口延所述第一方向的长度。
  13. 如权利要求10至12中任一项所述的设备，其特征在于，所述确定单元，还用于以第二长度为步长，将所述输入图像沿第二方向的至少一部分 划分为具有相同大小的至少两个区域窗口，所述第二方向与所述第一方向垂直。
  14. 如权利要求13所述的设备，其特征在于，所述第二长度大于或等于沿所述第二方向的两个相邻的相位像素对之间的距离。
  15. 如权利要求13或14所述的设备，其特征在于，所述第二长度小于所述每个区域窗口延所述第二方向的长度。
  16. 如权利要求9至15中任一项所述的设备，其特征在于，所述确定单元，具体用于根据所述至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定所述每个区域窗口中的第一相位像素和第二相位像素的互相关；根据所述每个区域窗口中的第一相位像素和第二相位像素的互相关，确定所述每个区域窗口对应的相位差。
  17. 一种处理图像的设备，其特征在于，所述设备包括：
    图像传感器，所述图像传感器包括多个感光单元，所述多个感光单元包括多个普通感光单元和多个相位感光单元对，所述多个普通感光单元用于获取多个普通像素，所述多个相位感光单元对用于获取多个相位像素对，所述多个相位感光单元中的每个相位感光单元包括第一相位感光单元和第二相位感光单元，所述第一相位感光单元为左侧被遮挡的普通感光单元，所述第二相位感光单元为右侧被遮挡的普通感光单元，所述第一相位感光单元用于获取第一相位像素，所述第二相位感光单元用于获取第二相位像素，所述多个普通像素和所述多个相位像素组成输入图像；
    处理器，用于将所述输入图像划分为至少两个区域窗口，所述至少两个区域窗口中的每个区域窗口包括所述多个相位像素对中的至少两个相邻的相位像素对；
    所述处理器，还用于根据所述至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定所述每个区域窗口对应的相位差；
    所述处理器，还用于根据所述每个区域窗口对应的相位差，确定与所述输入图像对应的深度图像。。
  18. 如权利要求17所述的设备，其特征在于，所述处理器，具体用于以第一长度为步长，将所述输入图像沿着第一方向的至少一部分划分为具有相同大小的至少两个区域窗口，所述第一方向为所述输入图像的水平方向或所述输入图像的竖直方向。
  19. 如权利要求18所述的设备，其特征在于，所述第一长度大于或等于沿所述第一方向的两个相邻的相位像素对之间的距离。
  20. 如权利要求18或19所述的设备，其特征在于，所述第一长度小于所述每个区域窗口延所述第一方向的长度。
  21. 如权利要求18至20中任一项所述的设备，其特征在于，所述处理器，还用于以第二长度为步长，将所述输入图像沿第二方向的至少一部分划分为具有相同大小的至少两个区域窗口，所述第二方向与所述第一方向垂直。
  22. 如权利要求21所述的设备，其特征在于，所述第二长度大于或等于沿所述第二方向的两个相邻的相位像素对之间的距离。
  23. 如权利要求21或22所述的设备，其特征在于，所述第二长度小于所述每个区域窗口延所述第二方向的长度。
  24. 如权利要求17至23中任一项所述的设备，其特征在于，所述处理器，具体用于根据所述至少两个区域窗口中每个区域窗口的至少两个相位像素对，确定所述每个区域窗口中的第一相位像素和第二相位像素的互相关；根据所述每个区域窗口中的第一相位像素和第二相位像素的互相关，确定所述每个区域窗口对应的相位差。
  25. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储的程序包括用于执行如权利要求1至8中任一项方法的指令。
PCT/CN2016/070275 2016-01-06 2016-01-06 处理图像的方法和设备 WO2017117743A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US16/068,372 US20190012797A1 (en) 2016-01-06 2016-01-06 Image processing method and device
CN201680009659.0A CN107211095B (zh) 2016-01-06 2016-01-06 处理图像的方法和设备
PCT/CN2016/070275 WO2017117743A1 (zh) 2016-01-06 2016-01-06 处理图像的方法和设备
EP16882888.7A EP3389256B1 (en) 2016-01-06 2016-01-06 Method and device for processing image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/070275 WO2017117743A1 (zh) 2016-01-06 2016-01-06 处理图像的方法和设备

Publications (1)

Publication Number Publication Date
WO2017117743A1 true WO2017117743A1 (zh) 2017-07-13

Family

ID=59273136

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/070275 WO2017117743A1 (zh) 2016-01-06 2016-01-06 处理图像的方法和设备

Country Status (4)

Country Link
US (1) US20190012797A1 (zh)
EP (1) EP3389256B1 (zh)
CN (1) CN107211095B (zh)
WO (1) WO2017117743A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020062916A1 (zh) * 2018-09-30 2020-04-02 华为技术有限公司 一种获取深度图像的方法、摄像装置以及终端
CN112750136A (zh) * 2020-12-30 2021-05-04 深圳英集芯科技股份有限公司 一种图像处理方法和***

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190253607A1 (en) * 2018-02-15 2019-08-15 Qualcomm Incorporated Object tracking autofocus
CN112929563B (zh) * 2021-01-21 2022-08-26 维沃移动通信有限公司 对焦方法、装置及电子设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103108193A (zh) * 2011-11-11 2013-05-15 奥林巴斯映像株式会社 摄像装置以及摄像装置的控制方法
CN103852967A (zh) * 2012-12-03 2014-06-11 北京大学 一种模拟小景深的景深范围检测装置
CN103852954A (zh) * 2012-12-03 2014-06-11 北京大学 一种实现相位对焦的方法
CN104052919A (zh) * 2013-03-11 2014-09-17 佳能株式会社 摄像装置、摄像***及信号处理装置
US20150350529A1 (en) * 2014-06-02 2015-12-03 Canon Kabushiki Kaisha Image pickup device and image pickup system

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013183406A1 (ja) * 2012-06-07 2013-12-12 富士フイルム株式会社 撮像装置及び画像表示方法
US9600859B2 (en) * 2012-08-31 2017-03-21 Sony Corporation Image processing device, image processing method, and information processing device
JP2015033103A (ja) * 2013-08-07 2015-02-16 ソニー株式会社 画像処理装置、画像処理方法、およびプログラム
KR20150077646A (ko) * 2013-12-30 2015-07-08 삼성전자주식회사 이미지 처리 장치 및 방법
KR20150096224A (ko) * 2014-02-14 2015-08-24 삼성전자주식회사 고체 촬상 소자, 전자기기 및 오토 포커싱 방법
US9445018B2 (en) * 2014-05-01 2016-09-13 Semiconductor Components Industries, Llc Imaging systems with phase detection pixels
US10279742B2 (en) * 2014-05-29 2019-05-07 Nikon Corporation Image capture device and vehicle
US9432568B2 (en) * 2014-06-30 2016-08-30 Semiconductor Components Industries, Llc Pixel arrangements for image sensors with phase detection pixels
US9338380B2 (en) * 2014-06-30 2016-05-10 Semiconductor Components Industries, Llc Image processing methods for image sensors with phase detection pixels
US9749556B2 (en) * 2015-03-24 2017-08-29 Semiconductor Components Industries, Llc Imaging systems having image sensor pixel arrays with phase detection capabilities
US20160373664A1 (en) * 2015-10-27 2016-12-22 Mediatek Inc. Methods And Apparatus of Processing Image And Additional Information From Image Sensor

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103108193A (zh) * 2011-11-11 2013-05-15 奥林巴斯映像株式会社 摄像装置以及摄像装置的控制方法
CN103852967A (zh) * 2012-12-03 2014-06-11 北京大学 一种模拟小景深的景深范围检测装置
CN103852954A (zh) * 2012-12-03 2014-06-11 北京大学 一种实现相位对焦的方法
CN104052919A (zh) * 2013-03-11 2014-09-17 佳能株式会社 摄像装置、摄像***及信号处理装置
US20150350529A1 (en) * 2014-06-02 2015-12-03 Canon Kabushiki Kaisha Image pickup device and image pickup system
JP2015228466A (ja) * 2014-06-02 2015-12-17 キヤノン株式会社 撮像装置及び撮像システム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3389256A4 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020062916A1 (zh) * 2018-09-30 2020-04-02 华为技术有限公司 一种获取深度图像的方法、摄像装置以及终端
US11425355B2 (en) 2018-09-30 2022-08-23 Huawei Technologies Co., Ltd. Depth image obtaining method, image capture device, and terminal
CN112750136A (zh) * 2020-12-30 2021-05-04 深圳英集芯科技股份有限公司 一种图像处理方法和***
CN112750136B (zh) * 2020-12-30 2023-12-05 深圳英集芯科技股份有限公司 一种图像处理方法和***

Also Published As

Publication number Publication date
CN107211095B (zh) 2020-07-28
US20190012797A1 (en) 2019-01-10
EP3389256B1 (en) 2023-04-12
EP3389256A1 (en) 2018-10-17
EP3389256A4 (en) 2018-12-26
CN107211095A (zh) 2017-09-26

Similar Documents

Publication Publication Date Title
US11615546B2 (en) Systems and methods for depth estimation using generative models
CN110428366B (zh) 图像处理方法和装置、电子设备、计算机可读存储介质
EP3595285A1 (en) Photography processing method and device for camera module, and terminal apparatus
US7333652B2 (en) System and method for efficiently performing a depth map recovery procedure
JP5980294B2 (ja) データ処理装置、撮像装置、およびデータ処理方法
US20150104074A1 (en) Depth map generation and post-capture focusing
WO2017117743A1 (zh) 处理图像的方法和设备
US9818199B2 (en) Method and apparatus for estimating depth of focused plenoptic data
US20220182582A1 (en) Image processing method and apparatus, device and storage medium
US11024048B2 (en) Method, image processing device, and system for generating disparity map
US20220086352A9 (en) Method and electronic device for switching between first lens and second lens
JP2013061850A (ja) ノイズ低減のための画像処理装置及び画像処理方法
TW202013956A (zh) 影像裝置的校正方法及其相關影像裝置和運算裝置
WO2018063606A1 (en) Robust disparity estimation in the presence of significant intensity variations for camera arrays
JP5900017B2 (ja) 奥行き推定装置、再構成画像生成装置、奥行き推定方法、再構成画像生成方法及びプログラム
TWI635256B (zh) 基於相位對焦的定位方法及其系統
US10122939B2 (en) Image processing device for processing image data and map data with regard to depth distribution of a subject, image processing system, imaging apparatus, image processing method, and recording medium
US20230262333A1 (en) Method and electronic device for switching between first lens and second lens
US10147169B2 (en) Image processing device and program
WO2022033088A1 (zh) 图像处理方法、装置、电子设备和计算机可读介质
US9659222B2 (en) Vehicle event data recorder and operation method thereof
CN112866547B (zh) 对焦方法和装置、电子设备、计算机可读存储介质
CN112862880B (zh) 深度信息获取方法、装置、电子设备和存储介质
KR20120039855A (ko) 카메라 모듈의 이미지 처리 방법
JP5887974B2 (ja) 類似画像領域探索装置、類似画像領域探索方法、及び類似画像領域探索プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16882888

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2016882888

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2016882888

Country of ref document: EP

Effective date: 20180710