WO2016180325A1 - 图像处理方法及装置 - Google Patents

图像处理方法及装置 Download PDF

Info

Publication number
WO2016180325A1
WO2016180325A1 PCT/CN2016/081599 CN2016081599W WO2016180325A1 WO 2016180325 A1 WO2016180325 A1 WO 2016180325A1 CN 2016081599 W CN2016081599 W CN 2016081599W WO 2016180325 A1 WO2016180325 A1 WO 2016180325A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
matching
function
matching cost
parallax
Prior art date
Application number
PCT/CN2016/081599
Other languages
English (en)
French (fr)
Inventor
戴向东
Original Assignee
努比亚技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 努比亚技术有限公司 filed Critical 努比亚技术有限公司
Publication of WO2016180325A1 publication Critical patent/WO2016180325A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures

Definitions

  • This application relates to, but is not limited to, the field of image processing technology.
  • binocular vision can be used to restore the three-dimensional information of the scene, which is a hot research content in computer vision.
  • stereo matching is the most important and difficult step in binocular vision.
  • the accuracy of stereo matching affects the subsequent depth of field estimation and 3D reconstruction, which brings technical bottleneck to the application of binocular vision.
  • stereo matching algorithms have emerged, which are mainly divided into global and local stereo matching algorithms, and also global and local semi-global stereo matching algorithms. These algorithms can be obtained on two visible binocular cameras to meet practical applications.
  • the effect but the stereoscopic matching of heterogeneous sensor images, such as infrared light and visible light images, has relatively little research work, and due to the different imaging principles of image sensors, the images of infrared light and visible light appear larger in color. The difference makes the difficulty of stereo matching larger than that of the dual visible light, so that the effect that satisfies the practical application cannot be obtained.
  • This paper proposes an image processing method and device, which aims to solve the problem of difficulty in stereo matching of infrared light and visible light images.
  • the step of calculating the initial disparity image of the corresponding pixel according to the matching cost and the preset stereo matching algorithm includes:
  • the multi-directional dynamic programming algorithm is used to optimize the matching cost accumulation function to obtain an initial parallax image of the corresponding pixel.
  • the method further includes:
  • the corrected parallax image is smoothed by a filtering algorithm.
  • the step of modifying the initial parallax image comprises:
  • the step of calculating a gradient pattern of the two heterogeneous sensor images comprises:
  • soble operator also known as a step operator
  • the gradient magnitude and direction are normalized to a range of values suitable for image processing [0255] to obtain a gradient pattern of the original image.
  • the two heterogeneous sensor images are an infrared image and a visible light image, respectively.
  • the mutual information is used to measure the correlation of the grayscale distribution of the two heterogeneous sensor images.
  • the traversing the gradient pattern and calculating the mutual information of the matching window as a matching cost step for a given disparity range [0, Dmax], when the disparity is D, the right viewing angle gradient pattern
  • the pixel point p in I 1 matches the pixel point q in the left-view gradient direction pattern I 2
  • the matching cost function C(p, D) is calculated as follows:
  • P I (i) is the probability that the pixel value in image I is i.
  • the corresponding H 1 is the entropy of image I 1
  • the global energy function E(D) of the disparity map D is determined as follows:
  • C(p, D p ) is the matching cost function of the pixel point p in the parallax D
  • the T[] function is a truncation function
  • the value is 1 when the argument is 1
  • the argument is 0,
  • the value is 0,
  • P 1 and P 2 are penalty weights for the parallax change of the pixel point p and the adjacent pixel point q.
  • P 2 is determined by:
  • P' 2 is a constant
  • the penalty weight P 1 is a preset fixed value
  • P 2 > P 1 .
  • the step of acquiring the initial cost difference function of the corresponding pixel point by using the multi-directional dynamic programming algorithm to optimize the matching cost accumulation function by combining the semi-global stereo matching algorithm comprises:
  • the matching cost function L r (p,d) of the pixel p at the disparity d in the direction r is calculated by the following equation:
  • C(p,d) is the matching cost function of the pixel point p at the parallax d
  • the initial value of the matching cost function L r (p,d) is obtained according to the global energy function E(d);
  • the initial parallax image is obtained by calculating the minimized S(p, d).
  • An embodiment of the present invention further provides an image processing apparatus, including:
  • An image acquisition module is configured to: acquire two heterogeneous sensor images
  • a gradient pattern calculation module configured to: calculate a gradient pattern of the two heterogeneous sensor images
  • the matching cost calculation module is configured to: traverse the gradient direction map, and calculate mutual information of the matching window as a matching cost
  • the global energy optimization module is configured to: calculate an initial parallax image of the corresponding pixel according to the matching cost and the preset stereo matching algorithm.
  • the global energy optimization module is configured to: construct a global energy function according to the matching cost as a matching cost accumulation function; and combine a semi-global stereo matching algorithm to optimize the matching cost by using a multi-directional dynamic programming algorithm.
  • the accumulation function acquires an initial parallax image of the corresponding pixel.
  • the device further includes:
  • the parallax image optimization module is configured to: correct the initial parallax image by using a cross-coherence algorithm and an image segmentation algorithm; and smooth the corrected parallax image by using a filtering algorithm.
  • the parallax image optimization module is configured to exclude an incorrect matching point in the initial parallax image.
  • the gradient pattern calculation module is configured to: respectively calculate a gradient size and a direction of the horizontal and vertical directions of the two heterogeneous sensor images by using a soble operator;
  • the small sum direction is normalized to a range of values suitable for image processing [0 255], resulting in a gradient pattern of the original image.
  • the mutual information is used to measure the correlation of the grayscale distribution of the two heterogeneous sensor images.
  • the matching cost calculation module is configured to: for a given parallax range [0, Dmax], when the parallax is D, right perspective gradient direction I in FIG left view and pixel p in a direction of the gradient The pixel points q in Fig. 12 are matched, and the matching cost function C(p, D) is calculated as follows:
  • P I (i) is the probability that the pixel value in image I is i.
  • the corresponding H 1 is the entropy of image I 1
  • the global energy optimization module is configured to:
  • the global energy function E(D) of the disparity map D is determined as follows:
  • C(p, D p ) is the matching cost function of the pixel point p in the parallax D
  • the T[] function is a truncation function
  • the value is 1 when the argument is 1
  • the argument is 0,
  • the value is 0,
  • P 1 and P 2 are penalty weights for the parallax change of the pixel point p and the adjacent pixel point q.
  • P 2 is determined by:
  • P' 2 is a constant
  • the penalty weight P 1 is a preset fixed value
  • P 2 > P 1 .
  • the global energy optimization module is configured to:
  • the matching cost function L r (p,d) of the pixel p at the disparity d in the direction r is calculated by the following equation:
  • C(p,d) is the matching cost function of the pixel point p at the parallax d
  • the initial value of the matching cost function L r (p,d) is obtained according to the global energy function E(d);
  • the initial parallax image is obtained by calculating the minimized S(p, d).
  • the two heterogeneous sensor images are an infrared image and a visible light image, respectively.
  • the embodiment of the invention further provides a computer readable storage medium storing computer executable instructions for performing the method of any of the above.
  • An image processing method and apparatus obtains two different source sensor images; calculates a gradient direction pattern of the two heterogeneous sensor images; traverses the gradient direction map, and calculates mutual information of the matching window as a matching cost Constructing a global energy function using the matching cost, As a matching cost accumulation function; the initial parallax image of the corresponding pixel is calculated according to the matching cost accumulation function, thereby solving the problem that the stereo matching of the infrared light and the visible light image is difficult based on the infrared visible stereo matching scheme of the gradient direction combined with the mutual information. To meet the actual application needs.
  • FIG. 1 is a schematic diagram of an optional hardware structure of a mobile terminal implementing an embodiment of the present invention
  • FIG. 2 is a schematic diagram of a wireless communication system of the mobile terminal shown in FIG. 1;
  • FIG. 3 is a schematic flow chart of a first embodiment of an image processing method according to the present invention.
  • FIG. 4 is a schematic diagram showing a matching cost function of a pixel point p and a parallax d in a direction r in an embodiment of the present invention
  • FIG. 5 is a schematic diagram of acquiring an initial parallax image based on a matching cost accumulation function in an embodiment of the present invention
  • FIG. 6 is a schematic flow chart of a second embodiment of an image processing method according to the present invention.
  • Figure 7 (a) is an original visible light image in an embodiment of the present invention.
  • Figure 7 (b) is an original infrared image in an embodiment of the present invention.
  • Figure 7 (c) is a visible light gradient pattern of Figure 7 (a);
  • Figure 7 (d) is a view of the infrared light gradient in Figure 7 (b);
  • FIG. 7(e) is a schematic diagram of an initial parallax image obtained by performing image processing on the original visible light image shown in FIG. 7(a) and the original infrared image shown in FIG. 7(b);
  • Figure 7 (f) is a schematic view of the parallax image in Figure 7 (e) after correction;
  • FIG. 8 is a schematic diagram of functional modules of a first embodiment of an image processing apparatus according to the present invention.
  • Figure 9 is a block diagram showing the functional blocks of the second embodiment of the image processing apparatus of the present invention.
  • the mobile terminal can be implemented in a variety of forms.
  • the terminals described herein may include, for example, mobile phones, smart phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablets), PMPs (Portable Multimedia Players), navigation devices, and the like.
  • Mobile terminals and fixed terminals such as digital TVs, desktop computers, and the like.
  • the terminal is a mobile terminal.
  • configurations in accordance with embodiments of the present invention can be applied to fixed type terminals in addition to components that are specifically for mobile purposes.
  • FIG. 1 is a schematic diagram of an optional hardware structure of a mobile terminal implementing an embodiment of the present invention.
  • the mobile terminal 100 may include a wireless communication unit 110, an A/V (Audio/Video) input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, an interface unit 170, a controller 180, and a power supply unit 190. and many more.
  • Figure 1 illustrates a mobile terminal having various components, but it should be understood that not all illustrated components are required to be implemented. More or fewer components can be implemented instead. The elements of the mobile terminal will be described in detail below.
  • Wireless communication unit 110 typically includes one or more components that permit radio communication between mobile terminal 100 and a wireless communication system or network.
  • the wireless communication unit may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless internet module 113, a short-range communication module 114, and a location information module 115.
  • the broadcast receiving module 111 receives a broadcast signal and/or broadcast associated information from an external broadcast management server via a broadcast channel.
  • the broadcast channel can include a satellite channel and/or a terrestrial channel.
  • the broadcast management server may be a server that generates and transmits a broadcast signal and/or broadcast associated information or a server that receives a previously generated broadcast signal and/or broadcast associated information and transmits it to the terminal.
  • the broadcast signal may include a TV broadcast signal, a radio broadcast signal, a data broadcast signal, and the like.
  • the broadcast signal may further include a broadcast signal combined with a TV or radio broadcast signal.
  • the broadcast associated information may also be provided via a mobile communication network, and in this case, the broadcast associated information may be received by the mobile communication module 112.
  • the broadcast signal may exist in various forms, for example, it may exist in the form of Digital Multimedia Broadcasting (DMB) Electronic Program Guide (EPG), Digital Video Broadcasting Handheld (DVB-H) Electronic Service Guide (ESG), and the like.
  • the broadcast receiving module 111 can receive a signal broadcast by using a plurality of types of broadcast systems.
  • the broadcast receiving module 111 can use forward link media (MediaFLO) by using, for example, multimedia broadcast-terrestrial (DMB-T), digital multimedia broadcast-satellite (DMB-S), digital video broadcast-handheld (DVB-H)
  • MediaFLO forward link media
  • DMB-T multimedia broadcast-terrestrial
  • DMB-S digital multimedia broadcast-satellite
  • DVD-H digital video broadcast-handheld
  • the digital broadcasting system of the @ data broadcasting system
  • the terrestrial digital broadcasting integrated service (ISDB-T) and the like receives digital broadcasting.
  • the broadcast receiving module 111 can be constructed as a broadcast system suitable for providing a broadcast signal as well as the above-described digital broadcast system.
  • the broadcast signal and/or broadcast associated information received via the broadcast receiving module 111 may be stored in the memory 160 (or other type of storage medium).
  • the mobile communication module 112 transmits the radio signals to and/or receives radio signals from at least one of a base station (e.g., an access point, a Node B, etc.), an external terminal, and a server.
  • a base station e.g., an access point, a Node B, etc.
  • Such radio signals may include voice call signals, video call signals, or multiple types of data transmitted and/or received in accordance with text and/or multimedia messages.
  • the wireless internet module 113 supports wireless internet access of the mobile terminal.
  • the module can be internally or externally coupled to the terminal.
  • the wireless Internet access technologies involved in the module may include WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless Broadband), Wimax (Worldwide Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), etc. .
  • the short range communication module 114 is configured to support short range communication.
  • Some examples of short-range communication technology include Bluetooth TM, a radio frequency identification (RFID), infrared data association (IrDA), ultra wideband (UWB), ZigBee, etc. TM.
  • the location information module 115 is configured to check or obtain location information of the mobile terminal.
  • a typical example of a location information module is GPS (Global Positioning System).
  • GPS Global Positioning System
  • the GPS module 115 calculates distance information and accurate time information from three or more satellites and applies triangulation to the calculated information to accurately calculate three-dimensional current position information based on longitude, latitude, and altitude.
  • the method for calculating position and time information uses three satellites and corrects the calculated position and time information errors by using another satellite.
  • the GPS module 115 is capable of calculating speed information by continuously calculating current position information in real time.
  • the A/V input unit 120 is arranged to receive an audio or video signal.
  • the A/V input unit 120 may include a camera 121 and a microphone 122, and the camera 121 is illustrated in a video capture mode or an image capture mode.
  • the image data of a still picture or video obtained by the capture device is processed.
  • the processed image frame can be displayed on the display unit 151.
  • the image frames processed by the camera 121 may be stored in the memory 160 (or other storage medium) or transmitted via the wireless communication unit 110, and two or more cameras 121 may be provided according to the configuration of the mobile terminal.
  • the microphone 122 can receive sound (audio data) via a microphone in an operation mode of a telephone call mode, a recording mode, a voice recognition mode, and the like, and can process such sound as audio data.
  • the processed audio (voice) data can be converted to a format output that can be transmitted to the mobile communication base station via the mobile communication module 112 in the case of a telephone call mode.
  • the microphone 122 may pass noise cancellation (or suppression) algorithms to cancel (or suppress) noise or interference generated during the process of receiving and transmitting audio signals.
  • the user input unit 130 may generate key input data according to a command input by the user to control the operation of the mobile terminal.
  • the user input unit 130 allows the user to input various types of information, and may include a keyboard, a pot, a touch pad (eg, a touch sensitive component that detects changes in resistance, pressure, capacitance, etc. due to contact), a scroll wheel , rocker, etc.
  • a touch screen can be formed.
  • the sensing unit 140 detects the current state of the mobile terminal 100 (eg, the open or closed state of the mobile terminal 100), the location of the mobile terminal 100, the presence or absence of contact (ie, touch input) by the user with the mobile terminal 100, and the mobile terminal.
  • the sensing unit 140 can sense whether the slide type phone is turned on or off.
  • the sensing unit 140 can detect whether the power supply unit 190 provides power or whether the interface unit 170 is coupled to an external device.
  • Sensing unit 140 may include proximity sensor 141 which will be described below in connection with a touch screen.
  • the interface unit 170 serves as an interface through which at least one external device can connect with the mobile terminal 100.
  • the external device may include a wired or wireless headset port, an external power (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, and an audio input/output. (I/O) port, video I/O port, headphone port, and more.
  • the identification module may be stored to verify a variety of information used by the user using the mobile terminal 100 and may include a User Identification Module (UIM), a Customer Identification Module (SIM), a Universal Customer Identification Module (USIM), and the like.
  • UIM User Identification Module
  • SIM Customer Identification Module
  • USB Universal Customer Identification Module
  • the device having the identification module may take the form of a smart card, and thus the identification device may be connected to the mobile terminal 100 via a port or other connection device.
  • the port unit 170 can be configured to receive input from an external device (eg, data information, power, etc.) and transmit the received input to one or more components within the mobile terminal 100 or can be used in the mobile terminal and external device Transfer data between.
  • an external device eg, data information, power, etc.
  • the interface unit 170 may function as a path through which power is supplied from the base to the mobile terminal 100 or may be used as a plurality of command signals allowed to be input from the base to be transmitted to the mobile The path to the terminal.
  • a variety of command signals or power input from the base can be used as a signal for identifying whether the mobile terminal is accurately mounted on the base.
  • Output unit 150 is configured to provide an output signal (eg, an audio signal, a video signal, an alarm signal, a vibration signal, etc.) in a visual, audio, and/or tactile manner.
  • the output unit 150 may include a display unit 151, an audio output module 152, an alarm unit 153, and the like.
  • the display unit 151 can display information processed in the mobile terminal 100. For example, when the mobile terminal 100 is in a phone call mode, the display unit 151 can display a user interface (UI) or a graphical user interface (GUI) related to a call or other communication (eg, text messaging, multimedia file download, etc.). When the mobile terminal 100 is in a video call mode or an image capturing mode, the display unit 151 may display a captured image and/or a received image, a UI or GUI showing a video or image and related functions, and the like.
  • UI user interface
  • GUI graphical user interface
  • the display unit 151 can function as an input device and an output device.
  • the display unit 151 may include at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), an organic light emitting diode (OLED) display, a flexible display, a three-dimensional (3D) display, and the like.
  • LCD liquid crystal display
  • TFT-LCD thin film transistor LCD
  • OLED organic light emitting diode
  • a flexible display a three-dimensional (3D) display, and the like.
  • 3D three-dimensional
  • Some of these displays may be configured to be transparent to allow a user to view from the outside, which may be referred to as a transparent display, and a typical transparent display may be, for example, a TOLED (Transparent Organic Light Emitting Diode) display or the like.
  • TOLED Transparent Organic Light Emitting Diode
  • the mobile terminal 100 may include two or more display units (or other display devices), for example, the mobile terminal may include an external display unit (not shown) and an internal display unit (not shown) .
  • the touch screen can be set to detect touch input pressure as well as touch input position and touch input area.
  • the audio output module 152 may convert audio data received by the wireless communication unit 110 or stored in the memory 160 when the mobile terminal is in a call signal receiving mode, a call mode, a recording mode, a voice recognition mode, a broadcast receiving mode, and the like.
  • the audio signal is output as sound.
  • the audio output module 152 can provide audio output (eg, call signal reception sound, message reception sound, etc.) associated with a particular function performed by the mobile terminal 100.
  • the audio output module 152 can To include speakers, buzzers, and more.
  • the alarm unit 153 can provide an output to notify the mobile terminal 100 of the occurrence of an event. Typical events may include call reception, message reception, key signal input, touch input, and the like. In addition to audio or video output, the alert unit 153 can provide an output in a different manner to notify of the occurrence of an event. For example, the alarm unit 153 can provide an output in the form of vibrations, and when a call, message, or some other incoming communication is received, the alarm unit 153 can provide a tactile output (ie, vibration) to notify the user of it. By providing such a tactile output, the user is able to recognize the occurrence of an event even when the user's mobile phone is in the user's pocket. The alarm unit 153 can also provide an output of the notification event occurrence via the display unit 151 or the audio output module 152.
  • the memory 160 may store a software program or the like for processing and control operations performed by the controller 180, or may temporarily store data (for example, a phone book, a message, a still image, a video, etc.) that has been output or is to be output. Moreover, the memory 160 may store data regarding vibration and audio signals of various manners that are output when a touch is applied to the touch screen.
  • the memory 160 may include at least one type of storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (eg, SD or DX memory, etc.), a random access memory (RAM), a static random access memory ( SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, and the like.
  • the mobile terminal 100 can cooperate with a network storage device that performs a storage function of the memory 160 through a network connection.
  • the controller 180 typically controls the overall operation of the mobile terminal. For example, the controller 180 performs the control and processing associated with voice calls, data communications, video calls, and the like.
  • the controller 180 may include a multimedia module 181 for reproducing (or playing back) multimedia data, which may be constructed within the controller 180 or may be configured to be separate from the controller 180.
  • the controller 180 may perform a pattern recognition process to recognize a handwriting input or a picture drawing input performed on the touch screen as a character or an image.
  • the power supply unit 190 receives external power or internal power under the control of the controller 180 and provides appropriate power required to operate each component and component.
  • the embodiments described herein can be implemented in a computer readable medium using, for example, computer software, hardware, or any combination thereof.
  • the embodiments described herein can be used Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Units (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), Processors, Controllers, Microcontrollers
  • ASICs Application Specific Integrated Circuits
  • DSPs Digital Signal Processors
  • DSPDs Digital Signal Processing Units
  • PLDs Programmable Logic Devices
  • FPGAs Field Programmable Gate Arrays
  • Processors Controllers
  • Microcontrollers The microprocessor, the at least one of the electronic units designed to perform the functions described herein, is implemented, and in some cases, such an embodiment may be implemented in the controller 180.
  • implementations such as procedures or functions may be implemented with separate software modules that permit the execution of at least one function or operation.
  • the software code can be implemented by
  • the mobile terminal has been described in terms of its function.
  • a slide type mobile terminal among a plurality of types of mobile terminals such as a folding type, a bar type, a swing type, a slide type mobile terminal, and the like will be described as an example. Therefore, the embodiment of the present invention can be applied to any type of mobile terminal, and is not limited to a slide type mobile terminal.
  • the mobile terminal 100 as shown in FIG. 1 may be configured to operate using a communication system such as a wired and wireless communication system and a satellite-based communication system that transmits data via frames or packets.
  • a communication system such as a wired and wireless communication system and a satellite-based communication system that transmits data via frames or packets.
  • a communication system in which a mobile terminal is operable according to an embodiment of the present invention will now be described with reference to FIG.
  • Such communication systems may use different air interfaces and/or physical layers.
  • air interfaces used by communication systems include, for example, Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), and Universal Mobile Telecommunications System (UMTS) (in particular, Long Term Evolution (LTE)). ), Global System for Mobile Communications (GSM), etc.
  • FDMA Frequency Division Multiple Access
  • TDMA Time Division Multiple Access
  • CDMA Code Division Multiple Access
  • UMTS Universal Mobile Telecommunications System
  • LTE Long Term Evolution
  • GSM Global System for Mobile Communications
  • the following description relates to a CDMA communication system, but such teachings are equally applicable to other types of systems.
  • a CDMA wireless communication system can include a plurality of mobile terminals 100, a plurality of base stations (BS) 270, a base station controller (BSC) 275, and a mobile switching center (MSC) 280.
  • the MSC 280 is configured to interface with a public switched telephone network (PSTN) 290.
  • PSTN public switched telephone network
  • the MSC 280 is also configured to interface with a BSC 275 that can be coupled to the base station 270 via a backhaul line.
  • the backhaul line can be constructed in accordance with any of a number of known interfaces including, for example, E1/T1, ATM, IP, PPP, Frame Relay, HDSL, ADSL, or xDSL. It will be appreciated that the system as shown in FIG. 2 can include multiple BSCs 275.
  • Each BS 270 can serve one or more partitions (or regions), each of which is covered by a multi-directional antenna or an antenna directed to a particular direction radially away from the BS 270. Or, each partition can be used Covered by two or more antennas received in the diversity.
  • Each BS 270 can be configured to support multiple frequency allocations, and each frequency allocation has a particular frequency spectrum (eg, 1.25 MHz, 5 MHz, etc.).
  • BS 270 may also be referred to as a Base Transceiver Subsystem (BTS) or other equivalent terminology.
  • BTS Base Transceiver Subsystem
  • the term "base station” can be used to generally refer to a single BSC 275 and at least one BS 270.
  • a base station can also be referred to as a "cell station.”
  • multiple partitions of a particular BS 270 may be referred to as multiple cellular stations.
  • a broadcast transmitter (BT) 295 transmits a broadcast signal to the mobile terminal 100 operating within the system.
  • a broadcast receiving module 111 as shown in FIG. 1 is provided at the mobile terminal 100 to receive a broadcast signal transmitted by the BT 295.
  • GPS Global Positioning System
  • the satellite 300 helps locate at least one of the plurality of mobile terminals 100.
  • a plurality of satellites 300 are depicted, but it is understood that useful positioning information can be obtained using any number of satellites.
  • the GPS module 115 as shown in Figure 1 is typically configured to cooperate with the satellite 300 to obtain desired positioning information. Instead of GPS tracking technology or in addition to GPS tracking technology, other techniques that can track the location of the mobile terminal can be used. Additionally, at least one GPS satellite 300 can selectively or additionally process satellite DMB transmissions.
  • the BS 270 receives a reverse link signal from the mobile terminal 100.
  • Mobile terminal 100 typically participates in calls, messaging, and other types of communications.
  • Each reverse link signal received by a particular base station 270 is processed within a particular BS 270.
  • the obtained data is forwarded to the relevant BSC 275.
  • the BSC provides call resource allocation and coordinated mobility management functions including a soft handoff procedure between the BSs 270.
  • the BSC 275 also routes the received data to the MSC 280, which provides additional routing services for interfacing with the PSTN 290.
  • PSTN 290 interfaces with MSC 280, which forms an interface with BSC 275, and BSC 275 controls BS 270 accordingly to transmit forward link signals to mobile terminal 100.
  • the effect of satisfying the practical application can be obtained on two visible binocular cameras, but the stereo matching of the heterogeneous sensor images, such as infrared light and visible light images, due to the imaging principle of the image sensor Differently, the images of infrared and visible light appear to have large differences in color, so that the difficulty of stereo matching is larger than that of dual-visible stereo matching, so that the effect of satisfying the practical application cannot be obtained.
  • the stereo matching of the heterogeneous sensor images such as infrared light and visible light images
  • this paper proposes a solution that can solve the problem of difficult stereo matching of infrared light and visible light images.
  • a first embodiment of the present invention provides an image processing method, including:
  • Step S101 acquiring two heterogeneous sensor images
  • the two heterogeneous sensor images may be respectively an infrared image and a visible light image, and may of course be two other different types of images.
  • an image taken by an infrared image and a visible light image are used as image processing.
  • Step S102 calculating a gradient pattern of the two heterogeneous sensor images
  • the soble operator is used to calculate the magnitude and direction of the gradient of the two heterogeneous sensor images in the horizontal and vertical directions.
  • the operator consists of two sets of 3x3 matrices, which are horizontal and vertical, respectively, and are planarly convolved with the image to obtain lateral and longitudinal luminance difference approximations.
  • Gx and Gy represent the images detected by the longitudinal and lateral edges, respectively, G is the gradient of the pixel at the image, and ⁇ is the gradient direction.
  • the formula is as follows:
  • the range of gradient directions calculated here is (- ⁇ /2, ⁇ /2), which will contain negative numbers.
  • ⁇ can be normalized to (0, ⁇ ), and then normalized to the range of [0, 255] suitable for image processing.
  • Step S103 traversing the gradient pattern, and calculating mutual information of the matching window as a matching cost
  • the mutual information of the matching window is calculated as a matching cost.
  • the significance of the matching cost is to find similar points on the feature, the principle is to find the point with the smallest difference as the similar point.
  • P I (i) is the probability that the pixel value in image I is i.
  • the corresponding H 1 is the entropy of image I 1
  • the mutual information of image I 1 and image I 2 can measure the correlation of the gray distribution of the two images. The larger the mutual information, the closer the two images are.
  • Step S104 Calculate an initial parallax image of the corresponding pixel point according to the matching cost and the preset stereo matching algorithm.
  • C(p, D p ) is the matching cost of the pixel point p at the parallax D, which is the MI calculated in step S103.
  • the T[] function is a truncation function.
  • the value is 1, the argument is 0, and the value is 0.
  • the multi-directional dynamic programming algorithm is used to optimize the matching cost accumulation function to obtain an initial parallax image of the corresponding pixel.
  • the minimum E(D) is calculated, and the initial parallax image can be acquired.
  • FIG. 4 is a schematic diagram showing the matching cost function of the pixel point p and the parallax d in the direction r in the embodiment of the present invention
  • FIG. 5 is based on the matching cost accumulation in the embodiment of the present invention. The function obtains a schematic diagram of the initial parallax image.
  • the matching cost function L r (p,d) of the pixel p at the parallax d can be calculated by:
  • C(p,d) is the matching cost function of the pixel point p at the parallax d
  • the initial value of the matching cost function L r (p,d) is obtained according to the global energy function E(d).
  • S(p,d) is the sum of the cumulative cost functions L r (p,d) of all directions r, so that the initial disparity image can be obtained by computing the minimized S(p,d).
  • the two heterogeneous sensor images by acquiring the two heterogeneous sensor images, calculating the gradient pattern of the two heterogeneous sensor images, traversing the gradient pattern, and calculating the mutual information of the matching window as a matching cost; constructing the global energy by using the matching cost
  • the function is used as a matching cost accumulation function; the initial parallax image of the corresponding pixel is obtained according to the matching cost accumulation function, thereby solving the stereo matching of the infrared light and the visible image based on the infrared visible stereo matching scheme of the gradient direction combined with the mutual information.
  • the problem is to meet the actual application needs.
  • the second embodiment of the present invention provides an image processing method.
  • the method further includes:
  • Step S105 correcting the initial parallax image by using a cross-coherence algorithm and an image segmentation algorithm
  • step S106 the corrected parallax image is smoothed by using a filtering algorithm.
  • the embodiment considers that there is no matching point in the initial parallax image due to noise, illumination, occlusion, and weak texture region, and reasonable optimization is needed to eliminate errors in the initial parallax image. Match points.
  • the left and right parallax images are respectively acquired, and the parallax error of the corresponding pixel points of the left and right parallax images should be limited to a small range.
  • the expression is as follows:
  • e 12 (p, D 1p ) is the pixel corresponding to the corresponding matching point q in the image 2 based on the disparity value in the image 1 .
  • Fig. 7(a) is the original visible light image
  • Fig. 7(b) is the original infrared image.
  • FIG. 7(c) is a visible light gradient pattern
  • FIG. 7(d) is an infrared light gradient direction.
  • the initial parallax image of the corresponding pixel point is calculated, as shown in FIG. 7(e).
  • the initial parallax image is corrected by using a cross-coherence algorithm and an image segmentation algorithm; the corrected parallax image is smoothed by a filtering algorithm, and the corrected parallax image is as shown in FIG. 7(f), in the parallax image, the pixel point The brighter the color, the larger the parallax of the object at that location and the closer the distance from the camera.
  • the embodiment of the invention further provides a computer readable storage medium storing computer executable instructions for executing the image processing method described above.
  • the first embodiment of the present invention provides an image processing apparatus, including: an image acquisition module 201, a gradient pattern calculation module 202, a matching cost calculation module 203, and a global energy optimization module 204, wherein:
  • the image obtaining module 201 is configured to: acquire two heterogeneous sensor images
  • the gradient pattern calculation module 202 is configured to: calculate a gradient pattern of the two heterogeneous sensor images
  • the matching cost calculation module 203 is configured to: traverse the gradient pattern, and calculate mutual information of the matching window as a matching cost;
  • the global energy optimization module 204 is configured to: calculate an initial parallax image of the corresponding pixel according to the matching cost and the preset stereo matching algorithm.
  • the two heterogeneous sensor images may be respectively an infrared image and a visible light image, and may of course be two other different types of images.
  • an image taken by an infrared image and a visible light image are used as image processing.
  • the operator consists of two sets of 3x3 matrices, which are horizontal and vertical, respectively, and are planarly convolved with the image to obtain lateral and longitudinal luminance difference approximations.
  • Gx and Gy represent the images detected by the longitudinal and lateral edges, respectively, G is the gradient of the pixel at the image, and ⁇ is the gradient direction.
  • the formula is as follows:
  • the range of gradient directions calculated here is (- ⁇ /2, ⁇ /2), which will contain negative numbers.
  • ⁇ can be normalized to (0, ⁇ ), and then normalized to the range of [0, 255] suitable for image processing.
  • the mutual information of the matching window is calculated as a matching cost.
  • the significance of the matching cost is to find similar points on the feature, the principle is to find the point with the smallest difference as the similar point.
  • P I (i) is the probability that the pixel value in image I is i.
  • the corresponding H 1 is the entropy of image I 1
  • the mutual information of image I 1 and image I 2 can measure the correlation of the gray distribution of the two images. The larger the mutual information, the closer the two images are.
  • an initial parallax image for acquiring a corresponding pixel point is calculated.
  • C (p, D p) p is the pixel matching cost when parallax D is obtained by the calculation MI.
  • the T[] function is a truncation function.
  • the value is 1, the argument is 0, and the value is 0.
  • the multi-directional dynamic programming algorithm is used to optimize the matching cost accumulation function to obtain an initial parallax image of the corresponding pixel.
  • the minimum E(D) is calculated, and the initial parallax image can be acquired.
  • FIG. 4 is a schematic diagram showing the matching cost function of the pixel point p and the parallax d in the direction r in the embodiment of the present invention
  • FIG. 5 is based on the matching cost accumulation in the embodiment of the present invention. The function obtains the initial parallax image Figure.
  • the matching cost function L r (p,d) of the pixel p at the parallax d can be calculated by:
  • C(p,d) is the matching cost function of the pixel point p at the parallax d
  • the initial value of the matching cost function L r (p,d) is obtained according to the global energy function E(d).
  • S(p,d) is the sum of the cumulative cost functions L r (p,d) of all directions r, so that the initial disparity image can be obtained by computing the minimized S(p,d).
  • the two heterogeneous sensor images by acquiring the two heterogeneous sensor images, calculating the gradient pattern of the two heterogeneous sensor images, traversing the gradient pattern, and calculating the mutual information of the matching window as a matching cost; constructing the global energy by using the matching cost
  • the function is used as a matching cost accumulation function; the initial parallax image of the corresponding pixel is obtained according to the matching cost accumulation function, thereby solving the stereo matching of the infrared light and the visible image based on the infrared visible stereo matching scheme of the gradient direction combined with the mutual information.
  • the problem is to meet the actual application needs.
  • the second embodiment of the present invention provides an image processing apparatus.
  • the apparatus further includes:
  • the parallax image optimization module 205 is configured to: correct the initial parallax image by using a cross-coherence algorithm and an image segmentation algorithm; and smooth the corrected parallax image by using a filtering algorithm.
  • the embodiment considers that there is no matching point in the initial parallax image due to noise, illumination, occlusion, and weak texture region, and reasonable optimization is needed to eliminate errors in the initial parallax image. Match points.
  • the left and right parallax images are respectively acquired, and the parallax error of the corresponding pixel points of the left and right parallax images should be limited to a small range.
  • the expression is as follows:
  • e 12 (p, D 1p ) is the pixel corresponding to the corresponding matching point q in the image 2 based on the disparity value in the image 1 .
  • An image processing method and apparatus by acquiring two heterogeneous sensor images; calculating a gradient direction pattern of the two heterogeneous sensor images; traversing the gradient direction map, calculating mutual information of the matching window as a matching cost; using a matching cost Constructing the global energy function as a matching cost accumulation function; calculating the initial parallax image of the corresponding pixel according to the matching cost accumulation function, thereby solving the stereoscopic image of the infrared light and the visible light image based on the infrared visible stereo matching scheme of the gradient direction combined with the mutual information Matching difficult problems to meet practical application needs.
  • the infrared visible light stereo matching scheme based on the gradient direction combined with the mutual information solves the problem that the stereo matching of the infrared light and the visible light image is difficult, and satisfies the practical application requirements.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

一种图像处理方法及装置，其方法包括：获取两异源传感器图像（S101）；计算两异源传感器图像的梯度方向图（S102）；遍历梯度方向图，计算匹配窗口的互信息，作为匹配代价（S103）；根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像（S104）。

Description

图像处理方法及装置 技术领域
本申请涉及但不限于图像处理技术领域。
背景技术
在图像处理技术中，双目视觉可以用来还原场景的三维信息，是计算机视觉中的一个热点研究内容。其中，立体匹配是双目视觉中最为核心和困难的一个步骤，立体匹配的精确程度影响着后续的景深估计和三维重建的效果，给双目视觉的应用带来了技术瓶颈。
相关技术中，出现了很多立体匹配算法，主要分为全局和局部的立体匹配算法，也有全局和局部相结合的半全局立体匹配算法，这些算法在两幅可见光双目相机上可以获取满足实际应用的效果，但是对异源传感器图像，如红外光和可见光图像的立体匹配，相关的研究工作比较少，并且由于图像传感器的成像原理不同，导致红外光与可见光的图像在色彩上出现较大的差异，使得立体匹配的难度相较双可见光立体匹配要大，从而无法获取满足实际应用的效果。
发明内容
以下是对本文详细描述的主题的概述。本概述并非是为了限制权利要求的保护范围。
本文提出一种图像处理方法及装置，旨在解决红外光与可见光图像的立体匹配难度大的问题。
本发明实施例提供的一种图像处理方法，包括：
获取两异源传感器图像；
计算所述两异源传感器图像的梯度方向图；
遍历所述梯度方向图，计算匹配窗口的互信息，作为匹配代价；
根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像。
可选地，所述根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像的步骤包括：
根据所述匹配代价构造全局能量函数，作为匹配代价累积函数；
结合半全局立体匹配算法，利用多方向的动态规划算法优化所述匹配代价累积函数，获取相应像素点的初始视差图像。
可选地，所述方法还包括：
利用交叉一致性算法和图像分割算法修正所述初始视差图像；
利用滤波算法平滑修正后的视差图像。
可选地，所述修正所述初始视差图像的步骤包括：
排除所述初始视差图像中的错误匹配点。
可选地，所述计算所述两异源传感器图像的梯度方向图的步骤包括：
利用soble算子(索贝尔算子，也称为一阶梯度算子)分别计算两异源传感器图像在水平和垂直方向的梯度大小和方向；
将所述梯度大小和方向归一化处理到适合图像处理的数值范围[0255]，得到原始图像的梯度方向图。
可选地，所述两异源传感器图像分别为红外图像和可见光图像。
可选地，所述互信息用于衡量两异源传感器图像的灰度分布的相关性。
可选地，所述遍历所述梯度方向图，计算匹配窗口的互信息，作为匹配代价的步骤中，对于给定的视差范围[0，Dmax],在视差为D时，右视角梯度方向图I1中的像素点p与左视角梯度方向图I2中的像素点q匹配，通过如下方式计算匹配代价函数C(p，D)：
Figure PCTCN2016081599-appb-000001
Figure PCTCN2016081599-appb-000002
Figure PCTCN2016081599-appb-000003
Figure PCTCN2016081599-appb-000004
其中，PI(i)为图像I中像素值大小为i的概率，
Figure PCTCN2016081599-appb-000005
为图像I1与图像I2的联合灰度对概率，相应的H1为图像I1的熵，
Figure PCTCN2016081599-appb-000006
为图像I1与图像I2的联合熵，
Figure PCTCN2016081599-appb-000007
为图像I1与图像I2的互信息量。
可选地，根据所述匹配代价构造全局能量函数，作为匹配代价累积函数的步骤中：
关于视差图D的全局能量函数E(D),通过如下方式确定：
Figure PCTCN2016081599-appb-000008
其中，C(p，Dp)为像素点p在视差D时的匹配代价函数，T[]函数为截断函数，自变量为1时，值为1，自变量为0，值为0，P1和P2是像素点p与相邻像素点q的视差变化的惩罚权值。
可选地，P2由下式确定：
Figure PCTCN2016081599-appb-000009
其中，P′2为常量，惩罚权值P1为预设的固定值，P2>＝P1。
可选地，所述结合半全局立体匹配算法，利用多方向的动态规划算法优化所述匹配代价累积函数，获取相应像素点的初始视差图像的步骤包括：
通过下式计算在方向r上，像素点p在视差d的匹配代价函数Lr(p,d)：
Figure PCTCN2016081599-appb-000010
其中，C(p,d)为像素点p在视差d时的匹配代价函数，根据全局能量函数E(d)，得到匹配代价函数Lr(p,d)的初始值；
通过下式计算所有方向r的匹配代价函数Lr(p,d)累积之和S(p,d)；
Figure PCTCN2016081599-appb-000011
通过计算最小化的S(p,d)获取初始的视差图像。
本发明实施例还提出一种图像处理装置，包括：
图像获取模块，设置为：获取两异源传感器图像；
梯度方向图计算模块，设置为：计算所述两异源传感器图像的梯度方向图；
匹配代价计算模块，设置为：遍历所述梯度方向图，计算匹配窗口的互信息，作为匹配代价；
全局能量优化模块，设置为：根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像。
可选地，所述全局能量优化模块，是设置为：根据所述匹配代价构造全局能量函数，作为匹配代价累积函数；结合半全局立体匹配算法，利用多方向的动态规划算法优化所述匹配代价累积函数，获取相应像素点的初始视差图像。
可选地，所述装置还包括：
视差图像优化模块，设置为：利用交叉一致性算法和图像分割算法修正所述初始视差图像；利用滤波算法平滑修正后的视差图像。
可选地，所述视差图像优化模块，是设置为：排除所述初始视差图像中的错误匹配点。
可选地，所述梯度方向图计算模块，是设置为：利用soble算子分别计算两异源传感器图像在水平和垂直方向的梯度大小和方向；将所述梯度大 小和方向归一化处理到适合图像处理的数值范围[0 255]，得到原始图像的梯度方向图。
可选地，所述互信息用于衡量两异源传感器图像的灰度分布的相关性。
可选地，所述匹配代价计算模块，是设置为：对于给定的视差范围[0，Dmax],在视差为D时，右视角梯度方向图I1中的像素点p与左视角梯度方向图I2中的像素点q匹配，通过如下方式计算匹配代价函数C(p，D)：
Figure PCTCN2016081599-appb-000012
Figure PCTCN2016081599-appb-000013
Figure PCTCN2016081599-appb-000014
Figure PCTCN2016081599-appb-000015
其中，PI(i)为图像I中像素值大小为i的概率，
Figure PCTCN2016081599-appb-000016
为图像I1与图像I2的联合灰度对概率，相应的H1为图像I1的熵，
Figure PCTCN2016081599-appb-000017
为图像I1与图像I2的联合熵，
Figure PCTCN2016081599-appb-000018
为图像I1与图像I2的互信息量。
可选地，所述全局能量优化模块，是设置为：
关于视差图D的全局能量函数E(D),通过如下方式确定：
Figure PCTCN2016081599-appb-000019
其中，C(p，Dp)为像素点p在视差D时的匹配代价函数，T[]函数为截断函数，自变量为1时，值为1，自变量为0，值为0，P1和P2是像素点p与相邻像素点q的视差变化的惩罚权值。
可选地，P2由下式确定：
Figure PCTCN2016081599-appb-000020
其中，P′2为常量，惩罚权值P1为预设的固定值，P2>＝P1。
可选地，所述全局能量优化模块，是设置为：
通过下式计算在方向r上，像素点p在视差d的匹配代价函数Lr(p,d)：
Figure PCTCN2016081599-appb-000021
其中，C(p,d)为像素点p在视差d时的匹配代价函数，根据全局能量函数E(d)，得到匹配代价函数Lr(p,d)的初始值；
通过下式计算所有方向r的匹配代价函数Lr(p,d)累积之和S(p,d)；
Figure PCTCN2016081599-appb-000022
通过计算最小化的S(p,d)获取初始的视差图像。
可选地，所述两异源传感器图像分别为红外图像和可见光图像。
本发明实施例还提出一种计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令用于执行上述任一项的方法。
本发明实施例提出的一种图像处理方法及装置，通过获取两异源传感器图像；计算所述两异源传感器图像的梯度方向图；遍历梯度方向图，计算匹配窗口的互信息，作为匹配代价；利用匹配代价构造全局能量函数， 作为匹配代价累积函数；根据匹配代价累积函数计算获取相应像素点的初始视差图像，由此基于梯度方向结合互信息的红外可见光立体匹配方案，解决了红外光与可见光图像的立体匹配难度大的问题，满足实际应用需求。
在阅读并理解了附图和详细描述后，可以明白其他方面。
附图概述
图1是实现本发明实施例的移动终端的可选的硬件结构示意图；
图2是如图1所示的移动终端的无线通信***示意图；
图3是本发明图像处理方法第一实施例的流程示意图；
图4是本发明实施例中像素点p与视差d在方向r上的匹配代价函数示意图；
图5是本发明实施例中基于匹配代价累积函数获取初始的视差图像的示意图；
图6是本发明图像处理方法第二实施例的流程示意图；
图7(a)是本发明实施例中一种原始可见光图像；
图7(b)是本发明实施例中一种原始红外图像；
图7(c)是图7(a)中可见光梯度方向图；
图7(d)是图7(b)中红外光梯度方向图；
图7(e)是图7(a)所示的原始可见光图像和图7(b)所示的原始红外图像进行图像处理后得到的初始视差图像示意图；
图7(f)是图7(e)中的视差图像修正后的示意图；
图8是本发明图像处理装置第一实施例的功能模块示意图；
图9是本发明图像处理装置第二实施例的功能模块示意图。
本发明的实施方式
应当理解，此处所描述的实施例仅仅用以解释本发明，并不用于限定本发明。
现在将参考附图描述实现本发明实施例的移动终端。在后续的描述中，使用用于表示元件的诸如“模块”、“部件”或“单元”的后缀仅为了有利于本发明实施例的说明，其本身并没有特定的意义。因此，"模块"与"部件"可以混合地使用。
移动终端可以以多种形式来实施。例如，本文中描述的终端可以包括诸如移动电话、智能电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、导航装置等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。下面，假设终端是移动终端。然而，本领域技术人员将理解的是，除了特别用于移动目的的元件之外，根据本发明的实施方式的构造也能够应用于固定类型的终端。
图1为实现本发明实施例的移动终端的可选的硬件结构示意图。
移动终端100可以包括无线通信单元110、A/V(音频/视频)输入单元120、用户输入单元130、感测单元140、输出单元150、存储器160、接口单元170、控制器180和电源单元190等等。图1示出了具有多种组件的移动终端，但是应理解的是，并不要求实施所有示出的组件。可以替代地实施更多或更少的组件。将在下面详细描述移动终端的元件。
无线通信单元110通常包括一个或多个组件，其允许移动终端100与无线通信***或网络之间的无线电通信。例如，无线通信单元可以包括广播接收模块111、移动通信模块112、无线互联网模块113、短程通信模块114和位置信息模块115中的至少一个。
广播接收模块111经由广播信道从外部广播管理服务器接收广播信号和/或广播相关信息。广播信道可以包括卫星信道和/或地面信道。广播管理服务器可以是生成并发送广播信号和/或广播相关信息的服务器或者接收之前生成的广播信号和/或广播相关信息并且将其发送给终端的服务器。广播信号可以包括TV广播信号、无线电广播信号、数据广播信号等等。而且，广播信号可以进一步包括与TV或无线电广播信号组合的广播信号。广播相关信息也可以经由移动通信网络提供，并且在该情况下，广播相关信息可以由 移动通信模块112来接收。广播信号可以以多种形式存在，例如，其可以以数字多媒体广播(DMB)的电子节目指南(EPG)、数字视频广播手持(DVB-H)的电子服务指南(ESG)等等的形式而存在。广播接收模块111可以通过使用多种类型的广播***接收信号广播。特别地，广播接收模块111可以通过使用诸如多媒体广播-地面(DMB-T)、数字多媒体广播-卫星(DMB-S)、数字视频广播-手持(DVB-H)，前向链路媒体(MediaFLO@)的数据广播***、地面数字广播综合服务(ISDB-T)等等的数字广播***接收数字广播。广播接收模块111可以被构造为适合提供广播信号的广播***以及上述数字广播***。经由广播接收模块111接收的广播信号和/或广播相关信息可以存储在存储器160(或者其它类型的存储介质)中。
移动通信模块112将无线电信号发送到基站(例如，接入点、节点B等等)、外部终端以及服务器中的至少一个和/或从其接收无线电信号。这样的无线电信号可以包括语音通话信号、视频通话信号、或者根据文本和/或多媒体消息发送和/或接收的多种类型的数据。
无线互联网模块113支持移动终端的无线互联网接入。该模块可以内部或外部地耦接到终端。该模块所涉及的无线互联网接入技术可以包括WLAN(无线LAN)(Wi-Fi)、Wibro(无线宽带)、Wimax(全球微波互联接入)、HSDPA(高速下行链路分组接入)等等。
短程通信模块114设置为支持短程通信。短程通信技术的一些示例包括蓝牙TM、射频识别(RFID)、红外数据协会(IrDA)、超宽带(UWB)、紫蜂TM等等。
位置信息模块115设置为检查或获取移动终端的位置信息。位置信息模块的典型示例是GPS(全球定位***)。根据当前的技术，GPS模块115计算来自三个或更多卫星的距离信息和准确的时间信息并且对于计算的信息应用三角测量法，从而根据经度、纬度和高度准确地计算三维当前位置信息。当前，用于计算位置和时间信息的方法使用三颗卫星并且通过使用另外的一颗卫星校正计算出的位置和时间信息的误差。此外，GPS模块115能够通过实时地连续计算当前位置信息来计算速度信息。
A/V输入单元120设置为接收音频或视频信号。A/V输入单元120可以包括相机121和麦克风122，相机121对在视频捕获模式或图像捕获模式中由图 像捕获装置获得的静态图片或视频的图像数据进行处理。处理后的图像帧可以显示在显示单元151上。经相机121处理后的图像帧可以存储在存储器160(或其它存储介质)中或者经由无线通信单元110进行发送，可以根据移动终端的构造提供两个或更多相机121。麦克风122可以在电话通话模式、记录模式、语音识别模式等等运行模式中经由麦克风接收声音(音频数据)，并且能够将这样的声音处理为音频数据。处理后的音频(语音)数据可以在电话通话模式的情况下转换为可经由移动通信模块112发送到移动通信基站的格式输出。麦克风122可以通过噪声消除(或抑制)算法以消除(或抑制)在接收和发送音频信号的过程中产生的噪声或者干扰。
用户输入单元130可以根据用户输入的命令生成键输入数据以控制移动终端的操作。用户输入单元130允许用户输入多种类型的信息，并且可以包括键盘、锅仔片、触摸板(例如，检测由于被接触而导致的电阻、压力、电容等等的变化的触敏组件)、滚轮、摇杆等等。特别地，当触摸板以层的形式叠加在显示单元151上时，可以形成触摸屏。
感测单元140检测移动终端100的当前状态，(例如，移动终端100的打开或关闭状态)、移动终端100的位置、用户对于移动终端100的接触(即，触摸输入)的有无、移动终端100的取向、移动终端100的加速或减速移动和方向等等，并且生成用于控制移动终端100的操作的命令或信号。例如，当移动终端100实施为滑动型移动电话时，感测单元140可以感测该滑动型电话是打开还是关闭。另外，感测单元140能够检测电源单元190是否提供电力或者接口单元170是否与外部装置耦接。感测单元140可以包括接近传感器141将在下面结合触摸屏来对此进行描述。
接口单元170用作至少一个外部装置与移动终端100连接可以通过的接口。例如，外部装置可以包括有线或无线头戴式耳机端口、外部电源(或电池充电器)端口、有线或无线数据端口、存储卡端口、用于连接具有识别模块的装置的端口、音频输入/输出(I/O)端口、视频I/O端口、耳机端口等等。识别模块可以是存储用于验证用户使用移动终端100的多种信息并且可以包括用户识别模块(UIM)、客户识别模块(SIM)、通用客户识别模块(USIM)等等。另外，具有识别模块的装置(下面称为"识别装置")可以采取智能卡的形式，因此，识别装置可以经由端口或其它连接装置与移动终端100连接。接 口单元170可以用于接收来自外部装置的输入(例如，数据信息、电力等等)并且将接收到的输入传输到移动终端100内的一个或多个元件或者可以用于在移动终端和外部装置之间传输数据。
另外，当移动终端100与外部底座连接时，接口单元170可以用作允许通过其将电力从底座提供到移动终端100的路径或者可以用作允许从底座输入的多种命令信号通过其传输到移动终端的路径。从底座输入的多种命令信号或电力可以用作用于识别移动终端是否准确地安装在底座上的信号。输出单元150被构造为以视觉、音频和/或触觉方式提供输出信号(例如，音频信号、视频信号、警报信号、振动信号等等)。输出单元150可以包括显示单元151、音频输出模块152、警报单元153等等。
显示单元151可以显示在移动终端100中处理的信息。例如，当移动终端100处于电话通话模式时，显示单元151可以显示与通话或其它通信(例如，文本消息收发、多媒体文件下载等等)相关的用户界面(UI)或图形用户界面(GUI)。当移动终端100处于视频通话模式或者图像捕获模式时，显示单元151可以显示捕获的图像和/或接收的图像、示出视频或图像以及相关功能的UI或GUI等等。
同时，当显示单元151和触摸板以层的形式彼此叠加以形成触摸屏时，显示单元151可以用作输入装置和输出装置。显示单元151可以包括液晶显示器(LCD)、薄膜晶体管LCD(TFT-LCD)、有机发光二极管(OLED)显示器、柔性显示器、三维(3D)显示器等等中的至少一种。这些显示器中的一些可以被构造为透明状以允许用户从外部观看，这可以称为透明显示器，典型的透明显示器可以例如为TOLED(透明有机发光二极管)显示器等等。根据特定想要的实施方式，移动终端100可以包括两个或更多显示单元(或其它显示装置)，例如，移动终端可以包括外部显示单元(未示出)和内部显示单元(未示出)。触摸屏可设置为检测触摸输入压力以及触摸输入位置和触摸输入面积。
音频输出模块152可以在移动终端处于呼叫信号接收模式、通话模式、记录模式、语音识别模式、广播接收模式等等模式下时，将无线通信单元110接收的或者在存储器160中存储的音频数据转换音频信号并且输出为声音。而且，音频输出模块152可以提供与移动终端100执行的特定功能相关的音频输出(例如，呼叫信号接收声音、消息接收声音等等)。音频输出模块152可 以包括扬声器、蜂鸣器等等。
警报单元153可以提供输出以将事件的发生通知给移动终端100。典型的事件可以包括呼叫接收、消息接收、键信号输入、触摸输入等等。除了音频或视频输出之外，警报单元153可以以不同的方式提供输出以通知事件的发生。例如，警报单元153可以以振动的形式提供输出，当接收到呼叫、消息或一些其它进入通信(incomingcommunication)时，警报单元153可以提供触觉输出(即，振动)以将其通知给用户。通过提供这样的触觉输出，即使在用户的移动电话处于用户的口袋中时，用户也能够识别出事件的发生。警报单元153也可以经由显示单元151或音频输出模块152提供通知事件的发生的输出。
存储器160可以存储由控制器180执行的处理和控制操作的软件程序等等，或者可以暂时地存储己经输出或将要输出的数据(例如，电话簿、消息、静态图像、视频等等)。而且，存储器160可以存储关于当触摸施加到触摸屏时输出的多种方式的振动和音频信号的数据。
存储器160可以包括至少一种类型的存储介质，所述存储介质包括闪存、硬盘、多媒体卡、卡型存储器(例如，SD或DX存储器等等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘等等。而且，移动终端100可以与通过网络连接执行存储器160的存储功能的网络存储装置协作。
控制器180通常控制移动终端的总体操作。例如，控制器180执行与语音通话、数据通信、视频通话等等相关的控制和处理。另外，控制器180可以包括用于再现(或回放)多媒体数据的多媒体模块181，多媒体模块181可以构造在控制器180内，或者可以构造为与控制器180分离。控制器180可以执行模式识别处理，以将在触摸屏上执行的手写输入或者图片绘制输入识别为字符或图像。
电源单元190在控制器180的控制下接收外部电力或内部电力并且提供操作每个元件和组件所需的适当的电力。
这里描述的实施方式可以以使用例如计算机软件、硬件或其任何组合的计算机可读介质来实施。对于硬件实施，这里描述的实施方式可以通过使用 特定用途集成电路(ASIC)、数字信号处理器(DSP)、数字信号处理装置(DSPD)、可编程逻辑装置(PLD)、现场可编程门阵列(FPGA)、处理器、控制器、微控制器、微处理器、被设计为执行这里描述的功能的电子单元中的至少一种来实施，在一些情况下，这样的实施方式可以在控制器180中实施。对于软件实施，诸如过程或功能的实施方式可以与允许执行至少一种功能或操作的单独的软件模块来实施。软件代码可以由以任何适当的编程语言编写的软件应用程序(或程序)来实施，软件代码可以存储在存储器160中并且由控制器180执行。
至此，己经按照其功能描述了移动终端。下面，为了简要起见，将描述诸如折叠型、直板型、摆动型、滑动型移动终端等等的多种类型的移动终端中的滑动型移动终端作为示例。因此，本发明实施例能够应用于任何类型的移动终端，并且不限于滑动型移动终端。
如图1中所示的移动终端100可以被构造为利用经由帧或分组发送数据的诸如有线和无线通信***以及基于卫星的通信***来操作。
现在将参考图2描述其中根据本发明实施例的移动终端能够操作的通信***。
这样的通信***可以使用不同的空中接口和/或物理层。例如，由通信***使用的空中接口包括例如频分多址(FDMA)、时分多址(TDMA)、码分多址(CDMA)和通用移动通信***(UMTS)(特别地，长期演进(LTE))、全球移动通信***(GSM)等等。作为非限制性示例，下面的描述涉及CDMA通信***，但是这样的教导同样适用于其它类型的***。
参考图2，CDMA无线通信***可以包括多个移动终端100、多个基站(BS)270、基站控制器(BSC)275和移动交换中心(MSC)280。MSC280被构造为与公共电话交换网络(PSTN)290形成接口。MSC280还被构造为与可以经由回程线路耦接到基站270的BSC275形成接口。回程线路可以根据多个己知的接口中的任一种来构造，所述接口包括例如E1/T1、ATM，IP、PPP、帧中继、HDSL、ADSL或xDSL。将理解的是，如图2中所示的***可以包括多个BSC275。
每个BS270可以服务一个或多个分区(或区域)，由多向天线或指向特定方向的天线覆盖的每个分区放射状地远离BS270。或者，每个分区可以由用 于分集接收的两个或更多天线覆盖。每个BS270可以被构造为支持多个频率分配，并且每个频率分配具有特定频谱(例如，1.25MHz,5MHz等等)。
分区与频率分配的交叉可以被称为CDMA信道。BS270也可以被称为基站收发器子***(BTS)或者其它等效术语。在这样的情况下，术语"基站"可以用于笼统地表示单个BSC275和至少一个BS270。基站也可以被称为"蜂窝站"。或者，特定BS270的多个分区可以被称为多个蜂窝站。
如图2中所示，广播发射器(BT)295将广播信号发送给在***内操作的移动终端100。如图1中所示的广播接收模块111被设置在移动终端100处以接收由BT295发送的广播信号。在图2中，示出了几个全球定位***(GPS)卫星300。卫星300帮助定位多个移动终端100中的至少一个。
在图2中，描绘了多个卫星300，但是理解的是，可以利用任何数目的卫星获得有用的定位信息。如图1中所示的GPS模块115通常被构造为与卫星300配合以获得想要的定位信息。替代GPS跟踪技术或者在GPS跟踪技术之外，可以使用可以跟踪移动终端的位置的其它技术。另外，至少一个GPS卫星300可以选择性地或者额外地处理卫星DMB传输。
作为无线通信***的一个典型操作，BS270接收来自移动终端100的反向链路信号。移动终端100通常参与通话、消息收发和其它类型的通信。特定基站270接收的每个反向链路信号被在特定BS270内进行处理。获得的数据被转发给相关的BSC275。BSC提供通话资源分配和包括BS270之间的软切换过程的协调的移动管理功能。BSC275还将接收到的数据路由到MSC280，其提供用于与PSTN290形成接口的额外的路由服务。类似地，PSTN290与MSC280形成接口，MSC与BSC275形成接口，并且BSC275相应地控制BS270以将正向链路信号发送到移动终端100。
基于上述移动终端硬件结构以及通信***，提出本发明方法实施例。
由于相关技术的双目视觉立体匹配算法，在两幅可见光双目相机上可以获取满足实际应用的效果，但是对异源传感器图像，如红外光和可见光图像的立体匹配，由于图像传感器的成像原理不同，导致红外与可见光的图像在色彩上出现较大的差异，使得立体匹配的难度相较双可见光立体匹配要大，从而无法获取满足实际应用的效果。
为此，本文提出一种解决方案，可以很好的解决红外光与可见光图像的立体匹配难度大的问题。
如图3所示，本发明第一实施例提出一种图像处理方法，包括：
步骤S101，获取两异源传感器图像；
其中，两异源传感器图像可以分别为红外图像和可见光图像，当然也可以为其他两种不同类型的图像，本实施例以红外拍摄的图片与可见光拍摄的图片做图像处理进行举例。
步骤S102，计算所述两异源传感器图像的梯度方向图；
利用soble算子计算两异源传感器图像在水平和垂直方向的梯度大小和方向。该算子包含两组3x3的矩阵，分别为横向及纵向，将之与图像作平面卷积，即可分别得出横向及纵向的亮度差分近似值。
如果以I代表原始图像，Gx及Gy分别代表经纵向及横向边缘检测的图像，G为该图像像素点处的梯度大小，θ为梯度方向，其公式如下：
Figure PCTCN2016081599-appb-000023
Figure PCTCN2016081599-appb-000024
这里计算的梯度方向范围是(-π/2，π/2)，会包含负数，首先可以将θ归一化到(0，π)，之后归一化到适合图像处理的[0,255]范围，得到原始图像的梯度方向图像：
Figure PCTCN2016081599-appb-000025
步骤S103，遍历所述梯度方向图，计算匹配窗口的互信息，作为匹配代价；
在得到梯度方向图后，计算匹配窗口的互信息，作为匹配代价。匹配代价的意义在于：找到特征上的相似点，原理是找差异最小的点作为相似点。
对于给定的视差范围[0，Dmax],在视差为D时，右视角梯度方向图I1中的像素点p与左视角梯度方向图I2中的像素点q匹配，匹配代价函数C(p，D)计算方式如下式：
Figure PCTCN2016081599-appb-000026
Figure PCTCN2016081599-appb-000027
Figure PCTCN2016081599-appb-000028
Figure PCTCN2016081599-appb-000029
其中，PI(i)为图像I中像素值大小为i的概率，
Figure PCTCN2016081599-appb-000030
为图像I1与图像I2的联合灰度对概率，相应的H1为图像I1的熵，
Figure PCTCN2016081599-appb-000031
为图像I1与图像I2的联合熵，
Figure PCTCN2016081599-appb-000032
为图像I1与图像I2的互信息量，图像的互信息可以衡量两幅图像的灰度分布的相关性，互信息量越大，说明两幅图像越相近。
步骤S104，根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像。
首先，利用所述匹配代价构造全局能量函数，作为匹配代价累积函数；
本实施例考虑到，初始的匹配代价函数容易受到噪声和视角遮挡产生误匹配，需要利用视差的连续性约束，添加一个视差变化的平滑项。因此，定义一个关于视差图D的全局能量函数E(D),如式子：
Figure PCTCN2016081599-appb-000033
其中，C(p，Dp)为像素点p在视差D时的匹配代价，是由步骤S103中计算的MI。
T[]函数是一个截断函数，自变量为1时，值为1，自变量为0，值为0。P1和P2是像素点p与相邻像素点q的视差变化的惩罚权值，如果p与q的视差变化较小，则给定一个固定较小的惩罚权值P1；如果变化较大，则给予一个较大的惩罚值P2，P2的计算方式如下：P′2为常量，一般P2>＝P1。
Figure PCTCN2016081599-appb-000034
然后，结合半全局立体匹配算法，利用多方向的动态规划算法优化所述匹配代价累积函数，获取相应像素点的初始视差图像。
对于全局能量函数E(D)，计算出最小的E(D)，可以获取初始的视差图像。
由于二维图像能量函数的不连续性，这个全局最小化问题一般需要转换到一维空间利用动态规划的方法来求解，单方向的动态规划优化方法会出现水平条纹的问题，这里采用多方向的动态规划优化策略，如图4及图5所示，图4是本发明实施例中像素点p与视差d在方向r上的匹配代价函数示意图；图5是本发明实施例中基于匹配代价累积函数获取初始的视差图像的示意图。
在方向r上，像素点p在视差d的匹配代价函数Lr(p,d)可以通过下式来计算：
Figure PCTCN2016081599-appb-000035
Figure PCTCN2016081599-appb-000036
其中，C(p,d)为像素点p在视差d时的匹配代价函数，根据全局能量函数E(d)得到匹配代价函数Lr(p,d)的初始值。
S(p,d)为所有方向r的匹配代价函数Lr(p,d)累积之和，这样通过计算最小化的S(p,d)即可获取初始的视差图像。
本实施例通过上述方案，通过获取两异源传感器图像；计算所述两异源传感器图像的梯度方向图；遍历梯度方向图，计算匹配窗口的互信息，作为匹配代价；利用匹配代价构造全局能量函数，作为匹配代价累积函数；根据匹配代价累积函数计算获取相应像素点的初始视差图像，由此基于梯度方向结合互信息的红外可见光立体匹配方案，解决了红外光与可见光图像的立体匹配难度大的问题，满足实际应用需求。
如图6所示，本发明第二实施例提出一种图像处理方法，基于上述图3所示的实施例，该方法还包括：
步骤S105，利用交叉一致性算法和图像分割算法修正所述初始视差图像；
步骤S106，利用滤波算法平滑修正后的视差图像。
相比上述实施例，本实施例考虑到，初始的视差图像中会存在由于噪声、光照、遮挡、弱纹理区域引起的无匹配点，需要进行合理的优化，排除所述初始视差图像中的错误匹配点。
首先，依据视差唯一性这一约束条件，分别获取左右视差图像，左右视差图像对应像素点的视差值误差应限定在较小的范围。表达式如下：
Figure PCTCN2016081599-appb-000037
q＝e12(p，D1p)；
式中e12(p，D1p)是像素点p根据其在图像1中的视差值计算出在图像2中的对应匹配点q。左右一致性检测后，视差图像中会出现空洞，需要通过邻域插值和中值滤波的方法来进行插值和平滑视差图像，获取最后的视差图像。
基于上述实施例原理，下面结合实例图像，对本发明实施例图像处理的过程进行说明：
首先，获取两异源传感器图像，分别如图7(a)及图7(b)所示，图 7(a)为原始可见光图像，图7(b)为原始红外图像。
然后，计算两异源传感器图像的梯度方向图，分别如图7(c)及图7(d)所示，图7(c)为可见光梯度方向图，图7(d)为红外光梯度方向图。
之后，遍历所述梯度方向图，计算匹配窗口的互信息，作为匹配代价；
然后，根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像，如图7(e)所示。
最后，利用交叉一致性算法和图像分割算法修正所述初始视差图像；利用滤波算法平滑修正后的视差图像，修正后的视差图像如图7(f)所示，在视差图像中，像素点的颜色越亮，表示该处的物体的视差越大，距离相机的距离越近。
本发明实施例还提出一种计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令用于执行上述图像处理方法。
对应地，提出本发明图像处理装置实施例。
如图8所示，本发明第一实施例提出一种图像处理装置，包括：图像获取模块201、梯度方向图计算模块202、匹配代价计算模块203及全局能量优化模块204，其中：
图像获取模块201，设置为：获取两异源传感器图像；
梯度方向图计算模块202，设置为：计算所述两异源传感器图像的梯度方向图；
匹配代价计算模块203，设置为：遍历所述梯度方向图，计算匹配窗口的互信息，作为匹配代价；
全局能量优化模块204，设置为：根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像。
其中，两异源传感器图像可以分别为红外图像和可见光图像，当然也可以为其他两种不同类型的图像，本实施例以红外拍摄的图片与可见光拍摄的图片做图像处理进行举例。
首先，计算所述两异源传感器图像的梯度方向图。
利用soble算子计算两异源传感器图像在水平和垂直方向的梯度大小和 方向。该算子包含两组3x3的矩阵，分别为横向及纵向，将之与图像作平面卷积，即可分别得出横向及纵向的亮度差分近似值。
如果以I代表原始图像，Gx及Gy分别代表经纵向及横向边缘检测的图像，G为该图像像素点处的梯度大小，θ为梯度方向，其公式如下：
Figure PCTCN2016081599-appb-000038
Figure PCTCN2016081599-appb-000039
这里计算的梯度方向范围是(-π/2，π/2)，会包含负数，首先可以将θ归一化到(0，π)，之后归一化到适合图像处理的[0,255]范围，得到原始图像的梯度方向图像：
Figure PCTCN2016081599-appb-000040
在得到梯度方向图后，计算匹配窗口的互信息，作为匹配代价。匹配代价的意义在于：找到特征上的相似点，原理是找差异最小的点作为相似点。
对于给定的视差范围[0，Dmax],在视差为D时，右视角梯度方向图I1中的像素点p与左视角梯度方向图I2中的像素点q匹配，匹配代价函数C(p，D)计算方式如下式：
Figure PCTCN2016081599-appb-000041
Figure PCTCN2016081599-appb-000042
Figure PCTCN2016081599-appb-000043
Figure PCTCN2016081599-appb-000044
其中，PI(i)为图像I中像素值大小为i的概率，
Figure PCTCN2016081599-appb-000045
为图像I1与图像I2的联合灰度对概率，相应的H1为图像I1的熵，
Figure PCTCN2016081599-appb-000046
为图像I1与图像I2的联合熵，
Figure PCTCN2016081599-appb-000047
为图像I1与图像I2的互信息量，图像的互信息可以衡量 两幅图像的灰度分布的相关性，互信息量越大，说明两幅图像越相近。
然后，根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像。
首先，利用所述匹配代价构造全局能量函数，作为匹配代价累积函数；
本实施例考虑到，初始的匹配代价函数容易受到噪声和视角遮挡产生误匹配，需要利用视差的连续性约束，添加一个视差变化的平滑项。因此，定义一个关于视差图D的全局能量函数E(D),如式子：
Figure PCTCN2016081599-appb-000048
其中，C(p，Dp)为像素点p在视差D时的匹配代价，是由上述计算得到的MI。
T[]函数是一个截断函数，自变量为1时，值为1，自变量为0，值为0。P1和P2是像素点p与相邻像素点q的视差变化的惩罚权值，如果p与q的视差变化较小，则给定一个固定较小的惩罚权值P1；如果变化较大，则给予一个较大的惩罚值P2，P2的计算方式如下：P′2为常量，一般P2>＝P1。
Figure PCTCN2016081599-appb-000049
然后，结合半全局立体匹配算法，利用多方向的动态规划算法优化所述匹配代价累积函数，获取相应像素点的初始视差图像。
对于全局能量函数E(D)，计算出最小的E(D)，可以获取初始的视差图像。
由于二维图像能量函数的不连续性，这个全局最小化问题一般需要转换到一维空间利用动态规划的方法来求解，单方向的动态规划优化方法会出现水平条纹的问题，这里采用多方向的动态规划优化策略，如图4及图5所示，图4是本发明实施例中像素点p与视差d在方向r上的匹配代价函数示意图；图5是本发明实施例中基于匹配代价累积函数获取初始的视差图像的示意 图。
在方向r上，像素点p在视差d的匹配代价函数Lr(p,d)可以通过下式来计算：
Figure PCTCN2016081599-appb-000050
Figure PCTCN2016081599-appb-000051
其中，C(p,d)为像素点p在视差d时的匹配代价函数，根据全局能量函数E(d)得到匹配代价函数Lr(p,d)的初始值。
S(p,d)为所有方向r的匹配代价函数Lr(p,d)累积之和，这样通过计算最小化的S(p,d)即可获取初始的视差图像。
本实施例通过上述方案，通过获取两异源传感器图像；计算所述两异源传感器图像的梯度方向图；遍历梯度方向图，计算匹配窗口的互信息，作为匹配代价；利用匹配代价构造全局能量函数，作为匹配代价累积函数；根据匹配代价累积函数计算获取相应像素点的初始视差图像，由此基于梯度方向结合互信息的红外可见光立体匹配方案，解决了红外光与可见光图像的立体匹配难度大的问题，满足实际应用需求。
如图9所示，本发明第二实施例提出一种图像处理装置，基于上述图8所示的实施例，所述装置还包括：
视差图像优化模块205，设置为：利用交叉一致性算法和图像分割算法修正所述初始视差图像；利用滤波算法平滑修正后的视差图像。
相比上述实施例，本实施例考虑到，初始的视差图像中会存在由于噪声、光照、遮挡、弱纹理区域引起的无匹配点，需要进行合理的优化，排除所述初始视差图像中的错误匹配点。
首先，依据视差唯一性这一约束条件，分别获取左右视差图像，左右视差图像对应像素点的视差值误差应限定在较小的范围。表达式如下：
Figure PCTCN2016081599-appb-000052
q＝e12(p，D1p)；
式中e12(p，D1p)是像素点p根据其在图像1中的视差值计算出在图像2中的对应匹配点q。左右一致性检测后，视差图像中会出现空洞，需要通过邻域插值和中值滤波的方法来进行插值和平滑视差图像，获取最后的视差图像。
本发明实施例图像处理方法及装置，通过获取两异源传感器图像；计算所述两异源传感器图像的梯度方向图；遍历梯度方向图，计算匹配窗口的互信息，作为匹配代价；利用匹配代价构造全局能量函数，作为匹配代价累积函数；根据匹配代价累积函数计算获取相应像素点的初始视差图像，由此基于梯度方向结合互信息的红外可见光立体匹配方案，解决了红外光与可见光图像的立体匹配难度大的问题，满足实际应用需求。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
上述本发明实施例序号仅仅为了描述，不代表实施例的优劣。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本文的技 术方案本质上或者说对已有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括多个指令用以使得一台终端设备(可以是手机，计算机，服务器，空调器，或者网络设备等)执行本发明实施例所述的方法。
以上仅为本发明的实施例，并非因此限制本发明的专利范围，凡是利用本发明说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本发明的专利保护范围内。
工业实用性
本发明实施例基于梯度方向结合互信息的红外可见光立体匹配方案，解决了红外光与可见光图像的立体匹配难度大的问题，满足实际应用需求。

Claims (20)

  1. 一种图像处理方法，包括：
    获取两异源传感器图像；
    计算所述两异源传感器图像的梯度方向图；
    遍历所述梯度方向图，计算匹配窗口的互信息，作为匹配代价；
    根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像。
  2. 根据权利要求1所述的方法，其中，所述根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像的步骤包括：
    根据所述匹配代价构造全局能量函数，作为匹配代价累积函数；
    结合半全局立体匹配算法，利用多方向的动态规划算法优化所述匹配代价累积函数，获取相应像素点的初始视差图像。
  3. 根据权利要求1或2所述的方法，所述方法还包括：
    利用交叉一致性算法和图像分割算法修正所述初始视差图像；
    利用滤波算法平滑修正后的视差图像。
  4. 根据权利要求3所述的方法，其中，所述修正所述初始视差图像的步骤包括：
    排除所述初始视差图像中的错误匹配点。
  5. 根据权利要求3所述的方法，其中，所述计算所述两异源传感器图像的梯度方向图的步骤包括：
    利用soble算子分别计算两异源传感器图像在水平和垂直方向的梯度大小和方向；
    将所述梯度大小和方向归一化处理到适合图像处理的数值范围[0 255]，得到原始图像的梯度方向图。
  6. 根据权利要求3所述的方法，其中，所述两异源传感器图像分别为红外图像和可见光图像。
  7. 根据权利要求3所述的方法，其中，所述互信息用于衡量两异源传感器图像的灰度分布的相关性。
  8. 根据权利要求7所述的方法，其中，所述遍历所述梯度方向图，计算匹配窗口的互信息，作为匹配代价的步骤中，对于给定的视差范围[0，Dmax],在视差为D时，右视角梯度方向图I1中的像素点p与左视角梯度方向图I2中的像素点q匹配，通过如下方式计算匹配代价函数C(p，D)：
    Figure PCTCN2016081599-appb-100001
    Figure PCTCN2016081599-appb-100002
    Figure PCTCN2016081599-appb-100003
    Figure PCTCN2016081599-appb-100004
    其中，PI(i)为图像I中像素值大小为i的概率，
    Figure PCTCN2016081599-appb-100005
    为图像I1与图像I2的联合灰度对概率，相应的H1为图像I1的熵，
    Figure PCTCN2016081599-appb-100006
    为图像I1与图像I2的联合熵，
    Figure PCTCN2016081599-appb-100007
    为图像I1与图像I2的互信息量。
  9. 根据权利要求2所述的方法，其中，根据所述匹配代价构造全局能量函数，作为匹配代价累积函数的步骤中：
    关于视差图D的全局能量函数E(D),通过如下方式确定：
    Figure PCTCN2016081599-appb-100008
    其中，C(p，Dp)为像素点p在视差D时的匹配代价函数，T[]函数为截断函数，自变量为1时，值为1，自变量为0，值为0，P1和P2是像素点p与相邻像素点q的视差变化的惩罚权值。
  10. 根据权利要求9所述的方法，其中，所述结合半全局立体匹配算法，利用多方向的动态规划算法优化所述匹配代价累积函数，获取相应像素点的初始视差图像的步骤包括：
    通过下式计算在方向r上，像素点p在视差d的匹配代价函数Lr(p,d)：
    Figure PCTCN2016081599-appb-100009
    其中，C(p,d)为像素点p在视差d时的匹配代价函数，根据全局能量函数E(d)，得到匹配代价函数Lr(p,d)的初始值；
    通过下式计算所有方向r的匹配代价函数Lr(p,d)累积之和S(p,d)；
    Figure PCTCN2016081599-appb-100010
    通过计算最小化的S(p,d)获取初始的视差图像。
  11. 一种图像处理装置，包括：
    图像获取模块，设置为：获取两异源传感器图像；
    梯度方向图计算模块，设置为：计算所述两异源传感器图像的梯度方向图；
    匹配代价计算模块，设置为：遍历所述梯度方向图，计算匹配窗口的互信息，作为匹配代价；
    全局能量优化模块，设置为：根据所述匹配代价并结合预设的立体匹配算法，计算获取相应像素点的初始视差图像。
  12. 根据权利要求11所述的装置，其中，
    所述全局能量优化模块，是设置为：根据所述匹配代价构造全局能量函数，作为匹配代价累积函数；结合半全局立体匹配算法，利用多方向的动态规划算法优化所述匹配代价累积函数，获取相应像素点的初始视差图像。
  13. 根据权利要求11或12所述的装置，所述装置还包括：
    视差图像优化模块，设置为：利用交叉一致性算法和图像分割算法修正所述初始视差图像；利用滤波算法平滑修正后的视差图像。
  14. 根据权利要求13所述的装置，其中，
    所述视差图像优化模块，是设置为：排除所述初始视差图像中的错误匹配点。
  15. 根据权利要求13所述的装置，其中，
    所述梯度方向图计算模块，是设置为：利用soble算子分别计算两异源传感器图像在水平和垂直方向的梯度大小和方向；将所述梯度大小和方向归一化处理到适合图像处理的数值范围[0 255]，得到原始图像的梯度方向图。
  16. 根据权利要求13所述的装置，其中，所述互信息用于衡量两异源传感器图像的灰度分布的相关性。
  17. 根据权利要求16所述的装置，其中，所述匹配代价计算模块，是设置为：对于给定的视差范围[0，Dmax],在视差为D时，右视角梯度方向图I1中的像素点p与左视角梯度方向图I2中的像素点q匹配，通过如下方式计算匹配代价函数C(p，D)：
    Figure PCTCN2016081599-appb-100011
    Figure PCTCN2016081599-appb-100012
    Figure PCTCN2016081599-appb-100013
    Figure PCTCN2016081599-appb-100014
    其中，PI(i)为图像I中像素值大小为i的概率，
    Figure PCTCN2016081599-appb-100015
    为图像I1与图像I2的联合灰度对概率，相应的H1为图像I1的熵，
    Figure PCTCN2016081599-appb-100016
    为图像I1与图像I2的联合熵，
    Figure PCTCN2016081599-appb-100017
    为图像I1与图像I2的互信息量。
  18. 根据权利要求12所述的装置，其中，所述全局能量优化模块，是 设置为：
    关于视差图D的全局能量函数E(D),通过如下方式确定：
    Figure PCTCN2016081599-appb-100018
    其中，C(p，Dp)为像素点p在视差D时的匹配代价函数，T[]函数为截断函数，自变量为1时，值为1，自变量为0，值为0，P1和P2是像素点p与相邻像素点q的视差变化的惩罚权值。
  19. 根据权利要求18所述的装置，其中，所述全局能量优化模块，是设置为：
    通过下式计算在方向r上，像素点p在视差d的匹配代价函数Lr(p,d)：
    Figure PCTCN2016081599-appb-100019
    其中，C(p,d)为像素点p在视差d时的匹配代价函数，根据全局能量函数E(d)，得到匹配代价函数Lr(p,d)的初始值；
    通过下式计算所有方向r的匹配代价函数Lr(p,d)累积之和S(p,d)；
    Figure PCTCN2016081599-appb-100020
    通过计算最小化的S(p,d)获取初始的视差图像。
  20. 根据权利要求13所述的装置，其中，所述两异源传感器图像分别为红外图像和可见光图像。
PCT/CN2016/081599 2015-05-12 2016-05-10 图像处理方法及装置 WO2016180325A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510240940.2 2015-05-12
CN201510240940.2A CN104835165B (zh) 2015-05-12 2015-05-12 图像处理方法及装置

Publications (1)

Publication Number Publication Date
WO2016180325A1 true WO2016180325A1 (zh) 2016-11-17

Family

ID=53813030

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/081599 WO2016180325A1 (zh) 2015-05-12 2016-05-10 图像处理方法及装置

Country Status (2)

Country Link
CN (1) CN104835165B (zh)
WO (1) WO2016180325A1 (zh)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108876841A (zh) * 2017-07-25 2018-11-23 成都通甲优博科技有限责任公司 一种视差图视差求精中插值的方法及***
CN110533712A (zh) * 2019-08-26 2019-12-03 北京工业大学 一种基于卷积神经网络的双目立体匹配方法
CN110866535A (zh) * 2019-09-25 2020-03-06 北京迈格威科技有限公司 视差图的获取方法、装置、计算机设备和存储介质
CN111008660A (zh) * 2019-12-03 2020-04-14 北京京东乾石科技有限公司 语义地图的生成方法、装置、***、存储介质及电子设备
CN111354032A (zh) * 2018-12-24 2020-06-30 杭州海康威视数字技术股份有限公司 一种生成视差图的方法及装置
CN111462195A (zh) * 2020-04-09 2020-07-28 武汉大学 基于主线约束的非规则角度方向代价聚合路径确定方法
CN111612898A (zh) * 2020-06-18 2020-09-01 腾讯科技（深圳）有限公司 图像处理方法、装置、存储介质及电子设备
CN113052862A (zh) * 2021-04-12 2021-06-29 北京机械设备研究所 基于多级优化的室外场景下立体匹配方法、装置、设备
CN113534176A (zh) * 2021-06-22 2021-10-22 武汉工程大学 基于图正则化的光场高精度三维测距方法
CN117408890A (zh) * 2023-12-14 2024-01-16 武汉泽塔云科技股份有限公司 一种视频图像传输质量增强方法及***

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104835165B (zh) * 2015-05-12 2017-05-24 努比亚技术有限公司 图像处理方法及装置
CN104902260B (zh) * 2015-06-30 2018-04-27 Tcl集团股份有限公司 一种图像视差的获取方法及***
CN105354838B (zh) * 2015-10-20 2018-04-10 努比亚技术有限公司 图像中弱纹理区域的深度信息获取方法及终端
CN108205658A (zh) * 2017-11-30 2018-06-26 中原智慧城市设计研究院有限公司 基于单双目视觉融合的障碍物检测预警***
WO2019127192A1 (zh) * 2017-12-28 2019-07-04 深圳市大疆创新科技有限公司 图像处理方法和设备
CN110033426B (zh) * 2018-01-12 2021-07-09 杭州海康威视数字技术股份有限公司 一种用于对视差估计图像进行处理的装置
CN109840894B (zh) * 2019-01-30 2021-02-09 湖北亿咖通科技有限公司 视差图精修方法、装置及存储介质
CN110060283B (zh) * 2019-04-17 2020-10-30 武汉大学 一种多测度半全局密集匹配方法
CN110136188B (zh) * 2019-05-16 2023-01-17 东莞职业技术学院 一种基于特征的立体图像匹配算法
CN112215876B (zh) * 2020-10-22 2022-10-04 烟台艾睿光电科技有限公司 一种双光谱图像配准融合方法、装置、设备及存储介质
CN113724308B (zh) * 2021-11-02 2022-03-15 南京理工大学 基于光度与对比度互注意力的跨波段立体匹配算法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100027911A1 (en) * 2008-08-04 2010-02-04 Siemens Corporate Research, Inc. Multilevel thresholding for mutual information based registration and image registration using a gpu
CN103679714A (zh) * 2013-12-04 2014-03-26 中国资源卫星应用中心 一种基于梯度互相关的光学和sar图像自动配准方法
CN104021548A (zh) * 2014-05-16 2014-09-03 中国科学院西安光学精密机械研究所 一种获取场景4d信息的方法
CN104021556A (zh) * 2014-06-13 2014-09-03 西南交通大学 一种基于几何结构相似性的异源遥感影像配准方法
CN104835165A (zh) * 2015-05-12 2015-08-12 努比亚技术有限公司 图像处理方法及装置

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006285952A (ja) * 2005-03-11 2006-10-19 Sony Corp 画像処理方法、画像処理装置、プログラムおよび記録媒体

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100027911A1 (en) * 2008-08-04 2010-02-04 Siemens Corporate Research, Inc. Multilevel thresholding for mutual information based registration and image registration using a gpu
CN103679714A (zh) * 2013-12-04 2014-03-26 中国资源卫星应用中心 一种基于梯度互相关的光学和sar图像自动配准方法
CN104021548A (zh) * 2014-05-16 2014-09-03 中国科学院西安光学精密机械研究所 一种获取场景4d信息的方法
CN104021556A (zh) * 2014-06-13 2014-09-03 西南交通大学 一种基于几何结构相似性的异源遥感影像配准方法
CN104835165A (zh) * 2015-05-12 2015-08-12 努比亚技术有限公司 图像处理方法及装置

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
LI, LONGXUN;: "The Registration and Fusion of Multi-Sensor Images Based on Mutual Information", ELECTRONIC TECHNOLOGY & INFORMATION SCIENCE , CHINA MASTER'S THESES FULL-TEXT DATABASE, 15 January 2013 (2013-01-15), pages 1138 - 1999 *
LIU, YUANJUN;: "Research of Object Recognition and Location of Fire Detection Based on Dual-Band Images", ELECTRONIC TECHNOLOGY & INFORMATION SCIENCE , CHINA MASTER'S THESES FULL-TEXT DATABASE, 15 December 2011 (2011-12-15), pages 1138 - 1319 *
ZOU, JIBIAO: "Research on Local Matching Algorithms of Stereo Vision", ELECTRONIC TECHNOLOGY & INFORMATION SCIENCE , CHINA MASTER'S THESES FULL-TEXT DATABASE, 15 October 2014 (2014-10-15), pages 1138 - 1074 *

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108876841A (zh) * 2017-07-25 2018-11-23 成都通甲优博科技有限责任公司 一种视差图视差求精中插值的方法及***
CN108876841B (zh) * 2017-07-25 2023-04-28 成都通甲优博科技有限责任公司 一种视差图视差求精中插值的方法及***
CN111354032A (zh) * 2018-12-24 2020-06-30 杭州海康威视数字技术股份有限公司 一种生成视差图的方法及装置
CN111354032B (zh) * 2018-12-24 2023-10-20 杭州海康威视数字技术股份有限公司 一种生成视差图的方法及装置
CN110533712A (zh) * 2019-08-26 2019-12-03 北京工业大学 一种基于卷积神经网络的双目立体匹配方法
CN110533712B (zh) * 2019-08-26 2022-11-04 北京工业大学 一种基于卷积神经网络的双目立体匹配方法
CN110866535A (zh) * 2019-09-25 2020-03-06 北京迈格威科技有限公司 视差图的获取方法、装置、计算机设备和存储介质
CN110866535B (zh) * 2019-09-25 2022-07-29 北京迈格威科技有限公司 视差图的获取方法、装置、计算机设备和存储介质
CN111008660A (zh) * 2019-12-03 2020-04-14 北京京东乾石科技有限公司 语义地图的生成方法、装置、***、存储介质及电子设备
CN111462195B (zh) * 2020-04-09 2022-06-07 武汉大学 基于主线约束的非规则角度方向代价聚合路径确定方法
CN111462195A (zh) * 2020-04-09 2020-07-28 武汉大学 基于主线约束的非规则角度方向代价聚合路径确定方法
CN111612898A (zh) * 2020-06-18 2020-09-01 腾讯科技（深圳）有限公司 图像处理方法、装置、存储介质及电子设备
CN113052862A (zh) * 2021-04-12 2021-06-29 北京机械设备研究所 基于多级优化的室外场景下立体匹配方法、装置、设备
CN113534176A (zh) * 2021-06-22 2021-10-22 武汉工程大学 基于图正则化的光场高精度三维测距方法
CN117408890A (zh) * 2023-12-14 2024-01-16 武汉泽塔云科技股份有限公司 一种视频图像传输质量增强方法及***
CN117408890B (zh) * 2023-12-14 2024-03-08 武汉泽塔云科技股份有限公司 一种视频图像传输质量增强方法及***

Also Published As

Publication number Publication date
CN104835165A (zh) 2015-08-12
CN104835165B (zh) 2017-05-24

Similar Documents

Publication Publication Date Title
WO2016180325A1 (zh) 图像处理方法及装置
WO2018019124A1 (zh) 一种图像处理方法及电子设备、存储介质
WO2017067526A1 (zh) 图像增强方法及移动终端
WO2017045650A1 (zh) 一种图片处理方法及终端
CN106530241B (zh) 一种图像虚化处理方法和装置
WO2017020836A1 (zh) 一种虚化处理深度图像的装置和方法
WO2017067390A1 (zh) 图像中弱纹理区域的深度信息获取方法及终端
US9113305B2 (en) Method and apparatus for estimating location of user equipment in wireless network
WO2017016511A1 (zh) 一种图像处理方法及装置、终端
US9083968B2 (en) Mobile terminal and image display method thereof
WO2017071456A1 (zh) 一种终端的处理方法、终端及存储介质
WO2018050014A1 (zh) 对焦方法及拍照设备、存储介质
WO2018019128A1 (zh) 一种夜景图像的处理方法和移动终端
CN106713716B (zh) 一种双摄像头的拍摄控制方法和装置
WO2017071476A1 (zh) 一种图像合成方法和装置、存储介质
WO2017041714A1 (zh) 一种获取rgb数据的方法和装置
WO2017071542A1 (zh) 图像处理方法及装置
WO2017071475A1 (zh) 一种图像处理方法及终端、存储介质
CN106534693B (zh) 一种照片处理方法、装置及终端
WO2017067523A1 (zh) 图像处理方法、装置及移动终端
WO2018076938A1 (zh) 图像处理装置及方法和计算机存储介质
WO2017088618A1 (zh) 图片合成方法及装置
CN105430258B (zh) 一种自拍合影的方法和装置
CN106482641B (zh) 一种尺寸测量装置和方法
CN105959551A (zh) 一种拍摄装置及方法、移动终端

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16792175

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 17.04.2018)

122 Ep: pct application non-entry in european phase

Ref document number: 16792175

Country of ref document: EP

Kind code of ref document: A1