CN111131718A - Multi-exposure image fusion method and system with LED flicker compensation function - Google Patents
Multi-exposure image fusion method and system with LED flicker compensation function Download PDFInfo
- Publication number
- CN111131718A CN111131718A CN201910638889.9A CN201910638889A CN111131718A CN 111131718 A CN111131718 A CN 111131718A CN 201910638889 A CN201910638889 A CN 201910638889A CN 111131718 A CN111131718 A CN 111131718A
- Authority
- CN
- China
- Prior art keywords
- fusion
- image
- led flicker
- unit
- exposure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
- Picture Signal Circuits (AREA)
Abstract
The invention discloses a multi-exposure image fusion method and a system with an LED flicker compensation function, wherein the system comprises the following steps: the image sensor comprises an image sensor unit, an interface unit, a line buffer unit, a fusion unit and an output unit; the method and the system do not need an LFM image sensor, use a conventional image sensor, compensate LED flicker by fusion processing, improve image real-time performance, reduce hardware cost, only need a small amount of line cache in the processing process, directly process Bayer format data, reduce system delay, reduce storage overhead, and can be conveniently expanded to various programmable devices and special integrated circuits.
Description
Technical Field
The invention discloses a multi-exposure image fusion method and a multi-exposure image fusion system with an LED flicker compensation function, and relates to the technical field of image processing.
Background
Because of the characteristics of energy conservation, environmental protection, safety, low power consumption, high brightness and the like, light emitting diodes (hereinafter referred to as LEDs) are widely applied to the fields of various daily illuminations, traffic indicator lamps, automobile headlamps, road illuminations and the like. The LED dimming uses a pulse width modulation mode, and the LED light emission in a daily scene has periodicity, namely, flicker, with the period beingToThe seconds are different, and no uniform standard is formed in all parts of the world at present. The human eye is hardly affected by the flickering of the LED due to the characteristic of persistence of vision.
In human daily life scenes, for example, when a high-brightness automobile LED headlamp appears at extremely dark nights, the brightness change range of different areas of the same scene exceeds 8 orders of magnitude, namely the dynamic range is larger than 160 dB. Human eyes can adapt to brightness change of more than 6 orders of magnitude due to long-term evolution, namely, the dynamic range is higher than 120 dB. The single exposure of the image sensor can only provide a dynamic range of about 70dB, and the requirement of reflecting and recording a high-dynamic scene in the real world can not be met far, so that a dynamic range of 100-144 dB can be provided by a multi-exposure method, a dark area of light in a scene is recorded by long exposure, a strong area of light in the scene is recorded by short exposure, and brightness information in the real world scene is reflected and recorded better.
In the prior art, in order to provide a dynamic range not inferior to that of human eyes, a conventional image sensor performs multiple exposures in a line-alternating manner at the same time, and the longest exposure time can be reachedSecond, the shortest exposure time can beThe short exposure time is far shorter than the LED light-emitting period, so that the flickering and the brightness rolling of the LED illumination area can be seen in the fused image, on one hand, a user feels uncomfortable, on the other hand, the real states of a traffic signal lamp and other LED illumination areas cannot be correctly reflected and recorded, and the application of automobile safety, automatic driving and the like is threatened fatally. To improve the LED flicker problem, the shorter exposure time is generally broken up and distributed uniformly over a longer period by improving the exposure timing of the image sensor, so as to reduce the LED flicker effect, which is called LED flicker reduction (LFM).
In the prior art, the line alternation and multiple exposure fusion method has the defects that the line alternation and multiple exposure fusion method is interfered by LED flicker with uncertain periods, so that visual discomfort of a user, identification errors of traffic lights and the like are caused.
In the prior art, the LFM has the disadvantages of prolonging the operation time of each exposure, increasing the time delay between multiple exposures, and finally causing the reduction of image real-time performance and dynamic range, etc.
In the prior art, the LFM has another disadvantage that the operation time of a single exposure is prolonged, and in order to increase the dynamic range by using multiple exposures, the image data which must be cached by a processing system is greatly increased, which causes the increase of the system overhead and the cost.
Disclosure of Invention
The invention aims to solve the technical problem of providing a multi-exposure image fusion method and system with LED flicker compensation function, which has low complexity, small operand, low cost and is suitable for a conventional image sensor.
The technical scheme adopted by the invention for solving the technical problems is as follows:
a multi-exposure image fusion method with an LED flicker compensation function comprises the following steps:
s2: from input raw dataAndextracting luminance information therefromAnd(ii) a The input image data can be from a line cache unit and also can be from the output of a previous-level fusion unit;
S4: according to the brightness informationAndcalculating a reference value reflecting the severity of LED flicker occurring at each location;
S6: according to the fusion weight coefficientAnd LED flicker compensation factorCalculating a fusion value of each pixelAnd obtaining a final output image.
Preferably, in S2, the method specifically includes the following substeps:
s11: denote a 3 × 3 convolution kernel by h;
preferably, the division operation in the convolution operation described in S13 realizes fast calculation using a shift operation, that is
Preferably, in S3, the method specifically includes the following sub-steps:
s33: constructing a plane rectangular coordinate system, wherein the horizontal axis is brightness, and the vertical axis is a fusion weight coefficient;
S36: using said abscissa、、、、And ordinate、、Constructing a piecewise polyline equation for saidCoefficient mapping is carried out, i.e.
obtaining current effective exposure time length from system configurationAndratio ofCalculating a reference value,
Preferably, in S5, the method specifically includes the following sub-steps:
S53: calculated by the formula, i.e.
Preferably, in S6, the method specifically includes the following sub-steps:
A multi-exposure image fusion system with LED flicker compensation, which is described below by taking only 4 exposures as an example, the system comprising: an image sensor unit, an interface unit, line buffer units (1 to 4), a fusion unit (1 to 3), and an output unit;
an image sensor unit for generating raw data of 4 different exposure time lengths in a line-alternating manner, the exposure time lengths being respectively used、、、Represents;
the interface unit is used for connecting the image sensor unit, separating the line alternation image data generated by the image sensor unit and inputting the line alternation image data into the system;
line buffer units (1 to 4) for buffering and aligning 4 sets of exposure image data generated by the image sensor, respectively、、、Represents;
fusion unit 1 for detection and compensationLED flicker in (1), fusionAndimage data, output fusion image;
A fusion unit 2 for detecting and compensatingThe LED in (1) flickers and is fusedAndimage data, output fusion image;
A fusion unit 3 for detecting and compensatingThe LED in (1) flickers and is fusedAndimage data, output fusion image;
An output unit for fusing the final image according to a certain interface formatAnd outputting to an external or next-stage processing unit.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The present invention will be described in detail below with reference to the accompanying drawings so that the above advantages of the present invention will be more apparent. Wherein the content of the first and second substances,
FIG. 1 is a block diagram of a pipeline of a multiple exposure image fusion system with LED flicker compensation according to the present invention.
FIG. 2 is a pipeline diagram of the multiple exposure image fusion method with LED flicker compensation according to the present invention.
Fig. 3 is a schematic diagram illustrating the principle of extracting luminance information by performing a convolution operation of 3 × 3 on a Bayer format according to the present invention.
FIG. 4 is a schematic diagram illustrating the principle of piecewise polygonal line mapping in the process of calculating the fusion coefficient by the multi-exposure image fusion method with the LED flicker compensation function according to the present invention.
Detailed Description
The following detailed description of the embodiments of the present invention will be provided with reference to the drawings and examples, so that how to apply the technical means to solve the technical problems and achieve the technical effects can be fully understood and implemented. It should be noted that, as long as there is no conflict, the embodiments and the features of the embodiments of the present invention may be combined with each other, and the technical solutions formed are within the scope of the present invention.
As shown in fig. 1-4, wherein,
a multiple exposure image fusion system with LED flicker compensation, which is described by taking a 4-exposure image fusion system as an example, the system comprising: the system comprises an image sensor unit, a pipeline structure consisting of an interface unit, line buffer units (1 to 4), fusion units (1 to 3) and an output unit, wherein the image sensor unit generates 4 kinds of original image data with different exposure durations according to the flow sequence of the data, the original image data passes through the interface unit in a line-alternating mode, is separated and flows into the line buffer units (1 to 4), further flows into the fusion units (1 to 3), and finally flows out of the system through the output unit; all or part (at least including fusion units 1 to 3) of the units except the image sensor unit are realized in the form of a programmable device, wherein one expression is to run software on a certain general processing chip (CPU or DSP), the other expression is to realize a special image processing pipeline on a Field Programmable Gate Array (FPGA), and the other expression is to realize the special image processing pipeline by using an application specific integrated circuit chip (ASIC) or a system on a chip (SoC); the interface unit is used for realizing a communication protocol with the image sensor unit, establishing a data transmission channel between the image sensor and the system, and separating the line alternation image data generated by the data transmission channel into 4 groups of exposure image data to be input into the system; a line buffer unit (1 to 4) for buffering and aligning 4 sets of exposure image data generated by the image sensor; and the fusion units (1 to 3) are used for detecting and compensating LED flicker in each group of short exposure image data in a cascading mode, fusing each group of image data to obtain a final fusion image, and outputting the final fusion image to an external or next-stage processing unit through the output unit according to a certain interface format.
The multi-exposure image fusion method with the LED flicker compensation function comprises the following steps:
taking the exposure duration from the input as a unit of lineImage data of (1) byMeans that the exposure time length is obtained asImage data of (1) byIs shown in which;
The input image data can be from a line cache unit and also can be from the output of a previous-level fusion unit;
extracting brightness information from the input original data according to a certain algorithm;
preferably, a calculation method for extracting luminance information from the input raw data is as follows:
denote a 3 × 3 convolution kernel by h;
preferably, the first and second electrodes are formed of a metal,andeach being a plane of pixels arranged periodically in a Bayer (2x2) format;
preferably, the division operation in the convolution operation uses a shift operation to achieve fast calculation, i.e.
Further, using the luminance informationAnd a set of parameters obtained from the system configuration, calculatingThe fusion weight coefficient of each pixel in the image;
preferably, one method for calculating the fusion weight coefficient is as follows:
constructing a plane rectangular coordinate system, wherein the horizontal axis is brightness, and the vertical axis is a fusion weight coefficient;
Further, using said abscissa、、、、And ordinate、、Constructing a piecewise polyline equation for saidCoefficient mapping is carried out, i.e.
Further, using the luminance informationAndand calculating a reference value reflecting the severity of the LED flicker at the current position:
preferably, one method of calculating the reference value is as follows:
Further, a group of parameters are obtained from the system configuration, and the LED flicker compensation coefficient of the current position is calculated as follows:
preferably, one method of calculating the flicker compensation coefficient is as follows:
Calculated by the formula, i.e.
Further, calculating a fusion value of the pixel at the current position according to the fusion weight coefficient, the LED flicker compensation coefficient and a group of parameters obtained from system configuration;
preferably, one method of calculating the fusion value is as follows:
In the invention, the image generation algorithm with high dynamic range can be conveniently realized in various integrated circuit forms, including ASIC, FPGA and the like, the whole system has good expansibility, is convenient for integrating other image algorithms, and can ensure the real-time operation of processing.
In the embodiment of the present invention, the embodiment based on an integrated circuit is implemented by taking an FPGA as an example, and those skilled in the art can easily extend the embodiment to other integrated circuits to implement the embodiment.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A multi-exposure image fusion method with an LED flicker compensation function is characterized by comprising the following steps:
S4: according to the brightness informationAndcalculating a reference value reflecting the severity of LED flicker occurring at each location;
2. The multi-exposure image fusion method with the LED flicker compensation function according to claim 1, wherein in S2, the method specifically comprises the following sub-steps:
s11: denote a 3 × 3 convolution kernel by h;
4. The multi-exposure image fusion method with the LED flicker compensation function according to claim 1, wherein in S3, the method specifically comprises the following sub-steps:
s33: constructing a plane rectangular coordinate system, wherein the horizontal axis is brightness, and the vertical axis is a fusion weight coefficient;
S36: using said abscissa、、、、And ordinate、、Constructing a piecewise polyline equation for saidCoefficient mapping is carried out, i.e.
7. The multi-exposure image fusion method with the LED flicker compensation function according to claim 1, wherein in S6, the method specifically comprises the following sub-steps:
8. A multi-exposure image fusion system with LED flicker compensation, comprising: the device comprises an image sensor unit, an interface unit, a line cache unit, a fusion unit and an output unit;
the image sensor unit is used for generating a plurality of groups of Bayer raw data with different exposure time lengths in a line-alternating mode;
the interface unit is connected with the image sensor unit, and is used for separating data generated by the image sensor and inputting the data into the system;
the line buffer unit is used for buffering and aligning a plurality of groups of exposure image data generated by the image sensor;
the fusion unit is used for detecting and compensating LED flicker in the shorter exposure image data, fusing each group of image data and outputting a fused image;
and the output unit is used for outputting the final fusion image to an external or next-stage processing unit.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910638889.9A CN111131718B (en) | 2019-07-16 | 2019-07-16 | Multi-exposure image fusion method and system with LED flicker compensation function |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910638889.9A CN111131718B (en) | 2019-07-16 | 2019-07-16 | Multi-exposure image fusion method and system with LED flicker compensation function |
Publications (2)
Publication Number | Publication Date |
---|---|
CN111131718A true CN111131718A (en) | 2020-05-08 |
CN111131718B CN111131718B (en) | 2021-05-14 |
Family
ID=70495189
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910638889.9A Active CN111131718B (en) | 2019-07-16 | 2019-07-16 | Multi-exposure image fusion method and system with LED flicker compensation function |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111131718B (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113038026A (en) * | 2021-03-01 | 2021-06-25 | 维沃移动通信有限公司 | Image processing method and electronic device |
CN113242389A (en) * | 2021-07-12 | 2021-08-10 | 深圳市艾为智能有限公司 | Multi-frame dynamic range extension method and system for RCCB (Rich communication Circuit Board) image sensor |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN1250570A (en) * | 1997-11-12 | 2000-04-12 | 皇家菲利浦电子有限公司 | Method and apparatus for scaling and reducing flicker with dynamic coefficient weighting |
JP2011160090A (en) * | 2010-01-29 | 2011-08-18 | Sony Corp | Image processing device and signal processing method, and program |
CN103379288A (en) * | 2012-04-12 | 2013-10-30 | 索尼公司 | Image processing device, image processing method, and program |
US20140153839A1 (en) * | 2012-12-04 | 2014-06-05 | Samsung Techwin Co., Ltd. | Apparatus and method for calculating flicker-evaluation value |
CN103975578A (en) * | 2011-12-08 | 2014-08-06 | 索尼公司 | Image processing device, image processing method, and program |
US20140267815A1 (en) * | 2013-03-15 | 2014-09-18 | Canon Kabushiki Kaisha | Image capturing apparatus and method of controlling the same |
US20150195441A1 (en) * | 2012-09-20 | 2015-07-09 | Huawei Technologies Co., Ltd. | Image Processing Method and Apparatus |
CN104780358A (en) * | 2014-01-03 | 2015-07-15 | 联发科技(新加坡)私人有限公司 | Method for flicker detection and associated circuit |
CN105659583A (en) * | 2013-12-04 | 2016-06-08 | 索尼公司 | Image processing device, image processing method, electronic apparatus, and program |
CN106303274A (en) * | 2016-08-01 | 2017-01-04 | 凌云光技术集团有限责任公司 | A kind of high dynamic-range image synthesis method and device |
CN108353130A (en) * | 2015-11-24 | 2018-07-31 | 索尼公司 | Image processor, image processing method and program |
-
2019
- 2019-07-16 CN CN201910638889.9A patent/CN111131718B/en active Active
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN1250570A (en) * | 1997-11-12 | 2000-04-12 | 皇家菲利浦电子有限公司 | Method and apparatus for scaling and reducing flicker with dynamic coefficient weighting |
JP2011160090A (en) * | 2010-01-29 | 2011-08-18 | Sony Corp | Image processing device and signal processing method, and program |
CN103975578A (en) * | 2011-12-08 | 2014-08-06 | 索尼公司 | Image processing device, image processing method, and program |
CN103379288A (en) * | 2012-04-12 | 2013-10-30 | 索尼公司 | Image processing device, image processing method, and program |
US20150195441A1 (en) * | 2012-09-20 | 2015-07-09 | Huawei Technologies Co., Ltd. | Image Processing Method and Apparatus |
US20140153839A1 (en) * | 2012-12-04 | 2014-06-05 | Samsung Techwin Co., Ltd. | Apparatus and method for calculating flicker-evaluation value |
US20140267815A1 (en) * | 2013-03-15 | 2014-09-18 | Canon Kabushiki Kaisha | Image capturing apparatus and method of controlling the same |
CN105659583A (en) * | 2013-12-04 | 2016-06-08 | 索尼公司 | Image processing device, image processing method, electronic apparatus, and program |
CN104780358A (en) * | 2014-01-03 | 2015-07-15 | 联发科技(新加坡)私人有限公司 | Method for flicker detection and associated circuit |
CN108353130A (en) * | 2015-11-24 | 2018-07-31 | 索尼公司 | Image processor, image processing method and program |
CN106303274A (en) * | 2016-08-01 | 2017-01-04 | 凌云光技术集团有限责任公司 | A kind of high dynamic-range image synthesis method and device |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113038026A (en) * | 2021-03-01 | 2021-06-25 | 维沃移动通信有限公司 | Image processing method and electronic device |
CN113242389A (en) * | 2021-07-12 | 2021-08-10 | 深圳市艾为智能有限公司 | Multi-frame dynamic range extension method and system for RCCB (Rich communication Circuit Board) image sensor |
Also Published As
Publication number | Publication date |
---|---|
CN111131718B (en) | 2021-05-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN103778887B (en) | The brightness correcting method of LED display unit and device | |
CN101790270B (en) | LED backlight brightness dynamic control method and system adopting same | |
CN111131718B (en) | Multi-exposure image fusion method and system with LED flicker compensation function | |
CN104702850B (en) | A kind of complementary metal oxide semiconductor video camera and its light compensation method | |
CN106470336A (en) | Camera color temperature compensating system and intelligent terminal | |
CN110164366A (en) | A kind of information processing method, electronic equipment | |
CN110728944B (en) | Display device and display method | |
JP2010250320A (en) | Method for correcting pixel data, and display apparatus for performing the method | |
JP2007193295A (en) | Led driving integrated circuit device capable of pulse width modulation | |
CN101105912A (en) | Display quality automatic regulation device and method based on environment induction | |
CN104639920A (en) | Wide dynamic fusion method based on single-frame double-pulse exposure mode | |
CN104575405A (en) | Method for adjusting backlight brightness of display device, display device | |
CN106782385A (en) | A kind of light-dimming method and its device for wearing display device | |
CN103489411B (en) | The color of display screen maintains method and system | |
JP2021168465A5 (en) | ||
CN117274107B (en) | End-to-end color and detail enhancement method, device and equipment under low-illumination scene | |
WO2018225516A1 (en) | Image processing device | |
US8670660B2 (en) | Image display device, brightness control method and brightness control program | |
TW202133601A (en) | A low-light imaging system | |
CN202487127U (en) | Backlight source and display device | |
JP4749142B2 (en) | Visual status measurement device | |
KR100961675B1 (en) | Traffic light lamp using light emitting diode | |
JP2012023459A (en) | On-vehicle monitor camera and imaging method thereof | |
US20230237948A1 (en) | Display screen adjustment method and apparatus, and device | |
TW202013332A (en) | Local dimming system adaptable to a backlight of a display |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |