CN112396670B - Image reconstruction method for novel binary image sensor - Google Patents
Image reconstruction method for novel binary image sensor Download PDFInfo
- Publication number
- CN112396670B CN112396670B CN201910752407.2A CN201910752407A CN112396670B CN 112396670 B CN112396670 B CN 112396670B CN 201910752407 A CN201910752407 A CN 201910752407A CN 112396670 B CN112396670 B CN 112396670B
- Authority
- CN
- China
- Prior art keywords
- image reconstruction
- binary data
- binary
- image
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Transforming Light Signals Into Electric Signals (AREA)
- Image Processing (AREA)
Abstract
The image reconstruction method for the novel binary image sensor comprises the steps of collecting optical signals, generating electric signals, generating binary data streams, collecting the binary data streams, configuring image reconstruction parameters, selecting an image reconstruction mode, processing the binary data streams, finishing image reconstruction and outputting reconstructed images; the image reconstruction method and the image reconstruction system are combined with the characteristics of a novel binary image sensor such as high resolution, low noise and high frame frequency, the operations of generating, collecting, processing and reconstructing binary data streams are completed under the appointed image reconstruction parameters and image reconstruction modes, and the reconstructed image is finally output.
Description
Technical Field
The invention relates to the field of image sensors and digital image processing, in particular to an image reconstruction method for a novel binary image sensor.
Background
An image sensor is a semiconductor light-sensing element that converts an optical signal into an electrical signal. Currently, the mainstream image sensor is a complementary metal-Oxide-Semiconductor (CMOS) image sensor. The CMOS image sensor has the inherent advantages of being compatible with a standard CMOS process, high in integration level, low in power consumption, high in speed, low in cost, high in irradiation resistance and the like; meanwhile, with the continuous deepening of related scientific research, the performance indexes such as quantum efficiency, dark current, sensitivity, signal-to-noise ratio, dynamic range, linearity and the like are continuously improved. For the above two reasons, the CMOS image sensor has gradually replaced the conventional Charge Coupled Device (CCD) image sensor, and has gained dominance in various fields of consumer electronics, industrial electronics, automotive electronics, scientific imaging, and the like.
A binary image sensor is an image sensor that converts an optical signal into a binary data stream and outputs the binary data stream, and output data thereof includes both 0 and 1. Data 0 represents that the pixel unit of the image sensor corresponding to the data does not receive enough photons, and data 1 represents that the pixel unit corresponding to the data receives enough photons. The threshold for converting data 0 to data 1 may be generally configurable. For example, the threshold is configured to be 10 photons, and when the pixel unit receives 10 photons or more, data 1 is output, otherwise, data 0 is output.
In recent years, image sensors have gradually exhibited several new characteristics: with the development of semiconductor technology, the pixel size is gradually reduced to below 1um to reach submicron level; with the development of low-noise pixel-related research, pixel readout noise can be reduced to below 0.5 electron to reach a sub-electron level; with the development of high-speed readout circuits, in combination with technologies such as digital pixels and integration of storage nodes in the pixels, the theoretical frame rate of the image sensor can exceed 1Mfps.
In combination with the above new characteristics, the novel binary image sensor has the following characteristics: first, high resolution. I.e. the pixel size is small enough to enable integration of a larger number of pixel cells, typically 10MP and above, on a smaller chip size. Second, low noise. That is, the image sensor has a capability of distinguishing one photon to a certain extent, and a single photon counting capability, with a typical value of 0.5 e-or less, when the readout noise reaches a sub-electronic level. The output data 0 represents no photon incidence, and the output data 1 represents at least 1 photon incidence. Third, high frame rate. I.e. a sufficient number of read frames per second, typically 1kfps and above.
Aiming at the novel binary image sensor and the characteristics thereof, an image reconstruction method and a system matched with the novel binary image sensor are required to be provided, the image reconstruction method and the system are required to complete the operations of binary data stream generation, acquisition, processing, reconstruction and the like, and finally output the reconstructed image.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides an image reconstruction method for a novel binary image sensor. The image reconstruction method and the image reconstruction system are combined with the characteristics of a novel binary image sensor such as high resolution, low noise and high frame frequency, the operations of generating, collecting, processing and reconstructing binary data streams are completed under the appointed image reconstruction parameters and image reconstruction modes, and the reconstructed image is finally output.
The image reconstruction method for the novel binary image sensor has the flow shown in fig. 1, and the image reconstruction system includes the following flows: (1) collecting optical signals: the signal electrons generated by the incident photons are collected and stored by the pixels and their arrays. (2) generating an electrical signal: the signal electrons are converted into signal voltage values through charge transfer, correlated double sampling and amplification operations. (3) generating a binary data stream: the signal voltage value is quantized to a digital value of 0 or a digital value of 1 by a binary quantizer, such as a 1-bit analog-to-digital converter (1-bit ADC). Where a digital value of 0 represents no photon incident and a digital value of 1 represents at least one photon incident. In each frame, the individual pixel output results form a binary data matrix. And (4) collecting a binary data stream: and collecting and storing each frame of binary data matrix generated by the image sensor, namely collecting binary data stream. And (5) configuring image reconstruction parameters: and configuring image reconstruction parameters for subsequent image reconstruction operation. (6) selecting an image reconstruction mode: an image reconstruction mode is selected for a subsequent image reconstruction operation. (7) processing the binary data stream: the binary data stream is data processed under specified image reconstruction parameters and modes. And (8) completing image reconstruction and outputting the reconstructed image.
The image reconstruction method for the novel binary image sensor has a novel binary image sensor structure as shown in fig. 2. The novel binary image sensor consists of V rows and H columns of V x H pixel units (each square unit in FIG. 2); because of adopting the process technology with lower characteristic dimension, the typical value of the dimension of each pixel unit is 1um × 1um and below, and the typical value of the total number of the pixel units is 10MP and above; controlling V x H pixel units through a row addressing circuit, a column addressing circuit and a related driving circuit; quantizing the output result of each column of pixels through a column reading circuit, and outputting each column of data including data 0 or data 1; and carrying out parallel-serial processing on each column of data through a basic data processing circuit to form a binary data stream.
Fig. 3 shows a schematic diagram of a binary data flow of the image reconstruction method for a novel binary image sensor. The data stream comprises F frames of binary data matrixes, each frame of binary data matrix comprises V rows and H columns, and V x H data units in total correspond to V x H pixel units. As shown in fig. 3, each white square represents at least one photon incident, outputting data 1, and each black square represents no photon incident, outputting data 0. The binary data stream contains V × H × F bits of data in total.
Fig. 4 shows a schematic diagram of image reconstruction parameters of the image reconstruction method for the novel binary image sensor. Before data processing and image reconstruction, configuring image reconstruction parameters as (k V, k H and k F); extracting k F frames in an F-frame binary data matrix based on the original binary data stream (V, H, F); extracting k V rows k H columns binary data submatrix from the binary data matrixes of the V rows and the H columns; and finally, extracting the binary data block with the specified size. Under the image reconstruction parameters (k V, k H, k F), the binary data submatrix of each frame has the maximum value of k V × k H and the minimum value of 0; the binary data block has a maximum value of k V × k H × k F and a minimum value of 0. For example, as shown in fig. 4, the image reconstruction parameters (k V, k F, k F) = (3,3,3), and the maximum value of the binary data block extracted under the reconstruction parameters is 3 × 3=27; the sum of the values of the binary data blocks with the specified size is 4+5+4=13, and if the binary data blocks are subjected to subsequent data processing and image reconstruction, the binary data blocks correspond to a pixel point with a gray value of 13/27 + 255=123 in the reconstructed image.
Fig. 5 shows a schematic diagram of an image reconstruction mode of the image reconstruction method for the novel binary image sensor. Before data processing and image reconstruction, a corresponding image reconstruction mode needs to be selected. As shown in fig. 5, based on the original binary data stream (V, H, F), k V rows, k H columns binary data submatrix is extracted and referred to as an image reconstruction window. As shown in fig. 5 (a) and (b), the first type of image reconstruction mode is called a non-data-overlapped image reconstruction mode, and is characterized as follows: when horizontal scanning and vertical scanning are carried out, data do not overlap among the image reconstruction windows. As shown in fig. 5 (c) and (d), the second type of image reconstruction mode is called a data-overlapped image reconstruction mode, and is characterized as follows: when horizontal scanning and vertical scanning are carried out, data overlap exists between the image reconstruction windows, and the overlap range can be adjusted according to actual conditions.
The image reconstruction method for the novel binary image sensor is combined with the novel binary image sensor, can configure image reconstruction parameters and select an image reconstruction mode, can completely complete a series of operations such as binary data stream generation, acquisition, processing, reconstruction and the like, and finally outputs a reconstructed image.
Drawings
FIG. 1 is a flow chart of an image reconstruction system of the present invention;
FIG. 2 is a diagram of the novel binary image sensor architecture of the present invention;
FIG. 3 is a schematic representation of the binary data stream of the present invention, depicting a particular form of the raw binary data stream (V, H, F) generated by the image sensor;
FIG. 4 is a schematic diagram of the image reconstruction parameters of the present invention, depicting the specific significance of three image reconstruction parameters;
FIG. 5 is a schematic diagram of the image reconstruction mode of the present invention, depicting the basic principles of two types of image reconstruction modes;
fig. 6 shows an embodiment of simulation based on the image reconstruction method of the present invention.
Detailed Description
Fig. 6 shows an embodiment of simulation based on the image reconstruction method of the present invention. The following describes a specific embodiment of the present invention with reference to this example.
This embodiment is based on the original binary data stream (V, H, F) = (4096,4096,16), i.e., the original data stream of this embodiment contains 16 4096 rows, 4096 columns of binary data matrices, i.e., 268435456 data units, totaling 268435456 bits of data. In practical application, the original binary data stream related to the present invention is output by the novel binary image sensor, i.e. four processes of collecting light signals, generating binary data streams and collecting binary data streams are performed. In this case, a 4096 row, 4096 column binary data matrix corresponds to a 4096 row, 4096 column pixel array, and 16 binary data matrices correspond to 16 frames of image data. In 268435456 data units, data 0 indicates that no photon is incident on the pixel unit corresponding to the data unit, and data 1 indicates that at least one photon is incident on the pixel unit corresponding to the data unit.
The image reconstruction parameters (k V, k H, k F) = (16,16,16) configured in this embodiment, that is, this embodiment selects all 16 frames 4096 rows, 4096 columns binary data matrix, and selects 16 rows, 16 columns binary submatrix therein based on the original binary data stream (4096,4096,16), thereby completing the extraction of 16 × 16 binary data block. The theoretical maximum value of the 16 × 16 binary data block is 16 × 1=4096 during the subsequent data processing and image reconstruction. When the actual value of the 16 × 16 binary data block is S, the gray value of the corresponding pixel point in the reconstructed image is S/4096 × 255.
The image reconstruction mode selected in this embodiment is a data-overlap-free image reconstruction mode, that is, when data processing and image reconstruction are performed based on the image reconstruction parameters (16,16,16), there is no data overlap between the respective image reconstruction windows. As a result, the number of pixel points corresponding to the reconstructed image is (4096/16) =256 × 256=65536, that is, the resolution of the reconstructed image is 256 × 256.
As shown in fig. 6, in this embodiment, based on a standard test picture "lena" in digital image processing, an original binary data stream (4096,4096,16) is generated by an analog simulation tool, an image reconstruction parameter is configured to be (16,16,16), a data-overlap-free image reconstruction mode is selected, and through data processing, grayscale image reconstruction with a resolution of 256 × 256 is completed.
As shown in fig. 6, in this embodiment, based on a standard test picture "lena" in digital image processing, an original binary data stream (4096,4096,16) is generated by an analog simulation tool, an image reconstruction parameter is configured to be (16,16,16), a data-overlap-free image reconstruction mode is selected, and through data processing, grayscale image reconstruction with a resolution of 256 × 256 is completed.
Claims (1)
1. The image reconstruction method for the novel binary image sensor is characterized by comprising the following steps: the method comprises the following steps: (1) collecting optical signals: collecting and storing signal electrons generated by incident photons through the pixels and the array thereof; (2) generating an electrical signal: signal electrons are converted into signal voltage values through charge transfer, correlated double sampling and amplification operations; (3) generating a binary data stream: quantizing the signal voltage value into a digital value 0 or a digital value 1 by a binary quantizer; wherein, a digital value of 0 represents no photon incidence, and a digital value of 1 represents at least one photon incidence; in each frame, the output results of each pixel form a binary data matrix, the binary data stream comprises F frames of binary data matrices, each frame of binary data matrix comprises V rows and H columns, and V x H data units are counted, and the V x H data units correspond to the V x H pixel units; and (4) collecting a binary data stream: collecting and storing each frame of binary data matrix generated by the image sensor, namely collecting binary data stream; and (5) configuring image reconstruction parameters: configuring image reconstruction parameters for subsequent image reconstruction operation, wherein the image reconstruction parameters specifically comprise: before data processing and image reconstruction, configuring image reconstruction parameters as (k V, k H and k F); extracting k F frames in an F-frame binary data matrix based on the original binary data stream (V, H, F); extracting k V rows and k H columns binary data submatrix from the binary data matrixes of the V rows and the H columns; finally, extracting the binary data block with the specified size; under the image reconstruction parameters (k V, k H, k F), the binary data submatrix of each frame has the maximum value of k V × k H and the minimum value of 0; the binary data block has a maximum value of k V × k H × k F and a minimum value of 0; (6) selecting an image reconstruction mode: selecting an image reconstruction mode for subsequent image reconstruction operation, wherein the image reconstruction mode specifically comprises the following steps: based on the original binary data stream (V, H, F), k V row k H column binary data submatrix is extracted, and the submatrix is called an image reconstruction window; the first type of image reconstruction mode is called a non-data overlapping image reconstruction mode, and when horizontal scanning and vertical scanning are carried out, data do not overlap among image reconstruction windows; the second type of image reconstruction mode is called a data overlapping image reconstruction mode, when horizontal scanning and vertical scanning are carried out, data overlapping exists between image reconstruction windows, and the overlapping range can be adjusted according to actual conditions; (7) processing the binary data stream: under the appointed image reconstruction parameter and mode, processing the data of the binary data stream; and (8) completing image reconstruction and outputting the reconstructed image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910752407.2A CN112396670B (en) | 2019-08-15 | 2019-08-15 | Image reconstruction method for novel binary image sensor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910752407.2A CN112396670B (en) | 2019-08-15 | 2019-08-15 | Image reconstruction method for novel binary image sensor |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112396670A CN112396670A (en) | 2021-02-23 |
| CN112396670B true CN112396670B (en) | 2023-01-10 |
Family
ID=74602784
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910752407.2A Active CN112396670B (en) | 2019-08-15 | 2019-08-15 | Image reconstruction method for novel binary image sensor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112396670B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09109461A (en) * | 1995-10-16 | 1997-04-28 | Seiko Epson Corp | Apparatus for restoring compressed two-value image data |
| KR20040061738A (en) * | 2002-12-31 | 2004-07-07 | 정보통신연구진흥원 | Method of decomposing and composing multimedia data based on discrete wavelet transformation and system for decomposing and composing the same |
| CN101218803A (en) * | 2005-07-08 | 2008-07-09 | 西门子公司 | Method for sending a media data stream and method for receiving and creating a reconstructed media data stream, and associated transmission apparatus and reception apparatus |
| DE102009013402A1 (en) * | 2009-04-01 | 2010-10-07 | Compugroup Services Gmbh | Method for transmitting binary data in stream of facsimile connection, involves processing images, and allowing insertion of to-be-transmitted meta data directly into binary image information by facsimile connection |
| CN102036023A (en) * | 2009-10-01 | 2011-04-27 | 索尼公司 | Image sensor, electronic system and method for controlling image sensor |
| CN103975580A (en) * | 2011-12-09 | 2014-08-06 | 索尼公司 | Image pickup device, electronic apparatus, optically stimulated luminescence detection scanner, and image pickup method |
| CN104380340A (en) * | 2012-03-20 | 2015-02-25 | 保罗－里德－史密斯－吉塔尔斯股份合作有限公司 | Scoring and adjusting pixels based on neighborhood relationships for revealing data in images |
| CN106656466A (en) * | 2016-12-20 | 2017-05-10 | 中国科学院理化技术研究所 | Correlated encryption transmission device and method of underwater optical image |
| CN109413407A (en) * | 2017-12-05 | 2019-03-01 | 西北工业大学 | High spatial resolution optical field acquisition device and image generating method |
| CN109672891A (en) * | 2018-12-25 | 2019-04-23 | 哈尔滨工业大学 | The lossless second-compressed method of jpeg image |
| CN109799731A (en) * | 2019-01-28 | 2019-05-24 | 北京环境特性研究所 | Multiband optics radiant type Hardware In The Loop Simulation Method and system |
| CN109816603A (en) * | 2018-12-30 | 2019-05-28 | 天津大学 | The imaging sensor image restoring method of single photon counting imaging |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102466520B (en) * | 2010-11-11 | 2014-12-17 | 香港纺织及成衣研发中心 | Multispectral imaging color measurement system and imaging signal processing method thereof |
| US10129495B2 (en) * | 2016-03-25 | 2018-11-13 | Qualcomm Incorporated | Apparatus and method for generating local binary patterns (LBPS) |
-
2019
- 2019-08-15 CN CN201910752407.2A patent/CN112396670B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09109461A (en) * | 1995-10-16 | 1997-04-28 | Seiko Epson Corp | Apparatus for restoring compressed two-value image data |
| KR20040061738A (en) * | 2002-12-31 | 2004-07-07 | 정보통신연구진흥원 | Method of decomposing and composing multimedia data based on discrete wavelet transformation and system for decomposing and composing the same |
| CN101218803A (en) * | 2005-07-08 | 2008-07-09 | 西门子公司 | Method for sending a media data stream and method for receiving and creating a reconstructed media data stream, and associated transmission apparatus and reception apparatus |
| DE102009013402A1 (en) * | 2009-04-01 | 2010-10-07 | Compugroup Services Gmbh | Method for transmitting binary data in stream of facsimile connection, involves processing images, and allowing insertion of to-be-transmitted meta data directly into binary image information by facsimile connection |
| CN102036023A (en) * | 2009-10-01 | 2011-04-27 | 索尼公司 | Image sensor, electronic system and method for controlling image sensor |
| CN103975580A (en) * | 2011-12-09 | 2014-08-06 | 索尼公司 | Image pickup device, electronic apparatus, optically stimulated luminescence detection scanner, and image pickup method |
| CN104380340A (en) * | 2012-03-20 | 2015-02-25 | 保罗－里德－史密斯－吉塔尔斯股份合作有限公司 | Scoring and adjusting pixels based on neighborhood relationships for revealing data in images |
| CN106656466A (en) * | 2016-12-20 | 2017-05-10 | 中国科学院理化技术研究所 | Correlated encryption transmission device and method of underwater optical image |
| CN109413407A (en) * | 2017-12-05 | 2019-03-01 | 西北工业大学 | High spatial resolution optical field acquisition device and image generating method |
| CN109672891A (en) * | 2018-12-25 | 2019-04-23 | 哈尔滨工业大学 | The lossless second-compressed method of jpeg image |
| CN109816603A (en) * | 2018-12-30 | 2019-05-28 | 天津大学 | The imaging sensor image restoring method of single photon counting imaging |
| CN109799731A (en) * | 2019-01-28 | 2019-05-24 | 北京环境特性研究所 | Multiband optics radiant type Hardware In The Loop Simulation Method and system |
Non-Patent Citations (2)
| Title |
|---|
| "A Method of Biomimetic Visual Perception and Image Reconstruction Based on Pulse Sequence of Events";Jiangtao Xu等;《 IEEE Sensors Journal 》;20181111;Page(s): 1008 - 1018 * |
| "基于CMOS图像传感器列级ADC的数字双采样";徐江涛等;《南 开 大 学 学 报（自然科学版)》;20150228;第61-66页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112396670A (en) | 2021-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20170310910A1 (en) | Oversampled image sensor with conditional pixel readout | |
| JP4485203B2 (en) | Noise elimination type CMOS image sensor | |
| KR102212100B1 (en) | Split-gate conditional-reset image sensor | |
| US7091466B2 (en) | Apparatus and method for pixel binning in an image sensor | |
| CN106657829B (en) | Reading circuit and reading method for high-density SPAD array level analog signals | |
| EP2323361B1 (en) | Image sensor with interleaved image output | |
| CN203775318U (en) | Ultraviolet focal plane readout circuit based on pixel-level analog-to-digital conversion | |
| CN103856730A (en) | Ultraviolet focal plane readout circuit and method based on pixel level analog-to-digital conversion | |
| CN101888492B (en) | Processor used for image sensor wtih multiple data readout scheme | |
| JP2000516774A (en) | Image sensor with high dynamic range linear output | |
| CN102625059A (en) | Dynamic range extension for CMOS image sensors for mobile applications | |
| US9391632B1 (en) | Method and system for implementing an extended range approximation analog-to-digital converter | |
| CN116349240A (en) | Digital pixel sensor with reduced quantization operation | |
| CN112396670B (en) | Image reconstruction method for novel binary image sensor | |
| EP3930312B1 (en) | Delta image sensor with digital pixel storage | |
| CN101014098A (en) | Imaging device | |
| CN110336953B (en) | Image sensor with four-pixel structure and reading control method | |
| Fowler | High dynamic range image sensor architectures | |
| JP4311095B2 (en) | Solid-state imaging device and driving method thereof | |
| CN111182246B (en) | CMS-based CMOS image sensor reading circuit | |
| US11284031B1 (en) | Systems and methods for low-power image digitization | |
| CN111669526B (en) | CMOS image sensor for improving frame frequency high-speed all-digital data reading | |
| Ma et al. | A 1.26-inch 40.7 Mega-Pixel Photon-Counting Quanta Image Sensor with 0.35 e-Read Noise and 95dB Single-Exposure Dynamic Range | |
| CN219420923U (en) | Pixel circuit and image sensor | |
| Elmezayen | Smart Readout Electronics for CMOS Image Sensors |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |