CN114785904A - Chip internal image processing architecture and image processing method - Google Patents
Chip internal image processing architecture and image processing method Download PDFInfo
- Publication number
- CN114785904A CN114785904A CN202210478262.3A CN202210478262A CN114785904A CN 114785904 A CN114785904 A CN 114785904A CN 202210478262 A CN202210478262 A CN 202210478262A CN 114785904 A CN114785904 A CN 114785904A
- Authority
- CN
- China
- Prior art keywords
- image data
- module
- storage module
- mux
- input
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/88—Camera processing pipelines; Components thereof for processing colour signals for colour balance, e.g. white-balance circuits or colour temperature control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/64—Circuits for processing colour signals
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an internal image processing architecture of a chip and an image processing method, wherein the internal image processing architecture of the chip comprises a MUX module, an ISP module, a storage module and a plurality of input interfaces; the input interfaces are connected with the storage module and are used for storing image data input by different cameras into the storage module; the MUX module is respectively connected with the storage module, the ISP module and the input interfaces and is used for receiving image data from the input interfaces or reading the image data from the storage module to perform image data fusion processing and then sending the image data after the fusion processing to the ISP module; the ISP module is connected with the MUX module and used for receiving the image data spliced by the MUX module, and the ISP module is connected with the storage module and used for reading the image data from the storage module or writing the image data into the storage module. The chip supports multiple cameras, the area of the chip is reduced, and the use cost is reduced.
Description
Technical Field
The invention relates to the technical field of chips, in particular to an internal image processing architecture and an image processing method of a chip.
Background
The camera is an image-based sensor, has a very important role in robot perception, image information contains a large amount of data information, and the potential of the image information is also continuously mined through an algorithm. In future applications, the system complexity is higher and higher, the demands on the cameras are more and more, the chip needs to support a plurality of cameras, the existing image processing architecture inside a single chip only supports a single camera, when the plurality of cameras are used, the plurality of image processing architectures are needed to perform image processing, and the use cost is higher.
Disclosure of Invention
In order to solve the above problems, the present invention provides an internal image processing architecture and an image processing method for a chip, which are low in cost. The specific technical scheme of the invention is as follows:
an internal image processing architecture of a chip comprises a MUX module, an ISP module, a storage module and a plurality of input interfaces; the input interfaces are connected with the storage module and are used for storing image data input by different cameras into the storage module; the MUX module is respectively connected with the storage module, the ISP module and the input interfaces and is used for receiving image data from the input interfaces or reading the image data from the storage module to perform image data fusion processing and then sending the image data after the fusion processing to the ISP module; the ISP module is connected with the MUX module and used for receiving the image data spliced by the MUX module, and the ISP module is connected with the storage module and used for reading the image data from the storage module or writing the image data into the storage module.
Further, the number of input interfaces is five mipi interfaces, and the five mipi interfaces include two 4-channel interfaces and three 2-channel interfaces.
Further, the input interfaces include a parallel port, and the parallel port is multiplexed with one of the mipi interfaces.
Further, the format of the image data input to the storage module by the input interfaces is RAW8, RAW10, RAW12 or YUV 422.
Further, the ISP module reads the image data from the storage module by directly reading the image data or by reading the address through the configuration register to time-divisionally read the image data in the storage module.
An image processing method, which performs image processing through the above-mentioned on-chip image processing architecture, includes the following steps: respectively receiving image data input by different cameras through a plurality of input interfaces; storing the input image data into a storage module or/and sending the input image data to a MUX module according to the control command or/and the working state of the MUX module; the MUX module reads the image data from the storage module or receives the image data sent by the input interfaces, then performs fusion processing on the image data, and sends the fused image data to the ISP module; and the ISP module receives the image data sent by the MUX module or reads the image data from the storage module, processes the image data, and writes the processed image data into the storage module.
Further, the method for storing the input image data into the storage module or/and sending the input image data to the MUX module according to the control command or/and the working state of the MUX module comprises the following steps: if the control command is to store the input image data into the storage module, directly storing the image data into the storage module; if the control command is to send the input image data to the MUX module, judging whether the MUX module is in a working state; if the MUX module is in a working state, the image data are stored in the storage module, the MUX module reads the image data from the storage module after working, and if the MUX module is not in the working state, the image data are stored in the storage module and sent to the MUX module.
Furthermore, the MUX module reads the image data from the storage module or receives the image data sent by the input interfaces, and then fusion processing is carried out on the image data, and the method comprises the following steps: and after the MUX module reads the image data from the storage module or receives the image data sent by the plurality of input interfaces, the image data input by the two different interfaces are spliced.
Further, the ISP module reads the image data from the storage module, and comprises the following steps: the ISP module reads the image data from the storage module directly or reads the address through the configuration register to read the image data in the storage module in a time-sharing manner.
Further, the ISP module receives the image data sent by the MUX module or reads the image data from the storage module, and then processes the image data, including the following steps: the processing of the image data by the ISP module includes at least any one of linear correction, noise removal, dead pixel removal, interpolation, white balancing, or automatic exposure control.
Compared with the prior art, the invention has the beneficial effects that: according to the technical scheme, the ISP is multiplexed through the image processing framework with the MUX module, so that the area of the chip is reduced while the chip supports multiple cameras, and the use cost is reduced.
Drawings
FIG. 1 is a schematic diagram of an internal chip image processing architecture according to the present invention;
FIG. 2 is a diagram illustrating an image processing method according to the present invention.
Detailed Description
Reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar functions throughout.
In the description of the present invention, it should be noted that, for the terms of orientation, such as "central", "lateral", "longitudinal", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., it indicates that the orientation and positional relationship shown in the drawings are based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, but does not indicate or imply that the device or element referred to must have a specific orientation, be constructed in a specific orientation, and be operated without limiting the specific scope of protection of the present invention.
Furthermore, if the terms "first" and "second" are used for descriptive purposes only, they are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features. Thus, a definition of "a first" or "a second" feature may explicitly or implicitly include one or more of the feature, and in the description of the invention, "at least" means one or more unless specifically defined otherwise.
In the present invention, unless otherwise expressly specified or limited, the terms "assembled", "connected", and "connected" are to be construed broadly, e.g., as meaning fixedly connected, detachably connected, or integrally connected; or may be a mechanical connection; the two elements can be directly connected with each other or connected with each other through an intermediate medium, and the two elements can be communicated with each other. The specific meanings of the above-mentioned terms in the present invention can be understood according to specific situations by those of ordinary skill in the art.
In the present invention, unless otherwise specified and limited, "a first feature" on "or" under "a second feature may include the first and second features being in direct contact, and may also include the first and second features not being in direct contact but being in contact through another feature therebetween. Also, the first feature "above," "below," and "over" the second feature may include the first feature being directly above and obliquely above the second feature, or simply an elevation where the first feature is at a higher level than the second feature. The first feature being "above", "below" and "beneath" the second feature includes the first feature being directly below or obliquely below the second feature, or merely means that the first feature is at a lower level than the second feature.
The technical scheme and the beneficial effects of the invention are clearer and clearer by further describing the specific embodiment of the invention with the accompanying drawings of the specification. The embodiments described below are exemplary and are intended to be illustrative of the invention, but are not to be construed as limiting the invention.
ISP (image Signal processor), i.e. image processing, mainly functions to perform post-processing on the Signal output by the front-end image sensor, and mainly functions such as linear correction, noise removal, dead pixel removal, interpolation, white balance, automatic exposure control, etc., and can better restore field details under different optical conditions only depending on the ISP, and the imaging quality of the camera is determined to a great extent by the ISP technology. It can be divided into two forms of independent and integrated. The Firmware of the ISP comprises three parts, wherein one part is an ISP control unit and a basic algorithm library, one part is an AE/AWB/AF algorithm library, and the other part is a sensor library. The basic idea of Firmware design is to provide a 3A algorithm library separately, schedule a basic algorithm library and the 3A algorithm library by an ISP control unit, and register function call-back to the ISP basic algorithm library and the 3A algorithm library respectively by a sensor library so as to realize differentiated sensor adaptation. Mux (multiplexer) multiplexer, 1. combine two signals (analog or digital) into one, which can be separated in the subsequent circuit. For example: OFDM, standard FM stereo broadcasting (left and right channel signals may be multiplexed into a baseband signal), standard television (video and multiple audio signals share a single channel), and time division multiplexing (signals provided in different time slots). 2. An analog switch matrix, usually on a CMOS chip, allows one input to be switched to any of several outputs, specifically controlled by digital signals. The multiplexer can also be used for reverse data transmission, and for the matrix switch, one of the several inputs can be switched to the output end, and the specific switching is determined by the control signal. DDR = Double Data Rate, DDR SDRAM = Double Rate Synchronous Dynamic Random Access Memory, commonly known as DDR, where SDRAM is an abbreviation of Synchronous Dynamic Random Access Memory.
As shown in fig. 1, an internal image processing architecture of a chip includes a MUX module, an ISP module, a storage module, and a plurality of input interfaces; the input interfaces are connected with the storage module and are used for storing image data input by different cameras into the storage module; the MUX module is respectively connected with the storage module, the ISP module and the input interfaces and is used for receiving image data from the input interfaces or reading the image data from the storage module to perform image data fusion processing and then sending the image data after the fusion processing to the ISP module; the ISP module is connected with the MUX module and used for receiving the image data spliced by the MUX module, and the ISP module is connected with the storage module and used for reading the image data from the storage module or writing the image data into the storage module.
As one example, the number of input interfaces is five mipi interfaces including two 4-channel interfaces and three 2-channel interfaces. A mipi (mobile Industry Processor interface) is an alliance established by companies such as ARM, Nokia, ST, TI, etc. in 2003, and aims to standardize interfaces inside a mobile phone, such as a camera, a display screen interface, a radio frequency/baseband interface, etc., thereby reducing the complexity of mobile phone design and increasing the design flexibility. Different WorkGroups exist below the MIPI alliance, and a series of internal interface standards of the mobile phone are respectively defined, such as a camera interface CSI, a display interface DSI, a radio frequency interface DigRF, a microphone/loudspeaker interface SLIMbus and the like.
In one embodiment, the input interfaces include a parallel port, and the parallel port is multiplexed with one mipi interface. The parallel port is a parallel interface for short, and refers to an interface standard for transmitting data by adopting a parallel transmission mode. 1. The width of the data channel transmitted in parallel, also called the number of bits transmitted by the interface, 2. the characteristics of the additional interface control lines or interactive signals used to coordinate the parallel data transmission.
The format of the image data input to the storage module by the input interfaces is RAW8, RAW10, RAW12 or YUV 422.
As one example, the ISP module reads the image data from the storage module by directly reading the image data or by reading the address through the configuration register to time-divisionally read the image data in the storage module.
As shown in fig. 2, an image processing method for performing image processing by the above-mentioned on-chip image processing architecture includes the following steps: respectively receiving image data input by different cameras through a plurality of input interfaces; storing the input image data into a storage module or/and sending the input image data to a MUX module according to the control command or/and the working state of the MUX module; the MUX module reads the image data from the storage module or receives the image data sent by the input interfaces, then performs fusion processing on the image data, and sends the fused image data to the ISP module; the ISP module receives the image data sent by the MUX module or reads the image data from the storage module, then processes the image data, and writes the processed image data into the storage module.
As one embodiment, the method for storing the input image data into the storage module or/and sending the input image data to the MUX module according to the control command or/and the operating state of the MUX module includes the following steps: if the control command is to store the input image data into the storage module, directly storing the image data into the storage module; if the control command is to send the input image data to the MUX module, judging whether the MUX module is in a working state; if the MUX module is in a working state, the image data is stored in the storage module, the MUX module reads the image data from the storage module after finishing working, and if the MUX module is not in the working state, the image data is stored in the storage module and sent to the MUX module.
As one embodiment, the MUX module reads image data from the storage module or receives image data sent by a plurality of input interfaces, and then performs fusion processing on the image data, including the following steps: and after the MUX module reads the image data from the storage module or receives the image data sent by the plurality of input interfaces, splicing the image data input by the two different interfaces.
As one embodiment, the ISP module reads image data from the storage module, and includes the following steps: the ISP module directly reads the image data from the storage module or reads the address through the configuration register to read the image data in the storage module in a time-sharing way.
As one embodiment, the ISP module receives the image data sent by the MUX module or reads the image data from the storage module, and then processes the image data, including the following steps: the processing of the image data by the ISP module includes at least any one of linear correction, noise removal, dead pixel removal, interpolation, white balancing, or automatic exposure control.
As shown in fig. 1, when the chip internal image processing architecture works, image Data input by different cameras are received through C0-C4 interfaces, then the interfaces perform conventional cropping processing on the received image Data through a Crop layer of the interfaces, then the cropped original Data is stored into a memory module (RAW Data to DDR) through DDR (double Data rate), wherein the memory module is DDR memory or is transmitted into a MUX module, the MUX module receives the image Data or reads the image Data (DDR Data) from the memory module, and when the image Data is processed by 3d, the images received by the two interfaces are spliced into a large image and then are input into an ISP for processing. When the ISP reads data from the DDR to process, the register can be configured to Read addresses, so that multiple paths of data can be stored in the DDR firstly, then the ISP is called in a time-sharing mode to process, and ISP configuration information of different paths is Read in the DDR (DDR ISP Reg Read). The 5 paths of input can be independently output to the DDR and then enter the ISP path independently, and the input can still be stored in the DDR after entering the ISP path. The ISP module writes the processed image Data into the memory module memory, i.e. DDR Data Write in the figure.
Compared with the prior art, the invention has the beneficial effects that: according to the technical scheme, the ISP is multiplexed through the image processing framework with the MUX module, so that the area of the chip is reduced and the use cost is reduced while the chip supports multiple cameras.
In the description of the specification, references to the description of "one embodiment," "preferably," "an example," "a specific example" or "some examples" or the like are intended to mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention, and the schematic representation of the terms in this specification does not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The connection mode of connection in the description of the specification has obvious effects and practical effectiveness.
By describing the above structures and principles, those skilled in the art will appreciate that the present invention is not limited to the above embodiments, and that modifications and substitutions based on the known art are within the scope of the present invention as defined by the appended claims.
Claims (10)
1. The chip internal image processing architecture is characterized by comprising a MUX module, an ISP module, a storage module and a plurality of input interfaces;
the input interfaces are connected with the storage module and are used for storing image data input by different cameras into the storage module;
the MUX module is respectively connected with the storage module, the ISP module and the input interfaces and is used for receiving image data from the input interfaces or reading the image data from the storage module to perform image data fusion processing and then sending the image data after the fusion processing to the ISP module;
the ISP module is connected with the MUX module and used for receiving the image data spliced by the MUX module, and the ISP module is connected with the storage module and used for reading the image data from the storage module or writing the image data into the storage module.
2. The on-chip image processing architecture of claim 1, wherein the number of input interfaces is five mipi interfaces, and the five mipi interfaces comprise two 4-channel interfaces and three 2-channel interfaces.
3. The on-chip image processing architecture as recited in claim 2, wherein the plurality of input interfaces comprise a parallel port, and the parallel port is multiplexed with one of the mipi interfaces.
4. The on-chip image processing architecture of claim 1, wherein the format of the image data input to the storage module by the plurality of input interfaces is RAW8, RAW10, RAW12 or YUV 422.
5. The architecture of claim 1, wherein the ISP module reads the image data from the storage module by directly reading the image data or by time-sharing reading the image data from the storage module by reading the address through the configuration register.
6. An image processing method, characterized in that the image processing method performs image processing by the on-chip image processing architecture of any one of claims 1 to 5, the image processing method comprising the steps of:
respectively receiving image data input by different cameras through a plurality of input interfaces;
storing the input image data into a storage module or/and sending the input image data to a MUX module according to the control command or/and the working state of the MUX module;
the MUX module reads the image data from the storage module or receives the image data sent by the input interfaces, then performs fusion processing on the image data, and sends the fused image data to the ISP module;
and the ISP module receives the image data sent by the MUX module or reads the image data from the storage module, processes the image data, and writes the processed image data into the storage module.
7. The image processing method according to claim 6, wherein the storing of the input image data in the storage module or/and the sending of the input image data to the MUX module according to the control command or/and the operation state of the MUX module comprises the steps of:
if the control command is to store the input image data into the storage module, directly storing the image data into the storage module;
if the control command is to send the input image data to the MUX module, judging whether the MUX module is in a working state;
if the MUX module is in a working state, the image data are stored in the storage module, the MUX module reads the image data from the storage module after working, and if the MUX module is not in the working state, the image data are stored in the storage module and sent to the MUX module.
8. The image processing method according to claim 6, wherein the MUX module reads the image data from the storage module or receives the image data sent by the input interfaces, and then performs the fusion processing on the image data, comprising the following steps:
and after the MUX module reads the image data from the storage module or receives the image data sent by the plurality of input interfaces, the image data input by the two different interfaces are spliced.
9. The image processing method according to claim 6, wherein the ISP module reads the image data from the storage module, and comprises the following steps:
the ISP module directly reads the image data from the storage module or reads the address through the configuration register to read the image data in the storage module in a time-sharing way.
10. The image processing method according to claim 6, wherein the ISP module receives the image data sent by the MUX module or reads the image data from the storage module, and then processes the image data, comprising the steps of:
the processing of the image data by the ISP module includes at least any one of linear correction, noise removal, dead pixel removal, interpolation, white balancing, or automatic exposure control.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210478262.3A CN114785904A (en) | 2022-05-05 | 2022-05-05 | Chip internal image processing architecture and image processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210478262.3A CN114785904A (en) | 2022-05-05 | 2022-05-05 | Chip internal image processing architecture and image processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114785904A true CN114785904A (en) | 2022-07-22 |
Family
ID=82435856
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210478262.3A Pending CN114785904A (en) | 2022-05-05 | 2022-05-05 | Chip internal image processing architecture and image processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114785904A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117729394A (en) * | 2024-02-18 | 2024-03-19 | 厦门瑞为信息技术有限公司 | Method for simultaneously processing multiple image sensors by single image processing module |
-
2022
- 2022-05-05 CN CN202210478262.3A patent/CN114785904A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117729394A (en) * | 2024-02-18 | 2024-03-19 | 厦门瑞为信息技术有限公司 | Method for simultaneously processing multiple image sensors by single image processing module |
| CN117729394B (en) * | 2024-02-18 | 2024-04-30 | 厦门瑞为信息技术有限公司 | Method for simultaneously processing multiple image sensors by single image processing module |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20080292219A1 (en) | Method And System For An Image Sensor Pipeline On A Mobile Imaging Device | |
| US20010001563A1 (en) | Method and apparatus to control the behavior of a digital camera by detecting connectivity to a universal serial bus | |
| KR20140114501A (en) | Image Data Processing Method And Electronic Device supporting the same | |
| CN111641777A (en) | Image processing method, image processing apparatus, image processor, electronic device, and storage medium | |
| CN114785904A (en) | Chip internal image processing architecture and image processing method | |
| US20080292216A1 (en) | Method and system for processing images using variable size tiles | |
| US9247168B2 (en) | Imaging device including focusing pixels | |
| US20110193988A1 (en) | Semiconductor device and semiconductor integrated circuit | |
| CN108540689B (en) | Image signal processor, application processor and mobile device | |
| CN217388784U (en) | Internal image processing architecture of chip | |
| US8804009B2 (en) | Multimedia information appliance | |
| US9288397B2 (en) | Imaging device, method for processing image, and program product for processing image | |
| JP2017054263A (en) | Image processing device, control method of image processing device, and imaging system | |
| JP2013211715A (en) | Imaging device | |
| US9058668B2 (en) | Method and system for inserting software processing in a hardware image sensor pipeline | |
| JP2013211724A (en) | Imaging apparatus | |
| JP3321904B2 (en) | Imaging device | |
| JP2002176611A (en) | Image processor | |
| JP6669196B2 (en) | Imaging device and imaging device | |
| US8049820B2 (en) | Video processing circuits and methods using same buffer for video decoder and cross color suppressor | |
| JP5906846B2 (en) | Electronic camera | |
| US20040196380A1 (en) | Image processing device, camera and image processing method | |
| US8411153B2 (en) | Camera unit and multimedia information appliance including camera unit | |
| JP2012209798A (en) | Image processing system and imaging apparatus | |
| JP2000316121A (en) | Multiple access mode picture buffer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |