CN217388784U - Internal image processing architecture of chip - Google Patents

Abstract

The utility model discloses a chip internal image processing framework, which comprises a MUX module, an ISP module, a storage module and a plurality of input interfaces; the input interfaces are connected with the storage module and are used for storing image data input by different cameras into the storage module; the MUX module is respectively connected with the storage module, the ISP module and the input interfaces and is used for receiving image data from the input interfaces or reading the image data from the storage module to perform image data fusion processing and then sending the image data after the fusion processing to the ISP module; the ISP module is connected with the MUX module and used for receiving the image data spliced by the MUX module, and the ISP module is connected with the storage module and used for reading the image data from the storage module or writing the image data into the storage module. The chip can support multiple cameras, reduce the area of the chip and reduce the use cost.

Description

Internal image processing architecture of chip

Technical Field

The utility model relates to a chip technology field, concretely relates to chip internal image processing framework.

Background

The camera is an image-based sensor, has very important function on robot perception, and the image information contains a large amount of data information, and the potential of the image information is also continuously mined through an algorithm. In future applications, the system complexity is higher and higher, the demands on the cameras are more and more, the chip needs to support a plurality of cameras, the existing image processing architecture inside a single chip only supports a single camera, when the plurality of cameras are used, the plurality of image processing architectures are needed to perform image processing, and the use cost is higher.

SUMMERY OF THE UTILITY MODEL

In order to solve the above problem, the utility model provides a chip internal image processing framework, use cost is lower. The utility model has the following concrete technical scheme:

an internal image processing architecture of a chip comprises a MUX module, an ISP module, a storage module and a plurality of input interfaces; the input interfaces are connected with the storage module and are used for storing image data input by different cameras into the storage module; the MUX module is respectively connected with the storage module, the ISP module and the input interfaces and is used for receiving image data from the input interfaces or reading the image data from the storage module to perform image data fusion processing and then sending the image data after the fusion processing to the ISP module; the ISP module is connected with the MUX module and used for receiving the image data spliced by the MUX module, and the ISP module is connected with the storage module and used for reading the image data from the storage module or writing the image data into the storage module.

Further, the input interfaces are five mipi interfaces, and the five mipi interfaces comprise two 4-channel interfaces and three 2-channel interfaces.

Further, the input interfaces comprise a parallel port, and the parallel port is multiplexed with one of the mipi interfaces.

Further, the format of the image data input to the storage module by the input interfaces is RAW8, RAW10, RAW12 or YUV 422.

Further, the ISP module reads the image data from the storage module by directly reading the image data or by reading the address through the configuration register to time-divisionally read the image data in the storage module.

Compared with the prior art, the beneficial effects of the utility model reside in that: according to the technical scheme, the ISP is multiplexed through the image processing framework with the MUX module, so that the area of the chip is reduced while the chip supports multiple cameras, and the use cost is reduced.

Drawings

Fig. 1 is a schematic structural diagram of the chip internal image processing architecture of the present invention.

Detailed Description

Reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar functions throughout.

In the description of the present invention, it should be noted that, for the orientation words, if there are terms such as "center", "lateral", "longitudinal", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., the orientation and positional relationship indicated are based on the orientation or positional relationship shown in the drawings, and only for the convenience of describing the present invention and simplifying the description, it is not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and not be construed as limiting the specific scope of the present invention.

Furthermore, if the terms "first" and "second" are used for descriptive purposes only, they are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features. Thus, the definition of "a first" or "a second" feature may explicitly or implicitly include one or more of the features, and in the description of the invention, "at least" means one or more unless specifically defined otherwise.

In the present invention, unless otherwise expressly specified or limited, the terms "assembled", "connected", and "connected", if any, are to be construed broadly, e.g., as meaning fixedly connected, detachably connected, or integrally connected; or may be a mechanical connection; the two elements can be directly connected or connected through an intermediate medium, and the two elements can be communicated with each other. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to specific situations.

In the present application, unless otherwise specified or limited, "above" or "below" a first feature may include the first and second features being in direct contact, and may also include the first and second features not being in direct contact but being in contact with each other through another feature therebetween. Also, the first feature being "above," "below," and "above" the second feature includes the first feature being directly above and obliquely above the second feature, or simply an elevation which indicates a level of the first feature being higher than an elevation of the second feature. The first feature being "above", "below" and "beneath" the second feature includes the first feature being directly below or obliquely below the second feature, or merely means that the first feature is at a lower level than the second feature.

The technical solution and the advantages of the present invention will be more clear and clear by further describing the embodiments of the present invention with reference to the drawings of the specification. The embodiments described below are exemplary and are intended to be illustrative of the present invention, but should not be construed as limiting the invention.

ISP (image Signal processor), i.e. image processing, mainly functions to perform post-processing on the Signal output by the front-end image sensor, and mainly functions such as linear correction, noise removal, dead pixel removal, interpolation, white balance, automatic exposure control, etc., and can better restore field details under different optical conditions only depending on the ISP, and the imaging quality of the camera is determined to a great extent by the ISP technology. It can be divided into two forms of independent and integrated. The Firmware of the ISP comprises three parts, wherein one part is an ISP control unit and a basic algorithm library, one part is an AE/AWB/AF algorithm library, and the other part is a sensor library. The basic idea of Firmware design is to provide a 3A algorithm library separately, schedule a basic algorithm library and the 3A algorithm library by an ISP control unit, and register function call-back to the ISP basic algorithm library and the 3A algorithm library respectively by a sensor library so as to realize differentiated sensor adaptation. Mux (multiplexer) multiplexer, 1. combine two signals (analog or digital) into one, which can be separated in the subsequent circuit. For example: OFDM, standard FM stereo broadcasting (left and right channel signals may be multiplexed into a baseband signal), standard television (video and multiple audio signals share a single channel), and time division multiplexing (signals provided in different time slots). 2. An analog switch matrix, usually on a CMOS chip, allows one input to be switched to any of several outputs, specifically controlled by digital signals. The multiplexer can also be used for reverse data transmission, and for the matrix switch, one of the several inputs can be switched to the output end, and the specific switching is determined by the control signal. DDR = Double Data Rate, DDR SDRAM = Double Rate Synchronous Dynamic Random Access Memory, commonly known as DDR, where SDRAM is an abbreviation of Synchronous Dynamic Random Access Memory.

As shown in fig. 1, an internal image processing architecture of a chip includes a MUX module, an ISP module, a storage module, and a plurality of input interfaces; the input interfaces are connected with the storage module and are used for storing image data input by different cameras into the storage module; the MUX module is respectively connected with the storage module, the ISP module and the input interfaces and is used for receiving image data from the input interfaces or reading the image data from the storage module to perform image data fusion processing and then sending the image data after the fusion processing to the ISP module; the ISP module is connected with the MUX module and used for receiving the image data spliced by the MUX module, and the ISP module is connected with the storage module and used for reading the image data from the storage module or writing the image data into the storage module.

As one example, the number of input interfaces is five mipi interfaces including two 4-channel interfaces and three 2-channel interfaces. A mipi (mobile Industry Processor interface) is an alliance established by companies such as ARM, Nokia, ST, TI, etc. in 2003, and aims to standardize interfaces inside a mobile phone, such as a camera, a display screen interface, a radio frequency/baseband interface, etc., thereby reducing the complexity of mobile phone design and increasing the design flexibility. Different WorkGroups exist below the MIPI alliance, and a series of internal interface standards of the mobile phone are respectively defined, such as a camera interface CSI, a display interface DSI, a radio frequency interface DigRF, a microphone/loudspeaker interface SLIMbus and the like.

In one embodiment, the input interfaces include a parallel port, and the parallel port is multiplexed with one of the mipi interfaces. The parallel port is a parallel interface for short, and refers to an interface standard for transmitting data by adopting a parallel transmission mode. 1. The width of the data channel transmitted in parallel, also called the number of bits transmitted by the interface, 2. the characteristics of the additional interface control lines or interaction signals for coordinating the parallel data transmission.

The format of the image data input to the storage module by the input interfaces is RAW8, RAW10, RAW12 or YUV 422.

As one example, the ISP module reads the image data from the storage module by directly reading the image data or by reading the address through the configuration register to time-divisionally read the image data in the storage module.

An image processing method, which performs image processing through the above-mentioned on-chip image processing architecture, includes the following steps: respectively receiving image data input by different cameras through a plurality of input interfaces; storing the input image data into a storage module or/and sending the input image data to a MUX module according to the control command or/and the working state of the MUX module; the MUX module reads the image data from the storage module or receives the image data sent by the input interfaces, then performs fusion processing on the image data, and sends the fused image data to the ISP module; the ISP module receives the image data sent by the MUX module or reads the image data from the storage module, then processes the image data, and writes the processed image data into the storage module.

As one embodiment, the method for storing the input image data in the storage module or/and sending the input image data to the MUX module according to the control command or/and the operation state of the MUX module includes the following steps: if the control command is to store the input image data into the storage module, directly storing the image data into the storage module; if the control command is to send the input image data to the MUX module, judging whether the MUX module is in a working state; if the MUX module is in a working state, the image data is stored in the storage module, the MUX module reads the image data from the storage module after finishing working, and if the MUX module is not in the working state, the image data is stored in the storage module and sent to the MUX module.

As an embodiment, the MUX module reads image data from the storage module or receives image data sent by a plurality of input interfaces, and then performs fusion processing on the image data, including the following steps: and after the MUX module reads the image data from the storage module or receives the image data sent by the plurality of input interfaces, splicing the image data input by the two different interfaces.

As one embodiment, the ISP module reads image data from the storage module, and includes the following steps: the ISP module directly reads the image data from the storage module or reads the address through the configuration register to read the image data in the storage module in a time-sharing way.

As one embodiment, the ISP module receives the image data sent by the MUX module or reads the image data from the storage module, and then processes the image data, including the following steps: the processing of the image data by the ISP module includes at least any one of linear correction, noise removal, dead pixel removal, interpolation, white balancing, or automatic exposure control.

As shown in fig. 1, when the chip internal image processing architecture works, image Data input by different cameras are received through C0-C4 interfaces, then the interfaces perform conventional cropping processing on the received image Data through a Crop layer of the interfaces, then the cropped original Data is stored into a memory module (RAW Data to DDR) through DDR (double Data rate), wherein the memory module is the DDR memory or is transmitted into a MUX module, the MUX module receives the image Data or reads the image Data (DDR Data Read) from the memory module through DDR, when the 3d processing is performed, the images received by the two interfaces are spliced into a large image, and then the large image is input into an ISP for processing. When the ISP reads data from the DDR for processing, the register can be configured to Read addresses, so that multiple paths of data can be stored in the DDR firstly, then the ISP is called in a time-sharing mode for processing, and ISP configuration information of different paths is Read in the DDR (DDR ISP Reg Read). The 5 paths of input can be independently output to the DDR and then enter the ISP path independently, and the input can still be stored in the DDR after entering the ISP path. The ISP module writes the processed image Data into the memory module memory, i.e. the DDR Data Write in the figure.

Compared with the prior art, the beneficial effects of the utility model reside in that: according to the technical scheme, the ISP is multiplexed through the image processing framework with the MUX module, so that the area of the chip is reduced and the use cost is reduced while the chip supports multiple cameras.

In the description of the specification, reference to the description of "one embodiment," "preferably," "an example," "a specific example" or "some examples" or the like means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention, and schematic representations of the terms in this specification do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The connection mode connected in the description of the specification has obvious effects and practical effectiveness.

With the above structure and principle in mind, those skilled in the art should understand that the present invention is not limited to the above embodiments, and all modifications and substitutions based on the present invention and adopting the known technology in the art are within the scope of the present invention, which should be limited by the claims.

Claims (5)

1. An internal image processing architecture of a chip is characterized in that the internal image processing architecture of the chip comprises a MUX module, an ISP module, a storage module and a plurality of input interfaces;

the input interfaces are connected with the storage module and are used for storing image data input by different cameras into the storage module;

the MUX module is respectively connected with the storage module, the ISP module and the input interfaces and is used for receiving image data from the input interfaces or reading the image data from the storage module to perform image data fusion processing and then sending the image data after the fusion processing to the ISP module;

the ISP module is connected with the MUX module and used for receiving the image data spliced by the MUX module, and the ISP module is connected with the storage module and used for reading the image data from the storage module or writing the image data into the storage module.

2. The on-chip image processing architecture of claim 1, wherein the number of input interfaces is five mipi interfaces, and the five mipi interfaces comprise two 4-channel interfaces and three 2-channel interfaces.

3. The on-chip image processing architecture as recited in claim 2, wherein the plurality of input interfaces comprise a parallel port, and the parallel port is multiplexed with one of the mipi interfaces.

4. The on-chip image processing architecture as claimed in claim 1, wherein the format of the image data inputted to the memory module by said plurality of input interfaces is RAW8, RAW10, RAW12 or YUV 422.

5. The architecture of claim 1, wherein the ISP module reads the image data from the storage module by directly reading the image data or by reading the address through the configuration register to time-share the image data in the storage module.

Images