CN112514364A - Image signal processing apparatus, image signal processing method, camera, and movable platform - Google Patents
Image signal processing apparatus, image signal processing method, camera, and movable platform Download PDFInfo
- Publication number
- CN112514364A CN112514364A CN201980050308.8A CN201980050308A CN112514364A CN 112514364 A CN112514364 A CN 112514364A CN 201980050308 A CN201980050308 A CN 201980050308A CN 112514364 A CN112514364 A CN 112514364A
- Authority
- CN
- China
- Prior art keywords
- image
- module
- pixel
- signal processing
- classification information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
Abstract
The application provides an image signal processing device, a method, a camera and a movable platform, wherein the device comprises a pixel classification module and at least two first image signal processing modules; the pixel classification module is used for classifying each pixel in an input image to generate a plurality of classification information, and transmitting the input image and the classification information to the first image signal processing module; the classification information is used for representing the image characteristics of the pixel; the first image signal processing module is used for executing corresponding image processing operation on pixels in an input image based on the plurality of classification information and transmitting the generated image and the plurality of classification information to a next first image signal processing module; the embodiment can reduce the repeated execution of the same or similar steps by the first image signal processing module, and reduce the hardware expenditure cost.
Description
Technical Field
The present application relates to the field of image processing, and in particular, to an image signal processing apparatus and method, a camera, and a movable platform.
Background
In various photographing devices (such as digital cameras, video cameras, camera phones, etc.), raw Image data collected by an Image sensor is usually processed by an Image Signal Processor (ISP), and the ISP includes a plurality of functional modules, such as a sensor correction module, a color correction module, etc., and finally presents an Image visible to the naked eye of people, wherein the processing level of the ISP largely determines the imaging quality. In a traditional ISP, each functional module is relatively independent to implement its respective function, but some modules have similar or identical basic operations, and the modules are relatively independent to each other, so that the similar or identical basic operations among the modules need to be repeated, and are repeatedly implemented by multiple hardware resources, which results in high hardware cost and long image signal processing time.
Disclosure of Invention
In view of the above, the present application provides an image signal processing apparatus, an image signal processing method, a camera and a movable platform.
First, a first aspect of the present application provides an image signal processing apparatus, including a pixel classification module and at least two first image signal processing modules;
at least two first image signal processing modules are connected in sequence after the pixel classification module;
the pixel classification module is used for classifying each pixel in an input image to generate a plurality of classification information, and transmitting the input image and the classification information to a first image signal processing module connected with the input image and the classification information; the classification information is used for representing the image characteristics of the pixel;
the first image signal processing module is used for receiving the image input by the pixel classification module or the previous first image signal processing module and the classification information; and based on the classification information, executing corresponding image processing operation on pixels in the input image, and transmitting the generated image and the classification information to a next first image signal processing module.
According to a second aspect of the embodiments of the present application, there is provided an image signal processing method applied to an image signal processing apparatus, the method including:
in the pixel classification module, classifying each pixel in an input image to generate a plurality of classification information; the classification information is used for representing the image characteristics of the pixel;
and in the first image signal processing module, receiving the image input by the pixel classification module or the previous first image signal processing module and the classification information, executing corresponding image processing operation on the pixels in the input image based on the classification information, and transmitting the generated image and the classification information to the next first image signal processing module.
According to a third aspect of embodiments of the present application, there is provided a camera comprising:
a housing;
the lens assembly is arranged inside the shell;
the image sensor is arranged in the shell and used for sensing the light passing through the lens assembly and generating an electric signal; and the number of the first and second groups,
the image signal processing apparatus according to any one of the first aspect.
According to a fourth aspect of embodiments of the present application, there is also provided a movable platform, including:
a body;
the power system is arranged in the machine body and used for providing power for the movable platform; and a camera as described in the third aspect.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
in the embodiment of the application, the image signal processing device comprises a pixel classification module and at least two first image signal processing modules, wherein the pixel classification module classifies each pixel in an input image to generate a plurality of classification information, the classification information represents the image characteristics of the pixel and transmits the generated classification information to the first image signal processing module, so that the first image signal processing module does not need to repeat the image characteristic determination step of the same or similar pixel, can directly determine the image processing operation to be executed by each pixel based on the plurality of classification information, reduces the hardware resources for performing the same or similar step, and effectively reduces the hardware cost, and the deletion of the repetition step also effectively reduces the duration of image signal processing and improves the image signal processing efficiency.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
Fig. 1 is a block diagram illustrating a first image signal processing apparatus 10 according to an exemplary embodiment of the present application.
Fig. 2 is a block diagram illustrating a second image signal processing apparatus 10 according to an exemplary embodiment of the present application.
Fig. 3 is a block diagram illustrating a third image signal processing apparatus 10 according to an exemplary embodiment of the present application.
Fig. 4 is a block diagram illustrating a fourth image signal processing apparatus 10 according to an exemplary embodiment of the present application.
Fig. 5 is a block diagram illustrating a fifth image signal processing apparatus 10 according to an exemplary embodiment of the present application.
Fig. 6 is a block diagram illustrating a sixth image signal processing apparatus 10 according to an exemplary embodiment of the present application.
Fig. 7 is a flowchart illustrating an image signal processing method according to an exemplary embodiment of the present application.
FIG. 8 is a block diagram illustrating a camera according to an exemplary embodiment of the present application.
FIG. 9 is a block diagram illustrating a movable platform according to an exemplary embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Aiming at the problems of high hardware cost and long image signal processing time caused by relative independence of each functional module in the traditional ISP, the embodiment of the application provides an image signal processing device 10, wherein the image signal processing device 10 integrates the same or similar operations among the functional modules into a pixel classification module 11, so that the repeated execution process is avoided, excessive hardware resources are not needed, the hardware cost is reduced, and the image signal processing time is shortened; the image signal processing apparatus 10 may be applied to a camera, a mobile terminal (such as a mobile phone, a tablet or a computer), a car camera, a monitor, an educational device or a medical device, and the like, which require processing of a captured image signal.
Referring to fig. 1, a block diagram of a first image signal processing apparatus 10 according to an exemplary embodiment is shown. The device comprises: a pixel classification module 11 and at least two first image signal processing modules 12 (2 first image signal processing modules 12 are illustrated in fig. 1 as an example); wherein, after the pixel classification module 11, at least two first image signal processing modules 12 are connected in sequence.
The pixel classification module 11 is configured to classify each pixel in an input image, generate a plurality of classification information, and transmit the input image and the plurality of classification information to a first image signal processing module 12 connected thereto; the classification information is used to characterize the image characteristics of the pixel.
It should be understood that, in this embodiment, a representation manner of the classification information is not limited in any way, and may be specifically set according to an actual situation, and in an example, the classification information may be represented in a manner of a hash value or a hash vector, where the hash value includes, but is not limited to, a reshaped value, a floating point type value, and the like, and the hash vector includes, but is not limited to, a reshaped vector, a floating point type vector, and the like; in one example, the classification information may be, for example, a sequence of shaping values: 01011010101010 or 1253716581651, etc., each location may represent a certain type of image feature, and the specific value at that location may represent the specific meaning of that type of image feature, such as the image feature including edge features and flat region features, wherein the 1 st location represents an edge feature, wherein the value portion "0" represents an edge and "1" represents no edge; the 2 nd position represents a flat region feature, where a numerical portion "0" represents a flat region, a "1" represents a non-flat region, and if there is classification information of "01", it represents that the image feature of the pixel is: has an edge and a non-flat area.
In the present embodiment, considering that most of the first image signal processing modules 12 in the image signal processing chain need to know the image characteristics of each pixel in the input image to decide to perform the corresponding image processing operation on the pixel, on the basis that the repeated determination of the image features in the first image signal processing module 12 not only lengthens the image signal processing time, but also drives up the hardware cost due to repeated hardware resources, the pixel classification module 11 is provided in the image signal processing apparatus 10 according to the embodiment of the present application, the pixel classification module 11 classifies each pixel in the input image, generates classification information of each pixel, the classification information characterizes the image feature of the pixel, and the embodiment does not limit the image feature, the specific setting can be performed according to an application scene, for example, the image feature may include at least one or more of the following: edge intensity features, edge direction features, flat zone features, texture features, color features, isolated point features, and field features; as an example, for example, the pixels in the image are divided into 3 types, type 1 indicates that the pixel is non-edge, flat, non-textured and non-isolated, and type 2 indicates that the pixel is weak edge and 0 ° direction, non-flat, textured and non-isolated; the 3 types represent that the pixel is a strong edge and 90-degree direction, a non-flat area, a texture and a non-isolated point and the like; in the present embodiment, by classifying the pixels in the pixel classification module 11 and transmitting the generated classification information to the first image signal processing module 12, the first image signal processing module 12 does not need to repeat the same or similar image feature determination process, and can determine the image processing operation to be performed by each pixel based on the several classification information directly.
In a possible implementation manner, the pixel classification module 11 classifies each pixel in an input image based on the pixel and a neighborhood pixel of the pixel, and generates classification information of the pixel, where the classification information is used to characterize an image feature of the pixel, and the image feature is not limited in this embodiment, and may be specifically set according to an actual situation, for example, the image feature may be an edge intensity feature, an edge direction feature, a flatness degree, a texture richness degree, whether there is an isolated point or not, and a neighborhood feature; wherein the neighborhood feature may be an image feature specifying a neighborhood, such as may be an image feature of a small neighborhood (e.g., 3 × 3) and an image feature of a large neighborhood (e.g., 7 × 7); as an example, the pixel classification module 11 may classify the pixels and the gray level change of the pixels in the neighborhood of the pixels, and generate the classification information of the pixels.
In an example, the pixel classification module 11 may calculate gradient matrices in a horizontal direction and a vertical direction of each pixel based on the pixel and pixels in a neighborhood of the pixel, then determine a gradient direction, a magnitude value and a correlation degree of the pixel based on the gradient matrices, and then classify the pixels according to the gradient direction, the magnitude value and the correlation degree to obtain classification information of the pixels, for example, an input image is totally divided into N (N ≧ 1) classes, each class is represented by a different hash value or hash vector, where if the gradient direction of one pixel is t, the magnitude value is s, and the correlation is c, then a sufficient requirement that the pixel block belongs to a kth (1 ≦ K ≦ N) class is given as: tlk≤f(t)<trkAnd sl isk≤f(s)<srkAnd clk≤f(c)<crkWherein, tlk，trk，slk，srk，clk，crkThe preset parameters can be specifically set according to actual conditionsWithout being limited thereto, f (t), f(s), f (c) are functions of gradient direction, amplitude and correlation degree, respectively.
In another possible implementation manner, the pixel classification module 11 classifies each pixel in the input image based on the pixel, the neighboring pixels of the pixel, and specified image information, and generates classification information of the pixel, where the classification information is used to characterize the image feature of the pixel; the image information is not limited in any way in the embodiments of the present application, and may be specifically set according to actual situations, for example, the image information may include but is not limited to exposure parameters such as sensitivity (ISO value).
The first image signal processing module 12 is configured to receive the image and the classification information input by the pixel classification module 11 or the previous first image signal processing module 12; based on the classification information, corresponding image processing operations are performed on pixels in the input image, and the generated image and the classification information are transmitted to the next first image signal processing module 12.
In one example, the first image signal processing module 12 includes, but is not limited to: the system comprises a dead pixel correction module, a black level correction module, a shadow correction module, a white balance correction module, a demosaicing module, a color correction module, a brightness adjustment module, a noise reduction module and a sharpening module; the dead pixel correction module is used for eliminating pixels which are obviously different from the change of surrounding pixel points in the pixel array; the black level correction module is used for subtracting a dark current signal from an input image signal; the shadow correction module is used for compensating the brightness loss of peripheral pixels; the white balance correction module is used for removing the influence of ambient light; the demosaicing module is used for reconstructing complete colors; the color correction module is used for correcting color deviation; the brightness adjusting module is used for adjusting the overall or local brightness; the noise reduction module and the sharpening module are used for restoring relevant details of the image.
The first image signal processing module 12 stores the corresponding relationship between the classification information and the image processing operation, in one example, in one first image processing module, for example, the pixels in the image are divided into 2 types, the 1 st type corresponds to a convolution kernel of 3 × 3, and the 2 nd type corresponds to a convolution kernel of 6 × 6; it should be noted that, correspondence between the classification information and the image processing operation stored in different first image signal processing modules 12 is also different, and specific setting can be performed according to the actual image processing function of the first image signal processing module 12; after receiving the image input by the pixel classification module 11 or the previous first image signal processing module 12 and the classification information, the first image signal processing module 12 obtains the image processing operation corresponding to each pixel in the input image from the corresponding relationship by using the classification information as an index, and executes the image processing operation to generate an image to be sent to the next first image signal processing module 12, and the first image signal processing module 12 sends the classification information to the next first image signal processing module 12 together with the generated image, so that the next first image signal processing module 12 can directly execute the corresponding image processing operation on the input image based on the classification information.
In this embodiment, the classification information is transmitted to the first image signal processing module 12 along with the image in parallel, so that the first image signal processing module 12 does not need to repeat the same or similar steps, and can directly perform corresponding image processing operations on each pixel in the input image based on the classification information, thereby reducing hardware resources for performing the same or similar steps, effectively reducing hardware cost, and reducing the duration of image signal processing due to the deletion of the repeated steps, and improving image signal processing efficiency.
It can be understood that, in the present application, no limitation is imposed on the storage manner of the correspondence, and the storage manner may be specifically set according to actual situations, for example, the correspondence may be stored in a hash table format, where the hash table includes one or more key-value pair relationships, where the classification information is used as a hash key, and the image processing operation is stored as a key-value pair relationship of a hash value; wherein, the key-value-pair relationship may be designed based on artificial experience, or may be obtained through an intelligent learning algorithm, the intelligent learning algorithm may be a machine learning algorithm such as a random forest model, a decision tree model, or the like, or a deep learning algorithm such as a neural network model, or may be other algorithms such as a least square method, in one example, a key-value-pair relationship sample (including a hash key sample and a hash value sample) may be obtained, the hash key sample in the key-value-pair relationship sample is input into a specified model (such as a random forest model), so as to obtain a prediction result of the specified model, according to a difference between the prediction result of the specified model and the hash value sample in the key-value-pair relationship sample, a parameter of the specified model is adjusted, so as to obtain a trained model, so that any hash key may be input into the trained model, and acquiring a corresponding hash value to obtain a key-value-pair relation, thereby effectively reducing the process of manual debugging and improving the development efficiency.
In an embodiment, the different classification information corresponds to different image processing operations, the image processing operations include, but are not limited to, filters with different scales and different types of image processing functions, for example, the first image signal processing module 12 is an image denoising module, the image denoising module achieves the purpose of image smoothing through a low-pass filtering manner, for each pixel in the input image, a corresponding smoothing operator is obtained based on the corresponding classification information, the different classification information corresponds to different smoothing operators, for example, if the classification information indicates that the pixel is in a flat region, the corresponding image processing operation may be an omnidirectional smoothing operator; if the classification information indicates that the pixel is in the texture region, the corresponding image processing operation may be a directional smoothing operator.
It should be noted that, in the embodiment of the present application, there is no limitation on the specific position of the pixel classification module 11 in the image signal processing apparatus 10, and the specific setting may be performed according to actual situations.
In an embodiment, please refer to fig. 2, which is a structural diagram illustrating a second image signal processing apparatus 10 according to an exemplary embodiment of the present application, where the pixel classification module 11 is located at a first position in an image signal processing link of the image signal processing apparatus 10, the pixel classification module 11 is directly connected to an image sensor 20, the image sensor 20 is configured to acquire an image and transmit the image to the pixel classification module 11, so that the pixel classification module 11 can classify each pixel in an input image and generate classification information corresponding to each pixel, where the classification information is used to characterize an image feature of the pixel, so that a subsequent first image signal processing module 12 does not need to repeat the same or similar steps; the image sensor 20 may be a CMOS image sensor 20 or a CCD image sensor 20, and the image sensor 20 converts the captured light source signal into a digital signal to complete the image acquisition.
In another embodiment, please refer to fig. 3, which is a structural diagram of a third image signal processing apparatus 10 according to an exemplary embodiment of the present application (fig. 3 illustrates 2 second image signal processing modules 13 by way of example), the pixel classification module 11 may be disposed behind the second image signal processing module 13 that does not need to determine the image characteristics of the pixels, the image signal processing apparatus 10 further includes one second image signal processing module 13 or a plurality of second image signal processing modules 13 connected in sequence, the pixel classification module 11 is connected to the image sensor 20 through the one or more second image signal processing modules 13 to receive the image transmitted by the image sensor 20 through the one or more second image signal processing modules 13; wherein, the second image signal processing module 13 is configured to perform a corresponding image signal processing operation on the input image.
In one example, the second image signal processing module 13 includes a dead pixel correction module (for removing pixels that show a significant difference from the change of surrounding pixels in the pixel array), a black level correction module (for subtracting a dark current signal from an input image signal), a shading correction module (for compensating for the loss of brightness of surrounding pixels), and a white balance correction module (for removing the influence of ambient light), and the first image signal processing module 12 includes a mosaic module (for reconstructing a complete color), a color correction module (for correcting color deviation), a noise removal module (for restoring details related to an image), and a sharpening module (for restoring details related to an image); the connection order of the image sensor 20 and the modules in the image signal processing apparatus 10 is, in turn: the image sensor 20 → the dead pixel correction module → the black level correction module → the shadow correction module → the white balance correction module → the pixel classification module 11 → the mosaic module → the color correction module → the noise removal module → the sharpening module.
In an embodiment, considering that if the input image is a high-resolution image, some first image signal processing modules 12 have poor processing effect on the high-resolution image, for example, the noise removal module needs to filter the image, and the filter in the noise removal module has a fixed scale size, and if the input image is a high-resolution image, which contains more pixels, the noise removal module may not be able to sufficiently identify the noise in the high-resolution image, so that the image details cannot be restored, and needs to perform down-sampling on the input image, so as to reduce the number of pixels, based on which, the image signal processing apparatus 10 further includes a down-sampling module 14, the down-sampling module 14 is configured to perform down-sampling on the image acquired by the image sensor 20 one or more times, and transmit the generated large-scale image to the pixel classification module 11 connected thereto, the pixel classification module 11 can classify each pixel in the large-scale image to obtain the classification information of each pixel, thereby ensuring that the first image signal processing module 12 which needs to process the large-scale image does not need to repeat the same or similar down-sampling processing step and image feature determination step, not only reducing the image signal processing time, but also reducing the hardware expenditure cost.
In one implementation, please refer to fig. 4, which is a structural diagram illustrating a fourth image signal processing apparatus 10 according to an exemplary embodiment of the present application, where the image sensor 20 transmits the acquired image to the down-sampling module 14 and the pixel classification module 11, the down-sampling module 14 performs one or more down-sampling processes on the input image, and transmits the generated large-scale image to the pixel classification module 11 connected thereto, that is, the pixel classification module 11 may perform a classification process on the image acquired by the image sensor 20 and the large-scale image generated by the down-sampling module 14.
In addition, the image sensor 20 transmits the acquired current image to the down-sampling module 14 and the pixel classification module 11 in real time, considering that a certain processing time is required in the down-sampling processing process, after the pixel classification module 11 classifies pixels of the current image to acquire a plurality of classification information, if it waits for a large-scale image of the current image after the down-sampling processing, and then transmits the large-scale image after the classification processing, this process will consume too much invalid waiting time, resulting in a longer delay of image signal processing, and thus reducing the image signal processing efficiency, in this embodiment, the inventor found that, when the image sensor 20 transmits the current acquired image to the pixel classification module 11, the down-sampling module 14 also completes the down-sampling processing of the last acquired image of the image sensor 20 and transmits the generated large-scale image to the pixel classification module 11 within a specified time range (less than the waiting time), considering that the difference between adjacent images acquired by the image sensor 20 is small, the pixel classification module 11 may classify the image currently acquired by the image sensor 20 and the large-scale image acquired by the image sensor 20 last time and subjected to downsampling respectively, generate a plurality of classification information respectively corresponding to the two images, and transmit the classification information to the first image signal processing module 12 connected thereto, thereby effectively saving the waiting time and improving the image signal processing efficiency.
In another implementation manner, please refer to fig. 5, which illustrates a structure diagram of a fifth image signal processing apparatus 10 according to an exemplary embodiment of the present application, in which the image sensor 20 transmits the acquired image to the down-sampling module 14 and the pixel classification module 11 through one or more second image signal processing modules 13, that is, after the image acquired by the image sensor is processed by each second image signal processing module 13, the image is output to the down-sampling module 14 and the pixel classification module 11 by a last second image signal processing module 13, the down-sampling module 14 performs one or more down-sampling processes on the input image, and transmits the generated large-scale image to the pixel classification module 11 connected thereto; wherein, the image inputted to the pixel classification module 11 includes: the image classification module transmits a plurality of classification information respectively corresponding to the two images to the first image signal processing module 12 connected with the image classification module, so as to be beneficial to shortening the waiting time and improving the image signal processing efficiency.
In an embodiment, considering that there are fewer first image signal processing modules 12 that need a plurality of classification information corresponding to the large-scale image, and if the plurality of classification information corresponding to the large-scale image is transmitted together in an image processing link, excessive transmission resources may be consumed, the image signal processing apparatus 10 may further include a storage module 15, please refer to fig. 6, which shows a structure diagram of a sixth image signal processing apparatus 10 according to an exemplary embodiment of the present application, where the storage module 15 is connected to the pixel classification module 11, and the pixel classification module 11 is further configured to store the plurality of classification information corresponding to the large-scale image in the storage module 15; the first image signal processing module 12 is further configured to obtain the classification information from the storage module 15, and perform a corresponding image processing operation on the input image based on the classification information; in this embodiment, the first image signal processing module 12, which needs to process based on a plurality of classification information corresponding to the large-scale image, may directly obtain the plurality of classification information from the storage module 15, thereby effectively saving transmission resources for transmitting the plurality of classification information corresponding to the large-scale image in the image processing link.
Accordingly, referring to fig. 7, a flowchart of an image signal processing method according to an exemplary embodiment of the present application is shown, wherein the image signal is applied to an image signal processing apparatus, and the image signal processing apparatus includes a pixel classification module and at least two first image signal processing modules; the method comprises the following steps:
in step S101, in the pixel classification module, each pixel in the input image is classified, and a plurality of pieces of classification information are generated; the classification information is used to characterize the image characteristics of the pixel.
In step S102, in the first image signal processing module, the image and the classification information input by the pixel classification module or the previous first image signal processing module are received, based on the classification information, corresponding image processing operations are performed on pixels in the input image, and the generated image and the classification information are transmitted to the next first image signal processing module.
In one embodiment, the classification information is represented by any one of the following ways: a hash value or a hash vector.
In one embodiment, the image features include at least one or more of: edge intensity features, edge direction features, plateau region features, texture features, color features, outlier features, and domain features.
In one embodiment, the step S102 includes: taking the classification information as an index, acquiring image processing operation corresponding to each pixel in the input image from a pre-stored corresponding relation, and executing the image processing operation; the correspondence relationship represents a correspondence relationship of the classification information and the image processing operation.
In an embodiment, the correspondence is stored in a hash table format.
The hash table comprises one or more key value pair relationships; and the classification information is used as a hash key, and the image processing operation is used as the key-value-pair relation of the hash value to be stored.
In an embodiment, the image processing operation includes filters of different scales and different types of image processing functions.
In one embodiment, the step S101 includes: for each pixel in the input image, classifying based on the pixel and the neighborhood pixels of the pixel, and generating the classification information of the pixel.
In an embodiment, the input image is acquired from an image sensor.
In one embodiment, the method further comprises: and carrying out one or more times of downsampling processing on the image acquired by the image sensor to generate a large-scale image.
In one embodiment, the images classified include: the image sensor is used for acquiring the current image and the large-scale image acquired last time and subjected to down-sampling processing.
In one embodiment, the method further comprises: storing a plurality of classification information corresponding to the large-scale image, so that the first image signal processing module acquires the stored plurality of classification information, and performing corresponding image processing operation on the input image based on the plurality of classification information.
In one embodiment, the first image signal processing module comprises at least one or more of: the system comprises a dead pixel correction module, a black level correction module, a shadow correction module, a white balance correction module, a demosaicing module, a color correction module, a brightness adjustment module, a noise reduction module and a sharpening module.
For the method embodiment, since it basically corresponds to the apparatus embodiment, the relevant points may be referred to the partial description of the apparatus embodiment, and are not described herein again.
Accordingly, referring to fig. 8, an embodiment of the present application further provides a camera 100, including:
a housing 30.
The lens assembly 40 is disposed inside the housing 30.
And an image sensor 20 disposed inside the housing 30 for sensing light passing through the lens assembly and generating an electrical signal.
And the image signal processing device 10 for processing the electric signal.
Those skilled in the art will appreciate that fig. 8 is merely an example of the camera 100 and does not constitute a limitation of the camera 100 and may include more or fewer components than illustrated, or combine certain components, or different components, e.g., the camera 100 may also include a network access device, etc.
Accordingly, referring to fig. 9, an embodiment of the present invention further provides a movable platform 001, including:
a machine body 02.
And the power system 03 is arranged in the machine body 02 and used for providing power for the movable platform 001.
And the camera 100 described above.
In an embodiment, the movable platform may be a drone, an unmanned vehicle, or an unmanned ship.
Those skilled in the art will appreciate that fig. 9 is merely an example of a movable platform 001, and does not constitute a limitation of the movable platform 001, and may include more or less components than those shown, or combine certain components, or be different components, for example, the movable platform 001 may also include input-output devices, network access devices, etc.; it is understood that the camera 100 may be fixedly mounted on the movable platform 001, or may be detachably mounted on the movable platform 001, which is not limited in this embodiment.
The method and apparatus provided by the embodiments of the present invention are described in detail above, and the principle and the embodiments of the present invention are explained in detail herein by using specific examples, and the description of the embodiments is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (28)
1. An image signal processing device is characterized by comprising a pixel classification module and at least two first image signal processing modules;
at least two first image signal processing modules are connected in sequence after the pixel classification module;
the pixel classification module is used for classifying each pixel in an input image to generate a plurality of classification information, and transmitting the input image and the classification information to a first image signal processing module connected with the input image and the classification information; the classification information is used for representing the image characteristics of the pixel;
the first image signal processing module is used for receiving the image input by the pixel classification module or the previous first image signal processing module and the classification information; and based on the classification information, executing corresponding image processing operation on pixels in the input image, and transmitting the generated image and the classification information to a next first image signal processing module.
2. The apparatus of claim 1, wherein the classification information is represented by any one of: a hash value or a hash vector.
3. The apparatus according to claim 1, wherein the first image signal processing module stores therein a correspondence relationship between classification information and image processing operation;
the first image signal processing module is further configured to obtain, from the correspondence, an image processing operation corresponding to each pixel in the input image, using the plurality of pieces of classification information as an index.
4. The apparatus of claim 3, wherein the correspondence is stored in a hash table format;
the hash table comprises one or more key value pair relationships; and the classification information is used as a hash key, and the image processing operation is used as the key-value-pair relation of the hash value to be stored.
5. The apparatus of claim 1, wherein the image processing operations comprise different scales of filters and different types of image processing functions.
6. The apparatus of claim 1,
the pixel classification module is specifically configured to: for each pixel in the input image, classifying based on the pixel and the neighborhood pixels of the pixel, and generating the classification information of the pixel.
7. The apparatus of claim 1, wherein the pixel classification module is connected to an image sensor, and the image sensor is configured to capture an image and transmit the image to the pixel classification module.
8. The apparatus according to claim 1, further comprising a second image signal processing module or a plurality of second image signal processing modules connected in sequence;
the pixel classification module is connected with the image sensor through the one or more second image signal processing modules so as to receive the images transmitted by the image sensor through the one or more second image signal processing modules.
9. The apparatus of claim 7 or 8, further comprising the downsampling module;
the down-sampling module is used for carrying out down-sampling processing for one time or more times on the image collected by the image sensor and transmitting the generated large-scale image to the pixel classification module connected with the down-sampling module.
10. The apparatus of claim 9, wherein inputting the image of the pixel classification module comprises: the image sensor is used for acquiring the current image and the large-scale image acquired last time and subjected to down-sampling processing.
11. The apparatus of claim 10, further comprising a storage module;
the pixel classification module is further used for storing a plurality of classification information corresponding to the large-scale image into the storage module;
the first image signal processing module is further configured to obtain the plurality of classification information from the storage module, and perform corresponding image processing operations on the input image based on the plurality of classification information.
12. The apparatus of claim 1, wherein the first image signal processing module comprises at least one or more of:
the system comprises a dead pixel correction module, a black level correction module, a shadow correction module, a white balance correction module, a demosaicing module, a color correction module, a brightness adjustment module, a noise reduction module and a sharpening module.
13. The apparatus of claim 1, wherein the image features comprise at least one or more of:
edge intensity features, edge direction features, plateau region features, texture features, color features, outlier features, and domain features.
14. The image signal processing method is applied to an image signal processing device, and the image signal processing device comprises a pixel classification module and at least two first image signal processing modules; the method comprises the following steps:
in the pixel classification module, classifying each pixel in an input image to generate a plurality of classification information; the classification information is used for representing the image characteristics of the pixel;
and in the first image signal processing module, receiving the image input by the pixel classification module or the previous first image signal processing module and the classification information, executing corresponding image processing operation on the pixels in the input image based on the classification information, and transmitting the generated image and the classification information to the next first image signal processing module.
15. The method of claim 13, wherein the classification information is represented by any one of: a hash value or a hash vector.
16. The method of claim 13, wherein the image features comprise at least one or more of:
edge intensity features, edge direction features, plateau region features, texture features, color features, outlier features, and domain features.
17. The method of claim 13, wherein performing respective image processing operations on pixels in the input image based on the number of classification information comprises:
taking the classification information as an index, acquiring image processing operation corresponding to each pixel in the input image from a pre-stored corresponding relation, and executing the image processing operation; the correspondence relationship represents a correspondence relationship of the classification information and the image processing operation.
18. The method of claim 17, wherein the correspondence is stored in a hash table format;
the hash table comprises one or more key value pair relationships; and the classification information is used as a hash key, and the image processing operation is used as the key-value-pair relation of the hash value to be stored.
19. The method of claim 13, wherein the image processing operations include filters of different scales and different types of image processing functions.
20. The method of claim 13, wherein the classifying each pixel in the input image generates a number of classification information, including:
for each pixel in the input image, classifying based on the pixel and the neighborhood pixels of the pixel, and generating the classification information of the pixel.
21. The method of claim 13, wherein the input image is acquired from an image sensor.
22. The method of claim 21, further comprising:
and carrying out one or more times of downsampling processing on the image acquired by the image sensor to generate a large-scale image.
23. The method of claim 22, wherein classifying the images comprises: the image sensor is used for acquiring the current image and the large-scale image acquired last time and subjected to down-sampling processing.
24. The method of claim 23, further comprising:
storing a plurality of classification information corresponding to the large-scale image, so that the first image signal processing module acquires the stored plurality of classification information, and performing corresponding image processing operation on the input image based on the plurality of classification information.
25. The method of claim 13, wherein the first image signal processing module comprises at least one or more of:
the system comprises a dead pixel correction module, a black level correction module, a shadow correction module, a white balance correction module, a demosaicing module, a color correction module, a brightness adjustment module, a noise reduction module and a sharpening module.
26. A camera, comprising:
a housing;
the lens assembly is arranged inside the shell;
the image sensor is arranged in the shell and used for sensing the light passing through the lens assembly and generating an electric signal; and the number of the first and second groups,
the image signal processing apparatus according to any one of claims 1 to 13.
27. A movable platform, comprising:
a body;
the power system is arranged in the machine body and used for providing power for the movable platform; and the number of the first and second groups,
and a camera as claimed in claim 26.
28. The movable platform of claim 27, wherein the movable platform comprises a drone, an unmanned vehicle, and an unmanned ship.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2019/122084 WO2021102947A1 (en) | 2019-11-29 | 2019-11-29 | Image signal processing apparatus and method, camera, and mobile platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112514364A true CN112514364A (en) | 2021-03-16 |
Family
ID=74953909
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201980050308.8A Pending CN112514364A (en) | 2019-11-29 | 2019-11-29 | Image signal processing apparatus, image signal processing method, camera, and movable platform |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112514364A (en) |
| WO (1) | WO2021102947A1 (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040001632A1 (en) * | 2002-04-25 | 2004-01-01 | Yasushi Adachi | Image processing apparatus, image processing method, program, recording medium, and image forming apparatus having the same |
| US20070070427A1 (en) * | 2005-08-18 | 2007-03-29 | Lexmark International, Inc. | Systems and methods for selective dithering using pixel classification |
| CN103828345A (en) * | 2011-09-29 | 2014-05-28 | 富士胶片株式会社 | Apparatus, method, and program for image processing, storage medium, and image pickup apparatus |
| CN106934812A (en) * | 2015-12-25 | 2017-07-07 | 北京展讯高科通信技术有限公司 | Image-signal processor and its image-signal processing method |
| CN108320263A (en) * | 2017-12-29 | 2018-07-24 | 维沃移动通信有限公司 | A kind of method, device and mobile terminal of image procossing |
| CN109064418A (en) * | 2018-07-11 | 2018-12-21 | 成都信息工程大学 | A kind of Images Corrupted by Non-uniform Noise denoising method based on non-local mean |
| CN109583477A (en) * | 2018-11-05 | 2019-04-05 | 北京达佳互联信息技术有限公司 | A kind of pixel classifications processing method, device and the electronic equipment of image |
| CN109859209A (en) * | 2019-01-08 | 2019-06-07 | 平安科技（深圳）有限公司 | Remote Sensing Image Segmentation, device and storage medium, server |
| CN110147805A (en) * | 2018-07-23 | 2019-08-20 | 腾讯科技（深圳）有限公司 | Image processing method, device, terminal and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8593483B2 (en) * | 2009-10-20 | 2013-11-26 | Apple Inc. | Temporal filtering techniques for image signal processing |
| KR102287043B1 (en) * | 2017-05-22 | 2021-08-06 | 삼성전자주식회사 | Electronic device for processing image acquired by using camera and method for operating thefeof |
| CN107613192A (en) * | 2017-08-09 | 2018-01-19 | 深圳市巨龙创视科技有限公司 | A kind of Digital Image Processing algorithm based on video camera module |
| CN110139109B (en) * | 2018-02-08 | 2023-01-10 | 北京三星通信技术研究有限公司 | Image coding method and corresponding terminal |
-
2019
- 2019-11-29 WO PCT/CN2019/122084 patent/WO2021102947A1/en active Application Filing
- 2019-11-29 CN CN201980050308.8A patent/CN112514364A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040001632A1 (en) * | 2002-04-25 | 2004-01-01 | Yasushi Adachi | Image processing apparatus, image processing method, program, recording medium, and image forming apparatus having the same |
| US20070070427A1 (en) * | 2005-08-18 | 2007-03-29 | Lexmark International, Inc. | Systems and methods for selective dithering using pixel classification |
| CN103828345A (en) * | 2011-09-29 | 2014-05-28 | 富士胶片株式会社 | Apparatus, method, and program for image processing, storage medium, and image pickup apparatus |
| CN106934812A (en) * | 2015-12-25 | 2017-07-07 | 北京展讯高科通信技术有限公司 | Image-signal processor and its image-signal processing method |
| CN108320263A (en) * | 2017-12-29 | 2018-07-24 | 维沃移动通信有限公司 | A kind of method, device and mobile terminal of image procossing |
| CN109064418A (en) * | 2018-07-11 | 2018-12-21 | 成都信息工程大学 | A kind of Images Corrupted by Non-uniform Noise denoising method based on non-local mean |
| CN110147805A (en) * | 2018-07-23 | 2019-08-20 | 腾讯科技（深圳）有限公司 | Image processing method, device, terminal and storage medium |
| CN109583477A (en) * | 2018-11-05 | 2019-04-05 | 北京达佳互联信息技术有限公司 | A kind of pixel classifications processing method, device and the electronic equipment of image |
| CN109859209A (en) * | 2019-01-08 | 2019-06-07 | 平安科技（深圳）有限公司 | Remote Sensing Image Segmentation, device and storage medium, server |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021102947A1 (en) | 2021-06-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102574141B1 (en) | Image display method and device | |
| CN109712102B (en) | Image fusion method and device and image acquisition equipment | |
| CN111698434B (en) | Image processing apparatus, control method thereof, and computer-readable storage medium | |
| US9805452B2 (en) | Image processing apparatus, image processing method, program, and recording medium | |
| CN1985274A (en) | Methods, system and program modules for restoration of color components in an image model | |
| CN108616700B (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| WO2020010638A1 (en) | Method and device for detecting defective pixel in image | |
| CN110855876B (en) | Image processing method, terminal and computer storage medium | |
| CN113379609B (en) | Image processing method, storage medium and terminal equipment | |
| CN115314617A (en) | Image processing system and method, computer readable medium, and electronic device | |
| CN112308785B (en) | Image denoising method, storage medium and terminal equipment | |
| CN110838088B (en) | Multi-frame noise reduction method and device based on deep learning and terminal equipment | |
| CN113273180B (en) | Image processing apparatus and method | |
| CN113781326A (en) | Demosaicing method and device, electronic equipment and storage medium | |
| CN108574796B (en) | Digital camera method and apparatus optimized for computer vision applications | |
| CN107424134B (en) | Image processing method, image processing device, computer-readable storage medium and computer equipment | |
| CN116258653B (en) | Low-light level image enhancement method and system based on deep learning | |
| CN111968039A (en) | Day and night universal image processing method, device and equipment based on silicon sensor camera | |
| CN112514364A (en) | Image signal processing apparatus, image signal processing method, camera, and movable platform | |
| CN112950509B (en) | Image processing method and device and electronic equipment | |
| CN115731115A (en) | Data processing method and device | |
| CN115835034A (en) | White balance processing method and electronic equipment | |
| KR20220001417A (en) | Electronic device and controlling method of electronic device | |
| JP5810778B2 (en) | Image processing apparatus and image processing method | |
| CN112907454A (en) | Method and device for acquiring image, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20221115 |
|
| AD01 | Patent right deemed abandoned |