WO2021161652A1 - Dispositif de traitement de signal, capteur d'image, dispositif d'imagerie et procédé de traitement de signal - Google Patents

Dispositif de traitement de signal, capteur d'image, dispositif d'imagerie et procédé de traitement de signal Download PDF

Info

Publication number
WO2021161652A1
WO2021161652A1 PCT/JP2020/047168 JP2020047168W WO2021161652A1 WO 2021161652 A1 WO2021161652 A1 WO 2021161652A1 JP 2020047168 W JP2020047168 W JP 2020047168W WO 2021161652 A1 WO2021161652 A1 WO 2021161652A1
Authority
WO
WIPO (PCT)
Prior art keywords
convolution
pixel
processing
pixels
unit
Prior art date
Application number
PCT/JP2020/047168
Other languages
English (en)
Japanese (ja)
Inventor
渡部　剛史
孝文 朝原
和幸 奥池
秀 小林
柴山　憲文
Original Assignee
ソニーセミコンダクタソリューションズ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーセミコンダクタソリューションズ株式会社 filed Critical ソニーセミコンダクタソリューションズ株式会社
Publication of WO2021161652A1 publication Critical patent/WO2021161652A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis

Definitions

  • the present technology relates to a signal processing device, an image sensor, an imaging device, and a signal processing method that perform processing related to DNN (Deep Neural Network) based on a received signal output from a pixel.
  • DNN Deep Neural Network
  • An image captured by an image pickup device such as a camera may be subjected to DNN-related processing such as image recognition processing for a subject. Since such image recognition processing requires a large amount of calculation, high-performance parts are required in the CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), etc. used for the processing. In addition, when performing image recognition processing based on the difference output using a so-called DVS (Dynamic Vision Sensor) that outputs the difference from the previous received signal, it is adjusted to the vertical synchronization signal of DVS that can operate at higher speed. In order to perform image recognition processing, it is necessary to provide a higher-performance arithmetic unit.
  • DVS Dynamic Vision Sensor
  • Patent Document 1 detecting features and objects in the field of view of a computing device using a camera requires a huge amount of calculation, and as a result, in a computing device such as a mobile device. There is a risk that the battery life will decrease as the power consumption increases. When the same thing is done with the above-mentioned camera equipped with DVS, such a problem may be further exacerbated.
  • This technology was made in view of the above circumstances, and aims to reduce the amount of calculation for processing related to image recognition processing using DNN.
  • a convolution process is performed on a convolution layer for a pixel identification unit that identifies a pixel whose light reception amount has changed and a pixel whose light reception amount has changed, based on the change amount of the light reception signal indicating the light reception amount.
  • the filter processing that includes a convolution processing unit to be performed and can be executed a plurality of times in the convolution processing, all the pixels included in the processing target area that is a partial area of the input image and has the same size as the filter size are described above.
  • the filter processing is not performed.
  • the amount of change in the received signal can be obtained, for example, based on the signal output from the DVS (Dynamic Vision Sensor).
  • the signal processing device described above may include a management unit that manages the feature map generated by the convolution process. As a result, the processing result of the convolution process is stored until the convolution process is executed again.
  • the convolution processing unit in the signal processing device described above is invariant when the input image input to the convolution layer includes a change pixel affected by the pixel whose light receiving amount has changed and an invariant pixel not affected by the light received.
  • past feature map data may be used. As a result, it is not necessary to update the value of the area to be processed that uses the past data.
  • the management unit in the signal processing device may manage a dirty map indicating whether each pixel of the input image input to the convolution layer is the change pixel or the invariant pixel.
  • the dirty map shows the distinction between changeable pixels and invariant pixels for each pixel in the input image. That is, when a certain pixel is a changing pixel, it is necessary to re-execute the filter processing in which the pixel is included in the processing target area.
  • the convolution processing unit in the above-mentioned signal processing device may perform all the filter processing in the convolution layer when the feature map is not stored. As a result, in the initial processing, the filtering processing is executed for the entire input image in each convolution layer.
  • the convolution processing unit in the signal processing device may perform convolution processing in a plurality of convolution layers, and the management unit may manage the dirty map for each input image for each of the plurality of convolution layers. As a result, it is possible to perform a process of determining whether or not a change pixel is included in the process target area in any of the convolution layers.
  • the management unit in the signal processing device described above may clear the dirty map of the input image that has completed the convolution process. For example, when the update of a part of the feature map that should be updated by the change pixel in the input image is completed, the part of the feature map is updated to the latest information. In such a case, clear the dirty map.
  • the convolution processing unit in the signal processing device described above does not have to perform the convolution processing when the number of pixels whose light receiving amount has changed is equal to or less than the threshold value.
  • the results of the classification processing in the fully connected layer and the output layer may not change even if the feature map is updated by performing the convolution processing or the pooling processing.
  • the signal processing device described above includes a management unit that manages the cumulative number of pixels that accumulates the number of pixels whose light reception amount has changed, and when the cumulative number of pixels exceeds a threshold value, the convolution processing unit performs convolution processing. Then, the management unit may reset the cumulative number of pixels. By managing the cumulative number of pixels, it is possible to appropriately perform the determination process of whether or not to execute the convolution process or the pooling process.
  • the signal processing device described above may include a threshold value adjusting unit that performs a threshold value adjusting process for adjusting the threshold value for the cumulative number of pixels based on the processing result of the classification process after the convolution process. If the convolution process or pooling process is executed too frequently, the processing load will increase. Further, if the execution frequency is too low, the classification process may not be performed correctly.
  • the threshold value adjusting unit in the signal processing device described above classifies the classification result of the classification process after the convolution process executed when the cumulative number of pixels exceeds the threshold value and the classification result of the classification process executed immediately before the convolution process.
  • the threshold value for the cumulative number of pixels may be increased in the threshold value adjustment process. If the classification result is unchanged as a result of executing the convolution process, the pooling process, and the classification process, the execution frequency of each process may be too high. Moreover, the high execution frequency may be due to the low threshold value.
  • the threshold value adjusting unit in the signal processing device described above classifies the classification result of the classification process after the convolution process executed when the cumulative number of pixels exceeds the threshold value and the classification result of the classification process executed immediately before the convolution process.
  • the threshold value for the cumulative number of pixels may be lowered in the threshold value adjustment process.
  • the execution frequency of each process may be too low. In addition, the low execution frequency may be due to the high threshold value.
  • the threshold adjusting unit in the above-mentioned signal processing device sets the threshold again. It is not necessary to perform the convolution process due to the excess.
  • the cumulative number of pixels which is the number of pixels whose light reception amount has changed, exceeds the threshold value
  • the cumulative number of pixels is reset and the convolution process, pooling process, and classification process are executed. If the cumulative number of pixels exceeds the threshold value again during the execution of each of these processes and the execution request of each process is received, a time lag occurs between the execution request and the process. If this is piled up, each process will not be executed at an appropriate timing.
  • the image pickup apparatus includes a light receiving unit that obtains a light receiving signal indicating the light receiving amount, a pixel specifying unit that specifies a pixel whose light receiving amount has changed based on the change amount of the light receiving signal, and the light receiving portion in which the light receiving amount has changed.
  • the filter processing that includes a convolution processing unit that performs convolution processing for pixels in the convolution layer and can be executed a plurality of times in the convolution processing
  • the processing target is a partial area of the input image and has the same size as the filter size.
  • the image pickup apparatus described above may include an image sensor in which the light receiving unit, the pixel identification unit, and the convolution processing unit are integrally formed.
  • the image sensor includes a light receiving unit that obtains a light receiving signal indicating the light receiving amount, a pixel specifying unit that specifies a pixel whose light receiving amount has changed based on the change amount of the light receiving signal, and the light receiving portion in which the light receiving amount has changed.
  • the filter processing that includes a convolution processing unit that performs the convolution processing for pixels in the convolution layer and can be executed a plurality of times in the convolution processing
  • the processing target is a partial area of the input image and has the same size as the filter size.
  • the pixel in which the received light amount has changed is specified based on the change amount of the received light signal indicating the received light amount, and the convolution process is performed on the convolution layer for the pixel in which the received light amount has changed, and the convolution is performed.
  • the filter processing that can be executed a plurality of times in the processing, all the pixels included in the processing target area which is a partial area of the input image and has the same size as the filter size are not affected by the pixel whose light receiving amount has changed. When it is an invariant pixel, the filter processing is not performed. Even by such a signal processing method, the same operation as that of the signal processing device according to the present technology can be obtained.
  • Imaging device configuration> ⁇ 2.
  • Functional configuration> ⁇ 3.
  • Image recognition processing example> ⁇ 4.
  • Dirty Map> ⁇ 5.
  • Threshold adjustment process> ⁇ 6.
  • Flowchart> ⁇ 7.
  • Modification example> ⁇ 8. Summary> ⁇ 9. This technology>
  • the configuration of the image pickup apparatus 1 according to the first embodiment will be described with reference to FIG.
  • the image pickup apparatus 1 includes an image pickup lens 2, an image sensor 3, a control unit 4, and a recording unit 5.
  • the image pickup device 1 is assumed to have various forms such as a camera mounted on an industrial robot, an in-vehicle camera, and a surveillance camera.
  • the image pickup lens 2 collects the incident light and guides it to the image sensor 3.
  • the image pickup lens 2 may be composed of a plurality of lenses.
  • the image sensor 3 is configured to include a plurality of light receiving elements, and outputs a signal obtained by photoelectric conversion.
  • the control unit 4 controls the shutter speed of the image sensor 3, gives instructions for various signal processing in each unit of the image pickup device 1, captures and records operations according to the user's operation, reproduces the recorded image file, and captures a lens.
  • 2 drive control for example, zoom control, focus control, aperture control, etc.
  • user interface control and the like are performed.
  • the recording unit 5 stores information and the like used for processing by the control unit 4.
  • the recording unit 5 for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like are comprehensively shown.
  • the recording unit 5 may be a memory area built in the microcomputer chip as the control unit 4, or may be configured by a separate memory chip.
  • the control unit 4 controls the entire image pickup apparatus 1 by executing a program stored in the ROM, flash memory, or the like of the recording unit 5.
  • the image sensor 3 will be specifically described. As shown in FIG. 2, the image sensor 3 includes a pixel array unit 11, an arbiter 12, a reading unit 13, a signal processing unit 14, a memory unit 15, and an output unit 16 that function as a so-called DVS.
  • the pixel array unit 11 is formed by arranging the pixels 17 in a two-dimensional array in the row direction (horizontal direction) and the column direction (vertical direction). Each pixel 17 detects the presence or absence of an event depending on whether or not the amount of change in the amount of received light exceeds a predetermined threshold value Th, and outputs a request to the arbiter 12 when the event occurs.
  • the arbiter 12 arbitrates the request from each pixel 17 and controls the reading operation by the reading unit 13.
  • the reading unit 13 performs a reading operation for each pixel 17 of the pixel array unit 11 based on the control of the arbiter 12.
  • the read operation is executed at a timing corresponding to, for example, the vertical synchronization signal XVS.
  • Each pixel 17 outputs a signal based on the difference between the reference level and the current received signal level according to the reading operation by the reading unit 13.
  • the signal read from each pixel 17 is stored in the memory unit 15 as a difference signal.
  • the pixel 17 in which the amount of change in the amount of received light exceeds a predetermined threshold Th will be referred to as a "change pixel”, and the other pixels 17 will be referred to as an "invariant pixel”.
  • the pixel 17 resets the reference level to the current level of the received light signal according to the output of the difference signal. This makes it possible to detect the amount of change in the amount of received light with respect to the reference level according to the next vertical synchronization signal XVS.
  • the difference signal is not read out and the reference level is not reset until the amount of change in the amount of received light exceeds a predetermined threshold value Th.
  • a difference signal corresponding to the amount of change in the integrated received light amount is output from the pixel 17. Further, in the pixel array unit 11, since the reading operation is executed only for the pixels 17 (that is, the changing pixels) in which the amount of change in the amount of received light is larger than the threshold Th, the time required for the reading operation can be shortened. , High-speed drive (for example, 2000 fps, etc.) is possible.
  • the signal processing unit 14 executes various signal processing, image recognition processing by DNN, and the like.
  • image recognition processing by CNN Convolutional Neural Network
  • DNN Deep Neural Network
  • the signal processing unit 14 updates all pixel data stored in the memory unit 15.
  • the all-pixel data is data based on the latest light-receiving amount of all the pixels 17 included in the pixel array unit 11.
  • the signal processing unit 14 adds the data of the difference signal to the data of all pixels in response to the new difference signal stored in the memory unit 15 from each pixel 17, and updates the data of all pixels.
  • the signal processing unit 14 can manage the latest received light amount data for all the pixels 17.
  • the signal processing unit 14 performs a process of determining whether or not the number of changing pixels is the threshold Th2 or more. When the number of changing pixels is the threshold Th2 or more, the signal processing unit 14 executes various processes for performing image recognition by CNN. In the following description, each process executed by the signal processing unit 14 in order to perform image recognition by CNN will be simply referred to as "image recognition process”.
  • the signal processing unit 14 is input with the coordinate information of the changing pixel from the reading unit 13 as information used for the image recognition processing.
  • the information input from the reading unit 13 to the signal processing unit 14 may be only information indicating the vertical position and the horizontal position of the changing pixel (that is, coordinate information), or the difference signal and the coordinate information. And may be included.
  • the number of changing pixels is integrated until the threshold Th2 is exceeded.
  • the number of changing pixels is "3".
  • the number of changing pixels "3" is equal to or less than the threshold Th2.
  • the integrated value of the number of changing pixels is one except for the overlapping (1,1). This is an increase of 1, resulting in a total of "4".
  • the signal processing unit 14 performs, for example, a convolution process by the convolution layer, a max pooling process by the pooling layer, a classification process by the fully connected layer and the output layer, and the like. The specific processing will be described later.
  • the signal processing unit 14 manages various types of information using the memory unit 15 in order to perform image recognition processing. Specifically, the signal processing unit 14 stores and manages the feature map as the processing result in the convolution layer and the pooling layer and the dirty map for each input image in the memory unit 15.
  • the dirty map is a map for determining whether or not the convolution process in the convolution layer and the max pooling process in the pooling layer need to be executed.
  • the dirty map is the same size as the input image, and the flag information for each pixel is shown so that the pixels that need to be processed can be known. Specifically, it will be described later.
  • the "pixel” here is a pixel in the input image for each layer of the CNN, and does not necessarily mean a pixel included in the pixel array unit 11. Also in the following description, the “pixel” may refer to the pixel 17 constituting the pixel array unit 11 or one pixel in the input image. One pixel in the input image may be a value calculated by the convolution process or a value calculated by the max pooling process.
  • the memory unit 15 is composed of a ROM, a RAM, or the like, and as described above, all pixel data, which is data based on the latest light received amount of all the pixels 17 included in the pixel array unit 11, a dirty map, and a CNN convolution.
  • the feature map and the like as the processing result in the layer and the pooling layer are stored.
  • the output unit 16 outputs the classification result by CNN to the control unit 4 in the subsequent stage based on a predetermined interface standard (for example, MIPI (Mobile Industry Processor Interface)).
  • a predetermined interface standard for example, MIPI (Mobile Industry Processor Interface)
  • the control unit 4 receives the classification result by CNN and uses it for various processes.
  • the functional configuration of the signal processing unit 14 will be described with reference to FIG.
  • the signal processing unit 14 includes a pixel identification unit 21, an image recognition processing unit 22, a threshold value adjusting unit 23, and a management unit 24.
  • the pixel specifying unit 21 specifies the position of the changing pixel based on the coordinate information of the changing pixel input from the reading unit 13.
  • the image recognition processing unit 22 includes, for example, convolution processing by a convolution layer, pooling processing by a pooling layer, binding processing by a fully connected layer, processing related to a drop function described later in the fully connected layer, and an output layer as image recognition processing. Performs various processing such as classification processing using the softmax function in.
  • the threshold value adjusting unit 23 adjusts the threshold value Th2 as a determination threshold value for determining whether or not to perform the image recognition processing by the image recognition processing unit 22.
  • the threshold value Th2 is a threshold value for determining the number of changing pixels.
  • the image recognition process is performed by increasing the threshold Th2 until the integrated value of the number of changed pixels becomes larger. Suppresses the execution of each process as.
  • the threshold value adjusting unit 23 lowers the threshold value Th2 so that each process as the image recognition process is executed at a stage where the integrated value of the number of changed pixels is smaller. ..
  • the management unit 24 manages the above-mentioned feature map, dirty map, and the like by using the memory unit 15. In addition, the management unit 24 manages the integrated value of the number of changed pixels (cumulative number of pixels for the changed pixel). In the management of the integrated value of the number of changing pixels, addition processing and reset processing of the integrated value are performed.
  • the signal processing unit 14 can use VGGish, AlexNet, GoogleNet, LSTM, RNN and the like.
  • FIG. 5 shows an example using AlexNet.
  • a plurality of rectangular boxes are arranged at a distance, and one box represents one layer in CNN (AlexNet).
  • CNN image recognition processing is advanced by performing predetermined processing in each layer.
  • the process proceeds from the “all pixel data” to the “output layer”.
  • the thickness of each box roughly represents the number of layers (number of images), and the length in the height direction (direction perpendicular to the processing progress direction) is the vertical and horizontal directions of the image.
  • the number of pixels (that is, the image size) of is roughly represented.
  • all pixel data is an image of R, G, and B, and one image is an image consisting of 224 pixels in each of the vertical and horizontal directions.
  • the number of layers of all pixel data is set to "3".
  • the three images as all pixel data are input images for the first convolution layer.
  • the first convolution layer performs convolution processing using a plurality of filters on all pixel data, and outputs data in which, for example, the number of layers is 96 and the image size is 55 pixels both vertically and horizontally.
  • the first pooling layer performs max pooling processing using the output data from the first convolution layer as an input image, and outputs data having, for example, 96 layers and 27 pixels in both vertical and horizontal directions.
  • the second convolution layer performs a convolution process using the output data from the first pooling layer as an input image, and outputs data in which, for example, the number of layers is 256 and the image size is 27 pixels both vertically and horizontally.
  • the second pooling layer performs max pooling processing using the output data from the second convolution layer as an input image, and outputs data in which, for example, the number of layers is 256 and the image size is 13 pixels both vertically and horizontally.
  • the third convolution layer performs convolution processing using the output data from the second convolution layer as an input image, and outputs data in which, for example, the number of layers is 384 and the image size is 13 pixels both vertically and horizontally.
  • the 4th convolution layer performs convolution processing using the output data from the 3rd convolution layer as an input image, and outputs data in which, for example, the number of layers is 384 and the image size is 13 pixels both vertically and horizontally.
  • the 5th convolution layer performs convolution processing using the output data from the 4th convolution layer as an input image, and outputs data in which, for example, the number of layers is 256 and the image size is 13 pixels both vertically and horizontally.
  • the third pooling layer performs max pooling processing using the output data from the fifth convolution layer as an input image, and outputs data in which, for example, the number of layers is 256 and the image size is 6 pixels both vertically and horizontally.
  • the first fully connected layer performs a binding process using the output data from the third pooling layer, and outputs 4096 pieces of data.
  • the second fully connected layer performs a binding process using the output data from the first fully connected layer, and outputs 4096 pieces of data.
  • the first fully connected layer and the second fully connected layer are equipped with a dropout function to prevent overfitting.
  • the dropout function is a function that randomly invalidates a part of the input data.
  • the output layer performs classification processing using a softmax function using the output data from the second fully connected layer, and outputs, for example, the likelihood of each of 1000 labels as the final output.
  • image recognition of the subject of all pixel data is executed.
  • the input image input to each of the folding layer and the pooling layer described above corresponds to the input image as referred to in the claim. That is, the all-pixel data is the input image according to the claim for the first convolution layer, which is the first layer, and the feature map output from each layer is the input image according to the claim for the next layer. It is said that.
  • the signal processing unit 14 in the present technology is characterized in that the amount of calculation related to the convolution process in the convolution layer and the max pooling process in the pooling layer is reduced. Therefore, the signal processing unit 14 uses a dirty map.
  • FIG. 6 shows an example of all pixel data input to the first convolution layer and an example of a dirty map corresponding thereto.
  • All pixel data consists of three pixel data (images): pixel data (R image) for red (R), pixel data (G image) for green (G), and pixel data (B image) for blue (B). ) Consists of.
  • Dirty maps are prepared for each image.
  • An example of a dirty map corresponding to one input image (for example, R image) is shown in FIGS. 7 and 8.
  • FIG. 7 shows one of all pixel data which is an input image for the first convolution layer.
  • the pixel array unit 11 outputs data about the changing pixels. That is, pixels (changed pixels) in which the amount of change in the amount of received light exceeds a predetermined threshold value Th can be identified.
  • the changing pixel is shown so that it can be seen by surrounding the changing pixel with a line.
  • FIG. 8 shows a dirty map corresponding to the input image. As shown in the figure, the information of "1" is stored in the position corresponding to the changing pixel, and the information of "0" is stored in the position corresponding to the invariant pixel.
  • the dirty map for all pixel data input to the first convolution layer which is the first layer of CNN, has flag information arranged on the map so that variable pixels and invariant pixels can be distinguished. ..
  • a dirty map corresponding to the output data of the first convolution layer will be described.
  • AlexNet a convolution process using a filter having a filter size of 11 ⁇ 11 is performed, but here, for simplification of the explanation, the filter size is set to 3 ⁇ 3 and the convolution in the first convolution layer is performed.
  • the stride is originally 4 pixels, but here, it will be described as 1 pixel.
  • the convolution process is to perform a filter process for a partial area having the same size as the filter size in the input image a plurality of times while sliding the partial area.
  • a partial area having the same size as the filter size in the input image is described as a “processing target area”.
  • the area surrounded by the thick line in FIG. 9 is an example of the processing target area.
  • the processing target area A shown in FIG. 9 includes both changeable pixels and invariant pixels. Although it depends on the type of the filter, since the processing target area A includes the changing pixels, it is highly possible that the result of the filtering processing before the change of the received signal and the result of the filtering processing after the change are different. Therefore, in this example, in the convolution processing of the first convolution layer, if the processing target area includes change pixels, the filter processing is performed.
  • the processing target area B shown in FIG. 10 does not include changing pixels and is limited to invariant pixels. Since the change pixel is not included in the processing target area A, the result of the filter processing before the change of the received light signal and the result of the filter processing after the change are the same. Therefore, it is not necessary to execute the filter processing for the processing target area B.
  • the dirty map is used to determine whether or not the processing target area includes changing pixels.
  • FIG. 11 shows a pixel whose value has been updated by the convolution process in the first convolution layer as “1” and a pixel whose value has not been updated as “0”.
  • the data shown in FIG. 11 can be said to be a map of update flags indicating whether or not the value has been updated by the convolution process in the first convolution layer.
  • the output value A obtained as a result of executing the filter processing for the processing target area A has been updated, so the update flag is set to "1".
  • the processing target area B is not filtered and the value is not updated. Therefore, the output value B that should have been obtained when the filter processing is executed for the processing target area B is left as the old value before the update, and the update flag is set to "0".
  • the map shown in FIG. 11 is a dirty map corresponding to one output data of the first convolution layer. Further, since the output data of the first convolution layer is also the input data of the first pooling layer in the subsequent stage, the dirty map of FIG. 11 can be rephrased as a dirty map corresponding to the input image of the first pooling layer.
  • each layer in the latter stage multiple dirty maps are generated according to the input image.
  • the dirty map generated in each layer is stored in the memory unit 15 and updated every time each process is executed.
  • the latest value for each pixel of the input image of each layer may not exist.
  • the filtering processing cannot be performed unless the value of the pixel whose dirty map value is “0” is used.
  • the pixel whose dirty map is set to "0" is an invariant pixel, so that the latest difference signal is not output.
  • old information stored in the memory unit 15 is used. As a result, even if only the difference signal is output from the pixel array unit 11, the convolution process or the like can be appropriately performed.
  • Old filtering results may be used.
  • the max pooling process for the input image of FIG. 11 is performed in the first pooling layer, the pixels whose values in the dirty map are set to "0" (for example, by the convolution process by the first convolution layer of the previous layer).
  • the filter processing result that is not updated and remains old is used.
  • the feature map for each layer obtained as a result of the convolution process and the max pooling process executed in each layer is stored in the memory unit 15.
  • the feature map stored in the memory unit 15 the processing of each layer can be appropriately executed.
  • Threshold adjustment process The threshold value adjustment process executed by the threshold value adjustment unit 23 of the signal processing unit 14 will be described. In the present embodiment, the execution of the image recognition process is reserved until the integrated value of the number of changed pixels exceeds a predetermined threshold Th2, and the image recognition process is performed when the integrated value of the number of changed pixels exceeds the predetermined threshold Th2. (See Fig. 3).
  • the threshold Th2 is adjusted. Specifically, when the threshold Th2 is increased, the execution frequency of the image recognition process decreases. Further, when the threshold value Th2 is lowered, the execution frequency of the image recognition process increases.
  • the processing frequency of image recognition processing in a situation where a landscape with little movement is being imaged and the processing frequency of image recognition processing in a situation where a sport with a lot of movement is being imaged should be different.
  • the threshold value adjustment process adjusts the threshold value Th2 to be changed according to such a difference in shooting conditions, and the threshold value Th2 can be automatically adjusted to the optimum value.
  • the threshold value Th2 is adjusted based on the image recognition result output from the output layer. For example, if the current image recognition result is significantly different from the previous result, the threshold Th2 is lowered, and if the change is small, the threshold Th2 is raised.
  • the processing frequency of the image recognition process can be optimized.
  • the execution frequency of the image recognition process may be too low.
  • the pixel array unit 11 outputs a difference signal at a high speed such as 2000 fps, it is assumed that the same subject appears in the same area in a certain input image and the input image 0.5 ms later. If the same subject cannot be detected in the same area as a result of reducing the execution frequency of the image recognition process in such a situation, it can be determined that the execution frequency of the image recognition process has been reduced too much.
  • the threshold Th2 is lowered in order to increase the execution frequency of the image recognition process. ..
  • the threshold Th2 is continuously adjusted by continuously performing such processing, there is a possibility that the label output last time and the label output this time may be different from the same case frequently. There is. Originally, in an input image for a certain period of time, the same subject is often shown in the same area, and it may be desirable that the same label is output for a certain period of time as an image recognition result.
  • the threshold Th2 may be lowered (for example, 20% lower than the current threshold Th2). Further, by stopping the subsequent adjustment of the threshold value Th2 for a certain period of time (for example, 10 minutes), it is possible to prevent the threshold value Th2 from becoming low again.
  • FIG. 12 shows an example of a flowchart for the signal processing unit 14 to execute various processes.
  • the signal processing unit 14 performs a process of clearing the dirty map. This process is, for example, a process of deleting the dirty map stored in the memory unit 15.
  • step S101 is a process executed when the image pickup is started by the image pickup device 1 immediately after the power is turned on, or when the image pickup is restarted by the image pickup device 1 immediately after returning from the sleep mode. It is considered that the dirty map stored at this point has nothing to do with the subject of the imaging operation to be performed in the future. Therefore, the signal processing unit 14 deletes the dirty map in step S101.
  • the feature map may be further deleted. Further, the dirty map and the feature map stored in the memory unit 15 may be deleted when the power of the image pickup apparatus 1 is turned off.
  • step S102 the signal processing unit 14 waits for an image input.
  • the image input refers to the input of a light receiving signal (difference signal) obtained by photoelectric conversion in the pixel array unit 11.
  • the reading unit 13 reads the difference signal from each pixel 17, but since there is no (or is not stored) the immediately preceding received signal as a comparison target for calculating the difference, the difference signal is received by each pixel 17. It becomes the received light signal itself that is output according to the amount.
  • the signal processing unit 14 finishes the process of step S102 and proceeds to step S103.
  • step S103 the signal processing unit 14 executes image recognition processing for all pixel data. It can be said that the processing of each layer executes each processing because the feature map as the processing result is not stored in the memory unit 15. As a result, a dirty map is generated in each layer for which image recognition processing is performed, and a feature map is generated. These pieces of information are stored in the memory unit 15. In the dirty map used in the subsequent processing, all the flags may be rewritten to 0 by the clear processing. By clearing the values of all the pixels of the dirty map to "0", it is possible to prevent unnecessary processing from being executed by mistake.
  • the all-pixel data referred to here refers to image information input to the first layer (for example, the first convolution layer) of the CNN, and does not necessarily refer to all the pixels 17 included in the pixel array unit 11. It does not have to be. That is, it may be an image obtained by a light receiving signal of a part of pixels 17 of the pixel array unit 11.
  • the signal processing unit 14 waits for the output of the difference signal in step S104. When the difference signal is output, the signal processing unit 14 proceeds to the process of step S105.
  • the signal processing unit 14 refers to the dirty map for all pixel data in step S105. In other words, it refers to a dirty map for the input image that is input to the first layer of the CNN.
  • step S106 the signal processing unit 14 determines whether or not the integrated value of the number of changed pixels in the referenced dirty map exceeds the threshold Th2.
  • the signal processing unit 14 waits for the output of the next difference signal in step S104.
  • the signal processing unit 14 executes the image recognition process in step S107. Specifically, the minimum necessary filtering process is performed as described above while referring to the dirty map and the already generated feature map in each layer of the CNN.
  • step S107 the dirty map and the feature map stored in the memory unit 15 are updated.
  • step S108 the signal processing unit 14 performs a process of clearing the integrated value of the number of changed pixels. This process may be performed before step S107.
  • step S109 the signal processing unit 14 determines whether or not the current result and the previous result of the image recognition process are different than expected. For example, as described above, it is determined whether or not the labels given to the subjects are different.
  • the signal processing unit 14 performs a process of lowering the threshold Th2 in step S110. On the other hand, when it is determined that the difference is less than expected, the signal processing unit 14 performs a process of raising the threshold Th2 in step S111.
  • FIGS. 1 and 2 are examples of an image pickup apparatus in which a signal processing unit 14 is integrally provided with an image sensor 3.
  • a signal processing unit 14 is integrally provided with an image sensor 3.
  • the pixel array unit 11 or the like is arranged on the front surface, and the GPU or DSP as the signal processing unit 14 is formed on the back surface.
  • the image sensor 3 does not have to include the signal processing unit 14. That is, the image sensor and the signal processing unit 14 may be provided as separate bodies.
  • the configuration in which the image sensor 3 includes the arbiter 12 and only the difference signal for the necessary pixels 17 is read by the reading unit 13, but the reading unit 13 includes all of the pixel array unit 11.
  • the received signal level may be read out from the pixel 17 of the above based on the vertical synchronization signal XVS. Further, in that case, the reading unit 13 outputs only the difference signal for some of the pixels 17 recognized as changing pixels from the difference signals for all the read pixels 17 to the signal processing unit 14 in the subsequent stage. You may.
  • the most significant bit (MSB: Most Significant Bit) in the read signal may be a flag indicating whether or not the amount of change in the amount of received light exceeds a predetermined threshold Th.
  • the reading unit 13 may confirm the most significant bit of the signal read from the pixel 17 and output only the signal for the changing pixel to the signal processing unit 14 in the subsequent stage. Alternatively, the reading unit 13 may output the difference signal for all the read pixels 17, including the flag indicating whether or not the pixel is a changing pixel, to the signal processing unit 14. The signal processing unit 14 determines whether or not the difference signal is for the changing pixel by referring to the flag. In addition, the reading unit 13 reads the difference signal from all the pixels 17 of the pixel array unit 11 and determines whether or not the amount of change in the received light amount exceeds the threshold Th, and if it exceeds the threshold value Th, the MSB is displayed. The process of changing from 0 to 1 may be performed. In this case, the difference signal is output to the signal processing unit 14 after performing the operation on the MSB.
  • the above-mentioned example of the dirty map is provided corresponding to the input image for each layer of the CNN, but may be provided corresponding to the output image.
  • the pixel (one pixel) to be processed in the output image is set to "1" as a flag
  • the pixel is processed (filtered) and "0" is set as the flag. If so, the processing for the pixel may not be performed.
  • the size of the dirty map can be reduced and the memory unit 15 can be effectively used. Further, it is possible to reduce the memory size of the memory unit 15, and it is possible to reduce the component cost and the component size.
  • the execution request of the next image recognition process may be made during the execution of the image recognition process. Even if the image recognition processing unit 22 of the signal processing unit 14 receives a request for the next image recognition processing during the execution of the image recognition processing unit, the image recognition processing unit 22 may ignore the request and continue the executing image recognition processing. good. Further, the request for the image recognition process received during the process may be discarded. As a result, it is possible to prevent the processing timing of the image recognition process from being deviated from the request.
  • the image recognition process may not be executed when the number of changing pixels calculated each time does not exceed the threshold value without using the integrated value of the number of changing pixels.
  • the number of changing pixels is calculated each time based on the output from the pixel array unit 11. If the number of changing pixels does not exceed the threshold value, the image recognition process is not executed and the number of changing pixels is not carried over.
  • the threshold value used in this case may be the same numerical value as the threshold value Th2, or may be a value smaller than the threshold value Th2 because the number of changed pixels is not integrated.
  • the pixel identification unit 21 that identifies the pixel 17 in which the light reception amount has changed and the pixel 17 in which the light reception amount has changed are based on the change amount of the light reception signal indicating the light reception amount.
  • the filter processing which is provided with a convolution processing unit (image recognition processing unit 22) that performs the convolution processing of When all the pixels included in the processing target area are invariant pixels that are not affected by the pixels whose light receiving amount has changed, the filter processing is not performed.
  • the amount of change in the received signal can be obtained, for example, based on the signal output from the DVS.
  • the amount of calculation required for image analysis can be reduced and the processing required for the calculation can be performed. It is possible to reduce the time and processing load. Further, the result of the filter processing may not change depending on the processing target area. Execution is avoided for such filtering. Therefore, the amount of calculation for the convolution process can be reduced, and the processing load of the signal processing device can be reduced. Further, the power consumption can be reduced by reducing the amount of calculation, and the life of the battery included in the signal processing device can be improved. In particular, it is suitable when the signal processing device is mounted on a portable device such as a digital camera or a smartphone.
  • the management unit 24 that manages the feature map generated by the convolution process may be provided. As a result, the processing result of the convolution process is stored until the convolution process is executed again. Therefore, the processing result of the convolution processing can be reused.
  • the convolution processing unit is a change pixel affected by the pixel whose light receiving amount has changed in the input image input to the convolution layer.
  • the data of the past feature map may be used for the invariant pixels.
  • the filter processing for the processing target area can be appropriately executed.
  • each pixel of the input image input to the convolution layer is a change pixel.
  • the dirty map shows the distinction between changeable pixels and invariant pixels for each pixel in the input image. That is, when a certain pixel is a changing pixel, it is necessary to re-execute the filter processing in which the pixel is included in the processing target area.
  • a dirty map is used as an index for determining whether or not to perform filter processing for each processing target area.
  • the convolution process can be efficiently performed by deciding whether or not to perform the filter process by referring to the dirty map.
  • the convolution processing unit may perform the filter processing in the convolution layer when the feature map is not stored.
  • the filtering processing is executed for the entire input image in each convolution layer. Therefore, each process (convolution process and pooling process) for the first time can be appropriately performed.
  • the convolution processing unit (image recognition processing unit 22) performs convolution processing in a plurality of convolution layers, and the management unit 24 creates a dirty map for each input image for each of the plurality of convolution layers. You may manage it. As a result, it is possible to perform a process of determining whether or not a change pixel is included in the process target area in any of the convolution layers. Therefore, it is possible to improve the efficiency of the convolution process for each convolution layer. Of course, even if only the dirty map for the input image input to the first layer of CNN is managed, the efficiency of the convolution process can be improved, but the dirty map is managed for each input image for all layers. Thereby, the efficiency of the convolution process can be further promoted.
  • the management unit 24 may clear the dirty map of the input image that has completed the convolution process. For example, when the update of a part of the feature map that should be updated by the change pixel in the input image is completed, the part of the feature map is updated to the latest information. In such a case, clear the dirty map. As a result, it is possible to prevent the flag indicating the changing pixel from being left uncleared, prevent the feature map from being updated many times unnecessarily, and reduce the processing load. Can be done.
  • the convolution processing unit performs the convolution processing when the number of pixels whose light receiving amount has changed is equal to or less than the threshold value (for example, the threshold value Th2). It does not have to be done.
  • the threshold value for example, the threshold value Th2
  • the results of the classification processing in the fully connected layer and the output layer may not change even if the feature map is updated by performing the convolution processing or the pooling processing. In such a case, it is preferable to omit the convolution process and the pooling process.
  • the processing load can be reduced.
  • the input signal to the signal processing device is only the signal for the pixel whose light receiving amount has changed
  • the input signal has a high speed such as 2000 fps.
  • a high-performance GPU or DSP is required, which may result in high cost.
  • the processing will not be in time in the first place.
  • the execution frequency of each process can be suppressed, and the performance is low and the GPU or DSP is inexpensive. Can be used. This can contribute to the reduction of component costs.
  • the management unit 24 for managing the cumulative number of pixels (integrated value of the number of changed pixels) obtained by accumulating the number of pixels whose light receiving amount has changed is provided, and the cumulative number of pixels is increased.
  • the convolution processing unit image recognition processing unit 22
  • the management unit 24 may reset the cumulative number of pixels.
  • the threshold adjustment unit 23 that performs the threshold adjustment process for adjusting the threshold Th2 for the cumulative number of pixels based on the processing result of the classification process after the convolution process is provided. May be good. If the convolution process or pooling process is executed too frequently, the processing load will increase. Further, if the execution frequency is too low, the classification process may not be performed correctly. Specifically, there is a possibility that the subject to be detected cannot be detected by a surveillance camera or the like. By automatically adjusting the threshold Th2 by feedback control based on the classification processing result, the image recognition processing and the image classification processing using the CNN can be appropriately performed.
  • the threshold value adjustment unit 23 is a classification process after the convolution process executed when the cumulative number of pixels exceeds the threshold value Th2 for the cumulative number of pixels.
  • the threshold Th2 for the cumulative number of pixels may be increased in the threshold adjustment process. If the classification result is unchanged as a result of executing the convolution process, the pooling process, and the classification process, the execution frequency of each process may be too high. Further, the high execution frequency may be due to the low threshold Th2.
  • the threshold Th2 when the threshold Th2 is too low, the execution frequency of each process can be reduced by automatically adjusting to raise the threshold Th2, and the processing of the signal processing device (signal processing unit 14) can be reduced. The burden can be reduced.
  • the threshold value adjustment unit 23 includes the classification result of the classification process after the convolution process executed when the cumulative number of pixels exceeds the threshold value Th2 and immediately before that.
  • the threshold Th2 for the cumulative number of pixels may be lowered in the threshold adjustment process.
  • the execution frequency of each process may be too low. Further, the low execution frequency may be due to the high threshold Th2.
  • the execution frequency of each process can be increased by automatically adjusting to lower the threshold Th2, and the classification process is adjusted to be performed at an appropriate frequency. can do.
  • the threshold Th2 is automatically adjusted to raise the threshold Th2, so that the threshold Th2 can be adjusted to an appropriate value, and the processing load can be reduced and the classification accuracy can be improved at the same time. Can be done.
  • the threshold value adjusting unit 23 sets the threshold value Th2 when the cumulative number of pixels exceeds the threshold value Th2 again during the execution of the process executed because the cumulative number of pixels exceeds the threshold value Th2. It is not necessary to perform the convolution process due to exceeding the above value again.
  • the cumulative number of pixels which is the number of pixels whose light reception amount has changed, exceeds the threshold Th2
  • the cumulative number of pixels is reset and the convolution process, pooling process, and classification process are executed. If the cumulative number of pixels exceeds the threshold Th2 again during the execution of each of these processes and the execution request of each process is received, a time lag occurs between the execution request and the process.
  • the image pickup device 1 as an embodiment includes a light receiving unit (pixel array unit 11) that obtains a light receiving signal indicating the light receiving amount, and a pixel specifying unit 21 that specifies a pixel whose light receiving amount has changed based on the change amount of the light receiving signal.
  • a convolution processing unit image recognition processing unit 22 that performs convolution processing for pixels whose light reception amount has changed is provided in the convolution layer.
  • An image pickup apparatus 1 that does not perform the filter processing when all the pixels included in the processing target area having the same size as the filter are invariant pixels that are not affected by the pixels whose light reception amount has changed. May be good.
  • the image pickup apparatus 1 may include an image sensor 3 in which a light receiving unit (pixel array unit 11), a pixel identification unit 21, and a convolution processing unit (image recognition processing unit 22) are integrally formed.
  • a light receiving unit pixel array unit 11
  • a pixel identification unit 21 pixel identification unit 21
  • a convolution processing unit image recognition processing unit 22
  • the image sensor 3 has a light receiving unit (pixel array unit 11) that obtains a light receiving signal indicating the light receiving amount, a pixel specifying unit 21 that specifies a pixel whose light receiving amount has changed based on the change amount of the light receiving signal, and a light receiving amount.
  • the filter processing that includes a convolution processing unit (image recognition processing unit 22) that performs the convolution processing for the changed pixel in the convolution layer and can be executed a plurality of times in the convolution process, it is regarded as a partial area of the input image and the filter. If all the pixels included in the processing target area having the same size as the size are invariant pixels that are not affected by the pixels whose light receiving amount has changed, the filter processing may not be performed.
  • a CIS image sensor such as DVS and a signal processing unit (DNN block) that performs image recognition processing by DNN may be integrally formed.
  • the processing for image recognition using DNN may affect the processing for reading the received signal. That is, there is a possibility that the light receiving signal reading process is not properly performed due to the noise generated by the execution of the image recognition process using the DNN.
  • the processing load and the processing frequency for image recognition using the CNN can be reduced, noise can be reduced and the light receiving signal reading process can be appropriately executed.
  • the pixel in which the light receiving amount has changed is specified based on the change amount of the light receiving signal indicating the light receiving amount, and the convolution process is performed on the convolution layer for the pixel in which the light receiving amount has changed, and the convolution is performed.
  • the filter processing that can be executed multiple times in the processing, all the pixels included in the processing target area, which is a partial area of the input image and has the same size as the filter size, are not affected by the pixels whose light reception amount has changed. When it is a pixel, the filter processing is not performed.
  • This technology> Based on the amount of change in the light-receiving signal indicating the amount of light received, a pixel identification unit that identifies the pixel in which the amount of light received has changed, and a pixel identification unit.
  • a convolution processing unit that performs a convolution process on the pixel whose light receiving amount has changed in the convolution layer is provided.
  • all the pixels included in the processing target area which is a partial area of the input image and has the same size as the filter size, are affected by the pixel whose light receiving amount has changed.
  • a signal processing device that does not perform the filter processing when it is an invariant pixel that does not receive it.
  • the signal processing device which includes a management unit that manages a feature map generated by the convolution process.
  • the convolution processing unit includes the change pixel and the invariant pixel affected by the pixel whose light receiving amount has changed in the input image input to the convolution layer, the convolution processing unit obtains the data of the past feature map for the invariant pixel.
  • the signal processing apparatus according to (2) above.
  • the management unit manages a dirty map indicating whether each pixel of the input image input to the convolution layer is the change pixel or the invariant pixel.
  • the threshold value adjusting unit when the classification result of the classification process after the convolution process executed because the cumulative number of pixels exceeds the threshold value and the classification result of the classification process executed immediately before the convolution process are the same.
  • the signal processing apparatus according to (10) above which raises the threshold value for the cumulative number of pixels in the threshold value adjustment process.
  • the threshold adjustment unit may use the threshold adjustment unit.
  • the threshold adjusting unit convolves the convolution because the cumulative number of pixels exceeds the threshold again.
  • the signal processing apparatus according to any one of (10) to (12) above, which does not perform processing.
  • a light receiving part that obtains a light receiving signal indicating the amount of light received
  • a pixel identification unit that identifies a pixel whose light receiving amount has changed based on the change amount of the light receiving signal. It is provided with a convolution processing unit that performs a convolution process on the pixel whose light receiving amount has changed in the convolution layer.
  • the imaging device which includes an image sensor in which the light receiving unit, the pixel specifying unit, and the convolution processing unit are integrally formed.
  • a light receiving part that obtains a light receiving signal indicating the amount of light received, A pixel identification unit that identifies a pixel whose light receiving amount has changed based on the change amount of the light receiving signal.
  • a convolution processing unit that performs a convolution process on the pixel whose light receiving amount has changed in the convolution layer.
  • the filter processing that can be executed a plurality of times in the convolution processing, all the pixels included in the processing target area, which is a partial area of the input image and has the same size as the filter size, are affected by the pixel whose light receiving amount has changed.
  • An image sensor that does not perform the filtering process when it is an invariant pixel that does not receive it.
  • the pixel in which the amount of light received has changed is specified. The convolution process is performed on the convolution layer for the pixels whose light reception amount has changed.
  • all the pixels included in the processing target area which is a partial area of the input image and has the same size as the filter size, are affected by the pixel whose light receiving amount has changed.
  • Imaging device 1 Imaging device 3 Image sensor 11 Pixel array unit (light receiving unit) 14 Signal processing unit 17 Pixels 21 Pixel identification unit 22 Image recognition processing unit (convolution processing unit) 23 Threshold adjustment unit 24 Management unit Th2 Threshold

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention concerne un dispositif de traitement de signal pourvu d'une unité d'identification de pixel qui identifie, sur la base d'une quantité de variation d'un signal lumineux reçu indiquant une quantité de lumière reçue, un pixel dans lequel a varié la quantité de lumière reçue, et d'une unité de traitement de convolution qui effectue, par rapport au pixel dans lequel a varié la quantité de lumière reçue, un traitement de convolution dans une couche de convolution. Concernant un traitement de filtre qui peut être exécuté plusieurs fois lors du traitement de convolution, si tous les pixels compris dans une zone à traiter qui est une zone partielle d'une image d'entrée et qui a la même taille qu'un filtre sont des pixels sans variation qui ne sont pas affectés par le pixel dans lequel a varié la quantité de lumière reçue, le traitement de filtre n'est pas effectué.
PCT/JP2020/047168 2020-02-10 2020-12-17 Dispositif de traitement de signal, capteur d'image, dispositif d'imagerie et procédé de traitement de signal WO2021161652A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-020931 2020-02-10
JP2020020931 2020-02-10

Publications (1)

Publication Number Publication Date
WO2021161652A1 true WO2021161652A1 (fr) 2021-08-19

Family

ID=77292362

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/047168 WO2021161652A1 (fr) 2020-02-10 2020-12-17 Dispositif de traitement de signal, capteur d'image, dispositif d'imagerie et procédé de traitement de signal

Country Status (1)

Country Link
WO (1) WO2021161652A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018101317A (ja) * 2016-12-21 2018-06-28 ホーチキ株式会社 異常監視システム
US20190065885A1 (en) * 2017-08-29 2019-02-28 Beijing Samsung Telecom R&D Center Object detection method and system
JP2019211879A (ja) * 2018-06-01 2019-12-12 株式会社デンソーアイティーラボラトリ ３次元畳込み演算装置、ビジュアルオドメトリシステム、及び３次元畳込みプログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018101317A (ja) * 2016-12-21 2018-06-28 ホーチキ株式会社 異常監視システム
US20190065885A1 (en) * 2017-08-29 2019-02-28 Beijing Samsung Telecom R&D Center Object detection method and system
JP2019211879A (ja) * 2018-06-01 2019-12-12 株式会社デンソーアイティーラボラトリ ３次元畳込み演算装置、ビジュアルオドメトリシステム、及び３次元畳込みプログラム

Similar Documents

Publication Publication Date Title
JP6023058B2 (ja) 画像処理装置、画像処理方法、プログラム、集積回路
CN107258077B (zh) 用于连续自动聚焦(caf)的***和方法
CN100502471C (zh) 图像处理装置、图像处理方法和成像装置
CN106412214A (zh) 一种终端及终端拍摄的方法
CN101543054A (zh) 图像处理装置、图像处理方法及程序
JP4876065B2 (ja) 動きベクトル検出装置及び動きベクトル検出方法及び撮像装置及びプログラム
JP2003172873A5 (fr)
US20010014215A1 (en) Distance measuring device
WO2013150789A1 (fr) Dispositif d'analyse vidéo, procédé d'analyse vidéo, programme et circuit intégré
CN113840076B (zh) 一种拍摄变焦方法、电子设备及存储介质
CN104954659A (zh) 一种处理图像信息的方法及装置
JP7292850B2 (ja) 画像処理装置、画像処理方法、及びプログラム
KR20230110618A (ko) 영상 보정 방법, 장치 및 시스템, 전자 장치
CN101621619A (zh) 对多张脸部同时对焦的拍摄方法及其数字取像装置
JP2006318364A (ja) 画像処理装置
WO2021161652A1 (fr) Dispositif de traitement de signal, capteur d'image, dispositif d'imagerie et procédé de traitement de signal
US9489721B2 (en) Image processing apparatus, image processing method, and storage medium
CN114998317A (zh) 镜头遮挡检测方法、装置、摄像装置和存储介质
US20130293741A1 (en) Image processing apparatus, image capturing apparatus, and storage medium storing image processing program
CN103139521B (zh) 图像采集与选取方法及图像采集装置
JP2018142828A (ja) 付着物検出装置および付着物検出方法
CN101841654B (zh) 图像处理装置以及图像处理方法
JP7251426B2 (ja) 付着物検出装置および付着物検出方法
US20130223697A1 (en) Face detection-processing circuit and image pickup device including the same
JP2007189432A (ja) 車両用撮像装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20918694

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20918694

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP