WO2016111072A1 - Image processing device, image pickup device, and image processing method - Google Patents

Image processing device, image pickup device, and image processing method Download PDF

Info

Publication number
WO2016111072A1
WO2016111072A1 PCT/JP2015/079798 JP2015079798W WO2016111072A1 WO 2016111072 A1 WO2016111072 A1 WO 2016111072A1 JP 2015079798 W JP2015079798 W JP 2015079798W WO 2016111072 A1 WO2016111072 A1 WO 2016111072A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
license plate
vehicle body
image signal
signal corresponding
Prior art date
Application number
PCT/JP2015/079798
Other languages
French (fr)
Japanese (ja)
Inventor
星野　和弘
Original Assignee
ソニー株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニー株式会社 filed Critical ソニー株式会社
Priority to CN202010970091.7A priority Critical patent/CN112055130B/en
Priority to US15/540,597 priority patent/US10217034B2/en
Priority to JP2016568287A priority patent/JP6757258B2/en
Priority to CN201580071105.9A priority patent/CN107113375B/en
Publication of WO2016111072A1 publication Critical patent/WO2016111072A1/en
Priority to US16/246,340 priority patent/US10885403B2/en
Priority to US17/013,385 priority patent/US11244209B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/04—Detecting movement of traffic to be counted or controlled using optical or ultrasonic detectors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/223—Analysis of motion using block-matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/147—Details of sensors, e.g. sensor lenses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/60—Extraction of image or video features relating to illumination properties, e.g. using a reflectance or lighting model
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/017—Detecting movement of traffic to be counted or controlled identifying vehicles
    • G08G1/0175—Detecting movement of traffic to be counted or controlled identifying vehicles by photographing vehicles, e.g. when violating traffic rules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/11—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/131—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing infrared wavelengths
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/135—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on four or more different wavelength filter elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • H04N25/618—Noise processing, e.g. detecting, correcting, reducing or removing noise for random or high-frequency noise
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/144—Movement detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/20—Circuitry for controlling amplitude response
    • H04N5/205—Circuitry for controlling amplitude response for correcting amplitude versus frequency characteristic
    • H04N5/208—Circuitry for controlling amplitude response for correcting amplitude versus frequency characteristic for compensating for attenuation of high frequency components, e.g. crispening, aperture distortion correction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/646—Circuits for processing colour signals for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20192—Edge enhancement; Edge preservation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/143—Sensing or illuminating at different wavelengths
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
    • G06V20/625—License plates
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/08—Detecting or categorising vehicles
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2209/00—Details of colour television systems
    • H04N2209/04—Picture signal generators
    • H04N2209/041—Picture signal generators using solid-state devices
    • H04N2209/042—Picture signal generators using solid-state devices having a single pick-up sensor
    • H04N2209/045—Picture signal generators using solid-state devices having a single pick-up sensor using mosaic colour filter
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2209/00—Details of colour television systems
    • H04N2209/04—Picture signal generators
    • H04N2209/041—Picture signal generators using solid-state devices
    • H04N2209/042—Picture signal generators using solid-state devices having a single pick-up sensor
    • H04N2209/045—Picture signal generators using solid-state devices having a single pick-up sensor using mosaic colour filter
    • H04N2209/046—Colour interpolation to calculate the missing colour values

Definitions

  • This technology relates to an image processing apparatus.
  • the present invention relates to an image processing apparatus, an imaging apparatus, and an image processing method used for a surveillance camera.
  • a monitoring device for a vehicle a monitoring device that acquires the characteristics of the vehicle and identifies the vehicle is used.
  • a monitoring device that detects the position of a license plate of a vehicle acquires the color of the vehicle body based on the position of the license plate, and identifies the vehicle has been proposed (see, for example, Patent Document 1).
  • the detected position of the license plate is used only for setting a vehicle body color discrimination area for specifying the vehicle body color. That is, no special processing is performed on the license plate region. For this reason, in this prior art, only the color of the vehicle body is detected, and there is a problem that processing for improving reading of the license plate, which is indispensable for specifying the vehicle, is not performed.
  • This technology has been developed in view of such a situation, and in order to facilitate the reading of the license plate and the acquisition of the color of the vehicle body, the visibility of the license plate and the color reproducibility of the vehicle body are improved. Objective.
  • the present technology has been made to solve the above-described problems.
  • the first aspect of the present technology includes a vehicle body region detection unit that detects a vehicle body region of a vehicle from an image signal, and a vehicle license plate from the image signal.
  • a license plate area detector for detecting an area; a vehicle body area image processor for processing the image signal corresponding to the detected vehicle body area; and the image signal corresponding to the detected license plate area.
  • a license plate region image processing unit that performs processing different from the processing of the image signal corresponding to the vehicle body region, the image signal corresponding to the processed vehicle body region, and the processed license plate region
  • An image processing apparatus including a combining unit that combines the image signal corresponding to the above.
  • the license plate region image processing unit includes a luminance signal conversion process for converting the image signal corresponding to the license plate region into a luminance signal corresponding to the license plate region, and the luminance signal.
  • the license plate area image processing unit belongs to the same frame as the luminance signal conversion process for converting the image signal corresponding to the license plate area into a luminance signal corresponding to the license plate area.
  • the vehicle body region image processing unit is continuous with luminance and color difference signal conversion processing for converting the image signal corresponding to the vehicle body region into a luminance signal and a color difference signal corresponding to the vehicle body region.
  • the image signal includes image signals corresponding to red light, green light, blue light, and infrared light
  • the vehicle body region image processing unit corresponds to the vehicle body region.
  • the image signal corresponding to the red light, the green light, and the blue light included in the image signal is processed, and the license plate area image processing unit is included in the image signal corresponding to the license plate area. Processing of an image signal corresponding to infrared light may be performed.
  • the image signal corresponding to the license plate region is processed by the vehicle region image processing unit for each of the image signals corresponding to the red light, the green light, and the blue light included in the image signal corresponding to the vehicle body region.
  • the license plate region image processing unit performs processing on an image signal corresponding to infrared light included in the image signal.
  • the image signal further includes an image signal corresponding to white light
  • the vehicle body region image processing unit includes red light and green light included in the image signal corresponding to the vehicle body region.
  • Each image signal corresponding to blue light and white light may be processed. Accordingly, there is an effect that the processing by the vehicle body region image processing unit is performed on each image signal corresponding to the red light, the green light, the blue light, and the white light included in the image signal corresponding to the vehicle body region.
  • the image signal includes image signals corresponding to red light, green light, blue light, and infrared light, and an image corresponding to the infrared light included in the image signal.
  • the license plate area image processing unit further detects whether or not the signal is saturated, and the license plate region image processing unit detects that the image signal corresponding to the infrared light is not saturated.
  • the image signal corresponding to the infrared light included in the image signal corresponding to the license plate region is processed, and the saturation detection unit is saturated with the image signal corresponding to the infrared light.
  • the image signals corresponding to the red light, the green light, and the blue light included in the image signal corresponding to the license plate region may be processed.
  • the license plate area image processing unit processes the image signal corresponding to the infrared light when the image signal corresponding to the infrared light is not saturated, and the image signal corresponding to the infrared light is saturated. In this case, the image signal corresponding to the red light, the green light, and the blue light is processed.
  • a second aspect of the present technology provides an image sensor that generates an image signal, a vehicle body region detection unit that detects a vehicle body region of the vehicle from the image signal, and a number that detects a license plate region of the vehicle from the image signal.
  • a license plate region image processing unit that performs processing different from the processing of the image signal corresponding to the image signal, the image signal corresponding to the processed vehicle body region, and the image corresponding to the processed license plate region.
  • An imaging apparatus including a combining unit that combines signals.
  • the third aspect of the present technology includes a vehicle body region detection procedure for detecting the vehicle body region of the vehicle from the image signal, a license plate region detection procedure for detecting the license plate region of the vehicle from the image signal, and the detected
  • a vehicle body region image processing procedure for processing the image signal corresponding to the vehicle body region; and the processing of the image signal corresponding to the vehicle body region with respect to the image signal corresponding to the detected license plate region; Includes a license plate area image processing procedure for performing different processing, and a synthesis procedure for synthesizing the image signal corresponding to the processed vehicle body area and the image signal corresponding to the processed license plate area.
  • This is an image processing method.
  • the present technology by improving the visibility on the license plate and the color reproducibility on the vehicle body, it is possible to obtain an excellent effect that the license plate can be easily read and the color of the vehicle body can be easily obtained.
  • the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.
  • FIG. 1 is a diagram illustrating a configuration example of an imaging device according to an embodiment of the present technology.
  • the imaging apparatus 10 in the figure includes a lens 100, an imaging element 200, an image processing apparatus 300, an image signal output unit 400, an infrared light irradiation unit 500, and a control unit 600.
  • the lens 100 optically forms a subject on the image sensor 200.
  • the image sensor 200 converts an optical image formed by the lens 100 into an image signal and outputs the image signal.
  • the imaging device 200 is configured by two-dimensionally arranging pixels that generate image signals on a surface on which an optical image is formed.
  • the image signal output from this pixel includes a visible light signal corresponding to the visible light component and an infrared signal corresponding to the infrared light component.
  • the following image sensor can be used as the image sensor 200 having such pixels.
  • a pixel that outputs image signals corresponding to red light and infrared light a pixel that outputs image signals corresponding to green light and infrared light, a pixel that outputs image signals corresponding to blue light and infrared light, and
  • An image sensor having a pixel that outputs an image signal corresponding to infrared light can be used.
  • an imaging device having a total of five types of pixels in which pixels that output image signals corresponding to white light and infrared light are added to these pixels may be used.
  • an image signal corresponding to red light and infrared light, an image signal corresponding to green light and infrared light, and an image signal corresponding to blue light and infrared light are referred to as an R + IR signal, a G + IR signal, and a B + IR signal, respectively.
  • an image signal corresponding to white light and infrared light and an image signal corresponding to infrared light are referred to as a W + IR signal and an IR signal, respectively.
  • pixels that output the R + IR signal, the G + IR signal, the B + IR signal, the W + IR signal, and the IR signal are referred to as an R + IR pixel, a G + IR pixel, a B + IR pixel, a W + IR pixel, and an IR pixel, respectively.
  • each pixel is provided with a color filter.
  • the above-described four types or five types of pixels are configured so that the spectral characteristics of the color filter correspond to specific light.
  • the image processing apparatus 300 processes an image signal.
  • the image processing apparatus 300 converts the image signal output from the image sensor 200 into a luminance signal and a color difference signal, and processes and outputs these signals.
  • the image signal output unit 400 outputs the image signal processed by the image processing apparatus 300 to the outside of the imaging apparatus 10.
  • the image signal output unit 400 converts and outputs an image signal so as to conform to the standard of the signal line interface to which the imaging device 10 is connected.
  • the control unit 600 controls the entire imaging apparatus 10.
  • the infrared light irradiation unit 500 irradiates the subject with infrared light.
  • the infrared light irradiation unit 500 is controlled by the control unit 600.
  • FIG. 2 is a diagram illustrating a configuration example of the image sensor 200 according to the embodiment of the present technology.
  • the figure shows the arrangement of the pixels 201 on the screen of the image sensor 200.
  • the symbol written in the pixel 201 represents the type of the pixel 201.
  • Pixels labeled R + IR, G + IR, B + IR, W + IR and IR represent R + IR pixel, G + IR pixel, B + IR pixel, W + IR pixel and IR pixel, respectively. Yes.
  • These pixels are arranged on the screen based on a certain rule.
  • a in the figure is obtained by replacing one of the two G + IR pixels with an IR pixel in the Bayer array of R + IR pixels, G + IR pixels, and B + IR pixels. Pixels corresponding to red, green, and blue are arranged in a well-balanced manner, resulting in an array with high color reproducibility.
  • b is a checkered arrangement of G + IR pixels in which R + IR pixels, B + IR pixels, and IR pixels are arranged at positions other than the pixel positions of the G + IR pixels. Since there are many pixels corresponding to green, the array has a high resolution.
  • c is a checkered array of IR pixels in which R + IR pixels, G + IR pixels, and B + IR pixels are arranged at positions other than the pixel positions of the IR pixels. A large number of IR pixels are arranged, and the array has a high resolution with respect to infrared light.
  • d is a checkered arrangement of W + IR pixels in which R + IR pixels, G + IR pixels, B + IR pixels, and IR pixels are arranged at positions other than the pixel positions of the W + IR pixels. Many pixels corresponding to white are arranged, and the arrangement is high in sensitivity.
  • the imaging device 200 is assumed to be an imaging device having the R + IR pixel, the G + IR pixel, the B + IR pixel, and the IR pixel listed in a to c in FIG. To do.
  • FIG. 3 is a diagram illustrating a configuration example of the image processing device 300 according to the first embodiment of the present technology.
  • the image processing apparatus 300 includes an image sensor control unit 310, a motion image detection unit 320, a demosaic unit 330, and a license plate area detection unit 340.
  • the image processing apparatus 300 further includes a license plate region image processing unit 360, a vehicle body region detection unit 350, a vehicle body region image processing unit 370, and a combining unit 390.
  • the image sensor control unit 310 controls the image sensor 200.
  • the image sensor control unit 310 performs focus control, exposure control, image signal output control, and the like on the image sensor 200.
  • the motion image detection unit 320 detects a moving image area in the image signal output from the image sensor 200.
  • a known method for example, a method based on block matching used in MPEG (Moving Picture Experts Group) or the like can be used to detect this moving image area.
  • MPEG Motion Picture Experts Group
  • the demosaic unit 330 performs demosaic processing. This demosaic process is to interpolate a deficient image signal with respect to a monochromatic image signal output from the image sensor 200. By this demosaic processing, the image signal per pixel increases to four, that is, an R + IR signal, a G + IR signal, a B + IR signal, and an IR signal. Further, the demosaic unit 330 further performs an infrared light component removal process for removing an infrared light component from the R + IR signal, the G + IR signal, and the B + IR signal.
  • R, G, B, and IR represent an R signal, a G signal, a B signal, and an IR signal, respectively.
  • R + IR , G + IR and B + IR represent the R + IR signal, the G + IR signal and the B + IR signal, respectively.
  • the vehicle body area detection unit 350 detects the vehicle body area from the moving image area detected by the motion image detection unit 320.
  • a known method for example, a method of detecting the vehicle body region by holding a light / dark template representing a vehicle image as a light / dark image and performing pattern matching between the input image and the light / dark template. Can be used.
  • the license plate area detector 340 detects the license plate area from the moving image area detected by the motion image detector 320.
  • a known method can be used to detect the license plate region. For example, it is possible to use a method of detecting the license plate position by generating an image obtained by binarizing the image signal of the vehicle body and searching for a portion matching the pattern arrangement unique to the license plate in this image area.
  • the vehicle body region image processing unit 370 processes an image signal corresponding to the vehicle body region detected by the vehicle body region detection unit 350.
  • the vehicle body region image processing unit 370 performs processing for improving color reproducibility in the vehicle body. Details of the vehicle body region image processing will be described later.
  • the license plate area image processing unit 360 processes an image signal corresponding to the license plate area detected by the license plate area detection unit 340.
  • the license plate region image processing unit 360 performs processing different from the processing in the vehicle body region image processing unit 370 described above. That is, processing for improving the visibility of the license plate is performed. Details of the license plate area image processing will be described later.
  • the synthesizing unit 390 synthesizes the image signals processed by the vehicle body region image processing unit 370 and the license plate region image processing unit 360.
  • Each process in the image processing apparatus 300 described above can be realized by hardware that performs these processes. Also, a microcomputer or DSP may be installed and realized as software processing.
  • FIG. 4 is a diagram illustrating a demosaic process according to the first embodiment of the present technology. This figure shows a case where a demosaic process is performed on the pixel 202.
  • the pixel 202 is a B + IR pixel and outputs a B + IR signal.
  • An image signal corresponding to another color is interpolated at the pixel position of the pixel 202.
  • a is an example in the case of interpolating a G + IR signal for the pixel 202.
  • the average value of the G + IR signals in the G + IR pixels arranged on the left and right of the pixel 202 is set as the G + IR signal in the pixel 202.
  • B in the figure is an example in the case of interpolating the IR signal for the pixel 202.
  • the average value of the IR signals in the IR pixels arranged above and below the pixel 202 is set as the IR signal in the pixel 202.
  • C in the figure is an example when the R + IR signal is interpolated for the pixel 202.
  • the average value of the R + IR signals adjacent to the pixel 202 in the diagonal direction of the pixel 202 is set to the R + IR signal in the pixel 202.
  • the demosaic process in FIG. 9 performs interpolation using only image signals of adjacent pixels. After this demosaic processing, infrared light component removal processing is performed.
  • FIG. 5 is a diagram illustrating a configuration example of the license plate area image processing unit 360 according to the first embodiment of the present technology.
  • the license plate area image processing unit 360 includes a luminance signal conversion unit 362, an edge enhancement unit 363, and a two-dimensional noise reduction unit 364.
  • the edge emphasizing unit 363 performs edge emphasis processing on the Y signal in the license plate area.
  • the Y_edge signal in the figure represents the Y signal that has undergone edge enhancement processing.
  • This edge enhancement process is a process for enhancing an edge portion of an image.
  • a known method can be used for this edge enhancement processing. For example, a method can be used in which an edge portion is detected from an image and the edge portion is enhanced by increasing the luminance of the detected edge portion.
  • the two-dimensional noise reduction unit 364 performs a two-dimensional noise reduction process on the Y_edge signal.
  • the Y_nr2 signal in the figure represents the Y signal that has been subjected to the two-dimensional noise reduction process.
  • the noise reduction process is a process for removing superimposed noise.
  • the two-dimensional noise reduction process is a process for removing noise by attenuating high-frequency components of image signals belonging to the same frame.
  • FIG. 6 is a diagram illustrating a two-dimensional noise reduction process according to the first embodiment of the present technology.
  • a case where an epsilon filter is used as a two-dimensional noise reduction process is illustrated.
  • a in the same figure represents the block diagram of a two-dimensional noise reduction process.
  • a Y signal is input to the input signal line 303.
  • the epsilon filter 306 is a filter that removes a signal that exceeds a predetermined level with respect to an input signal.
  • the subtracter 308 subtracts the separated high frequency component from the input signal. Thereby, noise is removed.
  • Signal lines 305 and 307 are signal lines connected to the outputs of the high-pass filter 304 and the epsilon filter 306, respectively.
  • the output signal line 309 is a signal line connected to the output of the two-dimensional noise reduction unit 364.
  • this two-dimensional noise reduction process is inferior to the three-dimensional noise reduction process, which will be described later, the noise removal capability is inferior, the blur of the edge portion of the image is relatively small, and a reduction in visibility can be prevented.
  • other two-dimensional noise reduction processing for example, processing using a bilateral filter or processing using smoothing may be used.
  • the image in the license plate area can be converted into a monochrome image with improved visibility by edge enhancement processing and two-dimensional noise reduction processing.
  • FIG. 7 is a diagram illustrating a configuration example of the vehicle body region image processing unit 370 according to the first embodiment of the present technology.
  • the vehicle body region image processing unit 370 includes a luminance / color difference signal conversion unit 372, a two-dimensional noise reduction unit 374, and a three-dimensional noise reduction unit 375.
  • the luminance / color difference signal conversion unit 372 converts the R, G, and B signals among the image signals corresponding to the vehicle body region into luminance signals (Y signals) and color difference signals (Cb and Cr signals).
  • the Cb signal is a signal based on the difference between the B signal and the Y signal
  • the Cr signal is a signal based on the difference between the R signal and the Y signal. This conversion can be performed as follows.
  • Y 0.2990 ⁇ R + 0.5870 ⁇ G + 0.1140 ⁇ B
  • the two-dimensional noise reduction unit 374 performs the two-dimensional noise reduction process described above.
  • the two-dimensional noise reduction unit 374 performs a two-dimensional noise reduction process on the Y, Cb, and Cr signals output from the luminance / color difference signal conversion unit 372.
  • the Y_nr2, Cb_nr2 and Cr_nr2 signals in the figure represent Y, Cb and Cr signals that have been subjected to two-dimensional noise reduction processing, respectively.
  • the 3D noise reduction unit 375 performs 3D noise reduction processing.
  • the three-dimensional noise reduction unit 375 performs a three-dimensional noise reduction process on the Y_nr2, Cb_nr2, and Cr_nr2 signals output from the two-dimensional noise reduction unit 374.
  • This three-dimensional noise reduction process is a process for removing noise using image signals belonging to a plurality of consecutive frames.
  • the Y_nr3, Cb_nr3, and Cr_nr3 signals in the figure represent Y, Cb, and Cr signals that have been subjected to three-dimensional noise reduction processing, respectively.
  • FIG. 8 is a diagram illustrating a three-dimensional noise reduction process according to the first embodiment of the present technology.
  • Images 701 to 703 in the figure are images corresponding to the vehicle body region.
  • an image signal of an image corresponding to the body area of K consecutive frames is added by an adder 378.
  • a divider 379 divides by K which is the number of added image signals, and an output image 704 is generated.
  • the three-dimensional noise reduction process has a property that the effect of noise removal increases as the number K of image signals to be added increases, and can have a higher noise removal capability than the two-dimensional noise reduction process.
  • the image quality similarly decreases. Therefore, it is possible to prevent deterioration in image quality by compensating for the movement of the vehicle body and performing the above-described addition.
  • a known method for example, a motion compensation method used in MPEG can be used.
  • These two-dimensional noise reduction processing and three-dimensional noise reduction processing can turn the image of the vehicle body region into an image with improved color reproducibility.
  • FIG. 9 is a diagram illustrating a composition process according to the first embodiment of the present technology.
  • the figure shows the state of the image signal in each unit of the image processing apparatus 300 and the synthesis process in the synthesis unit 390.
  • An image signal constituting the image 705 including the vehicle is input to the motion image detection unit 320, and an image signal corresponding to the image 706 having motion is output.
  • This image signal is processed by the vehicle body region detection unit 350 and the vehicle body region image processing unit 370 to become an image signal corresponding to the image 707 with improved color reproducibility in the vehicle body.
  • the image signal corresponding to the moving image 706 is processed by the license plate area detection unit 340 and the license plate area image processing unit 360 to correspond to the image 708 with improved visibility on the license plate.
  • Image signal is processed by the license plate area detection unit 340 and the license plate area image processing unit 360 to correspond to the image 708 with improved visibility on the license plate.
  • an image 709 represents a combined image.
  • the synthesis by the synthesis unit 390 can be performed as follows, for example. An input image 705 is used as a background image, and an image 709 is synthesized by superimposing an image 707 in the vehicle body area and an image 708 in the license plate area on this image. As a result, the image with improved color reproducibility for the vehicle body region and the image with improved visibility for the license plate region are represented in the same image. If the license plate area is not detected by the license plate area detection unit 340, the synthesis unit 390 performs synthesis by superimposing only the image 707 of the vehicle body area on the background image.
  • FIG. 10 is a diagram illustrating an image display example according to the first embodiment of the present technology.
  • a represents an example of image display including the image display area 711 and the image information display area 712.
  • the image display area 711 is an area for displaying an image constituted by the image signal output from the synthesizing unit 390, and is processed by the image 715 processed by the vehicle body area image processing unit 370 and the license plate area image processing unit 360.
  • An image including the image 716 is displayed.
  • the image information display area 712 is an area for displaying information such as date.
  • b represents an example of display in which a license plate display area 713 is further added to the display of a in the figure. In the license plate display area 713, an image 714 obtained by enlarging the image 716 is displayed. For this reason, in the display of b in the same figure, the character string of a license plate can be read easily.
  • FIG. 11 is a diagram illustrating an example of an image processing procedure according to the first embodiment of the present technology.
  • the image processing apparatus 300 starts this processing when an image signal of one frame is input from the image sensor 200.
  • the moving image detection unit 320 detects a moving image area from the input image signal (step S901).
  • the image processing apparatus 300 proceeds to the process of step S906 and outputs the input image signal to the image signal output unit 400. (Step S906).
  • step S902 when a moving image region is detected (step S902: Yes), the vehicle body region detection unit 350 detects the vehicle region (step S903). At this time, when the vehicle body area is not detected (step S904: No), the image processing apparatus 300 proceeds to the process of step S906 and outputs the input image signal to the image signal output unit 400 (step). S906).
  • step S904 when the vehicle body area is detected (step S904: Yes), the license plate area detection unit 340 detects the license plate area (step S905).
  • step S910 the image processing apparatus 300 performs vehicle body and license plate region processing (step S910), and proceeds to the processing of step S906.
  • step S906 the image processing apparatus 300 outputs the image obtained in step S910 to the image signal output unit 400 (step S906). Thereafter, the image processing apparatus 300 ends the image processing.
  • FIG. 12 is a diagram illustrating an example of a processing procedure of the vehicle body and license plate region processing according to the first embodiment of the present technology.
  • the process in FIG. 11 corresponds to step S910 described in FIG.
  • the vehicle body region image processing unit 370 performs vehicle body region image processing (step S920).
  • the license plate area image processing unit 360 performs license plate area image processing (step S930).
  • the combining unit 390 combines the images processed by the vehicle body region image processing and the license plate region image processing (step S913).
  • the license plate area detection unit 340 does not detect the license plate area
  • the synthesis unit 390 performs synthesis using the image processed by the vehicle body area image processing.
  • the image processing apparatus 300 ends the vehicle body and license plate area processing.
  • FIG. 13 is a diagram illustrating an example of a processing procedure of the vehicle body region image processing according to the first embodiment of the present technology.
  • the process in FIG. 12 corresponds to step S920 described in FIG.
  • the vehicle body region image processing unit 370 converts the R, G, and B signals into luminance signals and color difference signals (step S923).
  • the vehicle body region image processing unit 370 performs a two-dimensional noise reduction process (step S925).
  • the vehicle body region image processing unit 370 performs three-dimensional noise reduction processing (step S926), and ends the vehicle body region image processing.
  • FIG. 14 is a diagram illustrating an example of a processing procedure of license plate area image processing according to the first embodiment of the present technology.
  • the process in FIG. 12 corresponds to step S930 described in FIG.
  • the license plate area image processing unit 360 converts the IR signal into a luminance signal (step S932).
  • the license plate area image processing unit 360 performs edge enhancement processing (step S934).
  • the license plate area image processing unit 360 performs two-dimensional noise reduction processing (step S935), and ends the license plate area image processing.
  • the visibility on the license plate and the color reproducibility on the vehicle body can be improved, and the vehicle to be monitored can be easily specified.
  • the imaging device 200 corresponding to both visible light and infrared light is used, and the visible light signal and the infrared light signal are separated and processed, thereby forming the imaging device 10 using a single imaging device. be able to. Thereby, the structure of the imaging device 10 can be simplified.
  • the license plate area detection unit 340 detects the license plate area for the image signal processed by the demosaic unit 330.
  • the license plate area may be detected for the image signal corresponding to the vehicle body area detected by the vehicle body area detection unit 350. This is because the license plate region is included in the vehicle body region. Thereby, the area of the image to be processed in the license plate area detection unit 340 is limited, and the processing time can be shortened.
  • the license plate image is processed by the IR signal.
  • the image of the license plate region is processed with the visible light signal (R, G, and B signals).
  • FIG. 15 is a diagram illustrating a configuration example of the image processing device 300 according to the second embodiment of the present technology.
  • the image processing apparatus 300 in the figure is different from the image processing apparatus 300 described in FIG. 3 in that it further includes a saturation detection unit 380.
  • the pixel corresponding to red light, green light, and blue light needs to be configured to include a color filter that removes infrared light. That is, it is necessary to use the image pickup device having the arrangement shown in FIGS. 2A to 2C among the image pickup devices described in FIG. Furthermore, the pixels in which R + IR, G + IR, and B + IR are described need to use an image sensor configured to detect red light, green light, and blue light and not detect an infrared light component, respectively.
  • the R + IR pixel, the G + IR pixel, and the B + IR pixel of the image sensor described in FIG. 2 have sensitivity to the infrared light component, respectively. Therefore, when excessive infrared light is incident, all of the IR pixels including the IR pixel are included. This is because the image signal output from the pixel is saturated.
  • the demosaic unit 330 removes the infrared component described above. Processing can be omitted.
  • Other configurations of the imaging apparatus 10 and the image processing apparatus 300 are the same as those of the imaging apparatus 10 and the image processing apparatus 300 described in the first embodiment of the present technology, and thus the description thereof is omitted.
  • the saturation detector 380 detects whether the IR signal is saturated. Since the image sensor control unit 310 controls the exposure as described above, normally, saturation of the IR signal does not occur. However, when imaging is performed while irradiating infrared light in the infrared light irradiation unit 500, if the distance between the vehicle that is the subject and the imaging device 10 is short, the saturation of the IR signal temporarily occurs. appear. Even in such a case, an image signal in which the IR signal is not saturated can be obtained by the exposure control of the image sensor control unit 310. However, when the vehicle enters at high speed, the exposure control of the image sensor control unit 310 cannot follow, and only the image signal in a state where the IR signal is saturated is acquired. In such a case, the saturation detector 380 detects the saturation of the IR signal.
  • the license plate region image processing unit 360 performs license plate region image processing based on the detection result by the saturation detection unit 380.
  • FIG. 16 is a diagram illustrating a configuration example of the license plate area image processing unit 360 according to the second embodiment of the present technology.
  • the license plate area image processing unit 360 is different from the license plate area image processing unit 360 described with reference to FIG. 5 in that a luminance signal conversion unit 366 is provided instead of the luminance signal conversion unit 362.
  • the luminance signal conversion unit 366 converts the R signal, the G signal, and the B signal into a Y signal when the IR signal is saturated. . For this reason, when the IR signal is saturated, it is possible to prevent the license plate area from being displayed in a so-called whiteout state. Thereby, the visibility of a license plate can be improved.
  • FIG. 17 is a diagram illustrating an example of a processing procedure of license plate area image processing according to the second embodiment of the present technology.
  • the process shown in FIG. 12 corresponds to the license plate area image process described with reference to FIG.
  • Note that the other image processing procedures are the same as the processing procedures described in the first embodiment of the present technology, and a description thereof will be omitted.
  • the license plate area image processing unit 360 determines whether or not the IR signal is saturated (step S951). Specifically, it is determined whether saturation of IR signal is detected by saturation detector 380. If the IR signal is not saturated (step S951: No), the license plate area image processing unit 360 converts the IR signal into a luminance signal (step S952), and the process proceeds to step S954. On the other hand, when the IR signal is saturated (step S951: Yes), the license plate area image processing unit 360 converts the R signal, the G signal, and the B signal into luminance signals (step S953). Transition to processing. In step S954, the license plate area image processing unit 360 performs edge enhancement processing (step S954). Next, the license plate area image processing unit 360 performs two-dimensional noise reduction processing (step S955), and ends the license plate area image processing.
  • the infrared light irradiation control in the infrared light irradiation unit 500 is not performed.
  • the saturation detection unit 380 of the image processing apparatus 300 detects the saturation of the IR signal
  • the control unit 600 controls the infrared light irradiation unit 500 to reduce the irradiation amount of the infrared light. Also good. Thereby, it can be set as a suitable infrared light irradiation amount, and the fall of the visibility in a license plate can be prevented.
  • the license plate region image is superimposed on the image of the vehicle body region and synthesized.
  • the registration of the license plate region image on the vehicle body region image is omitted.
  • FIG. 18 is a diagram illustrating a composition process according to the third embodiment of the present technology.
  • “a” represents the synthesis processing in the first embodiment of the present technology.
  • the license plate region image 708 is superimposed on the vehicle body region image 707 and displayed in the image display region 711 in the combining unit 390. For this reason, the license plate portion of the image 707 is overwritten with the monochrome license plate image, and the color of the license plate cannot be specified.
  • the composition processing according to the third embodiment of the present technology in which the registration of the license plate region image with the vehicle body region image 707 is omitted.
  • the license plate area of the image 707 of the vehicle body area has improved color reproducibility like the other vehicle body areas, and the color of the license plate can be specified.
  • the license plate area image 708 is displayed in the license plate display area 713.
  • the color of the license plate can be specified together with the color of the vehicle body.
  • the image sensor 200 that does not include a pixel corresponding to white light is used.
  • the image sensor 200 including pixels corresponding to white light is used.
  • the imaging device 10 according to the fourth embodiment of the present technology uses an imaging device having the pixel arrangement illustrated in d in FIG.
  • the motion image detection unit 320 and the demosaic unit 330 of the image processing device 300 according to the fourth embodiment of the present technology need to perform processing on an image signal including a W + IR signal. Since the configuration of the imaging apparatus 10 other than this is the same as that of the imaging apparatus 10 described in the first embodiment of the present technology, the description thereof is omitted.
  • FIG. 19 and 20 are diagrams illustrating demosaic processing according to the fourth embodiment of the present technology.
  • FIG. 19 shows the arrangement of the pixels of the image sensor 200, and it is assumed that signals other than the G + IR signal are interpolated in the pixel 203 arranged in the center.
  • a represents an example in the case of interpolating the R + IR signal.
  • the average value of the signals of the R + IR pixels around the pixel 203 is set as the R + IR signal in the pixel 203. This average is calculated by a weighted average using each R + IR signal.
  • Each R + IR signal is multiplied by a predetermined weight.
  • This weight is set to a larger value as the pixel is closer to the pixel 203. That is, in a in FIG. 20, a greater weight is set for the pixel 204 adjacent to the pixel 203 than for the pixel 205 arranged at a relatively distant position.
  • the interpolated image signal is described with the suffix “_L”. That is, the R + IR signal, the G + IR signal, the B + IR signal, the W + IR signal, and the signal after interpolation with respect to the IR signal become an R + IR_L signal, a G + IR_L signal, a B + IR_L signal, a W + IR_L signal, and an IR_L signal, respectively. Since these signals are average values of image signals in a relatively wide range, they become image signals in which high frequency components are lost. Therefore, when processing is performed using these signals, the image signal corresponding to the vehicle body region becomes an image with a blurred outline.
  • W + IR_H The interpolated W + IR signal is referred to as W + IR_H. Since this signal is the average value of the image signals of adjacent pixels, it becomes an image signal containing many high-frequency components.
  • the W + IR signal includes a signal having a wide range of wavelengths ranging from visible light to infrared light. As described later, this signal can be used to restore a high-frequency component for a signal after other demosaic processing.
  • the IR signal component is removed from the R + IR_L signal, the G + IR_L signal, and the B + IR_L signal. This can be performed by subtracting the IR_L signal from the R + IR_L signal, the G + IR_L signal, and the B + IR_L signal, similarly to the method described in the first embodiment of the present technology.
  • the image signals from which the IR signal component is removed are referred to as R_L signal, G_L signal, and B_L signal, respectively.
  • R R_L + W + IR_H ⁇ R_L / W + IR_L
  • G G_L + W + IR_H ⁇ G_L / W + IR_L
  • B B_L + W + IR_H ⁇ B_L / W + IR_L
  • W + IR _H and W + IR _L each represents a W + IR_H signals and W + IR_L signal.
  • R_L, G_L, and B_L represent an R_L signal, a G_L signal, and a B_L signal, respectively.
  • R signal, G signal, and B signal from which the high frequency component was restored can be obtained. Since these signals process the image signal corresponding to the vehicle body region, it is possible to obtain the image signal of the vehicle body region with improved blur and improved visibility.
  • the imaging element 200 having an image signal corresponding to white light when used, the visibility of the vehicle body region can be improved together with the color reproducibility. .
  • the single image sensor 200 is used.
  • the saturation detector 380 described with reference to FIG. 15 may be provided.
  • the IR signal When the IR signal is saturated, the visibility of the license plate region by the visible light signal can be improved.
  • the influence of infrared light can be removed by attaching an infrared cut filter to an image pickup device corresponding to visible light.
  • the embodiment of the present technology it is possible to read the license plate and acquire the color of the vehicle body by improving the visibility of the license plate and the color reproducibility of the vehicle body. Thereby, it is possible to easily identify the vehicle to be monitored.
  • the processing procedure described in the above embodiment may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program. You may catch it.
  • a recording medium for example, a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disc), a memory card, a Blu-ray disc (Blu-ray (registered trademark) Disc), or the like can be used.
  • this technique can also take the following structures.
  • a vehicle body region detection unit that detects a vehicle body region of the vehicle from the image signal;
  • a license plate area detector for detecting a license plate area of the vehicle from the image signal;
  • a vehicle body region image processing unit for processing the image signal corresponding to the detected vehicle body region;
  • a license plate region image processing unit that performs processing different from the processing of the image signal corresponding to the vehicle body region on the image signal corresponding to the detected license plate region;
  • An image processing apparatus comprising: a combining unit configured to combine the image signal corresponding to the processed vehicle body region and the image signal corresponding to the processed license plate region.
  • the license plate region image processing unit emphasizes an edge portion of an image in the luminance signal and luminance signal conversion processing for converting the image signal corresponding to the license plate region into a luminance signal corresponding to the license plate region.
  • the license plate region image processing unit includes a high-frequency component of the luminance signal belonging to the same frame as the luminance signal conversion processing for converting the image signal corresponding to the license plate region into a luminance signal corresponding to the license plate region.
  • the image processing apparatus according to (1) wherein a two-dimensional noise reduction process that removes noise of the luminance signal by attenuating the luminance signal is performed.
  • the vehicle body region image processing unit belongs to a plurality of frames that are continuous with luminance and color difference signal conversion processing for converting the image signal corresponding to the vehicle body region into a luminance signal and a color difference signal corresponding to the vehicle body region.
  • the image signal includes respective image signals corresponding to red light, green light, blue light, and infrared light
  • the vehicle body region image processing unit performs processing of each image signal corresponding to red light, green light, and blue light included in the image signal corresponding to the vehicle body region
  • the image processing apparatus according to any one of (1) to (4), wherein the license plate region image processing unit performs processing of an image signal corresponding to infrared light included in the image signal corresponding to the license plate region.
  • the image signal further includes an image signal corresponding to white light, The vehicle body region image processing unit performs processing of each image signal corresponding to red light, green light, blue light, and white light included in the image signal corresponding to the vehicle body region.
  • the image signal includes respective image signals corresponding to red light, green light, blue light, and infrared light
  • a saturation detector for detecting whether an image signal corresponding to infrared light included in the image signal is saturated;
  • the license plate region image processing unit includes an infrared signal included in the image signal corresponding to the license plate region.
  • an image sensor that generates an image signal; A vehicle body region detection unit for detecting a vehicle body region of the vehicle from the image signal; A license plate area detector for detecting a license plate area of the vehicle from the image signal; A vehicle body region image processing unit for processing the image signal corresponding to the detected vehicle body region; A license plate region image processing unit that performs processing different from the processing of the image signal corresponding to the vehicle body region on the image signal corresponding to the detected license plate region; An image pickup apparatus comprising: a combining unit that combines the image signal corresponding to the processed vehicle body region and the image signal corresponding to the processed license plate region.
  • a vehicle body region detection procedure for detecting a vehicle body region from the image signal;
  • a license plate area detection procedure for detecting a license plate area of a vehicle from the image signal;
  • a vehicle body region image processing procedure for processing the image signal corresponding to the detected vehicle body region;
  • a license plate area image processing procedure for performing processing different from the processing of the image signal corresponding to the vehicle body area on the image signal corresponding to the detected license plate area;
  • An image processing method comprising: a combining procedure for combining the image signal corresponding to the processed vehicle body region and the image signal corresponding to the processed license plate region.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Vascular Medicine (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)

Abstract

The purpose of the present invention is, with respect to a surveillance camera, to improve the visibility of a license plate and the ability to reproduce the color of a vehicle body. In the present invention, a vehicle body region detection unit detects the vehicle body region of a vehicle from an image signal. A license plate region detection unit detects the license plate region of a vehicle from an image signal. A vehicle body region image processing unit performs processing of the image signal corresponding to the detected vehicle body region. A license plate region image processing unit performs processing, which is different from the processing of the image signal corresponding to the vehicle body region, of the image signal corresponding to the detected license plate region. A synthesizing unit synthesizes the processed image signal corresponding to the vehicle body region and the processed image signal corresponding to the license plate region.

Description

画像処理装置、撮像装置および画像処理方法Image processing apparatus, imaging apparatus, and image processing method
　本技術は、画像処理装置に関する。詳しくは、監視カメラに使用される画像処理装置、撮像装置および画像処理方法に関する。 This technology relates to an image processing apparatus. Specifically, the present invention relates to an image processing apparatus, an imaging apparatus, and an image processing method used for a surveillance camera.
　従来、車輌を対象とする監視装置として、車輌の特徴を取得して車輌を特定する監視装置が使用されている。例えば、車輌のナンバープレートの位置を検出して、このナンバープレートの位置に基づいて車体の色を取得し、車輌を特定する監視装置が提案されている（例えば、特許文献１参照。）。 Conventionally, as a monitoring device for a vehicle, a monitoring device that acquires the characteristics of the vehicle and identifies the vehicle is used. For example, a monitoring device that detects the position of a license plate of a vehicle, acquires the color of the vehicle body based on the position of the license plate, and identifies the vehicle has been proposed (see, for example, Patent Document 1).
特開２００６－２０１８１７号公報JP 2006-201817 A
　上述の従来技術では、検出したナンバープレートの位置は車体の色を特定する車体色判別エリアを設定するためにのみ使用される。すなわち、ナンバープレート領域については特段の処理を行っていない。このため、この従来技術では、車体の色のみが検出され、車輌の特定に欠かせないナンバープレートの読取りを改善する処理が行われていないという問題がある。 In the above-described prior art, the detected position of the license plate is used only for setting a vehicle body color discrimination area for specifying the vehicle body color. That is, no special processing is performed on the license plate region. For this reason, in this prior art, only the color of the vehicle body is detected, and there is a problem that processing for improving reading of the license plate, which is indispensable for specifying the vehicle, is not performed.
　本技術はこのような状況に鑑みて生み出されたものであり、ナンバープレートの読取りおよび車体の色の取得を容易にするため、ナンバープレートにおける視認性および車体における色の再現性を向上することを目的とする。 This technology has been developed in view of such a situation, and in order to facilitate the reading of the license plate and the acquisition of the color of the vehicle body, the visibility of the license plate and the color reproducibility of the vehicle body are improved. Objective.
　本技術は、上述の問題点を解消するためになされたものであり、その第１の側面は、画像信号から車輌の車体領域を検出する車体領域検出部と、上記画像信号から車輌のナンバープレート領域を検出するナンバープレート領域検出部と、上記検出された上記車体領域に対応する上記画像信号の処理を行う車体領域画像処理部と、上記検出された上記ナンバープレート領域に対応する上記画像信号に対して上記車体領域に対応する上記画像信号の上記処理とは異なる処理を行うナンバープレート領域画像処理部と、上記処理された上記車体領域に対応する上記画像信号と上記処理された上記ナンバープレート領域に対応する上記画像信号とを合成する合成部とを具備する画像処理装置である。これにより、ナンバープレート領域に対応する画像信号と車体領域に対応する画像信号とには異なる処理が行われるという作用をもたらす。 The present technology has been made to solve the above-described problems. The first aspect of the present technology includes a vehicle body region detection unit that detects a vehicle body region of a vehicle from an image signal, and a vehicle license plate from the image signal. A license plate area detector for detecting an area; a vehicle body area image processor for processing the image signal corresponding to the detected vehicle body area; and the image signal corresponding to the detected license plate area. On the other hand, a license plate region image processing unit that performs processing different from the processing of the image signal corresponding to the vehicle body region, the image signal corresponding to the processed vehicle body region, and the processed license plate region An image processing apparatus including a combining unit that combines the image signal corresponding to the above. Thus, there is an effect that different processing is performed on the image signal corresponding to the license plate region and the image signal corresponding to the vehicle body region.
　また、この第１の側面において、上記ナンバープレート領域画像処理部は、上記ナンバープレート領域に対応する上記画像信号を上記ナンバープレート領域に対応する輝度信号に変換する輝度信号変換処理と上記輝度信号における画像のエッジ部分を強調するエッジ強調処理とを行ってもよい。これにより、ナンバープレート領域に対応する輝度信号における画像のエッジ部分が強調されるという作用をもたらす。 In the first aspect, the license plate region image processing unit includes a luminance signal conversion process for converting the image signal corresponding to the license plate region into a luminance signal corresponding to the license plate region, and the luminance signal. You may perform the edge enhancement process which emphasizes the edge part of an image. This brings about the effect that the edge portion of the image in the luminance signal corresponding to the license plate region is enhanced.
　また、この第１の側面において、上記ナンバープレート領域画像処理部は、上記ナンバープレート領域に対応する上記画像信号を上記ナンバープレート領域に対応する輝度信号に変換する輝度信号変換処理と同一フレームに属する上記輝度信号の高周波成分を減衰させることにより上記輝度信号のノイズを除去する２次元ノイズリダクション処理とを行ってもよい。これにより、ナンバープレート領域に対応する輝度信号における画像のノイズが除去されるという作用をもたらす。 In this first aspect, the license plate area image processing unit belongs to the same frame as the luminance signal conversion process for converting the image signal corresponding to the license plate area into a luminance signal corresponding to the license plate area. You may perform the two-dimensional noise reduction process which removes the noise of the said luminance signal by attenuating the high frequency component of the said luminance signal. This brings about the effect that the noise of the image in the luminance signal corresponding to the license plate area is removed.
　また、この第１の側面において、上記車体領域画像処理部は、上記車体領域に対応する上記画像信号を上記車体領域に対応する輝度信号および色差信号に変換する輝度および色差信号変換処理と連続する複数のフレームに属する上記輝度信号および上記色差信号を用いて上記輝度信号および上記色差信号のノイズを除去する３次元ノイズリダクション処理とを行ってもよい。これにより、車体領域に対応する輝度信号および色差信号のノイズが除去されるという作用をもたらす。 In the first aspect, the vehicle body region image processing unit is continuous with luminance and color difference signal conversion processing for converting the image signal corresponding to the vehicle body region into a luminance signal and a color difference signal corresponding to the vehicle body region. You may perform the three-dimensional noise reduction process which removes the noise of the said luminance signal and the said color difference signal using the said luminance signal and the said color difference signal which belong to a some flame | frame. As a result, the noise of the luminance signal and the color difference signal corresponding to the vehicle body region is removed.
　また、この第１の側面において、上記画像信号は、赤色光と緑色光と青色光と赤外光とに対応するそれぞれの画像信号を備え、上記車体領域画像処理部は、上記車体領域に対応する上記画像信号に含まれる赤色光と緑色光と青色光とに対応するそれぞれの画像信号の処理を行い、上記ナンバープレート領域画像処理部は、上記ナンバープレート領域に対応する上記画像信号に含まれる赤外光に対応する画像信号の処理を行ってもよい。これにより、車体領域に対応する画像信号に含まれる赤色光と緑色光と青色光とに対応するそれぞれの画像信号について上記車体領域画像処理部による処理が行われ、ナンバープレート領域に対応する画像信号に含まれる赤外光に対応する画像信号について上記ナンバープレート領域画像処理部による処理が行われるという作用をもたらす。 In the first aspect, the image signal includes image signals corresponding to red light, green light, blue light, and infrared light, and the vehicle body region image processing unit corresponds to the vehicle body region. The image signal corresponding to the red light, the green light, and the blue light included in the image signal is processed, and the license plate area image processing unit is included in the image signal corresponding to the license plate area. Processing of an image signal corresponding to infrared light may be performed. Thus, the image signal corresponding to the license plate region is processed by the vehicle region image processing unit for each of the image signals corresponding to the red light, the green light, and the blue light included in the image signal corresponding to the vehicle body region. The license plate region image processing unit performs processing on an image signal corresponding to infrared light included in the image signal.
　また、この第１の側面において、上記画像信号は、白色光に対応する画像信号をさらに備え、上記車体領域画像処理部は、上記車体領域に対応する上記画像信号に含まれる赤色光と緑色光と青色光と白色光とに対応するそれぞれの画像信号の処理を行ってもよい。これにより、車体領域に対応する画像信号に含まれる赤色光と緑色光と青色光と白色光とに対応するそれぞれの画像信号について上記車体領域画像処理部による処理が行われるという作用をもたらす。 In the first aspect, the image signal further includes an image signal corresponding to white light, and the vehicle body region image processing unit includes red light and green light included in the image signal corresponding to the vehicle body region. Each image signal corresponding to blue light and white light may be processed. Accordingly, there is an effect that the processing by the vehicle body region image processing unit is performed on each image signal corresponding to the red light, the green light, the blue light, and the white light included in the image signal corresponding to the vehicle body region.
　また、この第１の側面において、上記画像信号は、赤色光と緑色光と青色光と赤外光とに対応するそれぞれの画像信号を備え、上記画像信号に含まれる赤外光に対応する画像信号が飽和しているか否かを検出する飽和検出部をさらに具備し、上記ナンバープレート領域画像処理部は、上記飽和検出部が上記赤外光に対応する画像信号が飽和していないことを検出した場合には上記ナンバープレート領域に対応する上記画像信号に含まれる赤外光に対応する画像信号の処理を行い、上記飽和検出部が上記赤外光に対応する画像信号が飽和していることを検出した場合には上記ナンバープレート領域に対応する上記画像信号に含まれる赤色光と緑色光と青色光とに対応するそれぞれの画像信号の処理を行ってもよい。これにより、ナンバープレート領域画像処理部は、赤外光に対応する画像信号が飽和していない場合には赤外光に対応する画像信号について処理を行い、赤外光に対応する画像信号が飽和している場合には赤色光と緑色光と青色光とに対応するそれぞれの画像信号について処理を行うという作用をもたらす。 In the first aspect, the image signal includes image signals corresponding to red light, green light, blue light, and infrared light, and an image corresponding to the infrared light included in the image signal. The license plate area image processing unit further detects whether or not the signal is saturated, and the license plate region image processing unit detects that the image signal corresponding to the infrared light is not saturated. In this case, the image signal corresponding to the infrared light included in the image signal corresponding to the license plate region is processed, and the saturation detection unit is saturated with the image signal corresponding to the infrared light. When the image signal is detected, the image signals corresponding to the red light, the green light, and the blue light included in the image signal corresponding to the license plate region may be processed. As a result, the license plate area image processing unit processes the image signal corresponding to the infrared light when the image signal corresponding to the infrared light is not saturated, and the image signal corresponding to the infrared light is saturated. In this case, the image signal corresponding to the red light, the green light, and the blue light is processed.
　また、本技術の第２の側面は、画像信号を生成する撮像素子と、上記画像信号から車輌の車体領域を検出する車体領域検出部と、上記画像信号から車輌のナンバープレート領域を検出するナンバープレート領域検出部と、上記検出された上記車体領域に対応する上記画像信号の処理を行う車体領域画像処理部と、上記検出された上記ナンバープレート領域に対応する上記画像信号に対して上記車体領域に対応する上記画像信号の上記処理とは異なる処理を行うナンバープレート領域画像処理部と、上記処理された上記車体領域に対応する上記画像信号と上記処理された上記ナンバープレート領域に対応する上記画像信号とを合成する合成部とを具備する撮像装置である。これにより、ナンバープレート領域に対応する画像信号と車体領域に対応する画像信号とには異なる処理が行われるという作用をもたらす。 In addition, a second aspect of the present technology provides an image sensor that generates an image signal, a vehicle body region detection unit that detects a vehicle body region of the vehicle from the image signal, and a number that detects a license plate region of the vehicle from the image signal. A plate region detection unit; a vehicle body region image processing unit for processing the image signal corresponding to the detected vehicle body region; and the vehicle body region for the image signal corresponding to the detected license plate region. A license plate region image processing unit that performs processing different from the processing of the image signal corresponding to the image signal, the image signal corresponding to the processed vehicle body region, and the image corresponding to the processed license plate region. An imaging apparatus including a combining unit that combines signals. Thus, there is an effect that different processing is performed on the image signal corresponding to the license plate region and the image signal corresponding to the vehicle body region.
　また、本技術の第３の側面は、画像信号から車輌の車体領域を検出する車体領域検出手順と、上記画像信号から車輌のナンバープレート領域を検出するナンバープレート領域検出手順と、上記検出された上記車体領域に対応する上記画像信号の処理を行う車体領域画像処理手順と、上記検出された上記ナンバープレート領域に対応する上記画像信号に対して上記車体領域に対応する上記画像信号の上記処理とは異なる処理を行うナンバープレート領域画像処理手順と、上記処理された上記車体領域に対応する上記画像信号と上記処理された上記ナンバープレート領域に対応する上記画像信号とを合成する合成手順とを具備する画像処理方法である。これにより、ナンバープレート領域に対応する画像信号と車体領域に対応する画像信号とには異なる処理が行われるという作用をもたらす。 The third aspect of the present technology includes a vehicle body region detection procedure for detecting the vehicle body region of the vehicle from the image signal, a license plate region detection procedure for detecting the license plate region of the vehicle from the image signal, and the detected A vehicle body region image processing procedure for processing the image signal corresponding to the vehicle body region; and the processing of the image signal corresponding to the vehicle body region with respect to the image signal corresponding to the detected license plate region; Includes a license plate area image processing procedure for performing different processing, and a synthesis procedure for synthesizing the image signal corresponding to the processed vehicle body area and the image signal corresponding to the processed license plate area. This is an image processing method. Thus, there is an effect that different processing is performed on the image signal corresponding to the license plate region and the image signal corresponding to the vehicle body region.
　本技術によれば、ナンバープレートにおける視認性および車体における色の再現性を向上させることにより、ナンバープレートの読取りおよび車体の色の取得が容易になるという優れた効果を奏し得る。なお、ここに記載された効果は必ずしも限定されるものではなく、本開示中に記載されたいずれかの効果であってもよい。 According to the present technology, by improving the visibility on the license plate and the color reproducibility on the vehicle body, it is possible to obtain an excellent effect that the license plate can be easily read and the color of the vehicle body can be easily obtained. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.
本技術の実施の形態における撮像装置の構成例を示す図である。It is a figure showing an example of composition of an imaging device in an embodiment of this art. 本技術の実施の形態における撮像素子２００の構成例を示す図である。It is a figure showing an example of composition of image sensor 200 in an embodiment of this art. 本技術の第１の実施の形態における画像処理装置３００の構成例を示す図である。It is a figure showing an example of composition of image processing device 300 in a 1st embodiment of this art. 本技術の第１の実施の形態におけるデモザイク処理を示す図である。It is a figure showing demosaic processing in a 1st embodiment of this art. 本技術の第１の実施の形態におけるナンバープレート領域画像処理部３６０の構成例を示す図である。It is a figure which shows the structural example of the license plate area | region image process part 360 in 1st Embodiment of this technique. 本技術の第１の実施の形態における２次元ノイズリダクション処理を示す図である。It is a figure which shows the two-dimensional noise reduction process in 1st Embodiment of this technique. 本技術の第１の実施の形態における車体領域画像処理部３７０の構成例を示す図である。It is a figure which shows the structural example of the vehicle body area | region image process part 370 in 1st Embodiment of this technique. 本技術の第１の実施の形態における３次元ノイズリダクション処理を示す図である。It is a figure which shows the three-dimensional noise reduction process in 1st Embodiment of this technique. 本技術の第１の実施の形態における合成処理を示す図である。It is a figure showing composition processing in a 1st embodiment of this art. 本技術の第１の実施の形態における画像表示例を示す図である。It is a figure showing an example of image display in a 1st embodiment of this art. 本技術の第１の実施の形態における画像処理手順の一例を示す図である。It is a figure showing an example of an image processing procedure in a 1st embodiment of this art. 本技術の第１の実施の形態における車体およびナンバープレート領域処理の処理手順の一例を示す図である。It is a figure which shows an example of the process sequence of the vehicle body and number plate area | region process in 1st Embodiment of this technique. 本技術の第１の実施の形態における車体領域画像処理の処理手順の一例を示す図である。It is a figure which shows an example of the process sequence of the vehicle body area | region image process in 1st Embodiment of this technique. 本技術の第１の実施の形態におけるナンバープレート領域画像処理の処理手順の一例を示す図である。It is a figure which shows an example of the process sequence of the license plate area | region image process in 1st Embodiment of this technique. 本技術の第２の実施の形態における画像処理装置３００の構成例を示す図である。It is a figure showing an example of composition of image processing device 300 in a 2nd embodiment of this art. 本技術の第２の実施の形態におけるナンバープレート領域画像処理部３６０の構成例を示す図である。It is a figure showing an example of composition of number plate field image processing part 360 in a 2nd embodiment of this art. 本技術の第２の実施の形態におけるナンバープレート領域画像処理の処理手順の一例を示す図である。It is a figure which shows an example of the process sequence of the license plate area | region image process in 2nd Embodiment of this technique. 本技術の第３の実施の形態における合成処理を示す図である。It is a figure showing composition processing in a 3rd embodiment of this art. 本技術の第４の実施の形態におけるデモザイク処理を示す図である。It is a figure showing demosaic processing in a 4th embodiment of this art. 本技術の第４の実施の形態におけるデモザイク処理を示す図である。It is a figure showing demosaic processing in a 4th embodiment of this art.
　以下、本技術を実施するための形態（以下、実施の形態と称する）について説明する。説明は以下の順序により行う。
　１．第１の実施の形態（赤外光信号によりナンバープレート領域の画像を処理する場合の例）
　２．第２の実施の形態（赤外光信号または可視光信号によりナンバープレート領域の画像を処理する場合の例）
　３．第３の実施の形態（ナンバープレートの色を表示する場合の例）
　４．第４の実施の形態（白色光に対応する画素を有する撮像素子を使用する場合の例）
　５．変形例 Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order.
1. 1st Embodiment (example in the case of processing the image of a license plate area | region by an infrared-light signal)
2. Second embodiment (an example of processing an image in a license plate region by an infrared light signal or a visible light signal)
3. Third embodiment (example of displaying license plate color)
4). Fourth embodiment (example in the case of using an image sensor having pixels corresponding to white light)
5. Modified example
　＜１．第１の実施の形態＞
　［撮像装置の構成］
　図１は、本技術の実施の形態における撮像装置の構成例を示す図である。同図における撮像装置１０は、レンズ１００と、撮像素子２００と、画像処理装置３００と、画像信号出力部４００と、赤外光照射部５００と、制御部６００とを備えている。 <1. First Embodiment>
[Configuration of imaging device]
FIG. 1 is a diagram illustrating a configuration example of an imaging device according to an embodiment of the present technology. The imaging apparatus 10 in the figure includes a lens 100, an imaging element 200, an image processing apparatus 300, an image signal output unit 400, an infrared light irradiation unit 500, and a control unit 600.
　レンズ１００は、撮像素子２００に対して光学的に被写体を結像するものである。撮像素子２００は、レンズ１００によって結像された光学画像を画像信号に変換し、出力するものである。この撮像素子２００は、光学画像が結像される面に画像信号を生成する画素が２次元に配置されて構成されている。この画素から出力される画像信号には可視光成分に対応する可視光信号および赤外光成分に対応する赤外信号が含まれる。 The lens 100 optically forms a subject on the image sensor 200. The image sensor 200 converts an optical image formed by the lens 100 into an image signal and outputs the image signal. The imaging device 200 is configured by two-dimensionally arranging pixels that generate image signals on a surface on which an optical image is formed. The image signal output from this pixel includes a visible light signal corresponding to the visible light component and an infrared signal corresponding to the infrared light component.
　このような画素を有する撮像素子２００として、次のような撮像素子を使用することができる。例えば、赤色光と赤外光に対応する画像信号を出力する画素、緑色光と赤外光に対応する画像信号を出力する画素、青色光と赤外光に対応する画像信号を出力する画素および赤外光に対応する画像信号を出力する画素を有する撮像素子を使用することができる。また、例えば、これらの画素に白色光と赤外光に対応する画像信号を出力する画素を加えた合計５種の画素を有する撮像素子を使用することもできる。 The following image sensor can be used as the image sensor 200 having such pixels. For example, a pixel that outputs image signals corresponding to red light and infrared light, a pixel that outputs image signals corresponding to green light and infrared light, a pixel that outputs image signals corresponding to blue light and infrared light, and An image sensor having a pixel that outputs an image signal corresponding to infrared light can be used. In addition, for example, an imaging device having a total of five types of pixels in which pixels that output image signals corresponding to white light and infrared light are added to these pixels may be used.
　以下、赤色光と赤外光に対応する画像信号、緑色光と赤外光に対応する画像信号および青色光と赤外光に対応する画像信号をそれぞれＲ＋ＩＲ信号、Ｇ＋ＩＲ信号およびＢ＋ＩＲ信号と称する。また、白色光と赤外光に対応する画像信号および赤外光に対応する画像信号をそれぞれＷ＋ＩＲ信号およびＩＲ信号と称する。さらに、Ｒ＋ＩＲ信号、Ｇ＋ＩＲ信号、Ｂ＋ＩＲ信号、Ｗ＋ＩＲ信号およびＩＲ信号を出力する画素をそれぞれＲ＋ＩＲ画素、Ｇ＋ＩＲ画素、Ｂ＋ＩＲ画素、Ｗ＋ＩＲ画素およびＩＲ画素と称する。このように異なる光に対応した画素として構成するために、各画素にはカラーフィルタが備えられている。このカラーフィルタの分光特性を特定の光に対応させて、上述した４種または５種の画素を構成している。 Hereinafter, an image signal corresponding to red light and infrared light, an image signal corresponding to green light and infrared light, and an image signal corresponding to blue light and infrared light are referred to as an R + IR signal, a G + IR signal, and a B + IR signal, respectively. In addition, an image signal corresponding to white light and infrared light and an image signal corresponding to infrared light are referred to as a W + IR signal and an IR signal, respectively. Furthermore, pixels that output the R + IR signal, the G + IR signal, the B + IR signal, the W + IR signal, and the IR signal are referred to as an R + IR pixel, a G + IR pixel, a B + IR pixel, a W + IR pixel, and an IR pixel, respectively. In order to configure as pixels corresponding to different light in this way, each pixel is provided with a color filter. The above-described four types or five types of pixels are configured so that the spectral characteristics of the color filter correspond to specific light.
　画像処理装置３００は、画像信号を処理するものである。この画像処理装置３００は、撮像素子２００が出力した画像信号を輝度信号および色差信号に変換し、これらの信号を処理して出力する。 The image processing apparatus 300 processes an image signal. The image processing apparatus 300 converts the image signal output from the image sensor 200 into a luminance signal and a color difference signal, and processes and outputs these signals.
　画像信号出力部４００は、画像処理装置３００により処理された画像信号を撮像装置１０の外部に出力するものである。この画像信号出力部４００は、撮像装置１０が接続される信号線インターフェースの規格に適合するように画像信号を変換して出力する。 The image signal output unit 400 outputs the image signal processed by the image processing apparatus 300 to the outside of the imaging apparatus 10. The image signal output unit 400 converts and outputs an image signal so as to conform to the standard of the signal line interface to which the imaging device 10 is connected.
　制御部６００は、撮像装置１０の全体を制御するものである。赤外光照射部５００は、被写体に赤外光を照射するものである。この赤外光照射部５００は、制御部６００により制御される。 The control unit 600 controls the entire imaging apparatus 10. The infrared light irradiation unit 500 irradiates the subject with infrared light. The infrared light irradiation unit 500 is controlled by the control unit 600.
　［撮像素子の構成］
　図２は、本技術の実施の形態における撮像素子２００の構成例を示す図である。同図は、撮像素子２００の画面上の画素２０１の配置を表したものである。なお、画素２０１に記載された記号は、画素２０１の種類を表すものである。Ｒ+ＩＲ、Ｇ+ＩＲ、Ｂ+ＩＲ、Ｗ+ＩＲおよびＩＲと表記された画素は、それぞれＲ+ＩＲ画素、Ｇ+ＩＲ画素、Ｂ+ＩＲ画素、Ｗ+ＩＲ画素およびＩＲ画素を表している。これらの画素が画面上に一定の規則に基づいて配置されている。 [Configuration of image sensor]
FIG. 2 is a diagram illustrating a configuration example of the image sensor 200 according to the embodiment of the present technology. The figure shows the arrangement of the pixels 201 on the screen of the image sensor 200. Note that the symbol written in the pixel 201 represents the type of the pixel 201. Pixels labeled R + IR, G + IR, B + IR, W + IR and IR represent R + IR pixel, G + IR pixel, B + IR pixel, W + IR pixel and IR pixel, respectively. Yes. These pixels are arranged on the screen based on a certain rule.
　同図におけるａは、Ｒ+ＩＲ画素、Ｇ+ＩＲ画素およびＢ+ＩＲ画素のベイヤー配列において、２つあるＧ+ＩＲ画素の一方をＩＲ画素に置き換えたものである。赤、緑および青に対応する画素がバランスよく配置されており、色再現性が高い配列となる。同図におけるｂは、Ｇ＋ＩＲ画素の市松配列において、Ｇ＋ＩＲ画素の画素位置以外においてＲ+ＩＲ画素、Ｂ+ＩＲ画素およびＩＲ画素が配置されたものである。緑に対応する画素が多いため、解像度の高い配列となる。同図におけるｃは、ＩＲ画素の市松配列において、ＩＲ画素の画素位置以外においてＲ+ＩＲ画素、Ｇ+ＩＲ画素およびＢ+ＩＲ画素が配置されたものである。ＩＲ画素が多く配置されており、赤外光に対する解像度が高い配列となる。同図におけるｄは、Ｗ＋ＩＲ画素の市松配列において、Ｗ＋ＩＲ画素の画素位置以外においてＲ+ＩＲ画素、Ｇ+ＩＲ画素、Ｂ+ＩＲ画素およびＩＲ画素が配置されたものである。白に対応する画素が多く配置されており、感度が高い配列となる。 A in the figure is obtained by replacing one of the two G + IR pixels with an IR pixel in the Bayer array of R + IR pixels, G + IR pixels, and B + IR pixels. Pixels corresponding to red, green, and blue are arranged in a well-balanced manner, resulting in an array with high color reproducibility. In the figure, b is a checkered arrangement of G + IR pixels in which R + IR pixels, B + IR pixels, and IR pixels are arranged at positions other than the pixel positions of the G + IR pixels. Since there are many pixels corresponding to green, the array has a high resolution. In the figure, c is a checkered array of IR pixels in which R + IR pixels, G + IR pixels, and B + IR pixels are arranged at positions other than the pixel positions of the IR pixels. A large number of IR pixels are arranged, and the array has a high resolution with respect to infrared light. In the figure, d is a checkered arrangement of W + IR pixels in which R + IR pixels, G + IR pixels, B + IR pixels, and IR pixels are arranged at positions other than the pixel positions of the W + IR pixels. Many pixels corresponding to white are arranged, and the arrangement is high in sensitivity.
　なお、本技術の第１の実施の形態における撮像素子２００としては、同図におけるａ乃至ｃに挙げたＲ+ＩＲ画素、Ｇ+ＩＲ画素、Ｂ+ＩＲ画素およびＩＲ画素を有する撮像素子を想定する。 Note that the imaging device 200 according to the first embodiment of the present technology is assumed to be an imaging device having the R + IR pixel, the G + IR pixel, the B + IR pixel, and the IR pixel listed in a to c in FIG. To do.
　［画像処理装置の構成］
　図３は、本技術の第１の実施の形態における画像処理装置３００の構成例を示す図である。この画像処理装置３００は、撮像素子制御部３１０と、動き画像検出部３２０と、デモザイク部３３０と、ナンバープレート領域検出部３４０とを備えている。また、この画像処理装置３００は、ナンバープレート領域画像処理部３６０と、車体領域検出部３５０と、車体領域画像処理部３７０と、合成部３９０とをさらに備えている。 [Configuration of image processing apparatus]
FIG. 3 is a diagram illustrating a configuration example of the image processing device 300 according to the first embodiment of the present technology. The image processing apparatus 300 includes an image sensor control unit 310, a motion image detection unit 320, a demosaic unit 330, and a license plate area detection unit 340. The image processing apparatus 300 further includes a license plate region image processing unit 360, a vehicle body region detection unit 350, a vehicle body region image processing unit 370, and a combining unit 390.
　撮像素子制御部３１０は、撮像素子２００の制御を行うものである。この撮像素子制御部３１０は、撮像素子２００に対して、フォーカス制御、露出制御、画像信号の出力制御等を行う。 The image sensor control unit 310 controls the image sensor 200. The image sensor control unit 310 performs focus control, exposure control, image signal output control, and the like on the image sensor 200.
　動き画像検出部３２０は、撮像素子２００から出力された画像信号において動きのある画像領域を検出するものである。この動きのある画像領域の検出には、公知の方法、例えば、ＭＰＥＧ（Moving Picture Experts Group）等において使用されるブロックマッチングによる方法を使用することができる。 The motion image detection unit 320 detects a moving image area in the image signal output from the image sensor 200. A known method, for example, a method based on block matching used in MPEG (Moving Picture Experts Group) or the like can be used to detect this moving image area.
　デモザイク部３３０は、デモザイク処理を行うものである。このデモザイク処理は、撮像素子２００から出力された単色の画像信号に対して不足する画像信号を補間するものである。このデモザイク処理により、１画素当たりの画像信号は、Ｒ+ＩＲ信号、Ｇ+ＩＲ信号、Ｂ+ＩＲ信号およびＩＲ信号の４つに増加することになる。また、デモザイク部３３０は、Ｒ+ＩＲ信号、Ｇ+ＩＲ信号およびＢ+ＩＲ信号から赤外光成分を除去する赤外光成分除去処理をさらに行う。ここで、Ｒ+ＩＲ信号、Ｇ+ＩＲ信号およびＢ+ＩＲ信号から赤外光成分が除去された信号をそれぞれＲ信号、Ｇ信号およびＢ信号と称する。この赤外光成分除去処理は、例えば、次式のように行うことができる。
　　Ｒ＝Ｒ＋ＩＲ－ＩＲ
　　Ｇ＝Ｇ＋ＩＲ－ＩＲ
　　Ｂ＝Ｂ＋ＩＲ－ＩＲ
ただし、Ｒ、Ｇ、ＢおよびＩＲは、それぞれＲ信号、Ｇ信号、Ｂ信号およびＩＲ信号を表している。Ｒ＋ＩＲ、Ｇ＋ＩＲおよびＢ＋ＩＲは、それぞれＲ＋ＩＲ信号、Ｇ＋ＩＲ信号およびＢ＋ＩＲ信号を表している。 The demosaic unit 330 performs demosaic processing. This demosaic process is to interpolate a deficient image signal with respect to a monochromatic image signal output from the image sensor 200. By this demosaic processing, the image signal per pixel increases to four, that is, an R + IR signal, a G + IR signal, a B + IR signal, and an IR signal. Further, the demosaic unit 330 further performs an infrared light component removal process for removing an infrared light component from the R + IR signal, the G + IR signal, and the B + IR signal. Here, signals obtained by removing infrared light components from the R + IR signal, the G + IR signal, and the B + IR signal are referred to as an R signal, a G signal, and a B signal, respectively. This infrared light component removal processing can be performed, for example, as in the following equation.
R = R + IR -IR
G = G + IR -IR
B = B + IR -IR
Here, R, G, B, and IR represent an R signal, a G signal, a B signal, and an IR signal, respectively. R + IR , G + IR and B + IR represent the R + IR signal, the G + IR signal and the B + IR signal, respectively.
　車体領域検出部３５０は、動き画像検出部３２０により検出された動きのある画像領域から車体領域を検出するものである。この車体領域の検出には、公知の方法、例えば、車体の画像を濃淡画像で表した濃淡テンプレートを保持し、入力された画像および濃淡テンプレートのパターンマッチングを行うことにより、車体領域を検出する方法を用いることができる。 The vehicle body area detection unit 350 detects the vehicle body area from the moving image area detected by the motion image detection unit 320. For the detection of the vehicle body region, a known method, for example, a method of detecting the vehicle body region by holding a light / dark template representing a vehicle image as a light / dark image and performing pattern matching between the input image and the light / dark template. Can be used.
　ナンバープレート領域検出部３４０は、動き画像検出部３２０により検出された動きのある画像領域からナンバープレート領域を検出するものである。このナンバープレート領域の検出には、公知の方法を使用することができる。例えば、車体の画像信号を２値化した画像を生成し、この画像の領域においてナンバープレート特有のパターン配列に一致する部分を検索することにより、ナンバープレート位置を検出する方法を用いることができる。 The license plate area detector 340 detects the license plate area from the moving image area detected by the motion image detector 320. A known method can be used to detect the license plate region. For example, it is possible to use a method of detecting the license plate position by generating an image obtained by binarizing the image signal of the vehicle body and searching for a portion matching the pattern arrangement unique to the license plate in this image area.
　車体領域画像処理部３７０は、車体領域検出部３５０が検出した車体領域に対応する画像信号を処理するものである。この車体領域画像処理部３７０は、車体における色の再現性を向上させる処理を行う。この車体領域画像処理の詳細については後述する。 The vehicle body region image processing unit 370 processes an image signal corresponding to the vehicle body region detected by the vehicle body region detection unit 350. The vehicle body region image processing unit 370 performs processing for improving color reproducibility in the vehicle body. Details of the vehicle body region image processing will be described later.
　ナンバープレート領域画像処理部３６０は、ナンバープレート領域検出部３４０により検出されたナンバープレート領域に対応する画像信号を処理するものである。このナンバープレート領域画像処理部３６０は、上述の車体領域画像処理部３７０における処理とは異なる処理を行う。すなわち、ナンバープレートにおける視認性を向上させる処理を行う。このナンバープレート領域画像処理の詳細については後述する。 The license plate area image processing unit 360 processes an image signal corresponding to the license plate area detected by the license plate area detection unit 340. The license plate region image processing unit 360 performs processing different from the processing in the vehicle body region image processing unit 370 described above. That is, processing for improving the visibility of the license plate is performed. Details of the license plate area image processing will be described later.
　合成部３９０は、車体領域画像処理部３７０およびナンバープレート領域画像処理部３６０により処理された画像信号を合成するものである。 The synthesizing unit 390 synthesizes the image signals processed by the vehicle body region image processing unit 370 and the license plate region image processing unit 360.
　なお、上述した画像処理装置３００における各処理は、これらの処理を行うハードウェアにより実現することができる。また、マイコンやＤＳＰを搭載し、ソフトウェアによる処理として実現してもよい。 Each process in the image processing apparatus 300 described above can be realized by hardware that performs these processes. Also, a microcomputer or DSP may be installed and realized as software processing.
　［デモザイク処理］
　図４は、本技術の第１の実施の形態におけるデモザイク処理を示す図である。同図は、画素２０２に対してデモザイク処理を行う場合を表したものである。この画素２０２は、Ｂ+ＩＲ画素であり、Ｂ+ＩＲ信号を出力する。この画素２０２の画素位置に、他の色に対応する画像信号を補間する。同図におけるａは、画素２０２に対してＧ+ＩＲ信号を補間する場合の例である。画素２０２の左右に配置されたＧ＋ＩＲ画素におけるＧ＋ＩＲ信号の平均値を画素２０２におけるＧ＋ＩＲ信号にする。同図におけるｂは、画素２０２に対してＩＲ信号を補間する場合の例である。画素２０２の上下に配置されたＩＲ画素におけるＩＲ信号の平均値を画素２０２におけるＩＲ信号にする。同図におけるｃは、画素２０２に対してＲ＋ＩＲ信号を補間する場合の例である。画素２０２の斜め方向において画素２０２と隣接するＲ＋ＩＲ信号の平均値を画素２０２におけるＲ＋ＩＲ信号にする。このように、同図におけるデモザイク処理は、隣接する画素の画像信号のみを用いて補間を行う。このデモザイク処理の後、赤外光成分除去処理が行われる。 [Demosaic processing]
FIG. 4 is a diagram illustrating a demosaic process according to the first embodiment of the present technology. This figure shows a case where a demosaic process is performed on the pixel 202. The pixel 202 is a B + IR pixel and outputs a B + IR signal. An image signal corresponding to another color is interpolated at the pixel position of the pixel 202. In the figure, a is an example in the case of interpolating a G + IR signal for the pixel 202. The average value of the G + IR signals in the G + IR pixels arranged on the left and right of the pixel 202 is set as the G + IR signal in the pixel 202. B in the figure is an example in the case of interpolating the IR signal for the pixel 202. The average value of the IR signals in the IR pixels arranged above and below the pixel 202 is set as the IR signal in the pixel 202. C in the figure is an example when the R + IR signal is interpolated for the pixel 202. The average value of the R + IR signals adjacent to the pixel 202 in the diagonal direction of the pixel 202 is set to the R + IR signal in the pixel 202. As described above, the demosaic process in FIG. 9 performs interpolation using only image signals of adjacent pixels. After this demosaic processing, infrared light component removal processing is performed.
　［ナンバープレート領域画像処理部］
　図５は、本技術の第１の実施の形態におけるナンバープレート領域画像処理部３６０の構成例を示す図である。このナンバープレート領域画像処理部３６０は、輝度信号変換部３６２と、エッジ強調部３６３と、２次元ノイズリダクション部３６４とを備えている。 [Number plate area image processing section]
FIG. 5 is a diagram illustrating a configuration example of the license plate area image processing unit 360 according to the first embodiment of the present technology. The license plate area image processing unit 360 includes a luminance signal conversion unit 362, an edge enhancement unit 363, and a two-dimensional noise reduction unit 364.
　輝度信号変換部３６２は、ナンバープレート領域に対応する画像信号のうちＩＲ信号を輝度信号（Ｙ信号）に変換するものである。この変換は、次式のように行う。
　　Ｙ＝ＩＲ
ただし、ＹおよびＩＲはそれぞれＹ信号およびＩＲ信号を表す。このように、輝度信号変換部３６２は、ナンバープレート領域のＩＲ信号をＹ信号に変換している。色差信号を使用していないため、ナンバープレート領域画像処理後のナンバープレート領域の画像は、モノクロの画像になる。 The luminance signal conversion unit 362 converts the IR signal among the image signals corresponding to the license plate region into a luminance signal (Y signal). This conversion is performed as follows.
Y = IR
Y and IR represent a Y signal and an IR signal, respectively. In this way, the luminance signal conversion unit 362 converts the IR signal in the license plate area into a Y signal. Since no color difference signal is used, the license plate area image after the license plate area image processing is a monochrome image.
　エッジ強調部３６３は、ナンバープレート領域のＹ信号に対してエッジ強調処理を行うものである。なお、同図におけるＹ＿ｅｄｇｅ信号は、エッジ強調処理が行われたＹ信号を表している。このエッジ強調処理は、画像のエッジ部分を強調する処理である。このエッジ強調処理には、公知の方法を用いることができる。例えば、画像からエッジ部分を検出し、この検出されたエッジ部分の輝度を高めることによりエッジ部分を強調する方法を用いることができる。 The edge emphasizing unit 363 performs edge emphasis processing on the Y signal in the license plate area. The Y_edge signal in the figure represents the Y signal that has undergone edge enhancement processing. This edge enhancement process is a process for enhancing an edge portion of an image. A known method can be used for this edge enhancement processing. For example, a method can be used in which an edge portion is detected from an image and the edge portion is enhanced by increasing the luminance of the detected edge portion.
　２次元ノイズリダクション部３６４は、Ｙ＿ｅｄｇｅ信号に対して２次元ノイズリダクション処理を行うものである。なお、同図におけるＹ＿ｎｒ２信号は、２次元ノイズリダクション処理が行われたＹ信号を表している。ここで、ノイズリダクション処理とは、重畳しているノイズの除去を行う処理である。また、２次元ノイズリダクション処理は、同一フレームに属する画像信号の高周波成分を減衰させることによりノイズを除去する処理である。 The two-dimensional noise reduction unit 364 performs a two-dimensional noise reduction process on the Y_edge signal. The Y_nr2 signal in the figure represents the Y signal that has been subjected to the two-dimensional noise reduction process. Here, the noise reduction process is a process for removing superimposed noise. The two-dimensional noise reduction process is a process for removing noise by attenuating high-frequency components of image signals belonging to the same frame.
　［２次元ノイズリダクション処理］
　図６は、本技術の第１の実施の形態における２次元ノイズリダクション処理を示す図である。同図においては、２次元ノイズリダクション処理としてイプシロンフィルタを使用した場合を例示する。同図におけるａは、２次元ノイズリダクション処理のブロック図を表したものである。なお、入力信号線３０３にはＹ信号が入力される。この入力された信号をハイパスフィルタ３０４およびイプシロンフィルタ３０６に通すことにより、入力信号から高周波成分が分離される。なお、イプシロンフィルタ３０６は、入力された信号に対して所定のレベルを超える信号の除去を行うフィルタである。次に、減算器３０８により、この分離された高周波成分を入力信号から減算する。これにより、ノイズが除去される。 [2D noise reduction processing]
FIG. 6 is a diagram illustrating a two-dimensional noise reduction process according to the first embodiment of the present technology. In the figure, a case where an epsilon filter is used as a two-dimensional noise reduction process is illustrated. A in the same figure represents the block diagram of a two-dimensional noise reduction process. Note that a Y signal is input to the input signal line 303. By passing the input signal through the high-pass filter 304 and the epsilon filter 306, high frequency components are separated from the input signal. The epsilon filter 306 is a filter that removes a signal that exceeds a predetermined level with respect to an input signal. Next, the subtracter 308 subtracts the separated high frequency component from the input signal. Thereby, noise is removed.
　この様子を同図におけるｂに表した。なお、信号線３０５および３０７は、それぞれハイパスフィルタ３０４およびイプシロンフィルタ３０６の出力に接続された信号線である。また、出力信号線３０９は、２次元ノイズリダクション部３６４の出力に接続された信号線である。この２次元ノイズリダクション処理は、後述する３次元ノイズリダクション処理と比較して、ノイズ除去能力は劣るものの、画像のエッジ部分のぼけが比較的少なく、視認性の低下を防ぐことができる。なお、本技術の第１の実施の形態においては、これ以外の２次元ノイズリダクション処理、例えば、バイラテラルフィルタによる処理や平滑化による処理を使用することもできる。 This is shown in b in the figure. Signal lines 305 and 307 are signal lines connected to the outputs of the high-pass filter 304 and the epsilon filter 306, respectively. The output signal line 309 is a signal line connected to the output of the two-dimensional noise reduction unit 364. Although this two-dimensional noise reduction process is inferior to the three-dimensional noise reduction process, which will be described later, the noise removal capability is inferior, the blur of the edge portion of the image is relatively small, and a reduction in visibility can be prevented. Note that in the first embodiment of the present technology, other two-dimensional noise reduction processing, for example, processing using a bilateral filter or processing using smoothing may be used.
　エッジ強調処理や２次元ノイズリダクション処理により、ナンバープレート領域の画像を、モノクロながら視認性が向上された画像にすることができる。 The image in the license plate area can be converted into a monochrome image with improved visibility by edge enhancement processing and two-dimensional noise reduction processing.
　［車体領域画像処理部］
　図７は、本技術の第１の実施の形態における車体領域画像処理部３７０の構成例を示す図である。この車体領域画像処理部３７０は、輝度色差信号変換部３７２と、２次元ノイズリダクション部３７４と、３次元ノイズリダクション部３７５とを備えている。 [Body area image processing unit]
FIG. 7 is a diagram illustrating a configuration example of the vehicle body region image processing unit 370 according to the first embodiment of the present technology. The vehicle body region image processing unit 370 includes a luminance / color difference signal conversion unit 372, a two-dimensional noise reduction unit 374, and a three-dimensional noise reduction unit 375.
　輝度色差信号変換部３７２は、車体領域に対応する画像信号のうちＲ、ＧおよびＢ信号を輝度信号（Ｙ信号）および色差信号（ＣｂおよびＣｒ信号）に変換するものである。なお、Ｃｂ信号はＢ信号とＹ信号との差分に基づく信号であり、Ｃｒ信号はＲ信号とＹ信号との差分に基づく信号である。この変換は、次式のように行うことができる。
　　Ｙ　＝　０．２９９０×Ｒ＋０．５８７０×Ｇ＋０．１１４０×Ｂ
　　Ｃｂ＝－０．１６９０×Ｒ－０．３３１６×Ｇ＋０．５０００×Ｂ
　　Ｃｒ＝　０．５０００×Ｒ－０．４１８６×Ｇ－０．０８１３×Ｂ The luminance / color difference signal conversion unit 372 converts the R, G, and B signals among the image signals corresponding to the vehicle body region into luminance signals (Y signals) and color difference signals (Cb and Cr signals). The Cb signal is a signal based on the difference between the B signal and the Y signal, and the Cr signal is a signal based on the difference between the R signal and the Y signal. This conversion can be performed as follows.
Y = 0.2990 × R + 0.5870 × G + 0.1140 × B
Cb = -0.1690 × R-0.3316 × G + 0.5000 × B
Cr = 0.5000 x R-0.4186 x G-0.0813 x B
　２次元ノイズリダクション部３７４は、前述した２次元ノイズリダクション処理を行うものである。この２次元ノイズリダクション部３７４は、輝度色差信号変換部３７２から出力されたＹ、ＣｂおよびＣｒ信号に対して２次元ノイズリダクション処理を行う。なお、同図のＹ＿ｎｒ２、Ｃｂ＿ｎｒ２およびＣｒ＿ｎｒ２信号は、それぞれ２次元ノイズリダクション処理が行われたＹ、ＣｂおよびＣｒ信号を表している。 The two-dimensional noise reduction unit 374 performs the two-dimensional noise reduction process described above. The two-dimensional noise reduction unit 374 performs a two-dimensional noise reduction process on the Y, Cb, and Cr signals output from the luminance / color difference signal conversion unit 372. The Y_nr2, Cb_nr2 and Cr_nr2 signals in the figure represent Y, Cb and Cr signals that have been subjected to two-dimensional noise reduction processing, respectively.
　３次元ノイズリダクション部３７５は、３次元ノイズリダクション処理を行うものである。この３次元ノイズリダクション部３７５は、２次元ノイズリダクション部３７４から出力されたＹ＿ｎｒ２、Ｃｂ＿ｎｒ２およびＣｒ＿ｎｒ２信号に対して、３次元ノイズリダクション処理を行う。この３次元ノイズリダクション処理は、連続する複数のフレームに属する画像信号を用いてノイズの除去を行う処理である。なお、同図のＹ＿ｎｒ３、Ｃｂ＿ｎｒ３およびＣｒ＿ｎｒ３信号は、それぞれ３次元ノイズリダクション処理が行われたＹ、ＣｂおよびＣｒ信号を表している。 The 3D noise reduction unit 375 performs 3D noise reduction processing. The three-dimensional noise reduction unit 375 performs a three-dimensional noise reduction process on the Y_nr2, Cb_nr2, and Cr_nr2 signals output from the two-dimensional noise reduction unit 374. This three-dimensional noise reduction process is a process for removing noise using image signals belonging to a plurality of consecutive frames. The Y_nr3, Cb_nr3, and Cr_nr3 signals in the figure represent Y, Cb, and Cr signals that have been subjected to three-dimensional noise reduction processing, respectively.
　［３次元ノイズリダクション処理］
　図８は、本技術の第１の実施の形態における３次元ノイズリダクション処理を示す図である。同図における画像７０１乃至７０３は、車体領域に対応する画像である。同図に表したように、Ｋ個の連続するフレームの車体領域に対応する画像の画像信号が加算器３７８によって加算される。その後、除算器３７９により、加算した画像信号数であるＫで除算され、出力画像７０４が生成される。これにより、ランダムに発生したノイズ成分を平均化してノイズ除去を行うことができる。３次元ノイズリダクション処理は、加算する画像信号数Ｋが大きいほど、ノイズ除去の効果が高まる性質があり、２次元ノイズリダクション処理よりも高いノイズ除去能力にすることができる。一方、加算する画像信号数Ｋが大きいほど画像が平準化され、画像がぼやける等画質が低下するという問題がある。しかし、車体領域に関しては、色の再現が可能であれば、多少の画質の低下は許容できるため、問題にはならない。 [3D noise reduction processing]
FIG. 8 is a diagram illustrating a three-dimensional noise reduction process according to the first embodiment of the present technology. Images 701 to 703 in the figure are images corresponding to the vehicle body region. As shown in the figure, an image signal of an image corresponding to the body area of K consecutive frames is added by an adder 378. Thereafter, a divider 379 divides by K which is the number of added image signals, and an output image 704 is generated. As a result, it is possible to perform noise removal by averaging randomly generated noise components. The three-dimensional noise reduction process has a property that the effect of noise removal increases as the number K of image signals to be added increases, and can have a higher noise removal capability than the two-dimensional noise reduction process. On the other hand, there is a problem that as the number K of image signals to be added is larger, the image is leveled and the image quality is degraded, such as the image being blurred. However, with respect to the vehicle body region, if color reproduction is possible, a slight deterioration in image quality can be tolerated.
　また、動きのある画像に対して３次元ノイズリダクション処理を行う場合にも、同様に画質が低下する。そこで、車体の動きを補償して、上述の加算を行うことにより、画質の低下を防ぐことができる。この動き補償については、公知の方法、例えば、ＭＰＥＧに使用される動き補償方法を使用することができる。 Also, when the 3D noise reduction process is performed on a moving image, the image quality similarly decreases. Therefore, it is possible to prevent deterioration in image quality by compensating for the movement of the vehicle body and performing the above-described addition. For this motion compensation, a known method, for example, a motion compensation method used in MPEG can be used.
　これら２次元ノイズリダクション処理および３次元ノイズリダクション処理により、車体領域の画像を、色再現性が向上された画像にすることができる。 These two-dimensional noise reduction processing and three-dimensional noise reduction processing can turn the image of the vehicle body region into an image with improved color reproducibility.
　［合成処理］
　図９は、本技術の第１の実施の形態における合成処理を示す図である。同図は、画像処理装置３００の各部における画像信号の様子と合成部３９０における合成処理を表したものである。車輌を含む画像７０５を構成する画像信号が動き画像検出部３２０に入力され、動きのある画像７０６に対応する画像信号が出力される。この画像信号は、車体領域検出部３５０および車体領域画像処理部３７０により処理されて、車体における色の再現性を向上させた画像７０７に対応する画像信号になる。これと平行して、動きのある画像７０６に対応する画像信号は、ナンバープレート領域検出部３４０およびナンバープレート領域画像処理部３６０により処理されて、ナンバープレートにおける視認性を向上させた画像７０８に対応する画像信号になる。 [Composition process]
FIG. 9 is a diagram illustrating a composition process according to the first embodiment of the present technology. The figure shows the state of the image signal in each unit of the image processing apparatus 300 and the synthesis process in the synthesis unit 390. An image signal constituting the image 705 including the vehicle is input to the motion image detection unit 320, and an image signal corresponding to the image 706 having motion is output. This image signal is processed by the vehicle body region detection unit 350 and the vehicle body region image processing unit 370 to become an image signal corresponding to the image 707 with improved color reproducibility in the vehicle body. In parallel with this, the image signal corresponding to the moving image 706 is processed by the license plate area detection unit 340 and the license plate area image processing unit 360 to correspond to the image 708 with improved visibility on the license plate. Image signal.
　これら２つの画像信号が合成部３９０により合成されて出力される。なお、画像７０９は合成後の画像を表している。この合成部３９０による合成は、例えば、次のように行うことができる。入力画像７０５を背景画像とし、この画像に車体領域の画像７０７およびナンバープレート領域の画像７０８を重ね合わせて、画像７０９を合成する。これにより、車体領域について色再現性を向上させた画像とナンバープレート領域について視認性を向上させた画像とが同一の画像に表されることになる。なお、ナンバープレート領域検出部３４０においてナンバープレート領域が検出されない場合には、合成部３９０は、車体領域の画像７０７のみを背景画像に重ね合わせて合成を行う。 These two image signals are synthesized by the synthesis unit 390 and output. Note that an image 709 represents a combined image. The synthesis by the synthesis unit 390 can be performed as follows, for example. An input image 705 is used as a background image, and an image 709 is synthesized by superimposing an image 707 in the vehicle body area and an image 708 in the license plate area on this image. As a result, the image with improved color reproducibility for the vehicle body region and the image with improved visibility for the license plate region are represented in the same image. If the license plate area is not detected by the license plate area detection unit 340, the synthesis unit 390 performs synthesis by superimposing only the image 707 of the vehicle body area on the background image.
　［画像の表示］
　図１０は、本技術の第１の実施の形態における画像表示例を示す図である。同図におけるａは、画像表示領域７１１および画像情報表示領域７１２を含む画像表示の例を表したものである。画像表示領域７１１は、合成部３９０により出力された画像信号により構成された画像を表示する領域であり、車体領域画像処理部３７０により処理された画像７１５とナンバープレート領域画像処理部３６０により処理された画像７１６とを含んだ画像が表示される。画像情報表示領域７１２は、日付等の情報を表示する領域である。同図におけるｂは、同図におけるａの表示に対して、さらにナンバープレート表示領域７１３を付加した表示の例を表している。このナンバープレート表示領域７１３には、画像７１６を拡大した画像７１４が表示されている。このため、同図におけるｂの表示では、ナンバープレートの文字列の読取りが容易な構成になっている。 [Display image]
FIG. 10 is a diagram illustrating an image display example according to the first embodiment of the present technology. In the figure, a represents an example of image display including the image display area 711 and the image information display area 712. The image display area 711 is an area for displaying an image constituted by the image signal output from the synthesizing unit 390, and is processed by the image 715 processed by the vehicle body area image processing unit 370 and the license plate area image processing unit 360. An image including the image 716 is displayed. The image information display area 712 is an area for displaying information such as date. In the figure, b represents an example of display in which a license plate display area 713 is further added to the display of a in the figure. In the license plate display area 713, an image 714 obtained by enlarging the image 716 is displayed. For this reason, in the display of b in the same figure, the character string of a license plate can be read easily.
　［画像処理手順］
　図１１は、本技術の第１の実施の形態における画像処理手順の一例を示す図である。画像処理装置３００は、撮像素子２００から１フレームの画像信号が入力されると、本処理を開始する。まず、動き画像検出部３２０が入力された画像信号から動きのある画像領域を検出する（ステップＳ９０１）。この際、動きのある画像領域が検出されない場合には（ステップＳ９０２：Ｎｏ）、画像処理装置３００は、ステップＳ９０６の処理に移行し、入力された画像信号を画像信号出力部４００に対して出力する（ステップＳ９０６）。 [Image processing procedure]
FIG. 11 is a diagram illustrating an example of an image processing procedure according to the first embodiment of the present technology. The image processing apparatus 300 starts this processing when an image signal of one frame is input from the image sensor 200. First, the moving image detection unit 320 detects a moving image area from the input image signal (step S901). At this time, if a moving image region is not detected (step S902: No), the image processing apparatus 300 proceeds to the process of step S906 and outputs the input image signal to the image signal output unit 400. (Step S906).
　一方、動きのある画像領域が検出された場合には（ステップＳ９０２：Ｙｅｓ）、車体領域検出部３５０が、車体領域の検出を行う（ステップＳ９０３）。この際、車体領域が検出されない場合には（ステップＳ９０４：Ｎｏ）、画像処理装置３００は、ステップＳ９０６の処理に移行し、入力された画像信号を画像信号出力部４００に対して出力する（ステップＳ９０６）。 On the other hand, when a moving image region is detected (step S902: Yes), the vehicle body region detection unit 350 detects the vehicle region (step S903). At this time, when the vehicle body area is not detected (step S904: No), the image processing apparatus 300 proceeds to the process of step S906 and outputs the input image signal to the image signal output unit 400 (step). S906).
　一方、車体領域が検出された場合には（ステップＳ９０４：Ｙｅｓ）、ナンバープレート領域検出部３４０がナンバープレート領域の検出を行う（ステップＳ９０５）。次に、画像処理装置３００は、車体およびナンバープレート領域処理を行い（ステップＳ９１０）、ステップＳ９０６の処理に移行する。ステップＳ９０６において、画像処理装置３００は、ステップＳ９１０により得られた画像を画像信号出力部４００に対して出力する（ステップＳ９０６）。その後、画像処理装置３００は、画像処理を終了する。 On the other hand, when the vehicle body area is detected (step S904: Yes), the license plate area detection unit 340 detects the license plate area (step S905). Next, the image processing apparatus 300 performs vehicle body and license plate region processing (step S910), and proceeds to the processing of step S906. In step S906, the image processing apparatus 300 outputs the image obtained in step S910 to the image signal output unit 400 (step S906). Thereafter, the image processing apparatus 300 ends the image processing.
　［車体およびナンバープレート領域処理］
　図１２は、本技術の第１の実施の形態における車体およびナンバープレート領域処理の処理手順の一例を示す図である。同図の処理は、図１１において説明したステップＳ９１０に対応する処理である。まず、車体領域画像処理部３７０が車体領域画像処理を行う（ステップＳ９２０）。次に、ナンバープレート領域画像処理部３６０が、ナンバープレート領域画像処理を行う（ステップＳ９３０）。次に、合成部３９０が車体領域画像処理およびナンバープレート領域画像処理により処理された画像を合成する（ステップＳ９１３）。この際、ナンバープレート領域検出部３４０においてナンバープレート領域が検出されない場合には、合成部３９０は、車体領域画像処理により処理された画像により合成を行う。その後、画像処理装置３００は、車体およびナンバープレート領域処理を終了する。 [Body and license plate area processing]
FIG. 12 is a diagram illustrating an example of a processing procedure of the vehicle body and license plate region processing according to the first embodiment of the present technology. The process in FIG. 11 corresponds to step S910 described in FIG. First, the vehicle body region image processing unit 370 performs vehicle body region image processing (step S920). Next, the license plate area image processing unit 360 performs license plate area image processing (step S930). Next, the combining unit 390 combines the images processed by the vehicle body region image processing and the license plate region image processing (step S913). At this time, when the license plate area detection unit 340 does not detect the license plate area, the synthesis unit 390 performs synthesis using the image processed by the vehicle body area image processing. Thereafter, the image processing apparatus 300 ends the vehicle body and license plate area processing.
　［車体領域画像処理］
　図１３は、本技術の第１の実施の形態における車体領域画像処理の処理手順の一例を示す図である。同図の処理は、図１２において説明したステップＳ９２０に対応する処理である。まず、車体領域画像処理部３７０は、Ｒ、ＧおよびＢ信号を輝度信号および色差信号に変換する（ステップＳ９２３）。次に、車体領域画像処理部３７０は、２次元ノイズリダクション処理を行う（ステップＳ９２５）。次に、車体領域画像処理部３７０は、３次元ノイズリダクション処理を行い（ステップＳ９２６）、車体領域画像処理を終了する。 [Body area image processing]
FIG. 13 is a diagram illustrating an example of a processing procedure of the vehicle body region image processing according to the first embodiment of the present technology. The process in FIG. 12 corresponds to step S920 described in FIG. First, the vehicle body region image processing unit 370 converts the R, G, and B signals into luminance signals and color difference signals (step S923). Next, the vehicle body region image processing unit 370 performs a two-dimensional noise reduction process (step S925). Next, the vehicle body region image processing unit 370 performs three-dimensional noise reduction processing (step S926), and ends the vehicle body region image processing.
　［ナンバープレート領域画像処理］
　図１４は、本技術の第１の実施の形態におけるナンバープレート領域画像処理の処理手順の一例を示す図である。同図の処理は、図１２において説明したステップＳ９３０に対応する処理である。まず、ナンバープレート領域画像処理部３６０は、ＩＲ信号を輝度信号に変換する（ステップＳ９３２）。次に、ナンバープレート領域画像処理部３６０は、エッジ強調処理を行う（ステップＳ９３４）。次に、ナンバープレート領域画像処理部３６０は、２次元ノイズリダクション処理を行い（ステップＳ９３５）、ナンバープレート領域画像処理を終了する。 [Number plate area image processing]
FIG. 14 is a diagram illustrating an example of a processing procedure of license plate area image processing according to the first embodiment of the present technology. The process in FIG. 12 corresponds to step S930 described in FIG. First, the license plate area image processing unit 360 converts the IR signal into a luminance signal (step S932). Next, the license plate area image processing unit 360 performs edge enhancement processing (step S934). Next, the license plate area image processing unit 360 performs two-dimensional noise reduction processing (step S935), and ends the license plate area image processing.
　このように、本技術の第１の実施の形態によれば、ナンバープレートにおける視認性および車体における色の再現性を向上させることができ、監視対象である車輌の特定を容易にすることができる。また、可視光と赤外光の両方に対応する撮像素子２００を使用し、可視光信号と赤外光信号とを分離して処理することにより、単一の撮像素子による撮像装置１０を構成することができる。これにより、撮像装置１０の構成を簡素化することができる。 As described above, according to the first embodiment of the present technology, the visibility on the license plate and the color reproducibility on the vehicle body can be improved, and the vehicle to be monitored can be easily specified. . Further, the imaging device 200 corresponding to both visible light and infrared light is used, and the visible light signal and the infrared light signal are separated and processed, thereby forming the imaging device 10 using a single imaging device. be able to. Thereby, the structure of the imaging device 10 can be simplified.
　［変形例］
　上述の実施の形態では、ナンバープレート領域検出部３４０は、デモザイク部３３０により処理された画像信号に対してナンバープレート領域の検出を行っていた。これに対し、車体領域検出部３５０により検出された車体領域に対応する画像信号に対してナンバープレート領域の検出を行ってもよい。ナンバープレート領域は車体領域に含まれるためである。これにより、ナンバープレート領域検出部３４０において処理を行う画像の領域が限定され、処理時間を短縮することができる。 [Modification]
In the above-described embodiment, the license plate area detection unit 340 detects the license plate area for the image signal processed by the demosaic unit 330. On the other hand, the license plate area may be detected for the image signal corresponding to the vehicle body area detected by the vehicle body area detection unit 350. This is because the license plate region is included in the vehicle body region. Thereby, the area of the image to be processed in the license plate area detection unit 340 is limited, and the processing time can be shortened.
　＜２．第２の実施の形態＞
　上述の実施の形態では、ＩＲ信号によりナンバープレート領域の画像を処理していた。これに対し、本技術の第２の実施の形態では、ＩＲ信号が飽和している際には、可視光信号（Ｒ、ＧおよびＢ信号）によりナンバープレート領域の画像の処理を行う。 <2. Second Embodiment>
In the above-described embodiment, the license plate image is processed by the IR signal. On the other hand, in the second embodiment of the present technology, when the IR signal is saturated, the image of the license plate region is processed with the visible light signal (R, G, and B signals).
　［画像処理装置の構成］
　図１５は、本技術の第２の実施の形態における画像処理装置３００の構成例を示す図である。同図の画像処理装置３００は、飽和検出部３８０をさらに備える点において図３において説明した画像処理装置３００と異なっている。 [Configuration of image processing apparatus]
FIG. 15 is a diagram illustrating a configuration example of the image processing device 300 according to the second embodiment of the present technology. The image processing apparatus 300 in the figure is different from the image processing apparatus 300 described in FIG. 3 in that it further includes a saturation detection unit 380.
　また、本技術の第２の実施の形態における撮像素子２００には、赤色光、緑色光、青色光および赤外光に対応する画素を有する撮像素子を使用する必要がある。さらに、赤色光、緑色光および青色光に対応する画素は赤外光を除去するカラーフィルタを備える構成にする必要がある。すなわち、図２において説明した撮像素子のうち図２におけるａ乃至ｃに表した配列の撮像素子を使用する必要がある。さらに、Ｒ＋ＩＲ、Ｇ＋ＩＲおよびＢ＋ＩＲが記載された画素は、それぞれ赤色光、緑色光および青色光を検出し、赤外光成分を検出しない構成とした撮像素子を使用する必要がある。図２おいて説明した撮像素子のＲ＋ＩＲ画素、Ｇ＋ＩＲ画素およびＢ＋ＩＲ画素は、それぞれ赤外光成分に対しても感度を有するため、過大な赤外光が入射した際に、ＩＲ画素を含む全ての画素において出力される画像信号が飽和状態になるためである。 Also, it is necessary to use an image sensor having pixels corresponding to red light, green light, blue light, and infrared light as the image sensor 200 in the second embodiment of the present technology. Furthermore, the pixel corresponding to red light, green light, and blue light needs to be configured to include a color filter that removes infrared light. That is, it is necessary to use the image pickup device having the arrangement shown in FIGS. 2A to 2C among the image pickup devices described in FIG. Furthermore, the pixels in which R + IR, G + IR, and B + IR are described need to use an image sensor configured to detect red light, green light, and blue light and not detect an infrared light component, respectively. The R + IR pixel, the G + IR pixel, and the B + IR pixel of the image sensor described in FIG. 2 have sensitivity to the infrared light component, respectively. Therefore, when excessive infrared light is incident, all of the IR pixels including the IR pixel are included. This is because the image signal output from the pixel is saturated.
　このような本技術の第２の実施の形態における撮像素子２００が出力する画像信号は、Ｒ信号、Ｇ信号、Ｂ信号およびＩＲ信号であるため、デモザイク部３３０においては、前述した赤外成分除去処理を省略することができる。これ以外の撮像装置１０および画像処理装置３００の構成は、本技術の第１の実施の形態において説明した撮像装置１０および画像処理装置３００と同様であるため説明を省略する。 Since the image signals output from the image sensor 200 according to the second embodiment of the present technology are the R signal, the G signal, the B signal, and the IR signal, the demosaic unit 330 removes the infrared component described above. Processing can be omitted. Other configurations of the imaging apparatus 10 and the image processing apparatus 300 are the same as those of the imaging apparatus 10 and the image processing apparatus 300 described in the first embodiment of the present technology, and thus the description thereof is omitted.
　飽和検出部３８０は、ＩＲ信号が飽和しているか否かを検出するものである。前述のように撮像素子制御部３１０が露出の制御を行っているため、通常は、ＩＲ信号の飽和は発生しない。しかし、赤外光照射部５００において赤外光を照射しながら撮影を行った際に、被写体である車輌と撮像装置１０との間の距離が近い場合には、一時的なＩＲ信号の飽和が発生する。このような場合においても、撮像素子制御部３１０の露出制御により、ＩＲ信号が飽和していない状態の画像信号を得ることができる。しかし、車輌が高速に侵入した場合には、撮像素子制御部３１０の露出制御が追従できず、ＩＲ信号が飽和した状態の画像信号のみが取得されることとなる。このような場合に、飽和検出部３８０は、ＩＲ信号の飽和を検出する。 The saturation detector 380 detects whether the IR signal is saturated. Since the image sensor control unit 310 controls the exposure as described above, normally, saturation of the IR signal does not occur. However, when imaging is performed while irradiating infrared light in the infrared light irradiation unit 500, if the distance between the vehicle that is the subject and the imaging device 10 is short, the saturation of the IR signal temporarily occurs. appear. Even in such a case, an image signal in which the IR signal is not saturated can be obtained by the exposure control of the image sensor control unit 310. However, when the vehicle enters at high speed, the exposure control of the image sensor control unit 310 cannot follow, and only the image signal in a state where the IR signal is saturated is acquired. In such a case, the saturation detector 380 detects the saturation of the IR signal.
　本技術の第２の実施の形態におけるナンバープレート領域画像処理部３６０は、飽和検出部３８０による検出の結果に基づいてナンバープレート領域の画像処理を行うものである。 The license plate region image processing unit 360 according to the second embodiment of the present technology performs license plate region image processing based on the detection result by the saturation detection unit 380.
　［ナンバープレート領域画像処理部の構成］
　図１６は、本技術の第２の実施の形態におけるナンバープレート領域画像処理部３６０の構成例を示す図である。このナンバープレート領域画像処理部３６０は、輝度信号変換部３６２の代わりに輝度信号変換部３６６を備える点において、図５において説明したナンバープレート領域画像処理部３６０と異なっている。 [Configuration of license plate image processing unit]
FIG. 16 is a diagram illustrating a configuration example of the license plate area image processing unit 360 according to the second embodiment of the present technology. The license plate area image processing unit 360 is different from the license plate area image processing unit 360 described with reference to FIG. 5 in that a luminance signal conversion unit 366 is provided instead of the luminance signal conversion unit 362.
　輝度信号変換部３６６は、Ｒ信号、Ｇ信号、Ｂ信号およびＩＲ信号をＹ信号に変換するものである。この際、飽和検出部３８０による検出の結果に基づいて、異なる変換を行う。この変換は、次のように行うことができる。
（ａ）飽和検出部３８０によりＩＲ信号の飽和が検出された時
　Ｒ信号、Ｇ信号およびＢ信号を用いて、次式のように変換を行う。
　　Ｙ＝０．２９９０×Ｒ＋０．５８７０×Ｇ＋０．１１４０×Ｂ
（ｂ）飽和検出部３８０によりＩＲ信号の飽和が検出されない時
　ＩＲ信号を用いて、次式のように変換を行う。
　　Ｙ＝ＩＲ The luminance signal conversion unit 366 converts the R signal, the G signal, the B signal, and the IR signal into a Y signal. At this time, different conversions are performed based on the detection result by the saturation detection unit 380. This conversion can be performed as follows.
(A) When the saturation of the IR signal is detected by the saturation detection unit 380, conversion is performed using the R signal, the G signal, and the B signal as shown in the following equation.
Y = 0.2990 × R + 0.5870 × G + 0.1140 × B
(B) When saturation of IR signal is not detected by saturation detector 380 Using the IR signal, conversion is performed as shown in the following equation.
Y = IR
　このように、輝度信号変換部３６６は、図５において説明した輝度信号変換部３６２とは異なり、ＩＲ信号が飽和している際には、Ｒ信号、Ｇ信号およびＢ信号をＹ信号に変換する。このため、ＩＲ信号が飽和した場合において、ナンバープレート領域の表示がいわゆる白飛び状態になることを防ぐことができる。これにより、ナンバープレートの視認性を向上させることができる。 Thus, unlike the luminance signal conversion unit 362 described in FIG. 5, the luminance signal conversion unit 366 converts the R signal, the G signal, and the B signal into a Y signal when the IR signal is saturated. . For this reason, when the IR signal is saturated, it is possible to prevent the license plate area from being displayed in a so-called whiteout state. Thereby, the visibility of a license plate can be improved.
　［ナンバープレート領域画像処理］
　図１７は、本技術の第２の実施の形態におけるナンバープレート領域画像処理の処理手順の一例を示す図である。同図の処理は、図１２において説明したナンバープレート領域画像処理に対応する処理である。なお、これ以外の画像処理の処理手順は本技術の第１の実施の形態において説明した処理手順と同様であるため説明を省略する。 [Number plate area image processing]
FIG. 17 is a diagram illustrating an example of a processing procedure of license plate area image processing according to the second embodiment of the present technology. The process shown in FIG. 12 corresponds to the license plate area image process described with reference to FIG. Note that the other image processing procedures are the same as the processing procedures described in the first embodiment of the present technology, and a description thereof will be omitted.
　まず、ナンバープレート領域画像処理部３６０は、ＩＲ信号が飽和しているか否かを判断する（ステップＳ９５１）。具体的には、飽和検出部３８０においてＩＲ信号の飽和が検出されているか否かを判断する。ＩＲ信号が飽和していない場合には（ステップＳ９５１：Ｎｏ）、ナンバープレート領域画像処理部３６０は、ＩＲ信号を輝度信号に変換し（ステップＳ９５２）、ステップＳ９５４の処理に移行する。一方、ＩＲ信号が飽和している場合には（ステップＳ９５１：Ｙｅｓ）、ナンバープレート領域画像処理部３６０は、Ｒ信号、Ｇ信号およびＢ信号を輝度信号に変換し（ステップＳ９５３）、ステップＳ９５４の処理に移行する。ステップＳ９５４において、ナンバープレート領域画像処理部３６０は、エッジ強調処理を行う（ステップＳ９５４）。次に、ナンバープレート領域画像処理部３６０は、２次元ノイズリダクション処理を行い（ステップＳ９５５）、ナンバープレート領域画像処理を終了する。 First, the license plate area image processing unit 360 determines whether or not the IR signal is saturated (step S951). Specifically, it is determined whether saturation of IR signal is detected by saturation detector 380. If the IR signal is not saturated (step S951: No), the license plate area image processing unit 360 converts the IR signal into a luminance signal (step S952), and the process proceeds to step S954. On the other hand, when the IR signal is saturated (step S951: Yes), the license plate area image processing unit 360 converts the R signal, the G signal, and the B signal into luminance signals (step S953). Transition to processing. In step S954, the license plate area image processing unit 360 performs edge enhancement processing (step S954). Next, the license plate area image processing unit 360 performs two-dimensional noise reduction processing (step S955), and ends the license plate area image processing.
　このように、本技術の第２の実施の形態によれば、ＩＲ信号が飽和した場合においてナンバープレートにおける視認性の低下を防止することができる。 As described above, according to the second embodiment of the present technology, it is possible to prevent the visibility of the license plate from being lowered when the IR signal is saturated.
　［変形例］
　上述の第２の実施の形態では、ＩＲ信号が飽和した場合において、赤外光照射部５００における赤外光の照射の制御を行っていなかった。これに対し、画像処理装置３００の飽和検出部３８０においてＩＲ信号の飽和が検出された際に、制御部６００が赤外光照射部５００を制御して、赤外光の照射量を減少させてもよい。これにより、適正な赤外光照射量にすることができ、ナンバープレートにおける視認性の低下を防止することができる。 [Modification]
In the above-described second embodiment, when the IR signal is saturated, the infrared light irradiation control in the infrared light irradiation unit 500 is not performed. On the other hand, when the saturation detection unit 380 of the image processing apparatus 300 detects the saturation of the IR signal, the control unit 600 controls the infrared light irradiation unit 500 to reduce the irradiation amount of the infrared light. Also good. Thereby, it can be set as a suitable infrared light irradiation amount, and the fall of the visibility in a license plate can be prevented.
　＜３．第３の実施の形態＞
　上述の実施の形態では、車体領域の画像にナンバープレート領域の画像を重ね合わせて合成していた。これに対し、本技術の第３の実施の形態では、車体領域の画像へのナンバープレート領域の画像の重ね合わせを省略する。 <3. Third Embodiment>
In the above-described embodiment, the license plate region image is superimposed on the image of the vehicle body region and synthesized. On the other hand, in the third embodiment of the present technology, the registration of the license plate region image on the vehicle body region image is omitted.
　［合成処理］
　図１８は、本技術の第３の実施の形態における合成処理を示す図である。同図におけるａは、比較のため本技術の第１の実施の形態における合成処理を表したものである。同図におけるａでは、合成部３９０において車体領域の画像７０７にナンバープレート領域の画像７０８を重ね合わせて、画像表示領域７１１に表示している。このため、画像７０７のナンバープレート部分にモノクロのナンバープレート領域の画像が上書きされてしまい、ナンバープレートの色を特定することができない。 [Composition process]
FIG. 18 is a diagram illustrating a composition process according to the third embodiment of the present technology. For comparison, “a” represents the synthesis processing in the first embodiment of the present technology. In FIG. 9A, the license plate region image 708 is superimposed on the vehicle body region image 707 and displayed in the image display region 711 in the combining unit 390. For this reason, the license plate portion of the image 707 is overwritten with the monochrome license plate image, and the color of the license plate cannot be specified.
　同図におけるｂは、本技術の第３の実施の形態における合成処理を表したものであり、車体領域の画像７０７に対するナンバープレート領域の画像の重ね合わせを省略したものである。このため、車体領域の画像７０７のナンバープレート領域は、他の車体領域と同様に色再現性が向上しており、ナンバープレートの色を特定することができる。なお、ナンバープレート領域の画像７０８は、ナンバープレート表示領域７１３に表示される。 B in the figure represents the composition processing according to the third embodiment of the present technology, in which the registration of the license plate region image with the vehicle body region image 707 is omitted. For this reason, the license plate area of the image 707 of the vehicle body area has improved color reproducibility like the other vehicle body areas, and the color of the license plate can be specified. The license plate area image 708 is displayed in the license plate display area 713.
　このように、本技術の第３の実施の形態によれば、車体の色とともにナンバープレートの色の特定が可能になる。 As described above, according to the third embodiment of the present technology, the color of the license plate can be specified together with the color of the vehicle body.
　＜４．第４の実施の形態＞
　上述の実施の形態では、白色光に対応する画素を含まない撮像素子２００を使用していた。これに対し、本技術の第４の実施の形態では、白色光に対応する画素を含む撮像素子２００を使用する。 <4. Fourth Embodiment>
In the above-described embodiment, the image sensor 200 that does not include a pixel corresponding to white light is used. On the other hand, in the fourth embodiment of the present technology, the image sensor 200 including pixels corresponding to white light is used.
　［撮像装置および画像処理装置の構成］
　本技術の第４の実施の形態における撮像装置１０は、撮像素子２００として、図２におけるｄに表した画素の配置とした撮像素子を使用する。また、本技術の第４の実施の形態における画像処理装置３００の動き画像検出部３２０およびデモザイク部３３０は、Ｗ＋ＩＲ信号を含む画像信号に対して、処理を行う必要がある。これ以外の撮像装置１０の構成は本技術の第１の実施の形態において説明した撮像装置１０と同様であるため説明を省略する。 [Configuration of Imaging Device and Image Processing Device]
The imaging device 10 according to the fourth embodiment of the present technology uses an imaging device having the pixel arrangement illustrated in d in FIG. In addition, the motion image detection unit 320 and the demosaic unit 330 of the image processing device 300 according to the fourth embodiment of the present technology need to perform processing on an image signal including a W + IR signal. Since the configuration of the imaging apparatus 10 other than this is the same as that of the imaging apparatus 10 described in the first embodiment of the present technology, the description thereof is omitted.
　［デモザイク処理］
　図１９および２０は、本技術の第４の実施の形態におけるデモザイク処理を示す図である。図１９は、撮像素子２００の画素の配置を表したものであり、中央に配置された画素２０３においてＧ＋ＩＲ信号以外の信号を補間する場合を想定する。図２０におけるａは、Ｒ＋ＩＲ信号を補間する場合の例を表したものである。このように、画素２０３の周囲のＲ＋ＩＲ画素の信号による平均値を画素２０３におけるＲ＋ＩＲ信号にする。この平均は各Ｒ＋ＩＲ信号による加重平均により算出される。各Ｒ＋ＩＲ信号には、所定の重みが乗算される。この重みは、画素２０３に近い画素ほど大きな値としている。すなわち、図２０におけるａでは、画素２０３に隣接する画素２０４には、比較的離れた位置に配置された画素２０５よりも大きな重みが設定される。 [Demosaic processing]
19 and 20 are diagrams illustrating demosaic processing according to the fourth embodiment of the present technology. FIG. 19 shows the arrangement of the pixels of the image sensor 200, and it is assumed that signals other than the G + IR signal are interpolated in the pixel 203 arranged in the center. In FIG. 20, a represents an example in the case of interpolating the R + IR signal. In this way, the average value of the signals of the R + IR pixels around the pixel 203 is set as the R + IR signal in the pixel 203. This average is calculated by a weighted average using each R + IR signal. Each R + IR signal is multiplied by a predetermined weight. This weight is set to a larger value as the pixel is closer to the pixel 203. That is, in a in FIG. 20, a greater weight is set for the pixel 204 adjacent to the pixel 203 than for the pixel 205 arranged at a relatively distant position.
　このようにして、補間された画像信号を添え字「＿Ｌ」を付して記載する。すなわちＲ＋ＩＲ信号、Ｇ＋ＩＲ信号、Ｂ＋ＩＲ信号、Ｗ＋ＩＲ信号およびＩＲ信号に対する補間後の信号は、それぞれＲ＋ＩＲ＿Ｌ信号、Ｇ＋ＩＲ＿Ｌ信号、Ｂ＋ＩＲ＿Ｌ信号、Ｗ＋ＩＲ＿Ｌ信号およびＩＲ＿Ｌ信号となる。これらの信号は、比較的広い範囲の画像信号の平均値であるため、高周波成分が失われた画像信号になる。このため、これらの信号により処理を行うと、車体領域に対応する画像信号は、輪郭がぼやけた画像になる。 In this way, the interpolated image signal is described with the suffix “_L”. That is, the R + IR signal, the G + IR signal, the B + IR signal, the W + IR signal, and the signal after interpolation with respect to the IR signal become an R + IR_L signal, a G + IR_L signal, a B + IR_L signal, a W + IR_L signal, and an IR_L signal, respectively. Since these signals are average values of image signals in a relatively wide range, they become image signals in which high frequency components are lost. Therefore, when processing is performed using these signals, the image signal corresponding to the vehicle body region becomes an image with a blurred outline.
　図２０におけるｂは、Ｗ＋ＩＲ信号に対して高周波成分を多く含む信号を生成する補間を表したものである。Ｗ＋ＩＲ画素は、他の画素よりも多く配置されているため、図４において説明した補間方法と同様の方法を使用することができる。すなわち、画素２０３に隣接するＷ＋ＩＲ画素の信号の平均値を画素２０３におけるＷ＋ＩＲ信号にすることができる。このようにして、補間されたＷ＋ＩＲ信号をＷ＋ＩＲ＿Ｈと称する。この信号は、隣接する画素の画像信号の平均値であるため、高周波成分を多く含む画像信号になる。Ｗ＋ＩＲ信号は、可視光から赤外光に亘る広い範囲の波長の信号を含んでおり、後述するようにこの信号を用いて他のデモザイク処理後の信号に対する高周波成分の復元を行うことができる。 B in FIG. 20 represents an interpolation for generating a signal including a large amount of high-frequency components with respect to the W + IR signal. Since more W + IR pixels are arranged than the other pixels, a method similar to the interpolation method described in FIG. 4 can be used. That is, the average value of the signals of the W + IR pixels adjacent to the pixel 203 can be set to the W + IR signal in the pixel 203. The interpolated W + IR signal is referred to as W + IR_H. Since this signal is the average value of the image signals of adjacent pixels, it becomes an image signal containing many high-frequency components. The W + IR signal includes a signal having a wide range of wavelengths ranging from visible light to infrared light. As described later, this signal can be used to restore a high-frequency component for a signal after other demosaic processing.
　次に、Ｒ＋ＩＲ＿Ｌ信号、Ｇ＋ＩＲ＿Ｌ信号およびＢ＋ＩＲ＿Ｌ信号からＩＲ信号成分を除去する。これは、本技術の第１の実施の形態において説明した方法と同様に、Ｒ＋ＩＲ＿Ｌ信号、Ｇ＋ＩＲ＿Ｌ信号およびＢ＋ＩＲ＿Ｌ信号からＩＲ＿Ｌ信号を減算することにより行うことができる。ＩＲ信号成分を除去した画像信号をそれぞれＲ＿Ｌ信号、Ｇ＿Ｌ信号およびＢ＿Ｌ信号と称する。 Next, the IR signal component is removed from the R + IR_L signal, the G + IR_L signal, and the B + IR_L signal. This can be performed by subtracting the IR_L signal from the R + IR_L signal, the G + IR_L signal, and the B + IR_L signal, similarly to the method described in the first embodiment of the present technology. The image signals from which the IR signal component is removed are referred to as R_L signal, G_L signal, and B_L signal, respectively.
　次に、Ｒ＿Ｌ信号、Ｇ＿Ｌ信号およびＢ＿Ｌ信号の高周波成分の復元を行う。これは、次式のように行うことができる。
　　Ｒ＝Ｒ＿Ｌ＋Ｗ＋ＩＲ＿Ｈ×Ｒ＿Ｌ／Ｗ＋ＩＲ＿Ｌ
　　Ｇ＝Ｇ＿Ｌ＋Ｗ＋ＩＲ＿Ｈ×Ｇ＿Ｌ／Ｗ＋ＩＲ＿Ｌ
　　Ｂ＝Ｂ＿Ｌ＋Ｗ＋ＩＲ＿Ｈ×Ｂ＿Ｌ／Ｗ＋ＩＲ＿Ｌ
ただし、Ｗ＋ＩＲ＿ＨおよびＷ＋ＩＲ＿Ｌは、それぞれＷ＋ＩＲ＿Ｈ信号およびＷ＋ＩＲ＿Ｌ信号を表す。また、Ｒ＿Ｌ、Ｇ＿ＬおよびＢ＿Ｌは、それぞれＲ＿Ｌ信号、Ｇ＿Ｌ信号およびＢ＿Ｌ信号を表す。これにより、高周波成分が復元されたＲ信号、Ｇ信号およびＢ信号を得ることができる。これらの信号により車体領域に対応する画像信号の処理が行われるため、ぼけが改善されて視認性が向上した車体領域の画像信号を得ることができる。 Next, the high frequency components of the R_L signal, the G_L signal, and the B_L signal are restored. This can be done as follows:
R = R_L + W + IR_H × R_L / W + IR_L
G = G_L + W + IR_H × G_L / W + IR_L
B = B_L + W + IR_H × B_L / W + IR_L
However, W + IR _H and W + IR _L each represents a W + IR_H signals and W + IR_L signal. R_L, G_L, and B_L represent an R_L signal, a G_L signal, and a B_L signal, respectively. Thereby, R signal, G signal, and B signal from which the high frequency component was restored can be obtained. Since these signals process the image signal corresponding to the vehicle body region, it is possible to obtain the image signal of the vehicle body region with improved blur and improved visibility.
　このように、本技術の第４の実施の形態によれば、白色光に対応する画像信号を有する撮像素子２００を使用した場合において、色再現性とともに車体領域の視認性を向上させることができる。 As described above, according to the fourth embodiment of the present technology, when the imaging element 200 having an image signal corresponding to white light is used, the visibility of the vehicle body region can be improved together with the color reproducibility. .
　＜５．変形例＞
　上述の実施の形態では、単一の撮像素子２００を使用していた。これに対し、可視光に対応する撮像素子および赤外光に対応する撮像素子の２個の撮像素子を使用してもよい。可視光に対応する撮像素子および赤外光に対応する撮像素子のそれぞれについて露出の制御を行うことにより、それぞれ最適な露出条件にすることができるためである。なお、本技術の実施の形態の変形例においては、プリズムおよびハーフミラー等により撮像装置１０の入射光をこれら２個の撮像素子に導く必要がある。 <5. Modification>
In the above-described embodiment, the single image sensor 200 is used. On the other hand, you may use two image pick-up elements, the image pick-up element corresponding to visible light, and the image pick-up element corresponding to infrared light. This is because by controlling exposure for each of the image sensor corresponding to visible light and the image sensor corresponding to infrared light, optimum exposure conditions can be obtained. In the modification of the embodiment of the present technology, it is necessary to guide the incident light of the imaging device 10 to these two imaging elements by a prism and a half mirror.
　また、これら２個の撮像素子を備えるとともに、図１５において説明した飽和検出部３８０を備えてもよい。ＩＲ信号が飽和した場合において、可視光信号によるナンバープレート領域の視認性を向上させることができる。なお、可視光に対応する撮像素子に赤外線カットフィルタを装着させることにより、赤外光の影響を除去することもできる。 In addition to the two image sensors, the saturation detector 380 described with reference to FIG. 15 may be provided. When the IR signal is saturated, the visibility of the license plate region by the visible light signal can be improved. In addition, the influence of infrared light can be removed by attaching an infrared cut filter to an image pickup device corresponding to visible light.
　以上のように、本技術の実施の形態によれば、ナンバープレートにおける視認性および車体における色の再現性を向上させることにより、ナンバープレートの読取りおよび車体の色の取得が可能になる。これにより、監視対象である車輌の特定を容易に行うことができる。 As described above, according to the embodiment of the present technology, it is possible to read the license plate and acquire the color of the vehicle body by improving the visibility of the license plate and the color reproducibility of the vehicle body. Thereby, it is possible to easily identify the vehicle to be monitored.
　なお、上述の実施の形態は本技術を具現化するための一例を示したものであり、実施の形態における事項と、特許請求の範囲における発明特定事項とはそれぞれ対応関係を有する。同様に、特許請求の範囲における発明特定事項と、これと同一名称を付した本技術の実施の形態における事項とはそれぞれ対応関係を有する。ただし、本技術は実施の形態に限定されるものではなく、その要旨を逸脱しない範囲において実施の形態に種々の変形を施すことにより具現化することができる。 The above-described embodiment shows an example for embodying the present technology, and the matters in the embodiment and the invention-specific matters in the claims have a corresponding relationship. Similarly, the invention specific matter in the claims and the matter in the embodiment of the present technology having the same name as this have a corresponding relationship. However, the present technology is not limited to the embodiment, and can be embodied by making various modifications to the embodiment without departing from the gist thereof.
　また、上述の実施の形態において説明した処理手順は、これら一連の手順を有する方法として捉えてもよく、また、これら一連の手順をコンピュータに実行させるためのプログラム乃至そのプログラムを記憶する記録媒体として捉えてもよい。この記録媒体として、例えば、ＣＤ（Compact Disc）、ＭＤ（MiniDisc）、ＤＶＤ（Digital Versatile Disc）、メモリカード、ブルーレイディスク（Blu-ray（登録商標）Disc）等を用いることができる。 Further, the processing procedure described in the above embodiment may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program. You may catch it. As this recording medium, for example, a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disc), a memory card, a Blu-ray disc (Blu-ray (registered trademark) Disc), or the like can be used.
　なお、本明細書に記載された効果はあくまで例示であって、限定されるものではなく、また、他の効果があってもよい。 It should be noted that the effects described in this specification are merely examples, and are not limited, and other effects may be obtained.
　なお、本技術は以下のような構成もとることができる。
（１）画像信号から車輌の車体領域を検出する車体領域検出部と、
　前記画像信号から車輌のナンバープレート領域を検出するナンバープレート領域検出部と、
　前記検出された前記車体領域に対応する前記画像信号の処理を行う車体領域画像処理部と、
　前記検出された前記ナンバープレート領域に対応する前記画像信号に対して前記車体領域に対応する前記画像信号の前記処理とは異なる処理を行うナンバープレート領域画像処理部と、
　前記処理された前記車体領域に対応する前記画像信号と前記処理された前記ナンバープレート領域に対応する前記画像信号とを合成する合成部と
を具備する画像処理装置。
（２）前記ナンバープレート領域画像処理部は、前記ナンバープレート領域に対応する前記画像信号を前記ナンバープレート領域に対応する輝度信号に変換する輝度信号変換処理と前記輝度信号における画像のエッジ部分を強調するエッジ強調処理とを行う前記（１）に記載の画像処理装置。
（３）前記ナンバープレート領域画像処理部は、前記ナンバープレート領域に対応する前記画像信号を前記ナンバープレート領域に対応する輝度信号に変換する輝度信号変換処理と同一フレームに属する前記輝度信号の高周波成分を減衰させることにより前記輝度信号のノイズを除去する２次元ノイズリダクション処理とを行う前記（１）に記載の画像処理装置。
（４）前記車体領域画像処理部は、前記車体領域に対応する前記画像信号を前記車体領域に対応する輝度信号および色差信号に変換する輝度および色差信号変換処理と連続する複数のフレームに属する前記輝度信号および前記色差信号を用いて前記輝度信号および前記色差信号のノイズを除去する３次元ノイズリダクション処理とを行う前記（１）に記載の画像処理装置。
（５）前記画像信号は、赤色光と緑色光と青色光と赤外光とに対応するそれぞれの画像信号を備え、
　前記車体領域画像処理部は、前記車体領域に対応する前記画像信号に含まれる赤色光と緑色光と青色光とに対応するそれぞれの画像信号の処理を行い、
　前記ナンバープレート領域画像処理部は、前記ナンバープレート領域に対応する前記画像信号に含まれる赤外光に対応する画像信号の処理を行う
前記（１）から（４）に記載の画像処理装置。
（６）前記画像信号は、白色光に対応する画像信号をさらに備え、
　前記車体領域画像処理部は、前記車体領域に対応する前記画像信号に含まれる赤色光と緑色光と青色光と白色光とに対応するそれぞれの画像信号の処理を行う
前記（５）に記載の画像処理装置。
（７）前記画像信号は、赤色光と緑色光と青色光と赤外光とに対応するそれぞれの画像信号を備え、
　前記画像信号に含まれる赤外光に対応する画像信号が飽和しているか否かを検出する飽和検出部をさらに具備し、
　前記ナンバープレート領域画像処理部は、前記飽和検出部が前記赤外光に対応する画像信号が飽和していないことを検出した場合には前記ナンバープレート領域に対応する前記画像信号に含まれる赤外光に対応する画像信号の処理を行い、前記飽和検出部が前記赤外光に対応する画像信号が飽和していることを検出した場合には前記ナンバープレート領域に対応する前記画像信号に含まれる赤色光と緑色光と青色光とに対応するそれぞれの画像信号の処理を行う
前記（１）に記載の画像処理装置。
（８）画像信号を生成する撮像素子と、
　前記画像信号から車輌の車体領域を検出する車体領域検出部と、
　前記画像信号から車輌のナンバープレート領域を検出するナンバープレート領域検出部と、
　前記検出された前記車体領域に対応する前記画像信号の処理を行う車体領域画像処理部と、
　前記検出された前記ナンバープレート領域に対応する前記画像信号に対して前記車体領域に対応する前記画像信号の前記処理とは異なる処理を行うナンバープレート領域画像処理部と、
　前記処理された前記車体領域に対応する前記画像信号と前記処理された前記ナンバープレート領域に対応する前記画像信号とを合成する合成部と
を具備する撮像装置。
（９）画像信号から車輌の車体領域を検出する車体領域検出手順と、
　前記画像信号から車輌のナンバープレート領域を検出するナンバープレート領域検出手順と、
　前記検出された前記車体領域に対応する前記画像信号の処理を行う車体領域画像処理手順と、
　前記検出された前記ナンバープレート領域に対応する前記画像信号に対して前記車体領域に対応する前記画像信号の前記処理とは異なる処理を行うナンバープレート領域画像処理手順と、
　前記処理された前記車体領域に対応する前記画像信号と前記処理された前記ナンバープレート領域に対応する前記画像信号とを合成する合成手順と
を具備する画像処理方法。 In addition, this technique can also take the following structures.
(1) a vehicle body region detection unit that detects a vehicle body region of the vehicle from the image signal;
A license plate area detector for detecting a license plate area of the vehicle from the image signal;
A vehicle body region image processing unit for processing the image signal corresponding to the detected vehicle body region;
A license plate region image processing unit that performs processing different from the processing of the image signal corresponding to the vehicle body region on the image signal corresponding to the detected license plate region;
An image processing apparatus comprising: a combining unit configured to combine the image signal corresponding to the processed vehicle body region and the image signal corresponding to the processed license plate region.
(2) The license plate region image processing unit emphasizes an edge portion of an image in the luminance signal and luminance signal conversion processing for converting the image signal corresponding to the license plate region into a luminance signal corresponding to the license plate region. The image processing apparatus according to (1), wherein the edge enhancement processing is performed.
(3) The license plate region image processing unit includes a high-frequency component of the luminance signal belonging to the same frame as the luminance signal conversion processing for converting the image signal corresponding to the license plate region into a luminance signal corresponding to the license plate region. The image processing apparatus according to (1), wherein a two-dimensional noise reduction process that removes noise of the luminance signal by attenuating the luminance signal is performed.
(4) The vehicle body region image processing unit belongs to a plurality of frames that are continuous with luminance and color difference signal conversion processing for converting the image signal corresponding to the vehicle body region into a luminance signal and a color difference signal corresponding to the vehicle body region. The image processing apparatus according to (1), wherein the luminance signal and the color difference signal are used to perform a three-dimensional noise reduction process for removing noise of the luminance signal and the color difference signal.
(5) The image signal includes respective image signals corresponding to red light, green light, blue light, and infrared light,
The vehicle body region image processing unit performs processing of each image signal corresponding to red light, green light, and blue light included in the image signal corresponding to the vehicle body region,
The image processing apparatus according to any one of (1) to (4), wherein the license plate region image processing unit performs processing of an image signal corresponding to infrared light included in the image signal corresponding to the license plate region.
(6) The image signal further includes an image signal corresponding to white light,
The vehicle body region image processing unit performs processing of each image signal corresponding to red light, green light, blue light, and white light included in the image signal corresponding to the vehicle body region. Image processing device.
(7) The image signal includes respective image signals corresponding to red light, green light, blue light, and infrared light,
A saturation detector for detecting whether an image signal corresponding to infrared light included in the image signal is saturated;
When the saturation detection unit detects that the image signal corresponding to the infrared light is not saturated, the license plate region image processing unit includes an infrared signal included in the image signal corresponding to the license plate region. When the image signal corresponding to the light is processed and the saturation detection unit detects that the image signal corresponding to the infrared light is saturated, it is included in the image signal corresponding to the license plate region The image processing apparatus according to (1), wherein each image signal corresponding to red light, green light, and blue light is processed.
(8) an image sensor that generates an image signal;
A vehicle body region detection unit for detecting a vehicle body region of the vehicle from the image signal;
A license plate area detector for detecting a license plate area of the vehicle from the image signal;
A vehicle body region image processing unit for processing the image signal corresponding to the detected vehicle body region;
A license plate region image processing unit that performs processing different from the processing of the image signal corresponding to the vehicle body region on the image signal corresponding to the detected license plate region;
An image pickup apparatus comprising: a combining unit that combines the image signal corresponding to the processed vehicle body region and the image signal corresponding to the processed license plate region.
(9) a vehicle body region detection procedure for detecting a vehicle body region from the image signal;
A license plate area detection procedure for detecting a license plate area of a vehicle from the image signal;
A vehicle body region image processing procedure for processing the image signal corresponding to the detected vehicle body region;
A license plate area image processing procedure for performing processing different from the processing of the image signal corresponding to the vehicle body area on the image signal corresponding to the detected license plate area;
An image processing method comprising: a combining procedure for combining the image signal corresponding to the processed vehicle body region and the image signal corresponding to the processed license plate region.
　１０　撮像装置
　１００　レンズ
　２００　撮像素子
　２０１～２０５　画素
　３００　画像処理装置
　３０４　ハイパスフィルタ
　３０６　イプシロンフィルタ
　３０８　減算器
　３１０　撮像素子制御部
　３２０　動き画像検出部
　３３０　デモザイク部
　３４０　ナンバープレート領域検出部
　３５０　車体領域検出部
　３６０　ナンバープレート領域画像処理部
　３６２、３６６　輝度信号変換部
　３６３　エッジ強調部
　３６４、３７４　２次元ノイズリダクション部
　３７０　車体領域画像処理部
　３７２　輝度色差信号変換部
　３７５　３次元ノイズリダクション部
　３７８　加算器
　３７９　除算器
　３８０　飽和検出部
　３９０　合成部
　４００　画像信号出力部
　５００　赤外光照射部
　６００　制御部 DESCRIPTION OF SYMBOLS 10 Image pick-up device 100 Lens 200 Image pick-up element 201-205 Pixel 300 Image processing apparatus 304 High pass filter 306 Epsilon filter 308 Subtractor 310 Image pick-up element control part 320 Motion image detection part 330 Demosaic part 340 Number plate area | region detection part 350 Car body area | region detection part 360 License plate region image processing unit 362, 366 Luminance signal conversion unit 363 Edge enhancement unit 364, 374 Two-dimensional noise reduction unit 370 Car body region image processing unit 372 Luminance color difference signal conversion unit 375 Three-dimensional noise reduction unit 378 Adder 379 Divider 380 Saturation detection unit 390 synthesis unit 400 image signal output unit 500 infrared light irradiation unit 600 control unit

Claims (9)

  1. 　画像信号から車輌の車体領域を検出する車体領域検出部と、
    　前記画像信号から車輌のナンバープレート領域を検出するナンバープレート領域検出部と、
    　前記検出された前記車体領域に対応する前記画像信号の処理を行う車体領域画像処理部と、
    　前記検出された前記ナンバープレート領域に対応する前記画像信号に対して前記車体領域に対応する前記画像信号の前記処理とは異なる処理を行うナンバープレート領域画像処理部と、
    　前記処理された前記車体領域に対応する前記画像信号と前記処理された前記ナンバープレート領域に対応する前記画像信号とを合成する合成部と
    を具備する画像処理装置。     A vehicle body region detection unit for detecting a vehicle body region from the image signal;
    A license plate area detector for detecting a license plate area of the vehicle from the image signal;
    A vehicle body region image processing unit for processing the image signal corresponding to the detected vehicle body region;
    A license plate region image processing unit that performs processing different from the processing of the image signal corresponding to the vehicle body region on the image signal corresponding to the detected license plate region;
    An image processing apparatus comprising: a combining unit configured to combine the image signal corresponding to the processed vehicle body region and the image signal corresponding to the processed license plate region.
  2. 　前記ナンバープレート領域画像処理部は、前記ナンバープレート領域に対応する前記画像信号を前記ナンバープレート領域に対応する輝度信号に変換する輝度信号変換処理と前記輝度信号における画像のエッジ部分を強調するエッジ強調処理とを行う請求項１記載の画像処理装置。 The license plate area image processing unit converts the image signal corresponding to the license plate area into a luminance signal corresponding to the license plate area, and edge enhancement that enhances an edge portion of the image in the luminance signal. The image processing apparatus according to claim 1, wherein the processing is performed.
  3. 　前記ナンバープレート領域画像処理部は、前記ナンバープレート領域に対応する前記画像信号を前記ナンバープレート領域に対応する輝度信号に変換する輝度信号変換処理と同一フレームに属する前記輝度信号の高周波成分を減衰させることにより前記輝度信号のノイズを除去する２次元ノイズリダクション処理とを行う請求項１記載の画像処理装置。 The license plate area image processing unit attenuates a high frequency component of the luminance signal belonging to the same frame as a luminance signal conversion process for converting the image signal corresponding to the license plate area into a luminance signal corresponding to the license plate area. The image processing apparatus according to claim 1, wherein two-dimensional noise reduction processing is performed to remove noise from the luminance signal.
  4. 　前記車体領域画像処理部は、前記車体領域に対応する前記画像信号を前記車体領域に対応する輝度信号および色差信号に変換する輝度および色差信号変換処理と連続する複数のフレームに属する前記輝度信号および前記色差信号を用いて前記輝度信号および前記色差信号のノイズを除去する３次元ノイズリダクション処理とを行う請求項１記載の画像処理装置。 The vehicle body region image processing unit includes the luminance signal belonging to a plurality of frames continuous with the luminance and color difference signal conversion processing for converting the image signal corresponding to the vehicle body region into a luminance signal and a color difference signal corresponding to the vehicle body region. The image processing apparatus according to claim 1, wherein a three-dimensional noise reduction process is performed to remove noise of the luminance signal and the color difference signal using the color difference signal.
  5. 　前記画像信号は、赤色光と緑色光と青色光と赤外光とに対応するそれぞれの画像信号を備え、
    　前記車体領域画像処理部は、前記車体領域に対応する前記画像信号に含まれる赤色光と緑色光と青色光とに対応するそれぞれの画像信号の処理を行い、
    　前記ナンバープレート領域画像処理部は、前記ナンバープレート領域に対応する前記画像信号に含まれる赤外光に対応する画像信号の処理を行う
    請求項１記載の画像処理装置。     The image signal includes respective image signals corresponding to red light, green light, blue light, and infrared light,
    The vehicle body region image processing unit performs processing of each image signal corresponding to red light, green light, and blue light included in the image signal corresponding to the vehicle body region,
    The image processing apparatus according to claim 1, wherein the license plate area image processing unit processes an image signal corresponding to infrared light included in the image signal corresponding to the license plate area.
  6. 　前記画像信号は、白色光に対応する画像信号をさらに備え、
    　前記車体領域画像処理部は、前記車体領域に対応する前記画像信号に含まれる赤色光と緑色光と青色光と白色光とに対応するそれぞれの画像信号の処理を行う
    請求項５記載の画像処理装置。     The image signal further comprises an image signal corresponding to white light,
    The image processing according to claim 5, wherein the vehicle body region image processing unit performs processing of respective image signals corresponding to red light, green light, blue light, and white light included in the image signal corresponding to the vehicle body region. apparatus.
  7. 　前記画像信号は、赤色光と緑色光と青色光と赤外光とに対応するそれぞれの画像信号を備え、
    　前記画像信号に含まれる赤外光に対応する画像信号が飽和しているか否かを検出する飽和検出部をさらに具備し、
    　前記ナンバープレート領域画像処理部は、前記飽和検出部が前記赤外光に対応する画像信号が飽和していないことを検出した場合には前記ナンバープレート領域に対応する前記画像信号に含まれる赤外光に対応する画像信号の処理を行い、前記飽和検出部が前記赤外光に対応する画像信号が飽和していることを検出した場合には前記ナンバープレート領域に対応する前記画像信号に含まれる赤色光と緑色光と青色光とに対応するそれぞれの画像信号の処理を行う
    請求項１記載の画像処理装置。     The image signal includes respective image signals corresponding to red light, green light, blue light, and infrared light,
    A saturation detector for detecting whether an image signal corresponding to infrared light included in the image signal is saturated;
    When the saturation detection unit detects that the image signal corresponding to the infrared light is not saturated, the license plate region image processing unit includes an infrared signal included in the image signal corresponding to the license plate region. When the image signal corresponding to the light is processed and the saturation detection unit detects that the image signal corresponding to the infrared light is saturated, it is included in the image signal corresponding to the license plate region The image processing apparatus according to claim 1, wherein each image signal corresponding to red light, green light, and blue light is processed.
  8. 　画像信号を生成する撮像素子と、
    　前記画像信号から車輌の車体領域を検出する車体領域検出部と、
    　前記画像信号から車輌のナンバープレート領域を検出するナンバープレート領域検出部と、
    　前記検出された前記車体領域に対応する前記画像信号の処理を行う車体領域画像処理部と、
    　前記検出された前記ナンバープレート領域に対応する前記画像信号に対して前記車体領域に対応する前記画像信号の前記処理とは異なる処理を行うナンバープレート領域画像処理部と、
    　前記処理された前記車体領域に対応する前記画像信号と前記処理された前記ナンバープレート領域に対応する前記画像信号とを合成する合成部と
    を具備する撮像装置。     An image sensor for generating an image signal;
    A vehicle body region detection unit for detecting a vehicle body region of the vehicle from the image signal;
    A license plate area detector for detecting a license plate area of the vehicle from the image signal;
    A vehicle body region image processing unit for processing the image signal corresponding to the detected vehicle body region;
    A license plate region image processing unit that performs processing different from the processing of the image signal corresponding to the vehicle body region on the image signal corresponding to the detected license plate region;
    An image pickup apparatus comprising: a combining unit that combines the image signal corresponding to the processed vehicle body region and the image signal corresponding to the processed license plate region.
  9. 　画像信号から車輌の車体領域を検出する車体領域検出手順と、
    　前記画像信号から車輌のナンバープレート領域を検出するナンバープレート領域検出手順と、
    　前記検出された前記車体領域に対応する前記画像信号の処理を行う車体領域画像処理手順と、
    　前記検出された前記ナンバープレート領域に対応する前記画像信号に対して前記車体領域に対応する前記画像信号の前記処理とは異なる処理を行うナンバープレート領域画像処理手順と、
    　前記処理された前記車体領域に対応する前記画像信号と前記処理された前記ナンバープレート領域に対応する前記画像信号とを合成する合成手順と
    を具備する画像処理方法。     A vehicle body region detection procedure for detecting a vehicle body region from an image signal;
    A license plate area detection procedure for detecting a license plate area of a vehicle from the image signal;
    A vehicle body region image processing procedure for processing the image signal corresponding to the detected vehicle body region;
    A license plate area image processing procedure for performing processing different from the processing of the image signal corresponding to the vehicle body area on the image signal corresponding to the detected license plate area;
    An image processing method comprising: a combining procedure for combining the image signal corresponding to the processed vehicle body region and the image signal corresponding to the processed license plate region.
PCT/JP2015/079798 2015-01-08 2015-10-22 Image processing device, image pickup device, and image processing method WO2016111072A1 (en)

Priority Applications (6)

Application Number Priority Date Filing Date Title
CN202010970091.7A CN112055130B (en) 2015-01-08 2015-10-22 Image processing apparatus, imaging apparatus, and image processing method
US15/540,597 US10217034B2 (en) 2015-01-08 2015-10-22 Image processing device, imaging device, and image processing method
JP2016568287A JP6757258B2 (en) 2015-01-08 2015-10-22 Image processing device, imaging device and image processing method
CN201580071105.9A CN107113375B (en) 2015-01-08 2015-10-22 Image processing apparatus, imaging apparatus, and image processing method
US16/246,340 US10885403B2 (en) 2015-01-08 2019-01-11 Image processing device, imaging device, and image processing method
US17/013,385 US11244209B2 (en) 2015-01-08 2020-09-04 Image processing device, imaging device, and image processing method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2015002390 2015-01-08
JP2015-002390 2015-01-08

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US15/540,597 A-371-Of-International US10217034B2 (en) 2015-01-08 2015-10-22 Image processing device, imaging device, and image processing method
US16/246,340 Continuation US10885403B2 (en) 2015-01-08 2019-01-11 Image processing device, imaging device, and image processing method

Publications (1)

Publication Number Publication Date
WO2016111072A1 true WO2016111072A1 (en) 2016-07-14

Family

ID=56355775

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2015/079798 WO2016111072A1 (en) 2015-01-08 2015-10-22 Image processing device, image pickup device, and image processing method

Country Status (4)

Country Link
US (3) US10217034B2 (en)
JP (1) JP6757258B2 (en)
CN (2) CN112055130B (en)
WO (1) WO2016111072A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024043485A1 (en) * 2022-08-26 2024-02-29 삼성전자주식회사 Electronic device and method for processing image including text

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112055130B (en) 2015-01-08 2023-03-17 索尼半导体解决方案公司 Image processing apparatus, imaging apparatus, and image processing method
JP2017112401A (en) * 2015-12-14 2017-06-22 ソニー株式会社 Imaging device, apparatus and method for image processing, and program
JP6720694B2 (en) * 2016-05-20 2020-07-08 富士通株式会社 Image processing program, image processing method, and image processing apparatus
DE102016213059A1 (en) * 2016-07-18 2018-01-18 Robert Bosch Gmbh Method and control unit for processing an image representing at least one halo and image recording system
US10609298B2 (en) * 2018-02-06 2020-03-31 Google Llc Adaptive infrared illumination for exposure correction
TWI672957B (en) * 2018-03-29 2019-09-21 瑞昱半導體股份有限公司 Image processing device and image processing method
US10559200B1 (en) * 2018-05-01 2020-02-11 Flock Group Inc. Method and system for capturing and storing significant surveillance images
CN111355900B (en) * 2020-03-12 2021-12-17 浙江大华技术股份有限公司 Image processing method and device, storage medium and electronic device
US11423704B2 (en) * 2020-07-01 2022-08-23 Sony Corporation Apparatus for controlling multiple gates through which vehicles travel
CN114095672A (en) * 2020-07-31 2022-02-25 北京小米移动软件有限公司 Imaging system, method and electronic device
EP4264475A1 (en) 2020-12-15 2023-10-25 Selex ES Inc. Systems and methods for electronic signature tracking
MX2023010504A (en) * 2021-03-10 2023-09-20 Selex Es Inc Systems and methods for vehicle information capture using white light.

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000182182A (en) * 1998-12-14 2000-06-30 Omron Corp Vehicle image pick up device
JP2006053681A (en) * 2004-08-10 2006-02-23 Nagoya Electric Works Co Ltd Vehicle photographing method and device thereof
JP2008042695A (en) * 2006-08-09 2008-02-21 Gm2:Kk Monitor camera apparatus, monitor camera system, and method of recording monitor camera image
JP2013149040A (en) * 2012-01-18 2013-08-01 Sumitomo Electric Ind Ltd Number plate color determining device, computer program, and number plate color determining method

Family Cites Families (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2132515C (en) * 1992-03-20 2006-01-31 Glen William Auty An object monitoring system
JPH11306283A (en) * 1998-04-24 1999-11-05 Chuo Spring Co Ltd Number plate reader
JP3987264B2 (en) * 2000-03-24 2007-10-03 富士通株式会社 License plate reader and method
JP2002316602A (en) * 2001-04-24 2002-10-29 Matsushita Electric Ind Co Ltd Pickup image displaying method of onboard camera, and device therefor
US8331621B1 (en) * 2001-10-17 2012-12-11 United Toll Systems, Inc. Vehicle image capture system
US20060030985A1 (en) * 2003-10-24 2006-02-09 Active Recognition Technologies Inc., Vehicle recognition using multiple metrics
JP2006201817A (en) 2005-01-17 2006-08-03 Hitachi Ltd Method and device for discriminating vehicle body color
US7095892B2 (en) * 2005-01-26 2006-08-22 Motorola, Inc. Object-of-interest image capture
JP4268208B2 (en) * 2005-02-03 2009-05-27 富士通株式会社 Vehicle image data generation program and vehicle image data generation device
JP4766302B2 (en) * 2005-03-22 2011-09-07 オムロン株式会社 Image processing apparatus and method, recording medium, and program
US7613328B2 (en) * 2005-09-09 2009-11-03 Honeywell International Inc. Label detection
JP4288623B2 (en) * 2007-01-18 2009-07-01 ソニー株式会社 Imaging device, noise removal device, noise removal method, noise removal method program, and recording medium recording noise removal method program
US7786897B2 (en) * 2007-01-23 2010-08-31 Jai Pulnix, Inc. High occupancy vehicle (HOV) lane enforcement
JP5210994B2 (en) * 2009-08-18 2013-06-12 東芝アルパイン・オートモティブテクノロジー株式会社 Image display device for vehicle
TWI409718B (en) * 2009-12-04 2013-09-21 Huper Lab Co Ltd Method of locating license plate of moving vehicle
EP2601617A4 (en) * 2010-08-05 2017-12-06 Hi-Tech Solutions Ltd. Method and system for collecting information relating to identity parameters of a vehicle
JP5651414B2 (en) * 2010-09-16 2015-01-14 株式会社東芝 Vehicle detection device
JP5740241B2 (en) * 2011-08-03 2015-06-24 株式会社東芝 Vehicle detection device
DE102011053052B3 (en) * 2011-08-26 2013-02-28 Jenoptik Robot Gmbh Method and device for identifying motor vehicles for traffic monitoring
US8823798B2 (en) * 2011-12-23 2014-09-02 Xerox Corporation Obscuring identification information in an image of a vehicle
US8731245B2 (en) * 2012-03-07 2014-05-20 Xerox Corporation Multiple view transportation imaging systems
JP5615862B2 (en) * 2012-03-07 2014-10-29 クラリオン株式会社 Vehicle perimeter monitoring device
WO2013181457A2 (en) * 2012-05-30 2013-12-05 Arboc, Llc Interactive advertising based on license plate recognition
JP6120169B2 (en) * 2012-07-25 2017-04-26 パナソニックＩｐマネジメント株式会社 Image editing device
JP2014115799A (en) * 2012-12-10 2014-06-26 Panasonic Corp License plate determination device
US9396403B2 (en) * 2012-12-31 2016-07-19 Instytut Badawczy Drog I Mostow Method of vehicle identification and a system for vehicle identification
CN203244413U (en) * 2013-05-07 2013-10-23 中国医学科学院生物医学工程研究所 Noise reduction device for ultrasonic imaging
US20150125042A1 (en) * 2013-10-08 2015-05-07 Smartlanes Technologies, Llc Method and system for data collection using processed image data
US9792301B2 (en) * 2014-09-26 2017-10-17 Conduent Business Services, Llc Multi-query privacy-preserving parking management system and method
JP6550723B2 (en) * 2014-10-31 2019-07-31 オムロン株式会社 Image processing apparatus, character recognition apparatus, image processing method, and program
CN112055130B (en) 2015-01-08 2023-03-17 索尼半导体解决方案公司 Image processing apparatus, imaging apparatus, and image processing method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000182182A (en) * 1998-12-14 2000-06-30 Omron Corp Vehicle image pick up device
JP2006053681A (en) * 2004-08-10 2006-02-23 Nagoya Electric Works Co Ltd Vehicle photographing method and device thereof
JP2008042695A (en) * 2006-08-09 2008-02-21 Gm2:Kk Monitor camera apparatus, monitor camera system, and method of recording monitor camera image
JP2013149040A (en) * 2012-01-18 2013-08-01 Sumitomo Electric Ind Ltd Number plate color determining device, computer program, and number plate color determining method

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024043485A1 (en) * 2022-08-26 2024-02-29 삼성전자주식회사 Electronic device and method for processing image including text

Also Published As

Publication number Publication date
US20200401859A1 (en) 2020-12-24
CN112055130A (en) 2020-12-08
US10217034B2 (en) 2019-02-26
JP6757258B2 (en) 2020-09-16
US11244209B2 (en) 2022-02-08
CN112055130B (en) 2023-03-17
US20170357881A1 (en) 2017-12-14
US20190147306A1 (en) 2019-05-16
CN107113375B (en) 2020-09-18
US10885403B2 (en) 2021-01-05
CN107113375A (en) 2017-08-29
JPWO2016111072A1 (en) 2017-10-12

Similar Documents

Publication Publication Date Title
WO2016111072A1 (en) Image processing device, image pickup device, and image processing method
JP4914303B2 (en) Image processing apparatus and imaging apparatus, image processing method and imaging method, and image processing program
US10298863B2 (en) Automatic compensation of lens flare
JP5040655B2 (en) Image processing apparatus and image processing method
JP5016255B2 (en) Noise reduction apparatus, control method thereof, control program thereof, imaging apparatus, and digital camera
JP4144630B2 (en) Imaging device
JP4084402B2 (en) Imaging device
JP2013219705A (en) Image processor, image processing method and program
JP2006180269A (en) Image processing apparatus, image processing method, imaging apparatus, program, and recording medium
WO2017159041A1 (en) Signal processing apparatus, image pickup apparatus, and signal processing method
JP4986820B2 (en) Image processing apparatus and image processing method
JP5948167B2 (en) Imaging device
JP2009194721A (en) Image signal processing device, image signal processing method, and imaging device
JP5919086B2 (en) Image processing apparatus, control method therefor, and program
JP4877402B2 (en) Image processing apparatus, image processing method, imaging apparatus, program, and recording medium
JP5227906B2 (en) Video recording system
JP4962293B2 (en) Image processing apparatus, image processing method, and program
JP2016082390A (en) Signal processing unit
WO2015156045A1 (en) Image processing device, image-capturing device, and image-capturing method
JP2010056817A (en) Imaging apparatus
CN105450909B (en) Information processing method and electronic equipment
JP2013055622A (en) Image processing apparatus, image processing method, information recording medium, and program
JP2013157807A (en) Image signal processing method, image signal processing circuit, and imaging device
JP2009017583A (en) Image processing device
JP2006245916A (en) Circuit and method for processing color separation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15876945

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2016568287

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 15540597

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15876945

Country of ref document: EP

Kind code of ref document: A1