EP3136726A1 - Method of pre-processing digital images, and digital image pre-processing system - Google Patents
Method of pre-processing digital images, and digital image pre-processing system Download PDFInfo
- Publication number
- EP3136726A1 EP3136726A1 EP15182692.2A EP15182692A EP3136726A1 EP 3136726 A1 EP3136726 A1 EP 3136726A1 EP 15182692 A EP15182692 A EP 15182692A EP 3136726 A1 EP3136726 A1 EP 3136726A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- processing
- movement
- image sensor
- image
- digital image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007781 pre-processing Methods 0.000 title claims abstract description 105
- 238000000034 method Methods 0.000 title claims abstract description 62
- 230000033001 locomotion Effects 0.000 claims abstract description 116
- 230000001419 dependent effect Effects 0.000 claims abstract description 7
- 238000001914 filtration Methods 0.000 claims description 13
- 238000013507 mapping Methods 0.000 claims description 8
- 238000002360 preparation method Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 238000005070 sampling Methods 0.000 claims description 6
- 238000012937 correction Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 4
- 238000004091 panning Methods 0.000 description 11
- 230000000007 visual effect Effects 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000002123 temporal effect Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 238000012886 linear function Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
- 230000002269 spontaneous effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/003—Reconstruction from projections, e.g. tomography
- G06T11/005—Specific pre-processing for tomographic reconstruction, e.g. calibration, source positioning, rebinning, scatter correction, retrospective gating
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/103—Selection of coding mode or of prediction mode
- H04N19/107—Selection of coding mode or of prediction mode between spatial and temporal predictive coding, e.g. picture refresh
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/117—Filters, e.g. for pre-processing or post-processing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/162—User input
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/423—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements
- H04N19/426—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements using memory downsizing methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/57—Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
- H04N23/6811—Motion detection based on the image signal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
Definitions
- the present invention relates to the field of digital image processing, and more particularly to pre-processing of digital images in preparation for encoding.
- the digital images are compressed using various encoding schemes for transmission or storage.
- pre-processing may for instance involve sharpening or noise filtering, in order to improve the quality of the images.
- Pre-processing may for instance involve sharpening or noise filtering, in order to improve the quality of the images.
- a problem arises when the image sensor capturing the images moves during capture. Firstly, the ensuing changes in the images lead to a higher output bit rate from the encoder, because there is less temporal redundancy between the images. If bandwidth is very restricted, such as when transmitting over a mobile network, it may even be impossible to transmit video from a camera when fast movement causes increased bitrate. Additionally, the movement may make it computationally heavier to pre-process the image.
- the pre-processing may add to the number of bits required for representing the images compared to a corresponding image captured when the image sensor is stationary.
- One solution to this problem is to freeze encoding of images during panning and/or tilting of a camera comprising the image sensor, and to resume encoding new input image frames only after movement has ceased.
- An additional object is to provide a pre-processing system enabling provision of images also when the image sensor is moving.
- Yet another object is to provide a computer program product that enables efficient bitrate also when the image sensor capturing the images is moving.
- these objects are achieved, in full or at least in part, by a method of pre-processing digital images captured by an image sensor for encoding, comprising: receiving a first digital image, receiving information representative of a movement of the image sensor at a time of capture of said first digital image, and pre-processing said first digital image, wherein at least one parameter of said pre-processing is dependent on said information representative of movement.
- pre-processing may be adapted such that the movement of the image sensor does not cause unnecessarily high bitrate.
- images may be encoded and stored or transmitted for displaying also during periods of movement of the image sensor, without the cost of increased bitrate.
- the images captured during movement of the image sensor e.g., during a fast panning movement, will generally be blurry, a loss of image quality possibly caused by modifying the pre-processing will have little impact on the usability of such images.
- said at least one parameter of said pre-processing may be modified such that the first digital image is less processed by said pre-processing than if the movement were equal to or below said threshold.
- said at least one parameter of said pre-processing may be modified such that at least one pre-processing procedure is disabled. For instance, sharpening may be dispensed with when the image sensor is moving, as the images would not really benefit from sharpening anyway.
- said at least one parameter is modified such that at least one pre-processing procedure processes said first digital image, but to a greater or lesser extent.
- filtering may be applied to a greater degree.
- tone mapping may be performed, but to a lesser degree.
- said at least one parameter of said pre-processing is modified such that at least one pre-processing procedure that is otherwise disabled is enabled. For instance, blurring may be applied during movement of the image sensor, thereby making blocks of pixels in the images more or less homogeneous, which leads to lower bit requirement for representation of the image.
- the pre-processing may comprise performing at least one pre-processing procedure chosen from the group consisting of sharpening, adding contrast, tone mapping, colour correction, spatial and/or temporal noise filtering, blurring, Bayer pattern sampling, demosaicing, HDR merging, controlling focus, controlling exposure, and controlling gain.
- the information representative of a movement of the image sensor may be provided by a pan, tilt and/or zoom controller controlling movement of said image sensor.
- adaptation of the pre-processing may be made in view of intentional movement of the image.
- zooming does not in itself imply a movement of the image sensor.
- zooming will induce changes in the captured images analogous to actual movement of the image sensor. Therefore, in the context of this application, a zoom operation will be seen as equivalent to a movement of the image sensor.
- the information representative of a movement of the image sensor may be provided by a motion sensor.
- Information from the motion sensor may also be used for providing information on pan, tilt and/or zoom operations.
- said at least one parameter is modified gradually in preparation for a planned movement of the image sensor.
- harder noise filtering may be added gradually leading up to a planned movement, or sharpening may be gradually decreased.
- the method may further comprise encoding said first digital image as an intra-frame in preparation for a planned movement of the image sensor. If there is movement in the captured images, inter-frames will become larger, due to larger differences between frames. At some point, the differences will be large enough that an intra-frame, not relying on other image frames for its encoding, will be more efficient. If it is known that there will be movement, an intra-frame can be forced, such that unnecessarily large inter-frames may be avoided. By forcing, or inserting, an intra-frame, the encoded frame may have higher quality, as may subsequent images encoded as inter-frames with reference to that intra-frame. The subsequent inter-frames may also require fewer bits because of having a more suitable reference frame.
- a digital image pre-processing system for pre-processing input images captured by an image sensor for encoding
- the system comprising: an image receiving module arranged to receive input images, a movement information receiving module arranged to receive information representative of a movement of the image sensor at a time of capture of said first digital image, and a pre-processing module arranged to pre-process the input images before encoding, wherein at least one parameter of said pre-processing is dependent on said information representative of movement.
- the pre-processing module may be arranged to perform at least one pre-processing procedure chosen from the group consisting of sharpening, adding contrast, tone mapping, colour correction, temporal and/or spatial noise filtering, blurring, Bayer pattern sampling, demosaicing, HDR merging, controlling focus, controlling exposure, and controlling gain.
- the movement information receiving module is communicatively connected to a pan, tilt and/or zoom controller controlling movement of said image sensor. Thereby, information on intentional movement may be provided.
- the movement information receiving module may be communicatively connected to a motion sensor. As discussed in connection with the method, information on unintentional as well as intentional movement may thereby be provided.
- a camera comprising a pre-processing system according to the second aspect.
- Such a camera may generally be embodied in the same ways as the second aspect, with accompanying advantages.
- a computer program product comprising a computer-readable storage medium with instructions adapted to carry out the method according to the first aspect when executed by a processor.
- the processor may be any kind of processor, e.g., a central processing unit (CPU), a graphics processing unit (GPU), a custom made processing device implemented in an integrated circuit, an ASIC, an FPGA, or logical circuitry including discrete components.
- the computer-program product may generally be varied in the same ways as the first aspect with accompanying advantages.
- a scene 1 is shown, which is monitored by a camera 2.
- An example of a camera is shown in Fig. 2 .
- This camera 2 has an image sensor (not shown in Fig. 2 ) and a lens 3.
- the camera 2 is a so called PTZ camera with capability to pan around a pan axis 4 and to tilt around a tilt axis 5.
- the camera 2 is also capable of zooming.
- Fig. 3 is a block diagram showing some of the components of the camera 2.
- an image sensor 6 arranged to capture images may be seen, as well as a PT controller 7 for controlling panning and tilting of the camera 2.
- the camera 2 has a zoom controller 8 for controlling zooming.
- the camera 2 will also have additional components, known in the art, which are not relevant for the invention, and which will therefore not be shown or discussed here.
- Fig. 4 The basic principles of a process for capturing and encoding images of the scene 1 with the camera 2 will now be described with reference to Fig. 4 .
- Light from the scene 1 enters the camera 2 through the lens 3 and is captured 10 by the image sensor.
- the data from the image sensor 6 is in the form of grey scale intensities, generally with each pixel being represented by a respective digital value.
- the captured image, or more specifically the data representing the image then undergoes pre-processing 11.
- Pre-processing 11 may involve one or more pre-processing procedures, such as Bayer pattern sampling, demosaicing, sharpening and tone mapping.
- the pre-processing procedures generally aim at providing a higher quality or more usable image.
- the pre-processed image is thereafter ready for encoding 12.
- Encoding may be done according to any compression format.
- the camera 2 moves during capture of a video sequence, be it because of a planned or spontaneous intentional panning, tilting or zooming operation or because of unintentional vibrations, the images may become blurry. Further, the movement leads to increased bitrates for representing images, since there is less redundancy between images of a moving scene than between images of a static scene. The increased bitrate may be too high for transmitting video if bandwidth is very restricted, such as on a mobile network, thereby making it impossible to transmit any images, e.g., during fast panning or tilting movements of the camera. As mentioned above, the bitrate problem is sometimes dealt with by freezing an image during panning, tilting and/or zooming.
- images should not be frozen during movement of the camera. Although this may lead to higher bitrate, the visual information available during movement may be worth it. However, the inventors have realised that part of the bitrate increase may be avoided by modifying the pre-processing applied before encoding.
- sharpening works by increasing differences in the image, and may thereby also lead to increased noise.
- the differences and noise lead to higher bitrates when encoding the sharpened image.
- the increase in bit rate may be curbed.
- tone mapping used for improving the dynamic range of an image adds differences in the bright and dark areas of the image and may add noise, thereby increasing bitrate. Disabling tone mapping during movement may inhibit part of the bitrate increase.
- Reduction or disabling of sharpening may even be taken as far as actually adding blur during movement of the camera.
- differences may be reduced, e.g., by calculating an average luminance value of a block or group of pixels and using that as the luminance value of all pixels in that block. In this manner pixel values are "smeared" over areas of the image.
- Some pre-processing procedures may be modified during movement of the camera although not thereby contributing to the goal of lowering bitrate, but actually improving image quality or image usability.
- Some cameras improve the dynamic range in the images by capturing two successive images within a short period of time. For one of the images a low exposure is used, resulting in an image where bright areas of the scene may be scene clearly, but where dark areas lack details and are more or less completely black. For the other image a high exposure is used, resulting in an image where the bright areas are overexposed or saturated, appearing completely white, and where the dark areas show details. By selectively using areas of one or the other image, a merged image is produced in which both dark and bright areas of the scene may be shown in detail.
- HDR high dynamic range
- WDR wide dynamic range
- Image or video statistics such as luminance values, chromaticity values, variance of luminance and/or chromaticity, motion information, autofocus values, moving object counter, bitrate, or filter parameters, gathered over time during a video sequence, may be reset when movement of the image sensor is determined. In this way, the movement will not corrupt the statistics.
- information on movement of the camera 2, or more precisely the image sensor 6, is obtained.
- This information may be provided by the PT controller 7, which has this information anyway for controlling panning and tilting of the camera 2.
- information may be provided by the zoom controller 8.
- the camera 2 may also have a motion sensor 9, e.g., an accelerometer, which senses movement of the image sensor.
- the motion sensor 9 may provide information on movement of the image sensor 6 when the camera 2 is unintentionally vibrating or shaking. It may also provide information on intentional movement of the image sensor 6 in addition to or instead of the PT controller 7.
- the movement may be a desired movement or an undesired movement. Desired movements may, e.g., be controlled using an input device such as a joystick. Desired movements may also be planned and pre-programmed as so called guard tours. Undesired movements may, e.g., be vibrations caused by wind or traffic, or by insufficient fixing of the camera.
- temporal noise filtering may be modified such that filtering is gradually made more severe, and may then be kept at an increased level until the end of the movement.
- An increased noise filtering may lead to loss of details in the pre-processed image, but again, during movement of the camera, this will in most cases not be of concern for the usability of the images.
- Filtering may be done temporally as well as spatially. One or both of these two types may be used, and a balance there between may be modified during movement, such that more or only temporal filtering is done when the image sensor is stationary, and more or only spatial filtering is done when the image sensor is moving.
- encoding may also be modified. If it is known that the camera will move, it is implicitly known that there will be apparent movement in images, leading to larger inter-frames. When movement starts, an intra-frame may therefore be forced, thereby providing a better reference for subsequent inter-frames.
- a first digital image is received (step S01) and information representative of a movement of the image sensor at a time of capture of the first image is received (step S02).
- the first digital image is pre-processed (step S03). Based on the information representative of movement, it is decided if and how one or more parameters of the pre-processing is to be modified (step S04). For instance, the movement may be compared to a predetermined threshold, and if the movement is above that threshold, one or more pre-processing procedures are disabled, enabled or otherwise modified.
- the pre-processing may be modified based on information representative of movement received for a previous image, such that a movement of the image sensor will affect the pre-processing only of subsequently captured images.
- parameters may be modified immediately, such that a movement of the image sensor will affect the pre-processing of the image for which the information representative of movement of the image sensor was received, and possibly also of subsequently captured images.
- the first image is encoded (step S05), such that it may be transmitted and/or stored.
- Fig. 6 an embodiment of a digital image pre-processing system 20 is shown.
- the system 20 has an image receiving module 21, which is arranged to receive input images, and a movement information receiving module 22, which is arranged to receive information representative of a movement of the image sensor at a time of capture of the input images.
- the system 20 has a pre-processing module 23 arranged to pre-process the input images. At least one parameter of the pre-processing is dependent on the information representative of movement of the image sensor.
- the images may be encoded by an encoder (not shown).
- the encoder may be integrated in the pre-processing system 20, or it may be separate from the pre-processing system 20.
- the pre-processing system 20 may be integrated in a camera, such as the camera 102 schematically shown in Fig. 7 .
- a camera such as the camera 102 schematically shown in Fig. 7 .
- components of the camera 102 that are the same as in the camera 2 shown in Fig. 3 are indicated with the same reference numerals, but with the addition of one hundred, and will not be discussed further here.
- the pre-processing system 20 may be a separate unit, operatively connected to a camera, such as the camera 2 in Fig. 3 .
- the pre-processing system may be embodied as hardware, firmware, software, or a combination thereof.
- the PTZ controller is described as one controller for panning and tilting and one for zooming. However, it may as well be made up of separate controllers for each of panning, tilting and zooming, respectively, or of one common controller for panning, tilting and zooming.
- the predetermined threshold against which the movement of the image sensor is compared may be pre-programmed or set by user input.
- a plurality of thresholds may be used. For instance, if movement is above a first, low threshold, one or more pre-processing procedures may be reduced, and if movement is above a second, higher threshold, these pre-processing procedures may be disabled.
- a parameter of the pre-processing may depend on the movement, e.g., as a linear function. Thus, with increasing movement, the parameter may be gradually decreased. As an example, if the camera is slowly panning or tilting, sharpening may be slightly reduced, and if the camera is moving more quickly, sharpening may be more severely reduced or even disabled, the sharpening thus being dependent on the movement level.
- the pre-processing system may be embodied as software, firmware, hardware, or a combination thereof.
- the pre-processing module of the pre-processing system may be arranged to perform all the pre-processing procedures to be applied on the images.
- the pre-processing module may be divided into several submodules, independent or cooperating, each performing just one or a few pre-processing procedures.
- the camera may be a monitoring camera. Further, the camera may be any type of camera, e.g., a camera employing visible light, an IR camera or a thermal camera.
- the camera may be a digital camera, but the invention may also be used with analogue cameras. In such case, images from an analogue camera may be converted to digital format using a digitalisation unit.
- the image sensor capturing the image frames may be arranged in another type of image capturing device.
- the images may also be generated by a visual light sensor, a thermal sensor, a time-of-flight sensor, or other types of image generating sensors.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Studio Devices (AREA)
- Geometry (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
Description
- The present invention relates to the field of digital image processing, and more particularly to pre-processing of digital images in preparation for encoding.
- In many uses of digital images, e.g., in monitoring systems, the digital images are compressed using various encoding schemes for transmission or storage. Before encoding, pre-processing of the digital images is performed. Pre-processing may for instance involve sharpening or noise filtering, in order to improve the quality of the images. However, a problem arises when the image sensor capturing the images moves during capture. Firstly, the ensuing changes in the images lead to a higher output bit rate from the encoder, because there is less temporal redundancy between the images. If bandwidth is very restricted, such as when transmitting over a mobile network, it may even be impossible to transmit video from a camera when fast movement causes increased bitrate. Additionally, the movement may make it computationally heavier to pre-process the image. On top of that, the pre-processing may add to the number of bits required for representing the images compared to a corresponding image captured when the image sensor is stationary. One solution to this problem is to freeze encoding of images during panning and/or tilting of a camera comprising the image sensor, and to resume encoding new input image frames only after movement has ceased. However, that means visual information is lost during movement, and an operator watching the transmitted image sequence may be disoriented. Thus, there is a need for an improved encoding approach.
- It is an object of the present invention to provide a method of pre-processing digital images for encoding, which allows a lower bitrate when the image sensor capturing the images is moving. Another object of the invention is to provide a method of pre-processing which makes it possible to avoid unnecessarily heavy computations when the image sensor is moving. A further object is to provide a method of pre-processing digital images for encoding, which enables visual information also during movement of the image sensor. Still another object is to provide a pre-processing method that may provide images with improved usability during movement of the image sensor.
- It is also an object of the invention to provide a pre-processing system that may avoid unnecessarily high bitrate and unnecessarily high computational usage when the image sensor capturing the images is moving. Further, it is an object of the invention to provide a pre-processing system which may enable improved usability of images during movement of the image sensor.
- An additional object is to provide a pre-processing system enabling provision of images also when the image sensor is moving.
- Yet another object is to provide a computer program product that enables efficient bitrate also when the image sensor capturing the images is moving.
- According to a first aspect, these objects are achieved, in full or at least in part, by a method of pre-processing digital images captured by an image sensor for encoding, comprising: receiving a first digital image, receiving information representative of a movement of the image sensor at a time of capture of said first digital image, and pre-processing said first digital image, wherein at least one parameter of said pre-processing is dependent on said information representative of movement. In this manner, pre-processing may be adapted such that the movement of the image sensor does not cause unnecessarily high bitrate. Thereby, images may be encoded and stored or transmitted for displaying also during periods of movement of the image sensor, without the cost of increased bitrate. Further, since the images captured during movement of the image sensor, e.g., during a fast panning movement, will generally be blurry, a loss of image quality possibly caused by modifying the pre-processing will have little impact on the usability of such images.
- If the movement of the image sensor is above a predetermined threshold, said at least one parameter of said pre-processing may be modified such that the first digital image is less processed by said pre-processing than if the movement were equal to or below said threshold.
- Further, said at least one parameter of said pre-processing may be modified such that at least one pre-processing procedure is disabled. For instance, sharpening may be dispensed with when the image sensor is moving, as the images would not really benefit from sharpening anyway.
- According to another variant, said at least one parameter is modified such that at least one pre-processing procedure processes said first digital image, but to a greater or lesser extent. As an example, filtering may be applied to a greater degree. As a further example, tone mapping may be performed, but to a lesser degree.
- According to yet another variant, said at least one parameter of said pre-processing is modified such that at least one pre-processing procedure that is otherwise disabled is enabled. For instance, blurring may be applied during movement of the image sensor, thereby making blocks of pixels in the images more or less homogeneous, which leads to lower bit requirement for representation of the image.
- The pre-processing may comprise performing at least one pre-processing procedure chosen from the group consisting of sharpening, adding contrast, tone mapping, colour correction, spatial and/or temporal noise filtering, blurring, Bayer pattern sampling, demosaicing, HDR merging, controlling focus, controlling exposure, and controlling gain.
- The information representative of a movement of the image sensor may be provided by a pan, tilt and/or zoom controller controlling movement of said image sensor. In this manner, adaptation of the pre-processing may be made in view of intentional movement of the image. It may here be noted, that zooming does not in itself imply a movement of the image sensor. However, for purposes of encoding images, zooming will induce changes in the captured images analogous to actual movement of the image sensor. Therefore, in the context of this application, a zoom operation will be seen as equivalent to a movement of the image sensor.
- The information representative of a movement of the image sensor may be provided by a motion sensor. Hereby, undesired movement caused by shaking may be taken into account. Information from the motion sensor may also be used for providing information on pan, tilt and/or zoom operations.
- According to a variant of the method, said at least one parameter is modified gradually in preparation for a planned movement of the image sensor. In this manner, e.g., harder noise filtering may be added gradually leading up to a planned movement, or sharpening may be gradually decreased. Thereby, abrupt changes in the appearance of the images may be avoided.
- The method may further comprise encoding said first digital image as an intra-frame in preparation for a planned movement of the image sensor. If there is movement in the captured images, inter-frames will become larger, due to larger differences between frames. At some point, the differences will be large enough that an intra-frame, not relying on other image frames for its encoding, will be more efficient. If it is known that there will be movement, an intra-frame can be forced, such that unnecessarily large inter-frames may be avoided. By forcing, or inserting, an intra-frame, the encoded frame may have higher quality, as may subsequent images encoded as inter-frames with reference to that intra-frame. The subsequent inter-frames may also require fewer bits because of having a more suitable reference frame.
- According to a second aspect, the above-mentioned objects are achieved, in full or at least in part, by a digital image pre-processing system for pre-processing input images captured by an image sensor for encoding, the system comprising: an image receiving module arranged to receive input images, a movement information receiving module arranged to receive information representative of a movement of the image sensor at a time of capture of said first digital image, and a pre-processing module arranged to pre-process the input images before encoding, wherein at least one parameter of said pre-processing is dependent on said information representative of movement. By means of such a system, it may be possible to provide images during movement of the image sensor, without the cost of unnecessarily high output bitrates.
- The pre-processing module may be arranged to perform at least one pre-processing procedure chosen from the group consisting of sharpening, adding contrast, tone mapping, colour correction, temporal and/or spatial noise filtering, blurring, Bayer pattern sampling, demosaicing, HDR merging, controlling focus, controlling exposure, and controlling gain.
- In an embodiment of the digital image pre-processing, the movement information receiving module is communicatively connected to a pan, tilt and/or zoom controller controlling movement of said image sensor. Thereby, information on intentional movement may be provided.
- Alternatively or additionally, the movement information receiving module may be communicatively connected to a motion sensor. As discussed in connection with the method, information on unintentional as well as intentional movement may thereby be provided.
- According to a third aspect, the above-mentioned objects are achieved, in full or at least in part, by a camera comprising a pre-processing system according to the second aspect. Such a camera may generally be embodied in the same ways as the second aspect, with accompanying advantages.
- According to a fourth aspect, the above-mentioned objects are achieved, in full or at least in part, by a computer program product comprising a computer-readable storage medium with instructions adapted to carry out the method according to the first aspect when executed by a processor. The processor may be any kind of processor, e.g., a central processing unit (CPU), a graphics processing unit (GPU), a custom made processing device implemented in an integrated circuit, an ASIC, an FPGA, or logical circuitry including discrete components. The computer-program product may generally be varied in the same ways as the first aspect with accompanying advantages.
- A further scope of applicability of the present invention will become apparent from the detailed description given below. However, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the scope of the invention will become apparent to those skilled in the art from this detailed description.
- Hence, it is to be understood that this invention is not limited to the particular component parts of the device described or steps of the methods described as such device and method may vary. It is also to be understood that the terminology used herein is for purpose of describing particular embodiments only, and is not intended to be limiting. It must be noted that, as used in the specification and the appended claim, the articles "a", "an", "the", and "said" are intended to mean that there are one or more of the elements unless the context clearly dictates otherwise. Thus, for example, a reference to "an object" or "the object" may include several objects, and the like. Furthermore, the word "comprising" does not exclude other elements or steps.
- The invention will now be described in more detail by way of example and with reference to the accompanying schematic drawings, in which:
-
Fig. 1 is an illustration of a scene being monitored by a camera, -
Fig. 2 is a front view of a camera with pan/tilt/zoom capability, -
Fig. 3 is a block diagram of the camera ofFig. 2 , -
Fig. 4 is a block diagram showing some functions in a process of capturing, pre-processing and encoding images, -
Fig. 5 is a flow chart of a method of pre-processing digital images, -
Fig. 6 is a block diagram of an embodiment of a digital image pre-processing system, and -
Fig. 7 is a block diagram of a camera incorporating a digital image pre-processing system. - In
Fig. 1 , ascene 1 is shown, which is monitored by acamera 2. An example of a camera is shown inFig. 2 . Thiscamera 2 has an image sensor (not shown inFig. 2 ) and alens 3. Thecamera 2 is a so called PTZ camera with capability to pan around apan axis 4 and to tilt around atilt axis 5. Thecamera 2 is also capable of zooming. -
Fig. 3 is a block diagram showing some of the components of thecamera 2. Here, animage sensor 6 arranged to capture images may be seen, as well as aPT controller 7 for controlling panning and tilting of thecamera 2. Further, thecamera 2 has azoom controller 8 for controlling zooming. Thecamera 2 will also have additional components, known in the art, which are not relevant for the invention, and which will therefore not be shown or discussed here. - The basic principles of a process for capturing and encoding images of the
scene 1 with thecamera 2 will now be described with reference toFig. 4 . Light from thescene 1 enters thecamera 2 through thelens 3 and is captured 10 by the image sensor. The data from theimage sensor 6 is in the form of grey scale intensities, generally with each pixel being represented by a respective digital value. The captured image, or more specifically the data representing the image, then undergoespre-processing 11. Pre-processing 11 may involve one or more pre-processing procedures, such as Bayer pattern sampling, demosaicing, sharpening and tone mapping. The pre-processing procedures generally aim at providing a higher quality or more usable image. The pre-processed image is thereafter ready for encoding 12. Encoding may be done according to any compression format. - If the
camera 2 moves during capture of a video sequence, be it because of a planned or spontaneous intentional panning, tilting or zooming operation or because of unintentional vibrations, the images may become blurry. Further, the movement leads to increased bitrates for representing images, since there is less redundancy between images of a moving scene than between images of a static scene. The increased bitrate may be too high for transmitting video if bandwidth is very restricted, such as on a mobile network, thereby making it impossible to transmit any images, e.g., during fast panning or tilting movements of the camera. As mentioned above, the bitrate problem is sometimes dealt with by freezing an image during panning, tilting and/or zooming. In this manner, no images are pre-processed and encoded during movement, thereby leading to a lower bitrate. However, such an approach means that the operator is provided with less or no visual information during movement. The operator therefore has little chance of noting suspicious objects or behaviour occurring in areas of the scene that are transitioned during movement. The lack of visual information during movement may also lead to disorientation, since the operator is only provided with images of the start position of the movement and of the end position of the movement, and none there between. These problems are the same for live display of video sequences as for recorded video sequences. - According to the invention, images should not be frozen during movement of the camera. Although this may lead to higher bitrate, the visual information available during movement may be worth it. However, the inventors have realised that part of the bitrate increase may be avoided by modifying the pre-processing applied before encoding.
- For instance, sharpening works by increasing differences in the image, and may thereby also lead to increased noise. The differences and noise, in turn, lead to higher bitrates when encoding the sharpened image. By reducing or even disabling sharpening during movement of the camera, or more precisely the image sensor, the increase in bit rate may be curbed.
- Similarly, tone mapping used for improving the dynamic range of an image adds differences in the bright and dark areas of the image and may add noise, thereby increasing bitrate. Disabling tone mapping during movement may inhibit part of the bitrate increase.
- In the same way, other pre-processing procedures that would otherwise add to the bitrate may be disabled or applied to a lesser extent. Intuitively, this will lower the image quality, but since images captured during movement of the camera are generally blurry, the lowered image quality will in most cases not be of any concern to the operator. Further, during fast movement, the images may not even benefit from pre-processing.
- Reduction or disabling of sharpening may even be taken as far as actually adding blur during movement of the camera. Thus, instead of increasing differences in the image, differences may be reduced, e.g., by calculating an average luminance value of a block or group of pixels and using that as the luminance value of all pixels in that block. In this manner pixel values are "smeared" over areas of the image.
- Some pre-processing procedures may be modified during movement of the camera although not thereby contributing to the goal of lowering bitrate, but actually improving image quality or image usability. Some cameras improve the dynamic range in the images by capturing two successive images within a short period of time. For one of the images a low exposure is used, resulting in an image where bright areas of the scene may be scene clearly, but where dark areas lack details and are more or less completely black. For the other image a high exposure is used, resulting in an image where the bright areas are overexposed or saturated, appearing completely white, and where the dark areas show details. By selectively using areas of one or the other image, a merged image is produced in which both dark and bright areas of the scene may be shown in detail. Such pre-processing procedures are referred to as HDR (high dynamic range) merging or WDR (wide dynamic range) merging. For such merging to give a good result, the two differently exposed images have to be captured with such short time between them that they show the same scene, or else objects moving in the scene between the two captures will appear blurry. However, if the camera is moving it might not be possible to capture the two images sufficiently close together in time for them to actually show the same scene. The entire image may therefore appear blurry. In order to remedy this problem, HDR merging may be disabled during movement of the camera. Thus, only one of the two differently exposed images may be encoded, or capturing of two differently exposed images may be disabled, such that just one image is captured and encoded.
- Image or video statistics, such as luminance values, chromaticity values, variance of luminance and/or chromaticity, motion information, autofocus values, moving object counter, bitrate, or filter parameters, gathered over time during a video sequence, may be reset when movement of the image sensor is determined. In this way, the movement will not corrupt the statistics.
- In order to ascertain that a modification of one or more of the pre-processing procedures should be made, information on movement of the
camera 2, or more precisely theimage sensor 6, is obtained. This information may be provided by thePT controller 7, which has this information anyway for controlling panning and tilting of thecamera 2. Similarly, such information may be provided by thezoom controller 8. Thecamera 2 may also have amotion sensor 9, e.g., an accelerometer, which senses movement of the image sensor. Themotion sensor 9 may provide information on movement of theimage sensor 6 when thecamera 2 is unintentionally vibrating or shaking. It may also provide information on intentional movement of theimage sensor 6 in addition to or instead of thePT controller 7. - As already discussed, the movement may be a desired movement or an undesired movement. Desired movements may, e.g., be controlled using an input device such as a joystick. Desired movements may also be planned and pre-programmed as so called guard tours. Undesired movements may, e.g., be vibrations caused by wind or traffic, or by insufficient fixing of the camera.
- If a movement of the camera is planned, it is possible to make modifications of one or more pre-processing procedures in preparation for the movement. For instance, temporal noise filtering may be modified such that filtering is gradually made more severe, and may then be kept at an increased level until the end of the movement. An increased noise filtering may lead to loss of details in the pre-processed image, but again, during movement of the camera, this will in most cases not be of concern for the usability of the images. Filtering may be done temporally as well as spatially. One or both of these two types may be used, and a balance there between may be modified during movement, such that more or only temporal filtering is done when the image sensor is stationary, and more or only spatial filtering is done when the image sensor is moving.
- In preparation of a planned movement, encoding may also be modified. If it is known that the camera will move, it is implicitly known that there will be apparent movement in images, leading to larger inter-frames. When movement starts, an intra-frame may therefore be forced, thereby providing a better reference for subsequent inter-frames.
- With reference to
Fig. 5 , the method will now be described in more general terms. A first digital image is received (step S01) and information representative of a movement of the image sensor at a time of capture of the first image is received (step S02). The first digital image is pre-processed (step S03). Based on the information representative of movement, it is decided if and how one or more parameters of the pre-processing is to be modified (step S04). For instance, the movement may be compared to a predetermined threshold, and if the movement is above that threshold, one or more pre-processing procedures are disabled, enabled or otherwise modified. - The pre-processing may be modified based on information representative of movement received for a previous image, such that a movement of the image sensor will affect the pre-processing only of subsequently captured images. For some pre-processing procedures, parameters may be modified immediately, such that a movement of the image sensor will affect the pre-processing of the image for which the information representative of movement of the image sensor was received, and possibly also of subsequently captured images.
- After pre-processing (S03), the first image is encoded (step S05), such that it may be transmitted and/or stored.
- In
Fig. 6 an embodiment of a digitalimage pre-processing system 20 is shown. Thesystem 20 has animage receiving module 21, which is arranged to receive input images, and a movementinformation receiving module 22, which is arranged to receive information representative of a movement of the image sensor at a time of capture of the input images. Further, thesystem 20 has apre-processing module 23 arranged to pre-process the input images. At least one parameter of the pre-processing is dependent on the information representative of movement of the image sensor. After pre-processing, the images may be encoded by an encoder (not shown). The encoder may be integrated in thepre-processing system 20, or it may be separate from thepre-processing system 20. Thepre-processing system 20 may be integrated in a camera, such as thecamera 102 schematically shown inFig. 7 . InFig. 7 , components of thecamera 102 that are the same as in thecamera 2 shown inFig. 3 are indicated with the same reference numerals, but with the addition of one hundred, and will not be discussed further here. - Alternatively, the
pre-processing system 20 may be a separate unit, operatively connected to a camera, such as thecamera 2 inFig. 3 . - The pre-processing system may be embodied as hardware, firmware, software, or a combination thereof.
- It will be appreciated that a person skilled in the art can modify the above described embodiments in many ways and still use the advantages of the invention as shown in the embodiments above. As an example, a couple of pre-processing procedures have been mentioned, but the invention is equally applicable to other pre-processing procedures applied in preparation for encoding a digital image, e.g., adding contrast, colour correction, Bayer pattern sampling, demosaicing or controlling gain. It will be understood that pre-processing procedures that tend to add noise may be reduced or disabled in order to lower the output bitrate, whereas pre-processing procedures that tend to reduce noise may be enabled or increased to the same end.
- In connection with
Fig. 3 , the PTZ controller is described as one controller for panning and tilting and one for zooming. However, it may as well be made up of separate controllers for each of panning, tilting and zooming, respectively, or of one common controller for panning, tilting and zooming. - The predetermined threshold against which the movement of the image sensor is compared may be pre-programmed or set by user input.
- Instead of using just one threshold, a plurality of thresholds may be used. For instance, if movement is above a first, low threshold, one or more pre-processing procedures may be reduced, and if movement is above a second, higher threshold, these pre-processing procedures may be disabled. A parameter of the pre-processing may depend on the movement, e.g., as a linear function. Thus, with increasing movement, the parameter may be gradually decreased. As an example, if the camera is slowly panning or tilting, sharpening may be slightly reduced, and if the camera is moving more quickly, sharpening may be more severely reduced or even disabled, the sharpening thus being dependent on the movement level.
- The pre-processing system may be embodied as software, firmware, hardware, or a combination thereof.
- The pre-processing module of the pre-processing system may be arranged to perform all the pre-processing procedures to be applied on the images. Alternatively, the pre-processing module may be divided into several submodules, independent or cooperating, each performing just one or a few pre-processing procedures.
- In the examples above, the invention is described in connection with a camera. The camera may be a monitoring camera. Further, the camera may be any type of camera, e.g., a camera employing visible light, an IR camera or a thermal camera. The camera may be a digital camera, but the invention may also be used with analogue cameras. In such case, images from an analogue camera may be converted to digital format using a digitalisation unit.
- Instead of in a camera, the image sensor capturing the image frames may be arranged in another type of image capturing device.
- The images may also be generated by a visual light sensor, a thermal sensor, a time-of-flight sensor, or other types of image generating sensors.
- Thus, the invention should not be limited to the shown embodiments but should only be defined by the appended claims.
Claims (15)
- A method of pre-processing digital images captured by an image sensor (6; 106) for encoding, comprising:receiving a first digital image (S01),receiving information representative of a movement of the image sensor at a time of capture of said first digital image (S02), andpre-processing (S03) said first digital image,wherein at least one parameter of said pre-processing is dependent on said information representative of movement (S04).
- The method according to claim 1, wherein if said movement of the image sensor is above a predetermined threshold, said at least one parameter of said pre-processing (S03) is modified (S04) such that the first digital image is less processed by said pre-processing than if said movement were equal to or below said threshold.
- The method according to claim 2, wherein said at least one parameter of said pre-processing is modified (S04) such that at least one pre-processing procedure is disabled.
- The method according to claim 2 or 3, wherein said at least one parameter is modified (S04) such that at least one pre-processing procedure processes said first digital image, but to a greater or lesser extent.
- The method according to claim 1, wherein said at least one parameter of said pre-processing is modified (S04) such that at least one pre-processing procedure that is otherwise disabled is enabled.
- The method according to any one of the preceding claims, wherein said pre-processing (S03) comprises performing at least one pre-processing procedure chosen from the group consisting of sharpening, adding contrast, tone mapping, colour correction, noise filtering, blurring, Bayer pattern sampling, demosaicing, HDR merging, controlling focus, controlling exposure, and controlling gain.
- The method according to any one of the preceding claims, wherein said information representative of a movement of the image sensor (6; 106) is provided by a pan, tilt and/or zoom controller (7, 8; 107, 108) controlling movement of said image sensor (6; 106).
- The method according to any one of the preceding claims, wherein said information representative of a movement of the image sensor is provided by a motion sensor (9; 109).
- The method according to any one of the preceding claims, wherein said at least one parameter is modified (S04) gradually in preparation for a planned movement of the image sensor (6; 106).
- The method according to any one of the preceding claims, further comprising encoding (S05) said first digital image as an intra-frame in preparation for a planned movement of the image sensor (6; 106).
- A digital image pre-processing system for pre-processing input images captured by an image sensor (6; 106) for encoding, the system (20) comprising:an image receiving module (21) arranged to receive input images,a movement information receiving module (22) arranged to receive information representative of a movement of the image sensor at a time of capture of said input images, anda pre-processing module (23) arranged to pre-process the input images before encoding,wherein at least one parameter of said pre-processing is dependent on said information representative of movement.
- The digital image pre-processing system according to claim 11, wherein said pre-processing module (23) is arranged to perform at least one pre-processing procedure chosen from the group consisting of sharpening, adding contrast, tone mapping, colour correction, noise filtering, blurring, Bayer pattern sampling, demosaicing, HDR merging, controlling focus, controlling exposure, and controlling gain.
- The digital image pre-processing system according to claim 11 or 12, wherein said movement information receiving module (22) is communicatively connected to a pan, tilt and/or zoom controller (7, 8; 107, 108) controlling movement of said image sensor (6; 106) and/or to a motion sensor (9; 109).
- A camera comprising a pre-processing system according to any one of claims 11-13.
- A computer program product comprising a computer-readable storage medium with instructions adapted to carry out the method according to any one of claims 1-10 when executed by a processor.
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP15182692.2A EP3136726B1 (en) | 2015-08-27 | 2015-08-27 | Pre-processing of digital images |
JP2016159858A JP6751319B2 (en) | 2015-08-27 | 2016-08-17 | Digital image preprocessing method and digital image preprocessing system |
KR1020160106315A KR102192063B1 (en) | 2015-08-27 | 2016-08-22 | Method of pre-processing digital images, and digital image pre-processing system |
CN201610717860.6A CN106488117B (en) | 2015-08-27 | 2016-08-24 | The preprocess method of digital picture and digital image preprocessing system |
US15/246,951 US10110929B2 (en) | 2015-08-27 | 2016-08-25 | Method of pre-processing digital images, and digital image preprocessing system |
TW105127565A TWI689203B (en) | 2015-08-27 | 2016-08-26 | Method of pre-processing digital images, and digital image pre-processing system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP15182692.2A EP3136726B1 (en) | 2015-08-27 | 2015-08-27 | Pre-processing of digital images |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3136726A1 true EP3136726A1 (en) | 2017-03-01 |
EP3136726B1 EP3136726B1 (en) | 2018-03-07 |
Family
ID=54150220
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15182692.2A Active EP3136726B1 (en) | 2015-08-27 | 2015-08-27 | Pre-processing of digital images |
Country Status (6)
Country | Link |
---|---|
US (1) | US10110929B2 (en) |
EP (1) | EP3136726B1 (en) |
JP (1) | JP6751319B2 (en) |
KR (1) | KR102192063B1 (en) |
CN (1) | CN106488117B (en) |
TW (1) | TWI689203B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3429196A1 (en) * | 2017-07-11 | 2019-01-16 | Hanwha Techwin Co., Ltd. | Apparatus for processing image and method of processing image |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10291849B1 (en) | 2015-10-16 | 2019-05-14 | Tribune Broadcasting Company, Llc | Methods and systems for determining that a video-capturing device is unsteady |
CN107016687A (en) * | 2017-03-25 | 2017-08-04 | 平顶山学院 | The hybrid algorithm of video frequency motion target detect and track |
EP3503548B1 (en) * | 2017-12-19 | 2019-10-23 | Axis AB | Video encoding method and system |
EP3579553B1 (en) | 2018-06-05 | 2020-05-20 | Axis AB | A method, controller, and system for encoding a sequence of video frames |
US11050938B2 (en) | 2019-07-03 | 2021-06-29 | Gopro, Inc. | Apparatus and methods for pre-processing and stabilization of captured image data |
KR102583686B1 (en) * | 2020-03-25 | 2023-09-27 | 한국전자통신연구원 | Method and apparatus for detecting objects |
EP3985976B1 (en) * | 2020-10-16 | 2022-09-28 | Axis AB | Method of encoding an image including a privacy mask |
CN115049636A (en) * | 2022-07-12 | 2022-09-13 | 郑州工商学院 | Computer image processing method |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0520741A1 (en) * | 1991-06-24 | 1992-12-30 | Canon Kabushiki Kaisha | Image compression associated with camera shake correction |
EP1679884A1 (en) * | 2004-12-28 | 2006-07-12 | Seiko Epson Corporation | Imaging apparatus and portable device and portable telephone using the same |
US20100183071A1 (en) * | 2009-01-19 | 2010-07-22 | Segall Christopher A | Methods and Systems for Enhanced Dynamic Range Images and Video from Multiple Exposures |
US20130076921A1 (en) * | 2011-09-26 | 2013-03-28 | Christopher Owen | Video Stabilization |
WO2013090097A1 (en) * | 2011-12-15 | 2013-06-20 | Apple Inc. | Motion sensor based virtual tripod method for video stabilization |
US20130342714A1 (en) * | 2012-06-22 | 2013-12-26 | Apple Inc. | Automated tripod detection and handling in video stabilization |
Family Cites Families (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2223643B (en) | 1988-10-05 | 1993-04-07 | Marconi Gec Ltd | Data processing apparatus |
JPH08297738A (en) * | 1995-04-26 | 1996-11-12 | Hitachi Ltd | Image pickup device and picture input device |
JPH10136309A (en) * | 1996-11-01 | 1998-05-22 | Canon Inc | Image compression/storage device |
JPH11112844A (en) * | 1997-09-30 | 1999-04-23 | Canon Inc | Image processor, image processing method and computer readable storage medium |
JPH11317950A (en) * | 1998-04-30 | 1999-11-16 | Sony Corp | Data encoding device/method and data transmitting method |
JP2002354461A (en) * | 2001-05-23 | 2002-12-06 | Matsushita Electric Ind Co Ltd | Image server, image server system using the same and method for transferring image data |
US7505673B2 (en) * | 2003-09-30 | 2009-03-17 | At&T Intellectual Property I, L.P. | Video recorder for detection of occurrences |
JP4112502B2 (en) * | 2004-01-09 | 2008-07-02 | 富士フイルム株式会社 | Image generation apparatus and method, image distribution apparatus, mail relay apparatus, and program |
US7382400B2 (en) * | 2004-02-19 | 2008-06-03 | Robert Bosch Gmbh | Image stabilization system and method for a video camera |
JP2006191185A (en) * | 2004-12-28 | 2006-07-20 | Toshiba Corp | Image pickup apparatus |
JP4179279B2 (en) * | 2004-12-28 | 2008-11-12 | セイコーエプソン株式会社 | IMAGING DEVICE AND MOBILE DEVICE AND MOBILE PHONE HAVING THE SAME |
US9479794B2 (en) * | 2005-11-10 | 2016-10-25 | Freescale Semiconductor, Inc. | Resource efficient video processing via prediction error computational adjustments |
JP4781200B2 (en) * | 2006-08-10 | 2011-09-28 | キヤノン株式会社 | Digital camera, digital camera control method, program, storage medium, and image processing apparatus |
US7956899B2 (en) * | 2007-08-29 | 2011-06-07 | Sanyo Electric Co., Ltd. | Imaging device and image processing apparatus |
JP5191224B2 (en) * | 2007-12-07 | 2013-05-08 | イーストマン コダック カンパニー | Image processing device |
JP4613990B2 (en) * | 2008-07-31 | 2011-01-19 | ソニー株式会社 | Image processing apparatus, image processing method, and program |
US9998697B2 (en) | 2009-03-02 | 2018-06-12 | Flir Systems, Inc. | Systems and methods for monitoring vehicle occupants |
WO2010109900A1 (en) * | 2009-03-26 | 2010-09-30 | パナソニック株式会社 | Video processor, video processing method, integrated circuit for video processing, video playback device |
JP5625543B2 (en) * | 2010-06-28 | 2014-11-19 | ソニー株式会社 | ENCODING DEVICE, ELECTRONIC DEVICE, IMAGING DEVICE, AND IMAGING SYSTEM |
US20130021512A1 (en) * | 2011-07-20 | 2013-01-24 | Broadcom Corporation | Framing of Images in an Image Capture Device |
JP2013031038A (en) * | 2011-07-29 | 2013-02-07 | Brother Ind Ltd | Communication device, image acquisition method, and image acquisition program |
CA2851259C (en) | 2011-10-07 | 2018-06-12 | Flir Systems, Inc. | Smart surveillance camera systems and methods |
JP6033972B2 (en) * | 2013-02-28 | 2016-11-30 | レイセオン カンパニー | Method and apparatus for gain correction and level correction of a multi-tap CCD camera |
TW201503678A (en) | 2013-07-05 | 2015-01-16 | Shao-Yang Wang | Medical grade calibration system for display device and method thereof |
-
2015
- 2015-08-27 EP EP15182692.2A patent/EP3136726B1/en active Active
-
2016
- 2016-08-17 JP JP2016159858A patent/JP6751319B2/en active Active
- 2016-08-22 KR KR1020160106315A patent/KR102192063B1/en active IP Right Grant
- 2016-08-24 CN CN201610717860.6A patent/CN106488117B/en active Active
- 2016-08-25 US US15/246,951 patent/US10110929B2/en active Active
- 2016-08-26 TW TW105127565A patent/TWI689203B/en active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0520741A1 (en) * | 1991-06-24 | 1992-12-30 | Canon Kabushiki Kaisha | Image compression associated with camera shake correction |
EP1679884A1 (en) * | 2004-12-28 | 2006-07-12 | Seiko Epson Corporation | Imaging apparatus and portable device and portable telephone using the same |
US20100183071A1 (en) * | 2009-01-19 | 2010-07-22 | Segall Christopher A | Methods and Systems for Enhanced Dynamic Range Images and Video from Multiple Exposures |
US20130076921A1 (en) * | 2011-09-26 | 2013-03-28 | Christopher Owen | Video Stabilization |
WO2013090097A1 (en) * | 2011-12-15 | 2013-06-20 | Apple Inc. | Motion sensor based virtual tripod method for video stabilization |
US20130342714A1 (en) * | 2012-06-22 | 2013-12-26 | Apple Inc. | Automated tripod detection and handling in video stabilization |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3429196A1 (en) * | 2017-07-11 | 2019-01-16 | Hanwha Techwin Co., Ltd. | Apparatus for processing image and method of processing image |
KR20190006725A (en) * | 2017-07-11 | 2019-01-21 | 한화테크윈 주식회사 | Image process device and image processing method |
US10778878B2 (en) | 2017-07-11 | 2020-09-15 | Hanwha Techwin Co., Ltd. | Apparatus for processing image and method of processing image |
Also Published As
Publication number | Publication date |
---|---|
CN106488117A (en) | 2017-03-08 |
EP3136726B1 (en) | 2018-03-07 |
US10110929B2 (en) | 2018-10-23 |
JP2017098934A (en) | 2017-06-01 |
TWI689203B (en) | 2020-03-21 |
JP6751319B2 (en) | 2020-09-02 |
US20170064338A1 (en) | 2017-03-02 |
TW201724841A (en) | 2017-07-01 |
KR102192063B1 (en) | 2020-12-16 |
KR20170026167A (en) | 2017-03-08 |
CN106488117B (en) | 2019-11-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3136726B1 (en) | Pre-processing of digital images | |
JP3870124B2 (en) | Image processing apparatus and method, computer program, and computer-readable storage medium | |
EP2742680B1 (en) | Video stabilisation | |
KR102067199B1 (en) | Method and encoder system for encoding video | |
WO2010095207A1 (en) | Video image pickup device | |
US8493499B2 (en) | Compression-quality driven image acquisition and processing system | |
EP3506228A1 (en) | Image capturing apparatus, image processing apparatus, control method, and program | |
KR100426174B1 (en) | Method for controlling a camera using video compression algorithm | |
CN111988610B (en) | Method and bit rate controller for controlling the output bit rate of a video encoder | |
US10142664B2 (en) | Method and device for determining properties of a graphical overlay for a video stream | |
KR101900889B1 (en) | A bit rate controller and a method for limiting output bit rate | |
EP3362988A1 (en) | A method and system for stabilizing video frames | |
CN105827935A (en) | Terminal screenshot method and terminal | |
CN113630600A (en) | Human visual system adaptive video coding | |
EP3468188B1 (en) | Method and encoder for encoding video streams | |
US10764508B2 (en) | Image processing apparatus, control method thereof, and storage medium | |
JP7435208B2 (en) | Image processing device and program | |
EP2495972A1 (en) | Monitoring device and method for monitoring a location |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20160922 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20170922 |
|
GRAJ | Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted |
Free format text: ORIGINAL CODE: EPIDOSDIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTC | Intention to grant announced (deleted) | ||
INTG | Intention to grant announced |
Effective date: 20171213 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP Ref country code: AT Ref legal event code: REF Ref document number: 977788 Country of ref document: AT Kind code of ref document: T Effective date: 20180315 |
|
REG | Reference to a national code |
Ref country code: SE Ref legal event code: TRGR |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602015008558 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20180307 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 4 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180607 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 977788 Country of ref document: AT Kind code of ref document: T Effective date: 20180307 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180607 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180608 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602015008558 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180709 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 |
|
26N | No opposition filed |
Effective date: 20181210 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180827 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180831 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180831 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20180831 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180831 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180827 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180307 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180307 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20150827 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180827 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20180707 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230505 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20230720 Year of fee payment: 9 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: SE Payment date: 20230720 Year of fee payment: 9 Ref country code: FR Payment date: 20230720 Year of fee payment: 9 Ref country code: DE Payment date: 20230720 Year of fee payment: 9 |