EP2396744B1 - High-resolution optical code imaging using a color imager - Google Patents
High-resolution optical code imaging using a color imager Download PDFInfo
- Publication number
- EP2396744B1 EP2396744B1 EP10741731.3A EP10741731A EP2396744B1 EP 2396744 B1 EP2396744 B1 EP 2396744B1 EP 10741731 A EP10741731 A EP 10741731A EP 2396744 B1 EP2396744 B1 EP 2396744B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- sensor elements
- light
- infrared
- optical code
- reading device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
- G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
- G06K7/10544—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation by scanning of the records by radiation in the optical part of the electromagnetic spectrum
- G06K7/10712—Fixed beam scanning
- G06K7/10722—Photodetector array or CCD scanning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
- G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
- G06K7/10544—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation by scanning of the records by radiation in the optical part of the electromagnetic spectrum
- G06K7/10712—Fixed beam scanning
- G06K7/10722—Photodetector array or CCD scanning
- G06K7/10732—Light sources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
- G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
- G06K7/12—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation using a selected wavelength, e.g. to sense red marks and ignore blue marks
Definitions
- the field of this disclosure relates generally to systems and methods of data reading, and more particularly but not exclusively to reading of optical codes (e.g., bar codes).
- optical codes e.g., bar codes
- Optical codes encode useful, optically-readable information about the items to which they are attached or otherwise associated.
- an optical code is the bar code.
- Bar codes are ubiquitously found on or associated with objects of various types, such as the packaging of retail, wholesale, and inventory goods; retail product presentation fixtures (e.g., shelves); goods undergoing manufacturing; personal or company assets; and documents.
- a bar code typically serves as an identifier of an object, whether the identification be to a class of objects (e.g., containers of milk) or a unique item.
- a typical linear or one-dimensional bar code such as a UPC code, consist of alternating bars (i.e., relatively dark areas) and spaces (i.e., relatively light areas).
- the pattern of alternating bars and spaces and the widths of those bars and spaces represent a string of binary ones and zeros, wherein the width of any particular bar or space is an integer multiple of a specified minimum width, which is called a "module" or "unit.”
- a bar code reader must be able to reliably discern the pattern of bars and spaces, such as by determining the locations of edges demarking adjacent bars and spaces from one another, across the entire length of the bar code.
- Bar codes are just one example of the many types of optical codes in use today.
- Higher-dimensional optical codes such as, two-dimensional matrix codes (e.g., MaxiCode) or stacked codes (e.g., PDF 417), which are also sometimes referred to as "bar codes," are also used for various purposes.
- optical code reading devices are available for capturing an optical code and for decoding the information represented by the optical code.
- image-based readers are available that include imagers, such as charge coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) imagers, that generate electronic image data that represent an image of a captured optical code.
- CCDs charge coupled devices
- CMOS complementary metal oxide semiconductor
- Image-based readers are used for reading one-dimensional optical codes and higher-dimensional optical codes. Because optical codes most often include dark and light patterns (e.g., black and white) that represent binary data, imagers of image-based readers are typically monochrome so that uniform sensitivity for each pixel of the imager is achieved.
- typical image-based readers include light sources that illuminate the image-based reader's field of view with narrowband visible light to achieve high optical resolution by avoiding chromatic aberration and polychromatic diffraction effects.
- Narrowband light sources typically used for imaging include laser diodes, having a bandwidth on the order of 5 nanometers (nm), and light emitting diodes (LEDs), having a bandwidth on the order of 50 nm.
- One embodiment is directed to an optical code reading device that includes a color image sensor array positioned to sense light reflected from an object in a field of view of the optical code reading device and to produce from the sensed reflected light image data representing an image of the object.
- the color image sensor array has a first set of sensor elements that are sensitive to a first visible wavelength band of light, and a second set of sensor elements that are sensitive to a second visible wavelength band of light. The first and second sets of sensor elements are also sensitive to light within an infrared wavelength band.
- the optical code reading device includes an artificial illumination source positioned to illuminate the field of view of the optical code reading device with light that is incident upon and reflected from the object in the field of view toward the image sensor array.
- the illumination source is operable to produce infrared light having wavelengths within the infrared wavelength band so that, upon illumination of the field of view, at least some sensor elements of each of the first and second sets are sensitive to the infrared light and contribute to production of the image data.
- FIG. 1 is a diagram of an imaging device 100, such as an optical code reading device, according to one embodiment.
- the imaging device 100 includes a color image sensor array 102 contained within a housing 103 of the imaging device 100.
- the color image sensor array 102 includes a first set of sensor elements 104 and a second set of sensor elements 106.
- the sensor elements of the color image sensor array 102 may be arranged in a one-dimensional array or, preferably, a two-dimensional array.
- the color image sensor array 102 may be a CCD, such as a frame-transfer or interline-transfer CCD.
- the color image sensor array 102 may alternatively be a CMOS imager, such as a global shuttered or rolling-reset CMOS imager.
- Suitable imagers or image sensor arrays are available, for example, from Aptina Imaging Corporation of San Jose, California, USA including, but not limited to, a model MT9V022 VGA color imager. Imagers are available from many manufacturers and are available in various resolutions (numbers of pixels). For higher resolution applications, other imagers from Aptina Imaging Corporation are suitable, including the model MT9M001 with 1.3 megapixels, the model MT9M002 with 1.6 megapixels, and the model MT9P001 with 5 megapixels. As imaging technology advances, imager resolution increases and other imagers may also be suitable.
- the color image sensor array 102 may include more than two sets of sensor elements.
- the color image sensor array 102 may include three sets of sensor elements 104, 106, and 202 arranged in a Bayer pattern as shown in Figure 2 . Each set of sensor elements corresponds to a different color.
- the first set 104 may be sensitive to light having wavelengths that correspond to the color green (G) (wavelengths ranging between about 500 nanometers (nm) and about 600 nm)
- the second set 106 may be sensitive to light having wavelengths that correspond to the color red (R) (wavelengths ranging between about 600 nm and about 750 nm)
- the third set 202 may be sensitive to light having wavelengths that correspond to the color blue (B) (wavelengths ranging between about 400 nm and about 500 nm).
- a color filter associated with each sensor element of the different sets appreciably filters out visible light that does not correspond to its color (i.e., the color filters associated with the first set 104 appreciably block out red and blue light).
- Figure 3 is a graph depicting an example of the quantum efficiency percentage versus the wavelength of light incident upon red, green and blue sensor elements of the model MT9V022 VGA color imager available from Aptina Imaging Corporation that may be used as the color image sensor array 102.
- a curve 104' corresponding to the spectral sensitivity of the sensor elements of the first set 104, has a local peak 104a' at a wavelength corresponding to the color green.
- a curve 106' corresponding to the spectral sensitivity of the sensor elements of the second set 106, has a local peak 106a' at a wavelength corresponding to the color red.
- a curve 202' corresponding to the spectral sensitivity of the sensor elements of the third set 202, has a local peak 202a' at a wavelength corresponding to the color blue.
- the curves 104', 106', and 202' also have respective local peaks 104b', 106b', and 202b' near a common wavelength that corresponds non-visible light - in this case infrared light.
- the first set 104, the second set 106, and the third set 202 are not only sensitive to green, red, and blue light, respectively, but also to light within an infrared wavelength band.
- the infrared wavelength band, to which the first set 104, the second set 106, and the third set 202 are sensitive may be relatively narrow (e.g., no more than about 100 nm) and may include 850 nm. Also, the quantum efficiency percentage associated with the local peaks 104b', 106b', and 202b' may be substantially the same or within a narrow percentage range.
- the sensitivity of the sets 104, 106, and 202 to infrared light may be substantially equal or within a narrow sensitivity range (e.g., about five percent in quantum efficiency) so that an average intensity value of light sensed by the first set 104 may be substantially equal to an average intensity value of light sensed by the second set 106 and an average intensity value of light sensed by the third set 202.
- a narrow sensitivity range e.g., about five percent in quantum efficiency
- the color image sensor array 102 need not be limited to three sets of sensor elements or the colors red, green, and blue, and the color image sensor array 102 may include color filter patterns other than the Bayer pattern.
- the color image sensor array 102 may include a cyan, yellow, green, and magenta (CYGM) filter or a red, green, blue, and emerald (RGBE) filter in which the sensor elements of the different colors are also sensitive to light within an infrared wavelength band.
- CYGM cyan, yellow, green, and magenta
- RGBE emerald
- the color filter pattern used on the color filter array 102 may be chosen to achieve accurate color rendition or to improve sensitivity in a color photograph application. While these distinctions are not necessary in the present embodiment, the imaging device 100 and its associated methods are flexible to compensate for the effects of these various filters.
- the imaging device 100 may also include one or more artificial illumination sources 108 (two illumination sources are depicted in Figure 1 ).
- the artificial illumination sources 108 may be mounted to a printed circuit board 110 upon which the color image sensor array 102 is also mounted.
- the artificial illumination sources 108 are operable to emit infrared illumination.
- the infrared illumination emitted by the artificial illumination sources 108 may be narrowband infrared illumination (e.g., illumination having a bandwidth less than about 100 nm).
- the wavelength bandwidth of light emitted by the artificial illumination sources 108 preferably includes 850 nm, when using a color image sensor array with characteristics shown in Figure 3 .
- the imaging device 100 typically includes a suitable optical system 112 positioned to focus light upon the color image sensor array 102.
- the optical system 112 may include conventional optical components, such as one or more lenses, an aperture, and, in some cases, a mechanical shutter.
- the color image sensor array 102 may include electronic shuttering means.
- the optical system 112 may also include one or more optical filters to block out certain wavelengths of light.
- the optical system 112 excludes an infrared filter, which is operable to block out infrared light, and may include one or more optical filters that are operable to block out light that does not have wavelengths within the infrared wavelength band.
- the artificial illumination sources 108 are shown as being mounted on the printed circuit board 110, the artificial illumination sources 108 may be positioned in other convenient locations to provide illumination of the object 114.
- the artificial illumination sources 108 illuminate the field of view 116 with infrared illumination (step 402) or, in an example not as claimed, illumination of another non-visible frequency at which all of the sensor elements have an acceptable response. If an object 114 (e.g., an optical code) is within the field of view 116 of the imaging device, infrared light reflects off the object 114 toward the optical system 112. Infrared light that is incident on the optical system 112 is focused by the optical system 112 onto the sensor elements of the color image sensor array 102 (step 404).
- infrared illumination e.g., an optical code
- Sensor elements of the first set 104, the second set 106, and the third set 202 sense the focused infrared light (steps 406a, 406b, and 406c).
- the color image sensor array 102 produces image data based upon the infrared light that is incident on, and sensed by, the sensor elements of the first, second, and third sets 104, 106, and 202 (step 408).
- An enclosure 118 may cover the color image sensor array 102 except where the optical system 112 is located, so that an appreciable amount of light from sources other than the artificial illumination sources 108 does not reach the color image sensor array 102.
- each of the sets 104, 106, and 202 are sensitive to infrared light, each of the sets 104, 106, and 202 contribute to production of the image data and high-resolution infrared imaging of the object 114 may be achieved.
- Each of the sets 104, 106, and 202 may contribute to the production of the image data to a sufficiently equal extent that no one of the set 104, 106, or 202 contributes to the image data appreciably more than the other two sets.
- the resolution of an infrared image represented by the image data may be substantially equal to a resolution produced by a monochrome image sensor array having the same size of sensor elements and the same number of sensor elements as the sum of the number of sensor elements in the sets 104, 106, and 202. In other words, when illuminated with infrared light, the color image sensor array 102 may achieve a resolution substantially equivalent to a monochrome imager.
Description
- The field of this disclosure relates generally to systems and methods of data reading, and more particularly but not exclusively to reading of optical codes (e.g., bar codes).
- Optical codes encode useful, optically-readable information about the items to which they are attached or otherwise associated. Perhaps the most common example of an optical code is the bar code. Bar codes are ubiquitously found on or associated with objects of various types, such as the packaging of retail, wholesale, and inventory goods; retail product presentation fixtures (e.g., shelves); goods undergoing manufacturing; personal or company assets; and documents. By encoding information, a bar code typically serves as an identifier of an object, whether the identification be to a class of objects (e.g., containers of milk) or a unique item. A typical linear or one-dimensional bar code, such as a UPC code, consist of alternating bars (i.e., relatively dark areas) and spaces (i.e., relatively light areas). The pattern of alternating bars and spaces and the widths of those bars and spaces represent a string of binary ones and zeros, wherein the width of any particular bar or space is an integer multiple of a specified minimum width, which is called a "module" or "unit." Thus, to decode the information, a bar code reader must be able to reliably discern the pattern of bars and spaces, such as by determining the locations of edges demarking adjacent bars and spaces from one another, across the entire length of the bar code.
- Bar codes are just one example of the many types of optical codes in use today. Higher-dimensional optical codes, such as, two-dimensional matrix codes (e.g., MaxiCode) or stacked codes (e.g., PDF 417), which are also sometimes referred to as "bar codes," are also used for various purposes.
- Different methods and types of optical code reading devices are available for capturing an optical code and for decoding the information represented by the optical code. For example, image-based readers are available that include imagers, such as charge coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) imagers, that generate electronic image data that represent an image of a captured optical code. Such a device is known from
WO 2005/072 193 A2 . Image-based readers are used for reading one-dimensional optical codes and higher-dimensional optical codes. Because optical codes most often include dark and light patterns (e.g., black and white) that represent binary data, imagers of image-based readers are typically monochrome so that uniform sensitivity for each pixel of the imager is achieved. Also, typical image-based readers include light sources that illuminate the image-based reader's field of view with narrowband visible light to achieve high optical resolution by avoiding chromatic aberration and polychromatic diffraction effects. Narrowband light sources typically used for imaging include laser diodes, having a bandwidth on the order of 5 nanometers (nm), and light emitting diodes (LEDs), having a bandwidth on the order of 50 nm. - Common imagers made for image capturing devices, such as still cameras and video cameras, however, are color imagers - not monochrome. Because imagers made for many image capturing devices are color, color imagers are generally made in higher volume and have become less expensive than monochrome imagers. Some image-based readers have included color imagers, but the present inventors have recognized that those readers do not effectively achieve high optical resolution comparable to monochrome image-based readers with the same number and size of pixels.
- This disclosure describes improved optical reading devices and associated methods. In accordance with the invention, there is provided the method of claim 1 and the device of claim 8.
- One embodiment is directed to an optical code reading device that includes a color image sensor array positioned to sense light reflected from an object in a field of view of the optical code reading device and to produce from the sensed reflected light image data representing an image of the object. The color image sensor array has a first set of sensor elements that are sensitive to a first visible wavelength band of light, and a second set of sensor elements that are sensitive to a second visible wavelength band of light. The first and second sets of sensor elements are also sensitive to light within an infrared wavelength band. The optical code reading device includes an artificial illumination source positioned to illuminate the field of view of the optical code reading device with light that is incident upon and reflected from the object in the field of view toward the image sensor array. The illumination source is operable to produce infrared light having wavelengths within the infrared wavelength band so that, upon illumination of the field of view, at least some sensor elements of each of the first and second sets are sensitive to the infrared light and contribute to production of the image data.
- Additional aspects and advantages will be apparent from the following detailed description of preferred embodiments, which proceeds with reference to the accompanying drawings.
-
-
Figure 1 is a diagram of an imaging system according to a preferred embodiment. -
Figure 2 is a diagram of a color filter pattern of a color image sensor array of the imaging system ofFigure 1 . -
Figure 3 is a graph of the sensitivity of blue, green, and red sensor elements as a function of light wavelength of an illustrative color image sensor array used in the imaging system ofFigure 1 . -
Figure 4 is a flowchart showing the operational steps of the imaging system ofFigure 1 . -
Figure 5 is a flowchart of an illumination calibration method according to one embodiment. - With reference to the above-listed drawings, this section describes particular embodiments and their detailed construction and operation.
-
Figure 1 is a diagram of animaging device 100, such as an optical code reading device, according to one embodiment. Theimaging device 100 includes a colorimage sensor array 102 contained within ahousing 103 of theimaging device 100. The colorimage sensor array 102 includes a first set ofsensor elements 104 and a second set ofsensor elements 106. The sensor elements of the colorimage sensor array 102 may be arranged in a one-dimensional array or, preferably, a two-dimensional array. The colorimage sensor array 102 may be a CCD, such as a frame-transfer or interline-transfer CCD. The colorimage sensor array 102 may alternatively be a CMOS imager, such as a global shuttered or rolling-reset CMOS imager. Suitable imagers or image sensor arrays are available, for example, from Aptina Imaging Corporation of San Jose, California, USA including, but not limited to, a model MT9V022 VGA color imager. Imagers are available from many manufacturers and are available in various resolutions (numbers of pixels). For higher resolution applications, other imagers from Aptina Imaging Corporation are suitable, including the model MT9M001 with 1.3 megapixels, the model MT9M002 with 1.6 megapixels, and the model MT9P001 with 5 megapixels. As imaging technology advances, imager resolution increases and other imagers may also be suitable. - The color
image sensor array 102 may include more than two sets of sensor elements. For example, the colorimage sensor array 102 may include three sets ofsensor elements Figure 2 . Each set of sensor elements corresponds to a different color. For example, thefirst set 104 may be sensitive to light having wavelengths that correspond to the color green (G) (wavelengths ranging between about 500 nanometers (nm) and about 600 nm), thesecond set 106 may be sensitive to light having wavelengths that correspond to the color red (R) (wavelengths ranging between about 600 nm and about 750 nm), and thethird set 202 may be sensitive to light having wavelengths that correspond to the color blue (B) (wavelengths ranging between about 400 nm and about 500 nm). Moreover, a color filter associated with each sensor element of the different sets appreciably filters out visible light that does not correspond to its color (i.e., the color filters associated with thefirst set 104 appreciably block out red and blue light). -
Figure 3 is a graph depicting an example of the quantum efficiency percentage versus the wavelength of light incident upon red, green and blue sensor elements of the model MT9V022 VGA color imager available from Aptina Imaging Corporation that may be used as the colorimage sensor array 102. A curve 104', corresponding to the spectral sensitivity of the sensor elements of thefirst set 104, has alocal peak 104a' at a wavelength corresponding to the color green. A curve 106', corresponding to the spectral sensitivity of the sensor elements of thesecond set 106, has a local peak 106a' at a wavelength corresponding to the color red. A curve 202', corresponding to the spectral sensitivity of the sensor elements of thethird set 202, has alocal peak 202a' at a wavelength corresponding to the color blue. The curves 104', 106', and 202' also have respectivelocal peaks 104b', 106b', and 202b' near a common wavelength that corresponds non-visible light - in this case infrared light. In other words, thefirst set 104, thesecond set 106, and thethird set 202 are not only sensitive to green, red, and blue light, respectively, but also to light within an infrared wavelength band. The infrared wavelength band, to which thefirst set 104, thesecond set 106, and thethird set 202 are sensitive, may be relatively narrow (e.g., no more than about 100 nm) and may include 850 nm. Also, the quantum efficiency percentage associated with thelocal peaks 104b', 106b', and 202b' may be substantially the same or within a narrow percentage range. In other words, the sensitivity of thesets first set 104 may be substantially equal to an average intensity value of light sensed by thesecond set 106 and an average intensity value of light sensed by thethird set 202. - The color
image sensor array 102 need not be limited to three sets of sensor elements or the colors red, green, and blue, and the colorimage sensor array 102 may include color filter patterns other than the Bayer pattern. For example, the colorimage sensor array 102 may include a cyan, yellow, green, and magenta (CYGM) filter or a red, green, blue, and emerald (RGBE) filter in which the sensor elements of the different colors are also sensitive to light within an infrared wavelength band. The color filter pattern used on thecolor filter array 102 may be chosen to achieve accurate color rendition or to improve sensitivity in a color photograph application. While these distinctions are not necessary in the present embodiment, theimaging device 100 and its associated methods are flexible to compensate for the effects of these various filters. - The
imaging device 100 may also include one or more artificial illumination sources 108 (two illumination sources are depicted inFigure 1 ). Theartificial illumination sources 108 may be mounted to a printedcircuit board 110 upon which the colorimage sensor array 102 is also mounted. In a first embodiment, theartificial illumination sources 108 are operable to emit infrared illumination. The infrared illumination emitted by theartificial illumination sources 108 may be narrowband infrared illumination (e.g., illumination having a bandwidth less than about 100 nm). Also, the wavelength bandwidth of light emitted by theartificial illumination sources 108 preferably includes 850 nm, when using a color image sensor array with characteristics shown inFigure 3 . - The
imaging device 100 typically includes a suitableoptical system 112 positioned to focus light upon the colorimage sensor array 102. Theoptical system 112 may include conventional optical components, such as one or more lenses, an aperture, and, in some cases, a mechanical shutter. As an alternative to a mechanical shutter, the colorimage sensor array 102 may include electronic shuttering means. Theoptical system 112 may also include one or more optical filters to block out certain wavelengths of light. In one example, when infrared illumination sources are selected for theartificial illumination sources 108, theoptical system 112 excludes an infrared filter, which is operable to block out infrared light, and may include one or more optical filters that are operable to block out light that does not have wavelengths within the infrared wavelength band. Although theartificial illumination sources 108 are shown as being mounted on the printedcircuit board 110, theartificial illumination sources 108 may be positioned in other convenient locations to provide illumination of the object 114. - A preferred operation of the
imaging device 100 will now be described with reference to aflowchart 400 ofFigure 4 . Theartificial illumination sources 108 illuminate the field ofview 116 with infrared illumination (step 402) or, in an example not as claimed, illumination of another non-visible frequency at which all of the sensor elements have an acceptable response. If an object 114 (e.g., an optical code) is within the field ofview 116 of the imaging device, infrared light reflects off the object 114 toward theoptical system 112. Infrared light that is incident on theoptical system 112 is focused by theoptical system 112 onto the sensor elements of the color image sensor array 102 (step 404). Sensor elements of thefirst set 104, thesecond set 106, and thethird set 202 sense the focused infrared light (steps image sensor array 102 produces image data based upon the infrared light that is incident on, and sensed by, the sensor elements of the first, second, andthird sets enclosure 118 may cover the colorimage sensor array 102 except where theoptical system 112 is located, so that an appreciable amount of light from sources other than theartificial illumination sources 108 does not reach the colorimage sensor array 102. Because each of thesets sets sets set sets image sensor array 102 may achieve a resolution substantially equivalent to a monochrome imager. - Though the present invention has been set forth in the form of its preferred embodiments, it is nevertheless intended that modifications to the disclosed systems and methods may be made without departing from inventive concepts set forth herein.
Claims (15)
- A method of imaging an object in a field of view of an optical code reading device, the optical code reading device having a color image sensor array, the method comprising:illuminating a field of view of the optical code reading device with artificial infrared light that is incident upon the object, thereby producing reflected infrared light, the reflected infrared light having wavelengths within an infrared wavelength band;sensing the reflected infrared light with a first set of sensor elements of the color image sensor array, the first set of sensor elements being sensitive to the infrared light and to visible light having wavelengths within a first visible wavelength band;sensing the reflected infrared light with a second set of sensor elements of the color image sensor array, the second set of sensor elements being sensitive to the infrared light and to visible light having wavelengths within a second visible wavelength band different from the first visible wavelength band; andproducing image data from the first and second sets of sensor elements, the image data being derived from the infrared light sensed by the first and second sets of sensor elements and representing a high-resolution infrared image of the object.
- A method according to claim 1, further comprising:sensing the reflected infrared light with a third set of sensor elements of the color image sensor array, the third set of sensor elements being sensitive to the infrared light and to visible light having wavelengths within a third visible wavelength band different from each of the first and second visible wavelength bands; andproducing the image data from the first, second and third sets of sensor elements, the image data being derived from the infrared light sensed by the first, second and third sets of sensor elements.
- A method according to claim 2, wherein the resolution of the infrared image represented by the image data is substantially equal to a resolution produced by a monochrome image sensor array having the same number of sensor elements as the sum of the numbers of sensor elements in the first, second and third sets of sensor elements.
- A method according to claim 2 or 3, wherein the first, second and third visible wavelength bands correspond to the colors red, green and blue.
- A method as in one of claims 1-4, wherein the infrared wavelength band includes 850 nm.
- A method as in one of claims 1-5, wherein upon illumination of the field of view, sensor elements of each of the first and second sets of sensor elements are sensitive to the infrared light to a sufficiently equal extent such that an average intensity value of light sensed by the first set of sensor elements is substantially equal to an average intensity value of light sensed by the second set of sensor elements.
- A method as in one of claims 1-6, further comprising:intermittently blocking the reflected infrared light from reaching the sensor elements of the first and second sets by a mechanical shutter.
- An optical code reading device (100) comprising:a color image sensor array (102) positioned to sense light reflected from an object in a field of view of the optical code reading device and to produce from the sensed reflected light image data representing an infrared image of the object, the color image sensor array comprising:a first set of sensor elements (104) that are sensitive to a first visible wavelength band of light, anda second set of sensor elements (106) that are sensitive to a second visible wavelength band of light, different from the first visible wavelength band, wherein the first and second sets of sensor elements are sensitive to light within an infrared wavelength band; andan artificial illumination source (108) positioned to illuminate the field of view of the optical code reading device with light that is incident upon and reflected from the object in the field of view toward the color image sensor array, the artificial illumination source being operable to produce infrared light having a wavelength within the infrared wavelength band so that, upon illumination of the field of view, at least some sensor elements of each of the first and second sets are sensitive to the infrared light and are adapted to contribute to production of the image data.
- An optical code reading device according to claim 8, wherein the color image sensor array includes a third set of sensor elements that are sensitive to a third visible wavelength band different from each of the first and second visible wavelength bands of light and to light within the infrared wavelength bands
- An optical code reading device according to claim 9, wherein the first, second and third visible wavelength bands correspond to the colors red, green and blue.
- An optical code reading device as in one of claims 8-10, wherein the infrared wavelength band includes 850 nm.
- An optical code reading device as in one of claims 8-11, wherein upon illumination of the field of view, sensor elements of each of the first and second sets of sensor elements are sensitive to the infrared light to a sufficiently equal extent such that an average intensity value of light sensed by the first set of sensor elements is substantially equal to an average intensity value of light sensed by the second set of sensor elements.
- An optical code reading device as in one of claims 8-12, wherein the color image sensor is part of a rolling-reset CMOS imager.
- An optical code reading device as in one of claims 8-13, further comprising:a mechanical shutter positioned between the color image sensor array and the field of view.
- An optical code reading device according to claim 9, wherein:the first set of sensor elements are characterized by a first spectral sensitivity that varies as a first function of wavelength, the first function having a local peak at a first peak sensitivity wavelength and also having a local peak near an infrared wavelength;the second set of sensor elements are characterized by a second spectral sensitivity that varies as a second function of wavelength, the second function having a local peak at a second peak sensitivity wavelength and also having a local peak near the infrared wavelength;the third set of sensor elements are characterized by a third spectral sensitivity that varies as a third function of wavelength, the third function having a local peak at a third peak sensitivity wavelength and also having a local peak near the infrared wavelength;the first, second and third peak sensitivity wavelengths are different wavelengths; andthe artificial illumination source is operable to produce radiation including light having the infrared wavelength.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15176809P | 2009-02-11 | 2009-02-11 | |
PCT/US2010/023918 WO2010093806A2 (en) | 2009-02-11 | 2010-02-11 | High-resolution optical code imaging using a color imager |
Publications (3)
Publication Number | Publication Date |
---|---|
EP2396744A2 EP2396744A2 (en) | 2011-12-21 |
EP2396744A4 EP2396744A4 (en) | 2013-12-25 |
EP2396744B1 true EP2396744B1 (en) | 2016-06-01 |
Family
ID=42539596
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP10741731.3A Active EP2396744B1 (en) | 2009-02-11 | 2010-02-11 | High-resolution optical code imaging using a color imager |
Country Status (4)
Country | Link |
---|---|
US (1) | US8118226B2 (en) |
EP (1) | EP2396744B1 (en) |
CN (1) | CN102317951B (en) |
WO (1) | WO2010093806A2 (en) |
Families Citing this family (52)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7442629B2 (en) | 2004-09-24 | 2008-10-28 | President & Fellows Of Harvard College | Femtosecond laser-induced formation of submicrometer spikes on a semiconductor substrate |
US7057256B2 (en) | 2001-05-25 | 2006-06-06 | President & Fellows Of Harvard College | Silicon-based visible and near-infrared optoelectric devices |
US8800874B2 (en) | 2009-02-20 | 2014-08-12 | Datalogic ADC, Inc. | Systems and methods of optical code reading using a color imager |
US8886206B2 (en) | 2009-05-01 | 2014-11-11 | Digimarc Corporation | Methods and systems for content processing |
US9749607B2 (en) | 2009-07-16 | 2017-08-29 | Digimarc Corporation | Coordinated illumination and image signal capture for enhanced signal detection |
US9673243B2 (en) | 2009-09-17 | 2017-06-06 | Sionyx, Llc | Photosensitive imaging devices and associated methods |
US9911781B2 (en) | 2009-09-17 | 2018-03-06 | Sionyx, Llc | Photosensitive imaging devices and associated methods |
TWI438699B (en) * | 2009-10-09 | 2014-05-21 | Primax Electronics Ltd | Processing method of barcode and related apparatus thereof |
US8692198B2 (en) | 2010-04-21 | 2014-04-08 | Sionyx, Inc. | Photosensitive imaging devices and associated methods |
US20120146172A1 (en) | 2010-06-18 | 2012-06-14 | Sionyx, Inc. | High Speed Photosensitive Devices and Associated Methods |
SE535853C2 (en) | 2010-07-08 | 2013-01-15 | Itab Scanflow Ab | checkout counter |
TW201216168A (en) * | 2010-10-08 | 2012-04-16 | Primax Electronics Ltd | Barcode evaluation method and barcode evaluation apparatus |
WO2012085863A1 (en) * | 2010-12-21 | 2012-06-28 | Zamir Recognition Systems Ltd. | A visible light and ir hybrid digital camera |
US9007689B2 (en) * | 2011-04-04 | 2015-04-14 | General Electric Company | Method and apparatus for forming multiple images |
US9496308B2 (en) | 2011-06-09 | 2016-11-15 | Sionyx, Llc | Process module for increasing the response of backside illuminated photosensitive imagers and associated methods |
EP2732402A2 (en) * | 2011-07-13 | 2014-05-21 | Sionyx, Inc. | Biometric imaging devices and associated methods |
US8556179B2 (en) * | 2011-08-16 | 2013-10-15 | Intermec Ip Corp. | Motion tolerant bar code reader |
US8523074B2 (en) * | 2011-08-26 | 2013-09-03 | Honeywell International Inc. | Bar code imagers |
US9064764B2 (en) | 2012-03-22 | 2015-06-23 | Sionyx, Inc. | Pixel isolation elements, devices, and associated methods |
JP5702894B2 (en) * | 2012-07-06 | 2015-04-15 | 富士フイルム株式会社 | Color imaging device and imaging apparatus |
CA2841464C (en) * | 2013-02-01 | 2016-10-25 | Centre De Recherche Industrielle Du Quebec | Apparatus and method for scanning a surface of an article |
WO2014127376A2 (en) | 2013-02-15 | 2014-08-21 | Sionyx, Inc. | High dynamic range cmos image sensor having anti-blooming properties and associated methods |
KR20140112874A (en) * | 2013-03-14 | 2014-09-24 | 삼성전자주식회사 | Electronic device with camera module |
US9939251B2 (en) | 2013-03-15 | 2018-04-10 | Sionyx, Llc | Three dimensional imaging utilizing stacked imager devices and associated methods |
WO2014209421A1 (en) | 2013-06-29 | 2014-12-31 | Sionyx, Inc. | Shallow trench textured regions and associated methods |
CN106575370A (en) * | 2014-08-19 | 2017-04-19 | 英派尔科技开发有限公司 | Machine recognizable pattern generation |
FR3026223B1 (en) * | 2014-09-22 | 2016-12-23 | Photonis France | APPARATUS FOR ACQUIRING PHOTOCATHODE BIMODE IMAGES. |
US9727941B1 (en) | 2014-11-19 | 2017-08-08 | Digimarc Corporation | Optimizing optical scanners for digital watermark detection |
EP3166048A1 (en) * | 2015-11-06 | 2017-05-10 | ams AG | Optical reader device, tag for use on a disposable or replaceable component, optical data validation system and method for optical data validation |
US9811705B1 (en) | 2016-05-10 | 2017-11-07 | Datalogic Ip Tech S.R.L. | Illumination system with active element for generating different illumination patterns for a data reader |
US10776659B2 (en) * | 2016-08-26 | 2020-09-15 | Goodrich Corporation | Systems and methods for compressing data |
WO2018048001A1 (en) * | 2016-09-12 | 2018-03-15 | 한국생산기술연구원 | Smart tag recognition system using infrared-sensitive code and recognition method using same |
US10768497B2 (en) * | 2016-10-03 | 2020-09-08 | Xerox Corporation | Hyperspectral imaging system |
US11473969B2 (en) | 2018-08-09 | 2022-10-18 | Ouster, Inc. | Channel-specific micro-optics for optical arrays |
US10739189B2 (en) * | 2018-08-09 | 2020-08-11 | Ouster, Inc. | Multispectral ranging/imaging sensor arrays and systems |
EP3881235B1 (en) * | 2018-11-12 | 2023-06-07 | Hewlett-Packard Development Company, L.P. | Multiple-pattern fiducial for heterogeneous imaging sensor systems |
US10496911B1 (en) | 2019-03-18 | 2019-12-03 | Capital One Services, Llc | Detection of images in relation to targets based on colorspace transformation techniques and utilizing ultraviolet and infrared light |
US10534948B1 (en) | 2019-03-18 | 2020-01-14 | Capital One Services, Llc | Optimizing detection of images in relation to targets based on colorspace transformation techniques |
US10509991B1 (en) | 2019-03-18 | 2019-12-17 | Capital One Services, Llc | Detection of images in relation to targets based on colorspace transformation techniques and utilizing infrared light |
US10496862B1 (en) | 2019-03-18 | 2019-12-03 | Capital One Services, Llc | Detection of images in relation to targets based on colorspace transformation techniques and utilizing ultraviolet light |
US10523420B1 (en) | 2019-04-18 | 2019-12-31 | Capital One Services, Llc | Transmitting encoded data along transmission mediums based on colorspace schemes |
US10504013B1 (en) | 2019-04-24 | 2019-12-10 | Capital One Services, Llc | Colorspace encoding multimedia data on a physical page |
US10529300B1 (en) | 2019-06-20 | 2020-01-07 | Capital One Services, Llc | Adaptive image display based on colorspace conversions |
US11062104B2 (en) * | 2019-07-08 | 2021-07-13 | Zebra Technologies Corporation | Object recognition system with invisible or nearly invisible lighting |
US10614635B1 (en) | 2019-07-25 | 2020-04-07 | Capital One Services, Llc | Augmented reality system with color-based fiducial marker |
CN110632735A (en) * | 2019-08-16 | 2019-12-31 | 俞庆平 | Method for searching optimal focal plane in laser direct imaging system |
US10833852B1 (en) | 2019-10-03 | 2020-11-10 | Capital One Services, Llc | Encoded data along tape based on colorspace schemes |
US10715183B1 (en) * | 2019-10-25 | 2020-07-14 | Capital One Services, Llc | Data encoding with error-correcting code pursuant to colorspace schemes |
US10867226B1 (en) | 2019-11-04 | 2020-12-15 | Capital One Services, Llc | Programmable logic array and colorspace conversions |
US10762371B1 (en) | 2019-11-14 | 2020-09-01 | Capital One Services, Llc | Object detection techniques using colorspace conversions |
US10878600B1 (en) | 2019-12-10 | 2020-12-29 | Capital One Services, Llc | Augmented reality system with color-based fiducial marker utilizing local adaptive technology |
US11302036B2 (en) | 2020-08-19 | 2022-04-12 | Capital One Services, Llc | Color conversion between color spaces using reduced dimension embeddings |
Family Cites Families (34)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3971065A (en) | 1975-03-05 | 1976-07-20 | Eastman Kodak Company | Color imaging array |
US5576529A (en) | 1986-08-08 | 1996-11-19 | Norand Technology Corporation | Hand-held optically readable information set reader focus with operation over a range of distances |
JP3054151B2 (en) * | 1988-07-27 | 2000-06-19 | キヤノン株式会社 | Color image forming apparatus and color image processing method |
US5786582A (en) * | 1992-02-27 | 1998-07-28 | Symbol Technologies, Inc. | Optical scanner for reading and decoding one- and two-dimensional symbologies at variable depths of field |
JPH06309486A (en) | 1993-04-22 | 1994-11-04 | Tokyo Electric Co Ltd | Bar code reader |
US5714745A (en) | 1995-12-20 | 1998-02-03 | Metanetics Corporation | Portable data collection device with color imaging assembly |
JP3662769B2 (en) | 1999-05-12 | 2005-06-22 | 株式会社東研 | Code reading apparatus and method for color image |
US6628330B1 (en) | 1999-09-01 | 2003-09-30 | Neomagic Corp. | Color interpolator and horizontal/vertical edge enhancer using two line buffer and alternating even/odd filters for digital camera |
US6642962B1 (en) | 1999-09-01 | 2003-11-04 | Neomagic Corp. | Merged pipeline for color interpolation and edge enhancement of digital images |
US6765703B1 (en) | 2000-09-27 | 2004-07-20 | Kabushiki Kaisha Toshiba | Method and apparatus for sensing image |
US7607581B2 (en) | 2003-11-13 | 2009-10-27 | Metrologic Instruments, Inc. | Digital imaging-based code symbol reading system permitting modification of system features and functionalities |
US7594609B2 (en) * | 2003-11-13 | 2009-09-29 | Metrologic Instruments, Inc. | Automatic digital video image capture and processing system supporting image-processing based code symbol reading during a pass-through mode of system operation at a retail point of sale (POS) station |
US6832729B1 (en) | 2001-03-23 | 2004-12-21 | Zih Corp. | Portable data collection device for reading fluorescent indicia |
US7331523B2 (en) | 2001-07-13 | 2008-02-19 | Hand Held Products, Inc. | Adaptive optical image reader |
US6722569B2 (en) | 2001-07-13 | 2004-04-20 | Welch Allyn Data Collection, Inc. | Optical reader having a color imager |
US7071978B2 (en) | 2001-07-18 | 2006-07-04 | Hewlett-Packard Development Company, L.P. | Image mosaic data reconstruction |
US7379613B2 (en) | 2002-02-27 | 2008-05-27 | Omnivision Cdm Optics, Inc. | Optimized image processing for wavefront coded imaging systems |
EP1485861A4 (en) | 2002-03-20 | 2005-10-26 | Symbol Technologies Inc | Image capture system and method using a common imaging array |
US6976629B2 (en) | 2002-03-20 | 2005-12-20 | Symbol Technologies, Inc. | Image capture system and method |
JP2003309707A (en) * | 2002-04-16 | 2003-10-31 | Canon Inc | Image reader |
US6871545B2 (en) * | 2003-06-20 | 2005-03-29 | Weyerhaeuser Company | System and method for measuring stiffness in standing trees |
US7028901B2 (en) * | 2003-07-17 | 2006-04-18 | Symbol Technologies, Inc. | System and method for reading and decoding optical codes using multiple color illumination |
US7014114B2 (en) | 2003-10-02 | 2006-03-21 | Symbol Technologies, Inc. | Image capture device for and method of electro-optically reading indicia at low ambient light levels |
US7036735B2 (en) * | 2004-01-21 | 2006-05-02 | The Code Corporation | Graphical code reader having illumination LEDs of different wavelengths |
US7163149B2 (en) | 2004-03-02 | 2007-01-16 | Symbol Technologies, Inc. | System and method for illuminating and reading optical codes imprinted or displayed on reflective surfaces |
US20060060653A1 (en) | 2004-09-23 | 2006-03-23 | Carl Wittenberg | Scanner system and method for simultaneously acquiring data images from multiple object planes |
US7224540B2 (en) | 2005-01-31 | 2007-05-29 | Datalogic Scanning, Inc. | Extended depth of field imaging system using chromatic aberration |
US7780089B2 (en) | 2005-06-03 | 2010-08-24 | Hand Held Products, Inc. | Digital picture taking optical reader having hybrid monochrome and color image sensor array |
US7770799B2 (en) * | 2005-06-03 | 2010-08-10 | Hand Held Products, Inc. | Optical reader having reduced specular reflection read failures |
US7387252B2 (en) | 2006-08-15 | 2008-06-17 | Hand Held Products, Inc. | Optical reader with improved lens focusing system |
US8091788B2 (en) | 2007-01-11 | 2012-01-10 | Datalogic Scanning, Inc. | Methods and systems for optical code reading using virtual scan lines |
KR100874159B1 (en) * | 2007-03-28 | 2008-12-15 | 주식회사 아이센스 | Electrochemical Biosensors and Measuring Instruments |
US20090159685A1 (en) * | 2007-12-20 | 2009-06-25 | Symbol Technologies, Inc. | Optimizing Optical Quality of a Sensor in a Bar Code Reader |
US8800874B2 (en) | 2009-02-20 | 2014-08-12 | Datalogic ADC, Inc. | Systems and methods of optical code reading using a color imager |
-
2010
- 2010-02-11 CN CN201080007498.4A patent/CN102317951B/en active Active
- 2010-02-11 WO PCT/US2010/023918 patent/WO2010093806A2/en active Application Filing
- 2010-02-11 US US12/704,313 patent/US8118226B2/en active Active
- 2010-02-11 EP EP10741731.3A patent/EP2396744B1/en active Active
Also Published As
Publication number | Publication date |
---|---|
EP2396744A2 (en) | 2011-12-21 |
WO2010093806A3 (en) | 2010-12-02 |
US8118226B2 (en) | 2012-02-21 |
EP2396744A4 (en) | 2013-12-25 |
CN102317951B (en) | 2015-01-21 |
WO2010093806A2 (en) | 2010-08-19 |
CN102317951A (en) | 2012-01-11 |
US20100200658A1 (en) | 2010-08-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2396744B1 (en) | High-resolution optical code imaging using a color imager | |
US8998092B2 (en) | Systems and methods of optical code reading using a color imager | |
US9489557B2 (en) | Decodable indicia reading terminal with optical filter | |
EP2541464B1 (en) | Optical filter for image and barcode scanning | |
US8505823B2 (en) | Noise removal from color barcode images | |
US20030024986A1 (en) | Molded imager optical package and miniaturized linear sensor-based code reading engines | |
US20120312877A1 (en) | System and method for providing sufficient illumination quality for barcodes captured with a color image sensor | |
WO2013067671A1 (en) | Optical indicia reading terminal with color image sensor | |
US7729061B2 (en) | Imaging system having anamorphic magnification | |
US9948868B2 (en) | Multi-point spectral system and multi-point spectral module to be mounted on, clipped on or connected to an image capturing module of an electronic device easily | |
US7014114B2 (en) | Image capture device for and method of electro-optically reading indicia at low ambient light levels | |
US20100123009A1 (en) | High-resolution interpolation for color-imager-based optical code readers | |
CN107248242B (en) | Watermark enabling scanning | |
CN116508316A (en) | Machine vision system and method with multi-aperture optical assembly | |
US20240031511A1 (en) | Digital Watermark Enabled Scanner with White Illumination Source |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20110909 |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
DAX | Request for extension of the european patent (deleted) | ||
A4 | Supplementary search report drawn up and despatched |
Effective date: 20131122 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06K 7/12 20060101AFI20131118BHEP Ipc: G06K 7/10 20060101ALI20131118BHEP |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20151111 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: DATALOGIC ADC, INC. |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP Ref country code: AT Ref legal event code: REF Ref document number: 804269 Country of ref document: AT Kind code of ref document: T Effective date: 20160615 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602010033781 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20160601 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160901 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 804269 Country of ref document: AT Kind code of ref document: T Effective date: 20160601 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160902 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161001 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161003 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: BE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602010033781 Country of ref document: DE |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed |
Effective date: 20170302 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20170211 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170228 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170228 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: ST Effective date: 20171031 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170211 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170211 Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170211 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170211 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20100211 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160601 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160601 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20230216 Year of fee payment: 14 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230528 |