EP3232334B1 - Information processing device, information processing method, information processing program, and recording medium - Google Patents

Information processing device, information processing method, information processing program, and recording medium Download PDF

Info

Publication number
EP3232334B1
EP3232334B1 EP17159552.3A EP17159552A EP3232334B1 EP 3232334 B1 EP3232334 B1 EP 3232334B1 EP 17159552 A EP17159552 A EP 17159552A EP 3232334 B1 EP3232334 B1 EP 3232334B1
Authority
EP
European Patent Office
Prior art keywords
circuit
arithmetic
preprocessing
image processing
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP17159552.3A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3232334A1 (en
Inventor
Toshinori TAMAI
Taku Oya
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Omron Corp
Original Assignee
Omron Corp
Omron Tateisi Electronics Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Omron Corp, Omron Tateisi Electronics Co filed Critical Omron Corp
Publication of EP3232334A1 publication Critical patent/EP3232334A1/en
Application granted granted Critical
Publication of EP3232334B1 publication Critical patent/EP3232334B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F15/00—Digital computers in general; Data processing equipment in general
    • G06F15/76—Architectures of general purpose stored program computers
    • G06F15/78—Architectures of general purpose stored program computers comprising a single central processing unit
    • G06F15/7867—Architectures of general purpose stored program computers comprising a single central processing unit with reconfigurable architecture
    • G06F15/7871—Reconfiguration support, e.g. configuration loading, configuration switching, or hardware OS
    • G06F15/7878—Reconfiguration support, e.g. configuration loading, configuration switching, or hardware OS for pipeline reconfiguration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/00962—Input arrangements for operating instructions or parameters, e.g. updating internal software
    • H04N1/00965—Input arrangements for operating instructions or parameters, e.g. updating internal software using a plug-in memory module, e.g. memory card, memory stick
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/23—Reproducing arrangements
    • H04N1/2307—Circuits or arrangements for the control thereof, e.g. using a programmed control device, according to a measured quantity
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/0083—Arrangements for transferring signals between different components of the apparatus, e.g. arrangements of signal lines or cables
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/00962—Input arrangements for operating instructions or parameters, e.g. updating internal software
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2201/00—Indexing scheme relating to scanning, transmission or reproduction of documents or the like, and to details thereof
    • H04N2201/0077—Types of the still picture apparatus
    • H04N2201/0094—Multifunctional device, i.e. a device capable of all of reading, reproducing, copying, facsimile transception, file transception

Definitions

  • the present invention relates generally to an image processing device and the like, and relates more specifically to an image processing device provided with a programmable logic circuit that can be partially reconfigured.
  • Japanese Patent Application Publication No. 2010-206704 describes an image processing device configured to store the register settings for the next print job in flash memory when processing the current print job; the next print job begins when the register settings stored in the flash memory is written to a programmable circuit. This image processing device is able to reduce the amount of time between print jobs.
  • the image processing device disclosed in Japanese Patent Application Publication No. 2015-228187 detects in advance the type of object to be acquired, reconfigures a programmable circuit according to the document type, and uses the reconfigured circuit to process the image data acquired by a camera. The image processing device is thus able to reduce the wait before processing starts.
  • Further prior art documents are US 2011/225415 A1 , US 2007/245131 A1 , US 2015/256687 A1 and EP 2680560A2 as well as " DELAHAYE J-P ET AL: Software radio and dynamic reconfiguration on a DSP/FPGA platform", FREQUENZ, SCHIELE UND SCHON, BERLIN, DE, vol. 58, no. 5-6, 1 May 2004 (2004-05-01), pages 152-159, XP002530814, ISSN: 0016-1136 ".
  • the continuous flow of inspection objects requires that images are processed quickly in order to use an image processing device to inspect and measure an object.
  • the time reconfiguration takes depends on the size of the circuit. Therefore it tends to be difficult to further reduce the reconfiguration time in order to reduce the time between print jobs, or to reduce the wait time until processing starts as described in Japanese Patent Application Publication No. 2010-206704 and Japanese Patent Application Publication No. 2015-228187 .
  • Embodiments of the present invention aim to address these problems by providing an image processing device, and the like capable of further reducing the amount of time partial reconfiguration takes.
  • An image processing device according to an aspect of the present invention is defined in claim 1.
  • a timing control circuit is provided between each of the plurality of arithmetic converter circuits, therefore no problems occur in the preprocessing circuit regardless of how the arithmetic converter circuitry is reconfigured.
  • the configuration is also such that partial reconfiguration is performed for at least only one arithmetic converter circuit in the preprocessing circuit while not being performed for the other arithmetic converter circuits or timing control circuits. Consequently, the circuitry for partial reconfiguration is a smaller size. Thus, partial reconfiguration takes less time.
  • an image processing device capable of further reducing the amount of time a programmable circuit, such as an FPGA, takes to perform a partial reconfiguration.
  • Modifying image processing parameters may also take even less time to complete, when the parameters for image processing have been modified.
  • the image processing device provided is capable of quickly performing image processing that matches the modifications made without pausing operations, even when the type of object or scene to be processed changes.
  • an image processing device according to another aspect of the present invention is defined in claim 2.
  • partial reconfiguration is performed for at least one arithmetic module in the preprocessing circuit while not being performed for the other circuits. Consequently, the circuitry for partial reconfiguration is a smaller size. Thus, partial reconfiguration takes less time.
  • an image processing device capable of further reducing the amount of time a programmable circuit, such as an FPGA, takes to perform a partial reconfiguration.
  • the image processing device may be configured such that the preprocessing circuit includes a plurality of the same type of arithmetic converter circuit; and the plurality of kinds of arithmetic converter circuits are connected in series.
  • the above described configuration is capable of executing processes continuously without storing a processing result in the external memory when the same type of arithmetic converter circuit repeatedly executes a process as a preprocess. It is thus possible to reduce the preprocessing execution time compared to existing techniques.
  • the aforementioned arithmetic converter circuit may be configured to perform a filtering process on image data.
  • filtering is a simple computational process, given the larger amount of data handled for image processing, filtering requires very fast computations.
  • implementing the preprocessing circuit as a programmable circuit such as an FPGA achieves faster processing compared to execution on a CPU, for instance.
  • the image processing device is configured according to claim 5.
  • the plurality of types of partial reconfiguration data stored in the non-volatile storage unit is retained even through the device is turned off.
  • the partial reconfiguration data may also be forwarded to the volatile storage unit whereby, reading the partial reconfiguration data from the volatile storage unit allows the circuit configuration controller to quickly perform partial reconfiguration.
  • the data may also be transferred via a memory transfer controller; therefore, the transfer of data from the non-volatile storage unit to the volatile storage unit does not consume CPU resources. This reduces the load on the CPU.
  • the configuration is also such that the circuit configuration control step partially reconfigures at least only one arithmetic converter circuit in the preprocessing circuit while not partially reconfiguring the other arithmetic converter circuits or timing control circuits. Consequently, the circuitry for partial reconfiguration is a smaller size. Thus, partial reconfiguration takes less time.
  • an image processing method can be provided that is capable of further reducing the amount of time a programmable circuit such as an FPGA takes to perform a partial reconfiguration.
  • an image processing method configured to process image data obtained externally, includes: a preprocessing step carried out by a preprocessing circuit configured to carry out preprocessing during image processing; and a circuit configuration controller configured to carry out partial reconfiguration of the preprocessing circuit; preprocessing step includes at least one arithmetic conversion step which includes performing arithmetic computations and conversions on image data, with the arithmetic conversion step configured to include an input step temporarily storing data input thereto, an arithmetic conversion step carrying out arithmetic computation on said temporarily stored data, and an output step temporarily storing the computed data; and the circuit configuration control step carries out partial reconfiguration of at least one arithmetic module in the preprocessing circuit configured to perform the arithmetic computation step, while not carrying out partial reconfiguration on an input module and an output module in the preprocessing circuit configured to perform the input step and the output step respectively.
  • the circuit configuration control step partially reconfigures at least one arithmetic module in the preprocessing circuit while not partially reconfiguring the other circuits. Consequently, the circuitry for partial reconfiguration is a smaller size. Thus, partial reconfiguration takes less time.
  • an image processing method can be provided that is capable of further reducing the amount of time a programmable circuit such as an FPGA takes to perform a partial reconfiguration.
  • the preprocessing step may include a plurality of the same type of arithmetic conversion steps; and the plurality of kinds of arithmetic conversion steps may be executed in series.
  • the above described configuration is capable of executing processes continuously without storing a processing result in external memory when the same type of arithmetic conversion step is repeatedly executed as a preprocessing step. It is thus possible to reduce the preprocessing execution time compared to existing techniques. Effects
  • Embodiments of the invention provide an image processing device, and the like capable of further reducing the amount of time it takes for partial reconfiguration.
  • FIG. 2 An overview of the image recognition system 1 that contains the image processing device 100 is described using FIG. 2 .
  • FIG. 2 diagrams an image recognition system 1 that includes the image processing device according to the embodiment. As illustrated in FIG. 2 , the image recognition system 1 includes a camera 2 and the image processing device 100.
  • the camera 2 captures an image of an inspection object, and the image recognition system 1 uses the image processing device 100 to process the image, whereby the image recognition system 1 identifies the position of, compares, evaluates, or classifies the inspection object.
  • the digital camera 2 converts the image information from analog to digital, and is provided with a Complementary Metal Oxide Semiconductor (CMOS) sensor that serves as the photovoltaic imaging element.
  • CMOS Complementary Metal Oxide Semiconductor
  • the imaging element may also be a charge-coupled device (CCD) sensor or any other type of sensor.
  • the image processing device 100 includes a CPU 110, ROM 120, RAM 130, DDR 140, a user setting interface 150 (also referred to below as user interface; user I/F), a network interface (I/F) 160, and an FPGA, which are connected to a system bus 101.
  • a user setting interface 150 also referred to below as user interface; user I/F
  • a network interface (I/F) 160 and an FPGA, which are connected to a system bus 101.
  • the CPU 110 is the central computer that controls all the operations of the image processing device 100, which primarily involve image processing.
  • the ROM 120 is non-volatile memory, and stores a booting program that starts up the CPU 110.
  • the ROM 120 also stores data for configuring the later-described FPGA 200.
  • the RAM 130 is volatile memory that serves as a work area for the CPU 110.
  • the DDR 140 is also volatile memory that serves as a work area when reconfiguring the FPGA 200.
  • the DDR 140 is faster at reading out data than the ROM 120. Therefore, the circuit reconfiguration process may be executed more quickly by transferring a portion of the data stored in the ROM 120 to the DDR 140 when reconfiguring the FPGA 200.
  • the user interface 150 allows a user to enter operational directives to the image processing device 100, and presents the user with information.
  • the user interface 150 controls the input and output to and from, for example, an image display device such as a display or a touchscreen panel, and input devices such as a keyboard, mouse, or button.
  • the network interface 160 transmits the various kinds of data resulting from image processing by the image processing device 100 to an external server device or storage device (not shown) via a communication line such as LAN or the Internet. EtherCAT (Registered Trademark) may be adopted as the communication line, for example.
  • the network interface 160 also receives various kinds of data from outside the image processing device 100 via the communication line.
  • the FPGA 200 is one kind of programmable circuit that functions as a preprocessing circuit performing preprocessing on an image captured by the camera 2.
  • the FPGA 200 (later described) can be partially reconfigured.
  • the image processing device 100 includes the FPGA 200, the image processing device 100 may include other programmable circuits that are preprocessing circuits besides the FPGA 200.
  • Each of the aforementioned components is mutually connected via the system bus 101, to allow for high-speed data exchange.
  • FIG. 1 A more detailed description of the image processing device 100 of the embodiment outlined above is provided with reference to FIG. 1 , FIG. 3 , and FIG. 4 .
  • the Image Processing Device The Image Processing Device
  • FIG. 1 is a block diagram representing a general configuration of the image processing device 100 according to the first embodiment.
  • the image processing device 100 is equipped with a storage unit 300, a controller 400, and an FPGA 200.
  • the storage unit 300 stores CPU boot data 310 used to start up the CPU 110, startup configuration data 320 used to initially configure the FPGA 200, and FPGA reconfiguration data 330 used for partial reconfiguration of the FPGA 200.
  • the storage unit 300 corresponds to ROM 120 ( FIG. 2 ).
  • the FPGA reconfiguration data 330 includes a plurality of types of circuit reconfiguration data and is used in partial reconfiguration of a reconfigurable module in the FPGA 200 (later described); the plurality of types of circuit reconfiguration data includes, first circuit reconfig data 331, second circuit reconfig data 332, third circuit reconfig data 333, and the like, for example.
  • the controller 400 is provided with a user interface 150 ( FIG. 2 ), a configuration controller 410, and an image processing unit 420.
  • the configuration controller 410 and the image processing unit 420 may be implemented in software and run by the CPU 110 ( FIG. 2 ).
  • the configuration controller 410 uses the aforementioned startup configuration data 320 to initially configure the FPGA 200. This is because data in the FPGA disappears when the image processing device 100 is turned off. A preprocessing circuit in the FPGA 200 may be initialized from this startup configuration. Configuration of the FPGA on startup may be implemented using another technique, and is not particularly limited to using this startup configuration method.
  • the aforementioned preprocessing circuit executes preprocessing within the process of the image processing device 100 processing images.
  • the FPGA 200 is described from the view of an FPGA 200 initially configured as a preprocessing circuit.
  • the description also assumes that the configuration controller 410 carries out partial reconfiguration with the circuit reconfiguration data selected from the FPGA reconfiguration data 330 stored in the storage unit 300.
  • the image processing controller 420 postprocesses the image data preprocessed by the FPGA; i.e., the image processing controller 420 identifies the position of, compares, evaluates, or classifies the inspection object, for example.
  • the image processing controller 420 primarily controls image processing in the image processing device 100, outputs instructions to the configuration controller 410 for partial reconfiguration of the FPGA 200, or outputs instructions for the FPGA 200 to re-execute preprocessing, or the like.
  • the controller 400 is preferably equipped with a direct memory access controller (DMAC; not shown) to serve as a memory transfer controller.
  • DMAC direct memory access controller
  • the DMAC transfers the FPGA reconfiguration data 330 from the storage unit 300 to the DDR 140 during reconfiguration of the FPGA 200. Additionally, the DMAC transfers data on the basis of directives from the CPU 110, thereby reducing the load on the CPU 110.
  • the configuration controller 410 can significantly increase its data transfer rate (e.g., by as much as four times) by reading the FPGA reconfiguration data 330 from the DDR 140.
  • the programmable circuit i.e., the FPGA 200 configured initially as a preprocessing circuit, is described below in detail.
  • the FPGA 200 which is the programmable circuit that makes up the preprocessing circuit, includes an image acquisition unit 210, a data controller 220, a preprocessor module 230, a configuration interface 240, and an interrupt controller 250.
  • the image acquisition unit 210 acquires the image captured by the camera 2 and forwards the image to the data controller 220.
  • the data controller 220 controls the later-described preprocessor module 230 while transferring the data forwarded from the image acquisition unit 210.
  • the data controller 220 may also control the later-described preprocessor module 230 while transferring the data forwarded from the image processing controller 420.
  • the preprocessor module 230 preprocesses the image data transferred from the data controller 220.
  • the preprocessing performed in the preprocessor module 230 is not fixed; the configuration controller 410 may use the plurality of types of circuit reconfiguration data stored in the storage unit 300 to partially reconfigure the FPGA 200 so that the preprocessor module 230 executes the desired preprocessing.
  • the configuration interface 240 receives instructions from the configuration controller 410 and partially reconfigures the preprocessor module 230.
  • the interrupt controller 250 receives information from the configuration interface 240 on whether or not partial reconfiguration was successful. Based on this information the interrupt controller 250 generates an interrupt in the CPU 110 which is serving as the controller 400.
  • the preprocessor module 230 of the embodiment is described below in detail.
  • the preprocessor module 230 in the embodiment contains three shared modules, and three reconfigurable modules.
  • the image data transferred from the data controller 220 is processed in order by the shared module 231, reconfigurable module 232, shared module 233, reconfigurable module 234, shared module 235, reconfigurable module 236.
  • the preprocessor module 230 includes three reconfigurable modules in this embodiment, the preprocessor module 230 may contain two or more reconfigurable modules; the number of reconfigurable modules is not limited if a shared module is provided between reconfigurable modules.
  • the configuration interface reconfigures a reconfigurable module on the bases of instructions from the configuration controller 410 in order to modify the particulars of a preprocess.
  • This circuit reconfiguration uses the desired circuit reconfiguration data selected from the FPGA reconfiguration data 330.
  • Modifying the particulars of the preprocess may occur based on an instruction from the user, or may occur when the preproccesing is replaced in accordance with a program; the particulars of the preprocess may also be modified when the image processing controller 420 itself detects that the scene in the image changed. In other words, the particulars of the preprocess is modified when it is necessary to change the particulars thereof due to some cause.
  • Each of the reconfigurable modules 232, 234, 236 is guaranteed certain amount of space in the FPGA 200 so that the circuit may be reconfigured using any of the circuit reconfiguration data in the FPGA reconfiguration data 330.
  • the reconfigurable modules 232, 234, 236 are each arithmetic converter circuits that perform computations on the image data to convert the image data.
  • the aforementioned arithmetic converter circuit is programmed via the configuration interface 240 using the circuit reconfiguration data.
  • Each of the arithmetic converter circuits constituting the reconfigurable modules 232, 234, 236 may be mutually identical or mutually different sizes on the FPGA 200.
  • the shared modules 231, 233, 235 are timing control circuits that ensure reliable exchange of image data.
  • the reconfigurable module and the shared module are described in detail with reference to FIG. 3 and FIG. 4 .
  • FIG. 3 is a block diagram representing a partial exploded view of a preprocessor module 230 within the preprocessing circuit of the image processing device 100. Data flows from left to right in FIG. 3 .
  • the shared module 233 connects between the reconfigurable module 232 and the reconfigurable module 234.
  • the arithmetic converter circuit constituting the reconfigurable module in the image processing device 100 is an image filter.
  • the image filter uses computations to convert the image data and perform edge detection or image expansion or contraction.
  • the reconfigurable module in the image processing device 100 is provided with three image filters, with an image filter corresponding to the R, G, and B in a color image, respectively.
  • the shared module in the image processing device 100 is a flip-flop circuit (referred to as an FF circuit below) that acts as a timing control circuit ensuring the reliable exchange of data between the image filters.
  • the FF circuits may also be referred to as slice registers.
  • the FF circuit i.e, the shared module 233 secures the timing for transferring data until the data is sent to the next image filter (i.e., the next reconfigurable module 234).
  • providing an FF circuit between the image filters ensures that the processed data is transferred to the next image filter (e.g. the reconfigurable module 234) only after the computations from the previous image filter of the three filters (e.g., the reconfigurable module 232) is reliably complete.
  • Providing an FF circuit between that the image filters also ensures that no issues occur during the operation of the preprocessing circuit regardless of how the reconfigurable modules are reprogrammed.
  • the image filter (i.e., the reconfigurable module 232) is made up of an input module 232A that temporarily stores data entering therein, an arithmetic module 232b that performs arithmetic computations on the temporarily stored data, and an output module 232c that temporarily stores the processed data.
  • the image filter (i.e., the reconfigurable module 234) is similarly configured with an input module 234a, and arithmetic module 234b, and an output module 234c.
  • FIG. 4 illustrates the data processing that takes place in the reconfigurable module 232 in the image processing device 100.
  • the image filter (i.e., the reconfigurable module 232) is configured by an input module 232a, and arithmetic module 232b, and an output module 232c.
  • the input image data entering the input module 232a of the image filter flows in from top to bottom. For instance, data flows into the input module as data in line (n - 2), to data in line (n - 1), to data in line n, in that order. The data in each of the lines flows into the input module in order from left to right.
  • the data that has entered the input module 232a is stored in the RAM which is a storage region provided on the FPGA 200.
  • the FPGA 200 in this embodiment is provided with two RAM modules so that the image filter can perform a 3 x 3 computation in the arithmetic module 232b. Note that the aforementioned RAM is provided to each image filter.
  • the input module 232a stores the data entered in line (n - 2) in the first RAM, and the data entered in line (n - 1) in the second RAM; data entered in line n is transferred as is to the arithmetic module 232b.
  • the arithmetic module 232b performs a 3 x 3 computation, taking the data from line (n - 2) read out from the first RAM, the data from line (n - 1) read out from the second RAM, and the data from line n transferred from the input module 232a; the arithmetic module 232b forwards the computed data to the output module 232c.
  • the output module 232e collects, for instance, the RGB data forwarded by the arithmetic module 232b and outputs the same as processed data.
  • the FPGA 200 would need four RAM modules per image filter.
  • the configuration controller 410 reconfigures the preprocessing circuit while the device is running.
  • a programmable logic circuit typically has only a single image filter, with that image filter built for partial reconfiguration.
  • the reconfiguration process takes a comparatively longer time in existing image processing devices because either the entire preprocessing circuit is reconfigured or the entire module that executes preprocessing is reconfigured.
  • preprocessing can be modified by reconfiguring the circuitry for the image filter; that is, a preprocess can be modified by reconfiguring the arithmetic converter circuit constituting the reconfigurable module.
  • FF circuits serving as the timing control circuits i.e., the shared modules
  • these circuits have nothing to do with modifying the particulars of preprocessing.
  • the image processing device 100 of the embodiment selects at least one of the plurality of image filters for partial reconfiguration while excluding the FF circuits from partial reconfiguration when modifying the particulars of preprocessing while the device is running.
  • the circuitry for partial reconfiguration is a smaller size.
  • the number of shared modules reduces the number of circuits for partial reconfiguration, even when all the reconfigurable modules are reprogrammed. Reducing the size of the circuits that will be partially reconfigured reduces the amount of time needed for data transfer. Thus, partial reconfiguration takes less time.
  • the image processing device in this embodiment is capable of further reducing the amount of time a programmable circuit such as an FPGA takes to be partially reconfigured.
  • the image processing device 100 in this embodiment also has the following advantages. These advantages are described on the basis of FIGS. 5A and 5B , and FIGS. 6A and 6B .
  • FIG. 5A is a block diagram of a configuration of a preprocessing circuit 900 in a typical image processing device containing a plurality of preprocessing functions.
  • FIG. 5B is a block diagram of a configuration of a preprocessing circuit in the image processing device 100 of the embodiment wherein a plurality of reconfigurable circuits is programmed with the same kind of preprocessing function.
  • the preprocessing circuit in the typical image processing device 900 includes a built-in camera interface 910, a switch 920, function A preprocessor circuit 930, function B preprocessor circuit 940, and function C preprocessor circuit 950 each connected to the switch 920.
  • the preprocessing circuit in the image processing device 900 is connected to an external camera 2 and a memory 960.
  • the data captured by the camera 2 is transferred to the camera interface 910 and thereafter transferred to the switch 920.
  • preprocessing in the typical image processing device 900 may be performed in the following cases when the function A preprocessor circuit carries out preprocessing three times.
  • the switch 920 transfers data to the function A preprocessor circuit 930, and data A1 resulting from operations and conversions performed on the data by the function A preprocessor circuit is transferred to and stored in the memory 960. At this point, processing is not performed again by the function A preprocessor circuit 930 until all the processing by the function A preprocessor circuit 930 is terminated.
  • the data A1 is then read out from memory the memory 960, again transferred to the function A preprocessor circuit 930 to carry out preprocessing again, and the data A2 output from the function A preprocessor circuit 930 output to the memory 960 and stored therein.
  • the data A2 is then read out from the memory 960, transferred to the function A preprocessor circuit 930 and processing performed a third time.
  • Preprocessing in this manner in the typical image processing device 900 requires repeating the processes of function switching by the switch 920, writing to the external memory 960, and reading out from memory, and thus preprocessing requires a relatively longer time.
  • the image processing device 100 operates as follows when a plurality of the same kind of preprocessing is performed.
  • the preprocessing circuit in the image processing device 100 includes a camera interface 170, and three reconfigurable circuits 180, 181, 182.
  • the camera interface 170 is equivalent to the image acquisition unit 210 and the data controller 220 ( FIG. 1 ).
  • the preprocessing circuit in the image processing device 100 is connected to an external camera 2 and memory 190.
  • the data captured by the camera 2 is transferred to the camera interface 170 and processed in series by three reconfigurable circuits 180, 181, 182 which each have the same kind of preprocessing function.
  • the data operated on and converted by the first reconfigurable circuit 180 can be sent to the next reconfigurable circuit 181.
  • FIG. 6A depicts a process where noise is emphasized by using a differential function n times.
  • FIG. 6B illustrates the processes of eliminating noise by using a reduction function n times, and returning an image to its original size by using an expansion function n times.
  • repeating the differential process on an image containing noise emphasizes the edges of the noise data.
  • the portion of the edge that is at or above a given threshold is considered noise, and removing that noise portion from the image, removes noise from the image.
  • a function reducing an image may be used repeatedly on an image including includes noise data, whereby the noise data is removed from the image.
  • the image, with the noise data removed can be passed through an expansion function repeatedly to return the image to its original size. This removes noise from the image.
  • the preprocessing circuit is partially reconfigured with a plurality of circuits having the same function; hereby this above kind of repeated processing can be carried out continuously without needing to store the processing results in the external memory. It is thus possible to reduce the preprocessing execution time compared to existing techniques.
  • FIG. 7 is a flowchart illustrating the flow of processes executed in the image processing device 100 immediately after the device is powered on.
  • the image processing device 100 begins to setup the FPGA 200 immediately after the device is turned on.
  • the configuration controller 410 reads the startup configuration data from the storage unit 300, executes startup configuration of the FPGA 200 on the basis of said data, and programs the FPGA 200 as the preprocessing circuit (S10)
  • the configuration controller 410 uses FPGA reconfiguration data 330 to reconfigure the circuitry of the reconfigurable module in the preprocessor module 230.
  • the configuration controller 410 continues the reconfiguration process if reconfiguration of all the reconfigurable modules in the preprocessor module 230 is incomplete (NO at S30).
  • the FPGA reconfiguration data 330 stored in the storage unit 300 which is equivalent to the ROM 120, is transferred to the DDR 140 and stored (S40).
  • the image processing device 100 terminates initial setup of the FPGA 200.
  • FIG. 8 is a flowchart illustrating the processes executed by the image processing device on the reconfigurable modules in the preprocessing circuit during a partial reconfiguration.
  • the configuration controller 410 selects from a plurality of types of circuit reconfiguration data included in the FPGA reconfiguration data, and transmits the desired circuit reconfiguration data to the FPGA 200 (S100).
  • the configuration interface 240 reconfigures the desired reconfigurable module on the basis of the circuit reconfiguration data sent from the configuration controller 410.
  • the interrupt controller 250 On receiving information from the configuration interface 240 that circuit reconfiguration was successful (YES at S110), the interrupt controller 250 generates a success interrupt in the CPU 110 serving as the controller 400 indicating that circuit reconfiguration was successful. The circuit reconfiguration of the reconfiguration module is then terminated in this case.
  • the configuration interface 240 retries reconfiguring the desired reconfigurable module.
  • the interrupt controller 250 On receiving information from the configuration interface 240 that circuit reconfiguration failed (YES at S120), the interrupt controller 250 generates a failure interrupt in the CPU 110 serving as the controller 400 indicating that circuit reconfiguration failed.
  • the CPU 110 acting as the controller 400 executes a circuit reconfiguration failure process that generates a notification indicating that reconfiguration of the reconfigurable module failed.
  • the configuration controller 410 When reconfiguration has failed less than three times in succession (NO at S130), the configuration controller 410 resends the desired circuit reconfiguration data to the FPGA 200, and the image processing device repeats the steps S100 through S130.
  • a procedure executed by the image processing device 100 to partially reconfigure the preprocessing circuit is described below with reference to FIG. 9 .
  • the procedure described takes place when a user or program modifies the parameters of the preprocess while the image processing device is running.
  • FIG. 9 is a flowchart illustrating the flow of processes during a partial reconfiguration executed by the image processing device on the reconfigurable module in the preprocessing circuit, when the preprocess parameters are modified while the device is running.
  • the image processing device 100 halts operation of the reconfigurable module in the FPGA 200 (S210).
  • the configuration controller 410 sends the FPGA 200 the desired circuit reconfiguration data selected from the plurality of kinds of circuit reconfiguration data contained in the FPGA reconfiguration data 330; and the configuration interface 240 executes a process to reconfigure the desired reconfigurable module (S220).
  • This step S220 is repeated for all the reconfigurable modules requiring reconfiguration so long as the circuit reconfiguration process has not ended (NO at S230).
  • the image processing device 100 starts operations of the reconfigurable module in the FPGA 200 when the circuit reconfiguration process ends for all the reconfigurable modules requiring reconfiguration (YES at S240).
  • the image processing device 100 ends the partial reconfiguration process for the reconfigurable module taking place in the preprocessing circuit while the device is running.
  • the image processing techniques executed by the image processing device 100 can be presented as follows. That is, the image processing techniques executed by the image processing device 100 is run on image data obtained externally, and includes preprocessing executed during the aforementioned image processing that is executed by a preprocessing circuit, and control of circuit configuration which takes place during partial reconfiguration of the preprocessing circuit.
  • the preprocessing step includes at least one arithmetic conversion step where an arithmetic converter circuit performs arithmetic computations and conversions on the image data, with the arithmetic conversion step configured to include an input step temporarily storing data input thereto, an arithmetic computation step carrying out arithmetic computations on said temporarily stored data, and an output step temporarily storing the computed data.
  • An input module, an arithmetic module, and an output module in the arithmetic converter circuit carry out the aforementioned steps.
  • the circuit configuration control step carries out partial reconfiguration on at least one arithmetic module that executes an arithmetic computation step within the preprocessing circuit, but does not partially reconfigure the input module and the output module that executes the input step and the output step in the preprocessing circuit.
  • the configuration is also such that the circuit configuration control step partially reconfigures at least only one arithmetic converter circuit in the preprocessing circuit while not partially reconfiguring the other arithmetic converter circuits or timing control circuits.
  • the circuitry for partial reconfiguration is therefore a smaller size. Thus, partial reconfiguration takes less time.
  • an image processing method can be provided that is capable of further reducing the amount of time a programmable circuit such as an FPGA takes to perform a partial reconfiguration.
  • configuration is a term specific to FPGAs within the realm of programmable logic circuits. It is therefore clear that when the image processing device is equipped with programmable logic circuits that are not FPGAs, an equivalent term may be used in place of "configuration" when endeavoring to understand the details of this description.
  • FIG. 10 Yet another embodiment can be described below based on FIG. 10 .
  • the configurations distinct from the first embodiment are described. That is, the makeup described for the first embodiment can be included in this embodiment. Additionally, the definitions for the terms described for the first embodiment are the same.
  • FIG. 10 is a block diagram representing a partial exploded view of a preprocessor module within the preprocessing circuit in the image processing device 100.
  • the image processing device 100 in the second embodiment partially reconfigures only the arithmetic modules in the arithmetic converter circuits making up the partial reconfiguration module in the preprocessing circuit.
  • the second embodiment differs from the first embodiment in that the input module and the output module are shared modules, and there are no FF circuits acting as shared modules between each of the arithmetic converter circuits.
  • the preprocessor module in the preprocessing circuit of the image processing device 100 includes two image filters 600, 610 that serve as the arithmetic converter circuits.
  • the image filter 600 is made up of an input module 601, an arithmetic module 602, and an output module 603; the image filter 610 is made up of an input module 611, and arithmetic module 612, and an output module 613.
  • the configuration controller 410 in the image processing device 100 partially reconfigures the arithmetic module 602 and the arithmetic module 612, while leaving the input module 601, the output module 603, the input module 611, and the output module 613 untouched.
  • the input module 601, the output module 603, the input module 611, and the output module 613 are shared modules.
  • the configuration controller 410 when the image processing device 100 is partially reconfiguring the preprocessing circuit, the configuration controller 410 partially reconfigures at least one of the aforementioned arithmetic modules while not partially reconfiguring the other circuits. Consequently, the circuitry for partial reconfiguration is a smaller size. Thus, partial reconfiguration takes less time.
  • an image processing device capable of further reducing the amount of time a programmable circuit, such as an FPGA, takes to perform a partial reconfiguration.
  • the input modules and the output modules of the image filters are treated as shared modules and thus not subject to partial reconfiguration. Therefore, the image processing device 100 is able to ensure reliable exchange of data even if there are no FF circuits provided between the image filter 600 and the image filter 610. Thus, preprocessor circuit uses less resources.
  • the image processing techniques run by the image processing device 100 can be presented as follows. That is, the image processing techniques run by the image processing device 100 is run on image data obtained externally, and includes preprocessing executed during the aforementioned image processing that is run by a preprocessing circuit, and control of circuit configuration which takes place during partial reconfiguration of the preprocessing circuit.
  • the aforementioned preprocessing step includes at least one arithmetic conversion step which includes performing arithmetic computations and conversions on the image data, with the arithmetic conversion step configured to include an input step temporarily storing data input thereto, an arithmetic computation step carrying out arithmetic computations on said temporarily stored data, and an output step temporarily storing the computed data.
  • the circuit configuration control step carries out partial reconfiguration on at least one arithmetic module that executes an arithmetic computation step within the preprocessing circuit, but does not partially reconfigure the input module and the output module that executes the input step and the output step in the preprocessing circuit.
  • the circuit configuration control step partially reconfigures at least one arithmetic module in the preprocessing circuit while not partially reconfiguring the other circuits. Consequently, the circuitry for partial reconfiguration is a smaller size. Thus, partial reconfiguration takes less time.
  • an image processing method can be provided that is capable of further reducing the amount of time a programmable circuit such as an FPGA takes to perform a partial reconfiguration.
  • the control block in the image processing device 100 may be implemented as a logic circuit created from integrated circuits (IC chips) or the like (i.e., in hardware), or as software run on a CPU.
  • IC chips integrated circuits
  • the control block in the image processing device 100 may be implemented as a logic circuit created from integrated circuits (IC chips) or the like (i.e., in hardware), or as software run on a CPU.
  • the image processing device 100 is equipped with, for example: a CPU that executes commands from a program, i.e. software that realizes each of the functions of the image processing device; a ROM or storage device (also referred to as a recording medium), onto which the above-mentioned program and various kinds of data are recorded and from which the same can be read by a computer (or CPU); and a RAM into which the above mentioned program can be expanded.
  • the computer may read the above mentioned program from the above mentioned recording medium, and execute the program whereby the computer realizes an embodiment of the invention.
  • the above mentioned recording medium may be a non-transitory physical medium such as a tape, disc, card, semiconductor memory, or programmable logic circuit.
  • the above mentioned program may also be supplied to the computer via any desired transmission medium capable of transferring the program (for example, a communication network, or broadcast waves).
  • any desired transmission medium capable of transferring the program for example, a communication network, or broadcast waves.
  • embodiments of the invention may also be implemented in the form of data signals encapsulated in carrier waves that are realized through the electronic transmission of the above mentioned program.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Stored Programmes (AREA)
  • Logic Circuits (AREA)
  • Image Processing (AREA)
EP17159552.3A 2016-04-15 2017-03-07 Information processing device, information processing method, information processing program, and recording medium Active EP3232334B1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2016082259A JP6816380B2 (ja) 2016-04-15 2016-04-15 画像処理装置、画像処理方法、情報処理プログラム、および記録媒体

Publications (2)

Publication Number Publication Date
EP3232334A1 EP3232334A1 (en) 2017-10-18
EP3232334B1 true EP3232334B1 (en) 2023-05-03

Family

ID=58360817

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17159552.3A Active EP3232334B1 (en) 2016-04-15 2017-03-07 Information processing device, information processing method, information processing program, and recording medium

Country Status (4)

Country Link
US (1) US10108875B2 (ja)
EP (1) EP3232334B1 (ja)
JP (1) JP6816380B2 (ja)
CN (1) CN107302641B (ja)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6891976B2 (ja) * 2017-12-12 2021-06-18 日本電気株式会社 Ｆｐｇａシステム、パーシャルリコンフィグレーション実行方法、記憶媒体及びデータ構造
JP7451407B2 (ja) * 2018-12-07 2024-03-18 ソニーセミコンダクタソリューションズ株式会社 センサ装置、電子機器、センサシステム及び制御方法
US10651853B1 (en) * 2019-05-23 2020-05-12 Xilinx, Inc. Timing insulation circuitry for partial reconfiguration of programmable integrated circuits

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7088860B2 (en) * 2001-03-28 2006-08-08 Canon Kabushiki Kaisha Dynamically reconfigurable signal processing circuit, pattern recognition apparatus, and image processing apparatus
JP4861030B2 (ja) 2006-03-24 2012-01-25 株式会社東芝 半導体装置
JP5202265B2 (ja) * 2008-12-08 2013-06-05 キヤノン株式会社 画像処理装置、プログラム及び画像処理装置の制御方法
JP5247542B2 (ja) 2009-03-05 2013-07-24 キヤノン株式会社 画像処理装置、画像処理装置の制御方法、及び、プログラム
JP5504985B2 (ja) * 2010-03-11 2014-05-28 富士ゼロックス株式会社 データ処理装置
KR20120036208A (ko) * 2010-10-07 2012-04-17 삼성전자주식회사 재구성 기반 컴퓨팅 장치 및 이의 메모리 의존성 보정방법
JP5943736B2 (ja) 2012-06-28 2016-07-05 キヤノン株式会社 情報処理装置、情報処理装置の制御方法及びプログラム
GB2526018B (en) * 2013-10-31 2018-11-14 Silicon Tailor Ltd Multistage switch
JP6370063B2 (ja) 2014-03-04 2018-08-08 キヤノン株式会社 画像処理装置及びその制御方法、並びにプログラム
JP2015228187A (ja) 2014-06-02 2015-12-17 キヤノン株式会社 画像処理装置及び回路を再構成するための方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
DELAHAYE J-P ET AL: "Software radio and dynamic reconfiguration on a DSP/FPGA platform", FREQUENZ, SCHIELE UND SCHON, BERLIN, DE, vol. 58, no. 5-6, 1 May 2004 (2004-05-01), pages 152 - 159, XP002530814, ISSN: 0016-1136 *

Also Published As

Publication number Publication date
EP3232334A1 (en) 2017-10-18
JP2017191566A (ja) 2017-10-19
US20170300774A1 (en) 2017-10-19
JP6816380B2 (ja) 2021-01-20
CN107302641B (zh) 2019-05-14
CN107302641A (zh) 2017-10-27
US10108875B2 (en) 2018-10-23

Similar Documents

Publication Publication Date Title
US20180082175A1 (en) Convolutional Neural Network Processing Method and Device
EP3232334B1 (en) Information processing device, information processing method, information processing program, and recording medium
US10614148B2 (en) Reconfigurable convolution accelerator
US7984266B2 (en) Integrated computer array with independent functional configurations
JP5943736B2 (ja) 情報処理装置、情報処理装置の制御方法及びプログラム
KR20200068564A (ko) Ssd들 상에서의 가속된 데이터 처리를 위한 시스템 및 방법
JP5016807B2 (ja) データ処理システム
CN115698952A (zh) 可伸缩的硬件线程调度器
Maggiani et al. Reconfigurable architecture for computing histograms in real-time tailored to FPGA-based smart camera
JP2016115979A (ja) 画像処理装置、その制御方法及びプログラム
US10769090B2 (en) Information processing apparatus, control method of information processing, and non-transitory computer-readable storage medium for storing program
CN113992909A (zh) 一种mipi d-phy接口摄像头模组的测试***及方法
US8803893B2 (en) Image data processing apparatus
Park et al. A reconfigurable platform for the design and verification of domain-specific accelerators
JP2002337400A (ja) 画像処理装置
US11966749B2 (en) Processor and booting method thereof
CN114817117B (zh) 协议兼容方法、电子设备及计算机可读存储装置
CN116561049A (zh) 一种基于片内处理器配置码流的fpga及码流配置方法
US7017033B2 (en) Arithmetic apparatus and arithmetic method
Wernsman Analyzing energy savings in an FPGA video processing system using dynamic partial reconfiguration
Nascimento et al. System-on-Chip design flow for the image signal processor of a nonlinear CMOS imaging system
JP2016133898A (ja) 画像処理装置、画像処理装置の制御方法、及びプログラム
Ackermann et al. Enabling self-reconfiguration on a video processing platform
CN112261308A (zh) 一种具有片内模式识别的感光集成电路
RU2179739C2 (ru) Устройство для обработки информации

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20170328

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20191219

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 1/00 20060101ALN20220811BHEP

Ipc: G06F 15/78 20060101AFI20220811BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 1/00 20060101ALN20220922BHEP

Ipc: G06F 15/78 20060101AFI20220922BHEP

INTG Intention to grant announced

Effective date: 20221013

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017068227

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1565260

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230515

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20230503

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1565260

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230503

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230904

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230803

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230903

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230804

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602017068227

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20240206

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230503