EP3316184A1 - Program generating device, program generating method, and generating program - Google Patents
Program generating device, program generating method, and generating program Download PDFInfo
- Publication number
- EP3316184A1 EP3316184A1 EP15896359.5A EP15896359A EP3316184A1 EP 3316184 A1 EP3316184 A1 EP 3316184A1 EP 15896359 A EP15896359 A EP 15896359A EP 3316184 A1 EP3316184 A1 EP 3316184A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- image
- program
- image processing
- evaluation value
- output
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims description 85
- 238000012545 processing Methods 0.000 claims abstract description 187
- 230000002068 genetic effect Effects 0.000 claims abstract description 18
- 238000011156 evaluation Methods 0.000 claims description 132
- 230000008569 process Effects 0.000 claims description 46
- 230000002194 synthesizing effect Effects 0.000 claims description 3
- 230000008859 change Effects 0.000 claims description 2
- 238000004590 computer program Methods 0.000 claims 2
- 230000004083 survival effect Effects 0.000 abstract description 20
- 102000003800 Selectins Human genes 0.000 abstract 1
- 108090000184 Selectins Proteins 0.000 abstract 1
- AAEVYOVXGOFMJO-UHFFFAOYSA-N prometryn Chemical compound CSC1=NC(NC(C)C)=NC(NC(C)C)=N1 AAEVYOVXGOFMJO-UHFFFAOYSA-N 0.000 abstract 1
- 238000004364 calculation method Methods 0.000 description 21
- 238000000605 extraction Methods 0.000 description 18
- 239000004065 semiconductor Substances 0.000 description 17
- 238000003384 imaging method Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 10
- 230000035772 mutation Effects 0.000 description 10
- 239000000284 extract Substances 0.000 description 8
- 230000000052 comparative effect Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 4
- 238000013500 data storage Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000010276 construction Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 238000005286 illumination Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 241001377084 Actites Species 0.000 description 1
- 238000007630 basic procedure Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005401 electroluminescence Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/36—Software reuse
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/12—Computing arrangements based on biological models using genetic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/12—Computing arrangements based on biological models using genetic models
- G06N3/126—Evolutionary algorithms, e.g. genetic algorithms or genetic programming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration using local operators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/94—Hardware or software architectures specially adapted for image or video understanding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/96—Management of image or video recognition tasks
Definitions
- the embodiments discussed herein relate to a program generation apparatus, a program generation method, and a generation program.
- a technology for automatically generating an image processing program that performs desired image processing, by using genetic programming, is drawing attention.
- This technology is designed to optimize an image processing program that is generated by combining partial programs for image processing (for example, image filtering programs), based on learning data such as pairs of an input image and an image obtained as a result of processing (a target image), by using genetic programming.
- NPTL 1 Shinya Aoki and Tomoharu Nagao, "ACTIT: Automatic Construction of Tree-structural Image Transformations", The Journal of The Institute of Image Information and Television Engineers, Vol. 53, No. 6, June 20, 1999, pp. 890-892
- the following survival selection method is used, for example. That is, an input image included in learning data is processed with a program corresponding to an individual generated in the course of learning. An output data that is output as the processing result is compared with a target image included in the learning data. Then, a determination is made as to whether to pass the individual to the next generation, based on the comparison result.
- One aspect of the present disclosure is to provide a program generation apparatus, a program generation method, and a generation program capable of performing appropriate survival selection when generating an image processing program by using genetic programming.
- a program generation apparatus that generates a program by using genetic programming.
- the program generation apparatus includes a storage unit and a processing unit.
- the storage unit stores learning data including an input image and a first target image.
- the first target image indicates an image that is output halfway through a process of converting the input image into a second target image.
- the processing unit selects a first program from among a plurality of image processing programs each generated by combining a plurality of partial programs, generates a second program by changing a part of the partial programs included in the first program, performs image processing on the input image using the second program, determines whether to pass the second program to a next generation, based on a comparison between one or more intermediate output images and the first target image, the one or more intermediate output images being output halfway through the image processing, and replaces one of the plurality of image processing programs with the second program when the second program is determined to be passed to the next generation.
- a program generation method executed by a computer to perform the same procedure as that performed by the program generation apparatus described above.
- a generation program that causes a computer to perform the same procedure as that performed by the program generation apparatus described above.
- FIG. 1 illustrates an exemplary configuration and operation of a program generation apparatus according to a first embodiment.
- a program generation apparatus 1 is an apparatus that generates an image processing program by using genetic programming.
- the program generation apparatus 1 includes a storage unit 1a and a processing unit 1b.
- the storage unit 1a is implemented, for example, as a volatile storage device such as random access memory (RAM) and the like, or a non-volatile storage device such as hard disk drive (HDD), flash memory, and the like.
- the processing unit 1b is, for example, a processor.
- the storage unit 1a stores learning data 10.
- the learning data 10 includes an input image 11, a first target image 12, and a second target image 13.
- the second target image 13 is a target image for an image that is output as the processing result of image processing performed on the input image 11.
- the first target image 12 is a target image for an image that is output at a certain step halfway through image processing performed for converting the input image 11 into the second target image 13.
- the first target image 12 may be an image in which a specific image region of the image is distinguished from the other image region (background region).
- the image regions may be distinguished by, for example, assigning pixel values, such as setting the pixel value of the specific image region to a maximum value and setting the pixel value of the background image region to a minimum value (0).
- the storage unit 1a may store a plurality of sets of learning data 10.
- the storage unit 1a may store a program group 20.
- the program group 20 is data used for operations performed by the processing unit 1b, and includes a plurality of image processing programs 21, 22, 23, and so on. Each of the image processing programs 21, 22, 23 and so on included in the program group 20 is generated by combining a plurality of partial programs.
- a partial program is a program component for performing image processing such as image filtering and the like. Every time a new generation of the program group 20 is created, the program generation apparatus 1 preferentially preserves an image processing program with a high fitness value in the program group 20, and thereby generates an image processing program that performs desired image processing.
- the processing unit 1b selects an image processing program from the program group 20 (step S1). Here, for example, an image processing program 21 is selected. Then, the processing unit 1b generates an image processing program 21a by changing a part of the partial programs included in the selected image processing program 21.
- This processing process is a process of evolving the image processing program 21 (step S2). This evolution process involves, for example, a crossover between the image processing program 21 and another image processing program selected from the program group 20, a mutation of the image processing program 21 or the program resulting from the crossover, and the like.
- the image processing program 21a is generated by combining partial programs P1 to P3, for example. Note that in FIG. 1 , “In” indicates an input part of the image processing program 21a, and “Out” indicates an output part of the image processing program 21a.
- the processing unit 1b performs image processing on the input image 11, using the image processing program 21a (step S3).
- the processing unit 1b outputs intermediate output images halfway through this image processing.
- the intermediate output images are images that are respectively output as the processing results of the partial programs P1 and P2, other than the partial program P3 incorporated in the final stage, from among the partial programs P1 to P3 of the image processing program 21a.
- an intermediate output image 31 is output as the processing result of the partial program P1
- an intermediate output image 32 is output as the processing result of the partial program P2.
- step S4 determines whether to pass the generated image processing program 21a to the next generation.
- the operation of step S4 includes an operation (step S4a) of comparing the intermediate output images 31 and 32 with the first target image 12.
- step S4a for example, the similarity between the images is output as the comparison result.
- the processing unit 1b determines whether to pass the image processing program 21a to the next generation.
- step S4 the determination may be made based not only on the comparison result of step S4a, but also on the comparison result between a final output image, which is output as the execution result of the image processing program 21a, and the second target image 13.
- step S4 If, in step S4, the image processing program 21a is determined to be passed to the next generation, the processing unit 1b replaces one of the image processing programs 21, 22, 23, and so on of the program group 20 with the image processing program 21a (step S5). Thus, a new generation of the program group 20 is created.
- One method of determining as to whether to pass the image processing program 21a to the next generation may be to make a determination based on a comparison between the final output image, which is output as the result of image processing by the image processing program 21a, and the second target image 13.
- the image processing program 21a is eliminated and not passed to the next generation of the program group 20. That is, an effective image processing program that is likely to contribute to promoting learning is eliminated. This may result in an increase in time taken to complete generation of an image processing program.
- the comparison result in step S4a represents the index indicating how close the intermediate output image that is output halfway through execution of the image processing program 21a is to the desired image. Since a determination is made based on this index in step S4, when the image processing program 21a is likely to be able to output an appropriate image halfway through execution of the image processing program 21a, the image processing program 21a is passed to the next generation of the program group 20 without being eliminated. Thus, appropriate survival selection is performed such that an image processing program whose processing process is specified as being appropriate is more likely to be passed to the next generation.
- steps S1 to S5 are repeatedly executed using the new generation of the program group 20 that is created by the procedure described above.
- the speed of increase in the similarity between images indicated by the comparison result of step S4a is increased, so that the learning speed is improved.
- This reduces the time taken for the maximum value among the fitness values of the image processing programs included in the program group 20 and the image processing programs generated from these image processing programs through the evolution process to reach a predetermined threshold. That is, the time taken to generate an image processing program that implements desired processing is reduced.
- the image processing apparatus according to the second embodiment has the processing functions similar to those of the program generation apparatus 1 of FIG. 1 , and also has a function of performing image processing by executing an image processing program generated by these functions.
- FIG. 2 illustrates a comparative example of a procedure for an image processing program generation process.
- the learning data 50 includes an input image 51 and a target image 52 that is obtained by performing image processing on the input image 51.
- the input image 51 may be obtained, for example, by capturing an image of an object with a camera.
- an individual is formed by combining one or more partial programs. For example, as illustrated in the upper left of FIG. 2 , an individual is defined by a tree structure.
- a plurality of partial programs that may be incorporated into an individual are also prepared in advance.
- image filters are used as an example of partial programs that are incorporated into an individual.
- the partial programs are not limited to image filters, and programs for performing other types of image processing may be used. Note that in the upper left of FIG. 2 , "F" represents an image filter; “I” represents an input terminal; and “O” represents an output terminal.
- the image processing program generation process using genetic programming is performed, for example, in the following manner.
- a population 61 including a plurality of initial individuals is generated (step S11).
- Image filters are randomly selected from among a plurality of image filters prepared in advance and incorporated into the nodes of each initial individual.
- two parent individuals are randomly extracted from the generated population 61 (step S12).
- the two parent individuals undergo an evolution process, so that two or more child individuals are generated (step S13).
- a crossover operation and a mutation operation are performed on the two parent individuals.
- Three or more child individuals may be generated by performing different crossover operations and mutation operations on the two parent individuals.
- a fitness value is calculated for each of the child individuals generated through the evolution process and the original parent individuals (step S14).
- image processing using each of these individuals is performed on the input image 51 of the learning data 50.
- an image obtained from the image processing is compared with a corresponding target image 52 to thereby calculate a fitness value of the individual.
- the average of fitness values that are obtained using the plurality of sets of learning data 50 is calculated for each individual.
- the individual is output as a final image processing program. Then, the image processing program generation process ends.
- survival selection is performed on a population 62 including the generated child individuals and the two original parent individuals (step S15). In the survival selection, an individual with the highest fitness value is selected from the population 62. Further, one individual is selected from among the remaining individuals of the population 62 by using a predetermined method. For example, one individual is selected from among the remaining individuals in accordance with the probabilities based on their fitness values.
- the two individuals selected by the survival selection replace two individuals included in the population 61 (step S16).
- the two individuals selected by the survival selection replace the two individuals extracted as the parent individuals, among the individuals included in the population 61.
- the individuals included in the population 61 are changed to individuals of the next generation. Then, the same procedure is repeated until an individual with a fitness value greater than or equal to the predetermined threshold is produced.
- FIG. 3 illustrates an example of a crossover.
- a crossover is performed between parent individuals 71a and 72a, thereby generating a child individual 71b derived from the parent individual 71a and a child individual 72b derived from the parent individual 72a.
- the parent individual 71a includes image filters F1, F2, F3, and F4, and the parent individual 72a includes image filters F2, F3, F5, and F6. Assume here that the node of the image filter F2 in the parent individual 71a and the node of the image filter F5 in the parent individual 72a are selected to be subjected to a crossover.
- a crossover operation for example, not only a selected node but also nodes at levels lower than the level of the selected node are subjected to a crossover. Accordingly, in the example of FIG. 3 , a section of the parent individual 71a including "the image filters F2 and F1; a node of an input terminal connected to one end of the image filter F2; and a node of an input terminal connected to the image filter F1" is swapped with a section of the parent individual 72a including "the image filter F5; and a node of an input terminal connected to the image filter F5".
- the crossover produces the child individual 71b including the image filters F3, F4, and F5, and the child individual 72b including single image filters F1, F2, and F4 and two image filters F3.
- FIG. 4 illustrates an example of a mutation.
- an individual 73a includes the image filters F3, F4, and F5.
- the individual 73a may be a parent individual extracted from the population 61, or an individual produced by performing a crossover on the parent individuals extracted from the population 61.
- the node of the image filter F3 in the individual 73a is selected as a site of mutation, and an image filter F7 is selected as a replacing image filter in the mutation operation.
- the replacing image filter in the mutation operation is randomly selected from among a plurality of image filters prepared in advance.
- the mutation produces a child individual 73b including the image filters F4, F5, and F7.
- Conceivable uses of an image processing program generated by the above procedure include, for example, achieving a desired effect by performing image processing on an image of a product in the factory automation (FA) field.
- the image processing program may be used to perform image processing on an image of the appearance of a product to thereby extract sites with defects, extract points for alignment, or recognize characters printed on a specific component.
- FIG. 5 illustrates an example of image processing.
- the image processing includes, for example, performing certain processing on a specific region of an input image.
- FIG. 5 illustrates a process of extracting, from a package region of a semiconductor chip mounted on a printed circuit board, only the portion of text printed in that region as a region of a specific color (for example, white).
- a final output image 81d that is finally output as the result of image processing, only the portion of text "ABC123" printed in a package region of a semiconductor chip is extracted as a white region.
- the image processing of FIG. 5 is performed as preprocessing for recognizing text in a specific region.
- the process of performing certain processing on a specific region may include a process of extracting, from a region where a specific component is mounted, a portion of symbols or patterns drawn on the component as a region of a specific color.
- Other examples may also include a process of extracting, from a specific region in an image, only a region where a component is mounted as a region of a specific color, in order to detect the position and inclination of the component in the specific region.
- extraction target region a specific region from which text, symbols, or patterns are extracted.
- the brightness of the object at the time of imaging may greatly vary from image to image.
- the degree of variation in the brightness of the extraction target region in the image often differs from the degree of variation in the brightness of the background region other than the extraction target region. This is because the cause of variation in brightness may differ between the extraction target region and the background region. Possible causes include, for example, the difference in light reflectance and color tendency between the extraction target region and the background region, and the difference in how light is received from the surroundings due to the difference in height between the extraction target region and the background region.
- the package region of the semiconductor chip is black, and therefore the luminance changes little with variation of illumination.
- the printed circuit board that is displayed as background has a lighter color, and therefore the luminance is likely to change greatly with variation of illumination.
- an intermediate-processing algorithm that extracts the package region of a semiconductor chip is often included. According to this intermediate-processing algorithm, for example, an image is obtained in which an extracted region is distinguished from a background region other than the extracted region based on the pixel value or the like. Then, a region of text is extracted using the obtained image.
- An example of an image in which an extracted region is distinguished from a background region other than the extracted region may be a mask image in which a background region other than an extracted region is masked.
- a mask image an extracted region is converted into a white region (that is, a region with the maximum pixel value), and a background region other than the extracted region is converted into a black region (that is, a region with the minimum pixel value).
- An intermediate output image 81b of FIG. 5 is an example of a mask image generated by the intermediate-processing algorithm described above.
- This intermediate-processing algorithm is one that extracts a region of color similar to that of a package region of a semiconductor chip from the input image 81a, for example.
- the image processing of FIG. 5 also includes, for example, a process of extracting, from the input image 81a, a region of a color similar to that of text to be extracted.
- An intermediate output image 81c of FIG. 5 is an example of an image obtained by this process. Then, in the image processing of FIG. 5 , the final output image 81d is generated by performing a logical AND between the intermediate processing image 81b and the intermediate processing image 81c.
- Another method for solving the above problem may be to generate an image processing program after specifying a region of interest (ROI) corresponding to an extraction target region on the input image in advance.
- ROI region of interest
- this method is applicable to only an image in which an extraction target region is located in a fixed position.
- Still another method may be to separately perform learning for a program that extracts a region of interest and learning for a program that performs subsequent operations.
- performing learning multiple times increases the overall time taken to generate a program.
- FIG. 6 illustrates the overview of the procedure for evaluation of an individual according to the second embodiment.
- the image processing apparatus of the present embodiment evaluates an individual with the procedure illustrated in FIG. 6 .
- a set of an input image 82a, an intermediate target image 82b, and a final target image 82c is prepared as learning data 82.
- the final target image 82c is a target image for an image that is finally output by image processing using an individual under evaluation.
- the intermediate target image 82b is a target image for an image that is output at a stage halfway through image processing using the individual under evaluation, that is, at any of the nodes of the individual excluding the final node of the individual.
- the intermediate target image 82b is a mask image that masks the region (background region) other than the extraction target region.
- an individual 74 including image filters F1 to F9 is illustrated as an example of an individual under evaluation.
- the image processing apparatus takes the input image 82a as an input, and performs image processing using the individual 74 under evaluation. Then, the image processing apparatus acquires output images that are output at the nodes each incorporating an image filter (a partial program) in the individual 74.
- the acquired output images are roughly classified as follows: intermediate output images 83a, 83b, 83c, and so on that are output at the respective nodes (intermediate nodes) excluding the final node, and a the final output image 84 that is output at the final node. That is, the image processing apparatus acquires, as the intermediate output images 83a, 83b, 83c, and so on, output images that are output from the image filters F1 to F9 excluding the image filter F9 incorporated in the final node of the individual 74. Further, the image processing apparatus acquires, as the final output image 84, an output image that is from the image filter F9 incorporated in the final node of the individual 74 (that is, the final output image of the individual 74).
- the image processing apparatus determines whether to pass the individual 74 to the next generation, based on the comparison result between the acquired final output image 84 and the final target image 82c, and the comparison result between each of the intermediate output images 83a, 83b, 83c, and so on and the intermediate target image 82b.
- the image processing apparatus compares the acquired final output image 84 with the final target image 82c, and calculates a final evaluation value 85b based on the comparison result. This calculation may be performed by the same method as that of step S14 of FIG. 2 . Thus, for example, the similarity between the final output image 84 and the final target image 82c is calculated as the final evaluation value 85b.
- the image processing apparatus compares each of the acquired intermediate output images 83a, 83b, 83c, and so on with the intermediate target image 82b, and calculates an intermediate evaluation value 85a based on the comparison result. For example, the image processing apparatus calculates the similarity between each of the intermediate output images 83a, 83b, 83c, and so on and the intermediate target image 82b, and obtains the maximum value among the calculated similarities as the intermediate evaluation value 85a.
- the image processing apparatus determines whether to pass the individual 74 to the next generation, based on the intermediate evaluation value 85a and the final evaluation value 85b. For example, the image processing apparatus determines whether to pass the individual 74 to the next generation, based on an evaluation value obtained by synthesizing the intermediate evaluation value 85a and the final evaluation value 85b at a ratio corresponding to a weight coefficient.
- an individual that generates not only a final output image close to a final target image but also an intermediate output image close to an intermediate target image is passed to the next generation.
- the final evaluation values of the parent individuals and child individuals generated from the population 61 are more likely to increase, which promotes learning. Accordingly, the time taken to complete generation of an image processing program is reduced.
- the image processing apparatus may use, as an evaluation value for determining whether to pass an individual to the next generation, a value calculated based on a weighted intermediate evaluation value and final evaluation value. For example, in the initial stage of learning, the image processing apparatus uses an evaluation value calculated with a greater weight assigned to the intermediate evaluation value. Then, as learning progresses, the image processing apparatus gradually increases the weight assigned to the final evaluation value for calculating the evaluation value. Accordingly, in the initial stage of learning, learning is performed with a focus on making the intermediate output image closer to the intermediate target image. Then, as learning progresses and the intermediate evaluation value converges, a greater focus is placed on making the final output image closer to the final target image.
- the time taken for the final evaluation value to reach the predetermined threshold is reduced. Therefore, by varying the weight as described above, the time taken to complete generation of an image processing program is reduced as a whole.
- FIG. 7 illustrates an exemplary hardware configuration of the image processing apparatus.
- An image processing apparatus 100 is implemented, for example, as a computer illustrated in FIG. 7 .
- the overall operation of the image processing apparatus 100 is controlled by a processor 101.
- the processor 101 may be a multiprocessor. Examples of the processor 101 include central processing unit (CPU), micro processing unit (MPU), digital signal processor (DSP), application specific integrated circuit (ASIC), and programmable logic device (PLD). Alternatively, the processor 101 may be a combination of two or more elements selected from CPU, MPU, DSP, ASIC, and PLD.
- a random access memory (RAM) 102 and a plurality of peripheral devices are connected to the processor 101 via a bus 109.
- the RAM 102 is used as a primary storage device of the image processing apparatus 100.
- the RAM 102 temporarily stores at least part of the operating system (OS) program and application programs that are executed by the processor 101.
- the RAM 102 also stores various types of data used for processing by the processor 101.
- the peripheral devices connected to the bus 109 include an HDD 103, a graphics processing unit 104, an input interface 105, a reader 106, a network interface 107, and a communication interface 108.
- the HDD 103 is used as a secondary storage device of the image processing apparatus 100.
- the HDD 103 stores the OS program, application programs, and various types of data.
- non-volatile storage devices such as SSD (Solid State Drive) and the like may be used as a secondary storage device.
- a display device 104a is connected to the graphics processing unit 104.
- the graphics processing unit 104 displays an image on the screen of the display device 104a in accordance with an instruction from the processor 101.
- Examples of the display device include liquid crystal display, organic electro-luminescence display, and the like.
- An input device 105a is connected to the input interface 105.
- the input interface 105 receives signals from the input device 105a, and transmits the received signals to the processor 101.
- Examples of the input device 105a include keyboard, pointing device, and the like.
- Examples of pointing devices include mouse, touch panel, tablet, touch pad, track ball, and the like.
- a portable storage medium 106a is loaded into or removed from the reader 106.
- the reader 106 reads data stored in the portable storage medium 106a, and transmits the read data to the processor 101.
- Examples of the portable storage medium 106a include optical disc, magneto-optical disk, semiconductor memory device, and the like.
- the network interface 107 transmits data to and receives data from other apparatuses via a network 107a.
- the communication interface 108 transmits data to and receives data from an external device connected thereto.
- a camera 108a is connected to the communication interface 108 as an external device.
- the communication interface 108 transmits, to the processor 101, image data transmitted from the camera 108a.
- FIG. 8 is a block diagram illustrating an exemplary configuration of the processing functions of the image processing apparatus.
- the image processing apparatus 100 includes an image acquisition unit 111, an image processing unit 112, a program generation unit 120, a program storage unit 130, an element storage unit 141, a learning data storage unit 142, a population storage unit 143, and an output image storage unit 144.
- Operations performed by the image acquisition unit 111, the image processing unit 112, and the program generation unit 120 are implemented, for example, by the processor 101 of the image processing apparatus 100 executing predetermined programs. Some of the operations performed by the image processing unit 112 are implemented, for example, by the processor 101 of the image processing apparatus 100 executing an image processing program stored in the program storage unit 130.
- the program storage unit 130, the element storage unit 141, and the learning data storage unit 142 are implemented, for example, as a storage area of the HDD 103 of the image processing apparatus 100.
- the population storage unit 143 and the output image storage unit 144 are implemented, for example, as a storage area of the RAM 102 of the image processing apparatus 100.
- the image acquisition unit 111 acquires data of a captured image from the camera 108a, and outputs the data to the program generation unit 120 or the image processing unit 112.
- the program generation unit 120 generates an image processing program by using genetic programming, and stores the generated image processing program in the program storage unit 130.
- the internal configuration of the program generation unit 120 will be described below.
- the image processing unit 112 acquires data of an image captured by the camera 108a via the image acquisition unit 111.
- the image processing unit 112 performs image processing on the data of the acquired image in accordance with an image processing program stored in the program storage unit 130.
- the processed image is displayed, for example, on the display device 104a.
- the program storage unit 130 stores the image processing program generated by the program generation unit 120.
- the element storage unit 141 stores data of elements that may be incorporated into each individual generated by the program generation unit 120. These elements are partial programs that serve as the components of the image processing program, and include, for example, various types of image filtering programs.
- the learning data storage unit 142 stores a plurality of sets of learning data, each including an input image, and its corresponding intermediate target image and final target image.
- the input image included in each set of learning data may be, for example, an image captured by the camera 108a connected to the image processing apparatus 100.
- Each of the intermediate target image and the final target image corresponding to the input image is generated by, for example, retouching the input image.
- the population storage unit 143 stores a population.
- the population includes a plurality of individuals (image processing programs), each generated by combining the elements (partial programs) stored in the element storage unit 141.
- the population storage unit 143 may store image processing programs corresponding to the respective individuals, or may store, for each individual, configuration information indicating the names of partial programs incorporated in the respective nodes of the individual, and the connection structure between the nodes. Further, the population storage unit 143 stores evaluation values calculated for the respective individuals, in association with the individuals.
- the output image storage unit 144 stores output images obtained by executing a program corresponding to an individual subjected to survival selection.
- the output images include an intermediate output image that is output at an intermediate node of the individual, and a final output image that is output at a final node.
- the program generation unit 120 includes a learning control unit 121, a program execution unit 122, and an evaluation value calculation unit 123.
- the learning control unit 121 controls the entire program generation process performed by the program generation unit 120. For example, the learning control unit 121 performs operations such as generating an initial population, evolving an individual, performing survival selection based on evaluation values and outputting a final image processing program, creating a new population with a survived individual, and so on.
- the program execution unit 122 executes an individual (image processing program), in response to an instruction from the learning control unit 121. Upon executing the individual, the program execution unit 122 outputs not only a final output image that is output at the final node of the individual, but also an intermediate output image that is output at an intermediate node of the individual, and stores the images in the output image storage unit 144.
- the evaluation value calculation unit 123 calculates an evaluation value for evaluating each individual, in response to an instruction from the learning control unit 121.
- the evaluation value that is calculated includes not only the intermediate evaluation value and the final evaluation value described above, but also a comprehensive evaluation value.
- FIG. 9 illustrates a first example of learning data.
- image processing is performed that extracts, from a package region of a semiconductor chip mounted on a printed circuit board, only the portion of text printed in the region as a white region.
- the extraction target region from which text is extracted is the package region of a semiconductor chip.
- sets of learning data 151 to 153 illustrated in FIG. 9 are used, for example.
- the learning data 151 includes an input image 151a, an intermediate target image 151b, and a final target image 151c.
- the input image 151a displays a printed circuit board.
- the intermediate target image 151b is a mask image in which a background region other than a package region of a semiconductor chip (extraction target region) in the input image 151a is masked.
- the final target image 151c is an image in which only the portion of text printed in the package region of the semiconductor chip in the input image 151a is white.
- the learning data 152 includes an input image 152a, an intermediate target image 152b, and a final target image 152c.
- the input image 152a displays a printed circuit board.
- the printed circuit board in the input image 152a may be different from the printed circuit board in the input image 151a.
- the mounting position of the semiconductor chip in the input image 152a may be different from the mounting position of the semiconductor chip in the input image 151a.
- the intermediate target image 152b is a mask image in which a background region in the input image 152a is masked.
- the final target image 152c is an image in which only the portion of text printed in the package region of the semiconductor chip in the input image 152a is white.
- the learning data 153 includes an input image 153a, an intermediate target image 153b, and a final target image 153c.
- the input image 153a displays a printed circuit board.
- the printed circuit board in the input image 153a may be different from the printed circuit boards in the input images 151a and 152a.
- the mounting position of the semiconductor chip in the input image 153a may be different from the mounting positions of the semiconductor chips in the input images 151a and 152a.
- the intermediate target image 153b is a mask image in which a background region in the input image 153a is masked.
- the final target image 153c is an image in which only the portion of text printed in the package region of the semiconductor chip in the input image 153a is white.
- FIG. 10 illustrates a second example of learning data.
- image processing is performed that extracts, from the region of a license plate attached to a vehicle traveling on the road, only the portion of text printed in the region as a white region.
- the extraction target region from which text is extracted is the license plate region.
- sets of learning data 161 to 163 illustrated in FIG. 10 are used, for example.
- the learning data 161 includes an input image 161a, an intermediate target image 161b, and a final target image 161c.
- the input image 161a displays a vehicle.
- the intermediate target image 161b is a mask image in which a background region other than a license plate region (extraction target region) in the input image 161a is masked.
- the final target image 161c is an image in which only the portion of text contained in the license plate region in the input image 161a is white.
- the learning data 162 includes an input image 162a, an intermediate target image 162b, and a final target image 162c.
- the input image 162a displays a vehicle.
- the vehicle in the input image 162a may be different from the vehicle in the input image 161a.
- the position of the license plate in the input image 162a may be different from the position of the license plate in the input image 161a.
- the intermediate target image 162b is a mask image in which a background region in the input image 162a is masked.
- the final target image 162c is an image in which only the portion of text printed in the license plate region in the input image 162a is white.
- the learning data 163 includes an input image 163a, an intermediate target image 163b, and a final target image 163c.
- the input image 163a displays a vehicle.
- the vehicle in the input image 163a may be different from the vehicles in the input images 161a and 162a.
- the position of the license plate in the input image 163a may be different from the positions of the license plates in the input images 161a and 162a.
- the intermediate target image 163b is a mask image in which a background region in the input image 163a is masked.
- the final target image 163c is an image in which only the portion of text contained in the license plate region in the input image 163a is white.
- the input images 151a, 152a, and 153a of FIG. 9 differ in the brightness of illumination on the object at the time of imaging.
- the input images 161a, 162a, and 163a of FIG. 10 differ in the distribution of light illuminating the object at the time of imaging.
- imaging conditions such as brightness and the like.
- FIGS. 11 and 12 are flowcharts illustrating an example of a procedure for the program generation process.
- the learning control unit 121 receives an input operation for specifying learning data. For example, a set of learning data to be used in this process is specified from among the sets of learning data stored in the learning data storage unit 142. In this example, n sets of learning data are used (n is an integer greater than or equal to 1).
- Step S22 The learning control unit 121 generates a plurality of initial individuals by combining the elements registered in the element storage unit 141, and stores the generated initial individuals in the population storage unit 143.
- a population generated by this operation corresponds to the population 61 of FIG. 3 , and therefore is hereinafter referred to as the "population 61".
- Step S23 An intermediate evaluation value F mid and a final evaluation value F last of each individual included in the population 61 are calculated with the following procedure.
- the learning control unit 121 selects one of the individuals included in the population 61, and causes the program execution unit 122 to execute the selected individual.
- the program execution unit 122 performs image processing on each input image included in the n sets of learning data specified in step S21, in accordance with the selected individual. In this image processing, the program execution unit 122 stores images that are output from the nodes of the selected individual, in the output image storage unit 144.
- the stored output images include an intermediate output image that is output at an intermediate node, and a final output image that is output at the final node. That is, for each of the n sets of learning data, one or more intermediate output images and one final output image are stored.
- the learning control unit 121 causes the evaluation value calculation unit 123 to calculate an intermediate evaluation value F mid and the final evaluation value F last of the selected individual.
- the evaluation value calculation unit 123 first calculates a preliminary evaluation value for each intermediate node included in the individual. More specifically, a preliminary evaluation value f(k) of a k-th intermediate node included in the individual is calculated in accordance with the following equation (1), using n intermediate output images that are output at the k-th intermediate node based on the n sets of learning data, respectively.
- V max W represents the number of pixels in the horizontal direction of the image; H represents the number of pixels in the perpendicular direction of the image; m (k,i) (x, y) represents the pixel value at coordinates (x, y) in an intermediate output image that is output at the k-th intermediate node, using an input image included in an i-th learning data; M i (x, y) represents the pixel value at coordinates (x, y) in an intermediate target image included in the i-th learning data; and V max represents the possible maximum pixel value. Note these pixel values are, for example, luminance values. According to equation (1), the preliminary evaluation value f(k) takes a value greater than or equal to 0 and less than or equal to 1.
- the evaluation value calculation unit 123 calculates a final evaluation value F last in accordance with the following equation (3), using n final output images that are output at the final node of the individual based on the n sets of learning data, respectively.
- o i (x, y) represents the pixel value at coordinates (x, y) in a final output image that is output using an input image included in the i-th learning data
- T i (x, y) represents the pixel value at coordinates (x, y) in a final target image that is included in the i-th learning data.
- the final evaluation value F last takes a value greater than or equal to 0 and less than or equal to 1.
- an intermediate evaluation value F mid and a final evaluation value F last are calculated for each individual included in the population 61.
- the evaluation value calculation unit 123 registers the calculated intermediate evaluation value F mid and the final evaluation value F last , in association with the individual, in the population storage unit 143.
- the learning control unit 121 instructs the evaluation value calculation unit 123 to calculate a weight coefficient t.
- the evaluation value calculation unit 123 calculates the weight coefficient t, based on the distribution of the intermediate evaluation values F mid of all the individuals included in the current population 61. For example, the weight coefficient t is calculated as the average value of the intermediate evaluation values F mid of all the individuals included in the population 61.
- Step S25 The learning control unit 121 randomly selects two parent individuals from among the individuals included in the population 61.
- Step S26 The learning control unit 121 performs a crossover between the two selected parent individuals to thereby generate a predetermined number of, two or more, child individuals.
- Step S27 The learning control unit 121 introduces a mutation into a node of one of the generated child individuals to replace an image filter incorporated in the original child node with another image filter registered in the element storage unit 141.
- Step S28 An intermediate evaluation value F mid and a final evaluation value F last of each child individual generated by the operations of steps S26 and S27 are calculated with the same procedure as that used for calculating the intermediate evaluation value F mid and the final evaluation value F last of the individual in step S23.
- Step S29 The learning control unit 121 compares the final evaluation value F last of each of the parent individuals selected in step S25 and the individuals generated in steps S26 and S27 with a predetermined threshold. The learning control unit 121 determines whether there is an individual whose final evaluation value F last is greater than the threshold. If there is no individual whose final evaluation value F last is greater than the threshold, the process moves to step S30. If there is an individual whose final evaluation value F last is greater than the threshold, the process moves to step S33.
- Step S30 The learning control unit 121 causes the evaluation value calculation unit 123 to calculate a comprehensive evaluation value F total of each of the parent individuals selected in step S25 and the child individuals generated in steps S26 and S27.
- the evaluation value calculation unit 123 calculates the comprehensive evaluation value F total of each of these individuals, in accordance with the following equation (4).
- F total 1 ⁇ t F mid + tF last
- Step S31 The learning control unit 121 selects the individual having the highest comprehensive evaluation value F total among those calculated in step S30 as an individual to be preserved, from among the parent individuals selected in step S25 and the child individuals generated in steps S26 and S27. Further, the learning control unit 121 selects another individual to be preserved, from among the remaining individuals. In this selection operation, for example, an individual is selected in accordance with the probabilities based on the calculated comprehensive evaluation values F total .
- Step S32 The learning control unit 121 replaces, among the individuals included in the population 61, the parent individuals selected in step S25 with the two individuals selected in step S31. Thus, a new generation of the population 61 is created. Further, the intermediate evaluation values F mid and the final evaluation values F last of the two individuals selected in step S31 are registered, in association with the individuals, in the population storage unit 143.
- At least one of the individuals of the population 61 that are replaced may be, for example, the individual having the lowest comprehensive evaluation value F total or the lowest final evaluation value F last .
- Step S33 The learning control unit 121 stores an image processing program corresponding to the individual that is determined to have a final evaluation value F last greater than the threshold in step S29, in the program storage unit 130. Then, the process ends. Note that if, in step S29, there are a plurality of individuals having a final evaluation value F last greater than the threshold, the learning control unit 121 stores an image processing program corresponding to the individual having the highest final evaluation value F last among these individuals, in the program storage unit 130.
- step S30 the comprehensive evaluation value F total of each individual to be subjected to survival selection is calculated based on the intermediate evaluation value F mid and the final evaluation value F last of the individual. Then, in step S31, an individual to be preserved is selected based on the comprehensive evaluation value F total .
- an individual to be preserved is selected based not only on the final output image that is output as the result of image processing by each individual, but also on the effectiveness of the intermediate output image that is output halfway through the image processing. Therefore, an individual a part of whose processing process is determined to be appropriate is more likely to survive in the population 61 without being eliminated.
- the maximum value among the final evaluation values F last of the individuals of the population 61 is more likely to increase. Accordingly, the learning speed is improved, and the time taken to complete generation of an image processing program is reduced.
- the weight coefficient t used for calculating the comprehensive evaluation value F total is calculated again in step S24, based on the distribution of the intermediate evaluation values F mid of the respective individuals of the population 61 of that generation. Accordingly, the comprehensive evaluation value F total varies as learning progresses.
- the weight coefficient t is calculated based on the distributions of the intermediate evaluation values F mid of the respective individuals of the population 61, the value of the weight coefficient t gradually increases as learning progresses. Therefore, upon calculating the comprehensive evaluation value F total , the synthesis ratio of the final evaluation value F last increases as learning progresses.
- survival selection of individuals is performed with a focus on the intermediate evaluation value F mid .
- survival selection of individuals is performed with a focus on the final evaluation value F last .
- the time taken for the final evaluation value F last to reach the predetermined threshold is reduced. Therefore, by varying the weight coefficient as described above, the time taken to complete generation of an image processing program is reduced as a whole.
- FIG. 13 illustrates an example of changes in final evaluation value and weight coefficient.
- FIG. 13 illustrates changes in the final evaluation value F last in the present embodiment, together with a comparative example of changes in the final evaluation value F last in the case where survival selection is performed based on the final evaluation value F last , in place of the comprehensive evaluation value F total , in step S30 of FIG. 12 .
- the final evaluation value F last indicated in FIG. 13 is the maximum value among the final evaluation values F last that are compared with the threshold in step S29 of FIG. 12 .
- the time take for the final evaluation value F last to exceed the predetermined threshold in the present embodiment is reduced to about a half compared to the comparative example.
- the weight coefficient t generally increases as the generation count of the population 61 increases.
- a third embodiment illustrates a modification of the second embodiment, in which the weight coefficient t is calculated based on the temporal progress of learning, instead of calculating the weight coefficient t based on the calculated intermediate evaluation value F mid .
- the basic configuration of an image processing apparatus of the third embodiment is the same as that of the second embodiment, and will be described using the same reference signs as those used in the second embodiment.
- FIG. 14 is a diagram for explaining a modulation table used for calculation of a weight coefficient.
- a graph 170 of FIG. 14 represents graphically the information registered in the modulation table.
- the weight coefficient t increases at three stages as a generation count g of the population 61 increases.
- the image processing apparatus 100 of the present embodiment calculates the weight coefficient t, based on the modulation table storing the corresponding relationship between the generation count g and the weight coefficient t illustrated in the graph 170.
- the method of calculating the weight coefficient t is not limited to the method using the modulation table, and may be any method as long as the weight coefficient t increases as learning progresses.
- the weight coefficient t may be calculated using a predetermined calculation formula.
- FIGS. 15 and 16 are flowcharts illustrating an example of a procedure for a program generation process according to the third embodiment. Note that, in FIGS. 15 and 16 , the same steps as those of FIGS. 11 and 12 are indicated by the same step numbers, and will not be described herein.
- FIGS. 15 and 16 is different from the process of FIGS. 11 and 12 in the following respects. Steps S21a and S21b are added between step S21 and step S22. Further, the operation of step S24 is eliminated, so that step S23 is followed by step S25. Further, steps S32a and S32b are added after step S32, so that step S32b is followed by step S25.
- Step S21a The learning control unit 121 sets the modulation table of the weight coefficient t. For example, the correspondence relationship between the generation count g and the weight coefficient t is specified by an input operation by the user.
- Step S21b The learning control unit 121 initializes the generation count g to 1, and instructs the evaluation value calculation unit 123 to set the weight coefficient t.
- the evaluation value calculation unit 123 refers to the modulation table, and sets a value of the weight coefficient t associated with the current generation number g.
- Step S32a The learning control unit 121 increments the generation count g by one.
- Step S32b The learning control unit 121 instructs the evaluation value calculation unit 123 to update the weight coefficient t.
- the evaluation value calculation unit 123 refers to the modulation table, and updates the setting value of the current weight coefficient t, using the value of the weight coefficient t associated with the current generation number g.
- step S21b may be performed at any time point after completion of step S21a and before execution of step S30. Further, the operations of steps S32a and S32b may be performed at any time point after completion of step S32 and before execution of step S30.
- the value of the weight coefficient t gradually increases as learning progresses.
- survival selection of individuals is performed with a focus on the intermediate evaluation value F mid .
- survival selection of individuals is performed with a focus on the final evaluation value F last . Accordingly, the time taken to complete generation of an image processing program is reduced.
- each of the apparatuses may be implemented on a computer.
- a program describing operations of the functions of each apparatus is provided.
- the program is executed by a computer, the above-described processing functions are implemented on the computer.
- the program describing operations of the functions may be stored in a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage device, optical disc, magneto-optical storage medium, semiconductor memory device, and the like. Examples of magnetic storage devices include hard disk drive (HDD), flexible disk (FD), magnetic tape, and the like.
- optical discs include digital versatile disc (DVD), DVD-RAM, compact disc read only memory (CD-ROM), CD-Recordable (CD-R), CD-Rewritable (CD-RW), and the like.
- magneto-optical storage media include magneto-optical disk (MO) and the like.
- the program may be stored and sold in the form of a portable storage medium such as DVD, CD-ROM, and the like, for example.
- the program may also be stored in a storage device of a server computer, and transferred from the server computer to other computers via a network.
- the computer For executing the program on a computer, the computer stores the program recorded on the portable storage medium or the program transmitted from the server computer in its storage device. Then, the computer reads the program from its storage device, and performs processing in accordance with the program. The computer may read the program directly from the portable storage medium, and execute processing in accordance with the program. Further, the computer may sequentially receive the program from a server computer connected over a network, and perform processing in accordance with the received program.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Mathematical Physics (AREA)
- Molecular Biology (AREA)
- Genetics & Genomics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Physiology (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
- Stored Programmes (AREA)
Abstract
Description
- The embodiments discussed herein relate to a program generation apparatus, a program generation method, and a generation program.
- A technology for automatically generating an image processing program that performs desired image processing, by using genetic programming, is drawing attention. This technology is designed to optimize an image processing program that is generated by combining partial programs for image processing (for example, image filtering programs), based on learning data such as pairs of an input image and an image obtained as a result of processing (a target image), by using genetic programming.
- As an example of an apparatus using genetic programming, there has been proposed a genetic processing apparatus that evolves a converter, using weight data of the current generation and weight data of the previous generations.
- PTL 1: Japanese Laid-Open Patent Publication No.
2011-14049 - NPTL 1: Shinya Aoki and Tomoharu Nagao, "ACTIT: Automatic Construction of Tree-structural Image Transformations", The Journal of The Institute of Image Information and Television Engineers, Vol. 53, No. 6, June 20, 1999, pp. 890-892
- In the process of automatically generating an image processing program by using genetic programming, the following survival selection method is used, for example. That is, an input image included in learning data is processed with a program corresponding to an individual generated in the course of learning. An output data that is output as the processing result is compared with a target image included in the learning data. Then, a determination is made as to whether to pass the individual to the next generation, based on the comparison result.
- However, a problem with this method is that an effective individual that promotes learning may be eliminated depending on the content of image processing. This problem may result in an increase in time taken to generate an image processing program.
- One aspect of the present disclosure is to provide a program generation apparatus, a program generation method, and a generation program capable of performing appropriate survival selection when generating an image processing program by using genetic programming.
- According to one embodiment, there is provided a program generation apparatus that generates a program by using genetic programming. The program generation apparatus includes a storage unit and a processing unit. The storage unit stores learning data including an input image and a first target image. The first target image indicates an image that is output halfway through a process of converting the input image into a second target image. The processing unit selects a first program from among a plurality of image processing programs each generated by combining a plurality of partial programs, generates a second program by changing a part of the partial programs included in the first program, performs image processing on the input image using the second program, determines whether to pass the second program to a next generation, based on a comparison between one or more intermediate output images and the first target image, the one or more intermediate output images being output halfway through the image processing, and replaces one of the plurality of image processing programs with the second program when the second program is determined to be passed to the next generation.
- According to another embodiment, there is provided a program generation method executed by a computer to perform the same procedure as that performed by the program generation apparatus described above.
- According to still another embodiment, there is provided a generation program that causes a computer to perform the same procedure as that performed by the program generation apparatus described above.
- According to one aspect, it is possible to perform appropriate survival selection when generating an image processing program by using genetic programming.
- The above and other objects, features and advantages of the present invention will become apparent from the following description when taken in conjunction with the accompanying drawings which illustrate preferred embodiments of the present invention by way of example.
-
- [
FIG. 1] FIG. 1 illustrates an exemplary configuration and operation of a program generation apparatus according to a first embodiment. - [
FIG. 2] FIG. 2 illustrates a comparative example of a procedure for an image processing program generation process. - [
FIG. 3] FIG. 3 illustrates an example of a crossover. - [
FIG. 4] FIG. 4 illustrates an example of a mutation. - [
FIG. 5] FIG. 5 illustrates an example of image processing. - [
FIG. 6] FIG. 6 illustrates the overview of the procedure for evaluation of an individual according to a second embodiment. - [
FIG. 7] FIG. 7 illustrates an exemplary hardware configuration of an image processing apparatus. - [
FIG. 8] FIG. 8 is a block diagram illustrating an exemplary configuration of the processing functions of the image processing apparatus. - [
FIG. 9] FIG. 9 illustrates a first example of learning data. - [
FIG. 10] FIG. 10 illustrates a second example of learning data. - [
FIG. 11] FIG. 11 is a flowchart (part 1) illustrating an example of a procedure for a program generation process. - [
FIG. 12] FIG. 12 is a flowchart (part 2) illustrating the example of a procedure for a program generation process. - [
FIG. 13] FIG. 13 illustrates an example of changes in final evaluation value and weight coefficient. - [
FIG. 14] FIG. 14 is a diagram for explaining a modulation table used for calculation of a weight coefficient. - [
FIG. 15] FIG. 15 is a flowchart (part 1) illustrating an example of a procedure for a program generation process according to a third embodiment. - [
FIG. 16] FIG. 16 is a flowchart (part 2) illustrating the example of a procedure for a program generation process according to the third embodiment. - Several embodiments will be described below with reference to the accompanying drawings.
-
FIG. 1 illustrates an exemplary configuration and operation of a program generation apparatus according to a first embodiment. Aprogram generation apparatus 1 is an apparatus that generates an image processing program by using genetic programming. - The
program generation apparatus 1 includes a storage unit 1a and aprocessing unit 1b. The storage unit 1a is implemented, for example, as a volatile storage device such as random access memory (RAM) and the like, or a non-volatile storage device such as hard disk drive (HDD), flash memory, and the like. Theprocessing unit 1b is, for example, a processor. - The storage unit 1a
stores learning data 10. Thelearning data 10 includes an input image 11, afirst target image 12, and asecond target image 13. Thesecond target image 13 is a target image for an image that is output as the processing result of image processing performed on the input image 11. Meanwhile, thefirst target image 12 is a target image for an image that is output at a certain step halfway through image processing performed for converting the input image 11 into thesecond target image 13. For example, in the case of image processing that performs certain processing on a specific image region of the input image 11, thefirst target image 12 may be an image in which a specific image region of the image is distinguished from the other image region (background region). The image regions may be distinguished by, for example, assigning pixel values, such as setting the pixel value of the specific image region to a maximum value and setting the pixel value of the background image region to a minimum value (0). - Note that the storage unit 1a may store a plurality of sets of
learning data 10. - Further, the storage unit 1a may store a
program group 20. Theprogram group 20 is data used for operations performed by theprocessing unit 1b, and includes a plurality ofimage processing programs image processing programs program group 20 is generated by combining a plurality of partial programs. A partial program is a program component for performing image processing such as image filtering and the like. Every time a new generation of theprogram group 20 is created, theprogram generation apparatus 1 preferentially preserves an image processing program with a high fitness value in theprogram group 20, and thereby generates an image processing program that performs desired image processing. - The
processing unit 1b selects an image processing program from the program group 20 (step S1). Here, for example, animage processing program 21 is selected. Then, theprocessing unit 1b generates animage processing program 21a by changing a part of the partial programs included in the selectedimage processing program 21. This processing process is a process of evolving the image processing program 21 (step S2). This evolution process involves, for example, a crossover between theimage processing program 21 and another image processing program selected from theprogram group 20, a mutation of theimage processing program 21 or the program resulting from the crossover, and the like. - As the result of the evolution process, the
image processing program 21a is generated by combining partial programs P1 to P3, for example. Note that inFIG. 1 , "In" indicates an input part of theimage processing program 21a, and "Out" indicates an output part of theimage processing program 21a. - Then, the
processing unit 1b performs image processing on the input image 11, using theimage processing program 21a (step S3). Theprocessing unit 1b outputs intermediate output images halfway through this image processing. For example, the intermediate output images are images that are respectively output as the processing results of the partial programs P1 and P2, other than the partial program P3 incorporated in the final stage, from among the partial programs P1 to P3 of theimage processing program 21a. In the example ofFIG. 1 , anintermediate output image 31 is output as the processing result of the partial program P1, and anintermediate output image 32 is output as the processing result of the partial program P2. - Then, the
processing unit 1b determines whether to pass the generatedimage processing program 21a to the next generation (step S4). The operation of step S4 includes an operation (step S4a) of comparing theintermediate output images first target image 12. In step S4a, for example, the similarity between the images is output as the comparison result. Further, as in the example ofFIG. 1 , in the case where the plurality ofintermediate output images first target image 12 in step S4a, the maximum value among the similarities of theintermediate output images first target image 12 is output as the comparison result, for example. Based on the comparison result of step S4a, theprocessing unit 1b determines whether to pass theimage processing program 21a to the next generation. - Note than in step S4, the determination may be made based not only on the comparison result of step S4a, but also on the comparison result between a final output image, which is output as the execution result of the
image processing program 21a, and thesecond target image 13. - If, in step S4, the
image processing program 21a is determined to be passed to the next generation, theprocessing unit 1b replaces one of theimage processing programs program group 20 with theimage processing program 21a (step S5). Thus, a new generation of theprogram group 20 is created. - One method of determining as to whether to pass the
image processing program 21a to the next generation may be to make a determination based on a comparison between the final output image, which is output as the result of image processing by theimage processing program 21a, and thesecond target image 13. However, with this method, even in the case where an image close to the desired image such as thefirst target image 12 is generated halfway through the image processing, if the final image is not similar to thesecond target image 13, theimage processing program 21a is eliminated and not passed to the next generation of theprogram group 20. That is, an effective image processing program that is likely to contribute to promoting learning is eliminated. This may result in an increase in time taken to complete generation of an image processing program. - Meanwhile, the comparison result in step S4a represents the index indicating how close the intermediate output image that is output halfway through execution of the
image processing program 21a is to the desired image. Since a determination is made based on this index in step S4, when theimage processing program 21a is likely to be able to output an appropriate image halfway through execution of theimage processing program 21a, theimage processing program 21a is passed to the next generation of theprogram group 20 without being eliminated. Thus, appropriate survival selection is performed such that an image processing program whose processing process is specified as being appropriate is more likely to be passed to the next generation. - Then, the operations of steps S1 to S5 are repeatedly executed using the new generation of the
program group 20 that is created by the procedure described above. Thus, the speed of increase in the similarity between images indicated by the comparison result of step S4a is increased, so that the learning speed is improved. This reduces the time taken for the maximum value among the fitness values of the image processing programs included in theprogram group 20 and the image processing programs generated from these image processing programs through the evolution process to reach a predetermined threshold. That is, the time taken to generate an image processing program that implements desired processing is reduced. - Next, an image processing apparatus according to a second embodiment will be described. The image processing apparatus according to the second embodiment has the processing functions similar to those of the
program generation apparatus 1 ofFIG. 1 , and also has a function of performing image processing by executing an image processing program generated by these functions. - In the following, a comparative example of a basic procedure for an image processing program generation process by using genetic programming will first be described with reference to
FIGS. 2 to 4 . A problem with the comparative example will then be described with reference toFIG. 5 . After that, the image processing apparatus according to the second embodiment will be described. -
FIG. 2 illustrates a comparative example of a procedure for an image processing program generation process. - Before starting an image processing program generation process, at least one set of learning
data 50 is prepared. The learningdata 50 includes aninput image 51 and atarget image 52 that is obtained by performing image processing on theinput image 51. Theinput image 51 may be obtained, for example, by capturing an image of an object with a camera. - In the image processing program generation process using genetic programming, an individual is formed by combining one or more partial programs. For example, as illustrated in the upper left of
FIG. 2 , an individual is defined by a tree structure. - A plurality of partial programs that may be incorporated into an individual are also prepared in advance. In the following description, image filters are used as an example of partial programs that are incorporated into an individual. However, the partial programs are not limited to image filters, and programs for performing other types of image processing may be used. Note that in the upper left of
FIG. 2 , "F" represents an image filter; "I" represents an input terminal; and "O" represents an output terminal. - The image processing program generation process using genetic programming is performed, for example, in the following manner. First, a
population 61 including a plurality of initial individuals is generated (step S11). Image filters are randomly selected from among a plurality of image filters prepared in advance and incorporated into the nodes of each initial individual. Then, two parent individuals are randomly extracted from the generated population 61 (step S12). - Subsequently, the two parent individuals undergo an evolution process, so that two or more child individuals are generated (step S13). In the evolution process, a crossover operation and a mutation operation are performed on the two parent individuals. Three or more child individuals may be generated by performing different crossover operations and mutation operations on the two parent individuals.
- Then, a fitness value is calculated for each of the child individuals generated through the evolution process and the original parent individuals (step S14). In this operation, image processing using each of these individuals is performed on the
input image 51 of the learningdata 50. Then, an image obtained from the image processing is compared with acorresponding target image 52 to thereby calculate a fitness value of the individual. In the case where there are a plurality of sets of learningdata 50, the average of fitness values that are obtained using the plurality of sets of learningdata 50 is calculated for each individual. - If the fitness value of any of the individuals is greater than or equal to a predetermined threshold, the individual is output as a final image processing program. Then, the image processing program generation process ends. On the other hand, if the fitness values of all the individuals are less than the predetermined threshold, survival selection is performed on a
population 62 including the generated child individuals and the two original parent individuals (step S15). In the survival selection, an individual with the highest fitness value is selected from thepopulation 62. Further, one individual is selected from among the remaining individuals of thepopulation 62 by using a predetermined method. For example, one individual is selected from among the remaining individuals in accordance with the probabilities based on their fitness values. - The two individuals selected by the survival selection replace two individuals included in the population 61 (step S16). For example, the two individuals selected by the survival selection replace the two individuals extracted as the parent individuals, among the individuals included in the
population 61. Thus, the individuals included in thepopulation 61 are changed to individuals of the next generation. Then, the same procedure is repeated until an individual with a fitness value greater than or equal to the predetermined threshold is produced. -
FIG. 3 illustrates an example of a crossover. In the example ofFIG. 3 , a crossover is performed betweenparent individuals - The parent individual 71a includes image filters F1, F2, F3, and F4, and the parent individual 72a includes image filters F2, F3, F5, and F6. Assume here that the node of the image filter F2 in the parent individual 71a and the node of the image filter F5 in the parent individual 72a are selected to be subjected to a crossover.
- In a crossover operation, for example, not only a selected node but also nodes at levels lower than the level of the selected node are subjected to a crossover. Accordingly, in the example of
FIG. 3 , a section of the parent individual 71a including "the image filters F2 and F1; a node of an input terminal connected to one end of the image filter F2; and a node of an input terminal connected to the image filter F1" is swapped with a section of the parent individual 72a including "the image filter F5; and a node of an input terminal connected to the image filter F5". As a result, the crossover produces the child individual 71b including the image filters F3, F4, and F5, and the child individual 72b including single image filters F1, F2, and F4 and two image filters F3. -
FIG. 4 illustrates an example of a mutation. InFIG. 4 , an individual 73a includes the image filters F3, F4, and F5. The individual 73a may be a parent individual extracted from thepopulation 61, or an individual produced by performing a crossover on the parent individuals extracted from thepopulation 61. - Assume here that the node of the image filter F3 in the individual 73a is selected as a site of mutation, and an image filter F7 is selected as a replacing image filter in the mutation operation. Note that the replacing image filter in the mutation operation is randomly selected from among a plurality of image filters prepared in advance. The mutation produces a child individual 73b including the image filters F4, F5, and F7.
- Conceivable uses of an image processing program generated by the above procedure include, for example, achieving a desired effect by performing image processing on an image of a product in the factory automation (FA) field. Specifically, the image processing program may be used to perform image processing on an image of the appearance of a product to thereby extract sites with defects, extract points for alignment, or recognize characters printed on a specific component.
- In such usage, reconstruction of an image processing algorithm is often needed due to alterations and improvements made to a product used as an imaging object, the associated changes in the imaging environment, and the like. Therefore, there is a demand for easy construction of an image processing algorithm. There is also a demand for construction of an image processing algorithm robust to changes in the imaging environment, such as changes in lighting conditions, variations in the shape, position, and orientation of the imaging object, and so on.
- With use of genetic programming, it is possible to easily generate an image processing program usable for such applications, by simply preparing in advance the
input image 51 and thetarget image 52 corresponding thereto. Further, it is possible to automatically generate an image processing algorithm robust to changes in the imaging environment, by preparing in advance a plurality of pairs of theinput image 51 and the target image 52 (a plurality of sets of learning data 50) whose imaging environments are different from each other. -
FIG. 5 illustrates an example of image processing. The image processing includes, for example, performing certain processing on a specific region of an input image. As an example of such processing,FIG. 5 illustrates a process of extracting, from a package region of a semiconductor chip mounted on a printed circuit board, only the portion of text printed in that region as a region of a specific color (for example, white). In afinal output image 81d that is finally output as the result of image processing, only the portion of text "ABC123" printed in a package region of a semiconductor chip is extracted as a white region. - Note that the image processing of
FIG. 5 is performed as preprocessing for recognizing text in a specific region. Further, other examples of the process of performing certain processing on a specific region may include a process of extracting, from a region where a specific component is mounted, a portion of symbols or patterns drawn on the component as a region of a specific color. Other examples may also include a process of extracting, from a specific region in an image, only a region where a component is mounted as a region of a specific color, in order to detect the position and inclination of the component in the specific region. - In the following description, a specific region from which text, symbols, or patterns are extracted is referred to as an "extraction target region".
- As for input images used in the image processing described above, the brightness of the object at the time of imaging may greatly vary from image to image. Moreover, the degree of variation in the brightness of the extraction target region in the image often differs from the degree of variation in the brightness of the background region other than the extraction target region. This is because the cause of variation in brightness may differ between the extraction target region and the background region. Possible causes include, for example, the difference in light reflectance and color tendency between the extraction target region and the background region, and the difference in how light is received from the surroundings due to the difference in height between the extraction target region and the background region.
- Accordingly, it is highly difficult to create a preprocessing algorithm that reduces the effect of variation in brightness by performing processing on the entire image. In order to reduce the effect of variation in brightness, it is preferable to perform different types of preprocessing on the extraction target region and the background region. Therefore, in order to improve the robustness in the image processing described above, it is important to include a process of separating the extraction target region and the background region.
- In an
input image 81a illustrated inFIG. 5 , the package region of the semiconductor chip is black, and therefore the luminance changes little with variation of illumination. However, the printed circuit board that is displayed as background has a lighter color, and therefore the luminance is likely to change greatly with variation of illumination. In the case where a programmer creates an image processing program that obtains thefinal output image 81d from theinput image 81a containing such an object, an intermediate-processing algorithm that extracts the package region of a semiconductor chip is often included. According to this intermediate-processing algorithm, for example, an image is obtained in which an extracted region is distinguished from a background region other than the extracted region based on the pixel value or the like. Then, a region of text is extracted using the obtained image. - An example of an image in which an extracted region is distinguished from a background region other than the extracted region may be a mask image in which a background region other than an extracted region is masked. In a mask image, an extracted region is converted into a white region (that is, a region with the maximum pixel value), and a background region other than the extracted region is converted into a black region (that is, a region with the minimum pixel value).
- An
intermediate output image 81b ofFIG. 5 is an example of a mask image generated by the intermediate-processing algorithm described above. This intermediate-processing algorithm is one that extracts a region of color similar to that of a package region of a semiconductor chip from theinput image 81a, for example. Meanwhile, the image processing ofFIG. 5 also includes, for example, a process of extracting, from theinput image 81a, a region of a color similar to that of text to be extracted. Anintermediate output image 81c ofFIG. 5 is an example of an image obtained by this process. Then, in the image processing ofFIG. 5 , thefinal output image 81d is generated by performing a logical AND between theintermediate processing image 81b and theintermediate processing image 81c. - In the case where an image processing program that performs the image processing described above is generated by the procedure illustrated in
FIG. 2 , evaluation of an individual for survival selection (that is, calculation of fitness) is performed based on comparison between thefinal output image 81d and its corresponding target image. In this case, even if effective processing that extracts an extraction target region is performed halfway through the image processing using an individual, the individual may be eliminated in accordance with the evaluation result based on the final output image, regardless of whether such effective processing is performed. This causes a problem that an individual that has produced an effective output halfway through the image processing is not passed to the next generation. - This indicates that if survival selection of individuals is performed using only the evaluation result based on the final output image, it takes time for the value indicating the evaluation result to converge, which contributes to an increase in time taken to generate an image processing program. In other words, time taken to generate an image processing program is likely to be reduced by performing survival selection using the evaluation result that is based not only on the final output image, but also on the intermediate output image that is obtained halfway through the image processing.
- Another method for solving the above problem may be to generate an image processing program after specifying a region of interest (ROI) corresponding to an extraction target region on the input image in advance. However, this method is applicable to only an image in which an extraction target region is located in a fixed position. Still another method may be to separately perform learning for a program that extracts a region of interest and learning for a program that performs subsequent operations. However, performing learning multiple times increases the overall time taken to generate a program.
-
FIG. 6 illustrates the overview of the procedure for evaluation of an individual according to the second embodiment. The image processing apparatus of the present embodiment evaluates an individual with the procedure illustrated inFIG. 6 . - In the present embodiment, a set of an
input image 82a, anintermediate target image 82b, and afinal target image 82c is prepared as learningdata 82. Thefinal target image 82c is a target image for an image that is finally output by image processing using an individual under evaluation. Theintermediate target image 82b is a target image for an image that is output at a stage halfway through image processing using the individual under evaluation, that is, at any of the nodes of the individual excluding the final node of the individual. In the example ofFIG. 6 , theintermediate target image 82b is a mask image that masks the region (background region) other than the extraction target region. - In
FIG. 6 , an individual 74 including image filters F1 to F9 is illustrated as an example of an individual under evaluation. The image processing apparatus according to the present embodiment takes theinput image 82a as an input, and performs image processing using the individual 74 under evaluation. Then, the image processing apparatus acquires output images that are output at the nodes each incorporating an image filter (a partial program) in the individual 74. - The acquired output images are roughly classified as follows:
intermediate output images final output image 84 that is output at the final node. That is, the image processing apparatus acquires, as theintermediate output images final output image 84, an output image that is from the image filter F9 incorporated in the final node of the individual 74 (that is, the final output image of the individual 74). - The image processing apparatus determines whether to pass the individual 74 to the next generation, based on the comparison result between the acquired
final output image 84 and thefinal target image 82c, and the comparison result between each of theintermediate output images intermediate target image 82b. - More specifically, the image processing apparatus compares the acquired
final output image 84 with thefinal target image 82c, and calculates afinal evaluation value 85b based on the comparison result. This calculation may be performed by the same method as that of step S14 ofFIG. 2 . Thus, for example, the similarity between thefinal output image 84 and thefinal target image 82c is calculated as thefinal evaluation value 85b. - Further, the image processing apparatus compares each of the acquired
intermediate output images intermediate target image 82b, and calculates anintermediate evaluation value 85a based on the comparison result. For example, the image processing apparatus calculates the similarity between each of theintermediate output images intermediate target image 82b, and obtains the maximum value among the calculated similarities as theintermediate evaluation value 85a. - Then, the image processing apparatus determines whether to pass the individual 74 to the next generation, based on the
intermediate evaluation value 85a and thefinal evaluation value 85b. For example, the image processing apparatus determines whether to pass the individual 74 to the next generation, based on an evaluation value obtained by synthesizing theintermediate evaluation value 85a and thefinal evaluation value 85b at a ratio corresponding to a weight coefficient. - With the procedure described above, an individual that generates not only a final output image close to a final target image but also an intermediate output image close to an intermediate target image is passed to the next generation. As the number of individuals whose intermediate output image is determined to be close to the intermediate target image increases as described above in the population 61 (see
FIG. 2 ), the final evaluation values of the parent individuals and child individuals generated from thepopulation 61 are more likely to increase, which promotes learning. Accordingly, the time taken to complete generation of an image processing program is reduced. - The image processing apparatus may use, as an evaluation value for determining whether to pass an individual to the next generation, a value calculated based on a weighted intermediate evaluation value and final evaluation value. For example, in the initial stage of learning, the image processing apparatus uses an evaluation value calculated with a greater weight assigned to the intermediate evaluation value. Then, as learning progresses, the image processing apparatus gradually increases the weight assigned to the final evaluation value for calculating the evaluation value. Accordingly, in the initial stage of learning, learning is performed with a focus on making the intermediate output image closer to the intermediate target image. Then, as learning progresses and the intermediate evaluation value converges, a greater focus is placed on making the final output image closer to the final target image. As the number of individuals having a high intermediate evaluation value increases in the
population 61, the time taken for the final evaluation value to reach the predetermined threshold is reduced. Therefore, by varying the weight as described above, the time taken to complete generation of an image processing program is reduced as a whole. - In the following, the image processing apparatus according to the second embodiment will be described in detail.
-
FIG. 7 illustrates an exemplary hardware configuration of the image processing apparatus. Animage processing apparatus 100 is implemented, for example, as a computer illustrated inFIG. 7 . - The overall operation of the
image processing apparatus 100 is controlled by aprocessor 101. Theprocessor 101 may be a multiprocessor. Examples of theprocessor 101 include central processing unit (CPU), micro processing unit (MPU), digital signal processor (DSP), application specific integrated circuit (ASIC), and programmable logic device (PLD). Alternatively, theprocessor 101 may be a combination of two or more elements selected from CPU, MPU, DSP, ASIC, and PLD. - A random access memory (RAM) 102 and a plurality of peripheral devices are connected to the
processor 101 via abus 109. - The
RAM 102 is used as a primary storage device of theimage processing apparatus 100. TheRAM 102 temporarily stores at least part of the operating system (OS) program and application programs that are executed by theprocessor 101. TheRAM 102 also stores various types of data used for processing by theprocessor 101. - The peripheral devices connected to the
bus 109 include anHDD 103, agraphics processing unit 104, aninput interface 105, areader 106, anetwork interface 107, and acommunication interface 108. - The
HDD 103 is used as a secondary storage device of theimage processing apparatus 100. TheHDD 103 stores the OS program, application programs, and various types of data. Note that other types of non-volatile storage devices such as SSD (Solid State Drive) and the like may be used as a secondary storage device. - A
display device 104a is connected to thegraphics processing unit 104. Thegraphics processing unit 104 displays an image on the screen of thedisplay device 104a in accordance with an instruction from theprocessor 101. Examples of the display device include liquid crystal display, organic electro-luminescence display, and the like. - An
input device 105a is connected to theinput interface 105. Theinput interface 105 receives signals from theinput device 105a, and transmits the received signals to theprocessor 101. Examples of theinput device 105a include keyboard, pointing device, and the like. Examples of pointing devices include mouse, touch panel, tablet, touch pad, track ball, and the like. - A
portable storage medium 106a is loaded into or removed from thereader 106. Thereader 106 reads data stored in theportable storage medium 106a, and transmits the read data to theprocessor 101. Examples of theportable storage medium 106a include optical disc, magneto-optical disk, semiconductor memory device, and the like. - The
network interface 107 transmits data to and receives data from other apparatuses via anetwork 107a. - The
communication interface 108 transmits data to and receives data from an external device connected thereto. In this embodiment, acamera 108a is connected to thecommunication interface 108 as an external device. Thus, thecommunication interface 108 transmits, to theprocessor 101, image data transmitted from thecamera 108a. - With the hardware configuration described above, it is possible to implement the processing functions of the
image processing apparatus 100. -
FIG. 8 is a block diagram illustrating an exemplary configuration of the processing functions of the image processing apparatus. Theimage processing apparatus 100 includes animage acquisition unit 111, animage processing unit 112, aprogram generation unit 120, aprogram storage unit 130, anelement storage unit 141, a learningdata storage unit 142, apopulation storage unit 143, and an outputimage storage unit 144. - Operations performed by the
image acquisition unit 111, theimage processing unit 112, and theprogram generation unit 120 are implemented, for example, by theprocessor 101 of theimage processing apparatus 100 executing predetermined programs. Some of the operations performed by theimage processing unit 112 are implemented, for example, by theprocessor 101 of theimage processing apparatus 100 executing an image processing program stored in theprogram storage unit 130. Theprogram storage unit 130, theelement storage unit 141, and the learningdata storage unit 142 are implemented, for example, as a storage area of theHDD 103 of theimage processing apparatus 100. Thepopulation storage unit 143 and the outputimage storage unit 144 are implemented, for example, as a storage area of theRAM 102 of theimage processing apparatus 100. - The
image acquisition unit 111 acquires data of a captured image from thecamera 108a, and outputs the data to theprogram generation unit 120 or theimage processing unit 112. - The
program generation unit 120 generates an image processing program by using genetic programming, and stores the generated image processing program in theprogram storage unit 130. The internal configuration of theprogram generation unit 120 will be described below. - The
image processing unit 112 acquires data of an image captured by thecamera 108a via theimage acquisition unit 111. Theimage processing unit 112 performs image processing on the data of the acquired image in accordance with an image processing program stored in theprogram storage unit 130. The processed image is displayed, for example, on thedisplay device 104a. - The
program storage unit 130 stores the image processing program generated by theprogram generation unit 120. - The
element storage unit 141 stores data of elements that may be incorporated into each individual generated by theprogram generation unit 120. These elements are partial programs that serve as the components of the image processing program, and include, for example, various types of image filtering programs. - The learning
data storage unit 142 stores a plurality of sets of learning data, each including an input image, and its corresponding intermediate target image and final target image. The input image included in each set of learning data may be, for example, an image captured by thecamera 108a connected to theimage processing apparatus 100. Each of the intermediate target image and the final target image corresponding to the input image is generated by, for example, retouching the input image. - The
population storage unit 143 stores a population. The population includes a plurality of individuals (image processing programs), each generated by combining the elements (partial programs) stored in theelement storage unit 141. Note that thepopulation storage unit 143 may store image processing programs corresponding to the respective individuals, or may store, for each individual, configuration information indicating the names of partial programs incorporated in the respective nodes of the individual, and the connection structure between the nodes. Further, thepopulation storage unit 143 stores evaluation values calculated for the respective individuals, in association with the individuals. - The output
image storage unit 144 stores output images obtained by executing a program corresponding to an individual subjected to survival selection. The output images include an intermediate output image that is output at an intermediate node of the individual, and a final output image that is output at a final node. - The
program generation unit 120 includes alearning control unit 121, aprogram execution unit 122, and an evaluationvalue calculation unit 123. - The
learning control unit 121 controls the entire program generation process performed by theprogram generation unit 120. For example, thelearning control unit 121 performs operations such as generating an initial population, evolving an individual, performing survival selection based on evaluation values and outputting a final image processing program, creating a new population with a survived individual, and so on. - The
program execution unit 122 executes an individual (image processing program), in response to an instruction from thelearning control unit 121. Upon executing the individual, theprogram execution unit 122 outputs not only a final output image that is output at the final node of the individual, but also an intermediate output image that is output at an intermediate node of the individual, and stores the images in the outputimage storage unit 144. - The evaluation
value calculation unit 123 calculates an evaluation value for evaluating each individual, in response to an instruction from thelearning control unit 121. The evaluation value that is calculated includes not only the intermediate evaluation value and the final evaluation value described above, but also a comprehensive evaluation value. - In the following, an example of learning data will be described.
-
FIG. 9 illustrates a first example of learning data. InFIG. 9 , for example, image processing is performed that extracts, from a package region of a semiconductor chip mounted on a printed circuit board, only the portion of text printed in the region as a white region. In this case, the extraction target region from which text is extracted is the package region of a semiconductor chip. In order to generate an image processing program for implementing such image processing using theimage processing apparatus 100, sets of learningdata 151 to 153 illustrated inFIG. 9 are used, for example. - The learning
data 151 includes aninput image 151a, anintermediate target image 151b, and afinal target image 151c. Theinput image 151a displays a printed circuit board. Theintermediate target image 151b is a mask image in which a background region other than a package region of a semiconductor chip (extraction target region) in theinput image 151a is masked. Thefinal target image 151c is an image in which only the portion of text printed in the package region of the semiconductor chip in theinput image 151a is white. - The learning
data 152 includes aninput image 152a, anintermediate target image 152b, and afinal target image 152c. Theinput image 152a displays a printed circuit board. The printed circuit board in theinput image 152a may be different from the printed circuit board in theinput image 151a. Further, the mounting position of the semiconductor chip in theinput image 152a may be different from the mounting position of the semiconductor chip in theinput image 151a. Theintermediate target image 152b is a mask image in which a background region in theinput image 152a is masked. Thefinal target image 152c is an image in which only the portion of text printed in the package region of the semiconductor chip in theinput image 152a is white. - The learning
data 153 includes aninput image 153a, anintermediate target image 153b, and afinal target image 153c. Theinput image 153a displays a printed circuit board. The printed circuit board in theinput image 153a may be different from the printed circuit boards in theinput images input image 153a may be different from the mounting positions of the semiconductor chips in theinput images intermediate target image 153b is a mask image in which a background region in theinput image 153a is masked. Thefinal target image 153c is an image in which only the portion of text printed in the package region of the semiconductor chip in theinput image 153a is white. -
FIG. 10 illustrates a second example of learning data. InFIG. 10 , for example, image processing is performed that extracts, from the region of a license plate attached to a vehicle traveling on the road, only the portion of text printed in the region as a white region. In this case, the extraction target region from which text is extracted is the license plate region. In order to generate an image processing program for implementing such image processing using theimage processing apparatus 100, sets of learningdata 161 to 163 illustrated inFIG. 10 are used, for example. - The learning
data 161 includes aninput image 161a, anintermediate target image 161b, and afinal target image 161c. Theinput image 161a displays a vehicle. Theintermediate target image 161b is a mask image in which a background region other than a license plate region (extraction target region) in theinput image 161a is masked. Thefinal target image 161c is an image in which only the portion of text contained in the license plate region in theinput image 161a is white. - The learning
data 162 includes aninput image 162a, anintermediate target image 162b, and afinal target image 162c. Theinput image 162a displays a vehicle. The vehicle in theinput image 162a may be different from the vehicle in theinput image 161a. Further, the position of the license plate in theinput image 162a may be different from the position of the license plate in theinput image 161a. Theintermediate target image 162b is a mask image in which a background region in theinput image 162a is masked. Thefinal target image 162c is an image in which only the portion of text printed in the license plate region in theinput image 162a is white. - The learning
data 163 includes aninput image 163a, anintermediate target image 163b, and afinal target image 163c. Theinput image 163a displays a vehicle. The vehicle in theinput image 163a may be different from the vehicles in theinput images input image 163a may be different from the positions of the license plates in theinput images intermediate target image 163b is a mask image in which a background region in theinput image 163a is masked. Thefinal target image 163c is an image in which only the portion of text contained in the license plate region in theinput image 163a is white. - As in the examples of
FIGS. 9 and10 described above, when generating an image processing program, it is preferable to use a plurality of sets of learning data that differ in the position of the extraction target region. Thus, even when image processing is performed on captured images that differ in the position of the extraction target region, it is possible to create an image processing program capable of efficiently achieving the desired result. - Further, the
input images FIG. 9 differ in the brightness of illumination on the object at the time of imaging. Theinput images FIG. 10 differ in the distribution of light illuminating the object at the time of imaging. As in these examples, when generating an image processing program, it is preferable to use a plurality of sets of learning data including input images that differ in imaging conditions such as brightness and the like. Thus, it is possible to generate an image processing program capable of stably achieving the desired result even in the case where the imaging conditions vary. - In the following, the image processing program generation process performed by the
image processing apparatus 100 will be described in detail with reference to a flowchart. -
FIGS. 11 and12 are flowcharts illustrating an example of a procedure for the program generation process. - (Step S21) The
learning control unit 121 receives an input operation for specifying learning data. For example, a set of learning data to be used in this process is specified from among the sets of learning data stored in the learningdata storage unit 142. In this example, n sets of learning data are used (n is an integer greater than or equal to 1). - (Step S22) The
learning control unit 121 generates a plurality of initial individuals by combining the elements registered in theelement storage unit 141, and stores the generated initial individuals in thepopulation storage unit 143. A population generated by this operation corresponds to thepopulation 61 ofFIG. 3 , and therefore is hereinafter referred to as the "population 61". - (Step S23) An intermediate evaluation value Fmid and a final evaluation value Flast of each individual included in the
population 61 are calculated with the following procedure. - The
learning control unit 121 selects one of the individuals included in thepopulation 61, and causes theprogram execution unit 122 to execute the selected individual. Theprogram execution unit 122 performs image processing on each input image included in the n sets of learning data specified in step S21, in accordance with the selected individual. In this image processing, theprogram execution unit 122 stores images that are output from the nodes of the selected individual, in the outputimage storage unit 144. The stored output images include an intermediate output image that is output at an intermediate node, and a final output image that is output at the final node. That is, for each of the n sets of learning data, one or more intermediate output images and one final output image are stored. - The
learning control unit 121 causes the evaluationvalue calculation unit 123 to calculate an intermediate evaluation value Fmid and the final evaluation value Flast of the selected individual. The evaluationvalue calculation unit 123 first calculates a preliminary evaluation value for each intermediate node included in the individual. More specifically, a preliminary evaluation value f(k) of a k-th intermediate node included in the individual is calculated in accordance with the following equation (1), using n intermediate output images that are output at the k-th intermediate node based on the n sets of learning data, respectively. - The evaluation
value calculation unit 123 calculates preliminary evaluation values f(k) of all the intermediate nodes included in the individual in the manner described above. Then, the evaluationvalue calculation unit 123 calculates an intermediate evaluation value Fmid corresponding to the individual in accordance with the following equation (2) . More specifically, the intermediate evaluation value Fmid corresponding to the individual is calculated as the maximum value among the preliminary evaluation values f(k) calculated for the individual. - Further, the evaluation
value calculation unit 123 calculates a final evaluation value Flast in accordance with the following equation (3), using n final output images that are output at the final node of the individual based on the n sets of learning data, respectively. - With the procedure described above, an intermediate evaluation value Fmid and a final evaluation value Flast are calculated for each individual included in the
population 61. The evaluationvalue calculation unit 123 registers the calculated intermediate evaluation value Fmid and the final evaluation value Flast, in association with the individual, in thepopulation storage unit 143. - (Step S24) The
learning control unit 121 instructs the evaluationvalue calculation unit 123 to calculate a weight coefficient t. The evaluationvalue calculation unit 123 calculates the weight coefficient t, based on the distribution of the intermediate evaluation values Fmid of all the individuals included in thecurrent population 61. For example, the weight coefficient t is calculated as the average value of the intermediate evaluation values Fmid of all the individuals included in thepopulation 61. - (Step S25) The
learning control unit 121 randomly selects two parent individuals from among the individuals included in thepopulation 61. - (Step S26) The
learning control unit 121 performs a crossover between the two selected parent individuals to thereby generate a predetermined number of, two or more, child individuals. - (Step S27) The
learning control unit 121 introduces a mutation into a node of one of the generated child individuals to replace an image filter incorporated in the original child node with another image filter registered in theelement storage unit 141. - (Step S28) An intermediate evaluation value Fmid and a final evaluation value Flast of each child individual generated by the operations of steps S26 and S27 are calculated with the same procedure as that used for calculating the intermediate evaluation value Fmid and the final evaluation value Flast of the individual in step S23.
- (Step S29) The
learning control unit 121 compares the final evaluation value Flast of each of the parent individuals selected in step S25 and the individuals generated in steps S26 and S27 with a predetermined threshold. Thelearning control unit 121 determines whether there is an individual whose final evaluation value Flast is greater than the threshold. If there is no individual whose final evaluation value Flast is greater than the threshold, the process moves to step S30. If there is an individual whose final evaluation value Flast is greater than the threshold, the process moves to step S33. - (Step S30) The
learning control unit 121 causes the evaluationvalue calculation unit 123 to calculate a comprehensive evaluation value Ftotal of each of the parent individuals selected in step S25 and the child individuals generated in steps S26 and S27. The evaluationvalue calculation unit 123 calculates the comprehensive evaluation value Ftotal of each of these individuals, in accordance with the following equation (4). - (Step S31) The
learning control unit 121 selects the individual having the highest comprehensive evaluation value Ftotal among those calculated in step S30 as an individual to be preserved, from among the parent individuals selected in step S25 and the child individuals generated in steps S26 and S27. Further, thelearning control unit 121 selects another individual to be preserved, from among the remaining individuals. In this selection operation, for example, an individual is selected in accordance with the probabilities based on the calculated comprehensive evaluation values Ftotal. - (Step S32) The
learning control unit 121 replaces, among the individuals included in thepopulation 61, the parent individuals selected in step S25 with the two individuals selected in step S31. Thus, a new generation of thepopulation 61 is created. Further, the intermediate evaluation values Fmid and the final evaluation values Flast of the two individuals selected in step S31 are registered, in association with the individuals, in thepopulation storage unit 143. - Note that at least one of the individuals of the
population 61 that are replaced may be, for example, the individual having the lowest comprehensive evaluation value Ftotal or the lowest final evaluation value Flast. - (Step S33) The
learning control unit 121 stores an image processing program corresponding to the individual that is determined to have a final evaluation value Flast greater than the threshold in step S29, in theprogram storage unit 130. Then, the process ends. Note that if, in step S29, there are a plurality of individuals having a final evaluation value Flast greater than the threshold, thelearning control unit 121 stores an image processing program corresponding to the individual having the highest final evaluation value Flast among these individuals, in theprogram storage unit 130. - According to the process illustrated in
FIGS. 11 and12 , in step S30, the comprehensive evaluation value Ftotal of each individual to be subjected to survival selection is calculated based on the intermediate evaluation value Fmid and the final evaluation value Flast of the individual. Then, in step S31, an individual to be preserved is selected based on the comprehensive evaluation value Ftotal. Thus, an individual to be preserved is selected based not only on the final output image that is output as the result of image processing by each individual, but also on the effectiveness of the intermediate output image that is output halfway through the image processing. Therefore, an individual a part of whose processing process is determined to be appropriate is more likely to survive in thepopulation 61 without being eliminated. Then, as the number of such individuals increases in thepopulation 61, the maximum value among the final evaluation values Flast of the individuals of thepopulation 61 is more likely to increase. Accordingly, the learning speed is improved, and the time taken to complete generation of an image processing program is reduced. - Further, after a new generation of the
population 61 is created in step S32, the weight coefficient t used for calculating the comprehensive evaluation value Ftotal is calculated again in step S24, based on the distribution of the intermediate evaluation values Fmid of the respective individuals of thepopulation 61 of that generation. Accordingly, the comprehensive evaluation value Ftotal varies as learning progresses. - Since the weight coefficient t is calculated based on the distributions of the intermediate evaluation values Fmid of the respective individuals of the
population 61, the value of the weight coefficient t gradually increases as learning progresses. Therefore, upon calculating the comprehensive evaluation value Ftotal, the synthesis ratio of the final evaluation value Flast increases as learning progresses. Thus, in the initial stage of learning, survival selection of individuals is performed with a focus on the intermediate evaluation value Fmid. Then, as learning progresses, survival selection of individuals is performed with a focus on the final evaluation value Flast. As the number of individuals having a high intermediate evaluation value Fmid increases in thepopulation 61, the time taken for the final evaluation value Flast to reach the predetermined threshold is reduced. Therefore, by varying the weight coefficient as described above, the time taken to complete generation of an image processing program is reduced as a whole. -
FIG. 13 illustrates an example of changes in final evaluation value and weight coefficient.FIG. 13 illustrates changes in the final evaluation value Flast in the present embodiment, together with a comparative example of changes in the final evaluation value Flast in the case where survival selection is performed based on the final evaluation value Flast, in place of the comprehensive evaluation value Ftotal, in step S30 ofFIG. 12 . Note that the final evaluation value Flast indicated inFIG. 13 is the maximum value among the final evaluation values Flast that are compared with the threshold in step S29 ofFIG. 12 . - According to the example of
FIG. 13 , the time take for the final evaluation value Flast to exceed the predetermined threshold in the present embodiment is reduced to about a half compared to the comparative example. Further, the weight coefficient t generally increases as the generation count of thepopulation 61 increases. - A third embodiment illustrates a modification of the second embodiment, in which the weight coefficient t is calculated based on the temporal progress of learning, instead of calculating the weight coefficient t based on the calculated intermediate evaluation value Fmid. Note that the basic configuration of an image processing apparatus of the third embodiment is the same as that of the second embodiment, and will be described using the same reference signs as those used in the second embodiment.
-
FIG. 14 is a diagram for explaining a modulation table used for calculation of a weight coefficient. Agraph 170 ofFIG. 14 represents graphically the information registered in the modulation table. In the example of thegraph 170, the weight coefficient t increases at three stages as a generation count g of thepopulation 61 increases. Theimage processing apparatus 100 of the present embodiment calculates the weight coefficient t, based on the modulation table storing the corresponding relationship between the generation count g and the weight coefficient t illustrated in thegraph 170. - Note that the method of calculating the weight coefficient t is not limited to the method using the modulation table, and may be any method as long as the weight coefficient t increases as learning progresses. For example, the weight coefficient t may be calculated using a predetermined calculation formula.
-
FIGS. 15 and16 are flowcharts illustrating an example of a procedure for a program generation process according to the third embodiment. Note that, inFIGS. 15 and16 , the same steps as those ofFIGS. 11 and12 are indicated by the same step numbers, and will not be described herein. - The process of
FIGS. 15 and16 is different from the process ofFIGS. 11 and12 in the following respects. Steps S21a and S21b are added between step S21 and step S22. Further, the operation of step S24 is eliminated, so that step S23 is followed by step S25. Further, steps S32a and S32b are added after step S32, so that step S32b is followed by step S25. - (Step S21a) The
learning control unit 121 sets the modulation table of the weight coefficient t. For example, the correspondence relationship between the generation count g and the weight coefficient t is specified by an input operation by the user. - (Step S21b) The
learning control unit 121 initializes the generation count g to 1, and instructs the evaluationvalue calculation unit 123 to set the weight coefficient t. The evaluationvalue calculation unit 123 refers to the modulation table, and sets a value of the weight coefficient t associated with the current generation number g. - (Step S32a) The
learning control unit 121 increments the generation count g by one. - (Step S32b) The
learning control unit 121 instructs the evaluationvalue calculation unit 123 to update the weight coefficient t. The evaluationvalue calculation unit 123 refers to the modulation table, and updates the setting value of the current weight coefficient t, using the value of the weight coefficient t associated with the current generation number g. - The operation of step S21b may be performed at any time point after completion of step S21a and before execution of step S30. Further, the operations of steps S32a and S32b may be performed at any time point after completion of step S32 and before execution of step S30.
- According to the third embodiment described above, the value of the weight coefficient t gradually increases as learning progresses. Thus, in the initial stage of learning, survival selection of individuals is performed with a focus on the intermediate evaluation value Fmid. Then, as learning progresses, survival selection of individuals is performed with a focus on the final evaluation value Flast. Accordingly, the time taken to complete generation of an image processing program is reduced.
- Note that the processing functions of each of the apparatuses (the
program generation apparatus 1 and the image processing apparatus 100) of the above embodiments may be implemented on a computer. In this case, a program describing operations of the functions of each apparatus is provided. When the program is executed by a computer, the above-described processing functions are implemented on the computer. The program describing operations of the functions may be stored in a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage device, optical disc, magneto-optical storage medium, semiconductor memory device, and the like. Examples of magnetic storage devices include hard disk drive (HDD), flexible disk (FD), magnetic tape, and the like. Examples of optical discs include digital versatile disc (DVD), DVD-RAM, compact disc read only memory (CD-ROM), CD-Recordable (CD-R), CD-Rewritable (CD-RW), and the like. Examples of magneto-optical storage media include magneto-optical disk (MO) and the like. - For distributing the program, the program may be stored and sold in the form of a portable storage medium such as DVD, CD-ROM, and the like, for example. The program may also be stored in a storage device of a server computer, and transferred from the server computer to other computers via a network.
- For executing the program on a computer, the computer stores the program recorded on the portable storage medium or the program transmitted from the server computer in its storage device. Then, the computer reads the program from its storage device, and performs processing in accordance with the program. The computer may read the program directly from the portable storage medium, and execute processing in accordance with the program. Further, the computer may sequentially receive the program from a server computer connected over a network, and perform processing in accordance with the received program.
- The above just illustrates the principle of the present disclosure. Further, various alterations and modifications can be made by those skilled in the art, and the present disclosure is not limited to the configuration illustrated and described above and application examples thereof, but all variant examples and equivalents thereof are assumed to be within the scope of the present disclosure according to the appended claims and their equivalents.
-
- 1:
- program generation apparatus
- 1a:
- storage unit
- 1b:
- processing unit
- 10:
- learning data
- 11:
- input image
- 12:
- first target image
- 13:
- second target image
- 20:
- program group
- 21, 21a, 22, 23:
- image processing program
- 31, 32:
- intermediate output image
- S1, S2, S3, S4, S4a, S5:
- step
Claims (7)
- A program generation apparatus that generates a program by using genetic programming, the program generation apparatus comprising:a storage unit for storing learning data including an input image and a first target image, the first target image indicating an image that is output halfway through a process of converting the input image into a second target image; anda processing unit for selecting a first program from among a plurality of image processing programs each generated by combining a plurality of partial programs, generating a second program by changing a part of the partial programs included in the first program, performing image processing on the input image using the second program, determining whether to pass the second program to a next generation, based on a comparison between one or more intermediate output images and the first target image, the one or more intermediate output images being output halfway through the image processing, and replacing one of the plurality of image processing programs with the second program when the second program is determined to be passed to the next generation.
- The program generation apparatus according to claim 1, wherein:the performing of the image processing includes executing non-final partial programs from among the partial programs included in the second program, the non-final partial programs being incorporated in positions other than a final stage, and outputting the intermediate output images for the respective non-final partial programs; andthe determining includes calculating evaluation values for the respective non-final partial programs, based on a comparison between each of the intermediate output images that are output for the respective non-final partial programs and the first target image, and determining whether to pass the second program to the next generation, based on a maximum value among the evaluation values.
- The program generation apparatus according to claim 1, wherein:the processing unit further performs processes on the input image using the respective image processing programs, and calculates a weight coefficient based on a comparison between each of images that are output halfway through the respective processes and the first target image; andin the determining, the processing unit calculates a first evaluation value based on the comparison between the one or more intermediate output images and the first target image, calculates a second evaluation value based on a comparison between a final output image and the second target image, the final output image being output as a result of the image processing, and determines whether to pass the second program to the next generation, based on a third evaluation value obtained by synthesizing the first evaluation value and the second evaluation value at a ratio corresponding to the weight coefficient.
- The program generation apparatus according to claim 1, wherein:the processing unit further calculates a weight coefficient, based on a number of times of generation change of a population that includes the plurality of image processing programs as individuals of a current generation; andin the determining, the processing unit calculates a first evaluation value based on the comparison between the one or more intermediate output images and the first target image, calculates a second evaluation value based on a comparison between a final output image and the second target image, the final output image being output as a result of the image processing, and determines whether to pass the second program to the next generation, based on a third evaluation value obtained by synthesizing the first evaluation value and the second evaluation value at a ratio corresponding to the weight coefficient.
- The program generation apparatus according to any one of claims 1 to 4, wherein:the first target image is an image obtained by distinguishing between a first image region on which specific processing is performed and a second image region other than the first image region in the input image; andthe second target image is an image obtained by performing the specific processing on the first image region of the input image.
- A program generation method, executed by a computer, for generating a program by using genetic programming, the program generation method comprising:selecting a first program from among a plurality of image processing programs each generated by combining a plurality of partial programs;generating a second program by changing a part of the partial programs included in the first program;performing image processing on an input image using the second program;determining whether to pass the second program to a next generation, based on a comparison between one or more intermediate output images and a first target image, the one or more intermediate output images being output halfway through the image processing, the first target image indicating an image that is output halfway through a process of converting the input image into a second target image; andreplacing one of the plurality of image processing programs with the second program when the second program is determined to be passed to the next generation.
- A computer program for generating a program by using genetic programming, the computer program causing a computer to perform a procedure comprising:selecting a first program from among a plurality of image processing programs each generated by combining a plurality of partial programs;generating a second program by changing a part of the partial programs included in the first program;performing image processing on an input image using the second program;determining whether to pass the second program to a next generation, based on a comparison between one or more intermediate output images and a first target image, the one or more intermediate output images being output halfway through the image processing, the first target image indicating an image that is output halfway through a process of converting the input image into a second target image; andreplacing one of the plurality of image processing programs with the second program when the second program is determined to be passed to the next generation.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2015/068371 WO2016208037A1 (en) | 2015-06-25 | 2015-06-25 | Program generating device, program generating method, and generating program |
Publications (3)
Publication Number | Publication Date |
---|---|
EP3316184A1 true EP3316184A1 (en) | 2018-05-02 |
EP3316184A4 EP3316184A4 (en) | 2018-07-18 |
EP3316184B1 EP3316184B1 (en) | 2020-03-11 |
Family
ID=57586632
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15896359.5A Active EP3316184B1 (en) | 2015-06-25 | 2015-06-25 | Program generating device, program generating method, and generating program |
Country Status (5)
Country | Link |
---|---|
US (1) | US10489710B2 (en) |
EP (1) | EP3316184B1 (en) |
JP (1) | JP6468356B2 (en) |
CN (1) | CN107636698B (en) |
WO (1) | WO2016208037A1 (en) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10838699B2 (en) | 2017-01-18 | 2020-11-17 | Oracle International Corporation | Generating data mappings for user interface screens and screen components for an application |
JP6663873B2 (en) * | 2017-02-22 | 2020-03-13 | 株式会社日立製作所 | Automatic program generation system and automatic program generation method |
US10489126B2 (en) * | 2018-02-12 | 2019-11-26 | Oracle International Corporation | Automated code generation |
JP7028317B2 (en) * | 2018-05-18 | 2022-03-02 | 富士通株式会社 | Information processing equipment, information processing methods and information processing programs |
US10936912B2 (en) * | 2018-11-01 | 2021-03-02 | International Business Machines Corporation | Image classification using a mask image and neural networks |
CN113168368B (en) * | 2018-11-28 | 2023-09-29 | 株式会社特拉斯特技术 | Programming device and recording medium |
JP7427337B2 (en) * | 2020-04-03 | 2024-02-05 | 株式会社ディスコ | Wafer inspection method |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1830320A4 (en) * | 2004-12-24 | 2010-10-20 | Nat Univ Corp Yokohama Nat Uni | Image processor |
JP4766030B2 (en) * | 2007-10-11 | 2011-09-07 | 富士ゼロックス株式会社 | Image processing apparatus and image processing program |
WO2009139161A1 (en) | 2008-05-15 | 2009-11-19 | 株式会社ニコン | Image processing device, image processing method, processing device, processing method, and program |
JP5461419B2 (en) * | 2008-10-27 | 2014-04-02 | 日本電信電話株式会社 | Pixel predicted value generation procedure automatic generation method, image encoding method, image decoding method, apparatus thereof, program thereof, and recording medium on which these programs are recorded |
JP2010232739A (en) * | 2009-03-25 | 2010-10-14 | Fuji Xerox Co Ltd | Image processing apparatus, image forming apparatus and program |
US20100277774A1 (en) * | 2009-05-04 | 2010-11-04 | Certifi Media Inc. | Image quality indicator responsive to image processing |
JP5313037B2 (en) * | 2009-05-11 | 2013-10-09 | パナソニック株式会社 | Electronic camera, image processing apparatus, and image processing method |
JP5310298B2 (en) * | 2009-06-24 | 2013-10-09 | 富士ゼロックス株式会社 | Image processing apparatus, image forming system, and program |
JP5359622B2 (en) * | 2009-07-03 | 2013-12-04 | 株式会社ニコン | Genetic processing apparatus, genetic processing method, and genetic processing program |
JP5088395B2 (en) * | 2010-04-15 | 2012-12-05 | 株式会社ニコン | Electronic camera |
US9171264B2 (en) * | 2010-12-15 | 2015-10-27 | Microsoft Technology Licensing, Llc | Parallel processing machine learning decision tree training |
JP6103243B2 (en) * | 2011-11-18 | 2017-03-29 | 日本電気株式会社 | Local feature quantity extraction device, local feature quantity extraction method, and program |
JP2014068273A (en) * | 2012-09-26 | 2014-04-17 | Olympus Imaging Corp | Image editing device, image editing method, and program |
JP6102947B2 (en) | 2012-12-28 | 2017-03-29 | 富士通株式会社 | Image processing apparatus and feature detection method |
EP2806374B1 (en) * | 2013-05-24 | 2022-07-06 | Tata Consultancy Services Limited | Method and system for automatic selection of one or more image processing algorithm |
JP6179224B2 (en) * | 2013-07-02 | 2017-08-16 | 富士通株式会社 | Image processing filter creation apparatus and method |
US9448771B2 (en) * | 2014-10-17 | 2016-09-20 | Duelight Llc | System, computer program product, and method for generating a lightweight source code for implementing an image processing pipeline |
WO2015194421A1 (en) * | 2014-06-16 | 2015-12-23 | オリンパス株式会社 | Medical treatment system and image processing setting method for same |
WO2015194006A1 (en) * | 2014-06-19 | 2015-12-23 | 富士通株式会社 | Program generation device, program generation method, and program |
CN104317556B (en) * | 2014-10-22 | 2018-03-16 | 华为技术有限公司 | A kind of streaming application upgrade method, main controlled node and stream calculation system |
-
2015
- 2015-06-25 JP JP2017524525A patent/JP6468356B2/en active Active
- 2015-06-25 EP EP15896359.5A patent/EP3316184B1/en active Active
- 2015-06-25 CN CN201580080754.5A patent/CN107636698B/en active Active
- 2015-06-25 WO PCT/JP2015/068371 patent/WO2016208037A1/en unknown
-
2017
- 2017-11-02 US US15/801,842 patent/US10489710B2/en active Active
Also Published As
Publication number | Publication date |
---|---|
CN107636698B (en) | 2021-10-19 |
CN107636698A (en) | 2018-01-26 |
EP3316184A4 (en) | 2018-07-18 |
JPWO2016208037A1 (en) | 2018-03-15 |
US10489710B2 (en) | 2019-11-26 |
US20180144249A1 (en) | 2018-05-24 |
JP6468356B2 (en) | 2019-02-13 |
EP3316184B1 (en) | 2020-03-11 |
WO2016208037A1 (en) | 2016-12-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3316184B1 (en) | Program generating device, program generating method, and generating program | |
US9697583B2 (en) | Image processing apparatus, image processing method, and computer-readable recording medium | |
US10303447B2 (en) | Program generating apparatus and method therefor | |
JP2017033529A (en) | Image recognition method, image recognition device and program | |
JP2017010475A (en) | Program generation device, program generation method, and generated program | |
CN110582783B (en) | Training device, image recognition device, training method, and computer-readable information storage medium | |
JP6577397B2 (en) | Image analysis apparatus, image analysis method, image analysis program, and image analysis system | |
JP6989450B2 (en) | Image analysis device, image analysis method and program | |
Sharma et al. | Implementation of CNN on Zynq based FPGA for Real-time Object Detection | |
US20220044147A1 (en) | Teaching data extending device, teaching data extending method, and program | |
JP2017162069A (en) | Optimization method, optimization device, program and image processing apparatus | |
CN112967180A (en) | Training method for generating countermeasure network, and image style conversion method and device | |
JP6208018B2 (en) | Image recognition algorithm combination selection device | |
JP2012048624A (en) | Learning device, method and program | |
CN111066061A (en) | Information processing apparatus, information processing method, and information processing program | |
CN109145991B (en) | Image group generation method, image group generation device and electronic equipment | |
JP2011014051A (en) | Generating device, generating method, and generation program | |
US11182650B2 (en) | Information processing apparatus to generate a next generation image processing program in genetic programming, control method, and non-transitory computer-readable storage medium for storage program | |
JP6331914B2 (en) | Algorithm generating apparatus, algorithm generating method and algorithm generating computer program | |
JPWO2007013425A1 (en) | Automatic image processing system | |
JP6633267B2 (en) | Dimension reduction device, method and program | |
JP2011014047A (en) | Image processing apparatus, image processing method, and image processing program | |
WO2017056320A1 (en) | Program generation device, program generation method and generation program | |
WO2024122356A1 (en) | Information processing device, information processing method, and program | |
KR100925146B1 (en) | Caption Extraction System and Control Method thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20171114 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602015048836 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: G06N0003120000 Ipc: G06F0008360000 |
|
A4 | Supplementary search report drawn up and despatched |
Effective date: 20180615 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06F 9/44 20180101ALI20180611BHEP Ipc: G06K 9/62 20060101ALI20180611BHEP Ipc: G06T 7/00 20170101ALI20180611BHEP Ipc: G06N 3/12 20060101ALI20180611BHEP Ipc: G06F 8/36 20180101AFI20180611BHEP Ipc: G06K 9/00 20060101ALI20180611BHEP Ipc: G06T 5/20 20060101ALI20180611BHEP |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20191002 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1244010 Country of ref document: AT Kind code of ref document: T Effective date: 20200315 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602015048836 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200611 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20200311 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200611 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200612 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200805 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200711 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1244010 Country of ref document: AT Kind code of ref document: T Effective date: 20200311 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602015048836 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602015048836 Country of ref document: DE Representative=s name: HL KEMPNER PATENTANWAELTE, SOLICITORS (ENGLAND, DE Ref country code: DE Ref legal event code: R082 Ref document number: 602015048836 Country of ref document: DE Representative=s name: HL KEMPNER PATENTANWALT, RECHTSANWALT, SOLICIT, DE |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
26N | No opposition filed |
Effective date: 20201214 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200625 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20200630 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200630 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200625 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200630 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200630 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20200311 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230510 Year of fee payment: 9 Ref country code: DE Payment date: 20230502 Year of fee payment: 9 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20230504 Year of fee payment: 9 |