WO2018186279A1 - Image processing device, image processing program, and recording medium - Google Patents

Image processing device, image processing program, and recording medium Download PDF

Info

Publication number
WO2018186279A1
WO2018186279A1 PCT/JP2018/013260 JP2018013260W WO2018186279A1 WO 2018186279 A1 WO2018186279 A1 WO 2018186279A1 JP 2018013260 W JP2018013260 W JP 2018013260W WO 2018186279 A1 WO2018186279 A1 WO 2018186279A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
attention
image processing
complementary
unit
Prior art date
Application number
PCT/JP2018/013260
Other languages
French (fr)
Japanese (ja)
Inventor
恭平 池田
山本　智幸
Original Assignee
シャープ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by シャープ株式会社 filed Critical シャープ株式会社
Publication of WO2018186279A1 publication Critical patent/WO2018186279A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/387—Composing, repositioning or otherwise geometrically modifying originals

Definitions

  • Patent Document 1 Patent Document 1
  • Patent Document 2 Patent Document 2
  • Japanese Patent Publication Japanese Patent Laid-Open No. 2006-301932 (published on November 7, 2006)” Japanese Patent Publication “Japanese Unexamined Patent Publication No. 2015-135540 (May 6, 2015)”
  • the non-target object is an object that hides the target area and does not need to be focused. What is unfocused can vary depending on the situation. For example, let us consider a case in which an operation is imaged using a camera placed behind the surgeon. At this time, if the surgeon wants to observe the surgical field from the viewpoint of the surgeon, the surgical field is the attention area, and the surgeon is the non-target object. On the other hand, if the surgeon wants to observe the surgeon's movements from the assistant's perspective, the surgeon becomes a region of interest rather than a non-attention object. There may also be an object (for example, a camera used for imaging) that is always desired to be set as a non-target object regardless of the situation.
  • an object for example, a camera used for imaging
  • the image to be generated can change depending on which object in the image is set as the non-target object.
  • image generation processing for example, every time a new non-attention object is selected, if an image that makes the non-attention object transparent is generated from scratch, it takes time to generate the image, so the image display is delayed. Occurs.
  • an image corresponding to each of a plurality of non-attention objects, such as a transparent non-interest object is generated in advance for each of the plurality of non-interest objects and all combinations of the plurality of non-interest objects, If stored, the amount of image data becomes enormous. Also, the processing settings (for example, the transparency of the non-attention object) cannot be changed later.
  • An object of one aspect of the present invention is to realize a technology that enables high-speed image generation while suppressing an increase in the amount of data to be stored in a process for generating an image in which a non-target object is made transparent. To do.
  • an image processing apparatus refers to attention image data including a complementary difference image generated in advance, and determines whether or not an object in a reference image is a non-attention object.
  • a non-target object selection unit that determines whether or not, a complementary difference image processing unit that performs image processing on the complementary difference image included in the target image data based on display conditions, and the complementary difference image and the reference image are combined
  • the image processing apparatus includes an attention image generation unit that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted.
  • an image processing system including a first image processing device and a second image processing device, wherein the first image processing device includes: The image processing device generates, for each of the plurality of reference images, a complementary difference image that is used to complement a non-target object included in the reference image, and generates attention image data including the generated complementary difference image The second image processing device selects whether or not the object in the reference image is a non-target object with reference to the target image data generated by the target image data generation unit.
  • a complementary differential image processing unit that performs image processing on the complementary differential image included in the target image data based on display conditions, and the complementary differential image and the reference image, Standard image Transparency of the non-interest of the inner is provided with a target image generator to obtain a target image that has been adjusted, the.
  • FIG. 1 is a schematic block diagram of an image processing apparatus according to Embodiment 1 of the present invention. It is a schematic block diagram of the attention image data generation part of the image processing device concerning Embodiment 1 of the present invention.
  • FIG. 6 illustrates a captured image of an input image according to one embodiment of the present invention. It is a figure explaining the complementary difference image of 1 aspect of this invention. It is a figure explaining the complementary difference image of 1 aspect of this invention.
  • It is a schematic block diagram of the attention image composition part of the image processing device concerning Embodiment 1 of the present invention. It is a figure explaining selection of the non-attention object by the image processing apparatus non-attention object selection part concerning Embodiment 1 of the present invention.
  • Embodiment 1 An embodiment of the present invention will be described with reference to FIGS.
  • the image processing system 1 includes an attention image data generation unit 30, an attention image synthesis unit 50, and a storage unit 20.
  • the attention image data generation unit 30 generates attention image data.
  • the attention image data includes a reference image described later, accompanying information of the reference image, a complementary difference image corresponding to the reference image, accompanying information of the complementary difference image, and three-dimensional information of the reference image.
  • the attention image data generation unit 30 outputs the generated attention image data to the storage unit 20.
  • the storage unit 20 stores the acquired attention image data.
  • the attention image synthesis unit 50 synthesizes the attention image using the attention image data acquired from the storage unit 20.
  • the attention image data generation unit 30, the attention image synthesis unit 50, and the storage unit 20 may be configured to be realized in an integrated image processing apparatus, or in a separate image processing apparatus. It is good also as a structure to implement
  • the first image processing device including the attention image data generation unit 30 and the storage unit 20 is disposed in the transmission station, and the second image processing device including the attention image synthesis unit 50 is disposed in the user's home. It may be configured.
  • each image processing apparatus includes a data transmission / reception unit, and the data transmission / reception unit
  • the image processing devices transmit and receive image data to each other. The same applies to each embodiment described later.
  • the attention image data generation unit 30 includes an acquisition unit 32, a reference image selection unit 34, an attention region complement data generation unit 36, a three-dimensional information acquisition unit 38, and a multiplexing unit 40.
  • the acquisition unit 32 acquires an input image group including a plurality of input images.
  • the input image is an image obtained by capturing an image of a certain scene or imaging target from various positions and angles.
  • the input image is not necessarily limited to an image captured by a general camera.
  • an image captured by a 360 degree camera may be included.
  • an image captured by an actual camera is not necessarily used as the input image.
  • the input image may be a virtual camera image.
  • the virtual camera image is, for example, an image synthesized from an image captured by an actual camera.
  • the virtual camera image may be an image drawn by computer graphics.
  • the acquisition unit 32 acquires camera parameters corresponding to each of the plurality of input images.
  • the camera parameters include at least information indicating the direction of the camera, information indicating the position of the camera, information indicating the angle of view of the camera, and information indicating the resolution of the image.
  • the acquired camera parameters are attached to the corresponding input image. In other words, there is always a corresponding camera parameter in the input image.
  • FIG. 3 shows an example of a situation where an input image is captured.
  • the attention area 500 is captured by the actual camera 100 and the actual camera 200.
  • the virtual camera 300 image is synthesized from the captured image of the camera 100 and the captured image of the camera 200.
  • the input image group can include an image of the attention area 500 captured by the camera 100, an image of the attention area 500 captured by the camera 200, and an image of the virtual camera 300.
  • the images of the camera 100, the camera 200, and the camera 300 include the worker 400 as an object that can be a non-target object.
  • the reference image selection unit 34 selects an image to be subjected to a complementing process related to a non-target object from the input image group acquired by the acquisition unit 32 and sets it as a reference image.
  • the reference image selection unit 34 can select a plurality of reference images. The selection of the reference image may be performed, for example, by the user selecting, or all input images included in the input image group may be selected as the reference image.
  • the attention area complementation data generation unit 36 to be described later may determine whether attention area complementation data can be generated, and select an input image that can generate attention area complementation data as a reference image.
  • the reference image selection unit 34 outputs the reference image to the attention area complement data generation unit 36 and the multiplexing unit 40.
  • the reference image selection unit 34 attaches camera parameters corresponding to the input image selected as the reference image to the reference image.
  • the camera position included in the camera parameters is referred to as a reference viewpoint position.
  • the reference viewpoint position is the position of the camera at the time of imaging if the reference image is an image captured by an actual camera.
  • the reference viewpoint position is a virtual viewpoint position if the reference image is an image synthesized from an image captured by an actual camera.
  • the attention area complement data generation unit 36 generates attention area complement data.
  • the attention area complement data is data used to complement a non-attention object shown in the reference image.
  • the term “complement” refers to processing for generating an image in which a non-target object is made transparent or translucent. In other words, complement refers to a process of generating an image in which the transparency of a non-target object is adjusted.
  • the attention area complement data generation unit 36 When there are a plurality of reference images, the attention area complement data generation unit 36 generates corresponding attention area complement data for each reference image. In the present embodiment, a complementary difference image is created as attention area complementary data.
  • the attention area complement data generation unit 36 first sets a non-target object candidate that is an object that can be selected as a non-target object in the non-target object selection unit 54.
  • the attention area complement data generation unit 36 first generates an image that makes the non-target object candidate transparent, in other words, a complementary image that is an image in which the transparency of the non-target object candidate is adjusted.
  • the method for synthesizing the complementary image is not limited.
  • a complementary image may be generated by applying a viewpoint conversion process to an input image captured from a position different from the reference viewpoint position.
  • the selection of a non-target object candidate may be performed by the user's selection, or all objects that can generate a complementary image may be selected as a non-target object candidate. Further, the correlation between a plurality of input images may be calculated, and an object existing in a region having a low correlation may be selected as a non-target object candidate.
  • the attention area complementary data generation unit 36 generates a complementary difference image by taking the difference between the complementary image and the reference image.
  • the attention area complement data generation unit 36 generates a complementary difference image by extracting an area having a pixel value equal to or less than a threshold value from the difference image between the reference image and the corresponding complement image.
  • the threshold value may be set arbitrarily.
  • the region-of-interest supplement data generation unit 36 extracts a complemented difference image from the complemented image so as to surround the region where the non-target object candidate exists.
  • a complementary difference image is generated separately for each non-target object candidate.
  • non-attention object candidate and the “non-attention object” may be simply described as “non-attention object”.
  • FIG. 4 is an example of a reference image and is represented as “reference image A”.
  • FIG. 4C is an example of a reference image and is represented as “reference image B”.
  • the reference image A and the reference image B are images obtained by capturing the hatched area from different quasi-viewpoint positions.
  • the reference image A includes a white cloud that is a non-target object in an area starting from the coordinates (x1, y1).
  • start point the upper left point in the target area
  • FIG. 4B An example of the complementary difference image corresponding to the reference image A is shown in FIG. 4B and is represented as “complementary difference image 1”.
  • the complementary differential image 1 shown in FIG. 4B includes the above-described part of the gray region that is shielded by the white cloud in FIG.
  • the gray region in the drawing shown in FIG. 4B is an image that is not shielded by a white cloud that is a non-target object.
  • the complementary difference image 1 includes information indicating that the corresponding reference image is “reference image A”, information indicating that the complementary image coordinate information indicating the corresponding position on the reference image A is (x1, y1), And information indicating that the name of the non-attention object is “baiyun”.
  • 4C includes a gray cloud that is a non-attention object in a region starting from coordinates (x2, y2).
  • An example of the complementary difference image corresponding to the reference image B is shown in FIG. 4D and is represented as “complementary difference image 2”.
  • the lower left part of the hatched area is shielded by a gray cloud that is a non-target object.
  • the complementary difference image 2 shown in (d) of FIG. 4 includes the part of the shaded area shielded by the gray clouds in (c) of FIG.
  • the hatched area in the drawing shown in FIG. 4D is an image that is not shielded by a gray cloud that is a non-target object.
  • the complementary difference image 2 includes information indicating that the corresponding reference image is “reference image B”, information indicating that the complementary image coordinate information indicating the corresponding position on the reference image B is (x2, y2), And information indicating that the name of the non-attention object is “gray cloud”.
  • the complementary difference image has alpha channel information composed of alpha values representing the opacity of each pixel.
  • the complementary difference image is smaller than the reference image. Therefore, even when there are a plurality of complementary difference images, the generation process can be performed with a relatively light process, and the amount of data to be stored can be suppressed.
  • FIG. 5 a specific example of a complementary difference image generation process is shown.
  • (A) of FIG. 5 is an example of a reference image, and includes a book end 600a selected as a non-target object and a medicine bottle 600b.
  • the complementary image with respect to the reference image is an image in which a non-attention object becomes transparent as shown in FIG.
  • the attention area complement data generation unit 36 is selected from the reference image shown in FIG. 5A and the complement image shown in FIG. 5B as the non-target object shown in FIG.
  • the complementary difference image 1 corresponding to the medicine bottle 600b and the complementary difference image 2 corresponding to the book end 600a selected as the non-target object shown in FIG. 5D are generated.
  • the 3D information acquisition unit 38 calculates 3D information of the captured situation.
  • a typical example of the three-dimensional information is a depth map.
  • the depth map is information indicating the depth value of each region of the reference image.
  • Each region of the reference image is, for example, a pixel unit of the reference image.
  • the depth value included in the depth map is based on the reference viewpoint position (origin). Further, it is assumed that the depth direction of each region of the depth map is parallel and not radial.
  • the depth map may be calculated by stereo matching. That is, a depth map of the reference image may be calculated from a plurality of input images including the reference image and camera parameters of the input image. Note that the depth map may be calculated using three or more input images. Further, the depth map may be measured using a distance sensor such as an infrared sensor, for example.
  • a depth map when all non-attention objects are made transparent a depth map when all non-attention objects are shown.
  • the former is referred to as “a depth map that does not include a non-target object”, and the latter is referred to as a “depth map that includes a non-target object”.
  • the multiplexing unit 40 multiplexes the reference image, the complementary difference image, and the three-dimensional information.
  • the reference image and the complementary difference image are multiplexed together with information attached to each.
  • the attention image synthesis unit 50 includes a demultiplexing unit 52, a non-target object selection unit 54, a complementary difference image processing unit 56, a display method acquisition unit 58, and a attention image generation unit 60. .
  • the demultiplexing unit 52 acquires the target image data from the storage unit 20, and demultiplexes the acquired target image data.
  • the demultiplexing unit 52 outputs the reference image included in the target image data and the accompanying information to the target image generating unit 60. Further, the three-dimensional information and the attention area complement data included in the attention image data are output to the non-attention object selection unit 54.
  • the non-target object selection unit 54 selects at least one of the objects shown in the reference image as the non-target object based on the information accompanying the complementary difference image or the three-dimensional information. A specific example of non-target object selection will be described later.
  • the non-attention object selection unit 54 outputs only the complementary difference image determined to contain a non-attention object to the complementary difference image processing unit 56 as the selected complementary difference image.
  • the non-target object selection unit 54 refers to information indicating the reference viewpoint position, a depth map that does not include the non-target object in the reference image, and a depth map of a complementary region described later in the reference image to determine whether the object is a non-target object. Can be determined.
  • the depth value of the depth map of the area on the attention area side with respect to the reference viewpoint position is a positive value. It becomes. Therefore, if the depth values of the depth map of the complementary region are all negative values, it can be said that the depth map exists behind the reference viewpoint.
  • the non-attention object selection unit 54 determines that an object that satisfies this condition, that is, an object whose depth values of the depth map are all negative values is a non-attention object. In other words, the non-attention object selection unit 54 determines that an object that exists within the transmission range illustrated in FIG. Then, the non-target object selection unit 54 outputs a complementary difference image that complements the non-target object to the complementary difference image processing unit 56.
  • the image at the reference viewpoint position is generated from each image captured by two cameras (camera A and camera B) located behind the reference viewpoint position.
  • the image captured by the camera A located behind the reference viewpoint position and on the extended line from the attention area to the non-target object includes the non-target object, but is behind the reference viewpoint position,
  • Non-display objects are not included in the image captured by the camera B that is not located on the extension line from the region toward the non-target object.
  • the image of the reference viewpoint position can be generated by complementing the image captured by the camera A with the image captured by the camera B.
  • An example of an object that is selected as a non-target object by the non-target object selection unit 54 is an object that overlaps the peripheral area of the reference viewpoint position.
  • the non-target object selection unit 54 refers to information indicating the reference viewpoint position, a depth map that does not include the non-target object in the reference image, and a depth map of the complementary region in the reference image to determine whether the object is a non-target object. Can be determined.
  • the non-target object selection unit 54 uses the reference viewpoint position as the origin, and the depth value of the depth map of the complementary region is equal to or less than a certain ratio, for example, ⁇ 10% of the maximum depth value.
  • An object including the following depth values is determined as a non-attention object.
  • the non-target object selection unit 54 determines that an object that exists within the transmission range illustrated in FIG. 7B is a non-target object. Then, the non-target object selection unit 54 outputs a complementary difference image that complements the non-target object to the complementary difference image processing unit 56.
  • the non-attention object selection unit 54 may make a determination based on a constant value instead of a ratio. For example, if the depth of the depth map of the complement region is the reference viewpoint position and the absolute value of the depth value is less than or equal to a certain value, the non-target object selection unit 54 does not pay attention to the non-target object candidate corresponding to the complement region. It can be judged as a thing. For example, if the depth of the depth map of the complementary region is included in a sphere with a certain radius centered on the reference viewpoint position, the non-target object candidate corresponding to the complementary region is selected as the non-target object. It can be judged.
  • the non-attention object selection unit 54 determines whether or not the object is a non-attention object based on information accompanying the complementary difference image.
  • the non-target object selection unit 54 displays a complementary difference image that complements the non-target object as a complementary difference image processing unit 56. Output to.
  • examples of objects selected as non-target objects by the non-target object selection unit 54 include all objects that are in front of the target area.
  • the non-target object selection unit 54 refers to the information indicating the reference viewpoint position, the depth map of the region not including the non-target object in the reference image, and the depth map of the complementary region in the reference image to determine whether the object is a non-target object. It can be determined whether or not. As shown in FIG. 7C, the direction toward the region of interest is positive with the reference viewpoint position as the origin. In this case, the non-target object selection unit 54 determines that the non-target object is included in the complementary difference image if all the depth values of the depth map of the complementary region are smaller than the depth map without the non-target object.
  • the non-attention object selection unit 54 determines that an object existing within the transmission range illustrated in (c) of FIG. 7 is a non-attention object. Then, the non-target object selection unit 54 outputs a complementary difference image that complements the non-target object to the complementary difference image processing unit 56.
  • the non-target object selection unit 54 may select a non-target object when the region of the non-target object candidate in the image at the reference viewpoint position is equal to or greater than a predetermined ratio.
  • the predetermined ratio is an arbitrary value. For example, assuming that the predetermined ratio is set to 50%, in the example shown in FIG. 7A, even if the ratio of the non-target object candidate area in the image captured by the rear camera is 20%, In the image of the reference viewpoint position, when the ratio of the non-target object candidate area is 50%, it is determined that the non-target object candidate is a non-target object, and a complementary difference image that complements the non-target object is The result is output to the complementary difference image processing unit 56.
  • a region where a non-target object candidate exists in the reference image is called a complementary region.
  • the complementary region refers to the inner region of the reference image that is complemented by the complementary image.
  • the complementary region is calculated from the complementary difference image corresponding to the reference image and the complementary image coordinate information attached to the complementary difference image.
  • the depth map of the complementary region is information obtained from the complementary region and the depth map with the non-target object, and is information indicating the depth value of the non-target object corresponding to the complementary difference image. That is, it refers to information obtained by cutting out a depth map corresponding to a complementary region from a depth map having a non-target object corresponding to the reference image.
  • the display method acquisition unit 58 acquires the display method selected and adjusted by the user. Further, the complementary difference image processing unit 56 acquires information indicating a display method related to complementation from the display method acquiring unit 58. The complementary difference image processing unit 56 performs processing according to the display condition on the complementary difference image, and outputs the processed complementary difference image to the attention image generation unit 60 as a processed complementary difference image.
  • the display method according to the present embodiment is a complementary region display method, and is selected and adjusted by the user. Below, the example of the display method which concerns on this embodiment is demonstrated concretely.
  • Display method example 1 As a first example, there can be mentioned a display method for switching and displaying whether or not a specific object is a non-target object, for example, whether or not to complement an object shown in a reference image.
  • the complementary difference image processing unit 56 sets the entire corresponding complementary difference image to be opaque. In other words, the overall alpha value of the corresponding complementary difference image is set to 1. Similarly, when not complementing, the entire alpha value of the corresponding complementary difference image is set to zero.
  • Display method example 2 As a second example, a display method that complements only a specified region in the display image, for example, complements only a part of a non-target object in the reference image can be mentioned.
  • the complementary difference image processing unit 56 can complement only the designated region by setting only the alpha value of the corresponding region of the complementary difference image corresponding to the designated region to 1.
  • the transparency of the non-target object is set, for example, the non-target object reflected in the reference image is made translucent.
  • the complementary difference image processing unit 56 can set the transparency of the non-attention area by changing the overall alpha value of the complementary difference image according to the set transparency.
  • the image processing system 1 is provided with a configuration for identifying the user's field of view, and the complementary difference image processing unit 56 is in the field of view when the user observes a part of the image captured by the 360 degree camera.
  • the display method which complements only a non-attention object can be mentioned.
  • the complementary difference image processing unit 56 sets the entire alpha value of the complementary difference image corresponding to the non-target object in the field of view to 1, and sets the entire alpha value of the complementary difference image corresponding to the non-target object other than the above to 0. By doing so, only the specified area can be complemented.
  • the complementary difference image processing unit 56 outputs the complementary difference image for which the above processing has been completed to the attention image generating unit 60.
  • the attention image generation unit 60 synthesizes the reference image supplied from the demultiplexing unit 52 and the processed complementary difference image supplied from the complementary difference image processing unit 56, and the attention of which the transparency of the non-target object is adjusted. Generate an image.
  • the attention image generation unit 60 synthesizes the complementary difference image with the reference image at a position based on the complementary image coordinate information of the reference image. That is, the complementary difference image is synthesized with respect to the reference image at a position where the complementary difference image is extracted in the complementary image corresponding to the reference image.
  • the attention image generation unit 60 synthesizes the complementary difference image at a position determined by the complementary image coordinate information that is the coordinate information in the reference image, so even if the complementary difference image is smaller than the reference image. Therefore, it is possible to suitably perform the synthesis process.
  • the attention image generation unit 60 alpha blends the reference image and the complementary difference image.
  • the attention image generation unit 60 sets the alpha value of the reference image according to the alpha value of the complementary difference image. Specifically, the attention image generation unit 60 sets the alpha value of each pixel of the reference image to a value obtained by subtracting from 1 the alpha value of each corresponding pixel of the complementary difference image.
  • step S ⁇ b> 10 the non-target object selection unit 54 acquires one of the complementary difference images from the demultiplexing unit 52.
  • step S12 the non-target object selection unit 54 acquires the depth map of the complementary region corresponding to the complementary difference image acquired in step S10 from the three-dimensional information of the reference image from the demultiplexing unit 52.
  • step S14 the non-attention object selection unit 54 refers to the three-dimensional information and determines whether or not the object included in the complementary difference image is a non-attention object. If it is a non-attention object (YES in S14), the process proceeds to step S16. If it is not a non-attention object (NO in S14), the process proceeds to S20.
  • the non-target object selection unit 54 determines that the medicine bottle is a non-target object for the complementary difference image 1 because the distance between the camera and the medicine bottle is short.
  • the non-focused object selection unit 54 determines that the complementary difference image 2 is not a non-focused object because the distance between the camera and the book end is long.
  • step S16 the display method acquisition unit 58 acquires information indicating the display method desired by the user.
  • step S18 the complementary difference image processing unit 56 changes the alpha value of the complementary difference image based on the display method acquired in S16.
  • the complementary difference image processing unit 56 changes the alpha value of all the pixels of the complementary difference image 1 to 0.5 based on the display conditions.
  • step S20 the target image composition unit 50 determines whether or not the processing for all the complementary difference images included in the target image data has been completed. If it is determined that the process has been completed (YES in S20), the process proceeds to step S22. If it is determined that the process has not been completed (NO in S20), the process returns to S10.
  • step S22 the attention image generation unit 60 generates an attention image from the reference image and the complementary difference image corresponding to the non-attention object.
  • the attention image generation unit 60 synthesizes the reference image and the complementary difference image 1 in which the alpha value is changed to 0.5. Since the alpha value of the complementary difference image 1 is 0.5, the non-target medicine bottle is translucent in the attention image, so that an object shielded by the medicine bottle can be seen through.
  • step S24 the attention image generation unit 60 outputs the generated attention image.
  • a plurality of complementary images are generated in advance according to the non-target object selection pattern, and only a region where a difference between the reference image and the complementary image is generated is stored as a complementary difference image.
  • a desired image that does not include a non-attention object can be generated with an arbitrary process setting and a relatively light process.
  • the amount of data to be stored can be suppressed.
  • processing settings can be changed flexibly.
  • a point on the space designated by the user (referred to as a “virtual viewpoint”.
  • the position of the “virtual viewpoint” can be indicated by coordinates in the space, and information indicating the coordinates of the virtual viewpoint is displayed as the virtual viewpoint.
  • an image (referred to as “virtual viewpoint image”) in which the attention area is observed is further generated from the position information.
  • the image processing system 1 a includes an attention image data generation unit 30, a virtual viewpoint image synthesis unit 70, and a storage unit 20.
  • the virtual viewpoint image composition unit 70 acquires information indicating the position of the virtual viewpoint.
  • the storage unit 20 and the attention image data generation unit 30 correspond to the storage unit 20 and the attention image data generation unit 30 of the image processing system 1 according to the first embodiment, respectively.
  • the virtual viewpoint image composition unit 70 includes a demultiplexing unit 52, a non-target object selection unit 54b, a complementary difference image processing unit 56, a display method acquisition unit 58, a target image generation unit 60, and a virtual viewpoint.
  • An image generation unit 72 is provided. Note that the demultiplexing unit 52, the non-target object selection unit 54a, the complementary difference image processing unit 56, the display method acquisition unit 58, and the target image generation unit 60 are each demultiplexed in the image processing system 1 according to the first embodiment. This corresponds to the conversion unit 52, the non-target object selection unit 54, the complementary difference image processing unit 56, the display method acquisition unit 58, and the target image generation unit 60.
  • the non-target object selection unit 54 of Embodiment 1 selects a non-target object based on the reference viewpoint position, but the non-target object selection unit 54a of the present embodiment replaces the reference viewpoint position with the position of the virtual viewpoint. Select a non-attention object as a reference.
  • Non-attention object selection unit 54 As an example of the object selected as the non-attention object by the non-attention object selection unit 54, an object positioned behind the virtual viewpoint position can be cited.
  • the non-target object selection unit 54 refers to information indicating the virtual viewpoint position, a depth map that does not include the non-target object in the reference image, and a depth map of a complementary region described later in the reference image to determine whether the object is a non-target object. Can be determined.
  • the virtual viewpoint position is the origin and the direction toward the attention area is positive
  • the depth value of the depth map of the area closer to the attention area than the virtual viewpoint position is a positive value.
  • the non-target object selection unit 54 determines that an object that satisfies this condition, that is, an object whose depth values in the depth map are all negative values, is a non-target object, and complements the non-target object. Is output to the complementary difference image processing unit 56.
  • the non-target object selection unit 54 refers to the information indicating the virtual viewpoint position, the depth map that does not include the non-target object in the reference image, and the depth map of the complementary region in the reference image to determine whether or not the object is a non-target object. Can be determined.
  • the non-attention object selection unit 54 uses a virtual viewpoint position as an origin, and removes an object in which the depth value of the depth map of the complementary region includes a depth value of a certain percentage or less, for example, ⁇ 10% or less of the maximum depth value. It determines with it being an attention object, and outputs the complementary difference image which complements the said non-attention object to the complementary difference image process part 56.
  • the non-attention object selection unit 54 may make a determination based on a constant value instead of a ratio. For example, if the depth of the depth map of the complementary region is included within ⁇ 10 cm in the depth direction starting from the virtual viewpoint position, the non-focused object selection unit 54 does not focus on the non-focused object candidate corresponding to the complementary region. It can be judged as a thing. Alternatively, the non-target object selection unit 54 may determine that the non-target object candidate is a non-target object on the condition that the absolute value of the depth value is not more than a certain value.
  • examples of objects selected as non-target objects by the non-target object selection unit 54 include all objects that are in front of the target area.
  • the non-target object selection unit 54 refers to the information indicating the virtual viewpoint position, the depth map of the region not including the non-target object in the reference image, and the depth map of the complementary region in the reference image to determine whether the object is a non-target object. It can be determined whether or not.
  • the direction toward the region of interest is positive with the virtual viewpoint position as the origin.
  • the non-target object selection unit 54 determines that the non-target object is included in the complementary difference image if all the depth values of the depth map of the complement region are smaller than the depth map without the non-target object,
  • the complementary difference image is output to the complementary difference image processing unit 56.
  • the non-attention object selection unit 54 may select a non-attention object as a non-attention object when the area of the non-attention object candidate in the image at the virtual viewpoint position is a predetermined ratio or more.
  • the predetermined ratio is an arbitrary value. For example, assuming that the predetermined ratio is set to 50%, even if the ratio of the non-target object candidate area in the image captured by the rear camera is 20%, the non-target object candidate is displayed in the virtual viewpoint position image. When the proportion of the area is 50%, it is determined that the non-target object candidate is a non-target object, and a complementary difference image that complements the non-target object is output to the complementary difference image processing unit 56.
  • the virtual viewpoint image generation unit 72 generates a virtual viewpoint image using the attention image generated by the attention image generation unit 60. That is, the virtual viewpoint image generation unit 72 generates, for example, an image in which a non-target object candidate determined to be a non-target object appears to be transparent and the target region is observed from the virtual viewpoint position. To do.
  • This embodiment is different from the above-described embodiment in that complementary difference information is generated and used instead of the complementary difference image.
  • the image processing system 1b includes an attention image data generation unit 30b, an attention image synthesis unit 50b, and a storage unit 20.
  • the storage unit 20 corresponds to the storage unit 20 of the image processing system 1 according to the first embodiment.
  • the attention image data generation unit 30 b includes an acquisition unit 32, a reference image selection unit 34, an attention region complement data generation unit 36 b, a three-dimensional information acquisition unit 38, and a multiplexing unit 40.
  • the acquisition unit 32, the reference image selection unit 34, the 3D information acquisition unit 38, and the multiplexing unit 40 are respectively the acquisition unit 32, the reference image selection unit 34, and the 3D information acquisition of the image processing system 1 according to the first embodiment. This corresponds to the unit 38 and the multiplexing unit 40.
  • the attention area complement data generation unit 36 of Embodiment 1 has generated attention image data including a complementary difference image
  • the attention area complement data generation unit 36b of the present embodiment generates complementary difference information instead of the complementary difference image. Then, attention image data including complementary difference information is generated.
  • an attention image that complements the region of the non-attention object included in the reference image is generated.
  • a reference image is complemented, a partial region of any one of the other reference images is used.
  • complementary difference information information indicating which region of another reference image should be used for which region of a certain reference image is referred to as complementary difference information.
  • an area hidden by a non-target object in one reference image is one of the other reference images (reference image B shown in FIG. 14B).
  • reference image A shown in FIG. 14A an area hidden by a non-target object in one reference image
  • reference image B shown in FIG. 14B an area hidden by a non-target object in one reference image
  • reference image B shown in FIG. 14B an area hidden by a non-target object in one reference image
  • reference image B shown in FIG. 14B
  • the reference image A includes a white cloud that is a non-target object in an area starting from the coordinates (x1, y1) and having a width of wa and a height of ha.
  • This region is a complement region to be complemented in the reference image A.
  • the complementary region in the third embodiment is equivalent to the complementary region shown in the description of the depth map of the complementary region in the first embodiment.
  • the complementary region here is not calculated from the complementary difference image corresponding to the reference image and the complementary image coordinate information attached to the complementary difference image, but is information included in the complementary difference information.
  • the reference image B has a width wb starting from the coordinates (x2, y2) corresponding to the coordinates (x1, y1) of the reference image A, There is no non-attention object in the area where the height is hb (reference area of the standard image B). Therefore, the reference area of the standard image B can be used to complement the complementary area of the standard image A.
  • the size and shape of the complementary region and the reference region are not necessarily the same.
  • the non-target object selection unit 54b described later deforms the partial image extracted from the reference region to fit the complementary region.
  • the complementary difference information 1 that is the complementary difference information of the reference image A is information indicating that the corresponding reference image is “reference image A”, the starting point (x1, information indicating the coordinates of y1), information indicating the width wa of the complementary region, information indicating the height ha of the complementary region, information indicating that the standard image used for complementing is “standard image B”, the starting point of the reference region ( information indicating the coordinates of x2, y2), information indicating the width wb of the complementary region, information indicating the height hb of the complementary region, and information indicating that the non-target object is a “white cloud”.
  • the reference image B includes a gray cloud that is a non-attention object in a region starting from the coordinates (xb1, yb1) and ending at the coordinates (xb2, yb2).
  • This region is a complement region to be complemented with the reference image B.
  • the lower right point in the target area is called an “end point”.
  • the reference image A starts from the coordinates (xa1, ya1) corresponding to the coordinates (xb1, yb1) of the reference image B, and the coordinates (xb2) of the reference image B , Yb2), there is no non-attention object in the area (reference area of the standard image A) whose end point is the coordinate (xa2, ya2) corresponding to. Therefore, the reference area of the standard image A can be used to complement the complementary area of the standard image B.
  • the complementary difference information 2 that is the complementary difference information of the reference image B is information indicating that the corresponding reference image is the “reference image B”, the starting point (xb1, Information indicating the coordinates of yb1), information indicating the coordinates of the end point (xb2, yb2) of the complementary region, information indicating that the standard image used for complementing is “standard image A”, and the starting point (xa1, ya1) of the reference region Information indicating the coordinates of the reference area, information indicating the coordinates of the end point (xa2, ya2) of the reference area, and information indicating that the non-target object is a “gray cloud”.
  • the supplemental area and the reference area may be expressed in any manner as long as the area is uniquely determined. For example, it may be represented by the coordinates of the four corners of the region. Further, it is not necessarily a rectangular area, and may be, for example, a circle.
  • the attention image synthesis unit 50 b includes a demultiplexing unit 52, a non-target object selection unit 54 b, a complementary difference image processing unit 56, a display method acquisition unit 58, and a attention image generation unit 60.
  • the demultiplexing unit 52, the complementary difference image processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60 are respectively the demultiplexing unit 52 and the complementary difference image of the image processing system 1 according to the first embodiment. This corresponds to the processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60.
  • the non-target object selection unit 54b first extracts pixels in the reference area of another standard image based on the complementary difference information corresponding to the standard image, and performs the complementary differential image corresponding to the selected non-target object. Is generated. At this time, information on the base image used for complementation and information other than the reference area included in the complementation difference information are handled as information accompanying the complementation difference image. Next, the same processing as that performed by the non-target object selection unit 54 of the first embodiment is performed. That is, a non-target object is selected based on information accompanying the complementary difference image or three-dimensional information, and the complementary difference image determined to contain the non-target object is output to the complementary difference image processing unit 56.
  • the image processing apparatus refers to the target image data including the complementary difference information generated in advance, and determines whether or not the object in the reference image is a non-target object. Based on the display unit, the selection unit 54b, the complementary difference image processing unit 56 that performs image processing on the complementary difference image specified by the complementary difference information included in the target image data, the complementary difference image, and the reference image And an attention image generation unit 60 that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted by combining.
  • a desired image that does not include a non-attention object can be generated with an arbitrary process setting and a relatively light process.
  • the amount of data to be stored can be suppressed.
  • This embodiment is different from the above-described embodiment in that three-dimensional information is not used.
  • the image processing system 1c includes an attention image data generation unit 30c, an attention image synthesis unit 50c, and a storage unit 20.
  • the storage unit 20 corresponds to the storage unit 20 of the image processing system 1 according to the first embodiment.
  • the attention image data generation unit 30c is different from the attention image data generation unit 30 of the first embodiment in that it does not have a function of a three-dimensional information acquisition unit.
  • the acquisition unit 32, the reference image selection unit 34, the attention area complement data generation unit 36, and the multiplexing unit 40c are respectively the acquisition unit 32, the reference image selection unit 34, and the attention area complement of the image processing system 1 according to the first embodiment. This corresponds to the data generation unit 36 and the multiplexing unit 40.
  • the attention image data since the three-dimensional information is not acquired, the attention image data includes a reference image and a complementary difference image.
  • the attention image synthesis unit 50 c includes a demultiplexing unit 52, a non-target object selection unit 54 c, a complementary difference image processing unit 56, a display method acquisition unit 58, and a attention image generation unit 60.
  • the demultiplexing unit 52, the complementary difference image processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60 are respectively the demultiplexing unit 52 and the complementary difference image of the image processing system 1 according to the first embodiment. This corresponds to the processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60.
  • the non-attention object selection unit 54c selects an unattention object without using the three-dimensional information. That is, the non-target object selection unit 54c selects a non-target object based on information accompanying the non-target object included in the complementary difference image. That is, the non-target object selection unit 54c identifies a specific object and selects it as a non-target object.
  • This embodiment is different from Embodiment 1 in that the non-attention object selection unit 54d uses user attention point information to select whether or not it is a non-attention object.
  • the image processing system 1 d includes an attention image data generation unit 30, an attention image synthesis unit 50 d, and a storage unit 20.
  • the attention image composition unit 50d in the present embodiment acquires information indicating the user's line-of-sight direction.
  • the storage unit 20 and the attention image data generation unit 30 correspond to the storage unit 20 and the attention image data generation unit 30 of the image processing system 1 according to the first embodiment, respectively.
  • the attention image data generation unit 30d includes an acquisition unit 32, a reference image selection unit 34, an attention region complement data generation unit 36, a three-dimensional information acquisition unit 38d, and a multiplexing unit 40.
  • the acquisition unit 32, the reference image selection unit 34, the attention area complement data generation unit 36, and the multiplexing unit 40 are respectively the acquisition unit 32, the reference image selection unit 34, and the attention area complementation of the image processing system 1 according to the first embodiment. This corresponds to the data generation unit 36 and the multiplexing unit 40.
  • the 3D information acquisition unit 38d acquires line-of-sight direction information in addition to the 3D information of the reference image.
  • the line-of-sight direction information includes information indicating the position of the eyes of a person included in the input image and information indicating a point that the person is interested in.
  • the line-of-sight direction information may be acquired using, for example, an eye tracking device, or may be acquired by performing image processing on an input image in which a person's eyes are reflected.
  • the line-of-sight direction information is output to the multiplexing unit 40 in association with the reference image. However, when a person is not included in the reference image, the line-of-sight direction information may not be associated with the reference image.
  • the three-dimensional information acquisition unit 38d acquires information on the position of the eyes of the person included in the line-of-sight direction information as information indicating a relative position with respect to the depth map.
  • the position of the eye can be represented by three-dimensional coordinates with reference to the origin of the depth map.
  • the information indicating the attention point included in the line-of-sight direction information is information indicating which coordinates in the reference image the person is observing. For example, when a person observes the center of an image with respect to a reference image having a width w and a height h, the coordinates of the attention point are (w / 2, h / 2).
  • the attention image synthesis unit 50 d includes a demultiplexing unit 52, a non-target object selection unit 54 d, a complementary difference image processing unit 56, a display method acquisition unit 58, and a attention image generation unit 60.
  • the demultiplexing unit 52, the complementary difference image processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60 are respectively the demultiplexing unit 52 and the complementary difference image of the image processing system 1 according to the first embodiment. This corresponds to the processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60.
  • the user attention point information is information representing coordinates in the attention image that the user observes. For example, when the user is observing the center of the attention image having the width w and the height h, the user attention point information is information indicating (w / 2, h / 2). For example, as the user attention point, if the user observes the attention image using a head mounted display (HMD), the center point of the area displayed on the HMD can be set as the attention point. Similarly, when the user observes an attention image using a flat panel display, the center point of the area displayed on the display can be set as the attention point. Further, for example, the measurement may be performed using an eye tracking device.
  • HMD head mounted display
  • the center point of the area displayed on the display can be set as the attention point.
  • the measurement may be performed using an eye tracking device.
  • the non-target object selection unit 54d calculates the angle of the line of sight of the person in the reference image from the line-of-sight direction information corresponding to the reference image. Next, the angle of the user's eye line is calculated from the reference viewpoint position and the user attention point information. The non-target object selection unit 54d determines whether or not the object is a non-target object from the former calculation result and the latter calculation result.
  • the non-target object selection unit 54d determines whether or not the object is a non-target object from the reference viewpoint position, the angle of the user's eyes, the position of the eyes of the person in the reference image, and the angle of the eyes.
  • the non-attention object selection unit 54d determines that the person in the reference image position is close to the eye position of the person in the reference image and the eye angle of the user is close to the eye angle of the person in the reference image. Can be made non-attention.
  • the user's line-of-sight direction when viewing the 360-degree camera image may be recorded and used as user attention point information.
  • control blocks of the image processing systems 1, 1 a, 1 b, 1 c, 1 d are integrated. It may be realized by a logic circuit (hardware) formed on a circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit).
  • the image processing apparatuses 1, 1 a, 1 b, 1 c, and 1 d record a CPU that executes instructions of a program, which is software that realizes each function, and the program and various data that can be read by a computer (or CPU).
  • ROM Read Only Memory
  • recording medium storage device
  • RAM Random Access Memory
  • the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it.
  • a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used.
  • the program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program.
  • an arbitrary transmission medium such as a communication network or a broadcast wave
  • one embodiment of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.
  • An image processing apparatus refers to a target image data including a complementary difference image generated in advance, and determines whether or not an object in a reference image is a non-target object (54, 54a, 54b, 54c, 54d) and a complementary difference image processing unit (56) that performs image processing on the complementary difference image included in the target image data based on display conditions; the complementary difference image; and An attention image generation unit (60) that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted by synthesizing the reference image;
  • a desired image that does not include a non-attention object can be generated with an arbitrary process setting and a relatively light process.
  • the amount of data to be stored can be suppressed.
  • the image processing apparatus is the image processing apparatus according to aspect 1, in which the complementary difference image is smaller than the reference image, the attention image data includes coordinate information, and the attention image generation unit (60) synthesizes the complementary difference image at a position determined by the coordinate information in the reference image.
  • the complementary difference image is smaller than the reference image, and the attention image generation unit (60) synthesizes the complementary difference image at a position determined by the coordinate information in the reference image. Even when there are a plurality of complementary difference images, the generation process can be performed with a lighter process, and the amount of data to be stored can be further suppressed. In addition, the complementary difference image can be suitably combined.
  • the non-target object selection unit (54, 54a, 54b, 54c, 54d) refers to the depth map of the reference image, and It is determined whether or not the object in the reference image is a non-target object.
  • the object in the reference image is a non-target object.
  • the depth map of the reference image includes a depth map that does not include the non-attention object and a depth map that includes the non-attention object.
  • the object in the reference image is a non-target object.
  • the image processing apparatus is the image processing apparatus according to any one of aspects 1 to 4, wherein the image is viewed from the position of the coordinates in the space based on the virtual viewpoint position information indicating the coordinates in the space and the attention image. You may further provide the virtual viewpoint image generation part (72) which synthesize
  • the non-target object selecting unit selects a specific object as the non-target object.
  • the region of interest is not hidden by what does not need a particular attention.
  • the non-target object selection unit refers to a user's line-of-sight direction to determine whether an object in the reference image is a non-target object. judge.
  • the object in the reference image is a non-attention object according to the user's line-of-sight direction.
  • the image processing apparatus refers to a target image data including preliminarily generated complementary difference information, and determines whether or not an object in the reference image is a non-target object.
  • a complementary difference image processing unit that performs image processing on a complementary difference image specified by the complementary difference information included in the target image data based on display conditions, and combining the complementary difference image and the reference image.
  • An attention image generation unit that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted.
  • An image processing system is an image processing system including a first image processing apparatus and a second image processing apparatus, and the first image processing apparatus includes a plurality of reference images.
  • the attention image data generating unit for generating a complementary difference image used to complement the non-target object included in the reference image, and generating attention image data including the generated complementary difference image
  • the second image processing apparatus refers to the target image data generated by the target image data generation unit, and selects a non-target object selection unit that selects whether or not the object in the reference image is a non-target object.
  • the transparency of the non-attention object in the reference image is obtained by combining the complementary difference image processing unit that performs image processing on the complementary difference image included in the attention image data, and the complementary difference image and the reference image. Adjusted A target image generator to obtain a target image, and a.
  • the image processing apparatus may be realized by a computer.
  • the image processing apparatus is operated on each computer by causing the computer to operate as each unit (software element) included in the image processing apparatus.
  • the image processing program of the image processing apparatus to be realized in this way and a computer-readable recording medium on which the image processing program is recorded also fall within the scope of the present invention.
  • Image processing system 54 1, 1a, 1b, 1c, 1d Image processing system 54, 54b, 54c, 54d Non-attention object selection unit 56 Complementary difference image processing unit 60 Attention image generation unit 36, 36b Attention region complementation data generation unit 72 Virtual viewpoint image generation unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

With regard to the process of generating such an image that a non-attention object is transparentized, the present invention makes it possible to generate the image at high speed while suppressing an increase in the data volume to be stored. An image processing device provided with a non-attention object selection unit (54) for determining whether or not an object in a reference image is a non-attention object, a complementary difference image processing unit (56) for performing image processing on a complementary difference image, and an attention object image generation unit (60) for synthesizing the complementary difference image and the reference image and thereby obtaining an attention object image in which the level of transparency of the non-attention object in the reference image is adjusted.

Description

画像処理装置、画像処理プログラム、及び記録媒体Image processing apparatus, image processing program, and recording medium
　本発明の一様態は、画像処理に関する。
　〔関連出願の相互参照〕
　本出願は、２０１７年４月４日に出願された出願番号２０１７－０７４５３３号の出願に関するものであって上記出願を基礎として優先権を主張するものである。上記出願の内容は、参照により本明細書に含まれる。 One embodiment of the present invention relates to image processing.
[Cross-reference of related applications]
This application is related to the application of application number 2017-074533 filed on Apr. 4, 2017, and claims priority from the above application. The contents of the above application are included herein by reference.
　ある視点から撮像された映像を観察しているときに、視聴者が注目したい領域（注目領域）が非注目物によって隠される状況が発生する場合がある。 When observing a video imaged from a certain viewpoint, there may be a situation in which a region (attention region) that the viewer wants to focus on is hidden by a non-attention object.
　このような場合に対し、注目領域を複数のカメラで撮像し、複数の画像を用いることで非注目物を透明にしたような画像を生成する技術が知られている（特許文献１、特許文献２等）。 In such a case, a technique is known in which a region of interest is captured by a plurality of cameras, and an image in which a non-attention object is made transparent by using a plurality of images is generated (Patent Document 1, Patent Document). 2 etc.).
日本国公開特許公報「特開２００６－３０１９３２号公報（２００６年１１月７日公開）」Japanese Patent Publication “Japanese Patent Laid-Open No. 2006-301932 (published on November 7, 2006)” 日本国特許公開公報「特開２０１５－１３５５４０号公報（２０１５年５月６日公開）」Japanese Patent Publication “Japanese Unexamined Patent Publication No. 2015-135540 (May 6, 2015)”
　ここで、非注目物とは、注目領域を隠す、注目する必要のない物体等のことである。何が非注目物であるかは、状況により変わり得る。例えば、手術の様子を、執刀医の後方に置かれたカメラを用いて撮像する場合を考える。このとき、執刀医の目線で手術野を観察したい場合は、手術野が注目領域であり、執刀医が非注目物となる。一方で、助手の目線で執刀医の動きを観察したい場合は、執刀医は非注目物ではなく注目領域となる。また、状況によらず常に非注目物と設定したい物体(例えば撮像に用いたカメラ)も存在し得る。 Here, the non-target object is an object that hides the target area and does not need to be focused. What is unfocused can vary depending on the situation. For example, let us consider a case in which an operation is imaged using a camera placed behind the surgeon. At this time, if the surgeon wants to observe the surgical field from the viewpoint of the surgeon, the surgical field is the attention area, and the surgeon is the non-target object. On the other hand, if the surgeon wants to observe the surgeon's movements from the assistant's perspective, the surgeon becomes a region of interest rather than a non-attention object. There may also be an object (for example, a camera used for imaging) that is always desired to be set as a non-target object regardless of the situation.
　上述のように、非注目物を透明にしたような画像を生成する場合、画像中のどの物体を非注目物に設定するかによって、生成すべき画像が変わり得る。このような画像生成処理において、例えば、新たに非注目物が選択される度に、非注目物を透明にしたような画像を一から生成すると、画像生成に時間がかかるため、画像表示に遅延が生じる。一方で、複数の非注目物の各々に対応する、非注目物を透明にしたような画像を、複数の非注目物毎、及び、複数の非注目物の全ての組み合わせについて、予め生成し、格納していると、画像デ－タ量が膨大になってしまう。また、後から処理の設定(例えば非注目物の透明度)を変更できない。 As described above, when generating an image in which a non-target object is made transparent, the image to be generated can change depending on which object in the image is set as the non-target object. In such image generation processing, for example, every time a new non-attention object is selected, if an image that makes the non-attention object transparent is generated from scratch, it takes time to generate the image, so the image display is delayed. Occurs. On the other hand, an image corresponding to each of a plurality of non-attention objects, such as a transparent non-interest object, is generated in advance for each of the plurality of non-interest objects and all combinations of the plurality of non-interest objects, If stored, the amount of image data becomes enormous. Also, the processing settings (for example, the transparency of the non-attention object) cannot be changed later.
　本発明の一態様は、非注目物を透明にしたような画像を生成する処理について、格納するデータ量の増大を抑制しつつ、高速な画像生成を可能とする技術を実現することを目的とする。 An object of one aspect of the present invention is to realize a technology that enables high-speed image generation while suppressing an increase in the amount of data to be stored in a process for generating an image in which a non-target object is made transparent. To do.
　上記の課題を解決するために、本発明の一態様に係る画像処理装置は、事前に生成された補完差分画像を含む注目画像データを参照して、基準画像内の物体が非注目物か否かを判定する非注目物選択部と、表示条件に基づき、前記注目画像データに含まれる前記補完差分画像に画像処理を行う補完差分画像処理部と、前記補完差分画像と前記基準画像を合成することで、前記基準画像内の前記非注目物の透明度が調整された注目画像を得る注目画像生成部と、を備えている。 In order to solve the above problems, an image processing apparatus according to an aspect of the present invention refers to attention image data including a complementary difference image generated in advance, and determines whether or not an object in a reference image is a non-attention object. A non-target object selection unit that determines whether or not, a complementary difference image processing unit that performs image processing on the complementary difference image included in the target image data based on display conditions, and the complementary difference image and the reference image are combined In this way, the image processing apparatus includes an attention image generation unit that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted.
　また、上記の課題を解決するために、本発明の一態様に係る画像処理システムは、第１の画像処理装置と第２の画像処理装置とを含む画像処理システムであって、前記第１の画像処理装置は、複数の基準画像の各々に関し、当該基準画像に含まれている非注目物を補完するために用いられる補完差分画像を生成し、生成した補完差分画像を含む注目画像データを生成する注目画像データ生成部を備え、前記第２の画像処理装置は、前記注目画像データ生成部が生成した注目画像データを参照して、基準画像内の物体が非注目物か否かを選択する非注目物選択部と、表示条件に基づき、前記注目画像データに含まれる前記補完差分画像に画像処理を行う補完差分画像処理部と、前記補完差分画像と前記基準画像を合成することで、前記基準画像内の前記非注目物の透明度が調整された注目画像を得る注目画像生成部と、を備えている。 In order to solve the above problem, an image processing system according to an aspect of the present invention is an image processing system including a first image processing device and a second image processing device, wherein the first image processing device includes: The image processing device generates, for each of the plurality of reference images, a complementary difference image that is used to complement a non-target object included in the reference image, and generates attention image data including the generated complementary difference image The second image processing device selects whether or not the object in the reference image is a non-target object with reference to the target image data generated by the target image data generation unit. By combining the non-target object selection unit, a complementary differential image processing unit that performs image processing on the complementary differential image included in the target image data based on display conditions, and the complementary differential image and the reference image, Standard image Transparency of the non-interest of the inner is provided with a target image generator to obtain a target image that has been adjusted, the.
　本発明の一態様によれば、格納するデータ量の増大を抑制しつつ、高速な画像の生成を可能とし、さらに処理の設定をフレキシブルに変更できる、という効果を奏する。 According to one aspect of the present invention, it is possible to generate a high-speed image while suppressing an increase in the amount of data to be stored, and further, it is possible to flexibly change processing settings.
本発明の実施形態１に係る画像処理装置の概略ブロック図である。1 is a schematic block diagram of an image processing apparatus according to Embodiment 1 of the present invention. 本発明の実施形態１に係る画像処理装置の注目画像データ生成部の概略ブロック図である。It is a schematic block diagram of the attention image data generation part of the image processing device concerning Embodiment 1 of the present invention. 本発明の一態様の入力画像の撮像イメージを説明する図である。FIG. 6 illustrates a captured image of an input image according to one embodiment of the present invention. 本発明の一態様の補完差分画像を説明する図である。It is a figure explaining the complementary difference image of 1 aspect of this invention. 本発明の一態様の補完差分画像を説明する図である。It is a figure explaining the complementary difference image of 1 aspect of this invention. 本発明の実施形態１に係る画像処理装置の注目画像合成部の概略ブロック図である。It is a schematic block diagram of the attention image composition part of the image processing device concerning Embodiment 1 of the present invention. 本発明の実施形態１に係る画像処理装置非注目物選択部による非注目物の選択を説明する図である。It is a figure explaining selection of the non-attention object by the image processing apparatus non-attention object selection part concerning Embodiment 1 of the present invention. 本発明の実施形態１の注目画像合成部が実行する処理のフローチャートである。It is a flowchart of the process which the attention image synthetic | combination part of Embodiment 1 of this invention performs. 本発明の実施形態１の注目画像合成部が実行する処理を説明する図である。It is a figure explaining the process which the attention image synthetic | combination part of Embodiment 1 of this invention performs. 本発明の実施形態２に係る画像処理装置の概略ブロック図である。全体構成It is a schematic block diagram of the image processing apparatus which concerns on Embodiment 2 of this invention. overall structure 本発明の実施形態２に係る画像処理装置の仮想視点画像合成部の概略ブロック図である。It is a schematic block diagram of the virtual viewpoint image synthetic | combination part of the image processing apparatus which concerns on Embodiment 2 of this invention. 本発明の実施形態３に係る画像処理装置の概略ブロック図である。It is a schematic block diagram of the image processing apparatus which concerns on Embodiment 3 of this invention. 本発明の実施形態３に係る画像処理装置の注目画像データ生成部の概略ブロック図である。It is a schematic block diagram of the attention image data generation part of the image processing device concerning Embodiment 3 of the present invention. 本発明の実施形態３の補完差分情報を説明する図である。It is a figure explaining the complementary difference information of Embodiment 3 of the present invention. 本発明の実施形態３に係る画像処理装置の注目画像合成部の概略ブロック図である。It is a schematic block diagram of the attention image composition part of the image processing device concerning Embodiment 3 of the present invention. 本発明の実施形態４に係る画像処理装置の概略ブロック図である。It is a schematic block diagram of the image processing apparatus which concerns on Embodiment 4 of this invention. 本発明の実施形態４に係る画像処理装置の注目画像データ生成部の概略ブロック図である。It is a schematic block diagram of the attention image data generation part of the image processing device concerning Embodiment 4 of the present invention. 本発明の実施形態４に係る画像処理装置の注目画像合成部の概略ブロック図である。It is a schematic block diagram of the attention image composition part of the image processing device concerning Embodiment 4 of the present invention. 本発明の実施形態５に係る画像処理装置の概略ブロック図である。It is a schematic block diagram of the image processing apparatus which concerns on Embodiment 5 of this invention. 本発明の実施形態５に係る画像処理装置の注目画像データ生成部の概略ブロック図である。It is a schematic block diagram of the attention image data generation part of the image processing device concerning Embodiment 5 of the present invention. 本発明の実施形態５に係る画像処理装置の注目画像合成部の概略ブロック図である。It is a schematic block diagram of the attention image composition part of the image processing device concerning Embodiment 5 of the present invention.
　〔実施形態１〕
　本発明の一実施形態について、図１から図９に基づいて説明する。 Embodiment 1
An embodiment of the present invention will be described with reference to FIGS.
　本実施形態に係る画像処理システム１の概要を図１に基づいて説明する。 An overview of the image processing system 1 according to the present embodiment will be described with reference to FIG.
　図１に示すように、本実施形態に係る画像処理システム１は、注目画像データ生成部３０、注目画像合成部５０、及び格納部２０を備えている。 As shown in FIG. 1, the image processing system 1 according to the present embodiment includes an attention image data generation unit 30, an attention image synthesis unit 50, and a storage unit 20.
　注目画像データ生成部３０は、注目画像データを生成する。注目画像データは、後述する基準画像、該基準画像の付随情報、基準画像に対応する補完差分画像、該補完差分画像の付随情報、及び基準画像の三次元情報を含む。注目画像データ生成部３０は、生成した注目画像データを、格納部２０に出力する。 The attention image data generation unit 30 generates attention image data. The attention image data includes a reference image described later, accompanying information of the reference image, a complementary difference image corresponding to the reference image, accompanying information of the complementary difference image, and three-dimensional information of the reference image. The attention image data generation unit 30 outputs the generated attention image data to the storage unit 20.
　格納部２０は、取得した注目画像データを格納する。 The storage unit 20 stores the acquired attention image data.
　注目画像合成部５０は、格納部２０から取得した注目画像データを用いて注目画像を合成する。 The attention image synthesis unit 50 synthesizes the attention image using the attention image data acquired from the storage unit 20.
　なお、画像処理システム１において、注目画像データ生成部３０、注目画像合成部５０、及び格納部２０を、一体の画像処理装置内で実現する構成としてもよいし、別体の画像処理装置内で実現する構成としてもよい。例えば、注目画像データ生成部３０及び格納部２０を含む第１の画像処理装置を、送信局に配置し、注目画像合成部５０を含む第２の画像処理装置を、ユーザの宅内に配置するといった構成でもよい。注目画像データ生成部３０、注目画像合成部５０、及び格納部２０を、別体の画像処理装置内で実現する構成の場合、各々の画像処理装置がデータ送受信部を備え、当該データ送受信部を介して各画像処理装置が互いに画像データの送受信を行う。後述する各実施形態においても同様である。 In the image processing system 1, the attention image data generation unit 30, the attention image synthesis unit 50, and the storage unit 20 may be configured to be realized in an integrated image processing apparatus, or in a separate image processing apparatus. It is good also as a structure to implement | achieve. For example, the first image processing device including the attention image data generation unit 30 and the storage unit 20 is disposed in the transmission station, and the second image processing device including the attention image synthesis unit 50 is disposed in the user's home. It may be configured. When the attention image data generation unit 30, the attention image synthesis unit 50, and the storage unit 20 are configured in a separate image processing apparatus, each image processing apparatus includes a data transmission / reception unit, and the data transmission / reception unit The image processing devices transmit and receive image data to each other. The same applies to each embodiment described later.
　＜注目画像データ生成部＞
　図２に示すように、注目画像データ生成部３０は、取得部３２、基準画像選択部３４、注目領域補完データ生成部３６、三次元情報取得部３８、及び多重化部４０を備えている。 <Attention image data generation unit>
As illustrated in FIG. 2, the attention image data generation unit 30 includes an acquisition unit 32, a reference image selection unit 34, an attention region complement data generation unit 36, a three-dimensional information acquisition unit 38, and a multiplexing unit 40.
　取得部３２は、複数の入力画像を含む入力画像群を取得する。入力画像は、ある一つのシーンや撮像対象を、様々な位置、角度から撮像した画像である。入力画像は、必ずしも一般的なカメラで撮像した画像に限らない。例えば、３６０度カメラで撮像した画像を含んでいてもよい。また、入力画像として、必ずしも実際のカメラで撮像した画像を用いなくともよい。例えば、入力画像は、仮想的なカメラの画像であってもよい。仮想的なカメラの画像とは、例えば、実際のカメラで撮像した画像から合成された画像である。また、仮想的なカメラの画像とは、コンピュータグラフィックで描画された画像であってもよい。 The acquisition unit 32 acquires an input image group including a plurality of input images. The input image is an image obtained by capturing an image of a certain scene or imaging target from various positions and angles. The input image is not necessarily limited to an image captured by a general camera. For example, an image captured by a 360 degree camera may be included. In addition, an image captured by an actual camera is not necessarily used as the input image. For example, the input image may be a virtual camera image. The virtual camera image is, for example, an image synthesized from an image captured by an actual camera. The virtual camera image may be an image drawn by computer graphics.
　また、取得部３２は、複数の入力画像の各々に対応するカメラパラメータも取得する。カメラパラメータには、少なくとも、カメラの向きを示す情報、カメラの位置を示す情報、カメラの画角を示す情報、画像の解像度を示す情報が含まれる。取得されたカメラパラメータは、対応する入力画像に付随される。言い換えると、入力画像には、必ず、対応するカメラパラメータが存在している。 Also, the acquisition unit 32 acquires camera parameters corresponding to each of the plurality of input images. The camera parameters include at least information indicating the direction of the camera, information indicating the position of the camera, information indicating the angle of view of the camera, and information indicating the resolution of the image. The acquired camera parameters are attached to the corresponding input image. In other words, there is always a corresponding camera parameter in the input image.
　図３は、入力画像を撮像する状況の一例を示している。図３では、注目領域５００を、実際のカメラ１００と、実際のカメラ２００で撮像している。さらに、カメラ１００の撮像画像と、カメラ２００の撮像画像から、仮想的なカメラ３００の画像を合成している。このとき、入力画像群には、カメラ１００で注目領域５００を撮像した画像、カメラ２００で注目領域５００を撮像した画像、及び仮想的なカメラ３００の画像を含むことができる。 FIG. 3 shows an example of a situation where an input image is captured. In FIG. 3, the attention area 500 is captured by the actual camera 100 and the actual camera 200. Further, the virtual camera 300 image is synthesized from the captured image of the camera 100 and the captured image of the camera 200. At this time, the input image group can include an image of the attention area 500 captured by the camera 100, an image of the attention area 500 captured by the camera 200, and an image of the virtual camera 300.
　また、図３に示す場合においては、カメラ１００、カメラ２００、及びカメラ３００の画像には、非注目物となり得る物体として作業者４００が含まれる。 In the case shown in FIG. 3, the images of the camera 100, the camera 200, and the camera 300 include the worker 400 as an object that can be a non-target object.
　基準画像選択部３４は、取得部３２が取得した入力画像群から、非注目物に関する補完処理の対象となる画像を選択し、基準画像とする。基準画像選択部３４は、複数の基準画像を選択することができる。基準画像の選択は、例えばユーザが選択することで行われてもよいし、入力画像群に含まれる全ての入力画像を基準画像として選択してもよい。また、後述する注目領域補完データ生成部３６において、注目領域補完データを生成可能かどうか判断し、注目領域補完データを生成可能な入力画像を、基準画像として選択してもよい。基準画像選択部３４は、基準画像を注目領域補完データ生成部３６、及び多重化部４０に出力する。 The reference image selection unit 34 selects an image to be subjected to a complementing process related to a non-target object from the input image group acquired by the acquisition unit 32 and sets it as a reference image. The reference image selection unit 34 can select a plurality of reference images. The selection of the reference image may be performed, for example, by the user selecting, or all input images included in the input image group may be selected as the reference image. In addition, the attention area complementation data generation unit 36 to be described later may determine whether attention area complementation data can be generated, and select an input image that can generate attention area complementation data as a reference image. The reference image selection unit 34 outputs the reference image to the attention area complement data generation unit 36 and the multiplexing unit 40.
　また、基準画像選択部３４は、基準画像として選択された入力画像に対応するカメラパラメータを、基準画像に付随する。上記カメラパラメータに含まれる、カメラの位置を、基準視点位置と呼称する。換言すれば、基準視点位置は、基準画像が実際のカメラで撮像した画像であれば、撮像時のカメラの位置である。また、基準視点位置は、基準画像が実際のカメラで撮像した画像から合成された画像であれば、仮想的な視点の位置である。 Also, the reference image selection unit 34 attaches camera parameters corresponding to the input image selected as the reference image to the reference image. The camera position included in the camera parameters is referred to as a reference viewpoint position. In other words, the reference viewpoint position is the position of the camera at the time of imaging if the reference image is an image captured by an actual camera. The reference viewpoint position is a virtual viewpoint position if the reference image is an image synthesized from an image captured by an actual camera.
　注目領域補完データ生成部３６は、注目領域補完データを生成する。注目領域補完データは、基準画像に映されている非注目物を補完するために用いられるデータである。ここで、補完とは、非注目物を透明もしくは半透明にしたような画像を生成する処理を指す。換言すれば、補完とは、非注目物の透明度が調整された画像を生成する処理を指す。注目領域補完データ生成部３６は、基準画像が複数ある場合、基準画像毎に、対応する注目領域補完データを生成する。本実施形態では、注目領域補完データとして、補完差分画像を作成する。 The attention area complement data generation unit 36 generates attention area complement data. The attention area complement data is data used to complement a non-attention object shown in the reference image. Here, the term “complement” refers to processing for generating an image in which a non-target object is made transparent or translucent. In other words, complement refers to a process of generating an image in which the transparency of a non-target object is adjusted. When there are a plurality of reference images, the attention area complement data generation unit 36 generates corresponding attention area complement data for each reference image. In the present embodiment, a complementary difference image is created as attention area complementary data.
　注目領域補完データ生成部３６は、まず、非注目物選択部５４において非注目物として選択され得る物体である非注目物候補を設定する。注目領域補完データ生成部３６は、まず、当該非注目物候補を透明にしたような画像、換言すれば非注目物候補の透明度が調整された画像である補完画像を生成する。補完画像の合成方法は問わない。例えば、基準視点位置とは別の位置から撮像された入力画像に対して、視点変換処理を適用することにより、補完画像を生成してもよい。 The attention area complement data generation unit 36 first sets a non-target object candidate that is an object that can be selected as a non-target object in the non-target object selection unit 54. The attention area complement data generation unit 36 first generates an image that makes the non-target object candidate transparent, in other words, a complementary image that is an image in which the transparency of the non-target object candidate is adjusted. The method for synthesizing the complementary image is not limited. For example, a complementary image may be generated by applying a viewpoint conversion process to an input image captured from a position different from the reference viewpoint position.
　非注目物候補の選択は、ユーザの選択によってもよいし、補完画像を生成可能な全ての物体を、非注目物候補に選択してもよい。また、複数の入力画像の相関を算出し、相関が低い領域に存在する物体を、非注目物候補と選択してもよい。 The selection of a non-target object candidate may be performed by the user's selection, or all objects that can generate a complementary image may be selected as a non-target object candidate. Further, the correlation between a plurality of input images may be calculated, and an object existing in a region having a low correlation may be selected as a non-target object candidate.
　続いて、注目領域補完データ生成部３６は、補完画像と基準画像の差分を取ることで、補完差分画像を生成する。 Subsequently, the attention area complementary data generation unit 36 generates a complementary difference image by taking the difference between the complementary image and the reference image.
　より具体的には、注目領域補完データ生成部３６は、基準画像と、対応する補完画像との差分画像から、画素値が閾値以下である領域を抜き出すことにより補完差分画像を生成する。閾値は任意に設定されてもよい。言い変えると、注目領域補完データ生成部３６は、補完画像から、非注目物候補が存在する領域を囲むように抜き出して補完差分画像を生成する。なお、一枚の基準画像に対し、複数の非注目物候補が存在する場合、補完差分画像は、非注目物候補毎に別個に生成される。 More specifically, the attention area complement data generation unit 36 generates a complementary difference image by extracting an area having a pixel value equal to or less than a threshold value from the difference image between the reference image and the corresponding complement image. The threshold value may be set arbitrarily. In other words, the region-of-interest supplement data generation unit 36 extracts a complemented difference image from the complemented image so as to surround the region where the non-target object candidate exists. When there are a plurality of non-target object candidates for one reference image, a complementary difference image is generated separately for each non-target object candidate.
　なお、以下の説明では、混乱を生じない限り、「非注目物候補」と「非注目物」とを特に区別せずに単に「非注目物」と表記することもある。 In the following description, unless a confusion arises, the “non-attention object candidate” and the “non-attention object” may be simply described as “non-attention object”.
　＜補完差分画像＞
　図４を参照して、補完差分画像の具体例を説明する。 <Complementary difference image>
A specific example of the complementary difference image will be described with reference to FIG.
　図４の（ａ）は、基準画像の一例であり、「基準画像Ａ」と表す。また図４の（ｃ）は、基準画像の一例であり、「基準画像Ｂ」と表す。基準画像Ａ及び基準画像Ｂは、それぞれ異なる準視点位置から、斜線領域を撮像した画像である。基準画像Ａは、座標（ｘ１，ｙ１）を始点とする領域に、非注目物である白雲を含む。なお、ここでは、対象の領域における左上端の点を「始点」とよぶ。この基準画像Ａに対応する補完差分画像の一例を図４の（ｂ）に示し、「補完差分画像１」と表す。 (A) of FIG. 4 is an example of a reference image and is represented as “reference image A”. FIG. 4C is an example of a reference image and is represented as “reference image B”. The reference image A and the reference image B are images obtained by capturing the hatched area from different quasi-viewpoint positions. The reference image A includes a white cloud that is a non-target object in an area starting from the coordinates (x1, y1). Here, the upper left point in the target area is referred to as a “start point”. An example of the complementary difference image corresponding to the reference image A is shown in FIG. 4B and is represented as “complementary difference image 1”.
　図４の（ａ）に示す基準画像Ａにおいて、灰色領域の右下の一部が、非注目物である白雲によって遮蔽されている。一方で、図４の（ｂ）に示す補完差分画像１には、図４の（ａ）において白雲によって遮蔽されていた灰色領域の上記一部が含まれている。換言すれば、図４の（ｂ）に示される図中の灰色領域は、非注目物である白雲に遮蔽されていない画像である。 In the reference image A shown in FIG. 4 (a), the lower right part of the gray area is shielded by a white cloud that is a non-target object. On the other hand, the complementary differential image 1 shown in FIG. 4B includes the above-described part of the gray region that is shielded by the white cloud in FIG. In other words, the gray region in the drawing shown in FIG. 4B is an image that is not shielded by a white cloud that is a non-target object.
　補完差分画像１には、対応する基準画像が「基準画像Ａ」であることを示す情報、基準画像Ａ上の対応位置を示す補完画像座標情報が（ｘ１，ｙ１）であることを示す情報、及び非注目物の名称が「白雲」であることを示す情報が付随される。 The complementary difference image 1 includes information indicating that the corresponding reference image is “reference image A”, information indicating that the complementary image coordinate information indicating the corresponding position on the reference image A is (x1, y1), And information indicating that the name of the non-attention object is “baiyun”.
　図４の（ｃ）の基準画像Ｂは、座標（ｘ２，ｙ２）を始点とする領域に、非注目物である灰色雲を含む。この基準画像Ｂに対応する補完差分画像の一例を図４の（ｄ）に示し、「補完差分画像２」と表す。 4C includes a gray cloud that is a non-attention object in a region starting from coordinates (x2, y2). An example of the complementary difference image corresponding to the reference image B is shown in FIG. 4D and is represented as “complementary difference image 2”.
　図４の（ｃ）に示す基準画像Ｂにおいて、斜線領域の左下の一部が、非注目物である灰色雲によって遮蔽されている。一方で、図４の（ｄ）に示す補完差分画像２には、図４の（ｃ）において灰色雲によって遮蔽されていた斜線領域の上記一部が含まれている。換言すれば、図４の（ｄ）に示される図中の斜線領域は、非注目物である灰色雲に遮蔽されていない画像である。 In the reference image B shown in FIG. 4 (c), the lower left part of the hatched area is shielded by a gray cloud that is a non-target object. On the other hand, the complementary difference image 2 shown in (d) of FIG. 4 includes the part of the shaded area shielded by the gray clouds in (c) of FIG. In other words, the hatched area in the drawing shown in FIG. 4D is an image that is not shielded by a gray cloud that is a non-target object.
　補完差分画像２には、対応する基準画像が「基準画像Ｂ」であることを示す情報、基準画像Ｂ上の対応位置を示す補完画像座標情報が（ｘ２，ｙ２）であることを示す情報、及び非注目物の名称が「灰色雲」であることを示す情報が付随される。また、補完差分画像は、各画素の不透明度を表すアルファ値から構成されるアルファチャンネル情報を有する。 The complementary difference image 2 includes information indicating that the corresponding reference image is “reference image B”, information indicating that the complementary image coordinate information indicating the corresponding position on the reference image B is (x2, y2), And information indicating that the name of the non-attention object is “gray cloud”. The complementary difference image has alpha channel information composed of alpha values representing the opacity of each pixel.
　図４に示すように、補完差分画像は、基準画像よりも小さい。したがって、補完差分画像が複数存在する場合にも、生成処理を比較的軽い処理で行うことができ、また、格納するデータ量を抑えられる。 As shown in FIG. 4, the complementary difference image is smaller than the reference image. Therefore, even when there are a plurality of complementary difference images, the generation process can be performed with a relatively light process, and the amount of data to be stored can be suppressed.
　図５を参照して、補完差分画像の生成過程の具体例を示す。図５の（ａ）は、基準画像の一例であり、非注目物に選択されたブックエンド６００ａ、及び薬瓶６００ｂを含んでいる。この基準画像に対する補完画像は、図５の（ｂ）に示す様に、非注目物が透明になったような画像である。注目領域補完データ生成部３６は、図５の（ａ）に示される基準画像、及び図５の（ｂ）に示される補完画像から、図５の（ｃ）に示される非注目物に選択された薬瓶６００ｂに対応する補完差分画像１、及び図５の（ｄ）に示される非注目物に選択されたブックエンド６００ａに対応する補完差分画像２を生成する。 Referring to FIG. 5, a specific example of a complementary difference image generation process is shown. (A) of FIG. 5 is an example of a reference image, and includes a book end 600a selected as a non-target object and a medicine bottle 600b. The complementary image with respect to the reference image is an image in which a non-attention object becomes transparent as shown in FIG. The attention area complement data generation unit 36 is selected from the reference image shown in FIG. 5A and the complement image shown in FIG. 5B as the non-target object shown in FIG. The complementary difference image 1 corresponding to the medicine bottle 600b and the complementary difference image 2 corresponding to the book end 600a selected as the non-target object shown in FIG. 5D are generated.
　三次元情報取得部３８は、撮像された状況の三次元情報を計算する。三次元情報の代表例として、デプスマップを挙げることができる。 The 3D information acquisition unit 38 calculates 3D information of the captured situation. A typical example of the three-dimensional information is a depth map.
　ここで、デプスマップとは、基準画像の各領域について、その奥行き値を示す情報である。基準画像の各領域とは、例えば基準画像の画素単位である。本実施形態では、デプスマップに含まれる奥行き値は、基準視点位置を基準（原点）とする。また、デプスマップの各領域の奥行きの方向は平行であり、放射状ではないものとする。デプスマップは、例えば、ステレオマッチングにより計算してもよい。すなわち、基準画像を含む複数の入力画像と、当該入力画像のカメラパラメータから当該基準画像のデプスマップを計算してもよい。なお、三枚以上の入力画像を利用して、デプスマップを計算してもよい。また、デプスマップは、例えば、赤外線センサ等の距離センサを利用して計測されてもよい。 Here, the depth map is information indicating the depth value of each region of the reference image. Each region of the reference image is, for example, a pixel unit of the reference image. In the present embodiment, the depth value included in the depth map is based on the reference viewpoint position (origin). Further, it is assumed that the depth direction of each region of the depth map is parallel and not radial. For example, the depth map may be calculated by stereo matching. That is, a depth map of the reference image may be calculated from a plurality of input images including the reference image and camera parameters of the input image. Note that the depth map may be calculated using three or more input images. Further, the depth map may be measured using a distance sensor such as an infrared sensor, for example.
　なお、デプスマップとして、一つの基準画像に対して、全ての非注目物を透明にした場合のデプスマップと、全ての非注目物が映されている場合のデプスマップの二種類が計算される。以降は、前者を「非注目物を含まないデプスマップ」、後者を「非注目物を含むデプスマップ」と呼ぶ。 Two types of depth maps are calculated for a single reference image: a depth map when all non-attention objects are made transparent, and a depth map when all non-attention objects are shown. . Hereinafter, the former is referred to as “a depth map that does not include a non-target object”, and the latter is referred to as a “depth map that includes a non-target object”.
　多重化部４０は、基準画像、補完差分画像、及び三次元情報を多重化する。基準画像と補完差分画像は、各々に付随されている情報も合わせて多重化される。 The multiplexing unit 40 multiplexes the reference image, the complementary difference image, and the three-dimensional information. The reference image and the complementary difference image are multiplexed together with information attached to each.
　＜注目画像合成部＞
　続いて、図６を参照して、注目画像合成部５０の構成例について説明する。 <Attention image composition unit>
Next, a configuration example of the attention image synthesis unit 50 will be described with reference to FIG.
　図６に示すように、注目画像合成部５０は、逆多重化部５２、非注目物選択部５４、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０を備えている。 As illustrated in FIG. 6, the attention image synthesis unit 50 includes a demultiplexing unit 52, a non-target object selection unit 54, a complementary difference image processing unit 56, a display method acquisition unit 58, and a attention image generation unit 60. .
　逆多重化部５２は、注目画像データを格納部２０から取得し、取得した注目画像データを逆多重化する。逆多重化部５２は、注目画像データに含まれる基準画像と、その付随情報を、注目画像生成部６０へ出力する。また、注目画像データに含まれる三次元情報及び注目領域補完データを非注目物選択部５４へ出力する。 The demultiplexing unit 52 acquires the target image data from the storage unit 20, and demultiplexes the acquired target image data. The demultiplexing unit 52 outputs the reference image included in the target image data and the accompanying information to the target image generating unit 60. Further, the three-dimensional information and the attention area complement data included in the attention image data are output to the non-attention object selection unit 54.
　非注目物選択部５４は、補完差分画像に付随する情報、または三次元情報に基づいて、基準画像に映されている物体の少なくとも何れかを非注目物として選択する。非注目物選択の具体例は後述する。 The non-target object selection unit 54 selects at least one of the objects shown in the reference image as the non-target object based on the information accompanying the complementary difference image or the three-dimensional information. A specific example of non-target object selection will be described later.
　非注目物選択部５４は、非注目物が含まれていると判断した補完差分画像のみを、選択済補完差分画像として補完差分画像処理部５６に出力する。 The non-attention object selection unit 54 outputs only the complementary difference image determined to contain a non-attention object to the complementary difference image processing unit 56 as the selected complementary difference image.
　＜非注目物の選択例＞
　以下では、図７を参照して、非注目物選択部５４による非注目物の選択例について説明する。 <Selection example of non-attention object>
In the following, with reference to FIG. 7, an example of non-target object selection by the non-target object selection unit 54 will be described.
　（選択例１）
　非注目物選択部５４に非注目物として選択される物体の例として、基準視点位置の後方に位置する物体が挙げられる。非注目物選択部５４は、基準視点位置を示す情報、基準画像における非注目物を含まないデプスマップ、及び基準画像における後述する補完領域のデプスマップを参照して、物体が非注目物か否かを判定することができる。 (Selection example 1)
As an example of the object selected as the non-attention object by the non-attention object selection unit 54, an object positioned behind the reference viewpoint position can be cited. The non-target object selection unit 54 refers to information indicating the reference viewpoint position, a depth map that does not include the non-target object in the reference image, and a depth map of a complementary region described later in the reference image to determine whether the object is a non-target object. Can be determined.
　図７の（ａ）に示されるように、基準視点位置を原点として、注目領域に向かう方向を正とした場合、基準視点位置よりも注目領域側の領域のデプスマップの奥行き値は正の値となる。したがって、補完領域のデプスマップの奥行き値が、全て負の値であれば、基準視点よりも後方に存在すると言える。非注目物選択部５４は、この条件に当てはまる物体、すなわち、デプスマップの奥行き値が全て負の値である物体を、非注目物であると判定する。換言すれば、非注目物選択部５４は、図７の（ａ）に示す透過範囲内に存在する物体を、非注目物であると判定する。そして、非注目物選択部５４は、当該非注目物を補完する補完差分画像を、補完差分画像処理部５６に出力する。 As shown in FIG. 7A, when the reference viewpoint position is the origin and the direction toward the attention area is positive, the depth value of the depth map of the area on the attention area side with respect to the reference viewpoint position is a positive value. It becomes. Therefore, if the depth values of the depth map of the complementary region are all negative values, it can be said that the depth map exists behind the reference viewpoint. The non-attention object selection unit 54 determines that an object that satisfies this condition, that is, an object whose depth values of the depth map are all negative values is a non-attention object. In other words, the non-attention object selection unit 54 determines that an object that exists within the transmission range illustrated in FIG. Then, the non-target object selection unit 54 outputs a complementary difference image that complements the non-target object to the complementary difference image processing unit 56.
　図７の（ａ）に示される例において、基準視点位置の画像は、基準視点位置よりも後方に位置する２台のカメラ（カメラＡ、カメラＢ）により撮像された各々の画像から生成される。基準視点位置よりも後方であって、注目領域から非注目物に向かう延長線上に位置するカメラＡで撮像した画像には非注目物が含まれるが、基準視点位置よりも後方ではあるが、注目領域から非注目物に向かう延長線上には位置しないカメラＢで撮像した画像には非表示物が含まれない。本例では、カメラＡで撮像した画像を、カメラＢで撮像した画像で補完することにより、基準視点位置の画像を生成することができる。 In the example shown in FIG. 7A, the image at the reference viewpoint position is generated from each image captured by two cameras (camera A and camera B) located behind the reference viewpoint position. . The image captured by the camera A located behind the reference viewpoint position and on the extended line from the attention area to the non-target object includes the non-target object, but is behind the reference viewpoint position, Non-display objects are not included in the image captured by the camera B that is not located on the extension line from the region toward the non-target object. In this example, the image of the reference viewpoint position can be generated by complementing the image captured by the camera A with the image captured by the camera B.
　（選択例２）
　また、非注目物選択部５４に非注目物として選択される物体の例として、基準視点位置の周辺領域と重なる物体が挙げられる。非注目物選択部５４は、基準視点位置を示す情報、基準画像における非注目物を含まないデプスマップ、及び基準画像における補完領域のデプスマップを参照して、物体が非注目物か否かを判定することができる。図７の（ｂ）に示されるように、非注目物選択部５４は、基準視点位置を原点として、補完領域のデプスマップの奥行き値に、一定の割合以下、例えば最大奥行き値の±１０％以下の奥行き値が含まれている物体を非注目物であると判定する。換言すれば、非注目物選択部５４は、図７の（ｂ）に示す透過範囲内に存在する物体を、非注目物であると判定する。そして、非注目物選択部５４は、当該非注目物を補完する補完差分画像を、補完差分画像処理部５６に出力する。 (Selection example 2)
An example of an object that is selected as a non-target object by the non-target object selection unit 54 is an object that overlaps the peripheral area of the reference viewpoint position. The non-target object selection unit 54 refers to information indicating the reference viewpoint position, a depth map that does not include the non-target object in the reference image, and a depth map of the complementary region in the reference image to determine whether the object is a non-target object. Can be determined. As shown in (b) of FIG. 7, the non-target object selection unit 54 uses the reference viewpoint position as the origin, and the depth value of the depth map of the complementary region is equal to or less than a certain ratio, for example, ± 10% of the maximum depth value. An object including the following depth values is determined as a non-attention object. In other words, the non-target object selection unit 54 determines that an object that exists within the transmission range illustrated in FIG. 7B is a non-target object. Then, the non-target object selection unit 54 outputs a complementary difference image that complements the non-target object to the complementary difference image processing unit 56.
　なお、非注目物選択部５４は、割合ではなく定数値で判断してもよい。例えば、非注目物選択部５４は、補完領域のデプスマップの奥行きが、基準視点位置を起点として奥行き値の絶対値が一定以下であれば、当該補完領域に対応する非注目物候補を非注目物だと判断することができる。また、例えば、補完領域のデプスマップの奥行きが、基準視点位置を中心とした、一定の半径を持つ球の中に含まれていれば、当該補完領域に対応する非注目物候補を非注目物だと判断することができる。 It should be noted that the non-attention object selection unit 54 may make a determination based on a constant value instead of a ratio. For example, if the depth of the depth map of the complement region is the reference viewpoint position and the absolute value of the depth value is less than or equal to a certain value, the non-target object selection unit 54 does not pay attention to the non-target object candidate corresponding to the complement region. It can be judged as a thing. For example, if the depth of the depth map of the complementary region is included in a sphere with a certain radius centered on the reference viewpoint position, the non-target object candidate corresponding to the complementary region is selected as the non-target object. It can be judged.
　（選択例３）
　また、非注目物選択部５４に非注目物として選択される物体の例として、特定の非注目物を挙げることができる。非注目物選択部５４は、補完差分画像に付随する情報に基づき、非注目物か否かを判断する。非注目物選択部５４は、上記付随する情報に含まれる非注目物候補の名称が、事前に指定された物体である場合、当該非注目物を補完する補完差分画像を補完差分画像処理部５６に出力する。 (Selection Example 3)
Further, as an example of the object selected as the non-target object by the non-target object selection unit 54, a specific non-target object can be cited. The non-attention object selection unit 54 determines whether or not the object is a non-attention object based on information accompanying the complementary difference image. When the name of the non-target object candidate included in the accompanying information is an object designated in advance, the non-target object selection unit 54 displays a complementary difference image that complements the non-target object as a complementary difference image processing unit 56. Output to.
　（選択例４）
　また、非注目物選択部５４に非注目物として選択される物体の例として、注目領域より手前にある全ての物体を挙げることができる。非注目物選択部５４は、基準視点位置を示す情報、基準画像における非注目物を含まない領域のデプスマップ、及び、基準画像における補完領域のデプスマップを参照して、物体が非注目物か否かを判断することができる。図７の（ｃ）に示されるように、基準視点位置を原点として、注目領域に向かう方向を正とする。この場合、非注目物選択部５４は、補完領域のデプスマップの全ての奥行き値が、非注目物の無いデプスマップより小さければ、補完差分画像に非注目物が含まれていると判定する。換言すれば、非注目物選択部５４は、図７の（ｃ）に示す透過範囲内に存在する物体を、非注目物であると判定する。そして、非注目物選択部５４は、当該非注目物を補完する補完差分画像を、補完差分画像処理部５６に出力する。 (Selection Example 4)
In addition, examples of objects selected as non-target objects by the non-target object selection unit 54 include all objects that are in front of the target area. The non-target object selection unit 54 refers to the information indicating the reference viewpoint position, the depth map of the region not including the non-target object in the reference image, and the depth map of the complementary region in the reference image to determine whether the object is a non-target object. It can be determined whether or not. As shown in FIG. 7C, the direction toward the region of interest is positive with the reference viewpoint position as the origin. In this case, the non-target object selection unit 54 determines that the non-target object is included in the complementary difference image if all the depth values of the depth map of the complementary region are smaller than the depth map without the non-target object. In other words, the non-attention object selection unit 54 determines that an object existing within the transmission range illustrated in (c) of FIG. 7 is a non-attention object. Then, the non-target object selection unit 54 outputs a complementary difference image that complements the non-target object to the complementary difference image processing unit 56.
　（選択例５）
　また、非注目物選択部５４は、基準視点位置の画像内に占める非注目物候補の領域が所定割合以上の場合、非注目物として選択してもよい。所定割合は任意の値とする。例えば、所定割合を５割に設定したとして、図７の（ａ）に示す例において、後方のカメラが撮像した画像において、非注目物候補の領域が占める割合が２割であったとしても、基準視点位置の画像において、非注目物候補の領域が占める割合が５割であった場合、非注目物候補は非注目物であると判定し、当該非注目物を補完する補完差分画像を、補完差分画像処理部５６に出力する。 (Selection Example 5)
Further, the non-target object selection unit 54 may select a non-target object when the region of the non-target object candidate in the image at the reference viewpoint position is equal to or greater than a predetermined ratio. The predetermined ratio is an arbitrary value. For example, assuming that the predetermined ratio is set to 50%, in the example shown in FIG. 7A, even if the ratio of the non-target object candidate area in the image captured by the rear camera is 20%, In the image of the reference viewpoint position, when the ratio of the non-target object candidate area is 50%, it is determined that the non-target object candidate is a non-target object, and a complementary difference image that complements the non-target object is The result is output to the complementary difference image processing unit 56.
　＜補完領域のデプスマップ＞
　基準画像内の非注目物候補が存在する領域のことを補完領域とよぶ。言い変えると、補完領域は、当該補完画像によって補完される、基準画像の内領域を指す。補完領域は、基準画像に対応する補完差分画像と、補完差分画像に付随する補完画像座標情報から計算される。 <Depth map of complementary area>
A region where a non-target object candidate exists in the reference image is called a complementary region. In other words, the complementary region refers to the inner region of the reference image that is complemented by the complementary image. The complementary region is calculated from the complementary difference image corresponding to the reference image and the complementary image coordinate information attached to the complementary difference image.
　また、補完領域のデプスマップは、補完領域と、非注目物のあるデプスマップから得られる情報であり、補完差分画像に対応する非注目物の奥行き値を示す情報である。即ち、基準画像に対応する非注目物のあるデプスマップから、補完領域に該当する箇所のデプスマップを切りだした情報を指す。 Further, the depth map of the complementary region is information obtained from the complementary region and the depth map with the non-target object, and is information indicating the depth value of the non-target object corresponding to the complementary difference image. That is, it refers to information obtained by cutting out a depth map corresponding to a complementary region from a depth map having a non-target object corresponding to the reference image.
　表示方法取得部５８は、ユーザにより選択、調整された表示方法を取得する。また、補完差分画像処理部５６は、表示方法取得部５８から、補完に関する表示方法を示す情報を取得する。補完差分画像処理部５６は、補完差分画像に対し、表示条件に応じた処理を施し、処理を施した補完差分画像を処理済補完差分画像として注目画像生成部６０に出力する。 The display method acquisition unit 58 acquires the display method selected and adjusted by the user. Further, the complementary difference image processing unit 56 acquires information indicating a display method related to complementation from the display method acquiring unit 58. The complementary difference image processing unit 56 performs processing according to the display condition on the complementary difference image, and outputs the processed complementary difference image to the attention image generation unit 60 as a processed complementary difference image.
　＜表示方法とアルファ値の例＞
　本実施形態に係る表示方法は、補完領域の表示方法であり、ユーザにより選択、調整される。以下では、本実施形態に係る表示方法の例について具体的に説明する。 <Example of display method and alpha value>
The display method according to the present embodiment is a complementary region display method, and is selected and adjusted by the user. Below, the example of the display method which concerns on this embodiment is demonstrated concretely.
　（表示方法例１）
　第１の例として、特定の物体を非注目物とするか否か、例えば、基準画像内に映された物体を補完するか否かを切り替えて表示する表示方法を挙げることができる。補完差分画像処理部５６は、特定の物体を非注目物として補完する場合は、対応する補完差分画像の全体を不透明に設定する。換言すれば、対応する補完差分画像の全体のアルファ値を１にする。同様に、補完をしない場合は、対応する補完差分画像の全体のアルファ値を０にする。 (Display method example 1)
As a first example, there can be mentioned a display method for switching and displaying whether or not a specific object is a non-target object, for example, whether or not to complement an object shown in a reference image. When complementing a specific object as a non-target object, the complementary difference image processing unit 56 sets the entire corresponding complementary difference image to be opaque. In other words, the overall alpha value of the corresponding complementary difference image is set to 1. Similarly, when not complementing, the entire alpha value of the corresponding complementary difference image is set to zero.
　（表示方法例２）
　第２の例として、表示画像内の指定した領域のみを補完する、例えば、基準画像内の非注目物の一部分のみを補完する表示方法を挙げることができる。補完差分画像処理部５６は、指定された領域に対応する補完差分画像の対応する領域のアルファ値のみを１にすることで、指定した領域のみを補完することができる。 (Display method example 2)
As a second example, a display method that complements only a specified region in the display image, for example, complements only a part of a non-target object in the reference image can be mentioned. The complementary difference image processing unit 56 can complement only the designated region by setting only the alpha value of the corresponding region of the complementary difference image corresponding to the designated region to 1.
　（表示方法例３）
　非注目物の透明度を設定する、例えば、基準画像内に映り込んだ非注目物を半透明にする。補完差分画像処理部５６は、設定された透明度に応じて補完差分画像の全体のアルファ値を変更することで、非注目領域の透明度を設定することができる。 (Display method example 3)
The transparency of the non-target object is set, for example, the non-target object reflected in the reference image is made translucent. The complementary difference image processing unit 56 can set the transparency of the non-attention area by changing the overall alpha value of the complementary difference image according to the set transparency.
　（表示方法例４）
　画像処理システム１に、ユーザの視野を特定する構成を設け、ユーザが、３６０度カメラで撮像された画像の一部を観察している場合に、補完差分画像処理部５６が、視界内にある非注目物のみを補完する表示方法を挙げることができる。補完差分画像処理部５６は、上記視界内にある非注目物に対応した補完差分画像の全体のアルファ値を１、上記以外の非注目物に対応した補完差分画像の全体のアルファ値を０にすることで、指定した領域のみを補完することができる。 (Display method example 4)
The image processing system 1 is provided with a configuration for identifying the user's field of view, and the complementary difference image processing unit 56 is in the field of view when the user observes a part of the image captured by the 360 degree camera. The display method which complements only a non-attention object can be mentioned. The complementary difference image processing unit 56 sets the entire alpha value of the complementary difference image corresponding to the non-target object in the field of view to 1, and sets the entire alpha value of the complementary difference image corresponding to the non-target object other than the above to 0. By doing so, only the specified area can be complemented.
　補完差分画像処理部５６は、上記処理の完了した補完差分画像を、注目画像生成部６０へ出力する。 The complementary difference image processing unit 56 outputs the complementary difference image for which the above processing has been completed to the attention image generating unit 60.
　注目画像生成部６０は、逆多重化部５２から供給される基準画像と、補完差分画像処理部５６から供給される処理済補完差分画像とを合成し、非注目物の透明度が調整された注目画像を生成する。 The attention image generation unit 60 synthesizes the reference image supplied from the demultiplexing unit 52 and the processed complementary difference image supplied from the complementary difference image processing unit 56, and the attention of which the transparency of the non-target object is adjusted. Generate an image.
　より具体的には、注目画像生成部６０は、補完差分画像を、基準画像に対し、基準画像の補完画像座標情報に基づいた位置に合成する。すなわち、補完差分画像は、基準画像に対して、基準画像に対応する補完画像内の、補完差分画像を抜き出した位置に合成される。 More specifically, the attention image generation unit 60 synthesizes the complementary difference image with the reference image at a position based on the complementary image coordinate information of the reference image. That is, the complementary difference image is synthesized with respect to the reference image at a position where the complementary difference image is extracted in the complementary image corresponding to the reference image.
　このように、注目画像生成部６０は、基準画像内の座標情報である補完画像座標情報によって定まる位置に補完差分画像を合成するので、補完差分画像が、基準画像よりも小さい場合であっても、好適に合成処理を行うことができる。 In this way, the attention image generation unit 60 synthesizes the complementary difference image at a position determined by the complementary image coordinate information that is the coordinate information in the reference image, so even if the complementary difference image is smaller than the reference image. Therefore, it is possible to suitably perform the synthesis process.
　注目画像生成部６０は、基準画像と補完差分画像とをアルファブレンドする。この実施形態において、注目画像生成部６０は、基準画像のアルファ値を、補完差分画像のアルファ値に応じて設定する。具体的には、注目画像生成部６０は、基準画像の各画素のアルファ値は、補完差分画像の対応する各画素のアルファ値を、１から引いた値に設定される。 The attention image generation unit 60 alpha blends the reference image and the complementary difference image. In this embodiment, the attention image generation unit 60 sets the alpha value of the reference image according to the alpha value of the complementary difference image. Specifically, the attention image generation unit 60 sets the alpha value of each pixel of the reference image to a value obtained by subtracting from 1 the alpha value of each corresponding pixel of the complementary difference image.
　＜注目画像合成部が実行する処理＞
　注目画像合成部５０が実行する処理の流れの一例を、図８に示すフローチャート、及び図９に示す処理フローに基づいて説明する。 <Processing executed by attention image composition unit>
An example of the flow of processing executed by the attention image synthesis unit 50 will be described based on the flowchart shown in FIG. 8 and the processing flow shown in FIG.
　ステップＳ１０では、非注目物選択部５４は、逆多重化部５２から補完差分画像の１つを取得する。 In step S <b> 10, the non-target object selection unit 54 acquires one of the complementary difference images from the demultiplexing unit 52.
　ステップＳ１２では、非注目物選択部５４は、逆多重化部５２から基準画像の三次元情報のうちステップＳ１０で取得された補完差分画像に対応する補完領域のデプスマップを取得する。 In step S12, the non-target object selection unit 54 acquires the depth map of the complementary region corresponding to the complementary difference image acquired in step S10 from the three-dimensional information of the reference image from the demultiplexing unit 52.
　ステップＳ１４では、非注目物選択部５４は、三次元情報を参照して、補完差分画像に含まれる物体が、非注目物か否かを判定する。非注目物である場合（Ｓ１４でＹＥＳ）、ステップＳ１６の処理に進む。非注目物ではない場合（Ｓ１４でＮＯ）、Ｓ２０に進む。 In step S14, the non-attention object selection unit 54 refers to the three-dimensional information and determines whether or not the object included in the complementary difference image is a non-attention object. If it is a non-attention object (YES in S14), the process proceeds to step S16. If it is not a non-attention object (NO in S14), the process proceeds to S20.
　図９に示す例においては、非注目物選択部５４は、補完差分画像１については、カメラと薬瓶との距離が近いので、薬瓶を非注目物と判定する。一方で、非注目物選択部５４は、補完差分画像２については、カメラとブックエンドとの距離が遠いので、非注目物ではないと判定する。 In the example shown in FIG. 9, the non-target object selection unit 54 determines that the medicine bottle is a non-target object for the complementary difference image 1 because the distance between the camera and the medicine bottle is short. On the other hand, the non-focused object selection unit 54 determines that the complementary difference image 2 is not a non-focused object because the distance between the camera and the book end is long.
　ステップＳ１６では、表示方法取得部５８は、ユーザの所望の表示方法を示す情報を取得する。 In step S16, the display method acquisition unit 58 acquires information indicating the display method desired by the user.
　ステップＳ１８では、補完差分画像処理部５６は、Ｓ１６で取得された表示方法に基づいて、補完差分画像のアルファ値を変更する。図９に示す例においては、補完差分画像処理部５６は、表示条件に基づいて、補完差分画像１の全画素のアルファ値を０．５に変更する。 In step S18, the complementary difference image processing unit 56 changes the alpha value of the complementary difference image based on the display method acquired in S16. In the example illustrated in FIG. 9, the complementary difference image processing unit 56 changes the alpha value of all the pixels of the complementary difference image 1 to 0.5 based on the display conditions.
　ステップＳ２０では、注目画像合成部５０は、注目画像データに含まれていた全ての補完差分画像についての処理が完了したか否かを判定する。処理が完了したと判定した場合（Ｓ２０でＹＥＳ）、ステップＳ２２の処理に進む。処理が完了していないと判定した場合（Ｓ２０でＮＯ）、Ｓ１０の処理に戻る。 In step S20, the target image composition unit 50 determines whether or not the processing for all the complementary difference images included in the target image data has been completed. If it is determined that the process has been completed (YES in S20), the process proceeds to step S22. If it is determined that the process has not been completed (NO in S20), the process returns to S10.
　ステップＳ２２では、注目画像生成部６０は、基準画像と、非注目物に対応する補完差分画像とから注目画像を生成する。図９に示す例においては、注目画像生成部６０は、基準画像と、アルファ値が０．５に変更された補完差分画像１とを合成する。補完差分画像１のアルファ値は０．５であるので、注目画像において、非注目物の薬瓶は半透明であるため、薬瓶に遮蔽されている物体が透けて見える。 In step S22, the attention image generation unit 60 generates an attention image from the reference image and the complementary difference image corresponding to the non-attention object. In the example illustrated in FIG. 9, the attention image generation unit 60 synthesizes the reference image and the complementary difference image 1 in which the alpha value is changed to 0.5. Since the alpha value of the complementary difference image 1 is 0.5, the non-target medicine bottle is translucent in the attention image, so that an object shielded by the medicine bottle can be seen through.
　ステップＳ２４では、注目画像生成部６０は、生成した注目画像を出力する。 In step S24, the attention image generation unit 60 outputs the generated attention image.
　以上のように、非注目物の選択パターンに応じて予め複数の補完画像を生成し、基準画像と補完画像の差分が発生する領域のみを補完差分画像として格納しておき、画像表示時に、当該複数の補完差分画像を含む注目画像データを用いて補完を行うことで、非注目物を含まない所望の画像を、任意の処理の設定かつ比較的軽い処理で生成できる。また、格納するデータ量を抑えられる。また、処理の設定をフレキシブルに変更可能である。 As described above, a plurality of complementary images are generated in advance according to the non-target object selection pattern, and only a region where a difference between the reference image and the complementary image is generated is stored as a complementary difference image. By performing complementation using attention image data including a plurality of complementary difference images, a desired image that does not include a non-attention object can be generated with an arbitrary process setting and a relatively light process. In addition, the amount of data to be stored can be suppressed. In addition, processing settings can be changed flexibly.
　〔実施形態２〕
　本発明の他の実施形態について、図１０～図１１に基づいて説明すれば、以下のとおりである。なお、説明の便宜上、前記実施形態にて説明した部材と同じ機能を有する部材については、同じ符号を付記し、その説明を省略する。 [Embodiment 2]
The following will describe another embodiment of the present invention with reference to FIGS. For convenience of explanation, members having the same functions as those described in the embodiment are given the same reference numerals, and descriptions thereof are omitted.
　本実施形態は、ユーザに指定された空間上の一点（「仮想視点」とよぶ。「仮想視点」の位置は、空間中の座標で示すことができ、仮想視点の座標を示す情報を仮想視点位置情報とよぶ。）から注目領域を観察しているような画像（「仮想視点画像」とよぶ）をさらに生成する点で、実施形態１と異なる。 In this embodiment, a point on the space designated by the user (referred to as a “virtual viewpoint”. The position of the “virtual viewpoint” can be indicated by coordinates in the space, and information indicating the coordinates of the virtual viewpoint is displayed as the virtual viewpoint. This is different from the first embodiment in that an image (referred to as “virtual viewpoint image”) in which the attention area is observed is further generated from the position information.
　図１０に示すように、本実施形態に係る画像処理システム１ａは、注目画像データ生成部３０、仮想視点画像合成部７０、及び格納部２０を備えている。仮想視点画像合成部７０は、仮想視点の位置を示す情報を取得する。なお、格納部２０、及び注目画像データ生成部３０は、それぞれ、実施形態１に係る画像処理システム１の格納部２０、及び注目画像データ生成部３０に対応する。 As shown in FIG. 10, the image processing system 1 a according to the present embodiment includes an attention image data generation unit 30, a virtual viewpoint image synthesis unit 70, and a storage unit 20. The virtual viewpoint image composition unit 70 acquires information indicating the position of the virtual viewpoint. The storage unit 20 and the attention image data generation unit 30 correspond to the storage unit 20 and the attention image data generation unit 30 of the image processing system 1 according to the first embodiment, respectively.
　図１１に示すように、仮想視点画像合成部７０は、逆多重化部５２、非注目物選択部５４ｂ、補完差分画像処理部５６、表示方法取得部５８、注目画像生成部６０、及び仮想視点画像生成部７２を備えている。なお、逆多重化部５２、非注目物選択部５４ａ、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０は、それぞれ、実施形態１に係る画像処理システム１の逆多重化部５２、非注目物選択部５４、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０に対応する。 As illustrated in FIG. 11, the virtual viewpoint image composition unit 70 includes a demultiplexing unit 52, a non-target object selection unit 54b, a complementary difference image processing unit 56, a display method acquisition unit 58, a target image generation unit 60, and a virtual viewpoint. An image generation unit 72 is provided. Note that the demultiplexing unit 52, the non-target object selection unit 54a, the complementary difference image processing unit 56, the display method acquisition unit 58, and the target image generation unit 60 are each demultiplexed in the image processing system 1 according to the first embodiment. This corresponds to the conversion unit 52, the non-target object selection unit 54, the complementary difference image processing unit 56, the display method acquisition unit 58, and the target image generation unit 60.
　実施形態１の非注目物選択部５４は、基準視点位置を基準として非注目物を選択したが、本実施形態の非注目物選択部５４ａは、基準視点位置に代えて、仮想視点の位置を基準として非注目物を選択する。 The non-target object selection unit 54 of Embodiment 1 selects a non-target object based on the reference viewpoint position, but the non-target object selection unit 54a of the present embodiment replaces the reference viewpoint position with the position of the virtual viewpoint. Select a non-attention object as a reference.
　＜非注目物の選択例＞
　（選択例１）
　非注目物選択部５４に非注目物として選択される物体の例として、仮想視点位置の後方に位置する物体が挙げられる。非注目物選択部５４は、仮想視点位置を示す情報、基準画像における非注目物を含まないデプスマップ、及び基準画像における後述する補完領域のデプスマップを参照して、物体が非注目物か否かを判定することができる。仮想視点位置を原点として、注目領域に向かう方向を正とした場合、仮想視点位置よりも注目領域側の領域のデプスマップの奥行き値は正の値となる。したがって、補完領域のデプスマップの奥行き値が、全て負の値であれば、基準視点よりも後方に存在すると言える。非注目物選択部５４は、この条件に当てはまる物体、すなわち、デプスマップの奥行き値が全て負の値である物体を、非注目物であると判定し、当該非注目物を補完する補完差分画像を、補完差分画像処理部５６に出力する。 <Selection example of non-attention object>
(Selection example 1)
As an example of the object selected as the non-attention object by the non-attention object selection unit 54, an object positioned behind the virtual viewpoint position can be cited. The non-target object selection unit 54 refers to information indicating the virtual viewpoint position, a depth map that does not include the non-target object in the reference image, and a depth map of a complementary region described later in the reference image to determine whether the object is a non-target object. Can be determined. When the virtual viewpoint position is the origin and the direction toward the attention area is positive, the depth value of the depth map of the area closer to the attention area than the virtual viewpoint position is a positive value. Therefore, if the depth values of the depth map of the complementary region are all negative values, it can be said that the depth map exists behind the reference viewpoint. The non-target object selection unit 54 determines that an object that satisfies this condition, that is, an object whose depth values in the depth map are all negative values, is a non-target object, and complements the non-target object. Is output to the complementary difference image processing unit 56.
　（選択例２）
　また、非注目物選択部５４に非注目物として選択される物体の例として、仮想視点位置の周辺領域と重なる物体を挙げられる。非注目物選択部５４は、仮想視点位置を示す情報、基準画像における非注目物を含まないデプスマップ、及び基準画像における補完領域のデプスマップを参照して、物体が非注目物か否かを判定することができる。非注目物選択部５４は、仮想視点位置を原点として、補完領域のデプスマップの奥行き値に、一定の割合以下、例えば最大奥行き値の±１０％以下の奥行き値が含まれている物体を非注目物であると判定し、当該非注目物を補完する補完差分画像を、補完差分画像処理部５６に出力する。 (Selection example 2)
Further, as an example of the object selected as the non-attention object by the non-attention object selection unit 54, an object overlapping the peripheral region of the virtual viewpoint position can be cited. The non-target object selection unit 54 refers to the information indicating the virtual viewpoint position, the depth map that does not include the non-target object in the reference image, and the depth map of the complementary region in the reference image to determine whether or not the object is a non-target object. Can be determined. The non-attention object selection unit 54 uses a virtual viewpoint position as an origin, and removes an object in which the depth value of the depth map of the complementary region includes a depth value of a certain percentage or less, for example, ± 10% or less of the maximum depth value. It determines with it being an attention object, and outputs the complementary difference image which complements the said non-attention object to the complementary difference image process part 56. FIG.
　なお、非注目物選択部５４は、割合ではなく定数値で判断してもよい。例えば、非注目物選択部５４は、補完領域のデプスマップの奥行きが、仮想視点位置を起点として奥行き方向±１０ｃｍ以内に含まれていれば、当該補完領域に対応する非注目物候補を非注目物だと判断することができる。または、非注目物選択部５４は、奥行き値の絶対値が一定以下、という条件で非注目物候補を非注目物だと判断してもよい。 It should be noted that the non-attention object selection unit 54 may make a determination based on a constant value instead of a ratio. For example, if the depth of the depth map of the complementary region is included within ± 10 cm in the depth direction starting from the virtual viewpoint position, the non-focused object selection unit 54 does not focus on the non-focused object candidate corresponding to the complementary region. It can be judged as a thing. Alternatively, the non-target object selection unit 54 may determine that the non-target object candidate is a non-target object on the condition that the absolute value of the depth value is not more than a certain value.
　（選択例３）
　また、非注目物選択部５４に非注目物として選択される物体の例として、注目領域より手前にある全ての物体を挙げることができる。非注目物選択部５４は、仮想視点位置を示す情報、基準画像における非注目物を含まない領域のデプスマップ、及び、基準画像における補完領域のデプスマップを参照して、物体が非注目物か否かを判断することができる。仮想視点位置を原点として、注目領域に向かう方向を正とする。この場合、非注目物選択部５４は、補完領域のデプスマップの全ての奥行き値が、非注目物の無いデプスマップより小さければ、補完差分画像に非注目物が含まれていると判定し、当該補完差分画像を、補完差分画像処理部５６に出力する。 (Selection Example 3)
In addition, examples of objects selected as non-target objects by the non-target object selection unit 54 include all objects that are in front of the target area. The non-target object selection unit 54 refers to the information indicating the virtual viewpoint position, the depth map of the region not including the non-target object in the reference image, and the depth map of the complementary region in the reference image to determine whether the object is a non-target object. It can be determined whether or not. The direction toward the region of interest is positive with the virtual viewpoint position as the origin. In this case, the non-target object selection unit 54 determines that the non-target object is included in the complementary difference image if all the depth values of the depth map of the complement region are smaller than the depth map without the non-target object, The complementary difference image is output to the complementary difference image processing unit 56.
　（選択例４）
　また、非注目物選択部５４は、仮想視点位置の画像内に占める非注目物候補の領域が所定割合以上の場合、非注目物として選択してもよい。所定割合は任意の値とする。例えば、所定割合を５割に設定したとして、後方のカメラが撮像した画像において、非注目物候補の領域が占める割合が２割であったとしても、仮想視点位置の画像において、非注目物候補の領域が占める割合が５割であった場合、非注目物候補は非注目物であると判定し、当該非注目物を補完する補完差分画像を、補完差分画像処理部５６に出力する。 (Selection Example 4)
Further, the non-attention object selection unit 54 may select a non-attention object as a non-attention object when the area of the non-attention object candidate in the image at the virtual viewpoint position is a predetermined ratio or more. The predetermined ratio is an arbitrary value. For example, assuming that the predetermined ratio is set to 50%, even if the ratio of the non-target object candidate area in the image captured by the rear camera is 20%, the non-target object candidate is displayed in the virtual viewpoint position image. When the proportion of the area is 50%, it is determined that the non-target object candidate is a non-target object, and a complementary difference image that complements the non-target object is output to the complementary difference image processing unit 56.
　仮想視点画像生成部７２は、注目画像生成部６０が生成した注目画像を用いて仮想視点画像を生成する。すなわち、仮想視点画像生成部７２は、例えば、非注目物と判断された非注目物候補が透明になったように見え、かつ、仮想視点位置から注目領域を観察しているような画像を生成する。 The virtual viewpoint image generation unit 72 generates a virtual viewpoint image using the attention image generated by the attention image generation unit 60. That is, the virtual viewpoint image generation unit 72 generates, for example, an image in which a non-target object candidate determined to be a non-target object appears to be transparent and the target region is observed from the virtual viewpoint position. To do.
　〔実施形態３〕
　本発明の他の実施形態について、図１２～図１５に基づいて説明すれば、以下のとおりである。なお、説明の便宜上、上述した実施形態にて説明した部材と同じ機能を有する部材については、同じ符号を付記し、その説明を省略する。 [Embodiment 3]
The following will describe another embodiment of the present invention with reference to FIGS. For convenience of explanation, members having the same functions as those described in the above-described embodiment are denoted by the same reference numerals and description thereof is omitted.
　本実施形態は、補完差分画像に代えて、補完差分情報を生成し、用いる点において、上述した実施形態と異なる。 This embodiment is different from the above-described embodiment in that complementary difference information is generated and used instead of the complementary difference image.
　図１２に示すように、本実施形態に係る画像処理システム１ｂは、注目画像データ生成部３０ｂ、注目画像合成部５０ｂ、及び、格納部２０を備えている。なお、格納部２０は、実施形態１に係る画像処理システム１の格納部２０に対応する。 As shown in FIG. 12, the image processing system 1b according to this embodiment includes an attention image data generation unit 30b, an attention image synthesis unit 50b, and a storage unit 20. The storage unit 20 corresponds to the storage unit 20 of the image processing system 1 according to the first embodiment.
　図１３に示すように、注目画像データ生成部３０ｂは、取得部３２、基準画像選択部３４、注目領域補完データ生成部３６ｂ、三次元情報取得部３８、及び多重化部４０を備えている。取得部３２、基準画像選択部３４、三次元情報取得部３８、及び多重化部４０は、それぞれ、実施形態１に係る画像処理システム１の取得部３２、基準画像選択部３４、三次元情報取得部３８、及び多重化部４０に対応する。 As illustrated in FIG. 13, the attention image data generation unit 30 b includes an acquisition unit 32, a reference image selection unit 34, an attention region complement data generation unit 36 b, a three-dimensional information acquisition unit 38, and a multiplexing unit 40. The acquisition unit 32, the reference image selection unit 34, the 3D information acquisition unit 38, and the multiplexing unit 40 are respectively the acquisition unit 32, the reference image selection unit 34, and the 3D information acquisition of the image processing system 1 according to the first embodiment. This corresponds to the unit 38 and the multiplexing unit 40.
　実施形態１の注目領域補完データ生成部３６は、補完差分画像を含む注目画像データを生成したが、本実施形態の注目領域補完データ生成部３６ｂは、補完差分画像に代えて、補完差分情報生成し、補完差分情報を含む注目画像データを生成する。 Although the attention area complement data generation unit 36 of Embodiment 1 has generated attention image data including a complementary difference image, the attention area complement data generation unit 36b of the present embodiment generates complementary difference information instead of the complementary difference image. Then, attention image data including complementary difference information is generated.
　本実施形態においても、上述した実施形態と同様に基準画像に含まれる非注目物の領域を補完した注目画像を生成する。本実施形態においては、基準画像を補完するときに、別の基準画像の何れかの一部の領域を用いる。このとき、ある基準画像のどの領域に対して、他の基準画像のどの領域を用いればよいのかを示す情報を、補完差分情報とよぶ。 Also in the present embodiment, as in the above-described embodiment, an attention image that complements the region of the non-attention object included in the reference image is generated. In the present embodiment, when a reference image is complemented, a partial region of any one of the other reference images is used. At this time, information indicating which region of another reference image should be used for which region of a certain reference image is referred to as complementary difference information.
　補完差分情報を、図１４を参照して説明する。例えば、一つの基準画像（図１４の（ａ）に示す基準画像Ａ）内の非注目物で隠された領域を、他の基準画像（図１４の（ｂ）に示す基準画像Ｂ）の一部領域（参照領域）を用いて補完する場合を考える。 Supplementary difference information will be described with reference to FIG. For example, an area hidden by a non-target object in one reference image (reference image A shown in FIG. 14A) is one of the other reference images (reference image B shown in FIG. 14B). Consider the case of complementing using a partial area (reference area).
　図１４の（ａ）に示すように、基準画像Ａは、座標（ｘ１，ｙ１）を始点とする、幅がｗａであり、高さがｈａである領域に非注目物である白雲を含む。この領域が、基準画像Ａにおいて補完の対象となる補完領域である。なお、実施形態３における補完領域は、実施形態１の補完領域のデプスマップの説明で示した補完領域と同等のものである。ただし、ここでの補完領域は、基準画像に対応する補完差分画像と、補完差分画像に付随する補完画像座標情報から計算されるものではなく、補完差分情報に含まれる情報である。 As shown in FIG. 14 (a), the reference image A includes a white cloud that is a non-target object in an area starting from the coordinates (x1, y1) and having a width of wa and a height of ha. This region is a complement region to be complemented in the reference image A. Note that the complementary region in the third embodiment is equivalent to the complementary region shown in the description of the depth map of the complementary region in the first embodiment. However, the complementary region here is not calculated from the complementary difference image corresponding to the reference image and the complementary image coordinate information attached to the complementary difference image, but is information included in the complementary difference information.
　これに対し、図１４の（ｂ）に示すように、基準画像Ｂは、基準画像Ａの座標（ｘ１，ｙ１）に対応する座標（ｘ２，ｙ２）を始点とする、幅がｗｂであり、高さがｈｂである領域（基準画像Ｂの参照領域）に非注目物が存在しない。したがって、基準画像Ａの補完領域を補完するのに、基準画像Ｂの参照領域を用いることができる。 On the other hand, as shown in FIG. 14B, the reference image B has a width wb starting from the coordinates (x2, y2) corresponding to the coordinates (x1, y1) of the reference image A, There is no non-attention object in the area where the height is hb (reference area of the standard image B). Therefore, the reference area of the standard image B can be used to complement the complementary area of the standard image A.
　なお、補完領域と参照領域との大きさ、及び形状は、必ずしも同一でなくともよい。補完領域と参照領域とが同じ大きさ、同じ形状でない場合、後述する非注目物選択部５４ｂにおいて、参照領域から抜き出された部分画像が、補完領域に合うように変形する。 Note that the size and shape of the complementary region and the reference region are not necessarily the same. When the complementary region and the reference region are not the same size and the same shape, the non-target object selection unit 54b described later deforms the partial image extracted from the reference region to fit the complementary region.
　図１４の（ｃ）に示すように、基準画像Ａの補完差分情報である補完差分情報１は、対応する基準画像が「基準画像Ａ」であることを示す情報、補完領域の始点（ｘ１，ｙ１）の座標を示す情報、補完領域の幅ｗａを示す情報、補完領域の高さｈａを示す情報、補完に用いる基準画像が「基準画像Ｂ」であることを示す情報、参照領域の始点（ｘ２，ｙ２）の座標を示す情報、補完領域の幅ｗｂを示す情報、補完領域の高さｈｂを示す情報、及び非注目物が「白雲」であることを示す情報を含む。 As shown in FIG. 14C, the complementary difference information 1 that is the complementary difference information of the reference image A is information indicating that the corresponding reference image is “reference image A”, the starting point (x1, information indicating the coordinates of y1), information indicating the width wa of the complementary region, information indicating the height ha of the complementary region, information indicating that the standard image used for complementing is “standard image B”, the starting point of the reference region ( information indicating the coordinates of x2, y2), information indicating the width wb of the complementary region, information indicating the height hb of the complementary region, and information indicating that the non-target object is a “white cloud”.
　一方、図１４の（ｂ）に示すように、基準画像Ｂは、座標（ｘｂ１，ｙｂ１）を始点とし、座標（ｘｂ２，ｙｂ２）を終点とする領域に非注目物である灰色雲を含む。この領域が、基準画像Ｂの補完対象の補完領域である。なお、ここでは、対象の領域における右下端の点を「終点」とよぶ。 On the other hand, as shown in FIG. 14B, the reference image B includes a gray cloud that is a non-attention object in a region starting from the coordinates (xb1, yb1) and ending at the coordinates (xb2, yb2). This region is a complement region to be complemented with the reference image B. Here, the lower right point in the target area is called an “end point”.
　これに対し、図１４の（ａ）に示すように、基準画像Ａは、基準画像Ｂの座標（ｘｂ１，ｙｂ１）に対応する座標（ｘａ１，ｙａ１）を始点とし、基準画像Ｂの座標（ｘｂ２，ｙｂ２）に対応する座標（ｘａ２，ｙａ２）を終点とする領域（基準画像Ａの参照領域）に非注目物が存在しない。したがって、基準画像Ｂの補完領域を補完するのに、基準画像Ａの参照領域を用いることができる。 On the other hand, as shown in FIG. 14A, the reference image A starts from the coordinates (xa1, ya1) corresponding to the coordinates (xb1, yb1) of the reference image B, and the coordinates (xb2) of the reference image B , Yb2), there is no non-attention object in the area (reference area of the standard image A) whose end point is the coordinate (xa2, ya2) corresponding to. Therefore, the reference area of the standard image A can be used to complement the complementary area of the standard image B.
　図１４の（ｄ）に示すように、基準画像Ｂの補完差分情報である補完差分情報２は、対応する基準画像が「基準画像Ｂ」であることを示す情報、補完領域の始点（ｘｂ１，ｙｂ１）の座標を示す情報、補完領域の終点（ｘｂ２，ｙｂ２）の座標を示す情報、補完に用いる基準画像が「基準画像Ａ」であることを示す情報、参照領域の始点（ｘａ１，ｙａ１）の座標を示す情報、参照領域の終点（ｘａ２，ｙａ２）の座標を示す情報、及び非注目物が「灰色雲」であることを示す情報を含む。 As shown in FIG. 14D, the complementary difference information 2 that is the complementary difference information of the reference image B is information indicating that the corresponding reference image is the “reference image B”, the starting point (xb1, Information indicating the coordinates of yb1), information indicating the coordinates of the end point (xb2, yb2) of the complementary region, information indicating that the standard image used for complementing is “standard image A”, and the starting point (xa1, ya1) of the reference region Information indicating the coordinates of the reference area, information indicating the coordinates of the end point (xa2, ya2) of the reference area, and information indicating that the non-target object is a “gray cloud”.
　なお、補完領域、及び参照領域は、領域が一意に定まるのであれば、どのような表現でもよい。例えば、領域の四隅の座標によって表されてもよい。また必ずしも矩形領域である必要は無く、例えば円形であってもよい。 Note that the supplemental area and the reference area may be expressed in any manner as long as the area is uniquely determined. For example, it may be represented by the coordinates of the four corners of the region. Further, it is not necessarily a rectangular area, and may be, for example, a circle.
　図１５に示すように、注目画像合成部５０ｂは、逆多重化部５２、非注目物選択部５４ｂ、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０を備えている。なお、逆多重化部５２、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０は、それぞれ、実施形態１に係る画像処理システム１の逆多重化部５２、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０に対応する。 As illustrated in FIG. 15, the attention image synthesis unit 50 b includes a demultiplexing unit 52, a non-target object selection unit 54 b, a complementary difference image processing unit 56, a display method acquisition unit 58, and a attention image generation unit 60. . Note that the demultiplexing unit 52, the complementary difference image processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60 are respectively the demultiplexing unit 52 and the complementary difference image of the image processing system 1 according to the first embodiment. This corresponds to the processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60.
　本実施形態の非注目物選択部５４ｂは、まず、基準画像に対応する補完差分情報に基づき、別の基準画像の参照領域の画素を抜き出して、選択された非注目物に対応する補完差分画像を生成する。この時、補完差分情報に含まれる、補完に用いる基準画像の情報と、参照領域以外の情報を、補完差分画像に付随する情報として扱う。次に、実施形態１の非注目物選択部５４と同様の処理を行う。すなわち、補完差分画像に付随する情報、または三次元情報に基づいて非注目物を選択し、非注目物が含まれていると判断した補完差分画像を補完差分画像処理部５６に出力する。 The non-target object selection unit 54b according to the present embodiment first extracts pixels in the reference area of another standard image based on the complementary difference information corresponding to the standard image, and performs the complementary differential image corresponding to the selected non-target object. Is generated. At this time, information on the base image used for complementation and information other than the reference area included in the complementation difference information are handled as information accompanying the complementation difference image. Next, the same processing as that performed by the non-target object selection unit 54 of the first embodiment is performed. That is, a non-target object is selected based on information accompanying the complementary difference image or three-dimensional information, and the complementary difference image determined to contain the non-target object is output to the complementary difference image processing unit 56.
　このように、本実施形態に係る画像処理装置は、事前に生成された補完差分情報を含む注目画像データを参照して、基準画像内の物体が非注目物か否かを判定する非注目物選択部５４ｂと、表示条件に基づき、前記注目画像データに含まれる前記補完差分情報によって指定される補完差分画像に画像処理を行う補完差分画像処理部５６と、前記補完差分画像と前記基準画像を合成することで、前記基準画像内の前記非注目物の透明度が調整された注目画像を得る注目画像生成部６０とを備えている。 As described above, the image processing apparatus according to the present embodiment refers to the target image data including the complementary difference information generated in advance, and determines whether or not the object in the reference image is a non-target object. Based on the display unit, the selection unit 54b, the complementary difference image processing unit 56 that performs image processing on the complementary difference image specified by the complementary difference information included in the target image data, the complementary difference image, and the reference image And an attention image generation unit 60 that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted by combining.
　本実施形態に係る構成によっても、非注目物を含まない所望の画像を、任意の処理の設定かつ比較的軽い処理で生成できる。また、格納するデータ量を抑えられる。 Also with the configuration according to the present embodiment, a desired image that does not include a non-attention object can be generated with an arbitrary process setting and a relatively light process. In addition, the amount of data to be stored can be suppressed.
　〔実施形態４〕
　本発明の他の実施形態について、図１６～図１８に基づいて説明すれば、以下のとおりである。なお、説明の便宜上、上述した実施形態にて説明した部材と同じ機能を有する部材については、同じ符号を付記し、その説明を省略する。 [Embodiment 4]
The following will describe another embodiment of the present invention with reference to FIGS. For convenience of explanation, members having the same functions as those described in the above-described embodiment are denoted by the same reference numerals and description thereof is omitted.
　本実施形態は、三次元情報を利用しない点で、上述した実施形態と異なる。 This embodiment is different from the above-described embodiment in that three-dimensional information is not used.
　図１６に示すように、本実施形態に係る画像処理システム１ｃは、注目画像データ生成部３０ｃ、注目画像合成部５０ｃ、及び格納部２０を備えている。なお、格納部２０は、実施形態１に係る画像処理システム１の格納部２０に対応する。 As shown in FIG. 16, the image processing system 1c according to the present embodiment includes an attention image data generation unit 30c, an attention image synthesis unit 50c, and a storage unit 20. The storage unit 20 corresponds to the storage unit 20 of the image processing system 1 according to the first embodiment.
　図１７に示すように、注目画像データ生成部３０ｃは、実施形態１の注目画像データ生成部３０と比較して、三次元情報取得部の機能を有さない点で異なる。取得部３２、基準画像選択部３４、注目領域補完データ生成部３６、及び多重化部４０ｃは、それぞれ、実施形態１に係る画像処理システム１の取得部３２、基準画像選択部３４、注目領域補完データ生成部３６、及び多重化部４０に対応する。 17, the attention image data generation unit 30c is different from the attention image data generation unit 30 of the first embodiment in that it does not have a function of a three-dimensional information acquisition unit. The acquisition unit 32, the reference image selection unit 34, the attention area complement data generation unit 36, and the multiplexing unit 40c are respectively the acquisition unit 32, the reference image selection unit 34, and the attention area complement of the image processing system 1 according to the first embodiment. This corresponds to the data generation unit 36 and the multiplexing unit 40.
　本実施形態においては、三次元情報を取得しないので、注目画像データは、基準画像、及び補完差分画像を含む。 In the present embodiment, since the three-dimensional information is not acquired, the attention image data includes a reference image and a complementary difference image.
　図１８に示すように、注目画像合成部５０ｃは、逆多重化部５２、非注目物選択部５４ｃ、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０を備えている。なお、逆多重化部５２、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０は、それぞれ、実施形態１に係る画像処理システム１の逆多重化部５２、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０に対応する。 As illustrated in FIG. 18, the attention image synthesis unit 50 c includes a demultiplexing unit 52, a non-target object selection unit 54 c, a complementary difference image processing unit 56, a display method acquisition unit 58, and a attention image generation unit 60. . Note that the demultiplexing unit 52, the complementary difference image processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60 are respectively the demultiplexing unit 52 and the complementary difference image of the image processing system 1 according to the first embodiment. This corresponds to the processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60.
　非注目物選択部５４ｃは、三次元情報を用いずに非注目物を選択する。すなわち、非注目物選択部５４ｃは、補完差分画像に含まれる非注目物に付随する情報に基づいて非注目物を選択する。すなわち、非注目物選択部５４ｃは、特定の物体を識別し非注目物として選択する。 The non-attention object selection unit 54c selects an unattention object without using the three-dimensional information. That is, the non-target object selection unit 54c selects a non-target object based on information accompanying the non-target object included in the complementary difference image. That is, the non-target object selection unit 54c identifies a specific object and selects it as a non-target object.
　〔実施形態５〕
　本発明の他の実施形態について、図１９～図２１に基づいて説明すれば、以下のとおりである。なお、説明の便宜上、前記実施形態にて説明した部材と同じ機能を有する部材については、同じ符号を付記し、その説明を省略する。 [Embodiment 5]
The following will describe another embodiment of the present invention with reference to FIGS. For convenience of explanation, members having the same functions as those described in the embodiment are given the same reference numerals, and descriptions thereof are omitted.
　本実施形態は、非注目物選択部５４ｄにおいて、非注目物か否かの選択に、ユーザ注目点情報を利用する点で、実施形態１と異なる。 This embodiment is different from Embodiment 1 in that the non-attention object selection unit 54d uses user attention point information to select whether or not it is a non-attention object.
　図１９に示すように、本実施形態に係る画像処理システム１ｄは、注目画像データ生成部３０、注目画像合成部５０ｄ、及び格納部２０を備えている。本実施形態における注目画像合成部５０ｄは、ユーザの視線方向を示す情報を取得する。なお、格納部２０、及び注目画像データ生成部３０は、それぞれ、実施形態１に係る画像処理システム１の格納部２０、及び注目画像データ生成部３０に対応する。 As shown in FIG. 19, the image processing system 1 d according to the present embodiment includes an attention image data generation unit 30, an attention image synthesis unit 50 d, and a storage unit 20. The attention image composition unit 50d in the present embodiment acquires information indicating the user's line-of-sight direction. The storage unit 20 and the attention image data generation unit 30 correspond to the storage unit 20 and the attention image data generation unit 30 of the image processing system 1 according to the first embodiment, respectively.
　図２０に示すように、注目画像データ生成部３０ｄは、取得部３２、基準画像選択部３４、注目領域補完データ生成部３６、三次元情報取得部３８ｄ、及び多重化部４０を備えている。取得部３２、基準画像選択部３４、注目領域補完データ生成部３６、及び多重化部４０は、それぞれ、実施形態１に係る画像処理システム１の取得部３２、基準画像選択部３４、注目領域補完データ生成部３６、及び多重化部４０に対応する。 As shown in FIG. 20, the attention image data generation unit 30d includes an acquisition unit 32, a reference image selection unit 34, an attention region complement data generation unit 36, a three-dimensional information acquisition unit 38d, and a multiplexing unit 40. The acquisition unit 32, the reference image selection unit 34, the attention area complement data generation unit 36, and the multiplexing unit 40 are respectively the acquisition unit 32, the reference image selection unit 34, and the attention area complementation of the image processing system 1 according to the first embodiment. This corresponds to the data generation unit 36 and the multiplexing unit 40.
　三次元情報取得部３８ｄは、基準画像の三次元情報に加え、視線方向情報を取得する。視線方向情報は、入力画像内に含まれる人物の目の位置を示す情報、及び該人物が注目する点を示す情報を含む。視線方向情報は、例えば、アイトラッキング機器を用いて取得してもよいし、人物の目が映っている入力画像を画像処理することにより取得してもよい。視線方向情報は、基準画像と対応付けられて、多重化部４０に出力される。ただし、基準画像内に人物が含まれない場合は、基準画像に視線方向情報を対応付けなくてもよい。 The 3D information acquisition unit 38d acquires line-of-sight direction information in addition to the 3D information of the reference image. The line-of-sight direction information includes information indicating the position of the eyes of a person included in the input image and information indicating a point that the person is interested in. The line-of-sight direction information may be acquired using, for example, an eye tracking device, or may be acquired by performing image processing on an input image in which a person's eyes are reflected. The line-of-sight direction information is output to the multiplexing unit 40 in association with the reference image. However, when a person is not included in the reference image, the line-of-sight direction information may not be associated with the reference image.
　三次元情報取得部３８ｄは、視線方向情報に含まれる人物の目の位置の情報を、デプスマップとの相対的な位置を表す情報として取得する。例えば、目の位置を、デプスマップの原点を基準とした三次元的な座標で表すことができる。 The three-dimensional information acquisition unit 38d acquires information on the position of the eyes of the person included in the line-of-sight direction information as information indicating a relative position with respect to the depth map. For example, the position of the eye can be represented by three-dimensional coordinates with reference to the origin of the depth map.
　視線方向情報に含まれる注目点を示す情報は、人物が基準画像内のどの座標を観察しているのかを示す情報である。例えば、幅ｗ高さｈの基準画像に対し、人物が画像の中心を観察している場合、注目点の座標は、（ｗ／２，ｈ／２）である。 The information indicating the attention point included in the line-of-sight direction information is information indicating which coordinates in the reference image the person is observing. For example, when a person observes the center of an image with respect to a reference image having a width w and a height h, the coordinates of the attention point are (w / 2, h / 2).
　図２１に示すように、注目画像合成部５０ｄは、逆多重化部５２、非注目物選択部５４ｄ、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０を備えている。なお、逆多重化部５２、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０は、それぞれ、実施形態１に係る画像処理システム１の逆多重化部５２、補完差分画像処理部５６、表示方法取得部５８、及び注目画像生成部６０に対応する。 As illustrated in FIG. 21, the attention image synthesis unit 50 d includes a demultiplexing unit 52, a non-target object selection unit 54 d, a complementary difference image processing unit 56, a display method acquisition unit 58, and a attention image generation unit 60. . Note that the demultiplexing unit 52, the complementary difference image processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60 are respectively the demultiplexing unit 52 and the complementary difference image of the image processing system 1 according to the first embodiment. This corresponds to the processing unit 56, the display method acquisition unit 58, and the attention image generation unit 60.
　＜ユーザ注目点情報＞
　ユーザ注目点情報は、ユーザが観察している注目画像内の座標を表す情報である。例えば、ユーザが幅ｗ高さｈの注目画像の中心を観察している場合、ユーザ注目点情報は（ｗ／２，ｈ／２）を示す情報である。例えば、ユーザ注目点は、ユーザがヘッドマウントディスプレイ（ＨＭＤ）を用いて注目画像を観察しているなら、ＨＭＤに表示されている領域の中心点を注目点とすることができる。ユーザがフラットパネルディスプレイを用いて注目画像を観察している場合も同様に、ディスプレイに表示されている領域の中心点を注目点とすることができる。また、例えば、アイトラッキング機器を用いて測定してもよい。 <User attention point information>
The user attention point information is information representing coordinates in the attention image that the user observes. For example, when the user is observing the center of the attention image having the width w and the height h, the user attention point information is information indicating (w / 2, h / 2). For example, as the user attention point, if the user observes the attention image using a head mounted display (HMD), the center point of the area displayed on the HMD can be set as the attention point. Similarly, when the user observes an attention image using a flat panel display, the center point of the area displayed on the display can be set as the attention point. Further, for example, the measurement may be performed using an eye tracking device.
　＜非注目物選択部＞
　非注目物選択部５４ｄは、まず、基準画像に対応する視線方向情報より、基準画像中の人物の目線の角度を計算する。次に、基準視点位置とユーザ注目点情報より、ユーザの目線の角度を計算する。非注目物選択部５４ｄは、前者の計算結果と、後者の計算結果から、非注目物か否かを判定する。 <Non-attention object selector>
First, the non-target object selection unit 54d calculates the angle of the line of sight of the person in the reference image from the line-of-sight direction information corresponding to the reference image. Next, the angle of the user's eye line is calculated from the reference viewpoint position and the user attention point information. The non-target object selection unit 54d determines whether or not the object is a non-target object from the former calculation result and the latter calculation result.
　非注目物選択部５４ｄは、基準視点位置、ユーザの目線の角度、基準画像中の人物の目の位置、及び目線の角度から、非注目物か否かを判定する。非注目物選択部５４ｄは、基準視点位置と、基準画像中の人物の目の位置とが近く、かつ、ユーザの目線の角度と、基準画像中の人物の目線の角度が近い場合、当該人物を非注目物とすることができる。 The non-target object selection unit 54d determines whether or not the object is a non-target object from the reference viewpoint position, the angle of the user's eyes, the position of the eyes of the person in the reference image, and the angle of the eyes. The non-attention object selection unit 54d determines that the person in the reference image position is close to the eye position of the person in the reference image and the eye angle of the user is close to the eye angle of the person in the reference image. Can be made non-attention.
　また、３６０度カメラの画像の視聴時のユーザの視線方向を記録し、ユーザ注目点情報として利用してもよい。 Also, the user's line-of-sight direction when viewing the 360-degree camera image may be recorded and used as user attention point information.
　〔ソフトウェアによる実現例〕
　画像処理システム１、１ａ、１ｂ、１ｃ、１ｄの制御ブロック（特に注目画像データ生成部３０、３０ｂ、３０ｃ、注目画像合成部５０、５０ｂ、５０ｃ、５０ｄ、仮想視点画像合成部７０）は、集積回路（ＩＣチップ）等に形成された論理回路（ハードウェア）によって実現してもよいし、ＣＰＵ（Central Processing Unit）を用いてソフトウェアによって実現してもよい。 [Example of software implementation]
The control blocks of the image processing systems 1, 1 a, 1 b, 1 c, 1 d (particularly the attention image data generation units 30, 30 b, 30 c, attention image synthesis units 50, 50 b, 50 c, 50 d, virtual viewpoint image synthesis unit 70) are integrated. It may be realized by a logic circuit (hardware) formed on a circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit).
　後者の場合、画像処理装置１、１ａ、１ｂ、１ｃ、１ｄは、各機能を実現するソフトウェアであるプログラムの命令を実行するＣＰＵ、上記プログラム及び各種データがコンピュータ（またはＣＰＵ）で読み取り可能に記録されたＲＯＭ（Read Only Memory）または記憶装置（これらを「記録媒体」と称する）、上記プログラムを展開するＲＡＭ（RandomAccess Memory）などを備えている。そして、コンピュータ（またはＣＰＵ）が上記プログラムを上記記録媒体から読み取って実行することにより、本発明の目的が達成される。上記記録媒体としては、「一時的でない有形の媒体」、例えば、テープ、ディスク、カード、半導体メモリ、プログラマブルな論理回路などを用いることができる。また、上記プログラムは、該プログラムを伝送可能な任意の伝送媒体（通信ネットワークや放送波等）を介して上記コンピュータに供給されてもよい。なお、本発明の一態様は、上記プログラムが電子的な伝送によって具現化された、搬送波に埋め込まれたデータ信号の形態でも実現され得る。 In the latter case, the image processing apparatuses 1, 1 a, 1 b, 1 c, and 1 d record a CPU that executes instructions of a program, which is software that realizes each function, and the program and various data that can be read by a computer (or CPU). ROM (Read Only Memory) or storage device (referred to as “recording medium”), RAM (Random Access Memory) for expanding the program, and the like. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one embodiment of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.
　〔まとめ〕
　本発明の態様１に係る画像処理装置は、事前に生成された補完差分画像を含む注目画像データを参照して、基準画像内の物体が非注目物か否かを判定する非注目物選択部（５４，５４ａ，５４ｂ，５４ｃ，５４ｄ）と、表示条件に基づき、前記注目画像データに含まれる前記補完差分画像に画像処理を行う補完差分画像処理部（５６）と、前記補完差分画像と前記基準画像を合成することで、前記基準画像内の前記非注目物の透明度が調整された注目画像を得る注目画像生成部（６０）と、を備えている。 [Summary]
An image processing apparatus according to an aspect 1 of the present invention refers to a target image data including a complementary difference image generated in advance, and determines whether or not an object in a reference image is a non-target object (54, 54a, 54b, 54c, 54d) and a complementary difference image processing unit (56) that performs image processing on the complementary difference image included in the target image data based on display conditions; the complementary difference image; and An attention image generation unit (60) that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted by synthesizing the reference image;
　上記の構成によれば、非注目物を含まない所望の画像を、任意の処理の設定かつ比較的軽い処理で生成できる。また、格納するデータ量を抑えられる。 According to the above configuration, a desired image that does not include a non-attention object can be generated with an arbitrary process setting and a relatively light process. In addition, the amount of data to be stored can be suppressed.
　本発明の態様２に係る画像処理装置は、上記態様１において、前記補完差分画像は、前記基準画像よりも小さく、前記注目画像データには、座標情報が含まれており、前記注目画像生成部（６０）は、前記基準画像内の前記座標情報によって定まる位置に前記補完差分画像を合成する。 The image processing apparatus according to aspect 2 of the present invention is the image processing apparatus according to aspect 1, in which the complementary difference image is smaller than the reference image, the attention image data includes coordinate information, and the attention image generation unit (60) synthesizes the complementary difference image at a position determined by the coordinate information in the reference image.
　上記の構成によれば、前記補完差分画像は、前記基準画像よりも小さく、前記注目画像生成部（６０）は、前記基準画像内の前記座標情報によって定まる位置に前記補完差分画像を合成するので、補完差分画像が複数存在する場合にも、生成処理をより軽い処理で行うことができ、また、格納するデータ量をより抑えることができる。また、補完差分画像を、好適に合成処理を行うことができる。 According to the above configuration, the complementary difference image is smaller than the reference image, and the attention image generation unit (60) synthesizes the complementary difference image at a position determined by the coordinate information in the reference image. Even when there are a plurality of complementary difference images, the generation process can be performed with a lighter process, and the amount of data to be stored can be further suppressed. In addition, the complementary difference image can be suitably combined.
　本発明の態様３に係る画像処理装置は、上記態様１～２において、前記非注目物選択部（５４，５４ａ，５４ｂ，５４ｃ，５４ｄ）は、前記基準画像のデプスマップを参照して、前記基準画像内の物体が非注目物か否かを判定する。 In the image processing apparatus according to aspect 3 of the present invention, in the above-described aspects 1 and 2, the non-target object selection unit (54, 54a, 54b, 54c, 54d) refers to the depth map of the reference image, and It is determined whether or not the object in the reference image is a non-target object.
　上記の構成によれば、基準画像内の物体が非注目物か否かを好適に判定することができる。 According to the above configuration, it can be suitably determined whether or not the object in the reference image is a non-target object.
　本発明の態様４に係る画像処理装置は、上記態様３において、前記基準画像のデプスマップには、前記非注目物を含まないデプスマップと、前記非注目物を含むデプスマップとが含まれる。 In the image processing device according to aspect 4 of the present invention, in the above aspect 3, the depth map of the reference image includes a depth map that does not include the non-attention object and a depth map that includes the non-attention object.
　上記の構成によれば、基準画像内の物体が非注目物か否かを好適に判定することができる。 According to the above configuration, it can be suitably determined whether or not the object in the reference image is a non-target object.
　本発明の態様５に係る画像処理装置は、上記態様１～４において、空間中の座標を示す仮想視点位置情報、及び前記注目画像を元に、前記空間中の座標の位置から見た画像である仮想視点画像を合成する仮想視点画像生成部（７２）をさらに備えてもよい。 The image processing apparatus according to aspect 5 of the present invention is the image processing apparatus according to any one of aspects 1 to 4, wherein the image is viewed from the position of the coordinates in the space based on the virtual viewpoint position information indicating the coordinates in the space and the attention image. You may further provide the virtual viewpoint image generation part (72) which synthesize | combines a certain virtual viewpoint image.
　上記の構成によれば、空間中の座標の位置から注目領域を観察しているような画像を得ることができる。 According to the above configuration, it is possible to obtain an image as if the region of interest is observed from the coordinate position in the space.
　本発明の態様６に係る画像処理装置は、上記態様１～２において、前記非注目物選択部は、特定の物体を前記非注目物として選択する。 In the image processing apparatus according to aspect 6 of the present invention, in the above-described aspects 1 and 2, the non-target object selecting unit selects a specific object as the non-target object.
　上記の構成によれば、特定の注目する必要がないものにより、注目領域が隠れることがない。 According to the above configuration, the region of interest is not hidden by what does not need a particular attention.
　本発明の態様７に係る画像処理装置は、上記態様１～２において、前記非注目物選択部は、ユーザの視線方向を参照して、前記基準画像内の物体が非注目物か否かを判定する。 In the image processing device according to aspect 7 of the present invention, in the above-described aspects 1 and 2, the non-target object selection unit refers to a user's line-of-sight direction to determine whether an object in the reference image is a non-target object. judge.
　上記の構成によれば、ユーザの視線方向に応じて、基準画像内の物体が非注目物か否かを好適に判定することができる。 According to the above configuration, it is possible to suitably determine whether or not the object in the reference image is a non-attention object according to the user's line-of-sight direction.
　本発明の態様８に係る画像処理装置は、事前に生成された補完差分情報を含む注目画像データを参照して、基準画像内の物体が非注目物か否かを判定する非注目物選択部と、表示条件に基づき、前記注目画像データに含まれる前記補完差分情報によって指定される補完差分画像に画像処理を行う補完差分画像処理部と、前記補完差分画像と前記基準画像を合成することで、前記基準画像内の前記非注目物の透明度が調整された注目画像を得る注目画像生成部と、を備えている。 The image processing apparatus according to the aspect 8 of the present invention refers to a target image data including preliminarily generated complementary difference information, and determines whether or not an object in the reference image is a non-target object. And a complementary difference image processing unit that performs image processing on a complementary difference image specified by the complementary difference information included in the target image data based on display conditions, and combining the complementary difference image and the reference image. An attention image generation unit that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted.
　上記の構成によれば、態様１と同様の効果を奏する。 According to the above configuration, the same effect as in the first aspect is obtained.
　本発明の態様９に係る画像処理システムは、第１の画像処理装置と第２の画像処理装置とを含む画像処理システムであって、前記第１の画像処理装置は、複数の基準画像の各々に関し、当該基準画像に含まれている非注目物を補完するために用いられる補完差分画像を生成し、生成した補完差分画像を含む注目画像データを生成する注目画像データ生成部を備え、前記第２の画像処理装置は、前記注目画像データ生成部が生成した注目画像データを参照して、基準画像内の物体が非注目物か否かを選択する非注目物選択部と、表示条件に基づき、前記注目画像データに含まれる前記補完差分画像に画像処理を行う補完差分画像処理部と、前記補完差分画像と前記基準画像を合成することで、前記基準画像内の前記非注目物の透明度が調整された注目画像を得る注目画像生成部と、を備えている。 An image processing system according to an aspect 9 of the present invention is an image processing system including a first image processing apparatus and a second image processing apparatus, and the first image processing apparatus includes a plurality of reference images. The attention image data generating unit for generating a complementary difference image used to complement the non-target object included in the reference image, and generating attention image data including the generated complementary difference image, The second image processing apparatus refers to the target image data generated by the target image data generation unit, and selects a non-target object selection unit that selects whether or not the object in the reference image is a non-target object. The transparency of the non-attention object in the reference image is obtained by combining the complementary difference image processing unit that performs image processing on the complementary difference image included in the attention image data, and the complementary difference image and the reference image. Adjusted A target image generator to obtain a target image, and a.
　上記の構成によれば、態様１と同様の効果を奏する。 According to the above configuration, the same effect as in the first aspect is obtained.
　本発明の各態様に係る画像処理装置は、コンピュータによって実現してもよく、この場合には、コンピュータを上記画像処理装置が備える各部（ソフトウェア要素）として動作させることにより上記画像処理装置をコンピュータにて実現させる画像処理装置の画像処理プログラム、及びそれを記録したコンピュータ読み取り可能な記録媒体も、本発明の範疇に入る。 The image processing apparatus according to each aspect of the present invention may be realized by a computer. In this case, the image processing apparatus is operated on each computer by causing the computer to operate as each unit (software element) included in the image processing apparatus. The image processing program of the image processing apparatus to be realized in this way and a computer-readable recording medium on which the image processing program is recorded also fall within the scope of the present invention.
　本発明は上述した各実施形態に限定されるものではなく、請求項に示した範囲で種々の変更が可能であり、異なる実施形態にそれぞれ開示された技術的手段を適宜組み合わせて得られる実施形態についても本発明の技術的範囲に含まれる。さらに、各実施形態にそれぞれ開示された技術的手段を組み合わせることにより、新しい技術的特徴を形成することができる。 The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.
　１，１ａ，１ｂ，１ｃ，１ｄ　画像処理システム
　５４，５４ｂ，５４ｃ，５４ｄ　非注目物選択部
　５６　補完差分画像処理部
　６０　注目画像生成部
　３６，３６ｂ　注目領域補完データ生成部
　７２　仮想視点画像生成部 1, 1a, 1b, 1c, 1d Image processing system 54, 54b, 54c, 54d Non-attention object selection unit 56 Complementary difference image processing unit 60 Attention image generation unit 36, 36b Attention region complementation data generation unit 72 Virtual viewpoint image generation unit

Claims (11)

  1. 　事前に生成された補完差分画像を含む注目画像データを参照して、基準画像内の物体が非注目物か否かを判定する非注目物選択部と、
    　表示条件に基づき、前記注目画像データに含まれる前記補完差分画像に画像処理を行う補完差分画像処理部と、
    　前記補完差分画像と前記基準画像を合成することで、前記基準画像内の前記非注目物の透明度が調整された注目画像を得る注目画像生成部と、
    を備えることを特徴とする画像処理装置。     A non-target object selection unit that determines whether or not the object in the reference image is a non-target object with reference to the target image data including the complementary difference image generated in advance;
    A complementary difference image processing unit that performs image processing on the complementary difference image included in the target image data based on display conditions;
    An attention image generation unit that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted by combining the complementary difference image and the reference image;
    An image processing apparatus comprising:
  2. 　前記補完差分画像は、前記基準画像よりも小さく、
    　前記注目画像データには、座標情報が含まれており、
    　前記注目画像生成部は、前記基準画像内の前記座標情報によって定まる位置に前記補完差分画像を合成する
    ことを特徴とする請求項１に記載の画像処理装置。     The complementary difference image is smaller than the reference image,
    The attention image data includes coordinate information,
    The image processing apparatus according to claim 1, wherein the attention image generation unit synthesizes the complementary difference image at a position determined by the coordinate information in the reference image.
  3. 　前記非注目物選択部は、前記基準画像のデプスマップを参照して、前記基準画像内の物体が非注目物か否かを判定することを特徴とする請求項１又は２に記載の画像処理装置。 The image processing according to claim 1, wherein the non-attention object selection unit determines whether or not an object in the reference image is a non-attention object with reference to a depth map of the reference image. apparatus.
  4. 　前記基準画像のデプスマップには、前記非注目物を含まないデプスマップと、前記非注目物を含むデプスマップとが含まれることを特徴とする請求項３に記載の画像処理装置。 The image processing apparatus according to claim 3, wherein the depth map of the reference image includes a depth map that does not include the non-target object and a depth map that includes the non-target object.
  5. 　空間中の座標を示す仮想視点位置情報、及び前記注目画像を元に、前記空間中の座標の位置から見た画像である仮想視点画像を合成する仮想視点画像生成部をさらに備える
    ことを特徴とする請求項１～４の何れか１項に記載の画像処理装置。     Further comprising: a virtual viewpoint image generation unit that synthesizes a virtual viewpoint image that is an image viewed from the position of the coordinates in the space based on the virtual viewpoint position information indicating coordinates in the space and the attention image. The image processing apparatus according to any one of claims 1 to 4.
  6. 　前記非注目物選択部は、特定の物体を前記非注目物として選択することを特徴とする請求項１又は２に記載の画像処理装置。 The image processing apparatus according to claim 1, wherein the non-attention object selection unit selects a specific object as the non-attention object.
  7. 　前記非注目物選択部は、ユーザの視線方向を参照して、前記基準画像内の物体が非注目物か否かを判定することを特徴とする請求項１又は２に記載の画像処理装置。 The image processing apparatus according to claim 1 or 2, wherein the non-attention object selection unit determines whether or not an object in the reference image is a non-attention object with reference to a user's line-of-sight direction.
  8. 　事前に生成された補完差分情報を含む注目画像データを参照して、基準画像内の物体が非注目物か否かを判定する非注目物選択部と、
    　表示条件に基づき、前記注目画像データに含まれる前記補完差分情報によって指定される補完差分画像に画像処理を行う補完差分画像処理部と、
    　前記補完差分画像と前記基準画像を合成することで、前記基準画像内の前記非注目物の透明度が調整された注目画像を得る注目画像生成部と、
    を備えることを特徴とする画像処理装置。     A non-target object selection unit that determines whether or not the object in the reference image is a non-target object with reference to the target image data including the complementary difference information generated in advance;
    Based on display conditions, a complementary difference image processing unit that performs image processing on the complementary difference image specified by the complementary difference information included in the target image data;
    An attention image generation unit that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted by combining the complementary difference image and the reference image;
    An image processing apparatus comprising:
  9. 　第１の画像処理装置と第２の画像処理装置とを含む画像処理システムであって、
    　前記第１の画像処理装置は、
    　　複数の基準画像の各々に関し、当該基準画像に含まれている非注目物を補完するために用いられる補完差分画像を生成し、生成した補完差分画像を含む注目画像データを生成する注目画像データ生成部を備え、
    　前記第２の画像処理装置は、
    　前記注目画像データ生成部が生成した注目画像データを参照して、基準画像内の物体が非注目物か否かを選択する非注目物選択部と、
    　表示条件に基づき、前記注目画像データに含まれる前記補完差分画像に画像処理を行う補完差分画像処理部と、
    　前記補完差分画像と前記基準画像を合成することで、前記基準画像内の前記非注目物の透明度が調整された注目画像を得る注目画像生成部と、
    を備えることを特徴とする画像処理システム。     An image processing system including a first image processing device and a second image processing device,
    The first image processing apparatus includes:
    Attention image data generation that generates a complementary difference image that is used to complement a non-attention object included in the reference image and generates attention image data including the generated complementary difference image for each of the plurality of reference images. Part
    The second image processing apparatus includes:
    A non-attention object selection unit that refers to the attention image data generated by the attention image data generation unit and selects whether or not the object in the reference image is a non-attention object;
    A complementary difference image processing unit that performs image processing on the complementary difference image included in the target image data based on display conditions;
    An attention image generation unit that obtains an attention image in which the transparency of the non-attention object in the reference image is adjusted by combining the complementary difference image and the reference image;
    An image processing system comprising:
  10. 　請求項１に記載の画像処理装置としてコンピュータを機能させるための画像処理プログラムであって、上記非注目物選択部、及び上記注目画像生成部としてコンピュータを機能させるための画像処理プログラム。 An image processing program for causing a computer to function as the image processing device according to claim 1, wherein the image processing program causes the computer to function as the non-attention object selection unit and the attention image generation unit.
  11. 　請求項１０に記載の画像処理プログラムを記録したコンピュータ読み取り可能な記録媒体。 A computer-readable recording medium on which the image processing program according to claim 10 is recorded.
PCT/JP2018/013260 2017-04-04 2018-03-29 Image processing device, image processing program, and recording medium WO2018186279A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2017-074533 2017-04-04
JP2017074533 2017-04-04

Publications (1)

Publication Number Publication Date
WO2018186279A1 true WO2018186279A1 (en) 2018-10-11

Family

ID=63712139

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/013260 WO2018186279A1 (en) 2017-04-04 2018-03-29 Image processing device, image processing program, and recording medium

Country Status (1)

Country Link
WO (1) WO2018186279A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004114224A1 (en) * 2003-06-20 2004-12-29 Nippon Telegraph And Telephone Corporation Virtual visual point image generating method and 3-d image display method and device
JP2005198007A (en) * 2004-01-07 2005-07-21 Ricoh Co Ltd Image processing device, image processing method, program, and information recording medium
JP2008117305A (en) * 2006-11-07 2008-05-22 Olympus Corp Image processor
WO2015011985A1 (en) * 2013-07-25 2015-01-29 ソニー株式会社 Information processing device, method, and program
JP2015126326A (en) * 2013-12-26 2015-07-06 株式会社東芝 Electronic apparatus and image processing method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004114224A1 (en) * 2003-06-20 2004-12-29 Nippon Telegraph And Telephone Corporation Virtual visual point image generating method and 3-d image display method and device
JP2005198007A (en) * 2004-01-07 2005-07-21 Ricoh Co Ltd Image processing device, image processing method, program, and information recording medium
JP2008117305A (en) * 2006-11-07 2008-05-22 Olympus Corp Image processor
WO2015011985A1 (en) * 2013-07-25 2015-01-29 ソニー株式会社 Information processing device, method, and program
JP2015126326A (en) * 2013-12-26 2015-07-06 株式会社東芝 Electronic apparatus and image processing method

Similar Documents

Publication Publication Date Title
WO2011033673A1 (en) Image processing apparatus
US9779539B2 (en) Image processing apparatus and image processing method
US9990738B2 (en) Image processing method and apparatus for determining depth within an image
JP6147464B2 (en) Image processing system, terminal device and method
JP7459051B2 (en) Method and apparatus for angle detection
CN105611267B (en) Merging of real world and virtual world images based on depth and chrominance information
US20160180514A1 (en) Image processing method and electronic device thereof
Kytö et al. Improving relative depth judgments in augmented reality with auxiliary augmentations
JP2016225811A (en) Image processing apparatus, image processing method, and program
CN107209949B (en) Method and system for generating magnified 3D images
CN113253845A (en) View display method, device, medium and electronic equipment based on eye tracking
JP2017158153A (en) Image processing apparatus and image processing method
KR101212223B1 (en) Device taking a picture and method to generating the image with depth information
US11043019B2 (en) Method of displaying a wide-format augmented reality object
WO2018186279A1 (en) Image processing device, image processing program, and recording medium
US20120249533A1 (en) Stereoscopic display apparatus
JP6004354B2 (en) Image data processing apparatus and image data processing method
JP2012065851A (en) Multi-view auto-stereoscopic endoscope system
US20210192782A1 (en) Methods and apparatuses for corner detection
JP2018129026A (en) Determination device, image processing apparatus, determination method, and determination program
JP5868055B2 (en) Image processing apparatus and image processing method
KR20190072742A (en) Calibrated Multi-Camera based Real-time Super Multi-View Image Synthesis Method and System
JP5281720B1 (en) 3D image processing apparatus and 3D image processing method
JP6085943B2 (en) Parallax control device and parallax control program
KR20230067307A (en) Method for moving in 3D modeling space using avatar

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18781911

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18781911

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP