CN104221368B - Picture coding device, picture decoding apparatus, method for encoding images, picture decoding method and program - Google Patents

Abstract

Make when visual point image and depth image are encoded or decode, it is possible to the unified multiple modes using coding different with the dependence of depth image with visual point image during decoding.Picture coding device, by predetermined each coded system switch data unit, determines any one coded system from the coded system that visual point image is different with the reference relation of depth image, passes through determined coded system and encode visual point image and depth image.Additionally, picture coding device would indicate that coding time visual point image and depth image reference relation image between be inserted in coded data string with reference to information.Picture decoding apparatus determines decoding process and decoding order according to the reference relation represented with reference to information between image, passes through determined decoding process and decoding order and is decoded visual point image and depth image.

Description

Picture coding device, picture decoding apparatus, Method for encoding images, picture decoding method and program

Technical field

The present invention relates to picture coding device, picture decoding apparatus, method for encoding images, image Coding/decoding method and program.

Background technology

By record or transmit the image of multiple viewpoint, and be rendered, it is possible to view and admire based on The image of viewing angle of hobby as the user of viewer.

As an example, the multi-angle image in DVD video has been prepared in advance based on thinking viewer The image carved while multiple viewpoints that possible interested or making Fang Xiang allows viewer see.With Family is by carrying out suitable operation when reproducing, it is possible to is switched to the reproduction of arbitrary image and regards Listen.

In order to realize such as the function of above-mentioned multi-angle image, need pre-recorded and each angle (depending on Point) corresponding multiple images whole.It is thus possible, for instance the number of viewpoint presentation content the most at most Data size the hugest.Therefore, in reality, such as, it is defined in making side and especially wants to allow audiovisual Person sees or viewer may cherish a special interest scene and prepare multi-angle image, such as not Exceed and in the range of the capacity of record medium, make presentation content.

Such as, especially, the image to physical culture, concert, legitimate drama etc., user is interested Viewpoint is various.From such a viewpoint, it is preferably able to provide user based on as far as possible The image of many viewpoints.

It is desired for background, it is known that following picture coding device: to multiple viewpoint figures with such As encoding, and also the depth information corresponding with these visual point images is encoded, generate Flow data (for example, referring to patent documentation 1) including these coded data.

This depth information is to represent that each subject in visual point image (is taken a picture with observation place Seat in the plane is put) the information of distance.By based on depth information and the letter relevant with position of camera Breath computing and obtain the position on three dimensions of each subject in visual point image, thus Taken scene can be reproduced virtually.Further, by with other position of camera pair On the screen answered, the scene reproduced is carried out projective transformation, it is possible to generate and from arbitrary viewpoint The identical image of image observed.

Depth information is by viewpoint position (photograph when being shot by the photographic attachment of photographing unit etc. Seat in the plane is put) to the distance (=degree of depth) of each subject in photographs at predetermined numerical value The information that scope (such as 8 bit) is quantized.And, depth information is by as above The shape of the monochrome image of the described brightness value that range conversion is pixel quantized Formula.Thus, it is also possible to carry out encoding (compression) as image using depth information.

In the picture coding device of patent documentation 1, about multiple visual point images of input, root According to as one of multi-view image coded system MVC (Multi-view Video Coding, many Viewpoint video encodes), use and used the predictive coding of time orientation and the prediction of viewpoint direction The coded system of coding.And, in the picture coding device of patent documentation 1, about the degree of depth Information, also by and improve code efficiency with time orientation and the predictive coding of viewpoint direction.

Additionally, as the video encoding method that multi-view image and depth image are encoded, Known following method.That is, in this video encoding method, there is following method: based on the degree of depth Image (range image) and the position relationship of photographing unit, generate in the viewpoint beyond referenced viewpoints Parallax compensation image, enter between parallax compensation image and the input picture of reality generated Row predictive coding (for example, referring to patent documentation 2).That is, this video encoding method attempts to Depth image improves the code efficiency of visual point image.In such video encoding method, need After obtaining identical parallax compensation image, and use once coding when encoding and when decoding The depth image that again decodes and generate parallax compensation image.Therefore, the coding of visual point image with And the coding result of decoding dependency depth image and decoded result.

It is also known that following video encoding method.That is, following method: with viewpoint figure As (Video) together to depth image (be defined as multiple auxiliary component (DEPTH: One of MultipleAuxiliary Components)) when encoding, pre-by visual point image The Information Pull of the motion vector etc. obtained when surveying coding is in coding (such as, the ginseng of depth image According to non-patent literature 1).It is in the case of this video encoding method, contrary with during patent documentation 2, The coding of depth image and the coding result of decoding dependency visual point image and decoded result.

Prior art literature

Patent documentation

Patent documentation 1: JP 2010-157823 publication

Patent documentation 2: JP 2007-36800 publication

Non-patent literature

Non-patent literature 1: " Coding of audio-visual objects:Visual ", ISO/IEC 14496-2:2001

Summary of the invention

The problem that invention is to be solved

By as above-mentioned patent documentation 2 or non-patent literature 1 to visual point image and depth map As encoding, it is possible to generate the image corresponding with multiple viewpoints with fewer data volume. But, these coded methods such as on the one hand by the Information Pull of depth image in visual point image Coding, the Information Pull of visual point image is the most mutual in the coding of depth image Dependence is different.Additionally, the coding of patent documentation 1 is between visual point image and depth image Do not utilize relation.

So, in these multi-view image coded systems, the dependence of visual point image and depth image is closed It is respectively different.And, these multi-view image coded systems are respectively provided with different advantages.

But, due to these picture coding modes coding and decoding time visual point image and the degree of depth The dependence of image is different, so can not simultaneously and use.Therefore, at present, to each machine Or service determines a picture coding mode, use this picture coding mode regularly.Now, Such as, even from the change etc. of content in a machine or service, create and make with determining Coded system compare the situation using other coded system more favourable, can not be tackled this Individual situation.

The present invention completes in view of such situation, its object is to to viewpoint When image and depth image encode or decode, it is possible to when unified use coding and decoding Multiple modes that visual point image is different with the dependence of depth image.

For solving the means of problem

(1) in order to solve above-mentioned problem, as the picture coding device of a mode of the present invention, Including: visual point image encoding section, when respectively to the multiple visual point images corresponding from different viewpoints When encoding, should refer to expression from viewpoint relatively in the space that is taken of described visual point image In comprise the depth image of the distance of object time, with reference to described depth image to coded system Described visual point image in switch data unit encodes, and reference should not be made to described depth image Time, without reference to described depth image to the described viewpoint in described coded system switch data unit Image encodes；Depth image encoding section, when described depth image is encoded, When should refer to described visual point image, with reference to described visual point image to described coded system switch data Described depth image in unit encodes, and when reference should not be made to described visual point image, does not joins According to described visual point image, the described depth image in described coded system switch data unit is carried out Coding；And with reference to information treatment part between image, will be by each described coded system switch data Unit represents the image of the reference relation of the described visual point image when coding and described depth image Between be inserted into reference to information and to include encoded visual point image and the coding of encoded depth image In serial data.

(2) additionally, in the picture coding device of the present invention, with reference at information between described image Reason portion is set as the situation of sequence corresponding to described coded system switch data unit, by described figure It is inserted in the head of the sequence in described coded data string with reference to information between Xiang.

(3) additionally, in the picture coding device of the present invention, with reference at information between described image Reason portion is set as the situation of picture corresponding to described coded system switch data unit, by described figure It is inserted in the head of the picture in described coded data string with reference to information between Xiang.

(4) additionally, in the picture coding device of the present invention, with reference at information between described image Reason portion is the situation of fragment corresponding to described coded system switch data unit, by between described image It is inserted in the head of the fragment in described coded data string with reference to information.

(5) additionally, in the picture coding device of the present invention, with reference at information between described image Reason portion is corresponding to the situation that described coded system switch data unit is coding unit unit, by institute State and be inserted in the head of the coding unit unit in described coded data string with reference to information between image.

(6) additionally, the picture decoding apparatus of a mode as the present invention, including: code Extraction unit, extracts from coded data string and carries out the visual point image corresponding from different viewpoints respectively The coded views image of coding, relatively it is taken sky at described visual point image from viewpoint to representing Coding depth image that the depth image of the distance of the object comprised between carries out encoding, by often Individual predetermined coded system switch data unit represents to described visual point image or the described degree of depth Between described visual point image when image encodes and the image of the reference relation of described depth image With reference to information；Visual point image lsb decoder, is decoded the described coded views image extracted； Depth image lsb decoder, is decoded the described coding depth image extracted；And decoding Control portion, based on the reference relation represented with reference to information between the described image extracted, determines institute State coded views image and the decoding order of described coding depth image.

(7) additionally, in the picture decoding apparatus of the present invention, described decoding control section is controlled System so that be shown as coded views image and coding depth figure with reference to information table between described image In the case of the decoded object images of in Xiang encodes with reference to another image, described The decoding of another image completes to start afterwards the decoding of described decoded object images, described decoding Control portion is controlled so that be shown as coded views image with reference to information table between described image Do not encode with reference to another image with the decoded object images of in coding depth image In the case of, even if the decoding of another image described is not fully complete also begins to described decoded object images Decoding.

(8) additionally, in the picture decoding apparatus of the present invention, described decoding control section based on from With reference to information between the described image extracted in the head of the sequence in described coded data string, determine As the described coded views image in the described sequence of described coded system switch data unit and The decoding order of described coding depth image.

(9) additionally, in the picture decoding apparatus of the present invention, described decoding control section based on from With reference to information between the described image extracted in the head of the picture in described coded data string, determine As the described coded views image in the described picture of described coded system switch data unit and The decoding order of described coding depth image.

(10) additionally, in the picture decoding apparatus of the present invention, described decoding control section based on With reference to information between the described image extracted in the head of the fragment from described coded data string, certainly It is set for as the described coded views image in the described fragment of described coded system switch data unit Decoding order with described coding depth image.

(11) additionally, in the picture decoding apparatus of the present invention, described decoding control section based on With reference to letter between the described image extracted in the head of the coding unit from described coded data string Breath, determines as the described volume in the described coding unit of described coded system switch data unit Code visual point image and the decoding order of described coding depth image.

(12) additionally, the method for encoding images of a mode as the present invention, including: regard Dot image coding step, when compiling the multiple visual point images corresponding from different viewpoints respectively During code, relatively comprise the space that is taken of described visual point image from viewpoint should refer to represent The depth image of distance of object time, with reference to described depth image, coded system is switched number Encode according to the described visual point image in unit, when reference should not be made to described depth image, no With reference to described depth image, the described visual point image in described coded system switch data unit is entered Row coding；Depth image coding step, when encoding described depth image, should join During according to described visual point image, with reference to described visual point image to described coded system switch data unit In described depth image encode, when reference should not be made to described visual point image, without reference to institute State visual point image the described depth image in described coded system switch data unit is compiled Code；And with reference to information handling step between image, will be by each described coded system switch data Unit represents the image of the reference relation of the described visual point image when coding and described depth image Between be inserted into reference to information and to include described coded views image and the coding of described coding depth image In serial data.

(13) additionally, the picture decoding method of a mode as the present invention, including: generation Code extraction step, extracts respectively to the visual point image corresponding from different viewpoints from coded data string The coded views image that carries out encoding, to representing from viewpoint being clapped at described visual point image relatively Coding depth image that the depth image of the distance taking the photograph the object comprised in space carries out encoding, Represent to described visual point image or described by each predetermined coded system switch data unit Described visual point image when depth image encodes and the figure of the reference relation of described depth image With reference to information between Xiang；Visual point image decoding step, enters the described coded views image extracted Row decoding；Depth image decoding step, is decoded the described coding depth image extracted； And decoding rate-determining steps, close based on the reference represented with reference to information between the described image extracted System, determines described coded views image and the decoding order of described coding depth image.

(14) additionally, the program of a mode as the present invention, it is used for making computer perform Following steps: visual point image coding step, when multiple regarding corresponding from different viewpoint respectively When dot image encodes, should refer to expression from viewpoint being clapped at described visual point image relatively During the depth image of the distance taking the photograph the object comprised in space, with reference to described depth image to volume Code mode switch data unit in described visual point image encode, reference should not be made to described deeply During degree image, without reference to described depth image to the institute in described coded system switch data unit State visual point image to encode；Depth image coding step, when compiling described depth image During code, when should refer to described visual point image, with reference to described visual point image to described coded system Described depth image in switch data unit encodes, and reference should not be made to described visual point image Time, without reference to described visual point image to the described degree of depth in described coded system switch data unit Image encodes；And with reference to information handling step between image, will be by each described coding staff Formula switch data unit represents the described visual point image when coding and the reference of described depth image It is inserted into reference to information between the image of relation and includes described coded views image and described coding depth In the coded data string of image.

(15) additionally, the program of a mode as the present invention, it is used for making computer perform Following steps: code extraction step, from coded data string extract respectively to from different viewpoints pair Coded views image that the visual point image answered carries out encoding, relatively regard described from viewpoint representing The depth image of the distance of the object comprised in the space that is taken of dot image carries out the volume encoded Code depth image, represent to described viewpoint by each predetermined coded system switch data unit Described visual point image when image or described depth image encode and described depth image With reference to information between the image of reference relation；Visual point image decoding step, to the described volume extracted Code visual point image is decoded；Depth image decoding step, to the described coding depth extracted Image is decoded；And decoding rate-determining steps, based on reference letter between the described image extracted The reference relation that breath represents, determines described coded views image and the solution of described coding depth image Code order.

Invention effect

As mentioned above, according to the present invention, visual point image and depth image are being encoded Or during decoding, it is possible to depending on of unified visual point image when using coding and decoding and depth image Multiple modes that the relation of relying is different.And, it is possible to obtain and suitably set according to dependence The effect of the decoding order of visual point image and depth image.

Accompanying drawing explanation

Fig. 1 is the figure of the structure example representing the picture coding device in embodiments of the present invention.

Fig. 2 is the reference relation example of the image in the first coded system representing present embodiment Figure.

Fig. 3 is the figure of the reference relation example of the image representing the coded object in present embodiment.

Fig. 4 is the figure of the structure example of the picture in the encoding target data representing present embodiment.

Fig. 5 is the figure of the structure example representing the coded data string in present embodiment.

Fig. 6 is to represent the classification according to the coded system switch data unit in present embodiment The figure of the example of the on position of reference information between image.

Fig. 7 is the figure of the process step example of the picture coding device execution representing present embodiment.

Fig. 8 is the figure of the structure example of the picture decoding apparatus representing present embodiment.

Fig. 9 is the knot of the visual point image correspondence table table corresponding with depth image representing present embodiment The figure of structure example.

Figure 10 is the figure of the process step example of the picture decoding apparatus execution representing present embodiment.

Detailed description of the invention

[structure of picture coding device]

Fig. 1 represents the structure example of the picture coding device 100 in embodiments of the present invention.

Picture coding device 100 shown in this figure includes visual point image encoding section 110, depth map As encoding section 120, coded system determination section 130, coded image storage part 140, photography conditions Between information encoding section 150, visual point image generating unit 160, image with reference to information treatment part 170 with And multiplexing unit 180.

Visual point image encoding section 110 inputs the multiple visual point images corresponding from different viewpoints respectively These multiple visual point image Pv are encoded by Pv.

It addition, the visual point image Pv corresponding with each viewpoint is such as to be separately positioned on different positions Put (viewpoint), the subject that shooting comprises in the same visual field (be taken space) The image of image.That is, one visual point image Pv is to be observed by some viewpoint to be taken The image of body.Additionally, the picture signal as visual point image Pv is, to being arranged in two dimensional surface In each pixel have and represent the subject that comprises in the space that is taken or the color of background Or deep or light signal value (brightness value) and there is the signal of the color space representing each pixel The picture signal of value.One example of such picture signal with the signal value representing color space It it is rgb signal.Rgb signal includes representing the R signal of brightness value of red component, expression The G-signal of the brightness value of green component, represent the B signal of the brightness value of blue component.

Depth image Pd is encoded by depth image encoding section 120.

Depth image (also referred to as " depth map (depth map) ", " depth image ", " away from From image ") Pd is to would indicate that the subject relatively comprised the space that is taken from viewpoint Or the distance of the object such as background signal value (also referred to as " depth value ", " depth value ", " degree of depth " etc.) it is set to the signal value (pixel value) of each pixel being arranged in two dimensional surface Picture signal.The pixel forming this depth image Pd is corresponding with the pixel forming visual point image. Depth image is for using the visual point image when space that is taken projects two dimensional surface and table The information in the existing three-dimensional space that is taken.

It addition, these visual point images Pv and depth image Pd both can be the figure corresponding with animation Picture, it is also possible to be the image corresponding with rest image.Additionally, depth image Pd can not also be right Each visual point image Pv of whole viewpoints is corresponding and prepares.As an example, there are three viewpoints In the case of three visual point image Pv of amount, depth image Pd can also prepare and these three Two corresponding images in visual point image Pv.

So, picture coding device 100 is by having visual point image encoding section 110 and depth map As encoding section 120, it is possible to carry out multi-view image coding.And, picture coding device 100 is made Encode for multi-view image, correspondence first～3 kinds of coded systems of the 3rd coded system.

First coded system is, to visual point image Pv and depth image Pd respectively such as by the time The predictive coding in direction and the predictive coding of viewpoint direction and with etc. and individually carry out the side encoded Formula.In this first coded system, the coding of visual point image Pv and decoding and depth image The coding of Pd and decoding the most mutually with reference to and independently carry out.That is, in the first coded system In the case of, the coding of visual point image Pv and decoding and the coding of depth image Pd and solution Code-phase is the most all independent of.

It addition, the coded method of the such as corresponding patent documentation 1 of the first coded system.

Second coded system is, based on depth image Pd and viewpoint (position of such as photographic attachment) Position relationship and generate the parallax compensation image in the viewpoint beyond referenced viewpoints, and utilize institute The parallax compensation image generated carries out the mode of the coding of visual point image Pv.At this second coding staff In formula, when coding and the decoding carrying out visual point image Pv, with reference to depth image Pd.That is, exist In the case of second coded system, the coding of visual point image Pv and decoding dependency depth image Pd.

It addition, the coded method of the such as corresponding patent documentation 2 of the second coded system.

3rd coded system is, the motion being obtained in that when the predictive coding of visual point image Pv The Information Pull of vector etc. is in the mode of the coding of depth image Pd.In the 3rd coded system, When coding and the decoding carrying out depth image Pd, with reference to visual point image Pv.That is, compile the 3rd In the case of code mode, the coding of depth image Pd and decoding dependency visual point image Pv.

It addition, the coded method of the such as corresponding non-patent literature 1 of the 3rd coded system.

And, first～the 3rd coded system be respectively provided with different advantages.

Such as, in the first coded system, due to visual point image and the coded data of depth image Do not interdepend, it is possible to suppression coding and decoding respective in process postpone.Additionally, Even in the case of the mass fraction of depth image or visual point image deteriorates, also because carrying out Separate coding, so the impact deteriorated will not pass between visual point image and depth image Broadcast.

Additionally, in the second coded system, due to coding and the decoding dependency degree of depth of visual point image The coding result of image and decoded result, so it is bigger to process retardation ratio.But, at this coding In method, if the quality of depth image is the highest, the generation precision of parallax compensation image is the highest, The compression efficiency that make use of the predictive coding of this parallax compensation image is greatly improved.

Additionally, in the 3rd coded system, regarding after utilizing coding when the coding of depth image The information of the motion vector etc. of dot image, utilizes decoded viewpoint when the decoding of depth image The information of the motion vector etc. of image.Thereby, it is possible to omit the motion with depth image as object The part process of exploration etc., such as, reduce treating capacity during coding/decoding.

Thus, picture coding device 100 can change unit by predetermined each coded system, Multi-view image volume is carried out while changing coded system between the first to the 3rd coded system Code.

Such as, by content of the presentation content according to coded object etc. and to apply flexibly its advantage Mode switches coded system, it is possible to take into account the raising of the quality of presentation content and proposing of code efficiency High.

Which in the such as first to the 3rd coded system coded system determination section 130 decision should pass through One coded system carries out multi-view image coding.When this decision, coded system determination section 130 See, for example the content of the coding parameter inputted from outside.Coding parameter is, such as, specify and The information of various parameters during row multi-view image coding.

In the case of coded system determination section 130 is determined as the first coded system, visual point image Encoding section 110 reference should not be made to depth image Pd when encoding visual point image Pv.Now, Visual point image encoding section 110, about visual point image Pv, encodes without reference to depth image Pd. Additionally, now, depth image encoding section 120 should not be joined when encoding depth image Pd According to visual point image Pv.Now, depth image encoding section 120 about depth image Pd, without reference to Visual point image Pv and encode.

Additionally, in the case of coded system determination section 130 is determined as the second coded system, depending on Dot image encoding section 110 should refer to depth image Pd when encoding visual point image Pv.This Time, visual point image encoding section 110, about visual point image Pv, is compiled with reference to depth image Pd Code.On the other hand, depth image Pd is being encoded by depth image encoding section 120 now Time, reference should not be made to visual point image Pv.Therefore, depth image encoding section 120 now is about deeply Degree image Pd, encodes without reference to visual point image Pv.

Additionally, in the case of coded system determination section 130 is determined as the 3rd coded system, depending on Dot image encoding section 110 reference should not be made to depth image Pd when encoding visual point image Pv. Now, visual point image encoding section 110, about visual point image Pv, is entered without reference to depth image Pd Row coding.On the other hand, now, depth image Pd is being carried out by depth image encoding section 120 Visual point image Pv is should refer to during coding.Now, depth image encoding section 120 is about depth image Pd, encodes with reference to visual point image Pv.

Coded image storage part 140 is stored in visual point image encoding section 110 and enters visual point image Pv The decoding visual point image generated during row coding.Additionally, coded image storage part 140 is deposited The decoding that storage generates during depth image Pd is encoded by depth image encoding section 120 Depth image.

In the case of the structure of Fig. 1, visual point image encoding section 110 is with reference to depth image Pd Time, the decoding depth image of storage in this coded image storage part 140 is utilized as reference Image.Additionally, depth image encoding section 120 is when with reference to visual point image Pv, will be at code pattern As in storage part 140, the decoding visual point image of storage utilizes as with reference to image.

Additionally, photography conditions information Ds is encoded and gives birth to by photography conditions information encoding section 150 Become encoded photogrammetric conditional information Ds_enc.

Photography conditions information Ds obtains based on being shot by photographic attachment at visual point image Pv In the case of the image of signal of video signal, as the information of the photography conditions representing this photographic attachment, Letter including the allocation position relation at the position of photographic attachment of the most each viewpoint or interval etc. Breath.Additionally, photography conditions information Ds is by such as CG (Computer at visual point image Pv Graphics, computer graphics) and in the case of the image that generates, including indicating a desire to shooting The information of the photography conditions of the virtual photographic attachment of this image.

Visual point image generating unit 160 regards based on the decoding of storage in coded image storage part 140 Dot image and decoding depth image and photography conditions information, generate visual point image Pv_i.Coding Image storage part 140 stores the visual point image Pv_i generated.The visual point image Pv so generated _ i is the visual point image of the object becoming View Synthesis predictive coding.Thus, for instance, it is possible to give birth to The coding becoming the arbitrary viewpoint beyond the visual point image Pv of visual point image encoding section 110 input regards Dot image.

Coded data string will be inserted into reference to information between image with reference to information treatment part 170 between image In STR.

That is, generate by each coded system switch data list with reference to information treatment part 170 between image With reference to information between visual point image during the presentation code of position and the image of the reference relation of depth image. Further, between image, its on position is specified with reference to information treatment part 170, between the image that will generate Multiplexing unit 180 is exported with reference to information.

" reference relation " that represent with reference to information between image represents specifically about to coding Depth image Pd whether is have references to or deep to coding when visual point image Pv_enc encodes Whether degree image Pd_enc have references to the relational of visual point image Pv when encoding.

It addition, can be based on visual point image encoding section 110 with reference to information treatment part 170 between image Coded treatment result and the coding result of depth image encoding section 120 identify this reference relation. Additionally it is possible to determination result based on coded system determination section 130 identifies.

Multiplexing unit 180, in predetermined timing, is properly entered what visual point image encoding section 110 generated The coding depth image Pd that coded views image Pv_enc, depth image encoding section 120 generate _ enc, encoded photogrammetric conditional information Ds_enc, carry out multiplexing by time division multiplex.Multiplexing unit 180 using defeated as the coded data string STR of bit stream form for the data that so carried out multiplexing Go out.

Now, multiplexing unit 180 will be inserted at coded data string with reference to information Dref between image On position specified in STR.It addition, specify with reference to information treatment part 170 between image On position according to being defined as the data unit of coded system switch data unit and different, about This point is described below.

[reference relation between image in each coded system]

Fig. 2 represents reference (dependence) the relation example of the image in the first coded system.It addition, In the figure, example when generating depth image Pd corresponding to each of whole viewpoints is represented.

In the figure, illustrate at viewpoint #0, three viewpoints of #1, #2 and time orientation Two dimension on 15 visual point image Pv0～Pv4, Pv10～Pv14, Pv20～Pv24 and The viewpoint identical with these and mutually depth image Pd0～Pd4 the most in the same time, Pd10～Pd14, Pd20～Pd24.

In the figure, the image of the terminal side of arrow is the object images of coding.Additionally, arrow The image of initial point side be when this object images is encoded reference with reference to image.

As an example, the visual point image Pv11 in viewpoint #1 is with reference in same viewpoint #1 The visual point image Pv10 in previous moment and the visual point image Pv12 in later moment and same Other viewpoint #0 in the individual moment, four viewpoint figures of visual point image Pv1 and Pv21 of #2 As Pv encodes.

Further, in the figure, for the ease of diagram easy to understand, visual point image Pv is merely illustrated Reference relation, but about depth image Pd, it is also possible to take same reference relation.

In fig. 2, viewpoint #0 is set as referenced viewpoints.Referenced viewpoints is to this viewpoint The image of other viewpoint is not used as with reference to image when image encodes or decodes Viewpoint.As shown in the figure, visual point image Pv0～Pv4 in viewpoint #0 all without reference to other Visual point image Pv10～Pv14 of viewpoint #1 or #2, Pv20～Pv24.

It addition, compiled by each visual point image Pv shown in Fig. 2 and depth image Pd When the image of code is decoded, also by the reference relation identical with Fig. 2 with reference to other figure As being decoded.

Will also appreciate that from described above, in the first coded system, when being predicted coding, Between visual point image Pv, carry out reference, between depth image Pd, carry out reference in the same manner.But no Carry out the reference between visual point image Pv and depth image Pd.

Fig. 3 represents and has used the first～the 3rd of present embodiment in the case of coded system The reference relation example of visual point image Pv and depth image Pd.As it was previously stated, first～the 3rd In coded system, owing to the reference relation of visual point image Pv and depth image Pd is respectively different, So can not use to the data of identical coded object and by multiple coded systems.But, In the present embodiment, the unit (coded system of each predetermined coding of picture etc. is such as pressed Switch data unit) switch coded system and use.Fig. 3 is such as to have switched by picture unit Example in the case of coded system.

In the figure, illustrate in viewpoint #0, two viewpoints of #1 and the two dimension of time orientation On six visual point image Pv0～Pv2, Pv10～Pv12 and corresponding six depth maps As Pd0～Pd2, Pd10～Pd12.

In the figure, the image of the terminal side of arrow is also intended to the object carrying out encoding or decode Image, the image of the initial point side of arrow is to join when this object images is encoded or decoded According to reference to image.

As an example, the depth image Pd11 in viewpoint #1 is with reference in same viewpoint #1 The depth image Pd10 of previous moment and the depth image Pd12 of later moment in time and same time The depth image Pd1 of other viewpoint #0 in quarter.Additionally, depth image Pd11 with reference to Same viewpoint and visual point image Pv11 corresponding to moment.

Additionally, the visual point image Pv11 of depth image Pd11 institute reference is with reference to identical viewpoint #1 In the visual point image Pv10 of previous moment and the visual point image Pv12 of later moment in time and identical Moment in the visual point image Pv1 of other viewpoint #0.Additionally, visual point image Pv11 ginseng According to corresponding to the viewpoint identical with visual point image Pv1 and the depth image Pd1 in moment.

According to the reference relation shown in Fig. 3, such as visual point image Pv0～Pv2 is respectively by first Coded system encodes.Additionally, visual point image Pv10～Pv12 is entered by the second coded system Row coding.Additionally, depth image Pd0～Pd2, Pd10～Pd12 are entered by the 3rd coded system Row coding.

Additionally, when image with reference to other as described above encodes, the image that should refer to Need once to encode.Therefore, the coded sequence root of visual point image Pv and depth image Pd Determine according to the reference relation between image.

Specifically, in the case of the reference relation of Fig. 3, coded sequence become Pv0, Pd0, Pv10、Pd10、Pv2、Pd2、Pv12、Pd12、Pv1、Pd1、Pv11、Pd11……。

[encoded data structure example]

Fig. 4 is set to the data of coded object as the picture coding device 100 of present embodiment One example, represents the picture 300 corresponding with visual point image Pv.

The picture 300 corresponding with visual point image Pv is, such as corresponding with the frame in image image Data.This picture 300 is formed by the pixel of predetermined number, and its least unit is to constitute a picture The signal (R, G, B signal or Y, Cb, Cr signal etc.) of the color component of element.

This picture 300 is divided into the unit of the block of the set of the pixel as predetermined number.And, Picture 300 in present embodiment is divided into the fragment of the set as block.In the figure, show The state that meaning property earth's surface diagram sheet 300 is formed by three fragments of fragment #1, #2, #3.Sheet Section is the ultimate unit of coding.

It addition, the picture corresponding with depth image Pd is also and the picture corresponding with visual point image Pv 300 in the same manner, predetermined pixel count formed.Additionally, be divided into the fragment of the set as block. But, be with the difference of visual point image Pv, depth image Pd only there is brightness value and not Has coloured information.

Fig. 5 schematically shows encoded picture 300 and has carried out the coded data string of multiplexing The structure example of STR.This coded data string STR in accordance with such as image encoding standards specification it H.264/AVC (Advanced Video Coding, advanced video coding) or the MVC of (Multi-view Video Coding, multiple view video coding).

Coded data string STR shown in Fig. 5 has stored towards rear successively from the front of data SPS (Sequence Parameter Set, sequence parameter set) #1, PPS (Picture Parameter Set, image parameters collection) #1, fragment #1, fragment #2, fragment #3, PPS#2, fragment #4……。

SPS is stored in including the letter of parameter public in the sequence entirety of the animation of multiple picture Breath, it may for example comprise constitute pixel count or the dot structure (bit number of pixel) etc. of picture.

PPS is the information of the parameter of storage picture unit, it may for example comprise represent picture unit The initial value etc. of the quantization parameter in the information of coding prediction mode or coding.

In the example of Fig. 5, SPS#1 storage includes the picture corresponding with PPS#1 and PPS#2 The public parameter of sequence.The SPS sequence number " 1 " of PPS#1 and PPS#2 storage SPS#1, Thus, it is understood that tackle each picture application SPS#1 corresponding with PPS#1 and PPS#2 Which interior parameter set.

Each fragment #1 of picture, #2, #3 of formation correspondence are applied by PPS#1 storage Parameter.Corresponding to this, fragment #1, #2, #3 storage PPS#1 sequence number " 1 ", thus, Recognize which parameter set in reply each fragment #1, #2, #3 application PPS#1.

Additionally, PPS#2 storage forms each fragment #4 of corresponding picture ... parameter. Corresponding to this, fragment #4 ... storage PPS#2 sequence number " 2 ", thus, it is understood that reply Each fragment #4 ... which parameter set in application PPS#2.

Additionally, the SPS that coded data string STR includes as shown in Figure 5, PPS, fragment etc. Data are stored in NAL (NetworkAbstractionLayer, network abstraction layer) unit (coding Unit) 400 data structure in.That is, NAL unit is storage SPS, PPS, fragment etc. The unit of unit information.

As shown in identical Fig. 5, NAL unit 400 is by NAL unit head and then this NAL RBSP (Raw Byte Sequence Payload, the raw byte sequence payload) shape of unit header Become.

Parameter set or the image coded data of the storage such as SPS, PPS, fragment are included in this RBSP In.NAL unit head includes the identification information of NAL unit.This identification information represents at RBSP The classification of the data of middle storage.

[example of coded system switch data unit]

Visual point image encoding section 110 and depth image encoding section 120 to visual point image Pv and When depth image Pd encodes, as illustrated in Figure 3, carry out based on reference time direction with And the inter prediction encoding of other the image in viewpoint direction.

And, visual point image encoding section 110 is when encoding visual point image Pv, it is possible to enter (View Synthesis prediction is compiled in the predictive coding of the composograph that row generates with utilizing depth image Pd Code).That is, visual point image encoding section 110 can carry out the second coded system.

Additionally, depth image encoding section 120 is when encoding depth image Pd, it is possible to enter Row make use of the coding encoding complete information (motion vector etc.) of visual point image Pv.Thus, Such as with only by the first coded system shown in Fig. 1 (at visual point image Pv and depth image Pd In individually only by the way of time orientation prediction encodes) the situation phase that encoded Ratio, it is possible to increase code efficiency.

Additionally, on the contrary, only by second or the 3rd feelings of being encoded of coded method Under condition, although the increase that there is process delay becomes adverse conditions, but passes through and use first to compile Code mode, it is possible to suppression processes the increase postponed, and also achieves the maintenance of image quality.

And, visual point image encoding section 110 and depth image encoding section 120 are as described above And when visual point image Pv and depth image Pd being encoded by multiple coded systems, as front State like that, by predetermined each coded system switch data unit switching coded system.Additionally, With reference to information treatment part 170, coded data string STR is inserted with reference to information between image between image, Make to be decoded accordingly with the coded system of each coded system switch data unit.

Therefore, illustrate coded system switch data unit in present embodiment example and with often With reference to information between the image in the coded data string STR that individual coded system switch data unit is corresponding The example of on position.

First, an example of coded system switch data unit is sequence.Now, coded system is certainly Determine portion 130 determines should apply which mode from first～the 3rd coded system to each sequence. Further, visual point image encoding section 110 and depth image encoding section 120 are by regarding in each sequence Dot image Pv and depth image Pd encode according to the coded system determined respectively.

Fig. 6 (a) represents the figure corresponding with the example that sequence is set to coded system switch data unit The example of the on position of reference information Dref between Xiang.It is sequence in coded system switch data unit In the case of, by reference letter between image between image with reference to information treatment part 170 as shown in the figure Breath Dref is inserted into the precalculated position in the RBSP of the SPS in coded data string STR.

That is, with reference to information Dref, this precalculated position is appointed as on position and by image between image Between export multiplexing unit 180 with reference to information Dref.Multiplexing unit 180 carries out coded data string STR Multiplexing process so that specified on position will be inserted into reference to information Dref between image.

Additionally, an example of coded system switch data unit is picture.Now, coded system is certainly Determine portion 130 determines should apply which mode from first～the 3rd coded system to each picture. Further, visual point image encoding section 110 and depth image encoding section 120 are by regarding in each picture Dot image Pv and depth image Pd encode according to the coded system determined respectively.

Fig. 6 (b) represents the figure corresponding with the example that picture is set to coded system switch data unit The example of the on position of reference information Dref between Xiang.It is picture in coded system switch data unit In the case of, as shown in the figure, with reference to information treatment part 170 by reference letter between image between image Breath Dref is inserted into the precalculated position in the RBSP of each PPS in coded data string STR.

Additionally, an example of coded system switch data unit is fragment.Now, coded system is certainly Determine portion 130 determines should apply which mode from first～the 3rd coded system to each fragment. Further, visual point image encoding section 110 and depth image encoding section 120 are by regarding in each fragment Dot image Pv and depth image Pd encode according to the coded system determined respectively.

Fig. 6 (c) represents the figure corresponding with the example that fragment is set to coded system switch data unit The example of the on position of reference information Dref between Xiang.It is fragment in coded system switch data unit In the case of, by reference letter between image between image with reference to information treatment part 170 as shown in the figure Breath Dref is inserted in the slice header of beginning configuration of the RBSP of NAL unit 400.

Fig. 6 (d) represents that the NAL unit head memory in NAL unit 400 has stored up between image Example with reference to information Dref.

As illustrated in fig. 5, NAL unit head is attached to the various of SPS, PPS, fragment etc. In data.Therefore, as shown in Fig. 6 (d), in NAL unit head, storage has reference between image In the case of information Dref, according to the information of this AL unit 400 storage, with reference to letter between image Coded system switch data unit corresponding for breath Dref changes.It means that carry out many During visual point image coding, such as, can switch coded system between sequence and picture and fragment and cut Change the classification of data unit.

That is, it is inserted into reference to information Dref between image in RBSP, stores the NAL of SPS In the case of in the NAL unit head of unit 400, coded system switch data unit becomes sequence.

Additionally, be inserted into the NAL storing PPS in RBSP between image with reference to information Dref In the case of in the NAL unit head of unit 400, coded system switch data unit becomes picture. Additionally, PPS can also specify the multiple pictures in a part for such as picture.Therefore, if In the case of multiple fractional unit switching coded system (reference relation), with Fig. 6 (c) Situation compare, it is possible to cut down coded data tediously long.

Additionally, deposit in the NAL unit head to the NAL unit 400 of RBSP Insert Fragment Containing with reference in the case of information Dref between image, coded system switch data unit becomes fragment.

Additionally, in the example of Fig. 6 (d), needing to distinguish with NAL unit unit is viewpoint figure Which in picture and depth image.As long as to this end, as represent image classification information and Component category information is stored in NAL unit head.Component refers to become the figure of coded object The classification of picture.Visual point image and depth image are the one of component respectively.

It addition, represent that the information of the classification of this image can also replace component category information to utilize The NAL unit identification information being included in standard in NAL unit head.I.e., it is also possible to root According to NAL unit identification information, identify the SPS of visual point image, the PPS of visual point image, viewpoint The fragment of image, the SPS of depth image, the PPS of depth image, the fragment etc. of depth image.

As long as additionally, with reference to information Dref such as about as visual point image or deep between image One of the component of degree image, represents the information that whether have references to another component when it encodes ?.Now, can be defined as carrying out table by " 1 " and " 0 " with reference to information Dref between image Show the labelling (inter_component_flag) of 1 bit of the image that whether have references to other.

Specifically, in the case of the first coded system, about coded views image Pv_enc Image between with reference to information Dref, storage represents not " 0 " with reference to depth image Pd.Additionally, Non-reference is represented about also storing with reference to information Dref between the image of coding depth image Pd_enc " 0 " of visual point image Pv.

Additionally, in the case of the second coded system, about coded views image Pv_enc's With reference to information Dref between image, storage represents " 1 " with reference to depth image Pd.On the other hand, About reference information Dref between the image of coding depth image Pd_enc, storage represents non-reference " 0 " of visual point image Pv.

Additionally, in the case of the 3rd coded system, about coded views image Pv_enc's With reference to information Dref between image, storage represents " 0 " of not reference depth image Pd.The opposing party Face, about reference information Dref between the image of coding depth image Pd_enc, storage represents ginseng " 1 " according to visual point image Pv.

Alternatively, it is also possible to reference to information Dref between replacement image, e.g. represent by first to Which mode in 3rd coded system and the information that encodes.

[the process step example of picture coding device]

The process step example that the flowchart representation picture coding device 100 of Fig. 7 performs.

Here, first, the coding from visual point image Pv starts explanation.Coded system determination section 130 pairs of predetermined each coded system switch data units determine the coding of visual point image Pv Mode (step S101).

Then, visual point image encoding section 110 is to comprise in coded system switch data unit Visual point image Pv is object, starts based on the coding of determined coded system.Starting this volume During code, visual point image encoding section 110 judges that whether the coded system determined is as should refer to other Component, i.e. the mode (step S102) of depth image Pd.

In the case of should refer to depth image Pd (step S102-is), visual point image encodes Portion 110 performs coding (step S103) with reference to the depth image Pd of the component as other. That is, as it was previously stated, visual point image encoding section 110 reads correspondence from coded image storage part 140 Decoding depth image, and utilize this reading decoding depth image and carry out visual point image Pv's Coding.

Further, generate expression with reference to information treatment part 170 between image to be carried out by step S103 The figure that the component (visual point image) of coding is the component (depth image) with reference to other and encodes With reference to information Dref (step S104) between Xiang.Specifically, with reference to information treatment part between image " 1 " is set with reference to information Dref between the image of 170 pairs of 1 bits.

On the other hand, in the case of reference should not be made to depth image Pd (step S102-is no), Visual point image encoding section 110, without reference to the depth image Pd of the component as other, is only passed through The identical predictive coding between component (visual point image) and perform coding (step S105).

Further, between image with reference to information treatment part 170 generate represent step S105 and encode point Amount (visual point image) is to join between the image not encoded with reference to other component (depth image) According to information Dref (step S106).Specifically, right with reference to information treatment part 170 between image " 0 " is set with reference to information Dref between the image of 1 bit.

Additionally, coded system determination section 130 is in step S101, also same to depth image Pd Sample ground determines coded system.Determining according to this, depth image encoding section 120 performs and step Process that S102, S103, S105 are corresponding and depth image Pd is encoded.Additionally, figure With reference to information treatment part 170 by generating figure with step S104, process identical for S106 between Xiang With reference to information Dref between Xiang.

Further, will join between image generated as above with reference to information treatment part 170 between image According to information Dref according to predetermined coded system switch data unit, as shown in Figure 6, will It is inserted into the precalculated position (step in coded data string STR with reference to information Dref between image S107).That is, its on position is specified and to multiplexing unit with reference to information treatment part 170 between image With reference to information Dref between 180 output images.

Though it addition, be shown without in the figure, with the component of step S103 and S105 While coding, also carry out the coding of the photography conditions information of photography conditions information encoding section 150. Further, multiplexing unit 180 inputs encoded component (coded views image Pv_enc and coding is deep Degree image Pd_enc) and encoded photogrammetric conditional information and the head that generated by step S108. Further, multiplexing unit 180 carries out time division multiplex so that by these data of input according to suitable Put in order arrangement, and exports (step S108) as coded data string STR.

[structure of picture decoding apparatus]

Fig. 8 represents the structure example of the picture decoding apparatus 200 in present embodiment.Shown in this figure Picture decoding apparatus 200 include code extraction unit 210, visual point image lsb decoder 220, the degree of depth Picture decoding portion 230, decoding image storage part 240, decoding control section 250, photography conditions letter Breath lsb decoder 260, visual point image generating unit 270, visual point image correspondence table storage part 280 and Depth image correspondence table storage part 290.

Code extraction unit 210 from input coded data string STR, extract auxiliary information Dsub, Coded views image Pv_enc, coding depth image Pd_enc and encoded photogrammetric conditional information Ds_enc.It addition, in auxiliary information Dsub, including between the image illustrated by Fig. 6 With reference to information Dref.

The visual point image lsb decoder 220 coded views image Pv to separating from coded data string STR _ enc is decoded and generates visual point image Pv_dec, and exports decoding image storage part 240.Visual point image lsb decoder 220, when being decoded coded views image Pv_enc, needs In the case of depth image, read the degree of depth of storage in decoding image storage part 240 Image Pd_dec.Further, utilize the depth image Pd_dec of this reading to coded views image Pv_enc is decoded.

The depth image lsb decoder 230 coding depth image Pd to separating from coded data string STR _ enc is decoded and generates depth image Pd_dec, and exports decoding image storage part 240.Depth image lsb decoder 230, when being decoded coding depth image Pd_enc, needs In the case of visual point image, read the viewpoint of storage in decoding image storage part 240 Image Pv_dec.Further, utilize the visual point image Pv_dec of this reading to coding depth image Pd_enc is decoded.

The visual point image Pv_ that visual point image lsb decoder 220 is decoded by decoding image storage part 240 The depth image Pd_dec that dec and depth image lsb decoder 230 generate stores.Additionally, The visual point image Pv_i generated by visual point image generating unit 270 described later is stored. Visual point image Pv_i is used in such as to the volume encoded by View Synthesis predictive coding Code visual point image Pv_enc is decoded.

As described above, the visual point image Pv_dec of decoding image storage part 240 storage utilizes When depth image lsb decoder 230 is decoded with reference to visual point image.Similarly, decoding image The depth image Pd_dec of storage part storage is used in visual point image lsb decoder 220 with reference to depth map During as being decoded.

Additionally, decoding image storage part 240 is by the visual point image Pv_dec stored and the degree of depth Image Pd_dec is such as sequentially outputted to outside according to according to the output of the DISPLAY ORDER etc. specified.

As described above, from the visual point image Pv_dec of picture decoding apparatus 200 output with deep Degree image Pd_dec is reproduced by not shown transcriber or application etc..Thus, such as enter The display of row multi-view image.

The content of decoding control section 250 auxiliary information Dsub based on input explains coded data String STR, and according to its explanation results, control visual point image lsb decoder 220 and depth image solution The decoding process in code portion 230.As one of control for this decoding process, decoding control section 250, based on reference information Dref between the image comprised in auxiliary information Dsub, are carried out as follows control System.

That is, it is set between image with reference to the solution in information Dref presentation code mode switch data unit The component (decoded object images) of code object is carried out with reference to other component (with reference to image) Coding.Now, decoding control section 250 controls visual point image lsb decoder 220 or depth image Lsb decoder 230 so that the component of decoder object is decoded by the component with reference to other.

Specifically, between image, represent that the component with reference to other is compiled with reference to information Dref In the case of Ma, the component at decoder object is visual point image and other component is depth image In the case of, decoding control section 250 is carried out as follows control.That is, decoding control section 250 controls Visual point image lsb decoder 220 so that coded views image Pv_enc is with reference to depth image Pd_ Dec and be decoded.

On the other hand, between image, represent that the component with reference to other is compiled with reference to information Dref In the case of Ma, the component at decoder object is depth image and other component is visual point image In the case of, decoding control section 250 is carried out as follows control.That is, decoding control section 250 controls Depth image lsb decoder 230 so that coding depth image Pd_enc is with reference to visual point image Pv_ Dec and be decoded.

Additionally, be set between image with reference in information Dref presentation code mode switch data unit The component of decoder object not component with reference to other is encoded.

Now, decoding control section 250 is controlled so that without reference to other component to decoding The component of object is decoded.

Specifically, decoding control section 250 now controls visual point image lsb decoder 220, makes Must be in the case of the component of decoder object be visual point image, coded views image Pv_enc does not joins It is decoded according to depth image Pd_dec.On the other hand, the component at decoder object is depth map In the case of Xiang, controlling depth picture decoding portion 230 so that coding depth image Pd_enc is not It is decoded with reference to visual point image Pv_dec.

Here, as described above, at the component with reference to other, component of decoder object is solved During code, other component of reference is needed to have been carried out decoding.Therefore, decoding control section 250 When the decoding of coded views image Pv_enc and coding depth image Pd_enc, control is right The order that coded views image Pv_enc and coding depth image Pd_enc is decoded so that The component that should refer to becomes the state that decoding is complete.

When this control, decoding control section 250 utilizes at visual point image correspondence table storage part 280 The visual point image correspondence table of middle storage is deep with storage in depth image corresponding table storage part 290 Degree image correspondence table.It addition, about make use of visual point image correspondence table table corresponding with depth image Decoding order control example, described below.

The encoded photogrammetric conditional information Ds_enc separated is entered by photography conditions information lsb decoder 260 Row decodes and generates photography conditions information Ds_dec.Photography conditions information Ds_dec exports outward Portion, and export visual point image generating unit 270.

Visual point image generating unit 270 utilizes the decoding of storage in decoding image storage part 240 to regard Dot image and decoding depth image and photography conditions information Ds_dec, generate visual point image Pv_i.Decoding image storage part 240 stores the visual point image Pv_i generated.

Visual point image correspondence table storage part 280 stores visual point image correspondence table.

Fig. 9 (a) represents the structure example of visual point image correspondence table 281.As shown in the drawing, viewpoint In image correspondence table 281, by each viewpoint sequence number, with reference to the value of information and decoded result between image Information sets up correspondence.

Viewpoint sequence number is that each viewpoint to the multiple viewpoints corresponding with visual point image Pv is composed in advance The sequence number given.Such as, viewpoint #0 shown in Fig. 2, #1, #2 are imparted viewpoint sequence respectively Numbers 0,1,2.

The coded views about each viewpoint sequence number the most in the same time is stored with reference to the value of information between image With reference to the content of information Dref between the image of image Pv_enc, i.e. with reference to information between storage image The value that Dref represents.As described above, table is carried out with reference to information Dref by the value of " 1 " between image Show the component with reference to other (being now depth image), by the value of " 0 " represent without reference to Other component.

Decoding result information represents the coded views image Pv_enc about corresponding viewpoint sequence number Decoding whether terminate.Now, decoding result information is such as set to the information of 1 bit, passes through The value of " 1 " represents and completes decoding, is represented by the value of " 0 " and does not complete decoding.

In the example of Fig. 9 (a), viewpoint sequence number represents " 0 "～" 5 ".I.e., now, table Show the example setting six different viewpoints.

And, although representing viewpoint sequence number " 0 " with reference to the value of information between the image in Fig. 9 (a) Coded views image Pv_enc encodes without reference to depth image, but remaining viewpoint sequence number The coded views image Pv_enc of " 1 "～" 5 " encodes with reference to depth image.This expression regards The coded views image Pv_enc of some sequence number " 0 " reference should not be made to depth image and decodes, but regards The coded views image Pv_enc of some sequence number " 1 "～" 5 " should refer to depth image and decodes.

Additionally, the decoding result information in Fig. 9 (a) represents at a time, viewpoint sequence number " 0 " The coded views image Pv_enc of " 1 " completes decoding, but viewpoint sequence number " 2 "～" 5 " Coded views image Pv_enc do not complete decoding.

Depth image correspondence table storage part 290 storage depth image correspondence table.

Fig. 9 (b) represents the structure example of depth image correspondence table 291.As shown in the drawing, the degree of depth In image correspondence table 291, by each viewpoint sequence number, with reference to the value of information and decoded result between image Information sets up correspondence.

Viewpoint sequence number is the every of the multiple viewpoints to the visual point image Pv corresponding with depth image Pd The sequence number that individual viewpoint gives in advance.

The coding depth about each viewpoint sequence number the most in the same time is stored with reference to the value of information between image The value represented with reference to information between the image of image Pd_enc.

Decoding result information represents the coding depth image Pd_enc about corresponding viewpoint sequence number Decoding whether terminate.Now, decoding result information is such as set to the information of 1 bit, passes through The value of " 1 " represents and completes decoding, is represented by the value of " 0 " and does not complete decoding.

In Fig. 9 (b), also illustrate that, as viewpoint sequence number, " 0 "～" 5 " is shown, set The example of six different viewpoints.

And, although representing viewpoint sequence number " 0 " with reference to the value of information between the image in Fig. 9 (b) The coding depth image Pd_enc of " 2 "～" 5 " encodes without reference to visual point image, but The coding depth image Pd_enc of viewpoint sequence number " 1 " encodes with reference to visual point image.This represents The coding depth image Pd_enc of viewpoint sequence number " 0 " and " 2 "～" 5 " reference should not be made to regard Dot image and decode, but the coding depth image Pd_enc of viewpoint sequence number " 1 " should refer to viewpoint Image and decode.

Additionally, the decoding result information in Fig. 9 (b) represents at a time, viewpoint sequence number " 0 "～ The depth image Pd_enc of " 2 " completes decoding, but viewpoint sequence number " 3 "～" 5 " is deep Degree image Pd_enc does not complete decoding.

The flowchart representation picture decoding apparatus 200 of Figure 10 is for regarding the coding of some viewpoint The process step example that dot image Pv_enc is decoded.

First, decoding control section 250 is with reference to the image comprised in auxiliary information Dsub of input Between with reference to information Dref (step S201), value with reference to information Dref between the image of reference is deposited Store up coded views image Pv_enc with decoder object in visual point image correspondence table 281 corresponding Viewpoint sequence number image between with reference to (step S202) in the value of information.

Additionally, meanwhile, decoding control section 250 in visual point image correspondence table 281 with solution In the decoding result information of the viewpoint sequence number corresponding for coded views image Pv_enc of code object, make Store for initial value and represent " 0 " (step S203) not completing decoding.

Then, decoding control section 250 is judged between the image stored by step S202 with reference to letter Whether breath value is " 1 " (step S204).This is equivalent to, it is determined that the coded views of decoder object Whether image Pv_enc is the image encoded with reference to depth image, i.e. decoder object Whether coded views image Pv_enc should refer to depth image and is decoded.

It is (step S204-is) in the case of " 1 " with reference to the value of information between image, decoding control Portion 250 processed etc. stays in the coded views image Pv in depth image correspondence table 291 with decoder object The decoding result information of viewpoint sequence number identical for _ enc becomes " 1 " (step S205-is no).

That is, decoding control section 250 is in the decoding of the coded views image Pv_enc of decoder object Time, till waiting until that the depth image Pd_dec (other components) that should refer to is decoded.

Further, if having carried out decoding according to depth image Pd_dec, decoding result information becomes " 1 " (step S205-is), then visual point image lsb decoder 220 is referred to by decoding control section 250 Show that decoding starts (step S206).

Additionally, be not (step S204-is no) in the case of " 1 " with reference to the value of information between image, Decoding control section 250 is skipped step S205 and is indicated decoding to start visual point image lsb decoder 220 (step S206).Decoding control section 250 i.e., now is to be not to wait for and identical viewpoint sequence The decoding of number coding depth image Pd_enc corresponding with the moment but to visual point image lsb decoder The parts that 220 instruction decodings start.

Starting instruction according to decoding, visual point image lsb decoder 220 judges at visual point image correspondence table With reference to information between the image of the viewpoint sequence number of the coded views image Pv_enc of decoder object in 281 Whether value is " 1 " (step S207).That is, visual point image lsb decoder 220 judges decoder object Coded views image Pv_enc whether should refer to depth image and decode.

It is (step S207-is) in the case of " 1 ", viewpoint figure with reference to the value of information between image As lsb decoder 220 has started with the decoding (step S208) of the encoded object image with reference to image.

That is, visual point image lsb decoder 220 is from decoding image storage part 240 as with reference to image Read in corresponding to the viewpoint sequence number identical with the coded views image Pv_enc of decoder object and time The depth image Pd_dec carved.Further, the depth image Pd_dec of this reading is utilized to start The decoding of coded views image Pv_enc.

On the other hand, between image, the reference value of information is in the case of " 0 " (step S207-is no), Visual point image lsb decoder 220 starts not utilize the coding of depth image Pd_dec (with reference to image) The decoding (step S209) of visual point image Pv_enc (decoded object images).

So, join between the image that visual point image lsb decoder 220 stores with reference to decoding control section 250 According to the value of information, decide whether to should refer to depth map to the coded views image Pv_enc of decoder object As and decode.This means that the decoding process of visual point image lsb decoder 220 is decoded control portion 250 Controlled.

Started step S208 or S209 coded views image Pv_enc decoding it After, decoding control section 250 waits that this decoding completes (step S210-is no).Further, if decoding Complete (step S210-is), then visual point image lsb decoder 220 is at visual point image correspondence table 281 In corresponding with the viewpoint sequence number of the coded views image Pv_enc of decoder object decoded result letter Breath storage represents " 1 " (step S211) completing decoding.

It addition, when coding depth image Pd_enc is decoded, also application and Figure 10 phase Same process.

Now, decoding control section 250 is with reference to the coding depth image Pd_enc with decoder object With reference to information Dref (step S201) between corresponding image.Further, decoding control section 250 will Between the image of reference, the value with reference to information Dref stores decoding in depth image correspondence table 291 With reference to the value of information between the image of the viewpoint sequence number corresponding to coding depth image Pd_enc of object In (step S202).Additionally, decoding control section 250 in depth image correspondence table 291 with In the decoding result information of the viewpoint sequence number corresponding for coding depth image Pd_enc of decoder object, Store as initial value and represent " 0 " (step S203) not completing decoding.

Decoding control section 250 be judged between image be " 1 " with reference to the value of information in the case of (step Rapid S204-is), wait the coding depth figure stayed in visual point image correspondence table 281 with decoder object As the decoding result information of viewpoint sequence number identical for Pd_enc becomes " 1 " (step S205-is no).

" 1 " (step S205-is), decoding control section 250 is become according to decoding result information Decoding is indicated to start (step S206) depth image lsb decoder 230.

Additionally, be not (step S204-is no) in the case of " 1 " with reference to the value of information between image, Decoding control section 250 is skipped step S205 and is indicated decoding to start depth image lsb decoder 230 (step S206).

Starting instruction according to decoding, depth image lsb decoder 230 judges at depth image correspondence table With reference to information between the image of the viewpoint sequence number of the coding depth image Pd_enc of decoder object in 291 Whether value is " 1 " (step S207).

It is (step S207-is) in the case of " 1 ", depth map with reference to the value of information between image As lsb decoder 230 utilizes the visual point image Pv_dec read in from decoding image storage part 240 to open The decoding of beginning coding depth image Pd_enc.

On the other hand, between image, the reference value of information is in the case of " 0 " (step S207-is no), Depth image lsb decoder 230 starts not utilize the coding of visual point image Pv_dec (with reference to image) The decoding (step S209) of depth image Pd_enc (decoded object images).

Started step S208 or S209 coding depth image Pd_enc decoding it After, decoding control section 250 waits that this decoding terminates (step S210-is no).Further, if decoding Terminate (step S210-is), then depth image lsb decoder 230 is at depth image correspondence table 291 In corresponding with the viewpoint sequence number of the coding depth image Pd_enc of decoder object decoded result letter Breath storage represents " 1 " (step S211) completing decoding.

Coded views image Pv_enc as illustrated by figure 3, in coded data string STR It is the order according to the reference relation encoded with putting in order of coding depth image Pd_enc.

It is thus possible, for instance the judgement of step S204 for Figure 10, at visual point image correspondence table 281 Or the timing with reference to the value of information has started reference object between the image of depth image correspondence table 291 The decoding of image.Therefore, by the coding that should refer to other the image of component and decode Step S204 and the S205 of Figure 10 is applied, it is possible to be reliably completed ginseng during the decoding of image After the decoding of the image of target, start the decoding of the coded image of decoder object.I.e., originally Embodiment can significantly suppress at the picture decoding of mode that decodes at the component with reference to other The delay of reason.

Alternatively, it is also possible to the program of the function of each several part in Fig. 1 and Fig. 8 will be used for realizing Record, in the record medium that computer can read, makes the program of record in this record medium Read in computer system and perform, thus carry out coding and the decoding of image.It addition, institute here " computer system " claimed includes the hardware of OS or ancillary equipment etc..

Additionally, " computer system " is as long as utilize the situation of WWW system, the most also include Homepage provides environment (or display environment).

Additionally, " the record medium that computer can read " refers to, floppy disk, photomagneto disk, ROM, The removable medium of CD-ROM etc., the storage device of the most built-in hard disk etc.. Additionally, " the record medium that computer can read " also includes as become via the Internet etc. Network or the communication line of telephone line etc. and the server in the case of have sent program or visitor The volatile memory (RAM) of the inside computer system of family machine is such, the set time keeps The medium of program.Additionally, said procedure both could be for realizing a part for aforesaid function Program, in addition it is also possible to be can be by already recorded in the program in computer system Combine and realize the program of aforesaid function.

Above, describe embodiments of the present invention the most in detail, but concrete structure is also It is not limited to this embodiment, also includes the design etc. of the scope of the purport without departing from the present invention.

Symbol description

100 picture coding devices

110 visual point image encoding section

120 depth image encoding section

130 coded system determination sections

140 coded image storage parts

150 photography conditions information encoding section

160 visual point image generating units

With reference to information treatment part between 170 images

180 multiplexing units

200 picture decoding apparatus

210 code extraction units

220 visual point image lsb decoders

230 depth image lsb decoders

240 decoding image storage parts

250 decoding control section

260 photography conditions information lsb decoders

270 visual point image generating units

280 visual point image correspondence table storage parts

281 visual point image correspondence tables

290 depth image correspondence table storage parts

291 depth image correspondence tables

Claims (6)

1. a picture coding device, it is characterised in that including:

Visual point image encoding section, when entering the multiple visual point images corresponding from different viewpoints respectively Row coding time, when should refer to the depth image corresponding with described visual point image, with reference to described deeply Described visual point image in coding units is encoded by degree image, reference should not be made to the described degree of depth During image, without reference to described depth image, the described visual point image in described coding units is carried out Coding；

Depth image encoding section, when encoding described depth image, described in should refer to During visual point image, with reference to described visual point image, the described depth image in described coding units is entered Row coding, when reference should not be made to described visual point image, without reference to described visual point image to described volume Described depth image in code unit encodes；And

With reference to information treatment part between image, it is right to be denoted as encoding by each described coding units Any one in the described visual point image of the component of elephant or described depth image is with reference to as other The described visual point image of component or described depth image carry out encoding the most not with reference to as other The described visual point image of component or described depth image carry out between the image of reference relation encoded With reference to information, it is inserted into and includes encoded visual point image and the coding of encoded depth image In serial data.

2. picture coding device as claimed in claim 1, it is characterised in that

Corresponding to described coding units with reference to information treatment part between described image is the situation of fragment, To be inserted in the head of the fragment in described coded data string with reference to information between described image, this institute State the institute of the component being denoted as coded object between image with reference to information by each described coding units Stating any one in visual point image or described depth image is with reference to as described in other component Visual point image or described depth image carry out encoding the most not with reference to as described in other component Visual point image or described depth image carry out the reference relation encoded.

3. a picture decoding apparatus, it is characterised in that including:

Code extraction unit, extracts respectively to the viewpoint corresponding from different viewpoints from coded data string Coded views image that image carries out encoding, the depth image corresponding with described visual point image is entered Row coding coding depth image, represent to described viewpoint figure by each predetermined coding units Described visual point image when picture or described depth image encode and the ginseng of described depth image According to reference information between the image of relation；

Visual point image lsb decoder, is decoded the described coded views image extracted；

Depth image lsb decoder, is decoded the described coding depth image extracted；And

Decoding control section, based on the reference relation represented with reference to information between the described image extracted, To any one in the coded views image of the component as decoder object or coding depth image it is Described visual point image or described depth image with reference to the component as other are decoded the most not Described visual point image or described depth image with reference to the component as other are decoded controlling System.

4. picture decoding apparatus as claimed in claim 3, it is characterised in that

Described decoding control section head based on the fragment from described coded data string is extracted With reference to information between described image, coded views image or the coding to the component as decoder object Any one in depth image be with reference to as other component described visual point image or described deeply Degree image be decoded or without reference to the component as other described visual point image or described deeply Degree image is decoded being controlled.

5. a method for encoding images, it is characterised in that including:

Visual point image coding step, when respectively to the multiple visual point images corresponding from different viewpoints When encoding, when should refer to the depth image corresponding with described visual point image, with reference to described Described visual point image in coding units is encoded by depth image, reference should not be made to described deeply During degree image, without reference to described depth image, the described visual point image in described coding units is entered Row coding；

Depth image coding step, when encoding described depth image, should refer to When stating visual point image, with reference to described visual point image to the described depth image in described coding units Encode, when reference should not be made to described visual point image, without reference to described visual point image to described Described depth image in coding units encodes；And

With reference to information handling step between image, coding will be denoted as by each described coding units Any one in the described visual point image of the component of object or described depth image is with reference to as it The described visual point image of his component or described depth image carry out encoding the most not with reference to as it The described visual point image of his component or described depth image carry out the image of the reference relation encoded Between with reference to information, be inserted into and include described coded views image and the volume of described coding depth image In code serial data.

6. a picture decoding method, it is characterised in that including:

Code extraction step, from coded data string extract respectively to from different corresponding the regarding of viewpoint Coded views image that dot image carries out encoding, to the depth image corresponding with described visual point image The coding depth image that carries out encoding, represent to described viewpoint by each predetermined coding units Described visual point image when image or described depth image encode and described depth image With reference to information between the image of reference relation；

Visual point image decoding step, is decoded the described coded views image extracted；

Depth image decoding step, is decoded the described coding depth image extracted；With And

Decoding rate-determining steps, closes based on the reference represented with reference to information between the described image extracted System, arbitrary in the coded views image of the component as decoder object or coding depth image Individual is to be decoded also with reference to as other the described visual point image of component or described depth image Be the described visual point image without reference to the component as other or described depth image be decoded into Row controls.