CN1507274A - Method for determining repeated pattern frame interpolating method and frame interpolating apparatus - Google Patents

Abstract

A repeated pattern judging method, a frame interpolation method using the same and an apparatus suitable for the same are provided to discriminate that a repeated pattern moves between frames. An MAE(Mean Ratio Error) calculating unit(1202) obtains MAEs with respect to respective reference blocks, which exist in a search range with an (M+2P)x(N+2P) size of a frame which refers for searching a reference block and a motion vector, by a full search method. An MAE map storing unit(1204) stores (M+P)x(N+P) MAEs as a map type with an (M+P)x(N+P) size. A motion vector extracting unit(1206) judges positions(x,y) of horizontal and vertical directions of a reference block having the least MAE among the MAEs stored in the MAE map storing unit(1204) as an interframe motion direction with respect to a current block. A repeated pattern recognizing unit(1210) discriminates whether a repeated pattern exists with reference to the MAEs stored in the MAE map storing unit(1204). A frame interpolation unit(1212) performs frame interpolation using a motion vector extracted in the motion vector extracting unit(1206) and linear interpolation.

Description

Determine that method, frame interpolation method and the frame interpolation of repeat patterns put

Technical field

The present invention relates to a kind of converting frame rate device, relate in particular to a kind of method, this method determines whether piece is included in the repeat patterns, is used to have the valid frame interpolation of the image of repeat patterns, and frame interpolation method and frame interpolation are put.

Background technology

The resolution of image depends on the pixel count that is comprised in the frame.1,920 * 1,080 ultimate resolution, a frame is made up of 1,920 pixel on each horizontal line and 1,080 pixel on each vertical line.Frame rate is represented the number of the frame that per second sends.When the picture signal that sends as TV signal, determine frame rate based on people's visual characteristic.

Usually, according to the requirement of locality, with the picture signal of various frequency broadcasting from image output device output.For example, in Europe and China, should export the picture signal that vertical frequency is 50Hz, and, should export the picture signal that vertical frequency is 60Hz in Korea S and North America.

When the picture signal of the various frequencies of output, image output device needs frequency translation.This frequency translation is called as converting frame rate.Particularly, when low frequency transform is arrived high frequency, must increase the number of frame.

In the past, by using the motion vector of estimating based on the difference between the consecutive frame, by repeating consecutive frame or creating the number that new frame increases frame.

High-resolution system checks the motion of present frame according to the movement tendency of image, constructs the image that seems nature to use the motion vector correction technology.This motion vector correction technology comprises: extract the piece with a plurality of motion vectors (motion vector that comprises correction target), and proofread and correct with this piece in the different motion vector of those nearby motion vectors directions, so the motion vector after the correction is identical with the nearby motion vectors direction.

As mentioned above, the conventional frame interpolating method uses the correlation of interframe to be provided for the superperformance of proper motion image or rest image.Yet, when the image with repeat patterns when interframe moves, the conventional frame interpolating method can't guarantee enough frame interpolation performances, because be difficult to accurate estimating motion vector.

For example, when the pattern that periodically repeats moved, for example striped shirt, striped tablecloth or the building of a string window is arranged were then compared with proper motion image or rest image, are difficult to the motion vector between accurate estimated frame.This is because the correlation of interframe changes significantly.

Therefore, need a kind of method, be used for the situation of determining that repeat patterns moves in interframe, with and effective frame interpolation method.

Summary of the invention

The invention provides a kind of method, be used for when the image with repeat patterns moves, determine the pattern that repeats.

The present invention also provides a kind of frame interpolation method, uses when the image with repeat patterns moves.

The present invention also provides a kind of frame interpolation to put, and uses this frame interpolation method.

According to an aspect of the present invention, a kind of method is provided, be used for determining when the image with repeat patterns moves, whether the reference block of M * N is included in the described repeat patterns, this method may further comprise the steps: use search fully, obtain the error of calibrated bolck and reference block, each reference block is comprised in the region of search of (M+2P) * (N+2P), and this region of search belongs to the frame of institute's reference in the motion vector sought process; According to the order of reference block, arrange (M+P) * (N+P) individual error with the form of (M+P) * (N+P) figure; In the left diagonal and right diagonal of this figure, obtain each deviation between current block and the adjacent block, and be accumulated in deviation that obtains in the left diagonal and the deviation that in right diagonal, obtains respectively; The deviation of accumulating in deviation of accumulating in the more left diagonal and the right diagonal, and choose bigger deviation; Selected deviation and threshold values 1 are compared; And if selected deviation determines then that greater than threshold values 1 this reference block is included in the repeat patterns.

This method also comprises: the sub-piece that this figure is divided into identical size; Calculate the ratio of the worst error and the minimal error of each sub-piece; Counting has the sum greater than the sub-piece of the ratio of threshold values 2; And if should sum greater than threshold values 3, then determine this a little and be included in the repeat patterns.

This method also comprises: inspection has the distribution greater than the sub-piece of the ratio of threshold values 2, and determines whether this sub-piece is included in the pseudo-repeat patterns.

In the method, by relatively on the level of this figure, vertical and diagonal, concentrating the pseudo-repeat patterns of the sub-piece that distributes, determine whether this sub-piece is included in this puppet repeat patterns.

This method also comprises: check the repetition degree that is confirmed as being included in the piece in the repeat patterns in reference block and the adjacent block; And, determine that reference block is included in this repeat patterns based on the degree that repeats.

According to a further aspect in the invention, a kind of frame interpolation method is provided, may further comprise the steps: (a) use search fully, obtain the error of calibrated bolck and reference block, each reference block is comprised in the region of search of (M+2P) * (N+2P), and this region of search belongs to the frame that is referenced in the motion vector sought process; (b) estimating motion vector is the positional information with reference block of minimal error; (c), determine whether described calibrated bolck and reference block are included in the repeat patterns according to the error that is obtained; (d) calculate the current block that will be interpolated and around the correlation between the adjacent block of this current block; And (e) according to the correlation of being calculated, by mixing piece that forms by linear interpolation and the piece that forms by estimation and motion compensation (ME/MC), the image after the acquisition interpolation.

Step (d) further comprises: count the number that all center on reference block and are included in all adjacent blocks in the repeat patterns; And calculate around the ratio of the number of the sum of the adjacent block of reference block and counting.

By interpolation the current block that is formed is represented as

$<\mathrm{PSTYLELSPACE}=130>f(i,j)=\frac{(\mathrm{\&alpha;}\&times;\mathrm{linear}(i,j)+(\mathrm{total}-\mathrm{\&alpha;})\&times;\mathrm{MC}(i,j))}{\mathrm{total}},$

Wherein, (α represents to be included in the number of the adjacent block in the repeat patterns to f for i, the j) location of pixels of expression current block, and total represents the sum of adjacent block.

According to a further aspect in the invention, provide a kind of frame interpolation to put, comprise: mean absolute error (MAE) computing unit, it uses search fully, MAE between basis of calculation piece and reference block, each reference block is comprised in the region of search of (M+2P) * (N+2P), and this region of search belongs to the frame of institute's reference in the motion vector sought process; MAE figure memory cell, it is according to the order of reference block, with form storage (M+P) * (N+P) individual error of (M+P) * (N+P) figure; Motion vector extraction unit, direction of motion of its identification current block be position with the reference block of the minimum MAE among each MAE that is stored among this MAE figure (x, y); The repeat patterns determining unit, it determines with reference to the MAE figure that is stored in the MAE figure memory cell whether piece is included in the repeat patterns; And the frame interpolation unit, use the motion vector and the linear interpolation of extracting to come interpolation frame.

Description of drawings

More than of the present invention and other aspects and advantage will become apparent when describing one exemplary embodiment with reference to the accompanying drawings, wherein:

Fig. 1 is the schematic diagram of symmetry blocks matching process;

The block diagram that Fig. 2 puts for the frame interpolation that uses conventional motion estimation and motion compensation (ME/MC);

Fig. 3 is the flow chart according to the method for definite repeat patterns of the present invention;

Fig. 4 illustrates being included in the piece in the repeat patterns, the figure that is formed by the MAE in the left and right sides diagonal that is presented on MAE figure;

Fig. 5 illustrates the MAE figure of general pattern;

Fig. 6 illustrates the MAE figure that is included in the piece in the repeat patterns, and wherein, each MAE figure is divided into 16 sub-pieces, and marks the zone that has greater than 3.0 MAE ratio with stain;

Fig. 7 illustrates the image with pseudo-repeat patterns;

Fig. 8 illustrates MAE figure, wherein marks the zone that has greater than 3.0 MAE ratio;

Fig. 9 illustrates for the sub-piece that has greater than 3.0 MAE ratio and carries out the arrowband classification, and described sub-piece concentrates on certain direction;

Figure 10 is for describing the flow chart according to frame interpolation of the present invention;

Figure 11 is the figure that explains the correlation between current block and the adjacent block;

The block diagram that Figure 12 puts for frame interpolation according to the present invention;

Figure 13 illustrates and is used for the adjacent block that correlation is checked;

Figure 14 A to 14D illustrates the test pattern with repeat patterns;

Figure 15 illustrates the image of handling by traditional MC, and the image that wherein has repeat patterns is disperseed;

Figure 16 illustrates the image of handling by according to the method for interpolation frame of the present invention, and wherein false picture is suppressed visibly.

Embodiment

Describe the present invention in detail referring now to accompanying drawing, wherein accompanying drawing shows the preferred embodiments of the present invention.

The block matching method that is used for estimation can be categorized as forward block matching process, back to block matching method and symmetry blocks matching process.Because the track of motion is very important to converting frame rate, thus generally use the symmetry blocks matching process to be used for converting frame rate, as shown in Figure 1.

Fig. 1 is the schematic diagram of symmetry blocks matching process.In Fig. 1, in the promptly symmetrical matching process, obtain each error of calibrated bolck with the reference block in the motion vector sought zone (M+2P) * (N+2P) that is positioned at reference frame of present frame.In the symmetry blocks matching process, this reference frame refers to former frame or back one frame.

Referring to Fig. 1, frame k-1 represents former frame, and frame k+1 represents back one frame, and frame K represents the present frame that will be interpolated.The calibrated bolck of present frame is by label 102 expressions.The motion vector sought zone of former frame is by label 110 expressions, and then the motion vector sought zone of a frame is by label 120 expressions.The area of tentative standard piece 102 equals M * N, and then the motion vector sought zone of calibrated bolck 102 will equal (M+2P) * (N+2P), wherein P represent from calibrated bolck to+/-x and+/-number of pixels that the y direction extends out.

Usually, the area of calibrated bolck 102 equals 8 * 8 pixels (comprising 64 pixels altogether), and the motion vector sought zone of calibrated bolck 102+/-the x direction accounts for 4 pixels ,+/-the y direction accounts for 4 pixels, promptly equals 16 * 16 pixels (comprising 256 pixels altogether).

Here, with classification for search for fully the search or diamond search.Search fully commonly used.

In search fully, be that unit moves and has and calibrated bolck reference block of the same area with a pixel, and compare with calibrated bolck.Thus, have 256 reference blocks.

The symmetry blocks matching process uses absolute difference sum (SAD) to obtain each error of the reference block in the motion vector sought zone of reference block in the motion vector sought zone of former frame and back one frame.

According to the symmetry blocks matching process, mutual benchmark piece 112 and reference block 122, the upper left side in the motion vector sought zone 120 of reference block 112 in frame k-1, and the lower right side in the motion vector sought zone 120 of reference block 122 in frame k+1.Equally, mutually benchmark piece 114 and reference block 124, reference block 114 be than reference block 112 pixel to the right, reference block 124 than reference block 122 to the first from left pixel.

Obtain each SAD by mutual more symmetrical reference block, and the result be stored in (M+P) * (N+P) SAD figure in.

The block diagram that Fig. 2 puts for the frame interpolation that uses conventional motion estimation and motion compensation (ME/MC).

SAD computing unit 202 uses symmetric motion to amount estimation method, the SAD between the symmetrical reference block in the motion vector sought zone 110 of calculating frame k-1 and the motion vector sought zone 120 of frame k+1.

$\mathrm{SAD}=\underset{i=0}{\overset{M-1}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{N-1}{\mathrm{\&Sigma;}}}\left(\right|f_2(i,j)-f_1(i,j)\left|\right)---\left(1\right),$

Wherein, M and N represent the wide and high of calibrated bolck, and f ₁(i, j) and f ₂(i, j) location of pixels in the expression former frame and the location of pixels in one frame of back.

Because the number of the reference block in each motion vector sought zone equals 256, so the number of the SAD that obtains equals 256.

This SAD figure of SAD figure memory cell 204 storages.576 SAD that this SAD figure storage is tried to achieve by SAD computing unit 202.

Motion vector extraction unit 206 is determined to have in the SAD figure memory cell 204 among each SAD of storage, and (x, y), and (x is y) as the direction of image motion in the current block to discern this position in the position of the reference block of the minimum SAD among the candidate that promptly moves.

Motion filter 208 is used to consider the correlation between current block and the adjacent block, and is used for preventing the motion vector mistake.Usually, use median filter or average filter as motion filter 208.Frame interpolation unit 210 uses the final motion vector interpolation present frame of being handled by motion filter 208, and wherein, this final motion vector is the mean value of the motion vector of former frame and back one frame.

As mentioned above, referring to Fig. 1 and Fig. 2, employed conventional motion estimation is identified as optimal motion vectors with the motion vector that has minimum SAD in the motion vector sought zone in converting frame rate.

Yet, even this optimal motion vectors can comprise the error of relevant motion total amount or direction.So, the correlation between motion filter 208 checking current blocks and the adjacent block.

Can use median filter or average filter as motion filter 208.Median filter is by the sequence arrangement current block from the minimum to the maximum and the motion vector of adjacent block, and the median in the motion vector of extraction arrangement.The mean value of all motion vectors of mean value filter calculations.

The final motion vector of being handled by motion filter 208 is considered to have the best information of current block, and comprised the motion vector of former frame and back one frame mean value (dx, dy).

Following equation 2 these frame interpolation methods of expression.

$f_k=\frac{f_{(k-1)}(i-\mathrm{dx},i-\mathrm{dy})+f_{(k+1)}(i+\mathrm{dx},i+\mathrm{dy})}{2}---\left(2\right),$

Wherein, f _k(i, j) location of pixels of the frame of expression interpolation.

The motion vector that use obtains by conventional motion estimation, conventional frame interpolating method have shown enough performances when supporting generally mobile or rest image.But, when the image with repeat patterns when interframe moves, be difficult to use the conventional motion estimation algorithm to obtain motion vector accurately.

For example, when having the image that periodically repeats pattern when moving, for example striped shirt, striped tablecloth or the building of a string window is arranged are difficult to accurate estimating motion vector.This is because the correlation of interframe changes significantly.

Briefly, use conventional motion estimation to be difficult to accurately estimate to have the movable information that periodically repeats the image of pattern, this has just caused the motion artifacts (artifact) of scattering as image.

Therefore, the present invention proposes a kind of algorithm, it allows to determine the pattern of repetition.The invention allows for a kind of frame interpolation method, by this method, be contained in piece in the repeat patterns, and use motion vector to handle other pieces by traditional ME/MC by the linear interpolation pack processing.In addition, the invention allows for a kind of frame interpolation of this frame interpolation method that uses puts.

Fig. 3 is for determining the flow chart of the method for repeat patterns according to the present invention.

This algorithm according to definite repeat patterns of the present invention carried out with three steps.

Change this fact largely based on the correlation between current block and the adjacent block, can make following two hypothesis.

The first, in repeat patterns, the correlation marked change between the mean absolute error among the MAE figure (MAE).Compare with normal situation mobile or rest image, the left side and the deviation between the MAE in the right diagonal that are included in the piece in the repeat patterns are bigger.

The second, the marked change of the correlation between MAE evenly distributes on the whole motion vector sought zone of this piece.

Under these hypothesis, determine that the algorithm of repeat patterns carried out with three steps.In the following description, the MAE map distance is represented the deviation between the MAE, and each piece error is represented MAE or SAD.

The MAE of the mean value of the difference sum between the remarked pixel can obtain with following formula 3

$\mathrm{MAE}=\frac{1}{\mathrm{MN}}\underset{i=0}{\overset{M-1}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{N-1}{\mathrm{\&Sigma;}}}\left(\right|f_2(i,j)-f_1(i,j)\left|\right)---\left(3\right),$

Wherein, M and N represent the wide and high of piece, and f ₁And f ₂Location of pixels in location of pixels in the expression former frame and back one frame.

In the step of determining the MAE map distance, according to first hypothesis, from the regional MAE figure (step S302) that obtains of the motion vector sought of former frame and back one frame.Then, each motion vector sought zone is divided into sub-piece (step S304).Obtain maximum MAE and minimum MAE from each sub-piece, and calculate the ratio of maximum MAE and minimum MAE, promptly MAE is than (Max/Min) (step S306).

Compare with proper motion or rest image, the correlation that has between image each MAE in MAE figure of repeat patterns is changed to bigger degree.By the left side in searching moving vector search zone and the motion in the right diagonal, can discern the motion of repeat patterns.

Can use following method to measure the degree that the correlation between each MAE changes among the MAE figure.

According to the order of reference block, be form arrangement (M+P) * (N+P) individual MAE of the MAE figure of (M+P) * (N+P) with size.In this MAE figure, try to achieve and accumulate the deviation between the adjacent MAE in left diagonal and the right diagonal.More left cornerwise accumulated value and right cornerwise accumulated value are obtained higher value, and this higher value and threshold values 1 are compared mutually.

Then MAE figure is divided into sub-piece.Obtain the MAE ratio from each sub-piece.Add up to the sub-piece number that has greater than the MAE ratio of threshold values 2.This count value and threshold values 3 are compared.

In this embodiment of the present invention, the motion vector sought zone+/-x and+/-the y direction on scope from-8～+ 7.The number of MAE among the MAE figure equals 256, comprises 16 MAE on each horizontal line and 16 MAE on each vertical line.The number of sub-piece equals 16, comprises 4 sub-pieces on each horizontal line and 4 sub-pieces on each vertical line.

Obtain and accumulate difference (step S308) between the adjacent MAE in left diagonal and the right diagonal.This accumulated value is represented total MAE map distance (step S310).

Obtain after total MAE map distance, the total MAE map distance of maximum is set to the MAE map distance of current block.If should the total MAE map distance of maximum greater than threshold values 1, then this handles according to second hypothesis and carries out (step S312).

Fig. 4 illustrates for the piece that is included in the repeat patterns, the figure that is formed by the MAE that presents in the left side of MAE figure and the right diagonal.As shown in Figure 4, the correlation marked change between MAE.

Fig. 5 illustrates the MAE figure to general pattern.As shown in Figure 5, the correlation between the MAE in the diagonal of the left and right sides does not have marked change.

In second step of the algorithm of determining repeat patterns, promptly in MAE figure classification, abandon pseudo-repeat patterns.This puppet repeat patterns is not real repeat patterns, but comprises and the similar characteristic of repeat patterns.If the distribution that has greater than the zone of 3.0 MAE ratio is twisted, promptly concentrate in any direction to present these zones, then really stator block is comprised in the pseudo-repeat patterns.

Have greater than threshold values 2 as the fruit piece, promptly be 3.0 MAE ratio in the present invention, and occupied more than 50% of whole sub-pieces, then really stator block is comprised in the repeat patterns, and this process proceeds to the distribution inspection of step 314.

Shown in equation 4, the zone of being represented by " 1 " is called as most zones, and other zones are called as a few regions.Each number that adds up to most zones and a few regions.

$\mathrm{majority}\_\mathrm{count}=\underset{i\&Element;\mathrm{majority}\_\mathrm{count}}{\mathrm{\&Sigma;f}\left(i\right)}---\left(4\right),$

$\min \mathrm{ority}\_\mathrm{count}=\underset{i\&Element;\min \mathrm{ority}\_\mathrm{count}}{\mathrm{\&Sigma;f}\left(i\right)}$

Wherein f (i) represents threshold values 2.

In step 314, count value is used to equation 5 to determine pseudo-repeat patterns.

(∧ of minority_count 〉=α * region_count) (minority_count≤β * (16-region_count) ... (5), wherein, region_count represents the number of the sub-piece in most zones, is set to " 8 " in the present invention.Factor alpha is set to 0.5, and factor beta is set to 0.25.

Fig. 6 illustrates for the MAE figure that is included in the piece in the repeat patterns, and wherein, each MAE figure is divided into 16 sub-pieces, and marks the zone that has greater than 3.0 MAE ratio with stain.Referring to Fig. 6, the sub-piece that has greater than the MAE ratio of threshold value 3 (promptly 7.0) is evenly distributed on the MAE figure.According to experiment, because the MAE of general pattern ratio is less than 2.0, so the image that threshold values 2 allows clearly to distinguish general pattern and has repeat patterns.

In step 316, check the distribution of MAE ratio, and in step 318, determine that the piece with arrowband is included in the pseudo-repeat patterns.

Fig. 7 illustrates the image with pseudo-repeat patterns.In Fig. 7, the part that is marked by circle is pseudo-repeat patterns.As shown in Figure 8, has regional centralized greater than 3.0 MAE ratio in diagonal.This result does not conform to the hypothesis of the characteristics of relevant image with repeat patterns.Therefore, must remove pseudo-repeat patterns.

Fig. 9 illustrates for the sub-piece that has greater than 3.0 MAE ratio and carries out the arrowband classification, and this a little concentrates on certain direction.Have zone greater than 3.0 MAE ratio by " 1 " expression, and mark, and other zones are represented by " 0 " with stain.

Determining of repeat patterns with improved median filter end.In improved median filter, not that (dependent) of subordinate but this fact of concentrating are checked the repetition of adjacent block, thereby only just approve the repetition of current block when adjacent block when being shown repetition greater than predeterminated level based on repeat patterns.In other words, be aggregated in 24 adjacent blocks and be included in 5 pieces and 5 pieces on each vertical line on each horizontal line, be included in the number of the piece in the repeat patterns.Adding up to greater than threshold values 4 if be included in the number of the piece in the repeat patterns, promptly is 6 in the present invention, determines that then current block is included in the repeat patterns.

In the present invention, can use the MAE ratio of adjacent block, rather than the MAE of sub-piece ratio.In other words, obtain the MAE ratio of adjacent block in the diagonal, add up to number, and this count value and threshold values 2 are compared greater than the MAE ratio of threshold values 4.

Used the symmetry blocks coupling in the present invention, but also may use additive method, mated to piece as forward block matching process or back.

In addition, the method for definite repeat patterns of the present invention can be used in motion vector estimation and the frame interpolation.

Figure 10 is for describing the flow chart according to frame interpolation method of the present invention.

At first step S1002, obtain MAE figure.

At next step S1004, based on this MAE figure estimating motion vector.

Then, at step S1006, determine whether the sub-piece of MAE figure is included in the repeat patterns.The flow chart of Fig. 3 has been described determining of repeat patterns.

In step S1008, check the correlation between current block and the adjacent block, with the ratio that obtains to be included in the piece in the repeat patterns and not have other pieces of repeat patterns, i.e. correlation between current block and the adjacent block.

Figure 11 has explained the correlation between current block and the adjacent block.As shown in figure 11, place piece 1102 between repeat patterns zone and the non-repeat patterns zone to be subjected to the two the influence of repeat patterns and non-repeat patterns.Therefore, the piece that must generate by linear interpolation and mix by the piece that ME/MC generates is with inserted block 1102 in correct.

In step S1110, mix piece that generates by linear interpolation and the piece that generates by ME/MC by ratio, and obtain the image after the interpolation based on acquisition.In other words, carry out the frame interpolation by soft handover.

The block diagram that Figure 12 puts for frame interpolation according to the present invention.

MAE computing unit 1202 uses the symmetric motion vector to estimate, the MAE between the reference block in the region of search of calculating former frame k-1 and back one frame k+1.

Because the number of the reference block in each motion vector sought zone is 256, so the number of the MAE that obtains is 256.

MAE figure memory cell 1204 storages one MAE figure.This MAE figure comprises 256 MAE that obtained by MAE computing unit 1202.

In the MAE that is stored in MAE figure memory cell 1204, among the candidate that promptly moves, the moving direction of motion vector extraction unit 1206 identification current blocks be position with reference block of minimum MAE (x, y).

Motion filter 1208 is used to consider the correlation between current block and the adjacent block, and is used for preventing the motion vector mistake.General median filter or the mean value filter of using is as motion filter 1208.

With reference to the MAE figure that is stored in the MAE figure memory cell 1204, repeat patterns determining unit 1210 determines whether piece is included in the repeat patterns.Repeat patterns determining unit 1210 comprises MAE map distance computing unit 1210a, MAE figure taxon 1210b and improved median filter 1210c.

MAE map distance computing unit 1210a calculates the difference between each MAE of the adjacent block that centers on current block on a left side and the right diagonal, and with result's accumulation, to calculate the distance between each MAE in a left side and right diagonal.

Ultimate range in institute's calculated distance is set to the MAE map distance, and compares with threshold values 1.If the MAE map distance is greater than threshold values 1, then MAE map distance computing unit 1210a determines to satisfy first hypothesis.

MAE map distance computing unit 1210a is divided into sub-piece with MAE figure then, and calculates minimum MAE and maximum MAE in each sub-piece, to obtain MAE than (Max/Min).

In the present invention, the motion vector sought zone+/-x and+/-the y direction is from-8-+7 expansion.The sum of MAE in MAE figure equals 256, is included in 16 MAE on each horizontal line and 16 MAE on each vertical line.MAE figure is divided into 16 sub-pieces, is included in 4 pieces on each horizontal line and 4 pieces on each vertical line.

MAE figure taxon 1210b removes pseudo-repeat patterns.Have greater than threshold values 2 as the fruit piece, promptly of the present invention 3.0 MAE ratio, and accounted for more than 50% of all sub-pieces, and then MAE figure taxon 1210b determines that this a little is included in the pseudo-repeat patterns, wherein threshold values 3 is set to 7.

MAE figure taxon 1210b checks the distribution have greater than the sub-piece of the MAE ratio of threshold values 2 then.Have the arrowband as the fruit piece, determine that then it is included in the pseudo-repeat patterns.

Median filter 1210c checks the repetition of adjacent block, only just approves the repetition of current block during greater than the repetition of predeterminated level when the adjacent block demonstration.In other words, at 24 adjacent blocks and be included in 5 pieces and 5 pieces on each vertical line on each horizontal line, add up to the number of piece with repeat patterns.Adding up to greater than threshold values 4 if be included in the number of the piece in the repeat patterns, promptly is 6 in the present invention, determines that then current block is included in the repeat patterns.

In the present invention, can use the MAE ratio of adjacent block, rather than the MAE of sub-piece ratio.In other words, obtain the MAE ratio of adjacent block in a left side and the right diagonal, add up to number, and this count value and threshold values 2 are compared greater than the MAE ratio of threshold values 4.

Frame interpolation unit 1212 uses the final motion vector of being handled by motion filter 1209 and uses linear interpolation that frame is carried out interpolation.

The adjacent block that frame interpolation unit 1212 is checked around current block is with the ratio that obtains to be included in the piece in the repeat patterns and not have other pieces of repeat patterns, i.e. correlation between current block and the adjacent block.Frame interpolation unit 1212 is by mixing piece that generates by linear interpolation and the piece that generates by ME/MC, the image after the acquisition interpolation then.

For example, by adding up to, check the correlation between current block and the adjacent block around the repetition of the adjacent block of current block.

Figure 13 illustrates and is used for the adjacent block that correlation is checked.In Figure 13,25 adjacent blocks are around current block.Add up to the number that is included in the adjacent block in the repeat patterns.

Frame interpolation based on the correlation that obtains is represented by equation 6.

$f(i,j)=\frac{(\mathrm{\&alpha;}\&times;\mathrm{linear}(i,j)+(\mathrm{total}-\mathrm{\&alpha;})\&times;\mathrm{MC}(i,j))}{\mathrm{total}}---\left(6\right),$

Wherein, (α represents to be included in the number of the adjacent block in the repeat patterns to f for i, the j) location of pixels of expression current block, and scope is 0≤α≤25 in the present invention.

According to the present invention, the image with repeat patterns is handled by linear interpolation, and does not have another image of repeat patterns to handle by ME/MC.

Yet simple hardware switches and can't show superior performance at the boundary between the piece in being included in repeat patterns zone and non-repeat patterns zone.Like this, based on the correlation between current block and the adjacent block, use soft handover.

In equation 6, α represents to be included in the number of the adjacent block in the repeat patterns.If α is very big, that is,, then use linear interpolation if image has many repeat patterns.Otherwise, then be used to mean value from the motion vector of former frame and back one frame, use interpolation based on MC.

Figure 14 A to 14D illustrates the test pattern with repeat patterns.Figure 14 A is illustrated in the grating image that the center has the repetition sliver.Figure 14 B illustrates has the radially Snell Wilcox of repeat patterns.Figure 14 C illustrates a Melco image, presents the building of the window with repetition.Figure 14 D illustrates a restaurant image, presents the tablecloth with repetition striped.

In order to compare the present invention and traditional MC, equation 7 and 8 defined mean square deviations (MSE, mean square error) or Y-PSNR (PSNR, peak signal to noise) have been used.

$\mathrm{MSE}=\underset{i=0}{\overset{M-1}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{N-1}{\mathrm{\&Sigma;}}}\left(\right|f_2(i,j)-f_1(i,j){|}^2)---\left(7\right),$

$\mathrm{PSNR}=20{\log }_{10}\left(\frac{255}{\sqrt{\mathrm{MSE}}}\right)---\left(8\right),$

Wherein, M and N represent the wide and high of piece, and f ₁(i, j) and f ₂(i, j) location of pixels of the location of pixels of expression former frame and back one frame.

Table 1 shows the PSNR that use equation 7 and 8 acquisitions.The PSNR that obtains is the mean value from the PSNR of frame.

[table 1]

Image	Tradition MC	The algorithm that proposes	Improve
				Grid	?23.6db	?26.2db	?2.6db
Snell?Wilcox	?15.8db	?24.9db	?9.1db
				Melco	?26.1db	?26.5db	?0.4db
The restaurant	?28.9db	?30.1db	?1.2db

Shown in top table 1, PSNR changes with the degree that the form of repeat patterns, motion total amount between the frame and repeat patterns occupy image.

Below, the vision difference that is obtained when using according to frame interpolation method of the present invention and using traditional MC is described with reference to Figure 15 and 16.

Figure 15 illustrates the image of handling by traditional MC, and the image that wherein has repeat patterns is disperseed.

Figure 16 illustrates the image of handling by according to the method for interpolation frame of the present invention, and wherein motion artifacts is suppressed visibly.

Result as a comparison, frame interpolation method according to the present invention has been eliminated the shown serious vacation picture of traditional MC.

As mentioned above, when the image with repeat patterns moves, allow effective frame interpolation according to the method for definite repeat patterns of the present invention.

When the image with repeat patterns moves, allow to use linear interpolation and traditional MC according to frame interpolation method of the present invention, thereby the image after the interpolation that does not have motion artifacts is provided.

In addition, when the image with repeat patterns moved, frame interpolation according to the present invention was put frame that mixing handles by linear interpolation and the frame by traditional MC use motion compensation process, and the image after the interpolation that does not have motion artifacts is provided thus.

Though concrete demonstration of the present invention and description it should be appreciated by those skilled in the art under the prerequisite of scope that does not break away from claim and spirit with reference to its exemplary embodiment, can make the various modifications on form and the details.

Claims (26)

1. method is used for determining whether the reference block of M * N is included in the repeat patterns when the image with repeat patterns moves, and this method may further comprise the steps:

Use search fully, obtain the error of calibrated bolck and reference block, each reference block is included in the region of search of (M+2P) * (N+2P), and this region of search belongs to the frame of institute's reference in the motion vector sought process;

According to the order of reference block, arrange (M+P) * (N+P) individual error with the form of (M+P) * (N+P) figure;

In the left diagonal and right diagonal of this figure, obtain each deviation between current block and the adjacent block, and be accumulated in deviation that obtains in the left diagonal and the deviation that in right diagonal, obtains respectively;

Compare deviation of in left diagonal, accumulating and the deviation of in right diagonal, accumulating, and select bigger deviation;

Deviation and the threshold values 1 selected are compared; And

If the deviation of selecting, determines then that this reference block is included in the repeat patterns greater than threshold values 1.

2. the method for claim 1, wherein the error of calibrated bolck and reference block is called as mean absolute error (MAE), because it is to obtain by the absolute difference sum between average each reference block.

3. the method for claim 1 also comprises:

The sub-piece that this figure is divided into identical size;

Calculate the ratio of the worst error and the minimal error of each sub-piece;

Counting has the sum greater than the sub-piece of the ratio of threshold values 2; And

If should sum greater than threshold values 3, then really stator block is included in the repeat patterns.

4. method as claimed in claim 3 also comprises:

Inspection has the distribution greater than the sub-piece of the ratio of threshold values 2, and determines whether this sub-piece is included in the pseudo-repeat patterns.

5. method as claimed in claim 4 wherein, by relatively concentrate the pseudo-repeat patterns of the sub-piece that distributes on the level of this figure, vertical and diagonal, determines whether described sub-piece is included in the described pseudo-repeat patterns.

6. method as claimed in claim 5 also comprises:

Check the repetition degree that is determined the piece that is included in the repeat patterns in reference block and the adjacent block; And

Based on the degree that repeats, determine that reference block is included in the described repeat patterns.

7. method as claimed in claim 6, wherein, the degree of repetition is represented by the sum that is confirmed as being included in the piece in the repeat patterns in reference block and the adjacent block.

8. frame interpolation method may further comprise the steps:

(a) use search fully, obtain the error of calibrated bolck and reference block, each reference block is included in the region of search of (M+2P) * (N+2P), and this region of search belongs to the frame of institute's reference in the motion vector sought process;

(b) estimating motion vector is the positional information with reference block of minimal error;

(c), determine whether described calibrated bolck and reference block are included in the repeat patterns according to the error that is obtained;

(d) calculate the current block that will be interpolated and around the correlation between the adjacent block of this current block; And

(e),, obtain the image after the interpolation by mixing piece that forms by linear interpolation and the piece that forms by estimation and motion compensation (ME/MC) according to the correlation of calculating.

9. frame interpolation method as claimed in claim 8, wherein, step (d) further comprises:

Counting centers on described reference block and is comprised in the number of all adjacent blocks in the repeat patterns; And

Calculate around the ratio of the number of the sum of the adjacent block of reference block and counting.

10. frame interpolation method as claimed in claim 9, wherein, the current block that forms by interpolation is expressed from the next,

$<\mathrm{PSTYLELSPACE}=130>f(i,j)=\frac{\mathrm{\&alpha;}\&times;\mathrm{linear}(i,j)+(\mathrm{total}-\mathrm{\&alpha;})\&times;\mathrm{MC}(i,j)}{\mathrm{total}},$

Wherein, (α represents to be included in the number of the adjacent block in the repeat patterns to f for i, the j) location of pixels of expression current block, and total represents the sum of adjacent block.

11. frame interpolation method as claimed in claim 8, wherein, step (c) further comprises:

When use is searched for fully, obtain the error of calibrated bolck and reference block, each reference block is included in the region of search of (M+2P) * (N+2P), and this region of search belongs to the frame of institute's reference in the motion vector sought process;

According to the order of reference block, arrange (M+P) * (N+P) individual error with the form of (M+P) * (N+P) figure;

In the left diagonal and right diagonal of this figure, obtain current block and around each deviation between the adjacent block of this current block, and be accumulated in deviation that obtains in the left diagonal and the deviation that in right diagonal, obtains respectively;

Compare deviation of in left diagonal, accumulating and the deviation of in right diagonal, accumulating, and choose bigger deviation;

Selected deviation and threshold values 1 are compared; And

If the deviation of selecting, determines then that this reference block is included in the repeat patterns greater than threshold values 1.

12. frame interpolation method as claimed in claim 11, wherein, the error of calibrated bolck and reference block is called as mean absolute error (MAE), because it is to try to achieve by the absolute difference sum between average each reference block.

13. frame interpolation method as claimed in claim 11 also comprises:

The sub-piece that this figure is divided into identical size;

Calculate the ratio of the worst error and the minimal error of each sub-piece;

Counting has the sum greater than the sub-piece of the ratio of threshold values 2; And

If should sum greater than threshold values 3, determine that then described sub-piece is included in the described repeat patterns.

14. frame interpolation method as claimed in claim 13 also comprises:

Inspection has the distribution greater than the sub-piece of the ratio of threshold values 2, and determines whether this sub-piece is included in the pseudo-repeat patterns.

15. frame interpolation method as claimed in claim 14 wherein, by relatively concentrate the pseudo-repeat patterns of the sub-piece that distributes on the level of described figure, vertical and diagonal, determines whether this sub-piece is included in this puppet repeat patterns.

16. frame interpolation method as claimed in claim 15 also comprises:

Inspection is confirmed as being included in the repetition degree of the piece in the repeat patterns in reference block and adjacent block; And

Based on the degree that repeats, determine that reference block is included in this repeat patterns.

17. frame interpolation method as claimed in claim 16, wherein, the degree of repetition is represented by the sum that is confirmed as being included in the piece in the repeat patterns in reference block and adjacent block.

18. a frame interpolation is put, and comprising:

Mean absolute error (MAE) computing unit, be used for using search fully, MAE between basis of calculation piece and the reference block, each reference block are comprised in the region of search of (M+2P) * (N+2P), and this region of search belongs to the frame of institute's reference in the motion vector sought process;

MAE figure memory cell is used for the order according to reference block, with form storage (M+P) * (N+P) individual error of (M+P) * (N+P) figure;

Motion vector extraction unit, the image moving direction that is used for discerning current block be position with the reference block of the minimum MAE among each MAE that is stored in this MAE figure (x, y);

The repeat patterns determining unit is used for determining with reference to the MAE figure that is stored in MAE figure memory cell whether piece is included in the repeat patterns; And

The frame interpolation unit is used to use the motion vector of extraction and linear interpolation to come interpolation frame.

19. frame interpolation as claimed in claim 18 is put, wherein, described repeat patterns determining unit also comprises:

MAE map distance computing unit is used for described MAE figure is divided into sub-piece, and calculates minimum MAE and maximum MAE in each sub-piece, to obtain MAE than (maximum/minimum);

Difference between the MAE of calculating adjacent block in left diagonal and right diagonal;

The difference that accumulation is calculated is to calculate the distance between each MAE in the diagonal of the left and right sides;

From each distance of calculating, choose maximum distance; And

If this ultimate range, determines then that reference block is included in this repeat patterns greater than threshold values 1.

20. frame interpolation as claimed in claim 19 is put, wherein, described repeat patterns determining unit also comprises:

MAE figure taxon is used for counting left and right sides diagonal and has sum greater than the sub-piece of the MAE ratio of threshold values 2; And have MAE ratio greater than threshold values 2 as the fruit piece, and accounted for more than 50% of whole sub-pieces, determine that then this sub-piece is included in the repeat patterns.

21. frame interpolation as claimed in claim 20 is put, wherein, the inspection of described MAE figure taxon has the distribution greater than the sub-piece of the MAE ratio of threshold values 2.

22. frame interpolation as claimed in claim 21 is put, wherein, described MAE figure taxon determines by relatively concentrate the pseudo-repeat patterns of the sub-piece that distributes on the level of this MAE figure, vertical and diagonal whether this sub-piece is included in this puppet repeat patterns.

23. frame interpolation as claimed in claim 20 is put, wherein, described repeat patterns determining unit also comprises:

Median filter is used for checking the reference block that is included in repeat patterns and the repetition of adjacent block, and based on the degree that repeats, determines that reference block is included in this repeat patterns.

24. frame interpolation as claimed in claim 18 is put, wherein, described frame interpolation unit calculates the current block that will be interpolated and around the correlation between the adjacent block of this current block, and according to the correlation of calculating, by mixing the piece that forms by linear interpolation and the piece by motion compensation (MC) formation, obtain the image after the interpolation.

25. frame interpolation as claimed in claim 24 is put, wherein, described frame interpolation unit counting is included in this repeat patterns and centers on all numbers around adjacent block of described calibrated bolck, calculate around the ratio of the number of the sum of the adjacent block of reference block and counting, and the current block that uses this ratio to calculate will to be interpolated and around the correlation between the adjacent block of this current block.

26. frame interpolation as claimed in claim 25 is put, wherein, the current block that forms by interpolation is expressed from the next,

$<\mathrm{PSTYLELSPACE}=130>f(i,j)=\frac{\mathrm{\&alpha;}\&times;\mathrm{linear}(i,j)+(\mathrm{total}-\mathrm{\&alpha;})\&times;\mathrm{Mc}(i,j)}{\mathrm{total}},$

Wherein, (α represents to be included in the number of the adjacent block in the repeat patterns to f for i, the j) location of pixels of expression current block, and total represents the sum of adjacent block.

Images