CN116091321A

CN116091321A - Image scaling method, device, equipment and storage medium

Info

Publication number: CN116091321A
Application number: CN202310377957.7A
Authority: CN
Inventors: 孙旭; 周玉龙; 刘刚; 李拓
Original assignee: Suzhou Inspur Intelligent Technology Co Ltd
Current assignee: Suzhou Inspur Intelligent Technology Co Ltd
Priority date: 2023-04-11
Filing date: 2023-04-11
Publication date: 2023-05-09
Anticipated expiration: 2043-04-11
Also published as: CN116091321B

Abstract

The invention provides an image scaling method, an image scaling device and a storage medium, and relates to the field of image processing, wherein the image scaling method comprises the following steps: dividing a source image into a plurality of image blocks with the same size, and determining the category of the image blocks based on the brightness components of all pixel points in the image blocks; determining a target interpolation method based on a first resolution and a first frame rate of the source image, a second resolution and a second frame rate of the target image, and an image scaling performance of an image scaling module that processes the source image; under the condition that the target interpolation method comprises at least two interpolation methods, an image scaling strategy is determined based on the maximum pixel number matched by the target interpolation method and the total pixel number in the image block of each category, and image scaling processing is carried out on the source image based on the image scaling strategy, so that the invention can realize image scaling with higher resolution and frame rate under the condition of fixed image scaling performance.

Description

Image scaling method, device, equipment and storage medium

Technical Field

The present invention relates to the field of image processing, and in particular, to an image scaling method, apparatus, device, and storage medium.

Background

Image scaling (image size) is an image compression part in image processing technology, and refers to the resizing of an image. Image scaling is a process of reconstructing pixel values of a source image according to the actual size requirement to output a new image after sampling the pixel values.

As a key in image processing technology, image scaling technology plays a vital role in acquiring more information. At present, when image scaling is performed, considering the total cost and performance of a chip, the overall image scaling performance of an image scaling module for processing image scaling in the chip is fixed, and the more complex the calculation process of an interpolation method is, the higher the image scaling performance of hardware required by the interpolation method is, and the better the display effect after image scaling is.

At present, the maximum resolution and the maximum frame rate of image scaling are generally limited, and a fixed interpolation method is adopted for processing, so that the basic function of hardware is ensured to be normal, but the method can limit the scaling processing capacity and the display effect, and the image scaling performance of the hardware can not be fully utilized, so that how to realize the image scaling with higher resolution and frame rate under the fixed image scaling performance is a practical problem which needs to be solved at present.

Disclosure of Invention

The invention provides an image scaling method, an image scaling device and a storage medium, which are used for solving the technical problem of how to realize image scaling with higher resolution and frame rate under the fixed image scaling performance.

In a first aspect, the present invention provides an image scaling method, including:

dividing a source image into a plurality of image blocks with the same size, and determining the category of the image block based on the brightness components of each pixel point in the image block;

determining a target interpolation method based on the first resolution and the first frame rate of the source image, the second resolution and the second frame rate of a target image and the image scaling performance of an image scaling module for processing the source image, wherein the target interpolation method is an interpolation method required to be used for scaling the source image to the target image, and the image scaling performance is the number of pixel points capable of completing scaling processing in unit time;

and under the condition that the target interpolation method comprises at least two interpolation methods, determining an image scaling strategy based on the maximum pixel point matched by the target interpolation method and the total pixel point in the image block of each category, and performing image scaling processing on the source image based on the image scaling strategy, wherein the image scaling strategy is the strategy of the interpolation method used by each pixel point in the source image.

According to the image scaling method provided by the invention, the determining a target interpolation method based on the first resolution and the first frame rate of the source image, the second resolution and the second frame rate of the target image, and the image scaling performance of the image scaling module for processing the source image comprises the following steps:

determining an interpolation factor for processing the source image matching based on the image scaling performance of an image scaling module for processing the source image, wherein the interpolation factor is a clock period required for processing one pixel point;

determining processing capacity based on a first resolution, a first frame rate and the interpolation factor of the source image, wherein the processing capacity is a clock period which can be completed every second when the source image is processed;

determining interpolation parameters based on a second resolution, a second frame rate, and the processing capability of the target image, and determining a target interpolation based on the interpolation parameters.

According to the image scaling method provided by the invention, the method for determining the target interpolation method based on the interpolation parameters comprises the following steps:

determining a target interpolation method as a first preset interpolation method corresponding to the minimum preset interpolation parameter when the interpolation parameter is smaller than or equal to the minimum preset interpolation parameter;

Determining a target interpolation method as a second preset interpolation method corresponding to the maximum preset interpolation parameter when the interpolation parameter is greater than or equal to the maximum preset interpolation parameter;

and determining a target interpolation method based on the interpolation parameter under the condition that the interpolation parameter is larger than the minimum preset interpolation parameter but smaller than the maximum preset interpolation parameter, wherein the target interpolation method comprises at least two interpolation methods.

According to the image scaling method provided by the invention, the interpolation parameters are obtained by the following formula:

wherein ,

for interpolation parameters +.>

For processing capacity, +.>

For the second resolution, ++>

At a second frame rate.

According to the image scaling method provided by the invention, the categories comprise a first category used for representing that the image block is in an image quasi-static region of the source image, a second category used for representing that the image block is in an image gentle change region of the source image and a third category used for representing that the image block is in an image sharp change region of the source image;

the variance of the brightness component of the image block of the image sharp change area is larger than the variance of the brightness component of the image block of the image gradual change area, and the variance of the brightness component of the image block of the image gradual change area is larger than the variance of the brightness component of the image block of the image quasi-static area.

According to the image scaling method provided by the invention, the determining the category of the image block based on the brightness component of each pixel point in the image block comprises the following steps:

determining a variance of the luminance component of the image block based on the luminance component of each pixel point in the image block;

determining a smoothness approximation value of the image block based on the variance of the brightness component of the image block and the variance of the brightness component of the adjacent image block corresponding to the image block;

a category of the image block is determined based on the smoothness approximation of the image block.

According to the image scaling method provided by the invention, the determining the category of the image block based on the smoothness approximation value of the image block comprises the following steps:

determining the category of the image block as the first category under the condition that the smoothness approximation value of the image block is smaller than or equal to a lowest preset threshold value;

determining the category of the image block as the second category when the smoothness approximation value of the image block is larger than the lowest preset threshold value but smaller than or equal to the highest preset threshold value;

and determining the category of the image block as the third category under the condition that the smoothness approximation value of the image block is larger than the highest preset threshold value.

According to the image scaling method provided by the invention, the smoothness approximation value of the image block is obtained by carrying out weighted summation on the variance of the brightness component of the image block and the variance of the brightness component of the adjacent image block corresponding to the image block; the difference between the weight value of the variance of the luminance component of the image block and the weight value of the variance of the luminance component of the neighboring image block increases as the number of total pixels in the image block increases.

According to the image scaling method provided by the invention, the determining the smoothness approximation value of the image block based on the variance of the brightness component of the image block and the variance of the brightness component of the adjacent image block corresponding to the image block further comprises:

and determining that the smoothness approximation of the image block is equal to the variance of the brightness component of the image block when the image block is a boundary image block, wherein the boundary image block comprises pixel points on the boundary of the source image.

The image scaling method provided by the invention further comprises the following steps:

and determining a smoothness approximation value of the image block based on the variance of the brightness component of the image block and the variance of the brightness component of the adjacent image block corresponding to the image block when the image block is a non-boundary image block, wherein the non-boundary image block is an image block except for the boundary image block in the source image.

According to the image scaling method provided by the invention, the target interpolation method comprises a first interpolation method and a second interpolation method, wherein the clock period required by the first interpolation method for processing one pixel point is larger than the clock period required by the second interpolation method for processing one pixel point.

According to the image scaling method provided by the invention, the determining an image scaling strategy based on the maximum pixel number matched by each target interpolation method and the total pixel number in each class of the image blocks comprises the following steps:

under the condition that the maximum pixel point number matched by the first interpolation method is smaller than or equal to the total pixel point number of the image blocks of the third category, traversing the categories of the image blocks where the pixel points are located one by one from the first pixel point of the source image;

and determining a target interpolation method required to be used for processing the pixel point as the second interpolation method under the condition that the class of the image block where the pixel point is located is not the third class.

According to the image scaling method provided by the invention, after traversing the categories of the image blocks where the pixel points are located one by one, the method further comprises:

determining a first accumulated total pixel point number of the pixel points currently required to use the first interpolation method under the condition that the class of the image block where the pixel points are located is the third class;

Determining a target interpolation method required to be used for processing the pixel points as the first interpolation method under the condition that the first accumulated total pixel point number is smaller than the total pixel point number of the image blocks of the third category;

and determining a target interpolation method required to be used for processing the pixel points as the second interpolation method under the condition that the first accumulated total pixel point number is larger than or equal to the total pixel point number of the image blocks of the third category.

According to the image scaling method provided by the invention, the image scaling strategy is determined based on the maximum pixel point number matched by each target interpolation method and the total pixel point number in the image block of each category, and the method further comprises the following steps:

when the maximum pixel point number matched by the first interpolation method is larger than the total pixel point number of the image blocks of the third category, but smaller than or equal to the sum of the total pixel point number of the image blocks of the third category and the total pixel point number of the image blocks of the second category, traversing the categories of the image blocks where the pixel points are located one by one from the first pixel point of the source image;

and determining a target interpolation method required to be used for processing the pixel point as the second interpolation method under the condition that the class of the image block where the pixel point is located is the first class.

and determining a target interpolation method required to be used for processing the pixel point as the first interpolation method under the condition that the class of the image block where the pixel point is located is the third class.

determining a second accumulated total pixel point number of the pixel points matched with the second category of the first interpolation method required to be used currently under the condition that the category of the image block where the pixel points are located is the second category;

determining a first interpolation method required to be used for processing the pixel points as the first interpolation method under the condition that the second accumulated total pixel points are smaller than a first target total pixel point, wherein the first target total pixel point is determined based on the maximum pixel point matched with the first interpolation method and the total pixel point of the image blocks of the third category;

and determining a target interpolation method required to be used for the pixel points as the second interpolation method under the condition that the second accumulated total pixel point number is larger than or equal to the first target total pixel point number.

determining a type of a target chip, wherein the target chip is a chip for distributing hardware resources to the image scaling module, when the maximum pixel point number matched by the first interpolation method is larger than the sum of the total pixel point number of the image blocks of the third category and the total pixel point number of the image blocks of the second category, but smaller than or equal to the total pixel point number in the source image;

and when the target chip is a resource consumption sensitive chip, determining that the target interpolation method required to be used for the pixel points in the third class of image blocks and the pixel points in the second class of image blocks is the first interpolation method, and determining that the target interpolation method required to be used for processing the pixel points in the first class of image blocks is the second interpolation method.

According to the image scaling method provided by the invention, after determining the type of the target chip, the method further comprises the following steps:

under the condition that the target chip is a resource consumption insensitive chip, starting from the first pixel point of the source image, traversing the categories of the image blocks where the pixel points are located one by one;

And determining a target interpolation method required to be used for processing the pixel point as the first interpolation method under the condition that the class of the image block where the pixel point is located is the third class or the second class.

determining a third accumulated total pixel point number of the pixel points currently required to use the first interpolation method under the condition that the category of the image block where the pixel points are located is the first category;

determining a first interpolation method required to process the pixel points as the first interpolation method under the condition that the third accumulated total pixel points are smaller than a second target total pixel points, wherein the second target total pixel points are determined based on the maximum pixel points matched by the first interpolation method, the total pixel points of the third class image blocks and the total pixel points of the second class image blocks;

and determining a target interpolation method required to be used for processing the pixel points as the second interpolation method under the condition that the third accumulated total pixel point number is larger than or equal to the second target total pixel point number.

and determining a target interpolation method required to be used for processing the pixel points in the first class of image blocks, the pixel points in the second class of image blocks and the pixel points in the third class of image blocks as the first interpolation method under the condition that the maximum pixel point matched by the first interpolation method is larger than the total pixel point in the source image.

According to the image scaling method provided by the invention, the maximum pixel number matched by the first interpolation method is obtained by the following formula:

wherein ,

maximum number of pixels matched for the first interpolation, +.>

Interpolation factor for the first interpolation, +.>

Interpolation factor for the second interpolation, +.>

For processing capacity, +.>

For the second resolution, ++>

At a second frame rate.

in the case where one interpolation is included in the target interpolation, the source image is scaled to the target image using the target interpolation.

In a second aspect, there is provided an image scaling apparatus comprising:

the classifying unit is used for dividing a source image into a plurality of image blocks and determining the category of the image blocks based on the brightness components of all pixel points in the image blocks;

a determining unit, configured to determine a target interpolation method based on a first resolution and a first frame rate of the source image, a second resolution and a second frame rate of a target image, and an image scaling performance of an image scaling module that processes the source image, where the image scaling performance is a number of pixel points that can complete scaling processing in a unit time;

and the scaling unit is used for determining an image scaling strategy based on the maximum pixel number matched by the target interpolation method and the total pixel number in the image block of each category and performing image scaling processing on the source image based on the image scaling strategy, wherein the image scaling strategy is a strategy of the interpolation method used by each pixel point in the source image when the target interpolation method comprises at least two interpolation methods.

In a third aspect, the present invention also provides a server comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing any one of the image scaling methods described above when executing the program.

In a fourth aspect, the present invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing any one of the image scaling methods described above when executing the program.

In a fifth aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements an image scaling method as described in any of the above.

The invention provides an image scaling method, which comprises the steps of firstly dividing a source image into a plurality of image blocks with the same size, determining the category of the image blocks based on the brightness components of all pixel points in the image blocks, then determining a target interpolation method required to be used for scaling the source image to the target image based on the first resolution and the first frame rate of the source image, the second resolution and the second frame rate of the target image and the image scaling performance of an image scaling module for processing the source image, and further determining a target interpolation method used by each pixel point in the source image based on the maximum pixel point matched by the target interpolation method and the total pixel point in the image blocks of each category under the condition that the target interpolation method comprises at least two interpolation methods, thereby realizing the self-adaptive scaling of the image in the image scaling process by analyzing the distribution characteristics of the brightness components of the source image, the resolution and the frame rate between the source image and the target image and the image scaling performance of the image scaling module, determining the target interpolation method with the best display effect under the image scaling performance of the image scaling module.

The invention provides an image scaling method, which is used for determining interpolation factors matched with a processing source image based on the image scaling performance of an image scaling module of the processing source image; determining a processing capability based on a first resolution, a first frame rate, and an interpolation factor of the source image; the interpolation parameters are determined based on the second resolution, the second frame rate and the processing capacity of the target image, the target interpolation method is determined based on the interpolation parameters, and the target interpolation method with the best display effect which can be used by the image scaling module in the adaptive switching mode is determined by analyzing the resolution and the frame rate between the source image and the target image, the image scaling performance of the image scaling module and the clock period required by each interpolation method for processing one pixel point, so that the image scaling with higher resolution and frame rate is realized under the fixed image scaling performance.

The invention provides an image scaling method, which is used for determining the variance of the brightness component of an image block based on the brightness component of each pixel point in the image block; and determining a smoothness approximation value of the image block based on the variances of the luminance components of the image block and the variances of the luminance components of the adjacent image blocks corresponding to the image block, thereby correcting the variances of the luminance components of the image block through the variances of the luminance components of the adjacent image blocks of the image block to realize the accurate division of the image block types, and improving the accuracy of the image scaling strategy generated subsequently through the accurate division of the image block types, so as to improve the display effect of the target image obtained by the adaptive switching interpolation method.

Drawings

In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic flow chart of an image scaling method according to an embodiment of the present invention;

FIG. 2 is a schematic view of a source image according to an embodiment of the present invention;

FIG. 3 is a schematic view of a scene of image blocks on image boundaries in a source image provided by an embodiment of the present invention;

fig. 4 is a schematic structural diagram of an image scaling device according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The image scaling method provided by the embodiment of the invention is described below with reference to fig. 1. As shown in fig. 1, a flowchart of an image scaling method according to an embodiment of the present invention is shown, and the method mainly includes the following steps:

step 101, dividing a source image into a plurality of image blocks with the same size, and determining the category of the image block based on the brightness component of each pixel point in the image block;

it will be understood that the source image refers to the original image before scaling, and the data format of the original image is typically RGB format, so before dividing the source image into image blocks, the source image in RGB format may be converted into YCbCr format, that is, the luminance component Y is separated from the chrominance components Cb and Cr.

The number of pixels in each image block is equal, in order to improve the accuracy of the subsequent processing result, the size of each image block may be set to n×n pixels, where the value of N may be flexibly adjusted along with the size of the source image, and in general, N may be 8, 16, 32, or 64.

As shown in fig. 2, each line in the source image has 1920 pixels, and the size of the image block is 16×16 pixels, so that each line in the source image has 120 image blocks, and the number of each image block in the figure is denoted by M-n, where M is the representation symbol of the image block, and n refers to the nth image block.

It should be noted that, in general, the content displayed in the image may be divided into a region with a gentle brightness change and a region with a severe brightness change, for example, background information in the image is generally a continuous pixel point with an approximate brightness component, and the main display object in the image, especially, the edge region of the display object, has a large brightness component change between the pixel points.

Step 102, determining a target interpolation method based on a first resolution and a first frame rate of the source image, a second resolution and a second frame rate of a target image, and an image scaling performance of an image scaling module for processing the source image, wherein the target interpolation method is an interpolation method required for scaling the source image to the target image;

in this embodiment, the image scaling performance of the hardware is the number of pixel points that can complete the scaling process in a unit time, which is related to the processing capability of the image scaling module itself and the processing capability of the image scaling module that is limited.

In particular, the processing power of the image scaling module itself is related to the clock frequency it allows and the computational complexity and hardware multiplier resources of the interpolation method it can use.

For example, the frequency of the image scaling module is 125MHZ, the maximum resolution is 1920×1080, the maximum frame rate is 30fps, and then it takes about 16ns to process a pixel in the image, and one clock period is about 8ns, and then it takes about 2 clock periods to process a pixel in the image. Typically, the better the image scaling performance of the image scaling module, the shorter the clock period it takes to process a pixel.

It should be noted that, because the calculation modes of the interpolation methods are different, the clock period required for processing a pixel point by adopting the interpolation methods is different, and for the interpolation method with better display effect, the longer the clock period required for processing a pixel point is, for example, bicubic interpolation calculation is required, 16 times of unitary cubic equation calculation is required, a great amount of hardware multiplier resources are consumed, if the hardware multiplier resources are fewer, multiple calculation can only be completed in series in sequence, and the number of required clock periods is more; if the hardware multiplier has a lot of resources, multiple computations can be performed in parallel, and the number of required clock cycles is much smaller. For bilinear interpolation calculation, a unitary quadratic equation needs to be calculated, the complexity is relatively low, and the required hardware multiplier resources are relatively less.

Therefore, in this embodiment, at least one suitable target interpolation method is selected according to the resolutions and frame rates of the source image and the target image, and the image scaling performance of the image scaling module that processes the source image.

It should be noted that, in the image scaling module, the processing capability of the image scaling module is limited only by the sub-modules in the system (or the chip) in general, and is related to hardware resources of a preceding module and a following module of the image scaling module, for example, in the image scaling module, the preceding module is a DDR (Double Data Rate) and the following module is an image transmission module, and the processing capability actually possessed by the image scaling module in this scenario is limited by the Data bandwidth C1 allocated to the image scaling module and the Data bandwidth C2 of the image transmission module by the DDR.

Step 103, determining an image scaling strategy based on the maximum pixel number matched by the target interpolation method and the total pixel number in the image block of each category and performing image scaling processing on the source image based on the image scaling strategy, wherein the image scaling strategy is a strategy of an interpolation method used by each pixel point in the source image when the target interpolation method comprises at least two interpolation methods.

Wherein the maximum pixel point number refers to the maximum number of pixels allowed to be processed using the target interpolation method, and the total pixel point number refers to the total number of pixels contained in all image blocks belonging to the category.

It can be understood that, when performing interpolation calculation for a region with gentle brightness change, different interpolation methods are used, the difference of final display effects is small, and different interpolation methods are used for a region with severe brightness change, and the difference of final display effects is large, so in this embodiment, based on the maximum pixel number matched by the target interpolation method and the total pixel number in each type of image block, a most suitable interpolation method is selected for each pixel point in the source image, so as to realize the maximum zoom capability under the fixed image zoom performance, and realize the image zoom with higher resolution and frame rate.

Further, in the case where one interpolation method is included in the target interpolation method, the target interpolation method is used to scale the source image to the target image, that is, when the target interpolation method is a single interpolation method, the target interpolation method is used to process the entire frame source image.

According to the image scaling method provided by the embodiment of the invention, the source image is divided into a plurality of image blocks with the same size, the category of the image blocks is determined based on the brightness components of all pixel points in the image blocks, then the target interpolation method required for scaling the source image to the target image is determined based on the first resolution and the first frame rate of the source image, the second resolution and the second frame rate of the target image and the image scaling performance of the image scaling module for processing the source image, and therefore, the image scaling performance of the image scaling module is determined by analyzing the distribution characteristics of the brightness components of the source image, the resolution and the frame rate between the source image and the target image and the image scaling performance of the image scaling module, and further, the target interpolation method used by each pixel point in the source image is determined based on the maximum pixel point number matched by the target interpolation method and the total pixel point number in the image blocks of each category under the condition that the target interpolation method is included, so that the image scaling performance can be adaptively realized in the image scaling process, and the image scaling performance can be realized under the condition that the current scaling performance is higher than that the image scaling performance can be realized by adopting the single scaling method in the prior art.

Further, in this embodiment, after determining the image scaling policy, each pixel point is sequentially transmitted to a corresponding computing unit, for example, a target interpolation method used by a certain pixel point is a bilinear interpolation method, then the pixel point is transmitted to a computing unit capable of performing bilinear interpolation processing, after being transmitted to the corresponding computing unit, each pixel point is further transmitted to a post-processing module after being subjected to interpolation processing in each computing unit, and image increment processing, for example, filtering processing of a scene, is performed in the post-processing module, and in addition, since the display format of an image is usually in RGB format, format conversion processing is performed on the image, so that a required target image can be obtained.

In some embodiments, the determining the target interpolation based on the first resolution and the first frame rate of the source image, the second resolution and the second frame rate of the target image, and an image scaling performance of an image scaling module that processes the source image comprises:

The interpolation factor for processing source image matching refers to a clock period required for processing one pixel point by the interpolation method with the best display effect under the image scaling performance of the hardware.

Because the calculation modes of the interpolation methods are different, the clock period required by processing one pixel point by adopting the interpolation methods is different, and the longer the clock period required by processing one pixel point is, for the interpolation method with better display effect, for example, 1 clock period is required by calculating one pixel point by using the nearest neighbor interpolation method, 2 clock periods are required by calculating one pixel point by using the bilinear interpolation method, and 4 clock periods are required by calculating one pixel point by using the bicubic interpolation method.

For ease of understanding, for this example, the image scaling module has a frequency of 125MHZ, a maximum resolution of 1920 x 1080, and a maximum frame rate of 30fps, and then it takes approximately 16ns to process a pixel in the image, and a clock period of approximately 8ns, and then it takes approximately 2 clock periods to process a pixel in the image, so the interpolation method with the best display effect that can be used is bilinear interpolation, and its interpolation factor is 2 clock periods.

Specifically, the processing power is obtained by the following formula:

wherein ,

for processing capacity, +.>

For interpolation factor +.>

At the level of the first resolution of the image,

is the first frame rate.

Specifically, the interpolation parameter is obtained by the following formula:

wherein ,

for interpolation parameters +.>

For processing capacity, +.>

For the second resolution, ++>

At a second frame rate.

According to the image scaling method provided by the embodiment of the invention, the target interpolation method with the best display effect which can be used under the image scaling performance of the image scaling module is determined by analyzing the resolution and the frame rate between the source image and the target image, the image scaling performance of the image scaling module and the clock period which is required by each interpolation method to process one pixel point.

In some embodiments, the determining a target interpolation based on the interpolation parameters includes:

The minimum preset interpolation parameter refers to the interpolation parameter of the first preset interpolation method with the lowest complexity and the worst display effect in the interpolation methods used by the image scaling module, and the maximum preset interpolation parameter refers to the interpolation parameter of the first preset interpolation method with the highest complexity and the best display effect in the interpolation methods used by the image scaling module.

For example, the interpolation methods that can be used by the image scaling module include a nearest field interpolation method, a bilinear interpolation method and a bicubic interpolation method, and the first preset interpolation method is the nearest field interpolation method, the minimum preset interpolation parameter is 1 clock cycle, the second preset interpolation method is the bicubic interpolation method, and the maximum preset interpolation parameter is 4 clock cycles.

In this embodiment, if the currently calculated interpolation parameter is smaller than or equal to the interpolation parameter of the first preset interpolation method with the lowest complexity and the worst display effect in the interpolation methods that can be used by the current image scaling module, it indicates that the source image that needs to be processed by the current image scaling module has exceeded the maximum scaling capability, and at this time, the whole frame of source image can only use the first preset interpolation method, but a situation that the image scaling process is blocked occurs, and if the currently calculated interpolation parameter is greater than or equal to the interpolation parameter of the second preset interpolation method with the highest complexity and the best display effect in the interpolation methods that can be used by the current image scaling module, it indicates that the current image scaling module can use the second preset interpolation method to process the whole frame of source image.

Further, when the interpolation parameter is greater than the minimum preset interpolation parameter but less than the maximum preset interpolation parameter, it indicates that the whole source image can be divided into at least two areas, and different interpolation methods are adopted for processing the pixel points in each area, for example, when the minimum preset interpolation parameter is 1 and the maximum preset interpolation parameter is 4, the interpolation parameter of the image scaling module is greater than 2 but less than or equal to 4, the source image can be processed by adopting a nearest field interpolation method, a bilinear interpolation method and a bicubic interpolation method, or can be processed by adopting only a bilinear interpolation method and a bicubic interpolation method.

Because the display effects after the processing of different interpolation methods are different, when the types of the interpolation methods are more, the more obvious image splitting condition is presented in the finally obtained target image, so that in general, at most, two different interpolation methods are adopted to process the source image.

Preferably, in one example, the target interpolation is determined in the following manner:

first case: and under the condition that the interpolation parameter is smaller than or equal to 1, determining that the target interpolation method is a nearest neighbor interpolation method, namely, when the source image processed by the current hardware exceeds the maximum scaling capability, the whole frame of source image uses the nearest neighbor interpolation method, but the condition that the image scaling process is blocked occurs.

Second case: and under the condition that the interpolation parameter is larger than 1 and smaller than or equal to 2, determining the target interpolation method to be a nearest neighbor interpolation method and a bilinear interpolation method, namely, a part of the area of the source image can use the bilinear interpolation method, and the part of the area uses the nearest neighbor interpolation method.

Third case: in the case that the interpolation parameter is greater than 2 but less than or equal to 4, the target interpolation method is determined to be bilinear interpolation method and bicubic interpolation method, that is, a part of the region of the source image can use bilinear interpolation method, and a part of the region uses bilinear interpolation method.

Fourth case: and under the condition that the interpolation parameter is larger than 4, determining that the target interpolation method is a bicubic interpolation method, namely that the bicubic interpolation method can be used for the whole frame of source image.

According to the image scaling method provided by the embodiment of the invention, the target interpolation method with the best display effect, which can be used under the image scaling performance of the image scaling module, is determined by analyzing the relation between the interpolation parameter and the minimum preset interpolation parameter and the maximum preset interpolation parameter, so that the image scaling with higher resolution and frame rate is realized under the premise of not influencing the display effect of the target image.

In some embodiments, the categories include a first category for characterizing the image block as being in an image quasi-stationary region of the source image, a second category for characterizing the image block as being in an image gradual change region of the source image, and a third category for characterizing the image block as being in an image gradual change region of the source image;

It should be noted that, in general, the content displayed in the image may be divided into a region with a gentle brightness change and a region with a severe brightness change, for example, background information in the image is generally a continuous pixel point with an approximate brightness component, and the main display object in the image, particularly, the edge region of the display object, has a large brightness component change between the pixel points.

Based on the above embodiment, the classifying the image block based on the brightness components of each pixel point in the image block, and determining the class of the image block includes:

and classifying the image blocks based on the smoothness approximation value of the image blocks, and determining the types of the image blocks.

Where adjacent image blocks refer to image blocks immediately adjacent to an image boundary of an image block, such as an image block immediately adjacent to an upper image boundary of an image block, or an image block immediately adjacent to a lower image boundary of an image block, or an image block immediately adjacent to a left image boundary of an image block, or an image block immediately adjacent to a right image boundary of an image block.

In one example, to improve the correction effect of an image block, an image block immediately adjacent to the image boundaries of four orientations of the upper, lower, left, and right sides of the image block is used to correct the luminance distribution condition of the image block.

In this embodiment, the variance is used to represent the brightness distribution condition of the pixels in the image blocks, specifically, the expectation of the brightness component of each image block may be calculated first, and then the variance of the brightness component of each image block may be calculated according to the expectation.

In this embodiment, for an image block near a dividing line between an image quasi-still region, an image gentle change region and an image severe change region, the brightness distribution condition of the image block is affected by the brightness component of a part of pixels, so as to affect the accuracy of the classification result of the image block.

In one embodiment, the smoothness approximation of the image block is obtained by weighted summation of the variance of the luminance component of the image block and the variance of the luminance component of the neighboring image block corresponding to the image block; the difference between the weight value of the variance of the luminance component of the image block and the weight value of the variance of the luminance component of the neighboring image block increases as the number of total pixels in the image block increases.

For example, if the adjacent image block includes an image block immediately adjacent to the image boundary of four directions of the upper, lower, left and right of the image block, the following formula may be used to obtain the smoothness approximation of the image block:

wherein ,

for the smoothness approximation of the nth image block, +.>

For the variance of the luminance component of the nth image block,/->

For the variance of the luminance component of the image block immediately above the nth image block, +.>

For the image immediately below the nth image blockVariance of luminance component of block, +.>

For the variance of the luminance component of the image block immediately to the left of the nth image block +.>

The variance of the luminance components of the image block to the right of the immediately n-th image block, a, b and c are conventional coefficients, respectively, where c is related to the number of image blocks per line in the source image.

For example, referring to FIG. 2, there are 1920 pixels per line in the source image, the size of the image block is 16×16 pixels, and thus there are 120 image blocks per line in the source image, each image block is denoted as M-n, where n refers to the nth image block, and then the smoothness approximation of the nth image block M-n

Is based on the variance of the luminance component of the image block M-n-120 above it +.>

Variance of luminance component of image block M-n+120 below it +. >

Variance of luminance component of image block M-n-1 to the left thereof +.>

Variance of luminance component of image block M-n+1 above it

And variance of luminance component of nth image block +.>

And (3) determining.

Wherein a is greater than b, and the difference between a and b is positively correlated with the number of pixels in the image block.

It should be noted that, the greater the number of pixels in an image block, the smaller the influence of the luminance component of a portion of the pixels on the luminance distribution of the image block, and therefore, when the greater the number of pixels in an image block, the difference between a and b can be appropriately increased.

In this embodiment, for an image block near a dividing line between an image quasi-still region, an image gentle change region and an image sharp change region, the brightness distribution condition of the image block is affected by the brightness component of a part of pixels, so as to affect the accuracy of the classification result of the image block, so that a weighted summation mode is adopted to calibrate the variance of the brightness component of the image block through the adjacent image blocks, and in addition, since the more the number of pixels in the image block is, the smaller the effect of the brightness component of the part of pixels is on the brightness distribution condition of the image block, the correction force of the variance of the image block can be adjusted by adjusting the difference between the weight value of the image block and the weight value of the adjacent image block according to the number of pixels in the image block, so as to further improve the accuracy of the classification of the subsequent image block.

In one example, the classifying the image block based on the smoothness approximation of the image block, determining the class of the image block includes:

In this embodiment, the lowest preset threshold and the highest preset threshold are inversely related to the image scaling performance. That is, when the image scaling performance of the image scaling module is higher, the values of the lowest preset threshold and the highest threshold can be properly reduced, so that the number of pixel points processed by the target interpolation method with better display effect is increased, and the image scaling effect is improved.

According to the image scaling method provided by the embodiment of the invention, the image blocks are precisely classified by the brightness components of the pixel points in the image blocks, so that the target interpolation method can be adaptively switched in the subsequent image scaling process.

In some embodiments, the determining the smoothness approximation of the image block based on the variance of the luminance component of the image block and the variance of the luminance component of the neighboring image block corresponding to the image block further includes:

Wherein, the image boundary refers to the boundary formed by a circle of pixel number points at the outermost side of the image, referring to fig. 3, 26 image blocks in fig. 3: the image blocks M-1, M-2 and …, M-6, M-7 and M-12 and … are boundary image blocks M-54.

In this embodiment, whether the image block is a boundary image block can be determined according to the number M-n of the image blocks and the number of image blocks in each row and each column in the source image, for example, 120 image blocks in each row and 75 image blocks in each column in the source image, and the image blocks with numbers M-1, M-2 … M-120, M-121 … M-240, M-241 … M-360 … M-8881 … M-9000 are boundary image blocks.

In this embodiment, since the main display objects in the image are mainly distributed in the middle position of the image, that is, the vicinity of the image boundary is usually the background information in the image, in order to reduce the amount of calculation and improve the image scaling efficiency, the adjacent image blocks are not used for correcting the brightness distribution of the image block for the boundary image block, and the adjacent image blocks are used for correcting the brightness distribution of the image block for the non-boundary image block.

In some embodiments, in the case that the number of the target interpolation methods is two, the target interpolation method includes a first interpolation method and a second interpolation method, where a clock period required for the first interpolation method to process one pixel point is greater than a clock period required for the second interpolation method to process one pixel point.

Based on the above embodiment, the determining an image scaling strategy based on the maximum pixel number matched by each target interpolation method and the total pixel number in the image block of each category includes:

In this embodiment, the maximum pixel number matched by the first interpolation method is obtained by the following formula:

wherein ,

maximum number of pixels matched for the first interpolation, +.>

Interpolation factor for the first interpolation, +.>

Interpolation factor for the second interpolation, +.>

For processing capacity, +.>

For the second resolution, ++>

At a second frame rate.

Further, after traversing the categories of the image blocks where the pixel points are located one by one, the method further includes:

In this embodiment, when the total number of pixels included in all the image blocks of the third category is not less than the maximum number of pixels allowed to be processed by the first interpolation method, a part of the pixels of the third category are selected to be processed by the first interpolation method, and the rest of the pixels of the third category are processed by the second interpolation method.

Specifically, in order to ensure the display effect of the picture in the scaled target image, the maximum number of pixels matched by the first interpolation method is sequentially selected from the first pixel of the source image to the third class of pixels.

In some embodiments, the determining the image scaling strategy based on the maximum pixel number matched by each target interpolation method and the total pixel number in the image block of each class of the class further includes:

Determining a target interpolation method required to be used for processing the pixel point as the second interpolation method under the condition that the class of the image block where the pixel point is located is the first class;

determining a target interpolation method required to be used for processing the pixel point as the first interpolation method under the condition that the class of the image block in which the pixel point is positioned is the third class;

and determining a target interpolation method required to be used for processing the pixel points as the second interpolation method under the condition that the second accumulated total pixel point number is larger than or equal to the first target total pixel point number.

In this embodiment, when the total number of pixels included in all the image blocks of the third category is smaller than the maximum number of pixels allowed to be processed by the first interpolation method, but the maximum number of pixels allowed to be processed by the first interpolation method does not exceed the total number of pixels included in all the image blocks of the third category and the second category, the pixels in all the image blocks of the third category are processed by the first interpolation method, the pixels in all the image blocks of the first category are processed by the second interpolation method, a part of the pixels in the second category are selected and processed by the first interpolation method, and the rest of the pixels in the second category are processed by the second interpolation method.

Specifically, in order to ensure the display effect of the frame in the scaled target image, from the first pixel point of the source image, sequentially selecting a first target total pixel point number of the second class of pixel points, where the second target total pixel point number is the maximum number of pixel points that can be processed by the first interpolation method after the first interpolation method has processed the total of pixel points contained in all the image blocks of the third class.

It should be noted that, the resource consumption sensitive chip refers to a case that the number of hardware resources is increased for the image scaling module, so that the number of allocated hardware resources of other modules in the chip is reduced.

For example, for a fully-customized or semi-customized chip, since each module in the chip has been pre-allocated with a fixed amount of hardware resources, if the hardware resource data is added to the image scaling module, the amount of the pre-allocated hardware resources of other modules will be reduced, so that the processing effect of other modules will be affected, and the chip is a resource consumption sensitive chip.

It should be noted that, since the image blocks of the first category refer to the image blocks located at the quasi-static region of the image of the source image, the display effects of the image blocks in the region taken by using different interpolation methods are not greatly different, so in this embodiment, when the target chip is a resource consumption sensitive chip, the resources representing the hardware are relatively tense, only the pixel points in the image blocks of the third category and the image blocks of the second category are processed by adopting the first interpolation method, so as to avoid the normal operation of the hardware.

In some embodiments, after determining the type of the target chip, the method further includes:

determining a target interpolation method required to be used for processing the pixel point as the first interpolation method under the condition that the class of the image block where the pixel point is located is the third class or the second class;

The resource consumption insensitive chip refers to that the number of hardware resources allocated by other modules in the chip is not reduced under the condition of increasing the number of hardware resources for the image scaling module.

In this embodiment, when the target chip is a resource consumption insensitive chip, the resources representing the hardware are sufficient, and besides the first interpolation method is adopted to process the pixel points in the third class image block and the second class image block, a part of the pixel points in the first class may be selected to process the pixel points in the first class, and the second interpolation method is adopted to process the pixel points in the rest of the first class, thereby improving the display effect of image scaling.

In this embodiment, the determining step of the image scaling policy is described only by taking the relationship between the maximum pixel number of the first interpolation method and the total pixel number of the image blocks of each category as a representative, and the determining step of the image scaling policy may also be described by taking the relationship between the maximum pixel number of the second interpolation method and the total pixel number of the image blocks of each category, which is not described herein again.

According to the image scaling method provided by the embodiment of the invention, the proper image scaling strategy is selected adaptively by analyzing the relation between the maximum pixel point number of the first interpolation method and the total pixel point number of the image blocks of each category, so that the image scaling with higher resolution and frame rate is realized under the fixed image scaling performance.

The image scaling device provided in the present application will be described below, and the image scaling device described below and the image scaling method described above may be referred to correspondingly to each other.

As shown in fig. 4, an image scaling apparatus according to this embodiment includes: a classification unit 410, configured to divide a source image into a plurality of image blocks, and determine a class of the image block based on a luminance component of each pixel point in the image block;

a determining unit 420, configured to determine a target interpolation method based on the first resolution and the first frame rate of the source image, the second resolution and the second frame rate of the target image, and an image scaling performance of an image scaling module that processes the source image, where the target interpolation method is an interpolation method that is required to be used for scaling the source image to the target image, and the image scaling performance is a number of pixel points that can complete scaling processing in a unit time;

and a scaling unit 430, configured to determine an image scaling policy based on a maximum pixel number matched by the target interpolation method and a total pixel number in each of the image blocks of the class, and perform image scaling processing on the source image based on the image scaling policy, where the image scaling policy is a policy of an interpolation method used by each pixel point in the source image, when the target interpolation method includes at least two interpolation methods.

It should be noted that, the image scaling device provided in this embodiment of the present application can implement all the method steps implemented in the embodiment of the image scaling method, and can achieve the same technical effects, and specific details of the same parts and beneficial effects as those of the embodiment of the method in this embodiment are not described herein.

In another aspect, the present invention further provides a server, where the server includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when executed by the processor, implements an image scaling method provided by the above methods, where the method includes:

Fig. 5 is a schematic structural diagram of an electronic device provided by the present invention. As shown in fig. 5, the electronic device may include: processor 510, communication interface (Communications Interface) 520, memory 530, and communication bus 540, wherein processor 510, communication interface 520, memory 530 complete communication with each other through communication bus 540. Processor 510 may invoke logic instructions in memory 530 to perform an image scaling method comprising:

determining a target interpolation method based on the first resolution and the first frame rate of the source image, the second resolution and the second frame rate of a target image and the image scaling performance of hardware for processing the source image, wherein the target interpolation method is an interpolation method required to be used for scaling the source image to the target image, and the image scaling performance is the number of pixel points capable of completing scaling processing in unit time;

Further, the logic instructions in the memory 530 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In another aspect, the present invention also provides a computer program product comprising a computer program storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, is capable of performing a method of image scaling provided by the methods described above, the method comprising:

In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the image scaling method provided by the above methods, the method comprising:

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image scaling method, comprising:

2. The image scaling method of claim 1, wherein the determining a target interpolation based on the first resolution and the first frame rate of the source image, the second resolution and the second frame rate of the target image, and an image scaling performance of an image scaling module that processes the source image comprises:

3. The image scaling method of claim 2, wherein said determining a target interpolation based on said interpolation parameters comprises:

4. The image scaling method of claim 2, wherein the interpolation parameter is obtained by the following formula:

；

wherein ,

for interpolation parameters +.>

For processing capacity, +.>

For the second resolution, ++>

At a second frame rate.

5. The image scaling method of claim 1 wherein the categories include a first category for characterizing the image block as being in an image quasi-stationary region of the source image, a second category for characterizing the image block as being in an image gradual change region of the source image, and a third category for characterizing the image block as being in an image gradual change region of the source image;

6. The image scaling method of claim 5, wherein said determining the class of the image block based on the luminance component of each pixel point in the image block comprises:

7. The image scaling method of claim 6, wherein the determining the category of the image block based on the smoothness approximation of the image block comprises:

8. The image scaling method of claim 6, wherein the smoothness approximation of the image block is obtained by weighted summation of the variance of the luminance component of the image block and the variance of the luminance component of the neighboring image block to which the image block corresponds; the difference between the weight value of the variance of the luminance component of the image block and the weight value of the variance of the luminance component of the neighboring image block increases as the number of total pixels in the image block increases.

9. The image scaling method of claim 6, wherein the determining the smoothness approximation of the image block based on the variance of the luminance component of the image block and the variance of the luminance component of the neighboring image block to which the image block corresponds further comprises:

and determining that the smoothness approximation value of the image block is equal to the variance of the brightness component of the image block when the image block is a boundary image block, wherein the boundary image block comprises pixel points on the image boundary of the source image.

10. The image scaling method of claim 9, further comprising:

11. The image scaling method of any one of claims 5 to 10, wherein the target interpolation method comprises a first interpolation method and a second interpolation method, wherein a clock period required for processing one pixel point by the first interpolation method is greater than a clock period required for processing one pixel point by the second interpolation method.

12. The method of claim 11, wherein determining an image scaling strategy based on the maximum number of pixels matched by the target interpolation and the total number of pixels in the image block of each of the categories comprises:

13. The image scaling method of claim 12, further comprising, after traversing the categories of the image blocks in which the pixels are located one by one:

14. The method of claim 11, wherein determining the image scaling strategy based on the maximum number of pixels matched by each of the target interpolations and the total number of pixels in each of the classes of image blocks, further comprises:

15. The image scaling method of claim 14, further comprising, after traversing the categories of the image blocks in which the pixels are located one by one:

16. The image scaling method of claim 14, further comprising, after traversing the categories of the image blocks in which the pixels are located one by one:

17. The method of claim 11, wherein determining the image scaling strategy based on the maximum number of pixels matched by each of the target interpolations and the total number of pixels in each of the classes of image blocks, further comprises:

18. The image scaling method of claim 17, further comprising, after said determining the type of the target chip:

19. The image scaling method of claim 18, further comprising, after traversing the categories of the image blocks in which the pixels are located one by one:

20. The method of claim 11, wherein determining the image scaling strategy based on the maximum number of pixels matched by each of the target interpolations and the total number of pixels in each of the classes of image blocks, further comprises:

21. The image scaling method of claim 11 wherein the maximum number of pixel points matched by said first interpolation is obtained by the following formula:

；

wherein ,

maximum number of pixels matched for the first interpolation, +.>

Interpolation factor for the first interpolation, +.>

Interpolation factor for the second interpolation, +.>

For processing capacity, +.>

For the second resolution, ++>

At a second frame rate.

22. The image scaling method of claim 1, further comprising:

23. An image scaling apparatus, comprising:

a determining unit, configured to determine a target interpolation method based on a first resolution and a first frame rate of the source image, a second resolution and a second frame rate of a target image, and an image scaling performance of an image scaling module that processes the source image, where the target interpolation method is an interpolation method that is required to be used for scaling the source image to the target image, and the image scaling performance is a number of pixel points that can complete scaling processing in a unit time;

24. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the image scaling method of any one of claims 1 to 22 when the program is executed.

25. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the image scaling method of any one of claims 1 to 22.