WO2022252080A1

WO2022252080A1 - Apparatus and method for generating a bloom effect

Info

Publication number: WO2022252080A1
Application number: PCT/CN2021/097482
Authority: WO
Inventors: Baoquan Liu; Shaozhuang SHI
Original assignee: Huawei Technologies Co.,Ltd.
Priority date: 2021-05-31
Filing date: 2021-05-31
Publication date: 2022-12-08
Also published as: EP4248396A4; EP4248396A1

Abstract

An image processing apparatus (1000) for generating a bloom effect comprising one or more processors (1001) and a memory (1002) storing in non-transient form data defining program code executable by the one or more processors (1001), wherein the program code, when executed by the one or more processors (1001), causes the image processing apparatus (1000) to: obtain an input image (901, 1701, 1801); generate a down-sampled image (502, 503) from the input image (902, 1802); select a region of one or more pixels of the down-sampled image (502, 503) in dependence on a brightness of the pixels (903, 1702, 1803); generate an up-sampled image region from the down-sampled image region (904, 1804); and generate an output bloom effect image (506) in dependence on the up-sampled image region (905, 1703, 1805). By generating an up-sampled image region from the down-sampled image region (904, 1804) and generating an output bloom effect image (506) in dependence on the up-sampled image region (905, 1703, 1805), it may be possible to enable the bloom effect algorithm (500) to reduce the number of sampling points to be up-sampled or filtered, which may reduce the computational load reducing the power consumption and rendering latency.

Description

APPARATUS AND METHOD FOR GENERATING A BLOOM EFFECT

FIELD OF THE INVENTION

This invention relates to image processing apparatuses and methods for generating a bloom effect, for example, for video games.

BACKGROUND

Bloom effect, sometimes referred to as light bloom, glow or halo, is a computer graphics effect. Bloom effect is often used in video games to reproduce an imaging effect of real-world camera. The effect produces fringes, or feathers, of light extending from the borders of bright areas in an image. This contributes to the illusion of an extremely bright light which overwhelms the camera, or eye, capturing the scene. Bloom effect is widely used in video games, particularly on a mobile platform, and may also be suitable for many other visual effects. Bloom effect may also be used as an indicator of the game engine rendering capability.

Bloom effect is usually implemented as a post-processing algorithm that obtains light bleeding effects through controllable rendering parameters. Bright areas, or light sources, are often difficult to express to the observer because the brightness range of the monitor may be limited. One way to distinguish bright light sources is to make them glow on the monitor, and the light sources bleeds around. This gives the observer a sense that the light sources, or the bright regions, may extend far beyond the original extent of the bright part of the image, so that they look more realistic. As a result, the image pixels of the bright light may appear to bleed beyond the original geometry borders.

The bloom effect may be achieved by using a post-processing, after the 3D scene rendering, to produce a bloom effect on bright areas in the rendered image. As shown in Figures 1A and 1B, the bloom effect may improve the illumination effect. Figure 1A illustrates an exemplary input image 101. Figure 1B illustrates an exemplary output image 103 with bloom effect. The input image 101 may comprise a bright region 102, or a plurality of bright regions 102, as shown in Figure 1A. The bloom effect algorithm may be applied to the bright region (s) 102 to produce the bloom regions 104, as shown in Figure 1B.

A common alternative to filtering high frequency signals is convolution filtering. Convolution filtering uses a Gaussian blur process to perform a convolution of an input image with a normal distribution function. Because the normal distribution function is also known as the "Gaussian distribution" , this technique is known as Gaussian blurring. Because the Fourier transform of Gaussian function is another Gaussian function, Gaussian blur is a low pass filtering to the input image.

Figures 2A and 2B illustrate an exemplary Gaussian blur used to perform bloom effect. Figure 2A illustrates an exemplary Gaussian kernel 201 used to perform bloom effect. Figure 2B illustrates an exemplary Gaussian matrix 202 used to perform bloom effect.

The Gaussian kernel 201 for Gaussian blur may be a 2D pixel array, preferably comprising a square arrangement, as shown in Figure 2A. In this way, the pixel values may correspond to the values of the 2D Gaussian function curve. The Gaussian kernel 201 illustrated in Figure 2A comprises a preferably 5x5 arrangement. Alternatively, the Gaussian kernel 201 may comprise different arrangements depending on the computational requirements.

To perform a Gaussian blur to an input image 203, each pixel, and neighbouring pixels, in the input image are multiplied by a Gaussian kernel 201 to weight each pixel. The combination of the resulting values are added together to produce the weighted sum to be stored in the output image 205. As shown in Figure 2B, the Gaussian blur comprises a 3x3 matrix 204 applied to the input image 203.

Figures 3A, 3B and 3C schematically illustrate components of an exemplary dual filtering algorithm 300 of the prior art. Figure 3A schematically illustrates a down-sample filter 301 for a dual filtering algorithm of the prior art. Figure 3B schematically illustrates an up-sample filter 302 for a dual filtering algorithm of the prior art. Figure 3C schematically illustrates a flow chart of an exemplary dual filtering algorithm 303 of the prior art.

A Kawase blur algorithm may be used for bloom post-processing effects. It may also be used as a general blur algorithm and may produce a very similar result to Gaussian blur in terms of blur appearance. The Kawase blur algorithm may sample four corners away from the current pixel. The Kawase blur algorithm then may perform a weighted sum operation at each of the iterative ping-pong processing between two textures of the same size. The Kawase blur algorithm may provide a multi-pass filter with a simple pattern of four samples that lie in the centre of four pixels. In this way. The Kawase blur algorithm may maximally exploit bilinear filtering and may be an effective approximation of a Gaussian blur.

A Kawase blur algorithm may use a four-pixel kernel which moves further away from the pixel be processed as the iteration steps increase. Conversely, a Gaussian blurmay use fixed kernel from the beginning to the end of the process. In this way, the Kawase blur algorithm may use less samples than Gaussian blur.

A dual filter algorithm 303, as shown in Figure 3C, may be used to further optimise a Kawase blur algorithm. The dual filter algorithm 303 may be derived from the Kawase blur algorithm. The dual filtering algorithm 303 may perform multiple passes of down-

sampling

305, 306, 307, 308 and up-

sampling

309, 310, 311, 312 with fixed filter stencils /patterns. In contrast to

mipmapping, the stencils used for up-

sampling

305, 306, 307, 308 and down-

sampling

309, 310, 311, 312 may produce a more circular shape which may reduce the likelihood of blocky artifacts.

The dual filtering may comprise some down-

sampling

305, 306, 307, 308 and up-

sampling

309, 310, 311, 312 iterative steps. The down-

sampling

305, 306, 307, 308 may be configured to capture five sampling points for each output pixel 301, as shown in Figure 3A. The texture size may be reduced to 1/4 of the original size after each down-

sampling

305, 306, 307, 308 iterative step. The up-

sampling

309, 310, 311, 312 may be performed to capture eight sampling points for each output pixel, as shown in Figure 3B. The texture size may be enlarged by four times after each up-

sampling

309, 310, 311, 312 iteration step.

Figure 4 illustrates an exemplary coding of a dual filtering algorithm of the prior art. Commonly, the code of shaders for down-

sampling

305, 306, 307, 308 and up-

sampling

309, 310, 311, 312 may be written as shown in Figure 4.

In contrast to a Kawase blur algorithm, not implemented into a dual filtering algorithm, which may perform iterative steps by ‘ping-pong’ processing between two textures of the same size, the dual blur algorithm 303 may perform a series of down-

sampling

305, 306, 307, 308 and up-

sampling

309, 310, 311, 312 iterative steps, each targeting on a different image size.

The prior art bloom effect algorithms often take a long time, with a big power cost, to process a frame image. This may be due to the large number of texture sampling points involved. This processing may be carried out on a mobile graphical processing unit (GPU) and may create a large cost in terms of the memory bandwidth. Additionally, the bloom may be used on videos, in particular video games. In this case, the bloom may be required to render each, or most, of the frames in the video game, which may put pressure on the power consumption for mobile devices. For example, for a 7x7 Gaussian blur filtering pass, the Gaussian kernel needs 49 texture sampling points for each output pixel. As a result, filtering a 1080p image (with 1920x1080 pixels, each has 4 bytes at RGBA8 format) would require approximately 400 MB byte of read bandwidth. This may be considered a large cost for a mobile GPU.

Furthermore, after one pass of blur filtering, the light source diffusion range is usually too small for a visually meaningful brightness diffusion effect. Therefore, multiple iterations of filtering passes are often required to obtain a visually meaningful brightness diffusion effect. Although the number of sampling points can be marginally reduced by using a horizontal and vertical separable filtering approach, the problem of a large number of sampling points may still not be solved.

As described herein, dual filtering is a possible solution for providing an optimized algorithm when compared to Gaussian blur and Kawase blur algorithms. Dual filtering may reduce the number of sampling points by down-sampling and up-sampling the frame image, which may improve performance and reduce power consumption. However, dual filtering may still have a large cost on mobile GPUs. In particular, dual filtering may carry out up-sampling passes which require 8 texture fetches for each output pixel, as shown in Figures 3A-3C.

The light source region of the input frame image often only accounts for a small portion of the image, particularly in video games. Existing dual filtering algorithms often process all the pixels of the entire frame input image, which wastes computing power.

To produce the bloom effect, the colour image in the frame buffer may be convolved with a convolution kernel in a post-processing step. The convolution step often requires the use of a large Gaussian kernel which is not practical for real-time graphics rendering speed.

Therefore, when using existing bloom effect algorithms, the bright regions, that require halo diffusion, in an image often only occupy a small portion of the image, and existing algorithms often use a full-image filtering solution, which may waste a lot of the computing power. Furthermore, rendering a large number of sampling results may consume a large amount of GPU memory bandwidth and may result in a slow rendering latency, particularly when using a Gaussian kernel.

It is desirable to develop an apparatus and method that overcomes the above problems.

SUMMARY

According to a first aspect there is provided an image processing apparatus for generating a bloom effect, the apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code, when executed by the one or more processors, causes the image processing apparatus to: obtain an input image; generate a down-sampled image from the input image; select a region of one or more pixels of the down-sampled image in dependence on a brightness of the pixels; generate an up-sampled image region from the down-sampled image region; and generate an output bloom effect image in dependence on the up-sampled image region.

The image processing apparatus aims to select a region of one or more pixels with particular brightness level and generate a bloom effect image in which the selected region receives the bloom effect.

Generating the output bloom effect image may use significant computational loading. Additionally, using a dual filtering method, down-sampling followed by up-sampling, may further increase the computational loading. By increasing the computational loading this may cause increased power consumption and a poorer latency of the processors. Power consumption can be a significant issue for the user, particularly if the apparatus is implemented on a mobile device, with a portable power supply. Additionally, poor latency can be issue, particularly if the input image is a frame image of a video or video game which may require a high-speed frame rendering rate.

By generating an up-sampled image region from the down-sampled image region, and generating an output bloom effect image in dependence on the up-sampled image region, it may be possible to enable the bloom effect algorithm to reduce the number of sampling points to be up-sampled, which may reduce the computational load reducing the power consumption and rendering latency.

In some implementations, the apparatus may be configured to generate the output bloom effect image by applying a blur diffusion algorithm only to the up-sampled image region.

The region of pixels may only comprise a portion of the overall image. For example, the region of pixels may comprise 10%of the image. By only applying the blur diffusion algorithm to the up-sampled image region, and not the whole of the image, this may significantly reduce the number of sampling points, depending on the size of the region of pixels, to which the blur diffusion algorithm is applied to. This may further reduce the computational load reducing the power consumption and rendering latency.

In some implementations, the apparatus may be configured to, after generating the down-sampled image from the input image, generate an up-sampled image from the down-sampled image.

By generating the up-sampled image from the down-sampled image, this may enable the up-sampling pass to output both the up-sampled image and the up-sampled image region. In this way, the output bloom effect image may comprise the entire image, including the up-sampled image and the up-sampled image region.

In some implementations, the apparatus may be configured to generate the output bloom effect image in dependence on the up-sampled image region and the up-sampled image.

By generating the output bloom effect image in dependence on the up-sampled image region and the up-sampled image, this may enable the output bloom effect image to incorporate features from both the up-sampled image and the up-sampled image region.

In some implementations, the apparatus may be configured to generate the output bloom effect image by combining the up-sampled image region and the up-sampled image.

By combining the up-sampled image region and the up-sampled image, this may enable the output bloom effect image to include the up-sampled image region features onto the up-sampled image.

In some implementations, the apparatus may be configured to combine the up-sampled image region and the up-sampled image by overlaying the up-sampled image region onto a corresponding region of the up-sampled image.

By overlaying the up-sampled image region onto a corresponding region of the up-sampled image, this may enable the up-sampled image region to be positioned onto the correct position on the up-sampled image. In other words, the blur halo may fit around the brighter region in the up-sampled image to generate the bloom effect.

In some implementations, the apparatus may be configured to select the region of one or more pixels of the down-sampled image by using a pixel minimum brightness threshold filter.

By using a pixel minimum brightness threshold filter to select the region of one or more pixels, this may enable the selected region to only comprise pixel which are bright. In this way, the bloom effect may only be applied to the brighter regions of the input image.

In some implementations, the apparatus may be configured to select the region of one or more pixels of the down-sampled image by generating a mask from the down-sampled image region.

In some implementations, the apparatus may be configured to generate the up-sampled image by using only portions of the down-sampled image not designated by the mask.

By masking the down-sampled image region with a mask, this may enable the apparatus to clearly distinguish between the region to receive the bloom effect. In this way, the area surrounding the mask is only up-sampled, and does not have the bloom effect applied. This may significantly reduce the number of sampling points, depending on the size of the region of pixels, to which the blur diffusion algorithm is applied to. This may further reduce the computational load reducing the power consumption and rendering latency.

In some implementations, the apparatus may be configured to generate the down-sampled image from the input image by using two or more down-sampling stages.

In some implementations, the apparatus may be configured to generate the up-sampled image region from the down-sampled image region by using two or more up-sampling stages.

In some implementations, the apparatus may be configured to generate the up-sampled image from the down-sampled image comprises using two or more up-sampling stages.

By using more than one down-sampling stage, or using more than one up-sampling stage, this may increase the magnitude of the bloom effect. In other words, the bloom effect, or blur halo, may diffuse further from the brighter region in the input image. By varying the number of down-sampling stages, this may enable the magnitude of the bloom effect to be varied.

In some implementations, the apparatus may be configured to repeat the steps of any of the preceding steps using the output bloom effect image as an input image for one or more iterations.

By re-inputting the output bloom effect image into the apparatus this may further increase the magnitude of the bloom effect. In other words, the bloom effect, or blur halo, may diffuse further from the brighter region in the input image. By varying the number of iterations, this may enable the magnitude of the bloom effect to be varied.

According to a second aspect there is provided a computer implemented method for generating a bloom effect, the method comprising: obtaining an input image; generating a down-sampled image from the input image; selecting a region of one or more pixels of the down-sampled image in dependence on a brightness of the pixels; generating an up-sampled image region from the down-sampled image region; and generating an output bloom effect image in dependence on the up-sampled image region.

By generating an up-sampled image region from the down-sampled image region, and generating an output bloom effect image in dependence on the up-sampled image region it may be possible to enable the bloom effect algorithm to reduce the number of sampling points to be up-sampled, which may reduce the computational load reducing the power consumption and rendering latency.

According to a third aspect there is provided an image processing apparatus for generating a bloom effect, the apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code, when executed by the one or more processors, causes the image processing apparatus to: obtain an input image; select a region of one or more pixels of the input image in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm; and generate an output bloom effect image in dependence on the filtered image region.

Generating the output bloom effect image may use significant computational loading. By increasing the computational loading this may cause increased power consumption and a poorer latency of the processors. Power consumption can be a significant issue for the user, particularly if the apparatus is implemented on a mobile device, with a portable power supply. Additionally, poor latency can be issue, particularly if the input image is a frame image of a video or video game which may require a high-speed frame rendering rate.

By selecting a region of one or more pixels of the input image in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm, and generating an output bloom effect image in dependence on the up-sampled image region it may be possible to enable the bloom effect algorithm to reduce the number of sampling points to be up-sampled, which may reduce the computational load reducing the power consumption and rendering latency.

In some implementations, the apparatus may be configured to filter the one or more pixels by using a cubic basis spline filter algorithm.

By using a cubic basis spline filter algorithm this may comprise a third order approximation. The third order approximation may be more accurate than a lower order approximation, but may be less accurate than a higher order approximation. Additionally, the third order approximation may require more sampling points than a lower order approximation, but may require less sampling points than a higher order approximation. The inventors have found that the third order approximation may provide a suitable level of accuracy while not requiring an excessive number of sampling points which may increase the computational loading.

In some implementations, the apparatus may be configured to generate the output bloom effect image by applying a blur diffusion algorithm to the filtered image region.

The region of pixels may only comprise a portion of the overall image. For example, the region of pixels may comprise 10%of the image. By only applying the blur diffusion algorithm to the filtered image region, and not the whole of the image, this may significantly reduce the number of sampling points, depending on the size of the region of pixels, to which the blur diffusion algorithm is applied to. This may further reduce the computational load reducing the power consumption and rendering latency.

In some implementations, the apparatus may be configured to generate the output bloom effect image in dependence on the filtered image region and the input image.

By generating the output bloom effect image in dependence on the filtered image region and the input image, this may enable the output bloom effect image to incorporate features from both the input image and the filtered image region.

In some implementations, the apparatus may be configured to generate the output bloom effect image by combining the filtered image region and the input image.

By combining the filtered image region and the input image, this may enable the output bloom effect image to include the filtered image region features onto the input image.

In some implementations, the apparatus may be configured to combine the filtered image region and the input image by overlaying the filtered image region onto a corresponding region of the input image.

By overlaying the filtered image region onto a corresponding region of the input image, this may enable the filtered image region to be positioned onto the correct position on the input image. In other words, the blur halo may fit around the brighter region in the input image to generate the bloom effect.

In some implementations, the apparatus may be configured to repeat the steps of any of the preceding steps by using the output bloom effect image as an input image for one or more iterations.

According to a fourth aspect there is provided a computer implemented method for generating a bloom effect, the method comprising: obtaining an input image; selecting a region of one or more pixels of the input image in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter; and generating an output bloom effect image in dependence on the up-sampled image region.

By selecting a region of one or more pixels of the input image in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm, and generating an output bloom effect image in dependence on the filtered image region it may be possible to enable the bloom effect algorithm to reduce the number of sampling points to be up-sampled, which may reduce the computational load reducing the power consumption and rendering latency.

According to a fifth aspect there is provided an image processing apparatus for generating a bloom effect, the apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code, when executed by the one or more processors, causes the image processing apparatus to: obtain an input image; generate a down-sampled image from the input image; select a region of one or more pixels of the down-sampled image in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm; generate an up-sampled image region from the down-sampled image region; and generate an output bloom effect image in dependence on the up-sampled image region.

According to a sixth aspect there is provided a computer implemented method for generating a bloom effect, the method comprising: obtaining an input image; generating a down-sampled image from the input image; selecting a region of one or more pixels of the down-sampled image in dependence on the brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm; generating an up-sampled image region from the down-sampled image region; and generating an output bloom effect image in dependence on the up-sampled image region.

By generating an up-sampled image region from the down-sampled image region, and generating an output bloom effect image in dependence on the up-sampled image region and wherein the one or more pixels are filtered by means of a basis-spline filter algorithm, it may be possible to enable the bloom effect algorithm to reduce the number of sampling points to be up-sampled, which may reduce the computational load reducing the power consumption and rendering latency. In particular, the combination of the selective up-sampling of the brighter image region and by carrying out the selection with a basis-spline filter algorithm may further reduce the number of sampling points to be up-sampled.

BRIEF DESCRIPTION OF THE FIGURES

The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

Figures 1A and 1B illustrate an exemplary input image and output bloom effect image. Figure 1A illustrates the exemplary input image. Figure 1B illustrates the exemplary output image.

Figures 2A and 2B illustrate an exemplary Gaussian blur used to perform bloom effect. Figure 2A illustrates an exemplary Gaussian kernel used to perform bloom effect. Figure 2B illustrates an exemplary Gaussian matrix used to perform bloom effect.

Figures 3A-3C schematically illustrate an exemplary dual filtering algorithm of the prior art. Figure 3A schematically illustrates a down-sample filter for a dual filtering algorithm of the prior art. Figure 3B schematically illustrates an up-sample filter for a dual filtering algorithm of the prior art. Figure 3C schematically illustrates a flowchart of an exemplary dual filtering algorithm of the prior art.

Figure 4 illustrates an exemplary coding of a dual filtering algorithm of the prior art.

Figure 5 schematically illustrates a first exemplary embodiment of a bloom effect algorithm.

Figures 6A and 6B schematically illustrate the dual filtering algorithm of the first exemplary embodiment of a bloom effect algorithm. Figure 6A schematically illustrates down-sampling and up-sampling passes of the first exemplary embodiment of a bloom effect algorithm. Figure 6B schematically illustrates a sampling pattern used to generate a mask texture of the first exemplary embodiment of a bloom effect algorithm.

Figures 7A and 7B illustrate an exemplary coding of the first exemplary embodiment of a bloom effect algorithm. Figure 7A illustrates an exemplary coding of generating a mask texture of the first exemplary embodiment of a bloom effect algorithm. Figure 7B illustrates an exemplary coding of an up-sampling pass of the first exemplary embodiment of a bloom effect algorithm.

Figures 8A-8C schematically illustrate the steps an image goes through during the first exemplary embodiment of a bloom effect algorithm. Figure 8A illustrates an input image. Figure 8B schematically illustrates a mask texture generated from the input image. Figure 8C illustrates an output image.

Figure 9 shows an example of a computer implemented method for generating bloom effect of the first exemplary embodiment of a bloom effect algorithm.

Figure 10 shows an example of an apparatus configured to perform the methods described herein.

Figure 11 schematically illustrates a second exemplary embodiment of a bloom effect algorithm.

Figure 12 graphically illustrates a weighted sum distribution for a cubic B-spline filtering algorithm of a second exemplary embodiment of a bloom effect algorithm.

Figure 13 schematically illustrates a texel-area of the second exemplary embodiment of a bloom effect algorithm.

Figure 14 graphically illustrates a comparison between a Gaussian filter kernel and a bicubic B-spline filter kernel.

Figures 15A-15D illustrate a first output image of the second exemplary embodiment of a bloom effect algorithm compared to a corresponding prior art first output image for the same input image. Figure 15A illustrates the first output image of the prior art. Figure 15B illustrates the first output image of the second exemplary embodiment of a bloom effect algorithm. Figure 15C illustrates a zoomed section of the first output image of the prior art. Figure 15D illustrates a zoomed section of the first output image of the second exemplary embodiment of a bloom effect algorithm.

Figures 16A-16C illustrate a second output image of the second exemplary embodiment of a bloom effect algorithm compared to a corresponding prior art second output image for the same input image. Figure 16A illustrates the input image. Figure 16B illustrates the second output image of the prior art. Figure 16C illustrates the second output image of the second exemplary embodiment of a bloom effect algorithm.

Figure 17 shows an example of a computer implemented method for generating bloom effect of the second exemplary embodiment of a bloom effect algorithm.

Figure 18 shows an example of a computer implemented method for generating bloom effect of the third exemplary embodiment of a bloom effect algorithm.

DETAILED DESCRIPTION

The apparatuses and methods described herein concern generating a bloom effect on an input image.

Embodiments of the present invention may tackle one or more of the problems previously mentioned by generating an output bloom effect image in dependence on the up-sampled image region. Additionally, embodiments of the present invention may tackle one or more of the problems previously mentioned problems by selecting a region of one or more pixels of the input image in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter. In this way, it may be possible to enable the bloom effect algorithm to reduce the number of sampling points to be up-sampled, which may reduce the computational load reducing the power consumption and rendering latency.

An image blurring algorithm may play an important role in generating a bloom effect in the image post-processing stage. In fact, many product-level post-processing implementations depend on image blurring algorithms. The blurring algorithms may include god ray based on radial blur, lens flare based on directional blur and bokeh blur in depth of field effects.

Gaussian blur is a known blurring algorithm. Gaussian blur has become synonymous for a blurring algorithm. Dual blur is also a known blurring algorithm which is known to achieve better performance.

Gaussian blur may be used to reduce high frequency image noise and details to produce an output image that looks blurry. The visual effect on the user, or viewer, may be similar to viewing images through a translucent screen. From the point of view of digital signal processing, the essence of image blur is a process of removing high-frequency signals and retaining low-frequency signals.

Figure 5 schematically illustrates an overview of a first exemplary embodiment of a bloom effect algorithm 500. The bloom effect algorithm 500 may be carried implemented by the image processing apparatus. The bloom effect algorithm 500 may be used to generate an output bloom effect image 506.

The apparatus may obtain an input image 501. The term obtain may include generate or receive. If the input image 501 is received, it may be received from a separate apparatus which generates the input image, or the input image 501 may be received from within the apparatus.

A down-sampled

image

502, 503 may be generated from the input image 501. The down-sampled

image

502, 503 may be generated using a down-sampling stage, or pass, indicated at 507. Figure 5 shows two down-

sampling stages

507, 508. The algorithm 500 may comprise two or more down-sampling stages. The further down-sampling stage 508 may provide a further down-sampled image 503. The algorithm 500 may suitably provide more than two down-

sampling stages

507, 508, for example four down-sampling stages, as shown in Figure 6 (described in more detail herein) . The term further down-sampled image 503 may be interchangeable with the down-sampled image 502. The term further down-sampling stage 508 may be interchangeable with the down-sampling stage 507.

By using more than one down-

sampling stage

507, 508, this may increase the magnitude of the bloom effect. In other words, the bloom effect, or blur halo, may diffuse further from the brighter region in the input image 501. By varying the number of down-sampling stages, this may enable the magnitude of the bloom effect to be varied.

The algorithm 500 may generate a mask texture 504 from the down-sampled image 503 in dependence on a brightness of the pixels. The mask texture 504 may provide an indication to designate the non-empty texels in 503. The brightness may suitably be measured based on the illuminance level, grey level, colour level or another brightness indicator level of the pixel. Preferably, the brightness is measured based on the colour level of the pixel.

Preferably, the algorithm 500 selects the region of one or more pixels of the down-sampled image 503 by using a pixel minimum brightness threshold filter. By using a pixel minimum brightness threshold filter to select the region of one or more pixels, this may enable the selected region to only comprise pixel which are bright. In this way, all the following filtering operations, such as upsampling and bloom effect, may only need to be applied to the brighter regions of an input image, or the image from the previous step.

The algorithm may select the region of one or more pixels of the down-sampled image 503 by masking the down-sampled image region with a mask. Preferably, the algorithm 500 generates the up-sampled image 505 by using only portions of the down-sampled image 503 designated by the mask texture 504. In other words, the empty texels designated by the mask texture 504 may be avoided when filtering the down-sampled image 503, so that generating the up-sampled image 505 does not need to fetch or filter those empty texels designated by the mask texture 504.

By masking the down-sampled image region with a mask texture 504, this may enable the apparatus to clearly distinguish between the region to receive the bloom effect or not. In this way, the area surrounding the mask is skipped for filtering, and hence does not have the bloom effect applied. This may significantly reduce the number of sampling points, depending on the size of the region of pixels, to which the blur diffusion algorithm is applied to. This may further reduce the computational load reducing the power consumption and rendering latency.

The algorithm 500 may generate an up-sampled image region from the down-sampled image region 504. The up-sampled image region may be generated using an up-sampling stage, or pass, 511, 512. Figure 5 shows two up-

sampling stages

511, 512. The algorithm 500 may comprise two or more up-sampling stages. The further up-sampling stage 512 may provide a further up-sampled image region. The algorithm 500 may suitably provide more than two up-

sampling stages

511, 512, for example four up-sampling stages, as shown in Figure 6. The term further up-sampled image region may be interchangeable with the up-sampled image region. The term further up-sampling stage 512 may be interchangeable with the up-sampling stage 511.

By using more than one up-

sampling stage

511, 512, this may increase the magnitude of the bloom effect. In other words, the bloom effect, or blur halo, may diffuse further from the brighter region in the input image 501. By varying the number of up-

sampling stages

511, 512, this may enable the magnitude of the bloom effect to be varied.

The algorithm 500 may generate an output bloom effect image 506 in dependence on the up-sampled image region. In other words, the up-sampled image region may provide the bloom effected region to produce the output bloom effect image 506.

Generating the output bloom effect image 506 may use significant computational loading. Additionally, using a dual filtering method, down-

sampling stages

507, 508 followed by up-sampling, may further increase the computational loading. By increasing the computational loading this may cause increased power consumption and a poorer latency of the processors. Power consumption can be a significant issue for the user, particularly if the apparatus is implemented on a mobile device, with a portable power supply. Additionally, poor latency can be issue, particularly if the input image is a frame image of a video or video game which may require a high-speed frame rendering rate.

By generating an up-sampled image region from the down-sampled image region 504, and generating an output bloom effect image 506 in dependence on the up-sampled image region it may be possible to enable the bloom effect algorithm to reduce the number of sampling points to be up-sampled, which may reduce the computational load reducing the power consumption and rendering latency.

Preferably, the algorithm 500 generates the output bloom effect image 506 by applying a blur diffusion algorithm only to the up-sampled image region. The region of pixels may only comprise a portion of the overall image, as illustrated by 104 in Figure 1B. For example, the region of pixels may comprise 10%of the image. By only applying the blur diffusion algorithm to the up-sampled image region, and not the whole of the image, this may significantly reduce the number of sampling points, depending on the size of the region of pixels, to which the blur diffusion algorithm is applied to. This may further reduce the computational load reducing the power consumption and rendering latency.

The algorithm 500, after generating the down-sampled image 503 from the input image 501, may also generate an up-sampled

image

505, 506 from the down- sampled image 503. The up-sampled

image

505, 506 may be generated using an up-sampling stage, or pass, 509. Figure 5 shows up down-

sampling stages

509, 510. The algorithm 500 may comprise two or more up-sampling stages. The further up-sampling stage 509 may provide a further up-sampled image 506. The algorithm 500 may suitably provide more than two up-

sampling stages

509, 510, for example four up-sampling stages, as shown in Figure 6. The term further up-sampled image 506 may be interchangeable with the up-sampled image 505. The term further down-sampling stage 510 may be interchangeable with the down-sampling stage 509.

By using more than up-

sampling stage

509, 510, this may increase the magnitude of the bloom effect. In other words, the bloom effect, or blur halo, may diffuse further from the brighter region in the input image 501. By varying the number of up-

sampling stages

509, 510, this may enable the magnitude of the bloom effect to be varied.

By generating the up-sampled

image

505, 506 from the down-sampled image 503, this may enable the up-

sampling pass

509, 510 to output both the up-sampled

image

505, 506 and the up-sampled image region. In this way, the output bloom effect image 506 may comprise the entire image, including the up-sampled

image

505, 506 and the up-sampled image region.

The algorithm 500 may generate the output bloom effect image in dependence on the up-sampled image region and the up-sampled

image

505, 506. In this way, this may enable the output bloom effect image 506 to incorporate features from both the up-sampled image 506 and the up-sampled image region.

In more detail, the algorithm may generate the output bloom effect image by combining the up-sampled image region and the up-sampled

image

505, 506. Preferably, the algorithm combines the up-sampled image region and the up-sampled

image

505, 506 by overlaying the up-sampled image region onto a corresponding region of the up-sampled

image

505, 506.

Figure 5 shows that the algorithm 500 may combine the up-sampled image 505 with the up-sampled image region using the up-

sampling stages

509 and 511. Additionally, the algorithm 500 may combine the further up-sampled image 506, with the further up-sampled image region using the up-

sampling stages

510 and 512. The up-sampling stage 510 may start at the up-sampled image 505 and the up-sampling stage 512 may start at the down-sampled image region 504. In other words, the up-sampled

image

505, 506 may be combined with the up-sampled image region after each up-sampling stage.

By overlaying the up-sampled image region onto a corresponding region of the up-sampled

image

505, 506, this may enable the up-sampled image region to be positioned onto the correct position on the up-sampled

image

505, 506. In other words, the blur halo may fit around the brighter region in the up-sampled

image

505, 506 to generate the bloom effect.

The algorithm 500 may repeat the steps shown in Figure 5 and described herein using the output bloom effect image 506 as an input image 501 for one or more iterations. By re-inputting the output bloom effect image 506 into the apparatus this may further increase the magnitude of the bloom effect. In other words, the bloom effect, or blur halo, may diffuse further from the brighter region in the input image 501. By varying the number of iterations, this may enable the magnitude of the bloom effect to be varied.

The first exemplary embodiment will now be described in more detail.

Bloom effect may be generated in multiple passes by first extracting the brightest part of the rendered image by applying a brightness threshold to each input pixel, so that only the pixels whose brightness is higher than the threshold needs to be blurred. The blurred result of the bright regions may be composited back onto the original input image to achieve the desired bloom effect, in which the dark regions will not be blurred. The blur passes may be expensive to compute which may slow the rending process.

As discussed herein, the dual filtering algorithm 303 may carry out a series of down-sampling and up-sampling passes to the input image. The dual filtering algorithm may use different filtering kernels, or stencils, for the down-sample and up-sample passes, as shown in Figures 3A and 3B. By adjusting the number of iterative passes the diffusion distance may be adjusted from the bright source pixels to their surrounding target pixels in order to produce a bigger or smaller blurring effects.

The dual filtering algorithm 303 may still have a large bandwidth cost on mobile GPUs. In particular, each of the up-sampling passes, which may use an eight texture fetch up-sample filter 302 for each output pixel, as shown in Figure 3B, may require significant bandwidth. Since there are more sampling points involved in the up-sampling stencil 302 than the down-sampling stencil 301, up-sampling may have a greater impact on performance and bandwidth than down-sampling. The inventors aimed to produce a bloom effect algorithm that optimises the dual filtering algorithm.

As shown in Figure 5, the first exemplary embodiment of the bloom effect algorithm may use a series of down-

sampling

507, 508 and up-

sampling

509, 510, 511 iterative passes. In the exemplary embodiment shown in Figure 5 the algorithm carries out two down-sampling passes 507, 508 and two up-sampling passes 509, 510, 511. The number of down-sampling passes 507, 508 and up-sampling passes 509, 510, 511 may be varied depending on the magnitude of the required bloom effect. An increased magnitude of bloom effect may generally require more passes.

A mask texture 504 may be generated based on the final down-sampled output image 503. The mask texture 504 may be utilised to increase the speed and efficiency of the upsampling stage to generate the output image 506, shown by 511. The mask texture 504 may also be utilised to increase the speed and efficiency of the upsampling stage to generate the intermediary up-sampled output image 505, shown by 510. The remainder of the output down-sampled image 503 may then be up-sampled to the intermediary up-sampled output image 505, shown by 509. The generated mask texture 504 may be used to speed up all the up-sampling passes.

As shown in Figure 5, at each down-

sampling pass

507, 508, the output texture size may decrease continuously by two times along both width and height dimensions. During the up-

sampling iterations

509, 510, 511, the texture size may increase continuously by two times along both width and height dimensions. For each up-

sampling pass

509, 510, 511 the halo diffusion range per input texels may be twice that of the output texels. In other words, the size in the normalized texture space of each texture may be in the range of 0 to 1. The actual distance of halo diffusion may become smaller as the output texture of each up-sampling pass is enlarged. The texture dimensions may enlarged by two times, but the distance per texel in normalized texture space may be 1/width and 1/height along X and Y dimensions, respectively.

Figures 6A and 6B further schematically illustrate the dual filtering algorithm 500 of the first exemplary embodiment of a bloom effect algorithm 500. Figure 6A schematically illustrates down-

sampling

603, 604, 605, 606 and up-sampling passes 607, 608, 609, 610 of the first exemplary embodiment of a bloom effect algorithm 500. Figure 6B schematically illustrates a sampling pattern 611 used to generate a mask texture of the first exemplary embodiment of a bloom effect algorithm 500 from the last down-sampled texture 606.

The last down-sampling pass 508 may generate the texture with the minimum size, shown as the texture 606 in Figure 6B. The diffusion distance covered per pixel in the texture 606 may be the largest. This means that the normalized pixel distance in the texture 606 may twice that of the texture 607, four times that of the texture 608, and so on. Using one single pixel distance in the texture 606 as the reference unit, the maximum diffusion range of one pixel in the texture 606, after all the up-sampling steps, may be calculated as follows: 1 + 1/2 + 1/4 + 1/8 +... < 2. The final halo diffusion range in the texture 610 may be less than two-pixel distance in the texture 606 in normalized texture space.

Figure 6B shows an exemplary embodiment of how mask texture 504 may be generated from the last down-sampled texture 606 with the minimum size. In this pass, both the input 606 and the output mask texture 504 may have the same pixel resolution. For each output pixel 612. The algorithm may perform four bilinear texture sampling at the four positions 613, which have the subpixel offsets of +/-0.5 pixel-size (i.e. a half pixel distance along both dimensions) from the centre of the output pixel 612.

Due to the bilinear texture sampling feature, the resulting colour for each sampling point 613 may be calculated as a weighted average of the four nearby pixel. If any of these four pixels comprise an RGB colour value greater than 0, then the resulting colour value of the bilinear sampling will be greater than 0. In this way, the algorithm may collect the final resulting value by summing up all the four bilinear sampling colour values at the four sampling points 613, and further summing up the final RBG values together into a single scalar sum, and check whether the final sum is greater than 0 or not. If the final sum is greater than 0, which means the 3x3=9 input pixels in the neighbourhood of the current pixel 612 must have at least one non-empty pixel, so the algorithm should output 1 as the final value of the current pixel for this single-channel mask texture. Otherwise (i.e. if the final sum is not greater than 0) , the algorithm will output 0 as the final value of the current pixel 612, which means that the 3x3=9 input pixels in the neighbourhood of the current pixel 612 are all empty pixels.

Figures 7A to 7C illustrate an exemplary coding of the first exemplary embodiment of a bloom effect algorithm. Figure 7A illustrates an exemplary coding of generating a mask texture of the first exemplary embodiment of a bloom effect algorithm. Figure 7C illustrates an exemplary coding of an up-sampling pass of the first exemplary embodiment of a bloom effect algorithm, which uses this mask texture to avoid filtering the empty region of the input image.

The GPU shader code 701 for generating the mask texture may be written as shown in Figure 7A.

After the mask texture 504 is generated, at each up-

sampling pass

509, 510, 511, before performing any blur filtering operations, the algorithm may perform a bilinear sampling to mask texture 504 to check if the current pixel’s neighbourhood has any non-empty pixels or not. As a result, the algorithm may avoid performing a costly blur filtering for those empty regions which do not need any halo diffusion.

Due to the nature of the up-sampling passes, a region with a 3x3 pixel-neighbourhood, as shown in Figure 6B, at an input texture (e.g., texture 606 in Figure 6A) is equivalent to a region with a 6x6 pixel-neighbourhood after the up-sampling pass (texture 607 in Figure 6A) . Therefore, an optimized up-sampling shader may be modified in which the fourth line of the code 702 may perform a test whether the mask is smaller than a very small threshold (0.001) , if so, an early exit can be performed to avoid all the eight followed texture fetches and their corresponding calculations.

As the halo regions may be sparcily arranged, most pixels may be directly returned, which is indicated by the mask texture testing in the code 702. Additionally, the mask texture 504 may be set to a small single channel color format (e.g. 8-bit per pixel) to further reduce bandwidth consumption.

Figures 8A-8C schematically illustrate the steps an image goes through during the first exemplary embodiment of a bloom effect algorithm. Figure 8A illustrates an input image 801. Figure 8B schematically illustrates a mask texture 804 generated from the input image 801. Figure 8C illustrates an output image 805 produced with the assistance of the mask texture.

The computational workload and bandwidth consumption for filtering the non-halo, or dark, diffusion region of the input image 801 may be avoided by filtering only the halo, or bright region, 802 of the input image 801. As shown in the following Figure 8A, the input image 801 may comprise a bright region 802, or a plurality of bright regions 802. Figure 8A shows an exemplary input image 801 used for analysis. The input image 801 may comprise bright regions 802 with different colours, shapes and sizes with a dark background, as shown in Figure 8A.

A mask texture 803 may be obtained by calculating the diffusion range of each bright region 801. The diffusion range of each bright region 801 may be marked with the mask texture 803. Once the mask texture 803 is generated from the input image 801 this may produce a masked non-empty texels 804. As a result of obtaining this mask texture 803, when filtering pixels in the dark background area, which can be indicated by the regions surrounding the non-empty texels 804 in the mask texture 803, the filtering function may be carried out without the need to perform any texture-sampling-based filtering operation for the empty texels 804. This may reduce the need for which memory-bandwidth heavy texture fetches and arithmetic logic unit (ALU) heavy weighted filtering calculation.

As the mask texture 803 is applied after all the down-sampling stages, as shown in Figure 5, the resolution of the mask texture 803 is often significantly smaller than the original input image 801. In most cases, this may be only 1/128 or 1/256 of the original input image. This smaller mask texture 803 is sampled and tested , which may enable a significant reduction in the bandwidth usage and increase performance.

The first exemplary embodiment of the bloom effect algorithm has been compared against the NetEase game’s bloom algorithm on a Huawei P40 phone.

In the preferred implementation, both algorithms involve four down-sampling passes and four up-sampling passes. Both algorithms use a 4-tap bilinear-box-filter for these down-sampling passes and up-sampling passes.

The first exemplary embodiment of the bloom effect algorithm differs from the prior method as it uses a mask texture to avoid blurring calculations for empty pixels in the input image, which has been thresholded as black colour.

In this comparison, the first exemplary embodiment of the bloom effect algorithm was found to provide better performance. For example, in some implementations, its use resulted in an FPS Performance gain of 35.39%, a power gain of 6.34%and the same image quality as previous methods.

Figure 9 summarises an example of a method 900 for generating a bloom effect. At step 901, the method comprises obtaining an input image. At step 902, the method comprises generating a down-sampled image from the input image. At step 903, the method comprises selecting a region of one or more pixels of the down-sampled image in dependence on a brightness of the pixels. At step 904, the method comprises generating an up-sampled image region from the down-sampled image region. At step 95, the method comprises generating an output bloom effect image in dependence on the up-sampled image region.

An example of an apparatus 1000 configured to implement the method 900 is schematically illustrated in Figure 10. The apparatus 1000 may be implemented on an electronic device, such as a laptop, tablet, smart phone or TV. In particular, the apparatus 1000 may be implemented using Vulkan, OpenGL, or otherApplication Programming Interfaces (APIs) and may be used on both PC and mobile devices.

The apparatus 1000 comprises a processor 1001 configured to process the datasets in the manner described herein. For example, the processor 1001 may be implemented as a computer program running on a programmable device such as a Central Processing Unit (CPU) . The apparatus 1000 comprises a memory 1002 which is arranged to communicate with the processor 1001. Memory 1002 may be a non-volatile memory. The processor 1001 may also comprise a cache (not shown in Figure 5) , which may be used to temporarily store data from memory 1002. The apparatus may comprise more than one processor and more than one memory. The memory may store data that is executable by the processor. The processor may be configured to operate in accordance with a computer program stored in non-transitory form on a machine readable storage medium. The computer program may store instructions for causing the processor to perform its methods in the manner described herein.

Figure 11 schematically illustrates an overview of a second exemplary embodiment of a bloom effect algorithm 1100. The bloom effect algorithm 1100 may be carried out on the image processing apparatus 1000. The bloom effect algorithm 1100 may be used to generate an output bloom effect image 1103.

The apparatus may obtain an input image 1101. The term obtain may include generate or receive. If the input image 1101 is received, it may be received from a separate apparatus which generates the input image, or the input image 501 may be received from within the apparatus.

The algorithm 1100 may select a region of one or more pixels of the input image 1101 in dependence on a brightness of the pixels. The selection of the region may provide an filtered image region 1102. The brightness may suitably be measured based on the illuminance level, grey level, colour level or another brightness indicator level of the pixel. Preferably, the brightness is measured based on the colour level of the pixel.

The algorithm 1100 may select the region of one or more pixels of the input image 1101 by using a pixel minimum brightness threshold filter. By using a pixel minimum brightness threshold filter to select the region of one or more pixels, this may enable the selected region to only comprise pixel which are bright. In this way, the bloom effect may only be applied to the brighter regions of the input image 1101.

Preferably, the one or more pixels of the filtered image region 1102 are filtered by means of a basis-spline filter algorithm.

Th algorithm may generate an output bloom effect image 1103 in dependence on the filtered image region 1102. In other words, the filtered image region 1102 may provide the bloom effected region to produce the output bloom effect image 1103.

Generating the output bloom effect image 1103 may use significant computational loading. By increasing the computational loading this may cause increased power consumption and a poorer latency of the processors. Power consumption can be a significant issue for the user, particularly if the apparatus is implemented on a mobile device, with a portable power supply. Additionally, poor latency can be issue, particularly if the input image is a frame image of a video or video game which may require a high-speed frame rendering rate.

By selecting a region of one or more pixels of the input image 1101 in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm, and generating an output bloom effect image in dependence on the filtered image region, it may be possible to enable the bloom effect algorithm to reduce the number of sampling points to be up-sampled, which may reduce the computational load reducing the power consumption and rendering latency.

The basis-spline filter algorithm may comprise a cubic basis spline filter algorithm. By using a cubic basis spline filter algorithm this may comprise a third order approximation. The third order approximation may be more accurate than a lower order approximation, but may be less accurate than a higher order approximation. Additionally, the third order approximation may require more sampling points than a lower order approximation, but may require less sampling points than a higher order approximation. The inventors have found that the third order approximation may provide a suitable level of accuracy while not requiring an excessive number of sampling points which may increase the computational loading.

The algorithm 1101 may generate the output bloom effect image by applying a blur diffusion algorithm to the filtered image region. The region of pixels may only comprise a portion of the overall image 1101. For example, the region of pixels may comprise 10%of the image 1101. By only applying the blur diffusion algorithm to the filtered image region 1102, and not the whole of the image 1101, this may significantly reduce the number of sampling points, depending on the size of the region of pixels, to which the blur diffusion algorithm is applied to. This may further reduce the computational load, reducing the power consumption and rendering latency.

The algorithm 1100 may generate the output bloom effect image 1103 in dependence on the filtered image region 1102 and the input image 1101. In this way, this may enable the output bloom effect image 1103 to incorporate features from both the input image 1101 and the filtered image region 1102.

In more detail, the algorithm 1100 may generate the output bloom effect image by combining the filtered image region 1102 and the input image 1101. Preferably, the algorithm 1100 combines the filtered image region 1102 and the input image 1101 by overlaying the filtered image region 1102 onto a corresponding region of the input image 1101.

By overlaying the filtered image region 1102 onto a corresponding region of the input image 1101, this may enable the filtered image region 1102 to be positioned onto the correct position on the input image 1101. In other words, the blur halo may fit around the brighter region in the input image 1101 to generate the bloom effect.

The algorithm 1100 may repeat the steps shown in Figure 11 and described herein using the output bloom effect image 1103 as an input image 1101 for one or more iterations. By re-inputting the output bloom effect image 1103 into the apparatus this may further increase the magnitude of the bloom effect. In other words, the bloom effect, or blur halo, may diffuse further from the brighter region in the input image 1101. By varying the number of iterations, this may enable the magnitude of the bloom effect to be varied.

The second embodiment will now be described in more detail.

Figure 12 graphically illustrates a weighted sum distribution 1200 for a cubic B-spline filtering apparatus of a second exemplary embodiment of a bloom effect algorithm.

A cubic B-spline filter 1200 may be used in image processing. The cubic B-spline filter 1200 may be applied in both x and y directions. This may be known as a bicubic B-spline filter 1200. The cubic B-spline filter 1200 is a cubic filter kernel with cubic polynomial weights, as shown in the Figure 12.

In Figure 12, the y value of the function is the relative weight which may be assigned to the texels that are distant from the centre of a given texture sampling coordinate x. Any texels that are positioned more than two texels from that centre may be ignored due to a zero weight value. Texels at the centre may be given highest weight. In particular, for cubic B-spline filter shown in Figure 12, all the weights are positive, i.e. greater than zero.

In 1D, the cubic B-spline filtering algorithm may be expressed by Equation 1:

f (x) = w0 (x) f _i-1 + w1 (x) f _i + w2 (x) f _i+1 + w3 (x) f _i+2 (1)

Where f _i is the indexed neighbouring texel value at 4 taps of integer sampling locations, which are multiplied by the corresponding cubic polynomial weights w _i (x) from the convolution kernel. The weighted sum is the final result of the filtering.

The general 1D cubic interpolation is a method for estimating a specific function value f (x) at an arbitrary continuous sampling point x by calculating the weighted sum of 4 taps of known functional values f _i-1, f _(i) , f _(i+1) and f _(i+2) at 4 integer grid locations (from i-1 to i+2) , where x = i + α, α ∈ [0, 1) , i.e., 1> α ≥ 0, and i ∈ Z being the integer and fractional parts of x, respectively.

The formula to calculate the B-spline weights using a third-order polynomial may be expressed by Equations 1, 2 and 3:

Where the four weights are determined by the fractional amount α of the present sampling coordinate x.

The bicubic filter may be a 2D extension of the 1D cubic filtering for interpolating data points on a 2D regular grid. The 2D interpolation function may be a separable extension of the 1D interpolation function. The 2D interpolation may be accomplished by two 1D interpolations with respect to each coordinate direction. Therefore,

Equations

1 and 2 may be used to produce the filter weights along both X and Y directions.

Figure 13 schematically illustrates a texel-area of the second exemplary embodiment of a bloom effect algorithm for a 2D cubic B-spline filtering.

The cubic B-spline filter in 2D may require a filter of 4x4 texel-area with 16 texture fetches, and with third-order weighted filtering calculation along x and y direction in a 4x4 local neighbourhood. This may involve 32 multiplications for a single pixel shading, as shown in the Figure 13. This may involve sampling a 4x4 grid of texels surrounding the target UV coordinate 1202 (shown by the cross) .

The filter may be simplified by utilising a GPU hardware bilinear sampling feature. This may mean that the algorithm needs only 4 taps of texture fetches to achieve the same filtering result for a 4x4 texel-area. This may effectively merge the sixteen sampling taps down to only four.

This result may be achieved by carefully tweaking the sampling coordinates. For 1D 1D cubic filtering, to merge the 4 taps down to 2, the algorithm may combine the first pair of taps into a linear tap and the second pair into another linear tap. Therefore, rather than perform two texture lookups at f _i and f _i+1, and then perform a linear combination a *f _i + b *f _i+1 with general a and b, the algorithm may perform a single texture lookup at i + b/ (a+ b) and simply multiply the result by (a+ b) . Consequently, a 2: 1 reduction may become a 4: 1 reduction in 2D. This may effectively take 16 taps down to 4 in 2D by using

Equations

4, 5, and 6.

Figure 14 graphically illustrates a comparison 1400 between the distributions of a Gaussian filter kernel 1402 and a bicubic B-spline filter kernel 1401 in 1D.

The inventors have found that the shape of the distribution of a Gaussian blur filter kernel 1402 and a cubic B-spline filter kernel 1401 may be similar, as shown in Figure 14. This has also been found to result in very similar blurring output from the bloom effect algorithm.

The B-spline filter algorithm 1401 has, for the reasons described herein, has also been found to be easier to calculate, in terms of computational loading, than the Gaussian blur algorithm 1402. This may be due to the reduction in sampling points. In particular, if the B-spline filter algorithm 1401 is applied to a dual filter algorithm, the effect may be magnified due to the increased number of down-sampling and up-sampling passes inherently increasing the number of sampling points. Therefore, the use of a cubic B-spline kernel 1401, instead of a Gaussian blur filter kernel 1402 may reduce the computational loading on the GPU while also maintaining an acceptable blurring output.

Preferably, the second exemplary embodiment of the algorithm may comprise a 4-tap bicubic B-spline kernel to replace the 8-tap bilinear up-sampling involved in the original dual filtering algorithm. As a result, the algorithm may only need four taps of texture fetches in the pixel neighbourhood to approximate the weighted sum of the eight sampling points for the up-sampling passes. This may reduce the GPU memory bandwidth for texture sampling.

The second exemplary embodiment of the bloom effect algorithm has been compared against the NetEase game’s bloom algorithm by using only one up-sampling pass on a Huawei Matebook laptop.

In this experiment, the number of down-sampling and up-sampling passes involved for both algorithms were the same. There were 4 down-sampling passes and one up-sampling pass. For both algorithms the mask texture of the first exemplary embodiment of the bloom effect algorithm was modified to avoid blurring calculations for the empty pixels in order to achieve better performance.

However, the second exemplary embodiment of the bloom effect algorithm used a bicubic B-Spline filter, while the NetEase game used 4-tap-bilinear-box-filter for the single up-sampling pass.

The second exemplary embodiment of the bloom effect algorithm may in some implementations provide better performance. For example, it may in some implementations result in a FPS improvement and improved bloom effect quality (as shown in Figures 15A-15B and 16A-16C) .

Figures 15A-15D illustrate a first output image 1503 of the second exemplary embodiment of a bloom effect algorithm compared to a corresponding prior art first output image 1501 for the same input image. Figure 15A shows the output image 1501 of the NetEase game algorithm. Figure 15B shows the output image 1503 of the second exemplary embodiment of a bloom effect algorithm. Figure 15C shows a zoomed section of the output image 1501 of the NetEase game algorithm. Figure 15D shows a zoomed section of the output image 1503 of the second exemplary embodiment of the bloom effect algorithm. The

output images

1501, 1503 comprise output

image bloom regions

1502, 1504. The output image bloom regions 1504 of the second exemplary embodiment of a bloom effect algorithm comprise a smoother bloom effect with less diamond shaped artifacts than the output image bloom regions 1502 of the NetEase game algorithm and thus a better quality bloom effect.

Figures 16A-16C illustrate a second output image 1605 of the second exemplary embodiment of a bloom effect algorithm compared to a corresponding prior art second output image 1603 for the same input image. Figure 16A illustrates the input image 1601. Figure 16B illustrates the second output image 1603 of the prior art. Figure 16C illustrates the second output image 1603 of the second exemplary embodiment of a bloom effect algorithm. Figures 16A-16C illustrate the same test as Figures 15A-15B with different shapes for the output

image bloom regions

1604, 1606. Again, the output image bloom regions 1606 of the second exemplary embodiment of a bloom effect algorithm compress a smoother bloom effect with less diamond shaped artifacts than the output image bloom regions 1602 of the NetEase game algorithm and thus a better quality bloom effect.

Table 1 shows the test data for the second output image 1605 of the second exemplary embodiment of a bloom effect algorithm compared to the corresponding prior art second output image 1603.

Table 1

In this exemplary implementation, the new algorithm's performance improved by 20.8%= (238-197) /197. The memory bandwidth was improved by up to 50%for the up-sampling stage and the rendering quality was similar to that achieved using previous techniques.

Figure 17 summarises an example of a method 1700 for generating a bloom effect. At step 1701, the method comprises obtaining an input image. At step 1702, the method comprises selecting a region of one or more pixels of the input image in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter. At step 1703, the method comprises generating an output bloom effect image in dependence on the up-sampled image region.

The method 1700 may be implemented on the apparatus 1000 illustrated in Figure 10 and described herein.

The first exemplary embodiment of a bloom effect algorithm 500 may be combined with the second exemplary embodiment of a bloom effect algorithm 1100 to form a third exemplary embodiment of a bloom effect algorithm. In other words, the algorithm 500 described in relation to Figure 5 may be combined with the algorithm 1100 described in relation to Figure 11.

In more detail, the algorithm may obtain an

input image

501, 1101; generate a down-sampled

image

502, 503 from the

input image

501, 1101; select a region of one or more pixels of the down-sampled image 503 in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm; generate an up-sampled image region from the down-sampled image region 504; and generate an output

bloom effect image

506, 1103 in dependence on the up-sampled image region.

The image processing apparatus aims to select a region of one or more pixels with particular brightness level and generate a

bloom effect image

506, 1103 in which the selected region receives the bloom effect.

Generating the output

bloom effect image

506, 1103 may use significant computational loading. Additionally, using a dual filtering method, down-sampling followed by up-sampling, may further increase the computational loading. By increasing the computational loading this may cause increased power consumption and a poorer latency of the processors. Power consumption can be a significant issue for the user, particularly if the apparatus is implemented on a mobile device, with a portable power supply. Additionally, poor latency can be issue, particularly if the input image is a frame image of a video or video game which may require a high-speed frame rendering rate.

By generating an up-sampled image region from the down-sampled image region 504, and generating an output

bloom effect image

506, 1103 in dependence on the up-sampled image region and wherein the one or more pixels are filtered by means of a basis-spline filter algorithm, it may be possible to enable the bloom effect algorithm to reduce the number of sampling points to be up-sampled, which may reduce the computational load reducing the power consumption and rendering latency.

In particular, the combination of the selective up-sampling of the brighter image region, of the first exemplary embodiment, and by carrying out the filtering with a basis-spline filter algorithm, of the second exemplary embodiment, may further reduce the number of sampling points to be up-sampled.

Table 2 shows the test data for an output image of the third exemplary embodiment of a bloom effect algorithm compared to the corresponding prior art output image.

Table 2

In this exemplary implementation, there was further optimization to the algorithm's performance, demonstrating an improvement of 66.5%= (328-197) /197.

Figure 18 summarises an example of a method 1800 for generating a bloom effect. At step 1801, the method comprises obtaining an input image. At step 1802, the method comprises generating a down-sampled image from the input image. At step 1803, the method comprises selecting a region of one or more pixels of the down-sampled image in dependence on the brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm. At step 1804, the method comprises generating an up-sampled image region from the down-sampled image region. At step 1805, the method comprises generating an output bloom effect image in dependence on the up-sampled image region.

The method 1800 may be implemented on the apparatus 1000 illustrated in Figure 10 and described herein.

The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

An image processing apparatus (1000) for generating a bloom effect, the apparatus (1000) comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code, when executed by the one or more processors, causes the image processing apparatus (1000) to:

obtain an input image (501) ;

generate a down-sampled image from the input image (502, 503) ;

select a region of one or more pixels of the down-sampled image (502, 503) in dependence on a brightness of the pixels;

generate an up-sampled image region from the down-sampled image region (504) ; and

generate an output bloom effect image (506) in dependence on the up-sampled image region.
The apparatus (1000) according to claim 1, wherein generating the output bloom effect image (506) comprises applying a blur diffusion algorithm only to the up-sampled image region.
The apparatus (1000) according to claim 1 or 2, wherein after generating the down-sampled image (503) from the input image (501) , the apparatus (1000) generates an up-sampled image (505) from the down-sampled image (503) .
The apparatus (1000) according to claim 3, wherein generating the output bloom effect image (506) is in dependence on the up-sampled image region and the up-sampled image (505) .
The apparatus (1000) according to claim 4, wherein generating the output bloom effect image (506) comprises combining the up-sampled image region and the up-sampled image (505) .
The apparatus (1000) according to claim 5, wherein combining the up-sampled image region and the up-sampled image (505) comprises overlaying the up-sampled image region onto a corresponding region of the up-sampled image (505) .
The apparatus (1000) according to any preceding claim, wherein selecting the region of one or more pixels of the down-sampled image (503) comprises using a pixel minimum brightness threshold filter.
The apparatus (1000) according to any preceding claim, wherein selecting the region of one or more pixels of the down-sampled image (503) comprises generating a mask from the down-sampled image region (504) .
The apparatus (1000) according to claim 8, wherein generating the up-sampled image (505) comprises using only portions of the down-sampled image (503) designated by the mask (504) .
The apparatus (1000) according to any preceding claim, wherein generating the down-sampled image (503) from the input image (501) comprises using two or more down-sampling stages (507, 508) .
The apparatus (1000) according to any preceding claim, wherein generating the up-sampled image region from the down-sampled image region (504) comprises using two or more up-sampling stages (511, 512) .
The apparatus (1000) according to any of claims 3 to 11, wherein generating the up-sampled image (505) from the down-sampled image (503) comprises using two or more up-sampling stages (509, 510) .
The apparatus (1000) according to any preceding claim, wherein the apparatus (1000) repeats the steps of any of the preceding claims using the output bloom effect image (506) as an input image (501) for one or more iterations.
A computer implemented method (900) for generating a bloom effect, the method (900) comprising:

obtaining (901) an input image;

generating (902) a down-sampled image from the input image;

selecting (903) a region of one or more pixels of the down-sampled image in dependence on a brightness of the pixels;

generating (904) an up-sampled image region from the down-sampled image region; and

generating (905) an output bloom effect image in dependence on the up-sampled image region.
An image processing apparatus (1000) for generating a bloom effect, the apparatus (1000) comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code, when executed by the one or more processors, causes the image processing apparatus (1000) to:

obtain an input image (1101) ;

select a region of one or more pixels of the input image in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm; and

generate an output bloom effect image (1103) in dependence on the filtered image region (1102) .
The apparatus (1000) of claim 15, wherein filtering the one or more pixels comprises using a cubic basis spline filter algorithm.
The apparatus (1000) according to claim 15 or 16, wherein generating the output bloom effect image (1103) comprises applying a blur diffusion algorithm to the filtered image region (1102) .
The apparatus (1000) according to claim 17, wherein generating the output bloom effect image (1103) is in dependence on the filtered image region (1102) and the input image (1101) .
The apparatus (1000) according to claim 18, wherein generating the output bloom effect image (1103) comprises combining the filtered image region (1102) and the input image (1101) .
The apparatus (1000) according to claim 19, wherein combining the filtered image region (1102) and the input image (1101) comprises overlaying the filtered image region (1102) onto a corresponding region of the input image (1101) .
The apparatus (1000) according to any of claims 15 to 20, wherein the apparatus (1000) repeats the steps of any of claims 15 to 20 using the output bloom effect image (1103) as an input image (1101) for one or more iterations.
A computer implemented method (1700) for generating a bloom effect, the method (1700) comprising:

obtaining (1701) an input image;

selecting (1702) a region of one or more pixels of the input image in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter; and

generating (1703) an output bloom effect image in dependence on the filtered image region.
An image processing apparatus (1000) for generating a bloom effect, the apparatus (1000) comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code, when executed by the one or more processors, causes the image processing apparatus (1000) to:

obtain an input image (501, 1101) ;

generate a down-sampled image from the input image (502) ;

select a region of one or more pixels of the down-sampled image in dependence on a brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm;

generate an up-sampled image region from the down-sampled image region (503) ; and

generate an output bloom effect image (506, 1103) in dependence on the up-sampled image region.
A computer implemented method (1800) for generating a bloom effect, the method (1800) comprising:

obtaining (1801) an input image;

generating (1802) a down-sampled image from the input image;

selecting (1803) a region of one or more pixels of the down-sampled image in dependence on the brightness of the pixels, wherein the one or more pixels are filtered by means of a basis-spline filter algorithm;

generating (1804) an up-sampled image region from the down-sampled image region; and

generating (1805) an output bloom effect image in dependence on the up-sampled image region.
A computer program comprising executable code which when executed by a computer causes the computer to perform the method of claim 24.