WO2014038927A1 - Load balancing of graphics processing units in an image processing system - Google Patents
Load balancing of graphics processing units in an image processing system Download PDFInfo
- Publication number
- WO2014038927A1 WO2014038927A1 PCT/MY2013/000165 MY2013000165W WO2014038927A1 WO 2014038927 A1 WO2014038927 A1 WO 2014038927A1 MY 2013000165 W MY2013000165 W MY 2013000165W WO 2014038927 A1 WO2014038927 A1 WO 2014038927A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- images
- video
- load
- graphics processing
- component
- Prior art date
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B13/00—Burglar, theft or intruder alarms
- G08B13/18—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength
- G08B13/189—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems
- G08B13/194—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
- G08B13/196—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
- G08B13/19665—Details related to the storage of video surveillance data
- G08B13/19667—Details realated to data compression, encryption or encoding, e.g. resolution modes for reducing data volume to lower transmission bandwidth or memory requirements
Definitions
- the invention relates to load balancing of graphics processing units in an image processing system.
- the frames thereof may be distributed by a controller to a plurality of graphics processing unit (GPU) for video analytics (VA) such as fire detection, motion detection, theft, etc.
- GPU graphics processing unit
- VA video analytics
- an image processing system comprising:
- At least one imaging source for capturing images and/or video
- one or more graphics processing units for processing the images and/or video; at least one monitoring component for monitoring the processing load of the graphics processing units;
- a load adjuster component for adjusting the processing load of the graphics processing units
- the load adjuster component adjusts the processing load by increasing or reducing the rate at which images and/or video are captured by the imaging source.
- the load adjuster component adjusts the processing load by increasing or reducing the rate at which images and/or video are captured by the imaging source.
- the imaging source is a camera.
- the load adjuster component reduces the capture rate of the imaging source by one frame per second. In one embodiment if the monitoring component detects that a graphics processing unit is not busy or underutilised, the load adjuster component increases the capture rate of the imaging source by one frame per second.
- configuration information for the imaging source is read from an XML configuration file.
- the initial frame rate of the imaging source is set at the imaging source's maximum capture rate e.g. 25 frames per second.
- the minimum frame rate is set at the minimum rate for effective video analytics e.g. 6 frames per second. It will be appreciated that any suitable rates could be applied depending on the requirements.
- the system includes a server in which the graphics processing units, monitoring component and load adjuster component are located.
- the images and/or video are transmitted to the server via a network such as the internet.
- the processed images and/or video are transmitted to a client for viewing, typically via a network such as the internet.
- the images and/or video are received by the server, decoded, and stored in a temporary buffer before they are processed by a graphics processing unit.
- the load adjuster component may discard images and/or video frames from the temporary buffer.
- the images and/or video are stored in a network buffer before being transmitted to the server.
- the load adjuster component may discard images and/or video frames from the network buffer. Thus network bandwidth is saved as the frames may be discarded prior to transmission.
- a processing component is provided to determine which images and/or video are to be processed by a graphics processing unit.
- the images and/or video are passed to a graphics processing unit for video analytics processing, typically based on image timestamp.
- the monitoring component includes a timer to measure the number of frames processed per second.
- a method of load balancing the processing of images and/or video in an image processing system comprising: capturing images and/or video via at least one imaging source; processing the images and/or video via a plurality of graphics processing units; monitoring the processing load of the graphics processing units via at least one monitoring component for; and adjusting the processing load of the graphics processing units via a load adjuster component; characterised in that the processing load is adjusted by the load adjuster component by increasing or reducing the rate at which images and/or video are captured by the imaging source.
- Figure 1 illustrates a schematic overview of the system according to an embodiment of the invention.
- Figure 2 illustrates a system architecture block diagram according to an embodiment of the invention.
- Figure 3 illustrates a block diagram of the image acquisition and storage process.
- Figure 4 illustrates a block diagram of the data buffering process.
- Figure 5 illustrates a block diagram of the processing component initialization sequence.
- Figure 6 illustrates a block diagram of the load adjuster module.
- Figure 7 illustrates a block diagram of the GPU monitoring component.
- Figure 8 illustrates a block diagram of the frame fetching and dropping mechanism.
- FIG. 1 there is illustrated the overall system 2 consisting of an imaging source in the form of a camera 4, a server 6 which has a central processing unit (CPU) 8, graphics processing unit (GPU) 10 and a network card 12, and viewing station 14 which receives processed images from the server 6.
- the CPU 8 manages the whole system
- the GPU is used for image processing 10
- the network card 12 is used to send processed images to a viewing client 16 on the viewing station 14 and receive the images from the camera 4, typically via the internet 18.
- Figure 2 depicts the system architecture block diagram, comprising a camera communication module 20 which receives images from the camera 4, a buffer management module 22 for temporarily storing images, a processing component 24 for determining which received images to process, a GPU monitoring thread 26 for monitoring GPU performance, a GPU card 28 for performing the image processing and streaming, a GPU processing load adjustment module 30, a client processing module 32, and a client communication module 34 for output to a viewing client 16 on the viewing station 14.
- the GPU processing load adjuster module 30 is responsible for adjusting the input load to the GPU 28. If the load adjuster module 30 finds the GPU 28 to be busy, it will reduce the camera acquisition rate by 1, drops frame at the network data buffer i.e. drops packets before decoding, and also drops frames at the temporary storage buffer which stores decoded images. If the GPU is free, the camera frame rate will be increased and each frame will be processed without frame drops.
- the GPU processing load adjuster component 30 interacts with the camera 4, camera communication module 20 and buffer management module 22 for load adjustment.
- the processing component 24 will pass the images to the GPU 28 for Video Analytics (VA) processing 34.
- the GPU monitoring component 26 is an object with an independent thread 36 monitoring the GPU device output buffer to determine GPU load. It provides feedback of GPU status 38 which is picked up by the Load Adjuster module 30.
- the camera communication module 20 connects 40 to the camera 4 and receives images 42.
- the images are decoded 44 based on received format e.g. MPEG or MJPEG and a temporary buffer is then checked 46 to determine 48 if it is full or not. If not full the frame is stored in the temporary buffer 50. If full the frame may be moved to a discard buffer 52 or the latest frame may be pushed into the temporary buffer 50.
- the discard buffer 52 does not immediately discard the frames, but retains them in round robin fashion.
- the decoded image will be fetched 54 by image data processing module 24 once the GPU is free for VA processing. Data availability is checked 56, and if available is sent 58 to the GPU 28 for processing 34. If data is not available the GPU status is checked 60 via the Load Adjuster module 30 to determine if it is busy or not 62. If the GPU is not busy, data is fetched 54 by the image data processing module 24. If the GPU is busy, the frame may be moved to the discard buffer 52. The camera acquisition rate may also be adjusted 64 accordingly.
- the frames in the temporary buffer 50 and discard buffer 52 need to be in sync based on time stamp, and the latest frame from the discard buffer 52 may be pushed back to the temporary buffer 50 accordingly. Older frames in the discard buffer 50 can be deleted.
- the streaming component through client communication module 34 will transmit and receive requests 66 for images from the view station 14.
- Data is read 68 from the GPU 28 and stored 70 in the processed data buffer.
- the data is fetched 72 by a specific thread and if available 74 is sent to the viewing client 16.
- Figure 3 illustrates further details of the image acquisition and storage process.
- the configuration information such as camera IP address, login and password are read 304 from the xml configuration file 306.
- the configuration information is passed to an acquisition component 308 and the connection to the camera is initialized through a web-url mechanism 310.
- the initial frame rate is set 312 at a maximum frame rate such as 25 fps.
- the data is stored 316 at the network buffer with timestamp information before being passed into a decoding buffer.
- the image is decoded 44 based on the acquired image format such as MJPEG or MPEG format, time stamped 318, then stored in the temporary buffer 50.
- Figure 4 shows data buffering details.
- the network buffer 404, decoding buffer 406 and processing buffer 408 are each managed in the same way.
- Once data is received 410 the next available slot is determined 412 to store the data.
- the buffer limit is checked 414, and if the limit is reached the pointer will be reset 416 to start position, otherwise it points to the next available buffer slot 418.
- the current available slot number is returned 420 so that data can be stored at that location 422.
- the buffers are managed in ring buffer fashion based on FIFO queues. It is assumed that data will be arriving in sequence as it is transmitted using a TCP/IP framework.
- Figure 5 shows the initialization sequence of the processing component 24. This starts 502 by reading 504 the same xml configuration file 306 as described above, as the load adjustment module 30 requires camera details to explicitly set the frame rate based on GPU load. Thus the configuration information is used to initialise 506 the processing component before initialising the load adjuster module 30 and the GPU monitoring thread 26.
- FIG. 6 shows the overall flow of the load adjuster module 30.
- load adjuster module operation 604 is determined by a GPU status flag 606, which is controlled by the GPU monitoring module 26.
- the status of the GPU flag is determined 608, and if found to be busy, the details of the current image source device are acquired 610 and the current frame rate of this device is obtained 612. If the frame rate is determined 614 to be above a threshold e.g. 6, the frame rate is decremented 616 by 1 and a new frame rate is explicitly set 618 at the device and stored locally 620. In addition, the communication module 20 and the buffer management module 22 are each respectively signalled 622, 624 to drop the frame.
- the GPU If the GPU is free, data for image processing operation using the GPU will be fetched 626, if available 628, from the buffer management module 22. Once data is fetched it will be sent 630 to the GPU 632 for processing.
- the processed data is read 634 from the GPU, the processed frame count is increased 636, and the processed data is stored 638 in the processed buffer, so that it can be fetched later by the streaming component.
- the acquisition device is identified 640 and the acquisition rate is increased 642 by 1. The maximum rate is taken care of by the device itself and no frames are dropped at the acquisition side.
- the GPU status is continued to be monitored 644, wherein the frame count is used to measure the processing rate by the GPU monitoring component 26 to determine if the GPU status is busy or free.
- Figure 7 shows the GPU monitoring component operation.
- the operation starts 702 and the GPU monitoring thread is initialised 704 wherein a timer is set 706 to activate timer code each second thus measuring the rate of operation.
- a timer is set 706 to activate timer code each second thus measuring the rate of operation.
- the timer code is activated 708, the current processed frame count of the device is acquired 710 and measured against the previous frame count 712, thereby providing a figure for how many frames are processed within a second.
- Processed frame count is always incremented to keep track of how many frames are processed. Overflow is taken care of during a comparison operation, which is required because the GPU monitoring thread is an independent thread from the processing thread.
- the processed frame count is checked 714, and the current GPU is checked 716 to determine its status 718. If the frame count exceeds the threshold value e.g. 6, for example because the GPU is processing the frames quickly, but the GPU status flag is set to busy, then the GPU status flag is set to free 720. If the frame count is below the threshold value i.e. the GPU is processing frames slowly due to heavy load, and the GPU status flag is set to free, then the GPU status flag is set to busy 722. If the frame count exceeds the threshold value and the GPU status is free, or the frame count is below the threshold value and the GPU is busy, the GPU status flag is not changed.
- the threshold value e.g. 6
- the GPU status flag is set to free
- the previous frame count is updated 724 to the current frame count and the code ends 726.
- FIG. 8 shows the frame fetching and dropping mechanism.
- Frame fetching and dropping mechanism is the same at the network buffer 404, buffer management module 22 and also at the streaming side via client processing module 32.
- the network communication and buffer management module handler 802 comprises a buffer data fetch handler, a frame drop handler, and a buffer store data handler. Basically frames are fetched based on position of frame fetch pointer once there is a request to get current frame.
- Buffer data Fetch handler 804 will receive a request to fetch 44, 54, 72 data for decoding (at network side), processing, or streaming respectively.
- Data is returned 806 based on the current location of the fetch data pointer which acts as an indicator for respective data. If the fetch Pointer reaches the end of the buffer 808 after the current data is returned, the pointer is shifted 810 to the first location i.e. the header of the buffer, else the current location is incremented 812 to point to the next location.
- the frame drop handler 814 receives requests 816 to drop frames, and in such circumstances the fetch pointer is incremented by one, thus skipping one frame.
- the request to discard a frame for network can be received from gpu processing load adjuster 30, processing 62, or streaming handled within the client communication module 34 (so as to ensure old frames are discarded if the client is down).
- Once notification is received if the current fetch pointer for frame dropping is pointing at the end of buffer, one frame is skipped by resetting the pointer to head/start location else one frame is dropped by incrementing the current pointer. There will be no discard requests from the processed data buffer, as processed frames are stored 70 in round robin fashion.
- buffer store data handler 818 data is stored at the buffer based on store pointer at the next available slot.
- a request to store data at a particular location for network can be received 820 from the camera 4, the buffer management module 22 (temporary buffer 50 or discard buffer 52) and for streaming (storing 70 in the processed data buffer).
- the location to store the data needs to be checked 822. If the location pointer is pointing to the end of buffer, it needs to be reset 824 to the head/start location else the current pointer is incremented 826 to point to the next empty slot. Data is then stored 828 at this location.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Closed-Circuit Television Systems (AREA)
- Studio Devices (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
An image processing system (2) comprising at least one imaging source (4) for capturing images and/or video, one or more graphics processing units (28) for processing the images and/or video, at least one monitoring component (26) for monitoring the processing load of the graphics processing units (28), and a load adjuster component (30) for adjusting the processing load of the graphics processing units (28) by increasing or reducing the rate at which images and/or video are captured by the imaging source (4).
Description
LOAD BALANCING OF GRAPHICS PROCESSING UNITS IN AN IMAGE
PROCESSING SYSTEM
Field of Invention The invention relates to load balancing of graphics processing units in an image processing system.
Background
In a conventional system for processing video, the frames thereof may be distributed by a controller to a plurality of graphics processing unit (GPU) for video analytics (VA) such as fire detection, motion detection, theft, etc.
However, if the processing load is too high for a particular GPU, a feedback signal is sent from that GPU to the controller to temporarily inhibit further transfer of frames thereto from the controller.
Unfortunately, this can result in frames being dropped when they are not processed, which reduces accuracy due to non-smooth transitions. An aim of the invention is to address the aforementioned issue.
Summary of Invention
In an aspect of the invention, there is provided an image processing system comprising:
at least one imaging source for capturing images and/or video;
one or more graphics processing units for processing the images and/or video; at least one monitoring component for monitoring the processing load of the graphics processing units; and
a load adjuster component for adjusting the processing load of the graphics processing units;
characterised in that the load adjuster component adjusts the processing load by increasing or reducing the rate at which images and/or video are captured by the imaging source. Thus by adjusting the image capture rate at source, overloading of the graphics processing units is minimised, resulting in less frames being dropped and smooth frame transitions.
In one embodiment the imaging source is a camera.
In one embodiment if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component reduces the capture rate of the imaging source by one frame per second.
In one embodiment if the monitoring component detects that a graphics processing unit is not busy or underutilised, the load adjuster component increases the capture rate of the imaging source by one frame per second.
In one embodiment configuration information for the imaging source is read from an XML configuration file. Typically the initial frame rate of the imaging source is set at the imaging source's maximum capture rate e.g. 25 frames per second. Typically the minimum frame rate is set at the minimum rate for effective video analytics e.g. 6 frames per second. It will be appreciated that any suitable rates could be applied depending on the requirements.
In one embodiment the system includes a server in which the graphics processing units, monitoring component and load adjuster component are located. Typically the images and/or video are transmitted to the server via a network such as the internet.
In one embodiment the processed images and/or video are transmitted to a client for viewing, typically via a network such as the internet. In one embodiment the images and/or video are received by the server, decoded, and stored in a temporary buffer before they are processed by a graphics processing unit.
In one embodiment if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component may discard images and/or video frames from the temporary buffer. In one embodiment the images and/or video are stored in a network buffer before being transmitted to the server.
In one embodiment if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component may discard images and/or video frames from the network buffer. Thus network bandwidth is saved as the frames may be discarded prior to transmission.
In one embodiment a processing component is provided to determine which images and/or video are to be processed by a graphics processing unit.
In one embodiment the images and/or video are passed to a graphics processing unit for video analytics processing, typically based on image timestamp.
In one embodiment the monitoring component includes a timer to measure the number of frames processed per second.
In a further aspect of the invention, there is provided a method of load balancing the processing of images and/or video in an image processing system comprising:
capturing images and/or video via at least one imaging source; processing the images and/or video via a plurality of graphics processing units; monitoring the processing load of the graphics processing units via at least one monitoring component for; and adjusting the processing load of the graphics processing units via a load adjuster component; characterised in that the processing load is adjusted by the load adjuster component by increasing or reducing the rate at which images and/or video are captured by the imaging source.
Brief Description of Drawings
It will be convenient to further describe the present invention with respect to the accompanying drawings that illustrate possible arrangements of the invention. Other arrangements of the invention are possible, and consequently the particularity of the accompanying drawings is not to be understood as superseding the generality of the preceding description of the invention.
Figure 1 illustrates a schematic overview of the system according to an embodiment of the invention.
Figure 2 illustrates a system architecture block diagram according to an embodiment of the invention.
Figure 3 illustrates a block diagram of the image acquisition and storage process. Figure 4 illustrates a block diagram of the data buffering process.
Figure 5 illustrates a block diagram of the processing component initialization sequence.
Figure 6 illustrates a block diagram of the load adjuster module.
Figure 7 illustrates a block diagram of the GPU monitoring component.
Figure 8 illustrates a block diagram of the frame fetching and dropping mechanism. Detailed Description
With reference to Figure 1 there is illustrated the overall system 2 consisting of an imaging source in the form of a camera 4, a server 6 which has a central processing unit (CPU) 8, graphics processing unit (GPU) 10 and a network card 12, and viewing station 14 which receives processed images from the server 6. The CPU 8 manages the whole system, the GPU is used for image processing 10 and the network card 12 is used to send processed images to a viewing client 16 on the viewing station 14 and receive the images from the camera 4, typically via the internet 18.
Figure 2 depicts the system architecture block diagram, comprising a camera communication module 20 which receives images from the camera 4, a buffer management module 22 for temporarily storing images, a processing component 24 for determining which received images to process, a GPU monitoring thread 26 for monitoring GPU performance, a GPU card 28 for performing the image processing and streaming, a GPU processing load adjustment module 30, a client processing module 32, and a client communication module 34 for output to a viewing client 16 on the viewing station 14.
The GPU processing load adjuster module 30 is responsible for adjusting the input load to the GPU 28. If the load adjuster module 30 finds the GPU 28 to be busy, it will reduce the camera acquisition rate by 1, drops frame at the network data buffer i.e. drops packets before decoding, and also drops frames at the temporary storage buffer which stores decoded images. If the GPU is free, the camera frame rate will be increased and each frame will be processed without frame drops.
The GPU processing load adjuster component 30 interacts with the camera 4, camera communication module 20 and buffer management module 22 for load adjustment. The processing component 24 will pass the images to the GPU 28 for Video Analytics (VA) processing 34.
The GPU monitoring component 26 is an object with an independent thread 36 monitoring the GPU device output buffer to determine GPU load. It provides feedback of GPU status 38 which is picked up by the Load Adjuster module 30. The camera communication module 20 connects 40 to the camera 4 and receives images 42. The images are decoded 44 based on received format e.g. MPEG or MJPEG and a temporary buffer is then checked 46 to determine 48 if it is full or not. If not full the frame is stored in the temporary buffer 50. If full the frame may be moved to a discard buffer 52 or the latest frame may be pushed into the temporary buffer 50. The discard buffer 52 does not immediately discard the frames, but retains them in round robin fashion.
The decoded image will be fetched 54 by image data processing module 24 once the GPU is free for VA processing. Data availability is checked 56, and if available is sent 58 to the GPU 28 for processing 34. If data is not available the GPU status is checked 60 via the Load Adjuster module 30 to determine if it is busy or not 62. If the GPU is not busy, data is fetched 54 by the image data processing module 24. If the GPU is busy, the frame may be moved to the discard buffer 52. The camera acquisition rate may also be adjusted 64 accordingly.
Thus if the GPU is busy 62, the frames in the temporary buffer 50 and discard buffer 52 need to be in sync based on time stamp, and the latest frame from the discard
buffer 52 may be pushed back to the temporary buffer 50 accordingly. Older frames in the discard buffer 50 can be deleted.
For the client 16 the streaming component through client communication module 34 will transmit and receive requests 66 for images from the view station 14. Data is read 68 from the GPU 28 and stored 70 in the processed data buffer. The data is fetched 72 by a specific thread and if available 74 is sent to the viewing client 16.
Figure 3 illustrates further details of the image acquisition and storage process. During initialization 302 the configuration information such as camera IP address, login and password are read 304 from the xml configuration file 306. The configuration information is passed to an acquisition component 308 and the connection to the camera is initialized through a web-url mechanism 310. The initial frame rate is set 312 at a maximum frame rate such as 25 fps. Thus as soon as the image is captured at the source device in this case is an IP cam, images will be streamed into acquisition component 314 at this rate. At this stage, the data is stored 316 at the network buffer with timestamp information before being passed into a decoding buffer. The image is decoded 44 based on the acquired image format such as MJPEG or MPEG format, time stamped 318, then stored in the temporary buffer 50.
Figure 4 shows data buffering details. The network buffer 404, decoding buffer 406 and processing buffer 408 are each managed in the same way. Once data is received 410 the next available slot is determined 412 to store the data. The buffer limit is checked 414, and if the limit is reached the pointer will be reset 416 to start position, otherwise it points to the next available buffer slot 418. The current available slot number is returned 420 so that data can be stored at that location 422. Thus the buffers are managed in ring buffer fashion based on FIFO queues. It is assumed that data will be arriving in sequence as it is transmitted using a TCP/IP framework. Figure 5 shows the initialization sequence of the processing component 24. This starts 502 by reading 504 the same xml configuration file 306 as described above, as the load adjustment module 30 requires camera details to explicitly set the frame rate based on GPU load. Thus the configuration information is used to initialise 506 the processing component before initialising the load adjuster module 30 and the GPU monitoring thread 26.
Figure 6 shows the overall flow of the load adjuster module 30. After initialisation 602, load adjuster module operation 604 is determined by a GPU status flag 606, which is controlled by the GPU monitoring module 26.
The status of the GPU flag is determined 608, and if found to be busy, the details of the current image source device are acquired 610 and the current frame rate of this device is obtained 612. If the frame rate is determined 614 to be above a threshold
e.g. 6, the frame rate is decremented 616 by 1 and a new frame rate is explicitly set 618 at the device and stored locally 620. In addition, the communication module 20 and the buffer management module 22 are each respectively signalled 622, 624 to drop the frame.
If the GPU is free, data for image processing operation using the GPU will be fetched 626, if available 628, from the buffer management module 22. Once data is fetched it will be sent 630 to the GPU 632 for processing. The processed data is read 634 from the GPU, the processed frame count is increased 636, and the processed data is stored 638 in the processed buffer, so that it can be fetched later by the streaming component. In addition, the acquisition device is identified 640 and the acquisition rate is increased 642 by 1. The maximum rate is taken care of by the device itself and no frames are dropped at the acquisition side. The GPU status is continued to be monitored 644, wherein the frame count is used to measure the processing rate by the GPU monitoring component 26 to determine if the GPU status is busy or free.
Figure 7 shows the GPU monitoring component operation. The operation starts 702 and the GPU monitoring thread is initialised 704 wherein a timer is set 706 to activate timer code each second thus measuring the rate of operation.
When the timer code is activated 708, the current processed frame count of the device is acquired 710 and measured against the previous frame count 712, thereby providing a figure for how many frames are processed within a second. Processed frame count is always incremented to keep track of how many frames are processed. Overflow is taken care of during a comparison operation, which is required because the GPU monitoring thread is an independent thread from the processing thread.
The processed frame count is checked 714, and the current GPU is checked 716 to determine its status 718. If the frame count exceeds the threshold value e.g. 6, for example because the GPU is processing the frames quickly, but the GPU status flag is set to busy, then the GPU status flag is set to free 720. If the frame count is below the threshold value i.e. the GPU is processing frames slowly due to heavy load, and the GPU status flag is set to free, then the GPU status flag is set to busy 722. If the frame count exceeds the threshold value and the GPU status is free, or the frame count is below the threshold value and the GPU is busy, the GPU status flag is not changed.
The previous frame count is updated 724 to the current frame count and the code ends 726.
Figure 8 shows the frame fetching and dropping mechanism. Frame fetching and dropping mechanism is the same at the network buffer 404, buffer management module 22 and also at the streaming side via client processing module 32.
The network communication and buffer management module handler 802 comprises a buffer data fetch handler, a frame drop handler, and a buffer store data handler. Basically frames are fetched based on position of frame fetch pointer once there is a request to get current frame. Buffer data Fetch handler 804 will receive a request to fetch 44, 54, 72 data for decoding (at network side), processing, or streaming respectively. Data is returned 806 based on the current location of the fetch data pointer which acts as an indicator for respective data. If the fetch Pointer reaches the end of the buffer 808 after the current data is returned, the pointer is shifted 810 to the first location i.e. the header of the buffer, else the current location is incremented 812 to point to the next location.
The frame drop handler 814 receives requests 816 to drop frames, and in such circumstances the fetch pointer is incremented by one, thus skipping one frame. The request to discard a frame for network can be received from gpu processing load adjuster 30, processing 62, or streaming handled within the client communication module 34 (so as to ensure old frames are discarded if the client is down). Once notification is received, if the current fetch pointer for frame dropping is pointing at the end of buffer, one frame is skipped by resetting the pointer to head/start location else one frame is dropped by incrementing the current pointer. There will be no discard requests from the processed data buffer, as processed frames are stored 70 in round robin fashion.
With regard to the buffer store data handler 818, data is stored at the buffer based on store pointer at the next available slot. A request to store data at a particular location for network can be received 820 from the camera 4, the buffer management module 22 (temporary buffer 50 or discard buffer 52) and for streaming (storing 70 in the processed data buffer). Once data is received, the location to store the data needs to be checked 822. If the location pointer is pointing to the end of buffer, it needs to be reset 824 to the head/start location else the current pointer is incremented 826 to point to the next empty slot. Data is then stored 828 at this location.
It will be appreciated by persons skilled in the art that the present invention may also include further additional modifications made to the device which does not affect the overall functioning of the device.
Claims
Claims
An image processing system (2) comprising
at least one imaging source (4) for capturing images and/or video;
one or more graphics processing units (28) for processing the images and/or video;
at least one monitoring component (26) for monitoring the processing load of the graphics processing units (28); and
a load adjuster component (30) for adjusting the processing load of the graphics processing units (28);
characterised in that the load adjuster component (30) adjusts the processing load by increasing or reducing the rate at which images and/or video are captured by the imaging source (4).
An image processing system according to claim 1 wherein if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component reduces the capture rate of the imaging source by one frame per second. 3. An image processing system according to claim 1 wherein if the monitoring component detects that a graphics processing unit is not busy or underutilised, the load adjuster component increases the capture rate of the imaging source by one frame per second.
An image processing system according to claim 1 wherein the images and/or video are transmitted to a server in which the graphics processing units, monitoring component and load adjuster component are located.
An image processing system according to claim 1 wherein the processed images and/or video are transmitted to a client for viewing.
An image processing system according to claim 1 wherein the images and/or video are decoded and stored in a temporary buffer before they are processed by a graphics processing unit.
An image processing system according to claim 6 wherein if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component may discard images and/or video frames from the temporary buffer.
An image processing system according to claim 1 wherein the images and/or video are stored in a network buffer before being transmitted from the imaging source, and if the monitoring component detects that a graphics processing unit is busy or overloaded, the load adjuster component may discard images and/or video frames from the network buffer.
9. An image processing system according to claim 1 wherein images and/or video are passed to a graphics processing unit for video analytics processing based on image timestamp. 10. A method of load balancing the processing of images and/or video in an image processing system (2) comprising:
capturing images and/or video via at least one imaging source (4);
processing the images and/or video via a plurality of graphics processing units (28);
monitoring the processing load of the graphics processing units via at least one monitoring component (26); and
adjusting the processing load of the graphics processing units via a load adjuster component (30);
characterised in that the processing load is adjusted by the load adjuster component (30) by increasing or reducing the rate at which images and/or video are captured by the imaging source (4).
1 1. A method according to claim 10 wherein if the monitoring component detects that a graphics processing unit is busy, the load adjuster component reduces the capture rate of the imaging source by one frame per second and/or discards images and/or video frames received therefrom.
12. A method according to claim 10 wherein if the monitoring component detects that a graphics processing unit is not busy or underutilised, the load adjuster component increases the capture rate of the imaging source by one frame per second.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
MYPI2012004012A MY162260A (en) | 2012-09-10 | 2012-09-10 | Load balancing of graphics processing units in an image processing system |
MYPI2012004012 | 2012-09-10 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2014038927A1 true WO2014038927A1 (en) | 2014-03-13 |
Family
ID=49356484
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/MY2013/000165 WO2014038927A1 (en) | 2012-09-10 | 2013-09-09 | Load balancing of graphics processing units in an image processing system |
Country Status (2)
Country | Link |
---|---|
MY (1) | MY162260A (en) |
WO (1) | WO2014038927A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016015667A1 (en) * | 2014-08-01 | 2016-02-04 | 广州金山网络科技有限公司 | Video stream decoding method and device |
CN114302140A (en) * | 2021-11-30 | 2022-04-08 | 山东云海国创云计算装备产业创新中心有限公司 | Pre-frame loss method, system, equipment and computer readable storage medium |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020126751A1 (en) * | 1998-05-22 | 2002-09-12 | Christoph E. Scheurich | Maintaining a frame rate in a digital imaging system |
US20030107648A1 (en) * | 2001-12-12 | 2003-06-12 | Richard Stewart | Surveillance system and method with adaptive frame rate |
EP1936576A1 (en) * | 2006-12-20 | 2008-06-25 | Axis AB | Camera tampering detection |
US20090135918A1 (en) * | 2007-11-23 | 2009-05-28 | Research In Motion Limited | System and method for providing a variable frame rate and adaptive frame skipping on a mobile device |
US20110211036A1 (en) * | 2010-02-26 | 2011-09-01 | Bao Tran | High definition personal computer (pc) cam |
-
2012
- 2012-09-10 MY MYPI2012004012A patent/MY162260A/en unknown
-
2013
- 2013-09-09 WO PCT/MY2013/000165 patent/WO2014038927A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020126751A1 (en) * | 1998-05-22 | 2002-09-12 | Christoph E. Scheurich | Maintaining a frame rate in a digital imaging system |
US20030107648A1 (en) * | 2001-12-12 | 2003-06-12 | Richard Stewart | Surveillance system and method with adaptive frame rate |
EP1936576A1 (en) * | 2006-12-20 | 2008-06-25 | Axis AB | Camera tampering detection |
US20090135918A1 (en) * | 2007-11-23 | 2009-05-28 | Research In Motion Limited | System and method for providing a variable frame rate and adaptive frame skipping on a mobile device |
US20110211036A1 (en) * | 2010-02-26 | 2011-09-01 | Bao Tran | High definition personal computer (pc) cam |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016015667A1 (en) * | 2014-08-01 | 2016-02-04 | 广州金山网络科技有限公司 | Video stream decoding method and device |
CN114302140A (en) * | 2021-11-30 | 2022-04-08 | 山东云海国创云计算装备产业创新中心有限公司 | Pre-frame loss method, system, equipment and computer readable storage medium |
CN114302140B (en) * | 2021-11-30 | 2023-11-03 | 山东云海国创云计算装备产业创新中心有限公司 | Pre-frame-loss method, system, equipment and computer readable storage medium |
Also Published As
Publication number | Publication date |
---|---|
MY162260A (en) | 2017-05-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6753902B2 (en) | Storage management of data streamed from video source devices | |
JP6169172B2 (en) | System and method for encoding video content using virtual intraframes | |
US8160129B2 (en) | Image pickup apparatus and image distributing method | |
US11683510B2 (en) | Method and devices for encoding and streaming a video sequence over a plurality of network connections | |
CN105991976A (en) | Method and system for generating event video sequence, and camera comprising the system | |
US20100033576A1 (en) | Data delivery device | |
US20150264411A1 (en) | Method and system for playback of motion video | |
US20150207715A1 (en) | Receiving apparatus, transmitting apparatus, communication system, control method for receiving apparatus, control method for transmitting apparatus, and recording medium | |
US20190191210A1 (en) | Systems and methods for transmitting a high quality video image from a low power sensor | |
WO2014038927A1 (en) | Load balancing of graphics processing units in an image processing system | |
EP3843411A1 (en) | Video frame-based media stream bandwidth reduction | |
US11438545B2 (en) | Video image-based media stream bandwidth reduction | |
JP2022541395A (en) | Systems and methods for collecting and transmitting real-time data | |
US20130007206A1 (en) | Transmission apparatus, control method for transmission apparatus, and storage medium | |
JP4533708B2 (en) | Surveillance camera system, control device for surveillance camera system, and control method | |
US20140327781A1 (en) | Method for video surveillance, a related system, a related surveillance server, and a related surveillance camera | |
JP6083964B2 (en) | Transmission device, transmission method, and program | |
JP2012090172A (en) | Monitoring camera, monitoring camera control method, and network camera system | |
JP6740002B2 (en) | Control device, control method and program | |
KR20140072668A (en) | Network camera server and method for processing video stream thereof | |
US8154656B2 (en) | Capture buffer control methodology for caption carriage switch | |
WO2021096159A1 (en) | System and method for ingesting live video | |
JP2016213611A (en) | Data distribution device and data distribution method | |
CN116614605A (en) | Large-scale video frame data extraction system and method | |
US20180262790A1 (en) | Systems and methods for adaptive streaming using jpeg 2000 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 13776869 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 13776869 Country of ref document: EP Kind code of ref document: A1 |