CN104301671A - Traffic monitoring video storing method in HDFS based on event intensity - Google Patents
Traffic monitoring video storing method in HDFS based on event intensity Download PDFInfo
- Publication number
- CN104301671A CN104301671A CN201410490195.2A CN201410490195A CN104301671A CN 104301671 A CN104301671 A CN 104301671A CN 201410490195 A CN201410490195 A CN 201410490195A CN 104301671 A CN104301671 A CN 104301671A
- Authority
- CN
- China
- Prior art keywords
- data
- video
- file
- event
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Closed-Circuit Television Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a traffic monitoring video storing method in an HDFS based on event intensity. The traffic monitoring video storing method is characterized by comprising the following steps that an intelligent camera obtains traffic monitoring data, detects traffic events and sends traffic video streaming data and event type description information included in the data to a video server; the video server encodes and segments the video data after receiving the video streaming data sent by the camera, a video file is generated, and the file is marked with the event type according to the event description information of the video data; the video data are uploaded to the HDFS; after receiving a data storage request, a Name Node in the HDFS calls a data placement strategy based on the event intensity to select a target data node for the data file to be stored; the file is stored in the target data node selected in the third step. The problem of node load unbalance caused when the HDFS stores mass traffic monitoring videos can be solved, and accordingly, the traffic monitoring data can be efficiently stored.
Description
Technical field
The present invention relates to the storage of massive video data, particularly relate to the Traffic Surveillance Video storage means based on event closeness in a kind of HDFS.
Background technology
Along with the development of video compression technology and network transmission technology, intelligent video monitoring system is popularized gradually.Municipal intelligent traffic video monitoring due to watch-dog increase and high Qinghua, the monitor video data volume produced is very huge, the 2000000 pixel high-definition cameras for functions such as violation event detections are example, its maximum video code check is 4Mbps/s, will produce the video data of 40TB in the guarded region 30 days of so 8 crossings (each crossing arranges 4 cameras).The memory property of data volume to supervisory control system of quick growth proposes acid test.Contain the various traffic events of generation in the monitor video simultaneously stored, these traffic events later stages can be carried out retrieval verification by relevant department.Need to make quick response to event inquiry demand in time while ensureing to preserve the new monitor video produced in real time, this again improves the performance requirement for supervisory control system.
Except Large Copacity, monitor video also has the requirement of the aspects such as high reliability (video data was correctly stored in memory space within least 30 days, in order to evidence obtaining) and extensibility.Meanwhile, urban traffic video monitoring stores and has again the features such as impermanency stores (generally only requiring storage 15 ~ 30d), coherence request is relatively low, video access time is long.The simulation experienced along with video monitoring, numeral, network three phases, the Main Morphology stored during video monitoring stores includes DVR, NVR, IPSAN tri-kinds, although the demand of system for storage aspect can be met, but the deployment of system and management maintenance cost higher, and be unfavorable for the expansion of the application such as video data analysis.
Along with the appearance of GFS (Googlefilesystem), Hadoop distributed data processing architecture, cloud memory technology is full-fledged gradually, its have Storage Virtualization, enhanced scalability, cost low, be easy to management and the advantage such as mode is flexible, breach performance and the capacity bottleneck of conventional store mode, have great importance in field of storage, especially be the storage of the Large Copacity such as video class, non-structured data, provide important solution.Wherein HDFS is the realization of increasing income of GFS, is one of two large cores of Hadoop framework, and design is used for being deployed on cheap hardware, can provide high transmission rates DDM.Utilize Hadoop to build the distributed memory system being applied to Traffic Surveillance Video storage, a low cost, high performance distributed storage scheme can be provided for intelligent traffic monitoring field.
But HDFS have employed random data placement strategy when the placement of data block.This may cause too much data to be relatively concentrated on some node thus produce and store focus, causes system load unbalanced, the throughput of influential system.Although Traffic Surveillance Video data scale is comparatively large, in the middle of practical application, the access request of user often concentrates on the video segment comprising traffic events.If too much event video data is left concentratedly at some back end because of random placement, then these nodes may be higher and become storage focus due to load.Therefore HDFS is used for the storage of Traffic Surveillance Video relative to other data-storage applications, adopts the frame perceptual strategy of acquiescence may cause more load imbalance situation.
Annotations and comments: Hadoop Distributed File System, being called for short HDFS, is a distributed file system.
Summary of the invention
The object of this invention is to provide the Traffic Surveillance Video storage means based on event closeness in a kind of HDFS, it can solve the unbalanced problem of node load produced when HDFS stores magnanimity Traffic Surveillance Video, thus realizes the efficient storage of traffic monitoring data.
For achieving the above object, the present invention adopts following technical scheme:
Based on the Traffic Surveillance Video storage means of event closeness in HDFS, specifically comprise the steps:
(1) intelligent video camera head obtains traffic monitoring data, carries out the detection of traffic events simultaneously, then the event type descriptor comprised in traffic video flow data and data is sent to video server simultaneously.
(2) video server receives to coding video data and segmentation after the video stream data that camera sends, then generating video file, and is that event type indicated by file according to the event description information of video data.Video data is uploaded to HDFS.
(3), after the NameNode node in HDFS receives data storage request, the data placement strategy called based on event closeness is the data file select target back end that will deposit.
(4) by file stored in the target data node chosen in step (3).
Described step (3) is based on the data placement strategy of event closeness, and concrete steps are as follows:
1) when system receives the file operation requests of client, judge to be operating as which kind of type;
2) if be operating as file write operations, the events affecting degree of the instant each event type safeguarded in acquisition system; The representative of events affecting degree comprises the video file of dissimilar event because the visit capacity difference of user is to its place back end load effect degree.The present invention defines the events affecting degree that the visit capacity of e type file in time interval T and the ratio of e type file total number of files are in systems in which e types of events, and per interval T upgrades once.If a
ethe disturbance degree of presentation of events type e, computing formula is as follows:
3) the event closeness of each back end is calculated; If the set of the back end in cluster is D, back end d ∈ D, F is the file set in node d, the event type that e (f) is file f.A
e (f)for the events affecting degree of file f.In the present invention define back end d event closeness be calculated as follows:
4) the event closeness of comprehensive each back end, the factor such as real time load, disk size of back end are chosen target and are placed node; If the choosing method that target places node arbitrary back end g that to be g, d be in cluster is as follows:
A) L
g≤ L
d, namely g is the node that in all back end, event closeness is minimum.
B) set the load of the current time of node d as N
d, N
drepresent with the client operation number of requests being currently connected to d.If the node selected 1) is multiple, so g is present load N
dminimum and have the node of enough memory capacity.
If obtain multiple node same c) 2), then d is according to the node that frame perceptual strategy is selected from these nodes.
5) obtain the event type e wanting store video, upgrade the stored number of e type file in destination node e and system;
6) if be operating as File read operation, then the access number of e type file is upgraded.System, every regular time T, is often kind of each event type renewal events affecting degree.
Innovative point of the present invention is embodied in:
HDFS is used to carry out the storage of mass transportation monitor video, utilize traffic video can carry out by event this feature of classifying, the video data content of being placed by back end is as one of primary reference point during data placement, when placing new video data, the load that the file that pre-estimation back end stores may cause it, simultaneously in conjunction with the real time load situation and space utilisation etc. of back end because usually evaluating node, optimal node is selected to carry out data placement, decrease the system load that the video file of depositing too much high attention rate due to node causes unbalance, improve the throughput of storage system.
Accompanying drawing explanation
Fig. 1: the overall structure figure of method;
Fig. 2: based on the data placement strategy works flow chart of event closeness.
Embodiment
Basic fundamental thinking of the present invention is: the feature that the memory property advantage of comprehensive utilization Hadoop distributed memory system and Traffic Surveillance Video possess solves the efficient storage problem of mass transportation monitor video.Traffic Surveillance Video data scale is comparatively large, but in the middle of practical application, the access request of user often concentrates on the video segment comprising traffic events.Too much event video data, because random data placement method, may be concentrated and leave some back end in by the frame perceptual strategy of HDFS, thus causes storing focus appearance.Comprehensive consideration above some, the present invention utilizes traffic video can carry out by event type this feature of classifying, the load that may be caused it by all types of event video stored in back end when data placement is as one of the primary evaluation factor of node, the combined factors such as real time load, disk size simultaneously in conjunction with node is evaluated, select best data placement node, thus the load of equilibrium criterion node.
Based on the Traffic Surveillance Video storage means of event closeness in HDFS, specifically comprise the following steps:
(1) crossing intelligent video camera head carries out the traffic incidents detection in video while generating monitor video, if event occurs, so camera generates event description information, together issues video server together with video data.
(2) video server receives to coding video data and segmentation after video original data, then generating video file, and is that event type indicated by file according to the event description information of data.Video data is uploaded to HDFS.
(3) present invention achieves the data placement strategy based on event closeness, for replacing the frame perceptual strategy given tacit consent in HDFS.Be different from the method for the random selecting back end of frame perceptual strategy, being the event closeness that calculate back end when choosing back end according to all kinds of event number of videos stored in back end based on the data placement strategy of event closeness, in conjunction with the present load of back end and disk size, back end being evaluated simultaneously.When HDFS receives file storage resource request, carry out file storage by going out back end based on the data placement policy selection of event closeness.
(4) HDFS safeguards the stored number of all types of file of each back end, upgrades the quantity of the type video file in target data node when storage file.
(5) when user's accessing video data, HDFS safeguards user's access number of every class event video, calculates the access temperature upgrading all kinds of event video, as the load effect degree of all kinds of event video council for back end every the set time.
Below in conjunction with accompanying drawing, technical solution of the present invention is described further.
As shown in Figure 1, the Traffic Surveillance Video storage means based on event closeness in a kind of HDFS is provided herein, specifically comprises the steps:
(1) intelligent video camera head obtains traffic monitoring data, carries out the detection of traffic events simultaneously, then the event type descriptor comprised in traffic video flow data and data is sent to video server simultaneously.
(2) video server receives to coding video data and segmentation after the video stream data that camera sends, then generating video file, and is that event type indicated by file according to the event description information of video data.Video data is uploaded to HDFS.
(3), after the NameNode node in HDFS receives data storage request, the data placement strategy called based on event closeness is the data file select target back end that will deposit.
(4) by file stored in the target data node chosen in (3).
As shown in Figure 2, concrete steps are as follows for the idiographic flow of the data placement strategy based on event closeness proposed in the present invention:
1) when system receives the file operation requests of client, judge to be operating as which kind of type;
2) if be operating as file write operations, the events affecting degree of the instant each event type safeguarded in acquisition system; The representative of events affecting degree comprises the video file of dissimilar event because the visit capacity difference of user is to its place back end load effect degree.The present invention defines the events affecting degree that the visit capacity of e type file in time interval T and the ratio of e type file total number of files are in systems in which e types of events, and per interval T upgrades once.If a
ethe disturbance degree of presentation of events type e, computing formula is as follows:
3) the event closeness of each back end is calculated; If the set of the back end in cluster is D, back end d ∈ D, F is the file set in node d, the event type that e (f) is file f.A
e (f)for the events affecting degree of file f.In the present invention define back end d event closeness be calculated as follows:
4) the event closeness of comprehensive each back end, the factor such as real time load, disk size of back end are chosen target and are placed node; If the choosing method that target places node arbitrary back end g that to be g, d be in cluster is as follows:
D) L
g≤ L
d, namely g is the node that in all back end, event closeness is minimum.
E) set the load of the current time of node d as N
d, N
drepresent with the client operation number of requests being currently connected to d.If the node selected 1) is multiple, so g is present load N
dminimum and have the node of enough memory capacity.
If obtain multiple node same f) 2), then d is according to the node that frame perceptual strategy is selected from these nodes.
5) obtain the event type e wanting store video, upgrade the stored number of e type file in destination node e and system;
6) if be operating as File read operation, then the access number of e type file is upgraded.System, every regular time T, is often kind of each event type renewal events affecting degree.
Claims (1)
- Based on the Traffic Surveillance Video storage means of event closeness in 1.HDFS, it is characterized in that, specifically comprise the steps:(1) intelligent video camera head obtains traffic monitoring data, carries out the detection of traffic events simultaneously, then the event type descriptor comprised in traffic video flow data and data is sent to video server simultaneously;(2) video server receives to coding video data and segmentation after the video stream data that camera sends, then generating video file, and is that event type indicated by file according to the event description information of video data.Video data is uploaded to HDFS;(3), after the NameNode node in HDFS receives data storage request, the data placement strategy called based on event closeness is the data file select target back end that will deposit;(4) by file stored in the target data node chosen in step (3);Described step (3) is based on the data placement strategy of event closeness, and concrete steps are as follows:1) when system receives the file operation requests of client, judge to be operating as which kind of type;2) if be operating as file write operations, the events affecting degree of the instant each event type safeguarded in acquisition system; The representative of events affecting degree comprises the video file of dissimilar event because the visit capacity difference of user is to its place back end load effect degree; The present invention defines the events affecting degree that the visit capacity of e type file in time interval T and the ratio of e type file total number of files are in systems in which e types of events, and per interval T upgrades once; If a ethe disturbance degree of presentation of events type e, computing formula is as follows:3) the event closeness of each back end is calculated; If the set of the back end in cluster is D, back end d ε D, F is the file set in node d, the event type that e (f) is file f; a e (f)for the events affecting degree of file f; In the present invention define back end d event closeness be calculated as follows:4) the event closeness of comprehensive each back end, the factor such as real time load, disk size of back end are chosen target and are placed node; If the choosing method that target places node arbitrary back end g that to be g, d be in cluster is as follows:A) L g≤ L d, namely g is the node that in all back end, event closeness is minimum;B) set the load of the current time of node d as N d, N drepresent with the client operation number of requests being currently connected to d; If the node selected 1) is multiple, so g is present load N dminimum and have the node of enough memory capacity;If obtain multiple node same c) 2), then d is according to the node that frame perceptual strategy is selected from these nodes;5) obtain the event type e wanting store video, upgrade the stored number of e type file in destination node e and system;6) if be operating as File read operation, then the access number of e type file is upgraded.System, every regular time T, is often kind of each event type renewal events affecting degree.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410490195.2A CN104301671B (en) | 2014-09-23 | 2014-09-23 | Traffic Surveillance Video storage method based on event closeness in HDFS |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410490195.2A CN104301671B (en) | 2014-09-23 | 2014-09-23 | Traffic Surveillance Video storage method based on event closeness in HDFS |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104301671A true CN104301671A (en) | 2015-01-21 |
| CN104301671B CN104301671B (en) | 2017-09-29 |
Family
ID=52321207
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410490195.2A Active CN104301671B (en) | 2014-09-23 | 2014-09-23 | Traffic Surveillance Video storage method based on event closeness in HDFS |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104301671B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106021585A (en) * | 2016-06-02 | 2016-10-12 | 同济大学 | Traffic incident video access method and system based on time-space characteristics |
| CN106506665A (en) * | 2016-11-18 | 2017-03-15 | 郑州云海信息技术有限公司 | A kind of load-balancing method of distributed video monitoring system and platform |
| CN106686108A (en) * | 2017-01-13 | 2017-05-17 | 中电科新型智慧城市研究院有限公司 | Video monitoring method based on distributed detection technology |
| CN107592506A (en) * | 2017-09-26 | 2018-01-16 | 英华达（上海）科技有限公司 | A kind of monitoring method and supervising device, monitoring system |
| CN109089075A (en) * | 2018-07-10 | 2018-12-25 | 浙江工商大学 | Embedded across cloud intelligence memory method and system |
| CN109379563A (en) * | 2018-10-30 | 2019-02-22 | 华南师范大学 | The method and system of monitor video data storage management |
| WO2019169998A1 (en) * | 2018-03-08 | 2019-09-12 | 华为技术有限公司 | Method, system, and related apparatus for selecting data node |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007067974A (en) * | 2005-08-31 | 2007-03-15 | Toshiba Corp | Video monitoring system and video monitoring method |
| US20120147192A1 (en) * | 2009-09-01 | 2012-06-14 | Demaher Industrial Cameras Pty Limited | Video camera system |
| CN202663496U (en) * | 2011-10-28 | 2013-01-09 | 唐玉勇 | Intelligent surveillance system by utilizing video monitoring |
| CN103294701A (en) * | 2012-02-24 | 2013-09-11 | 联想(北京)有限公司 | Distributed file system and data processing method |
-
2014
- 2014-09-23 CN CN201410490195.2A patent/CN104301671B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007067974A (en) * | 2005-08-31 | 2007-03-15 | Toshiba Corp | Video monitoring system and video monitoring method |
| US20120147192A1 (en) * | 2009-09-01 | 2012-06-14 | Demaher Industrial Cameras Pty Limited | Video camera system |
| CN202663496U (en) * | 2011-10-28 | 2013-01-09 | 唐玉勇 | Intelligent surveillance system by utilizing video monitoring |
| CN103294701A (en) * | 2012-02-24 | 2013-09-11 | 联想(北京)有限公司 | Distributed file system and data processing method |
Non-Patent Citations (1)
| Title |
|---|
| 刘琨，钮文良: "一种改进的Hadoop数据负载均衡算法", 《河南理工大学学报（自然科学版）》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106021585A (en) * | 2016-06-02 | 2016-10-12 | 同济大学 | Traffic incident video access method and system based on time-space characteristics |
| CN106506665A (en) * | 2016-11-18 | 2017-03-15 | 郑州云海信息技术有限公司 | A kind of load-balancing method of distributed video monitoring system and platform |
| CN106506665B (en) * | 2016-11-18 | 2019-09-24 | 郑州云海信息技术有限公司 | A kind of load-balancing method and platform of distributed video monitoring system |
| CN106686108A (en) * | 2017-01-13 | 2017-05-17 | 中电科新型智慧城市研究院有限公司 | Video monitoring method based on distributed detection technology |
| CN107592506A (en) * | 2017-09-26 | 2018-01-16 | 英华达（上海）科技有限公司 | A kind of monitoring method and supervising device, monitoring system |
| CN107592506B (en) * | 2017-09-26 | 2020-06-30 | 英华达（上海）科技有限公司 | Monitoring method, monitoring device and monitoring system |
| WO2019169998A1 (en) * | 2018-03-08 | 2019-09-12 | 华为技术有限公司 | Method, system, and related apparatus for selecting data node |
| CN109089075A (en) * | 2018-07-10 | 2018-12-25 | 浙江工商大学 | Embedded across cloud intelligence memory method and system |
| CN109379563A (en) * | 2018-10-30 | 2019-02-22 | 华南师范大学 | The method and system of monitor video data storage management |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104301671B (en) | 2017-09-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104301671A (en) | Traffic monitoring video storing method in HDFS based on event intensity | |
| US11375250B2 (en) | Dynamic load balancing for video analytics pipelines | |
| US11836533B2 (en) | Automated reconfiguration of real time data stream processing | |
| US10855545B2 (en) | Centralized resource usage visualization service for large-scale network topologies | |
| US10594571B2 (en) | Dynamic scaling of storage volumes for storage client file systems | |
| CN105791431A (en) | On-line distributed monitoring video processing task scheduling method and device | |
| CN108701076B (en) | Distributed data set storage and retrieval | |
| US9432430B1 (en) | Event stream processing device providing real-time incident identification | |
| US10129118B1 (en) | Real time anomaly detection for data streams | |
| US20190230000A1 (en) | Intelligent analytic cloud provisioning | |
| US9647904B2 (en) | Customer-directed networking limits in distributed systems | |
| EP3074876B1 (en) | Customer-directed networking limits in distributed systems | |
| EP3361703B1 (en) | Load balancing method, related device and system | |
| US9436692B1 (en) | Large scale video analytics architecture | |
| KR20190111937A (en) | Apparatus and method for providing a virtual device | |
| CN104966006A (en) | Intelligent face identification system based on cloud variation platform | |
| CN108076317B (en) | Video data storage method and system, video access method and node | |
| KR20150052540A (en) | Method for transcoding mutimedia, and hadoop-based mulimedia transcoding system operating the same | |
| US9167313B1 (en) | Methods and system for transferring data for remote storage | |
| CN105683918A (en) | Centralized networking configuration in distributed systems | |
| US20160173620A1 (en) | Time-based data placement in a distributed storage system | |
| US9009533B2 (en) | Home/building fault analysis system using resource connection map log and method thereof | |
| CN113505260A (en) | Face recognition method and device, computer readable medium and electronic equipment | |
| CN202872848U (en) | Cloud storage terminal equipment based on cloud information and cloud computing services | |
| CN104811646A (en) | Multi-video stream data concurrent modulation and buffering storage method based on continuous storage model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |