CN104036025A - Distribution-base mass log collection system - Google Patents
Distribution-base mass log collection system Download PDFInfo
- Publication number
- CN104036025A CN104036025A CN201410299857.8A CN201410299857A CN104036025A CN 104036025 A CN104036025 A CN 104036025A CN 201410299857 A CN201410299857 A CN 201410299857A CN 104036025 A CN104036025 A CN 104036025A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- distributed
- storage
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a distribution-base mass log collection system comprising a data source layer, a distribution-type cache layer, a distribution-type storing and calculating layer, a bushiness processing layer, a visible display layer and a unified dispatching and managing module. The system has the advantages that the problem of log collection and high-speed storage can be solved effectively, the distribution-type storage and search engine technology is adopted, querying and retrieval speed can be increased, and the mass logs can be collected and analyzed accurately and accurately in high speed.
Description
Technical field
The present invention relates to field of computer technology, relate in particular to a kind of based on distributed massive logs acquisition system.
Background technology
Along with emergence and the development of cloud computing, mobile Internet, Internet of Things, the epoch of large data arrive.Large data service system is carried out to log collection and analysis, be evaluation node main frame health degree, problem that system is occurred analyze, show all kinds of forms according to place.
The massive logs solution of increasing income has a lot, such as the Scribe of FaceBook, the Kafka of the Chukwa of Apache, LinkedIn, the Flume of Cloudera etc., also there is comprehensive solution, for example Kibana+Elasticsearch+LogStash, LogAnalyzer+MySQL+RSyslog, Splunk.These solutions, they possess three basic modules, be respectively agent, collector and store, wherein agent encapsulation of data source, sends to collector by the data in data source, and collector receives the data of a plurality of agent, and gather in the store of rear importing rear end, store is centralized storage system, should have extensibility and reliability, should support current popular HDFS.The necessary installation and deployment of agent, on carry out the main frame of log collection, also will configure correlation parameter to determine the position of collector place main frame.But in the method, the installation and deployment of server end are cumbersome, major part all needs to use source code to compile installation, and, the software of increasing income is external software substantially, due to China's Mainland to the blockade of external most of website and abroad some website to Chinese blockade, therefore, when likely downloading depended software, the problem that there will be refusal to connect.
Storm is a real-time streams Computational frame, can to input source, process with nearly real-time speed then output.Storm has following characteristic:
Be easy to expansion.Storm is used Zookeeper to carry out cluster coordination, can fully guarantee so the good operation of large-scale cluster.
The processing of every message can be guaranteed.
Storm cluster management is simple and easy.
The fault-tolerant function of Storm is fine.Once Topology submits, Storm always bootup window until Topology abolished or be closed.And in commission occur when wrong, also can redistributing task by Storm.
General use Java, but the Topology in Storm can use any language design.
But the service logic of the Topology of Storm, needs oneself to write code and realize, must in Spout, originate by specific data.
Summary of the invention
The present invention is in order to solve deficiency of the prior art and defect, propose a kind of based on distributed massive logs acquisition system, can successfully manage log collection and the problem of putting in storage at a high speed, simultaneously, use distributed storage and search engine technique, accelerate to search and the speed of retrieving, thereby realized at a high speed, accurately, reliably, massive logs is gathered and analyze.
A kind of based on distributed massive logs acquisition system, it by installing Agent process on destination host, the text of destination host, application program, database log information are carried out to the directed unified access interface that is pushed to server cluster selectively, and server end has adopted distributed caching and real-time streams to process framework technology; This system comprises data source layer, distributed caching layer, distributed storage and computation layer, Business treatment, visual presentation layer and United Dispatching and administration module.
Data source layer, by data acquisition assembly (producer) module, to each, the text above node, application program, database gather, and are pushed to distributed caching layer.
Distributed caching layer, by LVS, the message queue assembly of each node is carried out to load balancing, provide a unified interface to receive also data writing source node and push the data of coming, wait for that the data acquisition assembly (consumer) of distributed storage and computation layer reads.
Distributed storage and computation layer, provide storage and the function of calculating, and comprises data acquisition assembly (consumer) module, calculated off-line module, real-time computing module, distributed storage and search engine; Wherein, data acquisition assembly (consumer) module is responsible for that distributed caching layer is carried out to data and is read; Calculated off-line module is comprised of Hadoop and the ecosystem thereof; Computing module is comprised of Storm in real time.
Business treatment, provides function and the service of statistical study and data mining, by upper strata, is called.
Visual presentation layer, provides common inquiry, full-text search, form the function such as to show, import and export.
United Dispatching and administration module, carry out unified scheduling and management to above-mentioned 5 layers, and based on workflow, robotization is processed.
The beneficial effect that technical solution of the present invention is brought:
1, owing to having used Storm real-time streams to process framework technology, therefore, can closely in real time the message in Kafka distributed caching be processed and be classified.
2, pushing daily record is asynchronous carrying out with reading daily record, can, without the mutual existence of perception, only need to write or read toward Distributed Message Queue Kafka the inside.
3, per minute can be processed 1,000,000 records, can successfully manage the problem of large data data acquisition, produces economic benefit.
4, system has high reliability, guarantees that every is recorded processed.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, to the accompanying drawing of required use in embodiment or description of the Prior Art be briefly described below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skills, do not paying under the prerequisite of creative work, can also obtain according to these accompanying drawings other accompanying drawing.
Fig. 1 is the system architecture diagram of distributed data acquisition system in the present invention;
Fig. 2 is the structure composed figure of data acquisition subsystem in the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is only the present invention's part embodiment, rather than whole embodiment.Embodiment based in the present invention, those of ordinary skills, not making the every other embodiment obtaining under creative work prerequisite, belong to the scope of protection of the invention.
Proposed a kind of based on distributed massive logs acquisition system herein, by Agent process is installed on destination host, the text of destination host, application program, database log information are carried out to the directed unified access interface that is pushed to server cluster selectively, and server end has adopted distributed caching and real-time streams to process framework technology.By the invention solves in prior art, use traditional log collection technology, in the process that massive logs is gathered, there will be uploading speed to surpass acquisition process storage, cause the seemingly-dead or log recording of system to be lost; The distributed information log acquisition component that use is increased income, need to be at each node installation and configuration Agent, some Agent need to rely on a lot of third party softwares, due to reasons such as networks, shortcoming or deficiency that cannot installation, can successfully manage log collection and the problem of warehouse-in at a high speed, simultaneously, use distributed storage and search engine technique, accelerate to search and the speed of retrieving, thereby realized at a high speed, accurately, reliably, massive logs is gathered and analyze.
Be illustrated in figure 1 a kind of system architecture diagram based on distributed massive logs acquisition system.
Native system comprises data source layer, distributed caching layer, distributed storage and computation layer, Business treatment, visual presentation layer and United Dispatching and administration module.
Data source layer, by data acquisition assembly (producer) module, to each, the text above node, application program, database gather, and are pushed to distributed caching layer.
Distributed caching layer, by LVS, the message queue assembly of each node is carried out to load balancing, provide a unified interface to receive also data writing source node and push the data of coming, wait for that the data acquisition assembly (consumer) of distributed storage and computation layer reads.
Distributed storage and computation layer, provide storage and the function of calculating, and comprises data acquisition assembly (consumer) module, calculated off-line module, real-time computing module, distributed storage and search engine.Wherein, data acquisition assembly (consumer) module is responsible for that distributed caching layer is carried out to data and is read; Calculated off-line module is comprised of Hadoop and the ecosystem thereof; Computing module is comprised of Storm in real time.
Business treatment, provides function and the service of statistical study and data mining, by upper strata, is called.
Visual presentation layer, provides common inquiry, full-text search, form the function such as to show, import and export.
United Dispatching and administration module, carry out unified scheduling and management to above-mentioned 5 layers, and based on workflow, robotization is processed.
Data acquisition subsystem is the core component of distributed information log acquisition system, comprises data source layer, distributed caching layer, data analysis layer and data persistence layer, and it can independently become a new system.Wherein, data source layer is the data source layer of distributed information log acquisition system; Distributed caching layer is the distributed caching layer of distributed information log acquisition system; Data analysis layer is the distributed storage of distributed information log acquisition system and the real-time computing module of computation layer and United Dispatching and administration module; Data persistence layer is the distributed storage of distributed information log acquisition system and the distributed storage module of computation layer.
The composition of data acquisition subsystem, as shown in Figure 2.
The core of data acquisition subsystem, mainly comprises data acquisition assembly (producer) module, distributed caching layer and the data analysis layer that are positioned at data source layer.
Data source layer, comprise a lot of back end main frames, data acquisition assembly (producer) module has been installed on each main frame, can start Agent process and be responsible for the acquisition instructions of reception service end, the text of this node, application program, database data-pushing in the unified access interface in server cluster.
Distributed caching layer, is positioned at server cluster the inside, and its certain node main frame being mainly responsible for data to be automatically forwarded in Kafka cluster by LVS writes, and is responsible for externally providing unified access interface and distributed caching service.Wherein unified access interface comprises unified domain name/host name and port.
Data analysis layer, mainly consists of Storm, and wherein Topology consists of 1 Spout and several Bolt.KafkaReaderSpout is responsible for reading the data of Kafka cluster the inside, then data transmission, give ExecutorBolt, ExecutorBolt is according to the type of service logic and daily record, hands down to transmit data in HbaseWriterBolt, HdfsWriterBolt or SolrWriterBolt, to carry out persistent storage.Wherein, HbaseWriterBolt is responsible for the persistent storage of HBase, and HdfsWriterBolt is responsible for the persistent storage of HDFS, the persistent storage that SolrWriterBolt is responsible for Solr-Cloud.
Data persistence layer, is mainly comprised of HBase, HDFS and Solr-Cloud.HBase is mainly key-value pair storage, the daily record after storage original log, classification are processed; HDFS mainly stores original log; Solr-Cloud mainly carries out index to the daily record of HBase storage, accelerates retrieval rate, can full-text search.
The workflow of data acquisition subsystem:
First, the text, application program, database data that the Agent process of all node main frames of data source can push this node is automatically in the unified access interface in server cluster.
Then, unified access interface receives data, by LVS, is automatically forwarded in certain Kafka node and data writing, and wait is read.
Finally, the Topology in real-time computing module Storm carries out business processing, and wherein Topology consists of 1 Spout and several Bolt.KafkaReaderSpout is responsible for reading the data in Kafka cluster, then data transmission, give ExecutorBolt, ExecutorBolt is according to the type of service logic and daily record, hands down to transmit data in HbaseWriterBolt, HdfsWriterBolt or SolrWriterBolt, to carry out persistent storage.Wherein, HbaseWriterBolt is responsible for the persistent storage of HBase, and HdfsWriterBolt is responsible for the persistent storage of HDFS, the persistent storage that SolrWriterBolt is responsible for Solr-Cloud.
What above the embodiment of the present invention is provided is a kind ofly described in detail based on distributed massive logs acquisition system, applied specific case herein principle of the present invention and embodiment are set forth, the explanation of above embodiment is just for helping to understand method of the present invention and core concept thereof; , for one of ordinary skill in the art, according to thought of the present invention, all will change in specific embodiments and applications, in sum, this description should not be construed as limitation of the present invention meanwhile.
Claims (5)
1. one kind based on distributed massive logs acquisition system, it is characterized in that, this system by installing Agent process on destination host, the text of destination host, application program, database log information are carried out to the directed unified access interface that is pushed to server cluster selectively, and server end has adopted distributed caching and real-time streams to process framework technology; This system comprises data source layer, distributed caching layer, distributed storage and computation layer, Business treatment, visual presentation layer and United Dispatching and administration module;
Data source layer, by data acquisition assembly (producer) module, to each, the text above node, application program, database gather, and are pushed to distributed caching layer;
Distributed caching layer, by LVS, the message queue assembly of each node is carried out to load balancing, provide a unified interface to receive also data writing source node and push the data of coming, wait for that the data acquisition assembly (consumer) of distributed storage and computation layer reads;
Distributed storage and computation layer, provide storage and the function of calculating, and comprises data acquisition assembly (consumer) module, calculated off-line module, real-time computing module, distributed storage and search engine; Wherein, data acquisition assembly (consumer) module is responsible for that distributed caching layer is carried out to data and is read; Calculated off-line module is comprised of Hadoop and the ecosystem thereof; Computing module is comprised of Storm in real time;
Business treatment, provides function and the service of statistical study and data mining, by upper strata, is called;
Visual presentation layer, provides common inquiry, full-text search, form the function such as to show, import and export;
United Dispatching and administration module, carry out unified scheduling and management to above-mentioned 5 layers, and based on workflow, robotization is processed.
2. system according to claim 1, it is characterized in that, the real-time computing module of the data source layer of this system, distributed caching layer, distributed storage and computation layer and distributed storage module, United Dispatching and administration module are the cores of distributed information log acquisition system, it is data acquisition subsystem, it comprises data source layer, distributed caching layer, data analysis layer and data persistence layer, and it can independently become a new system; Wherein, data source layer is the data source layer of distributed information log acquisition system; Distributed caching layer is the distributed caching layer of distributed information log acquisition system; Data analysis layer is the distributed storage of distributed information log acquisition system and the real-time computing module of computation layer and United Dispatching and administration module; Data persistence layer is the distributed storage of distributed information log acquisition system and the distributed storage module of computation layer.
3. system according to claim 2, it is characterized in that, data source layer, comprise a lot of back end main frames, data acquisition assembly (producer) module has been installed on each main frame, can start Agent process and be responsible for the acquisition instructions of reception service end, the text of this node, application program, database data-pushing in the unified access interface in server cluster;
Distributed caching layer, is positioned at server cluster the inside, and its certain node main frame being mainly responsible for data to be automatically forwarded in Kafka cluster by LVS writes, and is responsible for externally providing unified access interface and distributed caching service; Wherein unified access interface comprises unified domain name/host name and port;
Data analysis layer, consists of Storm, and wherein Topology consists of 1 Spout and several Bolt.KafkaReaderSpout is responsible for reading the data of Kafka cluster the inside, then data transmission, give ExecutorBolt, ExecutorBolt is according to the type of service logic and daily record, hands down to transmit data in HbaseWriterBolt, HdfsWriterBolt or SolrWriterBolt, to carry out persistent storage; Wherein, HbaseWriterBolt is responsible for the persistent storage of HBase, and HdfsWriterBolt is responsible for the persistent storage of HDFS, the persistent storage that SolrWriterBolt is responsible for Solr-Cloud;
Data persistence layer, is comprised of HBase, HDFS and Solr-Cloud.HBase is mainly key-value pair storage, the daily record after storage original log, classification are processed; HDFS mainly stores original log; Solr-Cloud mainly carries out index to the daily record of HBase storage, accelerates retrieval rate, can full-text search.
4. system according to claim 3, is characterized in that, the core of data acquisition subsystem comprises data acquisition assembly (producer) module, distributed caching layer and the data analysis layer that are positioned at data source layer.
5. according to the system described in claim 3 or 4, it is characterized in that, the workflow of data acquisition subsystem is: first, the text, application program, database data that the Agent process of all node main frames of data source can push this node is automatically in the unified access interface in server cluster;
Then, unified access interface receives data, by LVS, is automatically forwarded in certain Kafka node and data writing, and wait is read;
Finally, the Topology in real-time computing module Storm carries out business processing, and wherein Topology consists of 1 Spout and several Bolt; KafkaReaderSpout is responsible for reading the data in Kafka cluster, then data transmission, give ExecutorBolt, ExecutorBolt is according to the type of service logic and daily record, hands down to transmit data in HbaseWriterBolt, HdfsWriterBolt or SolrWriterBolt, to carry out persistent storage; Wherein, HbaseWriterBolt is responsible for the persistent storage of HBase, and HdfsWriterBolt is responsible for the persistent storage of HDFS, the persistent storage that SolrWriterBolt is responsible for Solr-Cloud.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410299857.8A CN104036025A (en) | 2014-06-27 | 2014-06-27 | Distribution-base mass log collection system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410299857.8A CN104036025A (en) | 2014-06-27 | 2014-06-27 | Distribution-base mass log collection system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104036025A true CN104036025A (en) | 2014-09-10 |
Family
ID=51466795
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410299857.8A Pending CN104036025A (en) | 2014-06-27 | 2014-06-27 | Distribution-base mass log collection system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104036025A (en) |
Cited By (103)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104239197A (en) * | 2014-10-10 | 2014-12-24 | 浪潮电子信息产业股份有限公司 | Administrative user abnormal behavior detection method based on big data log analysis |
| CN104468710A (en) * | 2014-10-31 | 2015-03-25 | 西安未来国际信息股份有限公司 | Mixed big data processing system and method |
| CN104579789A (en) * | 2015-01-23 | 2015-04-29 | 广东能龙教育股份有限公司 | Massive user behavior data acquisition method and system based on message queue |
| CN104615777A (en) * | 2015-02-27 | 2015-05-13 | 浪潮集团有限公司 | Method and device for real-time data processing based on stream-oriented calculation engine |
| CN104714875A (en) * | 2015-03-11 | 2015-06-17 | 浪潮集团有限公司 | Distributed automatic collecting method |
| CN104750870A (en) * | 2015-04-20 | 2015-07-01 | 河海大学 | Log storage system based on HBase and implementation method |
| CN104915259A (en) * | 2015-06-15 | 2015-09-16 | 浪潮软件集团有限公司 | Task scheduling method applied to distributed acquisition system |
| CN104933160A (en) * | 2015-06-26 | 2015-09-23 | 河海大学 | ETL (Extract Transform and Load) framework design method for safety monitoring business analysis |
| CN105138592A (en) * | 2015-07-31 | 2015-12-09 | 武汉虹信技术服务有限责任公司 | Distributed framework-based log data storing and retrieving method |
| CN105205105A (en) * | 2015-08-27 | 2015-12-30 | 浪潮集团有限公司 | Data ETL (Extract Transform Load) system based on storm and treatment method based on storm |
| CN105227619A (en) * | 2015-07-06 | 2016-01-06 | 国网天津市电力公司 | A kind of method of the remote monitoring data storehouse running status based on Agent |
| CN105243147A (en) * | 2015-10-22 | 2016-01-13 | 浪潮(北京)电子信息产业有限公司 | Slow query log management method and system of MySQL database |
| CN105260452A (en) * | 2015-10-12 | 2016-01-20 | 成都视达科信息技术有限公司 | System and method for collecting, searching, and analyzing offline log |
| CN105391584A (en) * | 2015-11-30 | 2016-03-09 | 用友网络科技股份有限公司 | Abnormity early warning system for use in distributed environment |
| CN105450618A (en) * | 2014-09-26 | 2016-03-30 | Tcl集团股份有限公司 | Operation method and operation system of big data process through API (Application Programming Interface) server |
| CN105468720A (en) * | 2015-11-20 | 2016-04-06 | 北京锐安科技有限公司 | Method for integrating distributed data processing systems, corresponding systems and data processing method |
| CN105577422A (en) * | 2015-11-11 | 2016-05-11 | 江苏瑞中数据股份有限公司 | Energy internet real-time data analysis system and method thereof |
| CN105577431A (en) * | 2015-12-11 | 2016-05-11 | 青岛云成互动网络有限公司 | User information identification and classification method based on internet application and system thereof |
| CN105608188A (en) * | 2015-12-23 | 2016-05-25 | 北京奇虎科技有限公司 | Data processing method and data processing device |
| CN105677836A (en) * | 2016-01-05 | 2016-06-15 | 北京汇商融通信息技术有限公司 | Big data processing and solving system simultaneously supporting offline data and real-time online data |
| CN105824618A (en) * | 2016-03-10 | 2016-08-03 | 浪潮软件集团有限公司 | Real-time message processing method for Storm |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN105824945A (en) * | 2016-03-21 | 2016-08-03 | 中国电力科学研究院 | Method for collecting global energy Internet technology resource data |
| CN105868075A (en) * | 2016-03-31 | 2016-08-17 | 浪潮通信信息***有限公司 | System and method for monitoring and analyzing great deal of logs in real time |
| CN105869396A (en) * | 2016-04-28 | 2016-08-17 | 泰华智慧产业集团股份有限公司 | Vehicle crossing index statistical method and system based on big data platform |
| CN105933736A (en) * | 2016-04-18 | 2016-09-07 | 天脉聚源(北京)传媒科技有限公司 | Log processing method and device |
| CN105979297A (en) * | 2016-06-14 | 2016-09-28 | 天脉聚源(北京)传媒科技有限公司 | Watching duration statistic method and system |
| WO2016187967A1 (en) * | 2015-05-28 | 2016-12-01 | 中兴通讯股份有限公司 | Method and apparatus for realizing log transmission |
| CN106227877A (en) * | 2016-08-02 | 2016-12-14 | 北京集奥聚合科技有限公司 | A kind of distributed information log acquisition system based on hadoop and method |
| CN106294556A (en) * | 2016-07-26 | 2017-01-04 | 江苏神州信源***工程有限公司 | A kind of method using Apache Drill to concentrate search large database concept |
| CN104331044B (en) * | 2014-10-17 | 2017-01-11 | 东北大学 | Electro-fused magnesia furnace power consumption remote monitoring system and method |
| CN106383758A (en) * | 2016-09-22 | 2017-02-08 | 郑州云海信息技术有限公司 | Operation system-based information acquisition method |
| CN106411639A (en) * | 2016-09-18 | 2017-02-15 | 合网络技术（北京）有限公司 | Method and system for monitoring access data |
| CN106445790A (en) * | 2016-10-12 | 2017-02-22 | 北京集奥聚合科技有限公司 | Counting and account-checking method and device used in distributed real-time computing system |
| CN106503079A (en) * | 2016-10-10 | 2017-03-15 | 语联网（武汉）信息技术有限公司 | A kind of blog management method and system |
| CN106528847A (en) * | 2016-11-24 | 2017-03-22 | 北京集奥聚合科技有限公司 | Multi-dimensional processing method and system for massive data |
| CN106557561A (en) * | 2016-11-16 | 2017-04-05 | 贵州大学 | Magnanimity sensing data storage system and method based on HBase |
| CN106649670A (en) * | 2016-12-14 | 2017-05-10 | 北京五八信息技术有限公司 | Streaming computing-based data monitoring method and apparatus |
| CN106649377A (en) * | 2015-11-02 | 2017-05-10 | 中兴通讯股份有限公司 | Image processing system and method |
| CN106682073A (en) * | 2016-11-14 | 2017-05-17 | 上海轻维软件有限公司 | HBase fuzzy retrieval system based on Elastic Search |
| CN106790572A (en) * | 2016-12-27 | 2017-05-31 | 广州华多网络科技有限公司 | The system and method that a kind of distributed information log is collected |
| CN106933720A (en) * | 2017-01-16 | 2017-07-07 | 国家电网公司 | Network log information security scene-type analysis system and its analysis method |
| CN106980678A (en) * | 2017-03-30 | 2017-07-25 | 温馨港网络信息科技（苏州）有限公司 | Data analysing method and system based on zookeeper technologies |
| CN107016133A (en) * | 2017-05-24 | 2017-08-04 | 成都享之道网络科技有限公司 | Based on the online big data system with offline double processing |
| CN107229639A (en) * | 2016-03-24 | 2017-10-03 | 上海宝信软件股份有限公司 | The storage system of distributing real-time data bank |
| CN107357804A (en) * | 2017-05-24 | 2017-11-17 | 上海你我贷互联网金融信息服务有限公司 | The analysis system and method for internet finance massive logs |
| WO2017198227A1 (en) * | 2016-05-19 | 2017-11-23 | 中兴通讯股份有限公司 | Interactive internet protocol television system and real-time acquisition method for user data |
| CN107463490A (en) * | 2017-08-15 | 2017-12-12 | 四川长虹电器股份有限公司 | A kind of cluster daily record centralized collection method being applied in platform development |
| CN107508888A (en) * | 2017-08-25 | 2017-12-22 | 同方(深圳)云计算技术股份有限公司 | A kind of car networking service platform |
| CN107590182A (en) * | 2017-08-03 | 2018-01-16 | 华南理工大学 | A kind of distributed information log collection method |
| CN107622084A (en) * | 2017-08-10 | 2018-01-23 | 深圳前海微众银行股份有限公司 | Blog management method, system and computer-readable recording medium |
| US9900317B2 (en) | 2016-02-25 | 2018-02-20 | Red Hat, Inc. | Access guards for multi-tenant logging |
| CN107766147A (en) * | 2016-08-23 | 2018-03-06 | 上海宝信软件股份有限公司 | Distributed data analysis task scheduling system |
| CN107786565A (en) * | 2017-11-02 | 2018-03-09 | 江苏物联网研究发展中心 | A kind of distributed real-time intrusion detection method and detecting system |
| CN107800592A (en) * | 2017-11-09 | 2018-03-13 | 郑州云海信息技术有限公司 | A kind of server test results acquisition method |
| CN107872465A (en) * | 2017-12-05 | 2018-04-03 | 全球能源互联网研究院有限公司 | A kind of distributed network security monitoring method and system |
| CN107959697A (en) * | 2016-10-17 | 2018-04-24 | 腾讯科技（深圳）有限公司 | Source Data Acquisition method and system in big data off-line calculation |
| CN108073625A (en) * | 2016-11-14 | 2018-05-25 | 北京京东尚科信息技术有限公司 | For the system and method for metadata information management |
| CN108073705A (en) * | 2017-12-18 | 2018-05-25 | 郑州云海信息技术有限公司 | A kind of distributed mass data polymerize acquisition method |
| CN108076111A (en) * | 2016-11-15 | 2018-05-25 | 亿阳安全技术有限公司 | A kind of system and method for distributing data in big data platform |
| CN108153828A (en) * | 2017-12-12 | 2018-06-12 | 顺丰科技有限公司 | A kind of persistence method of real time data, device and equipment, storage medium |
| WO2018130222A1 (en) * | 2017-01-16 | 2018-07-19 | 贵州白山云科技有限公司 | Method for writing to log, system, medium, and device |
| CN108647323A (en) * | 2018-05-11 | 2018-10-12 | 重庆工商职业学院 | A kind of data summarization method of vocational ability |
| CN108710563A (en) * | 2018-05-16 | 2018-10-26 | 广州市千钧网络科技有限公司 | A kind of Application Logging method and device |
| CN108717632A (en) * | 2018-05-29 | 2018-10-30 | 广东通莞科技股份有限公司 | A kind of mobile payment storage protection and recovery system |
| CN108763310A (en) * | 2018-04-25 | 2018-11-06 | 江苏鸣鹤云科技有限公司 | A kind of big data platform of High Availabitity |
| CN108810146A (en) * | 2018-06-14 | 2018-11-13 | 鹤壁北岩科技有限公司 | Culture control system based on distributed cloud platform and control method |
| CN108829879A (en) * | 2018-06-26 | 2018-11-16 | 天津城建大学 | A kind of charging pile data monitoring method |
| CN108932352A (en) * | 2018-09-26 | 2018-12-04 | 江苏曲速教育科技有限公司 | Distributed cache system based on statistical server |
| CN108985981A (en) * | 2018-06-28 | 2018-12-11 | 北京奇虎科技有限公司 | Data processing system and method |
| CN109151464A (en) * | 2018-11-14 | 2019-01-04 | 江苏鸿信***集成有限公司 | IPTV set top box failure real-time detection method based on high amount of traffic processing |
| CN109167672A (en) * | 2018-07-13 | 2019-01-08 | 腾讯科技（深圳）有限公司 | One kind returning source location of mistake method, apparatus, storage medium and system |
| CN109445949A (en) * | 2018-12-07 | 2019-03-08 | 武汉轻工大学 | A kind of data collection system and collecting method |
| CN109542638A (en) * | 2018-10-26 | 2019-03-29 | 深圳点猫科技有限公司 | A kind of document handling method and device based on educational system |
| CN109542750A (en) * | 2018-11-26 | 2019-03-29 | 深圳天源迪科信息技术股份有限公司 | Distributed information log system |
| CN109657125A (en) * | 2018-12-14 | 2019-04-19 | 平安城市建设科技(深圳)有限公司 | Data processing method, device, equipment and storage medium based on web crawlers |
| CN109710731A (en) * | 2018-11-19 | 2019-05-03 | 北京计算机技术及应用研究所 | A kind of multidirectional processing system of data flow based on Flink |
| US10289383B2 (en) | 2016-07-28 | 2019-05-14 | International Business Machines Corporation | Cross object synchronization |
| CN109857613A (en) * | 2018-12-25 | 2019-06-07 | 南京南瑞信息通信科技有限公司 | A kind of automation operational system based on acquisition cluster |
| CN109886327A (en) * | 2019-02-12 | 2019-06-14 | 北京奇艺世纪科技有限公司 | The processing system and method for Java data in a kind of distributed system |
| CN110232054A (en) * | 2019-06-19 | 2019-09-13 | 北京百度网讯科技有限公司 | Log transmission system and streaming log transmission method |
| CN110245120A (en) * | 2019-06-19 | 2019-09-17 | 北京百度网讯科技有限公司 | The daily record data processing method of streaming computing system and streaming computing system |
| CN110262951A (en) * | 2019-06-10 | 2019-09-20 | 天翼电子商务有限公司 | A kind of business second grade monitoring method and system, storage medium and client |
| CN110309130A (en) * | 2018-03-21 | 2019-10-08 | 中国人民财产保险股份有限公司 | A kind of method and device for host performance monitor |
| US10467757B2 (en) | 2015-11-30 | 2019-11-05 | Shanghai United Imaging Healthcare Co., Ltd. | System and method for computer aided diagnosis |
| CN110442503A (en) * | 2019-07-29 | 2019-11-12 | 深圳数位传媒科技有限公司 | A kind of alarm method and device using log index |
| CN110659306A (en) * | 2019-08-29 | 2020-01-07 | 达疆网络科技(上海)有限公司 | Store system all-level cache synchronization mode |
| CN110688399A (en) * | 2019-08-26 | 2020-01-14 | 远光软件股份有限公司 | Stream type calculation real-time report system and method |
| CN111157838A (en) * | 2018-11-08 | 2020-05-15 | 中国铁路沈阳局集团有限公司科学技术研究所 | Intelligent management system for big data of running state of railway power distribution network |
| CN111200637A (en) * | 2019-12-20 | 2020-05-26 | 新浪网技术(中国)有限公司 | Cache processing method and device |
| CN111209314A (en) * | 2020-01-13 | 2020-05-29 | 国网浙江省电力有限公司信息通信分公司 | System for processing massive log data of power information system in real time |
| CN111258979A (en) * | 2020-01-16 | 2020-06-09 | 山东大学 | Cloud protection log system and working method thereof |
| CN111355788A (en) * | 2020-02-21 | 2020-06-30 | 深圳供电局有限公司 | Distributed data center management system |
| CN111552628A (en) * | 2020-03-20 | 2020-08-18 | 北京海致星图科技有限公司 | Distributed pressure measurement system and method for graph database and graph service interface |
| WO2020168756A1 (en) * | 2019-02-19 | 2020-08-27 | 平安科技（深圳）有限公司 | Cluster log feature extraction method, and apparatus, device and storage medium |
| CN111695126A (en) * | 2020-05-28 | 2020-09-22 | 武汉中海庭数据技术有限公司 | Crowdsourcing data decryption method and device, electronic equipment and storage medium |
| CN112163060A (en) * | 2020-09-16 | 2021-01-01 | 安徽龙运智能科技有限公司 | System for processing mass GPS data by big data technology |
| CN112347073A (en) * | 2020-10-27 | 2021-02-09 | 山东开创云计算有限公司 | Multiple data system |
| CN112433998A (en) * | 2020-11-20 | 2021-03-02 | 广东电网有限责任公司佛山供电局 | Multisource heterogeneous data acquisition and convergence system and method based on power system |
| CN112818045A (en) * | 2021-01-22 | 2021-05-18 | 辽宁长江智能科技股份有限公司 | Data access unified management platform for big data |
| CN113220655A (en) * | 2021-04-30 | 2021-08-06 | 中核武汉核电运行技术股份有限公司 | Data access method, device, equipment and readable storage medium |
| CN114996335A (en) * | 2022-08-03 | 2022-09-02 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
| CN115022402A (en) * | 2022-07-01 | 2022-09-06 | 杭州乘云数字技术有限公司 | Agent acquisition method and system based on one-stack integration technology |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050049924A1 (en) * | 2003-08-27 | 2005-03-03 | Debettencourt Jason | Techniques for use with application monitoring to obtain transaction data |
| CN103138989A (en) * | 2013-02-25 | 2013-06-05 | 武汉华工安鼎信息技术有限责任公司 | System and method for analyzing large number of logs |
| CN103399887A (en) * | 2013-07-19 | 2013-11-20 | 蓝盾信息安全技术股份有限公司 | Query and statistical analysis system for mass logs |
| CN103401934A (en) * | 2013-08-06 | 2013-11-20 | 广州唯品会信息科技有限公司 | Method and system for acquiring log data |
-
2014
- 2014-06-27 CN CN201410299857.8A patent/CN104036025A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050049924A1 (en) * | 2003-08-27 | 2005-03-03 | Debettencourt Jason | Techniques for use with application monitoring to obtain transaction data |
| CN103138989A (en) * | 2013-02-25 | 2013-06-05 | 武汉华工安鼎信息技术有限责任公司 | System and method for analyzing large number of logs |
| CN103399887A (en) * | 2013-07-19 | 2013-11-20 | 蓝盾信息安全技术股份有限公司 | Query and statistical analysis system for mass logs |
| CN103401934A (en) * | 2013-08-06 | 2013-11-20 | 广州唯品会信息科技有限公司 | Method and system for acquiring log data |
Non-Patent Citations (1)
| Title |
|---|
| 陈涛: ""分布式日志数据采集代理框架的研究与设计"", 《万方数据企业知识服务平台》 * |
Cited By (130)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105450618A (en) * | 2014-09-26 | 2016-03-30 | Tcl集团股份有限公司 | Operation method and operation system of big data process through API (Application Programming Interface) server |
| CN104239197A (en) * | 2014-10-10 | 2014-12-24 | 浪潮电子信息产业股份有限公司 | Administrative user abnormal behavior detection method based on big data log analysis |
| CN104331044B (en) * | 2014-10-17 | 2017-01-11 | 东北大学 | Electro-fused magnesia furnace power consumption remote monitoring system and method |
| CN104468710A (en) * | 2014-10-31 | 2015-03-25 | 西安未来国际信息股份有限公司 | Mixed big data processing system and method |
| CN104579789A (en) * | 2015-01-23 | 2015-04-29 | 广东能龙教育股份有限公司 | Massive user behavior data acquisition method and system based on message queue |
| CN104615777A (en) * | 2015-02-27 | 2015-05-13 | 浪潮集团有限公司 | Method and device for real-time data processing based on stream-oriented calculation engine |
| CN104714875A (en) * | 2015-03-11 | 2015-06-17 | 浪潮集团有限公司 | Distributed automatic collecting method |
| CN104750870A (en) * | 2015-04-20 | 2015-07-01 | 河海大学 | Log storage system based on HBase and implementation method |
| CN104750870B (en) * | 2015-04-20 | 2018-11-27 | 河海大学 | A kind of log storage system and implementation method based on HBase |
| CN106301844A (en) * | 2015-05-28 | 2017-01-04 | 中兴通讯股份有限公司 | A kind of method and device realizing log transmission |
| CN106301844B (en) * | 2015-05-28 | 2020-03-24 | 中兴通讯股份有限公司 | Method and device for realizing log transmission |
| WO2016187967A1 (en) * | 2015-05-28 | 2016-12-01 | 中兴通讯股份有限公司 | Method and apparatus for realizing log transmission |
| CN104915259A (en) * | 2015-06-15 | 2015-09-16 | 浪潮软件集团有限公司 | Task scheduling method applied to distributed acquisition system |
| CN104933160B (en) * | 2015-06-26 | 2019-06-28 | 河海大学 | A kind of ETL frame design method towards safety monitoring business diagnosis |
| CN104933160A (en) * | 2015-06-26 | 2015-09-23 | 河海大学 | ETL (Extract Transform and Load) framework design method for safety monitoring business analysis |
| CN105227619A (en) * | 2015-07-06 | 2016-01-06 | 国网天津市电力公司 | A kind of method of the remote monitoring data storehouse running status based on Agent |
| CN105138592A (en) * | 2015-07-31 | 2015-12-09 | 武汉虹信技术服务有限责任公司 | Distributed framework-based log data storing and retrieving method |
| CN105138592B (en) * | 2015-07-31 | 2019-03-26 | 武汉虹信技术服务有限责任公司 | A kind of daily record data storage and search method based on distributed structure/architecture |
| CN105205105B (en) * | 2015-08-27 | 2019-04-16 | 浪潮集团有限公司 | A kind of ETL process system and processing method based on storm |
| CN105205105A (en) * | 2015-08-27 | 2015-12-30 | 浪潮集团有限公司 | Data ETL (Extract Transform Load) system based on storm and treatment method based on storm |
| CN105260452A (en) * | 2015-10-12 | 2016-01-20 | 成都视达科信息技术有限公司 | System and method for collecting, searching, and analyzing offline log |
| CN105243147A (en) * | 2015-10-22 | 2016-01-13 | 浪潮(北京)电子信息产业有限公司 | Slow query log management method and system of MySQL database |
| CN106649377A (en) * | 2015-11-02 | 2017-05-10 | 中兴通讯股份有限公司 | Image processing system and method |
| CN105577422A (en) * | 2015-11-11 | 2016-05-11 | 江苏瑞中数据股份有限公司 | Energy internet real-time data analysis system and method thereof |
| CN105468720A (en) * | 2015-11-20 | 2016-04-06 | 北京锐安科技有限公司 | Method for integrating distributed data processing systems, corresponding systems and data processing method |
| US10467757B2 (en) | 2015-11-30 | 2019-11-05 | Shanghai United Imaging Healthcare Co., Ltd. | System and method for computer aided diagnosis |
| CN105391584A (en) * | 2015-11-30 | 2016-03-09 | 用友网络科技股份有限公司 | Abnormity early warning system for use in distributed environment |
| US10825180B2 (en) | 2015-11-30 | 2020-11-03 | Shanghai United Imaging Healthcare Co., Ltd. | System and method for computer aided diagnosis |
| CN105577431A (en) * | 2015-12-11 | 2016-05-11 | 青岛云成互动网络有限公司 | User information identification and classification method based on internet application and system thereof |
| CN105608188A (en) * | 2015-12-23 | 2016-05-25 | 北京奇虎科技有限公司 | Data processing method and data processing device |
| CN105677836A (en) * | 2016-01-05 | 2016-06-15 | 北京汇商融通信息技术有限公司 | Big data processing and solving system simultaneously supporting offline data and real-time online data |
| US10609035B2 (en) | 2016-02-25 | 2020-03-31 | Red Hat, Inc. | Access guards for multi-tenant logging |
| US10263993B2 (en) | 2016-02-25 | 2019-04-16 | Red Hat, Inc. | Access guards for multi-tenant logging |
| US9900317B2 (en) | 2016-02-25 | 2018-02-20 | Red Hat, Inc. | Access guards for multi-tenant logging |
| CN105824618A (en) * | 2016-03-10 | 2016-08-03 | 浪潮软件集团有限公司 | Real-time message processing method for Storm |
| CN105824744B (en) * | 2016-03-21 | 2018-06-15 | 焦点科技股份有限公司 | A kind of real-time logs capturing analysis method based on B2B platform |
| CN105824744A (en) * | 2016-03-21 | 2016-08-03 | 焦点科技股份有限公司 | Real-time log collection and analysis method on basis of B2B (Business to Business) platform |
| CN105824945A (en) * | 2016-03-21 | 2016-08-03 | 中国电力科学研究院 | Method for collecting global energy Internet technology resource data |
| CN107229639B (en) * | 2016-03-24 | 2020-07-28 | 上海宝信软件股份有限公司 | Storage system of distributed real-time database |
| CN107229639A (en) * | 2016-03-24 | 2017-10-03 | 上海宝信软件股份有限公司 | The storage system of distributing real-time data bank |
| CN105868075A (en) * | 2016-03-31 | 2016-08-17 | 浪潮通信信息***有限公司 | System and method for monitoring and analyzing great deal of logs in real time |
| CN105933736A (en) * | 2016-04-18 | 2016-09-07 | 天脉聚源(北京)传媒科技有限公司 | Log processing method and device |
| CN105869396A (en) * | 2016-04-28 | 2016-08-17 | 泰华智慧产业集团股份有限公司 | Vehicle crossing index statistical method and system based on big data platform |
| WO2017198227A1 (en) * | 2016-05-19 | 2017-11-23 | 中兴通讯股份有限公司 | Interactive internet protocol television system and real-time acquisition method for user data |
| CN105979297A (en) * | 2016-06-14 | 2016-09-28 | 天脉聚源(北京)传媒科技有限公司 | Watching duration statistic method and system |
| CN105979297B (en) * | 2016-06-14 | 2019-03-19 | 天脉聚源(北京)传媒科技有限公司 | One kind watching duration statistical method and system |
| CN106294556A (en) * | 2016-07-26 | 2017-01-04 | 江苏神州信源***工程有限公司 | A kind of method using Apache Drill to concentrate search large database concept |
| US10289383B2 (en) | 2016-07-28 | 2019-05-14 | International Business Machines Corporation | Cross object synchronization |
| CN106227877A (en) * | 2016-08-02 | 2016-12-14 | 北京集奥聚合科技有限公司 | A kind of distributed information log acquisition system based on hadoop and method |
| CN107766147A (en) * | 2016-08-23 | 2018-03-06 | 上海宝信软件股份有限公司 | Distributed data analysis task scheduling system |
| CN106411639A (en) * | 2016-09-18 | 2017-02-15 | 合网络技术（北京）有限公司 | Method and system for monitoring access data |
| CN106383758A (en) * | 2016-09-22 | 2017-02-08 | 郑州云海信息技术有限公司 | Operation system-based information acquisition method |
| CN106503079A (en) * | 2016-10-10 | 2017-03-15 | 语联网（武汉）信息技术有限公司 | A kind of blog management method and system |
| CN106445790A (en) * | 2016-10-12 | 2017-02-22 | 北京集奥聚合科技有限公司 | Counting and account-checking method and device used in distributed real-time computing system |
| CN107959697A (en) * | 2016-10-17 | 2018-04-24 | 腾讯科技（深圳）有限公司 | Source Data Acquisition method and system in big data off-line calculation |
| CN107959697B (en) * | 2016-10-17 | 2019-12-06 | 腾讯科技（深圳）有限公司 | Source data acquisition method and system in big data offline calculation |
| CN106682073A (en) * | 2016-11-14 | 2017-05-17 | 上海轻维软件有限公司 | HBase fuzzy retrieval system based on Elastic Search |
| CN108073625A (en) * | 2016-11-14 | 2018-05-25 | 北京京东尚科信息技术有限公司 | For the system and method for metadata information management |
| CN108073625B (en) * | 2016-11-14 | 2021-03-30 | 北京京东尚科信息技术有限公司 | System and method for metadata information management |
| CN108076111A (en) * | 2016-11-15 | 2018-05-25 | 亿阳安全技术有限公司 | A kind of system and method for distributing data in big data platform |
| CN108076111B (en) * | 2016-11-15 | 2021-07-09 | 亿阳安全技术有限公司 | System and method for distributing data in big data platform |
| CN106557561A (en) * | 2016-11-16 | 2017-04-05 | 贵州大学 | Magnanimity sensing data storage system and method based on HBase |
| CN106528847A (en) * | 2016-11-24 | 2017-03-22 | 北京集奥聚合科技有限公司 | Multi-dimensional processing method and system for massive data |
| CN106649670B (en) * | 2016-12-14 | 2020-07-17 | 北京五八信息技术有限公司 | Data monitoring method and device based on stream computing |
| CN106649670A (en) * | 2016-12-14 | 2017-05-10 | 北京五八信息技术有限公司 | Streaming computing-based data monitoring method and apparatus |
| CN106790572A (en) * | 2016-12-27 | 2017-05-31 | 广州华多网络科技有限公司 | The system and method that a kind of distributed information log is collected |
| WO2018130222A1 (en) * | 2017-01-16 | 2018-07-19 | 贵州白山云科技有限公司 | Method for writing to log, system, medium, and device |
| CN106933720A (en) * | 2017-01-16 | 2017-07-07 | 国家电网公司 | Network log information security scene-type analysis system and its analysis method |
| CN106980678A (en) * | 2017-03-30 | 2017-07-25 | 温馨港网络信息科技（苏州）有限公司 | Data analysing method and system based on zookeeper technologies |
| CN107016133A (en) * | 2017-05-24 | 2017-08-04 | 成都享之道网络科技有限公司 | Based on the online big data system with offline double processing |
| CN107357804A (en) * | 2017-05-24 | 2017-11-17 | 上海你我贷互联网金融信息服务有限公司 | The analysis system and method for internet finance massive logs |
| CN107590182B (en) * | 2017-08-03 | 2020-06-19 | 华南理工大学 | Distributed log collection method |
| CN107590182A (en) * | 2017-08-03 | 2018-01-16 | 华南理工大学 | A kind of distributed information log collection method |
| CN107622084A (en) * | 2017-08-10 | 2018-01-23 | 深圳前海微众银行股份有限公司 | Blog management method, system and computer-readable recording medium |
| CN107463490B (en) * | 2017-08-15 | 2020-06-30 | 四川长虹电器股份有限公司 | Cluster log centralized collection method applied to platform development |
| CN107463490A (en) * | 2017-08-15 | 2017-12-12 | 四川长虹电器股份有限公司 | A kind of cluster daily record centralized collection method being applied in platform development |
| CN107508888A (en) * | 2017-08-25 | 2017-12-22 | 同方(深圳)云计算技术股份有限公司 | A kind of car networking service platform |
| CN107786565A (en) * | 2017-11-02 | 2018-03-09 | 江苏物联网研究发展中心 | A kind of distributed real-time intrusion detection method and detecting system |
| CN107800592A (en) * | 2017-11-09 | 2018-03-13 | 郑州云海信息技术有限公司 | A kind of server test results acquisition method |
| CN107872465A (en) * | 2017-12-05 | 2018-04-03 | 全球能源互联网研究院有限公司 | A kind of distributed network security monitoring method and system |
| CN108153828A (en) * | 2017-12-12 | 2018-06-12 | 顺丰科技有限公司 | A kind of persistence method of real time data, device and equipment, storage medium |
| CN108073705A (en) * | 2017-12-18 | 2018-05-25 | 郑州云海信息技术有限公司 | A kind of distributed mass data polymerize acquisition method |
| CN108073705B (en) * | 2017-12-18 | 2022-06-14 | 浪潮云信息技术股份公司 | Distributed mass data aggregation acquisition method |
| CN110309130A (en) * | 2018-03-21 | 2019-10-08 | 中国人民财产保险股份有限公司 | A kind of method and device for host performance monitor |
| CN108763310A (en) * | 2018-04-25 | 2018-11-06 | 江苏鸣鹤云科技有限公司 | A kind of big data platform of High Availabitity |
| CN108647323A (en) * | 2018-05-11 | 2018-10-12 | 重庆工商职业学院 | A kind of data summarization method of vocational ability |
| CN108710563A (en) * | 2018-05-16 | 2018-10-26 | 广州市千钧网络科技有限公司 | A kind of Application Logging method and device |
| CN108717632A (en) * | 2018-05-29 | 2018-10-30 | 广东通莞科技股份有限公司 | A kind of mobile payment storage protection and recovery system |
| CN108810146A (en) * | 2018-06-14 | 2018-11-13 | 鹤壁北岩科技有限公司 | Culture control system based on distributed cloud platform and control method |
| CN108829879A (en) * | 2018-06-26 | 2018-11-16 | 天津城建大学 | A kind of charging pile data monitoring method |
| CN108985981B (en) * | 2018-06-28 | 2021-04-23 | 北京奇虎科技有限公司 | Data processing system and method |
| CN108985981A (en) * | 2018-06-28 | 2018-12-11 | 北京奇虎科技有限公司 | Data processing system and method |
| CN109167672A (en) * | 2018-07-13 | 2019-01-08 | 腾讯科技（深圳）有限公司 | One kind returning source location of mistake method, apparatus, storage medium and system |
| CN109167672B (en) * | 2018-07-13 | 2020-07-10 | 腾讯科技（深圳）有限公司 | Return source error positioning method, device, storage medium and system |
| CN108932352A (en) * | 2018-09-26 | 2018-12-04 | 江苏曲速教育科技有限公司 | Distributed cache system based on statistical server |
| CN109542638A (en) * | 2018-10-26 | 2019-03-29 | 深圳点猫科技有限公司 | A kind of document handling method and device based on educational system |
| CN111157838A (en) * | 2018-11-08 | 2020-05-15 | 中国铁路沈阳局集团有限公司科学技术研究所 | Intelligent management system for big data of running state of railway power distribution network |
| CN109151464A (en) * | 2018-11-14 | 2019-01-04 | 江苏鸿信***集成有限公司 | IPTV set top box failure real-time detection method based on high amount of traffic processing |
| CN109710731A (en) * | 2018-11-19 | 2019-05-03 | 北京计算机技术及应用研究所 | A kind of multidirectional processing system of data flow based on Flink |
| CN109542750A (en) * | 2018-11-26 | 2019-03-29 | 深圳天源迪科信息技术股份有限公司 | Distributed information log system |
| CN109445949A (en) * | 2018-12-07 | 2019-03-08 | 武汉轻工大学 | A kind of data collection system and collecting method |
| CN109657125A (en) * | 2018-12-14 | 2019-04-19 | 平安城市建设科技(深圳)有限公司 | Data processing method, device, equipment and storage medium based on web crawlers |
| CN109857613A (en) * | 2018-12-25 | 2019-06-07 | 南京南瑞信息通信科技有限公司 | A kind of automation operational system based on acquisition cluster |
| CN109857613B (en) * | 2018-12-25 | 2021-10-08 | 南京南瑞信息通信科技有限公司 | Automatic operation and maintenance system based on collection cluster |
| CN109886327A (en) * | 2019-02-12 | 2019-06-14 | 北京奇艺世纪科技有限公司 | The processing system and method for Java data in a kind of distributed system |
| WO2020168756A1 (en) * | 2019-02-19 | 2020-08-27 | 平安科技（深圳）有限公司 | Cluster log feature extraction method, and apparatus, device and storage medium |
| CN110262951A (en) * | 2019-06-10 | 2019-09-20 | 天翼电子商务有限公司 | A kind of business second grade monitoring method and system, storage medium and client |
| CN110232054A (en) * | 2019-06-19 | 2019-09-13 | 北京百度网讯科技有限公司 | Log transmission system and streaming log transmission method |
| CN110232054B (en) * | 2019-06-19 | 2021-07-20 | 北京百度网讯科技有限公司 | Log transmission system and streaming log transmission method |
| CN110245120B (en) * | 2019-06-19 | 2021-06-11 | 北京百度网讯科技有限公司 | Stream type computing system and log data processing method thereof |
| CN110245120A (en) * | 2019-06-19 | 2019-09-17 | 北京百度网讯科技有限公司 | The daily record data processing method of streaming computing system and streaming computing system |
| CN110442503A (en) * | 2019-07-29 | 2019-11-12 | 深圳数位传媒科技有限公司 | A kind of alarm method and device using log index |
| CN110688399A (en) * | 2019-08-26 | 2020-01-14 | 远光软件股份有限公司 | Stream type calculation real-time report system and method |
| CN110659306A (en) * | 2019-08-29 | 2020-01-07 | 达疆网络科技(上海)有限公司 | Store system all-level cache synchronization mode |
| CN111200637A (en) * | 2019-12-20 | 2020-05-26 | 新浪网技术(中国)有限公司 | Cache processing method and device |
| CN111200637B (en) * | 2019-12-20 | 2022-07-08 | 新浪网技术(中国)有限公司 | Cache processing method and device |
| CN111209314A (en) * | 2020-01-13 | 2020-05-29 | 国网浙江省电力有限公司信息通信分公司 | System for processing massive log data of power information system in real time |
| CN111258979A (en) * | 2020-01-16 | 2020-06-09 | 山东大学 | Cloud protection log system and working method thereof |
| CN111258979B (en) * | 2020-01-16 | 2022-04-15 | 山东大学 | Cloud protection log system and working method thereof |
| CN111355788A (en) * | 2020-02-21 | 2020-06-30 | 深圳供电局有限公司 | Distributed data center management system |
| CN111552628A (en) * | 2020-03-20 | 2020-08-18 | 北京海致星图科技有限公司 | Distributed pressure measurement system and method for graph database and graph service interface |
| CN111695126A (en) * | 2020-05-28 | 2020-09-22 | 武汉中海庭数据技术有限公司 | Crowdsourcing data decryption method and device, electronic equipment and storage medium |
| CN112163060A (en) * | 2020-09-16 | 2021-01-01 | 安徽龙运智能科技有限公司 | System for processing mass GPS data by big data technology |
| CN112347073A (en) * | 2020-10-27 | 2021-02-09 | 山东开创云计算有限公司 | Multiple data system |
| CN112433998B (en) * | 2020-11-20 | 2022-01-21 | 广东电网有限责任公司佛山供电局 | Multisource heterogeneous data acquisition and convergence system and method based on power system |
| CN112433998A (en) * | 2020-11-20 | 2021-03-02 | 广东电网有限责任公司佛山供电局 | Multisource heterogeneous data acquisition and convergence system and method based on power system |
| CN112818045A (en) * | 2021-01-22 | 2021-05-18 | 辽宁长江智能科技股份有限公司 | Data access unified management platform for big data |
| CN113220655A (en) * | 2021-04-30 | 2021-08-06 | 中核武汉核电运行技术股份有限公司 | Data access method, device, equipment and readable storage medium |
| CN115022402A (en) * | 2022-07-01 | 2022-09-06 | 杭州乘云数字技术有限公司 | Agent acquisition method and system based on one-stack integration technology |
| CN114996335A (en) * | 2022-08-03 | 2022-09-02 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104036025A (en) | Distribution-base mass log collection system | |
| Zicari | Big data: Challenges and opportunities | |
| Shree et al. | KAFKA: The modern platform for data management and analysis in big data domain | |
| Das et al. | Big data analytics: A framework for unstructured data analysis | |
| CN111400326B (en) | Smart city data management system and method thereof | |
| US9935912B2 (en) | Ad hoc message sharing between email and social networks | |
| Xhafa et al. | Processing and analytics of big data streams with yahoo! s4 | |
| CN103399887A (en) | Query and statistical analysis system for mass logs | |
| CN108021809A (en) | A kind of data processing method and system | |
| CN104426713A (en) | Method and device for monitoring network site access effect data | |
| CN105518644B (en) | Method for processing and displaying social data on map in real time | |
| Saxena et al. | Practical real-time data processing and analytics: distributed computing and event processing using Apache Spark, Flink, Storm, and Kafka | |
| CN112948492A (en) | Data processing system, method and device, electronic equipment and storage medium | |
| CN105391777A (en) | Algorithm escrow PaaS platform for decoupling logic code and performance code | |
| Sánchez et al. | Distributed data collection for the ATLAS EventIndex | |
| CN112328569A (en) | Construction method based on Flume distributed data collection architecture | |
| Di Stefano et al. | Prometheus and aiops for the orchestration of cloud-native applications in ananke | |
| Park et al. | An implementation of a high throughput data ingestion system for machine logs in manufacturing industry | |
| CN111625532A (en) | Data blood relationship processing method and device, computer equipment and storage medium | |
| US11256713B2 (en) | Virtual transaction queues for database replication | |
| CN108763562A (en) | A kind of construction method based on big data skill upgrading data exchange efficiency | |
| Riaz et al. | Filtering the big data based on volume, variety and velocity by using Kalman filter recursive approach | |
| CN111049898A (en) | Method and system for realizing cross-domain architecture of computing cluster resources | |
| Wadhera et al. | A systematic Review of Big data tools and application for developments | |
| CN108959041B (en) | Method for transmitting information, server and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20140910 |