CN110995815B - Information transmission method based on Gaia big data analysis system - Google Patents
Information transmission method based on Gaia big data analysis system Download PDFInfo
- Publication number
- CN110995815B CN110995815B CN201911181992.1A CN201911181992A CN110995815B CN 110995815 B CN110995815 B CN 110995815B CN 201911181992 A CN201911181992 A CN 201911181992A CN 110995815 B CN110995815 B CN 110995815B
- Authority
- CN
- China
- Prior art keywords
- data
- server
- key
- gaia
- attribute
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- PFFIDZXUXFLSSR-UHFFFAOYSA-N 1-methyl-N-[2-(4-methylpentan-2-yl)-3-thienyl]-3-(trifluoromethyl)pyrazole-4-carboxamide Chemical compound S1C=CC(NC(=O)C=2C(=NN(C)C=2)C(F)(F)F)=C1C(C)CC(C)C PFFIDZXUXFLSSR-UHFFFAOYSA-N 0.000 title claims abstract description 28
- 230000005540 biological transmission Effects 0.000 title claims abstract description 26
- 238000007405 data analysis Methods 0.000 title claims abstract description 24
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000004891 communication Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 4
- 238000010801 machine learning Methods 0.000 abstract description 4
- 238000007418 data mining Methods 0.000 abstract description 3
- 230000008901 benefit Effects 0.000 abstract description 2
- 238000005457 optimization Methods 0.000 description 9
- 238000004458 analytical method Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 6
- 230000004927 fusion Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 239000008186 active pharmaceutical agent Substances 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000005111 flow chemistry technique Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 238000012517 data analytics Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 230000004083 survival effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
- H04L67/5651—Reducing the amount or size of exchanged application data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/141—Setup of application sessions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/161—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
- H04L69/162—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields involving adaptations of sockets based mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer And Data Communications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
An information transmission method based on a Gaia big data analysis system belongs to the field of distributed information transmission such as data mining, machine learning and application statistics under a big data environment. The technical points are as follows: marking key attributes of the data in the historical statistical information by using a data analysis tool; using the key attribute marked in step S1 to call the key attribute of each piece of data with the key attribute to the front; the client transmits the data after the order is adjusted in step S2 to the server; when receiving data, the server side firstly analyzes the first few key attributes, if no key attribute exists, the server side does not continue to analyze, and if the key attribute exists, the server side continues to analyze. Has the advantages that: according to the invention, by adjusting the sequence of the attributes in each piece of data transmitted by the client, the number of pieces of data required to be analyzed is reduced when the server analyzes the data, only the required data is analyzed, and the required time can be greatly reduced.
Description
Technical Field
The invention belongs to the field of distributed information transmission such as data mining, machine learning and application statistics under a big data environment, and particularly relates to an information transmission method based on a Gaia big data analysis system.
Background
Gaia is a high-timeliness and extensible new generation big data analysis system oriented to multi-computing model mixing and coexistence. The method solves a series of key technical problems at several core levels of big data analysis systems such as self-adaption, telescopic big data storage, batch flow fusion big data calculation, high-dimensional large-scale machine learning, high-timeliness big data intelligent interaction guide and the like, constructs a new generation of autonomous and controllable high-timeliness telescopic big data analysis system, and masters the international leading big data analysis system core technology.
The novel big data computing system has a full-period multi-scale optimization and unified computing engine for batch flow mixing tasks. The existing big data computing system simulates the behavior of another type of framework by depending on a computing engine of the existing big data computing system, or defines a set of general interfaces to shield the difference of underlying computing engines, and has weak support for batch flow fusion. Meanwhile, the optimization is mostly located at a specific period or a specific level of execution, and the optimization capability for high-complexity tasks is insufficient. Aiming at the problems, a high-performance batch flow fusion big data computing technology based on a unified computing engine and full-period multi \ scale optimization is innovatively developed. The technology provides a unified expression logic support for batch flow fusion processing, and realizes the real fusion of batch and flow processing by integrating a calculation model, a data model, a transformation model and an action model of batch flow processing through unified expression modeling. Aiming at the characteristics of diversity, persistence, iteration and the like of the operation, optimization strategies facing to multi-operation, multi-task, iterative computation, persistent computation and the like are provided, and the optimization pertinence is stronger. Meanwhile, full-period optimization before and during execution is provided, and the optimization is subdivided into a plurality of scales such as a job level, a task level and a transformation level, so that extremely-fast response and massive throughput are realized.
Gaia is an open source computing platform facing distributed data stream processing and batch processing, and supports distributed parallel computing, including model parallel and data parallel. Big data computing engines are common and various in the society of today, and in recent years, the general computing framework comprising Hadoop and Spark and the special computing framework comprising TensorFlow are provided. This is a result of our diversification of the computational model requirements. In addition, large data frameworks based on resource management, support services and storage services are available. Big data is closely related to the development of people's survival. The content supported by the big data engine comprises various applications such as batch processing, flow calculation, machine learning and interactive analysis.
Two of the most important tasks are unique to data mining for large data. Firstly, real-time performance, such massive data scale needs real-time analysis and rapid result feedback. Secondly, accuracy is achieved, valuable information needed by users and hidden in the valuable information is required to be accurately extracted from massive data, the information obtained through mining is converted into organized knowledge to be expressed in modes of models and the like, and therefore the analysis models are applied to real life to improve production efficiency and optimize marketing schemes.
Therefore, whichever large data computing engine requires a fast response, i.e., low latency, there are many problems in life that require feedback in real time. In the case of a rapid increase in data, it is important to make a high throughput, low latency system.
Gaia integrates a plurality of advantages, including rapidness, reliability, expandability, complete compatibility with Hadoop, simple use and excellent performance. Using memory-based data flow and deep integration of iterative processing algorithms into the runtime of the system, Gaia enables the system to process data-intensive and iterative tasks at extremely fast speeds. Gaia also has high reliability and good scalability. And is Hadoop compatible. In addition, Gaia includes Java-based APIs for bulk and stream data-based analysis, optimizers, and distributed runtimes with custom memory management functions, etc., and thus can be fully compatible with Hadoop. In summary, Gaia has the high efficiency, flexibility and extensibility of a distributed MapReduce class of platform, and parallel database query optimization scheme, while it supports batch and stream-based data analysis, and provides Java-based APIs. In general, Gaia is an efficient, distributed, Java-based implementation of a general big data analytics engine.
One important characteristic of Gaia is low latency, but the original transmission method may waste a lot of resources during the information transmission process.
Disclosure of Invention
In order to enable each node to fully utilize resources and save the time for filtering information, the invention provides an information transmission method based on a Gaia big data analysis system, which can save time and improve efficiency when transmitting and filtering information.
The technical scheme is as follows:
an information transmission method based on a Gaia big data analysis system comprises the following steps:
s1, marking key attributes of the data in the historical statistical information by using a data analysis tool;
s2, using the key attribute marked in the step S1 to adjust the key attribute of each piece of data with the key attribute to the front;
s3, the client sends the data after the sequence is adjusted in the step S2 to the server;
s4, when the server receives the data, the first several key attributes are analyzed, if no key attribute exists, the analysis is not continued, and if the key attribute exists, the analysis is continued.
Further, a Socket is used for establishing communication connection between the client and the server.
Further, the client sends data to the server by using the IO stream.
Further, the data analysis tool is Python or Mysql.
Further, if there are several key attributes, the several key attributes are adjusted to the front according to the front-back order before adjustment.
According to the information transmission method based on the Gaia big data analysis system, the sequence of the attributes in each piece of data transmitted by the client is adjusted, so that the number of pieces of data to be analyzed is reduced when the server analyzes the data, only the needed data is analyzed, and the needed time can be greatly reduced; the method can filter out the unnecessary information, effectively save the time for transmitting the information, greatly improve the efficiency of information transmission and reasonably utilize resources.
Drawings
FIG. 1 is a process diagram for adjusting the key attribute order in an embodiment of the present invention;
fig. 2 is a basic framework diagram of information transmission in the embodiment of the present invention.
Detailed Description
The information transmission method based on the Gaia big data analysis system is further explained with reference to fig. 1-2.
Example 1
An information transmission method based on a Gaia big data analysis system comprises the following steps:
s1, marking key attributes of the data by using historical statistical information and a data analysis tool;
s2, using the key attribute analyzed in the step S1 to adjust the key attribute of each piece of data with the key attribute to the front;
s3, the client sends the data after the sequence is adjusted in the step S2 to the server;
s4, the server side analyzes the first few key attributes when receiving data, if no key attribute exists, the analysis is not continued, and if the key attribute is concerned, the analysis is continued;
further, the data analysis tool is Python or Mysql.
Further, if there are several key attributes, the several key attributes are adjusted to the front according to the front-back order before adjustment. The criteria for determining the key attributes is, regardless of the other attributes, as long as the attributes can identify the tag of the piece of data. For example, if a certain attribute in a piece of data is 1, the tag of the piece of data will not change regardless of other attributes.
Further, in step S1, the client and the server are connected to each other to ensure mutual communication.
Further, in step S2, each piece of data has one or more key attributes that can be determined for the information tag. Such as in a piece of intrusion detection data: 2, tcp, smtp, SF,1684,363,0,0,0,0,0,1,0,0,0,0,0,0, 1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,104,66,0.63,0.03,0.01,0.00,0.00,0.00,0.00,0.00, normal. The last attribute is a label, the data is marked as a normal type and is not invaded, and the data is of the normal type if the 11 th attribute and the 12 th attribute are combined in a way of 01.
If bits 11 and 12 are other numbers, such as a piece of data: 0, tcp, private, REJ,0,0,0,0,0,0,0,0,0,0,0, 38,1,0.00,0.00,1.00,1.00,0.03,0.55,0.00,208,1,0.00,0.11,0.18,0.00,0.01,0.00,0.42,1.00, portsweet, this piece of data is labeled as portsweet (an attack) type.
Further, in step S3, the client sends the newly adjusted sequence of data to the server, where the data takes the first data in S2 as an example, and the data to be sent to the server after the adjustment is: 0,1,2, tcp, smtp, SF,1684,363,0,0,0,0,0, 0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,104,66,0.63,0.03,0.01,0.00,0.00, 0.00. Since the normal is a label and is marked later, the real data is unmarked, so the normal does not need to be sent to the server side.
Further, in step S4, in the server-side parsing process, taking the previous intrusion detection data as an example: 0,1,2, tcp, smtp, SF,1684,363,0,0,0,0,0, 0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,104,66,0.63,0.03,0.01,0.00,0.00,0.00, resolving the first two bits of data at the server end to be 0,1, then continuing to resolve the whole piece of data, if not 0,1, then not resolving the rest of data.
Example 2
Gaia supports high performance pipeline information transfer. In the information transmission process, a client and a server are firstly connected, the client adjusts the key attributes of each piece of information to the first few bits, and then the adjusted data is sent to the server. The client analyzes the attributes of the first data bits at first, if the attributes of the first data bits are related to the key attributes, the data is continuously analyzed, otherwise, the data is not continuously analyzed.
A data transmission method is mainly characterized in that a client side adjusts key attributes of each piece of data to be transmitted to the front before transmitting the data to a server, then the server can analyze the key attributes first when analyzing, judges whether the data is needed, and does not need to analyze the rest data if the data is not needed, so that the time is saved, the resources are saved, and the efficiency is improved.
The embodiment of the invention is described by taking information transmission in intrusion detection as an example, and the information transmission method based on the Gaia system comprises the following steps:
step 1: the client server establishes a connection. After establishing the connection, the client and the server may communicate.
Step 2: the order of the key attributes of each piece of information to be transmitted is adjusted. The key attribute needed to identify this piece of information is adjusted to the front.
And step 3: and sending the information. And establishing connection in the first two steps, and sending information to the server after adjusting the sequence of the key attributes.
And 4, step 4: and (6) analyzing the information. After the information sent by the client, the server firstly analyzes the first key attributes of the information, if the first key attributes are relevant, the whole information is continuously analyzed, otherwise, the information is not analyzed any more.
Example 3
Information transmission becomes an important approach for solving problems in the field of big data application, is frequently used in big data processing algorithms, is more important for novel big data computing platforms Gaia, and the important characteristic of Gaia is low delay. Gaia is used as a real-time stream processing system, and the parallel mechanism of the Gaia is carried out by slicing nodes, and the task of information transmission is completed by a job.
The invention completes the information transmission through the following steps, as shown in fig. 2, the specific steps are as follows:
step 1: and establishing connection between the server and the client. And establishing the connection between the server and the client by using the Socket. In order to communicate between two computers, a network line must be used to connect the two computers. The Server (Server) refers to a computer or a program for providing information, the Client (Client) refers to a computer or a program for requesting information, and the network is used for connecting the Server and the Client to realize the mutual communication, and a Socket is used for monitoring a fixed port.
Step 2: and adjusting the sequence of key attributes of the information sent by the client. In the vast amount of information, including both information that we need and information that we do not need, there must be key attributes that mark this information. These key attributes are brought to the front before sending the information to the server.
And step 3: the client sends information to the server. And sending data to the server side by using the IO stream. Figure 1 illustrates the process of adjusting key attributes before sending data.
And 4, step 4: the server parses the information. The server firstly analyzes the first few bits of the information, if the first few bits are the key attributes, the whole piece of data is continuously analyzed, otherwise, the analysis is not continuously performed.
In the Gaia system, the flow of information transmission is shown in fig. 2. Suppose that each piece of information has 1000 attributes and 2 key attributes, and the two key attributes are the values that we need, and the whole piece of data is the value we need. The first 2 bits of data are parsed first, and if the first 2 bits of data are needed, the parsing is continued, otherwise, the parsing is not performed. The algorithm pseudo code for the server to parse the information is as follows:
the above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be able to cover the technical solutions and the inventive concepts of the present invention within the technical scope of the present invention.
Claims (3)
1. An information transmission method based on a Gaia big data analysis system is characterized by comprising the following steps:
s1, marking key attributes of the data in the historical statistical information by using a data analysis tool:
in data of intrusion detection, if 11 th bit and 12 th bit are 01, the key attribute is the last attribute of the data, and the key attribute is marked as a normal type; if the 11 th bit and the 12 th bit are not 01 and the key attribute is the last attribute of the data, marking the key attribute as an attack type;
s2, adjusting the 11 th bit attribute and the 12 th bit attribute to the forefront of the data by using the key attribute marked in the step S1;
s3, the client sends the data after the sequence is adjusted in the step S2 to the server; establishing communication connection between the client and the server by using Socket sockets; the client sends data to the server by using the IO stream;
s4, when the server receives the data, the first two bits of the data are firstly analyzed, if the data are '01', the whole data are continuously analyzed; if not "01," the remaining data is not parsed.
2. The information transmission method based on the Gaia big data analysis system according to claim 1, wherein the data analysis tool is Python or Mysql.
3. The information transmission method based on the Gaia big data analysis system according to claim 1, wherein if there are several key attributes, the several key attributes are adjusted to the front in a front-to-back order before the adjustment.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911181992.1A CN110995815B (en) | 2019-11-27 | 2019-11-27 | Information transmission method based on Gaia big data analysis system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911181992.1A CN110995815B (en) | 2019-11-27 | 2019-11-27 | Information transmission method based on Gaia big data analysis system |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110995815A CN110995815A (en) | 2020-04-10 |
CN110995815B true CN110995815B (en) | 2022-08-05 |
Family
ID=70087334
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911181992.1A Active CN110995815B (en) | 2019-11-27 | 2019-11-27 | Information transmission method based on Gaia big data analysis system |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110995815B (en) |
Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1412876A2 (en) * | 2001-03-19 | 2004-04-28 | International Business Machines Corporation | Using continuous optimization for ordering categorical data sets in a data processing system |
CN102799682A (en) * | 2012-05-10 | 2012-11-28 | 中国电力科学研究院 | Massive data preprocessing method and system |
CN105183914A (en) * | 2015-10-14 | 2015-12-23 | 合一网络技术(北京)有限公司 | Data characteristic formatting method and device |
CN105447172A (en) * | 2015-12-07 | 2016-03-30 | 北京先进数通信息技术股份公司 | Data processing method and system under Hadoop platform |
CN105468995A (en) * | 2015-12-15 | 2016-04-06 | 吉林大学 | Data mining based invasion detection system with Oracle as core |
CN105701161A (en) * | 2015-12-31 | 2016-06-22 | 深圳先进技术研究院 | Real-time big data user label system |
CN107026764A (en) * | 2016-02-02 | 2017-08-08 | 腾讯科技(深圳)有限公司 | A kind of remote debugging method, equipment, server and system |
CN107800787A (en) * | 2017-10-23 | 2018-03-13 | 广州百兴网络科技有限公司 | A kind of shared computer network system of distributed big data real-time exchange |
CN108566364A (en) * | 2018-01-15 | 2018-09-21 | 中国人民解放军国防科技大学 | Intrusion detection method based on neural network |
CN109698836A (en) * | 2019-02-01 | 2019-04-30 | 重庆邮电大学 | A kind of method for wireless lan intrusion detection and system based on deep learning |
CN109962909A (en) * | 2019-01-30 | 2019-07-02 | 大连理工大学 | A kind of network intrusions method for detecting abnormality based on machine learning |
CN110140126A (en) * | 2016-12-30 | 2019-08-16 | 微软技术许可有限责任公司 | Logarithm factually when adjustment to model management attribute |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106647724B (en) * | 2017-02-15 | 2017-12-26 | 北京航空航天大学 | A kind of detection of T BOX information securities and means of defence based on vehicle abnormality data monitoring |
-
2019
- 2019-11-27 CN CN201911181992.1A patent/CN110995815B/en active Active
Patent Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1412876A2 (en) * | 2001-03-19 | 2004-04-28 | International Business Machines Corporation | Using continuous optimization for ordering categorical data sets in a data processing system |
CN1659546A (en) * | 2001-03-19 | 2005-08-24 | 国际商业机器公司 | Using continuous optimization for ordering categorical data sets in a data processing system |
CN102799682A (en) * | 2012-05-10 | 2012-11-28 | 中国电力科学研究院 | Massive data preprocessing method and system |
CN105183914A (en) * | 2015-10-14 | 2015-12-23 | 合一网络技术(北京)有限公司 | Data characteristic formatting method and device |
CN105447172A (en) * | 2015-12-07 | 2016-03-30 | 北京先进数通信息技术股份公司 | Data processing method and system under Hadoop platform |
CN105468995A (en) * | 2015-12-15 | 2016-04-06 | 吉林大学 | Data mining based invasion detection system with Oracle as core |
CN105701161A (en) * | 2015-12-31 | 2016-06-22 | 深圳先进技术研究院 | Real-time big data user label system |
CN107026764A (en) * | 2016-02-02 | 2017-08-08 | 腾讯科技(深圳)有限公司 | A kind of remote debugging method, equipment, server and system |
CN110140126A (en) * | 2016-12-30 | 2019-08-16 | 微软技术许可有限责任公司 | Logarithm factually when adjustment to model management attribute |
EP3563287A1 (en) * | 2016-12-30 | 2019-11-06 | Microsoft Technology Licensing, LLC | Real time pivoting on data to model governance properties |
CN107800787A (en) * | 2017-10-23 | 2018-03-13 | 广州百兴网络科技有限公司 | A kind of shared computer network system of distributed big data real-time exchange |
CN108566364A (en) * | 2018-01-15 | 2018-09-21 | 中国人民解放军国防科技大学 | Intrusion detection method based on neural network |
CN109962909A (en) * | 2019-01-30 | 2019-07-02 | 大连理工大学 | A kind of network intrusions method for detecting abnormality based on machine learning |
CN109698836A (en) * | 2019-02-01 | 2019-04-30 | 重庆邮电大学 | A kind of method for wireless lan intrusion detection and system based on deep learning |
Non-Patent Citations (6)
Title |
---|
"A Method for Real-Time Identification of Malformed BGP Messages";Masafumi Watari; Peng Jiang; Atsuo Tachibana; Shigehiro Ano;《2011 IEEE/IPSJ International Symposium on Applications and the Internet》;20110830;第508-512页 * |
"Gaia Scheduler: A Kubernetes-Based Scheduler Framework";Shengbo Song;Lelai Deng;Jun Gong;Hanmei Luo,;《2018 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications》;20190321;第252-259页 * |
"Investigation of network intrusion detection using data visualization methods";Viktoras Bulavas;《2018 59th International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS)》;20181203;第878-984页 * |
"KDD 99数据集的应用";zhangtian6691844;《CSDN论坛》;20161205;全文 * |
"Parallel online sequential extreme learning machine based on MapReduce";BotaoWang n, ShanHuang,JunhaoQiu,YuLiu,GuorenWang;《Neurocomputing, 2015》;20150203;第224-232页 * |
"基于关联规则的网络入侵检测技术的应用研究’";黄晓燕,许龙飞;《第二十三届中国数据库学术会议论文集-技术报告篇》;20061110;第245-248页 * |
Also Published As
Publication number | Publication date |
---|---|
CN110995815A (en) | 2020-04-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6112196A (en) | Method and system for managing connections to a database management system by reusing connections to a database subsystem | |
CN108694221B (en) | Data real-time analysis method, module, equipment and device | |
US8812645B2 (en) | Query optimization in a parallel computer system with multiple networks | |
CN110912782B (en) | Data acquisition method, device and storage medium | |
CN108228743A (en) | Real-time big data search engine system | |
CN114356971A (en) | Data processing method, device and system | |
CN108509453B (en) | Information processing method and device | |
CN110705716A (en) | Multi-model parallel training method | |
CN114402315A (en) | Cloud-based query workload optimization | |
CN114036183A (en) | Data ETL processing method, device, equipment and medium | |
CN117278661B (en) | Industrial Internet of things multi-protocol analysis method and system | |
CN112182031B (en) | Data query method and device, storage medium and electronic device | |
CN114253798A (en) | Index data acquisition method and device, electronic equipment and storage medium | |
CN112883088B (en) | Data processing method, device, equipment and storage medium | |
CN110891083B (en) | Agent method for supporting multi-job parallel execution in Gaia | |
CN110995815B (en) | Information transmission method based on Gaia big data analysis system | |
Bruda et al. | Real-time computation: A formal definition and its applications | |
CN115048913B (en) | Command processing method and device and electronic equipment | |
CN110728118A (en) | Cross-data-platform data processing method, device, equipment and storage medium | |
CN113886422A (en) | Data extraction method, device, equipment and storage medium | |
CN115168389A (en) | Request processing method and device | |
CN111290855B (en) | GPU card management method, system and storage medium for multiple GPU servers in distributed environment | |
CN113934430A (en) | Data retrieval analysis method and device, electronic equipment and storage medium | |
Ogboada et al. | A model for optimizing the runtime of GraphQL queries | |
CN112564984A (en) | Distributed safe operation and maintenance method of Internet of things based on big data |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |