CN107295087B - System and method for realizing data aggregation between network systems - Google Patents
System and method for realizing data aggregation between network systems Download PDFInfo
- Publication number
- CN107295087B CN107295087B CN201710512510.0A CN201710512510A CN107295087B CN 107295087 B CN107295087 B CN 107295087B CN 201710512510 A CN201710512510 A CN 201710512510A CN 107295087 B CN107295087 B CN 107295087B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- network system
- acquired
- informationized
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
- G06F16/183—Provision of network file services by network file servers, e.g. by using NFS, CIFS
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a system and a method for realizing data aggregation between network systems, which are applied to an information system comprising a main network system and at least one slave network system, and the system comprises: the data acquisition module is respectively arranged on each application server; the data transmission module is respectively arranged in each slave network system; the data sharing module is arranged in the main network system; each data acquisition module acquires informationized data from the application server; each data transmission module is used for acquiring the information data acquired by each data acquisition module in the network system where the data transmission module is located and sending the information data to the data sharing module; and the data sharing module is used for acquiring the informationized data acquired by each data acquisition module in the main network system, receiving the informationized data sent by each data transmission module, and storing the acquired and received informationized data into a storage space in the main network system. The scheme can aggregate the informationized data included by different network systems.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a system and a method for implementing data aggregation between network systems.
Background
The people's court is the national judge organ, and the law is independently exercised to judge cases such as criminals, civil affairs, economy, administration and the like, and the working relation of the people's court is fair and impartial in all aspects of society and stable in society. Due to the importance of the work of the people's court, the work of the people's court has strict requirements on the stability, safety, efficiency, and failure recovery of application systems, databases, infrastructure, safety protection, and the like. In order to ensure that the work of the court can be performed safely and efficiently, the informatization system of the court usually comprises a plurality of network systems, and each network system is physically isolated.
Because the network systems in the information system of the court are physically isolated, the information data among the network systems cannot be interacted, so that the information data included by different network systems cannot be aggregated, and the overall monitoring of the operation body of the information system of the court is not facilitated.
Disclosure of Invention
The embodiment of the invention provides a system and a method for realizing data aggregation among network systems, which can aggregate informationized data included by different network systems.
The embodiment of the invention provides a system for realizing data aggregation between network systems, which is applied to an information system comprising a main network system and at least one slave network system, and comprises:
the data acquisition module is respectively arranged on each application server, wherein the master network system and each slave network system comprise at least one application server;
the data transmission module is respectively arranged in each slave network system;
the data sharing module is arranged in the main network system;
each data acquisition module is used for acquiring informationized data from the application server according to the received index parameters;
each data transmission module is used for acquiring the informationized data acquired by the data acquisition module arranged on each application server in the slave network system where the data transmission module is positioned, and sending the acquired informationized data to the data sharing module;
the data sharing module is configured to acquire the informationized data acquired by the data acquisition module arranged on each application server in the master network system, receive the informationized data sent by each data transmission module, and store the acquired and received informationized data in a storage space in the master network system.
Alternatively,
the system further comprises: a background control module;
the background control module is used for respectively forming the index parameters corresponding to each data acquisition module according to external triggering and sending the formed index parameters to the corresponding data acquisition modules.
Alternatively,
each of the data transmission modules includes: the system comprises a relational data transmission unit, a file transmission unit and a message queue unit;
the relational data transmission unit is used for acquiring the structured information data acquired by the data acquisition module arranged on each application server in the slave network system, and sending the acquired structured information data to the message queue unit;
the file transmission unit is used for acquiring unstructured information data acquired by the data acquisition module arranged on each application server in the slave network system, and sending the acquired unstructured information data to the message queue unit;
the message queue unit is used for sending the structured information data from the relational data transmission unit and the unstructured information data from the file transmission unit to the data sharing module in a message queue mode.
Alternatively,
the data sharing module comprises: the data integration unit and the data storage unit;
the data integration unit is used for carrying out standardization processing and labeling processing on the acquired and received information data;
the data storage unit is used for storing the informationized data processed by the data integration unit into the storage space.
Alternatively,
the data storage unit is used for storing the structured information data into a relational database included in the storage space and storing the unstructured information data into a non-relational database included in the storage space according to the type of the information data.
Alternatively,
the system further comprises: the data analysis module and the data display module;
the data analysis module is used for analyzing the development trend of the information system by using the information data stored in the storage space according to external triggering to obtain an analysis result;
and the data display module is used for displaying the analysis result obtained by the data analysis module.
In a second aspect, an embodiment of the present invention further provides a system for implementing data aggregation between network systems, where the system is provided in the first aspect, and a method for implementing data aggregation between network systems includes:
respectively utilizing each data acquisition module to acquire informationized data from the application server where the acquisition module is located according to the received index parameters;
acquiring the information data acquired by the data acquisition module arranged on each application server in the network system where each data transmission module is positioned by utilizing each data transmission module;
and acquiring the informationized data acquired by the data acquisition module arranged on each application server in the main network system by using the data sharing module, receiving the informationized data sent by each data transmission module, and storing the acquired and received informationized data into a storage space in the main network system.
Alternatively,
before the respectively using each data acquisition module to acquire the informationized data from the application server where the acquisition module is located according to the received index parameter, the method further includes:
and respectively forming the index parameters corresponding to each data acquisition module according to external trigger by using the background control module, and sending the formed index parameters to the corresponding data acquisition modules.
Alternatively,
the obtaining, by using each data transmission module, the information data acquired by the data acquisition module arranged on each application server in the network system where the data transmission module is located includes:
for each of the data transmission modules, performing:
acquiring structured information data acquired by the data acquisition module arranged on each application server in the network system where the data transmission module is positioned by using a relational data transmission unit included by the data transmission module;
acquiring unstructured information data, which are acquired by the data acquisition modules arranged on the application servers in the network system and are located by the data transmission modules, by using file transmission units included by the data transmission modules;
and sending the structured information data acquired by the relational data transmission unit and the unstructured information data acquired by the file transmission unit to the data sharing module in a message queue mode by using a message queue unit included in the data transmission module.
Alternatively,
the storing the acquired and received information data into a storage space in the master network system includes:
the acquired and received information data are subjected to standardization processing and labeling processing by using a data integration unit included by the data sharing module;
and storing the informationized data processed by the data integration unit into the storage space by using a data storage unit included in the data sharing module.
Alternatively,
after the storing the acquired and received information data into the storage space in the master network system, further comprising:
analyzing the development trend of the information system by utilizing the information data stored in the storage space according to external triggering by utilizing the data analysis module to obtain an analysis result;
and displaying the analysis result obtained by the data analysis module by using the data display module.
The system comprises a plurality of data acquisition modules respectively arranged on each application server, a plurality of data transmission modules respectively arranged in each slave network system and a data sharing module arranged in a master network system. Each data acquisition module acquires informationized data from the application server where the data acquisition module is located according to the index parameters, wherein the data acquisition module on the application server included in the slave network system transmits the acquired informationized data to the data transmission module arranged in the slave network system, and the data acquisition module on the application server included in the master network system transmits the acquired informationized data to the data sharing module arranged in the master network system. And the data sharing module stores the informationized data sent by the data acquisition module and the data transmission module into a storage space in the main network system. Therefore, after the informationized data of the application servers included in each slave network system are collected, the collected informationized data are transmitted to the data sharing module located in the master network system through the data transmission module, and finally the informationized data from the master network system and each slave network system are stored in the storage space located in the master network system through the data sharing module, so that the informationized data included in different network systems are aggregated.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic diagram of a system for implementing data aggregation between networks according to an embodiment of the present invention;
fig. 2 is a schematic diagram of another system for implementing data aggregation between networks according to an embodiment of the present invention;
fig. 3 is a schematic diagram of another system for implementing data aggregation between networks according to an embodiment of the present invention;
fig. 4 is a schematic diagram of another system for implementing data aggregation between networks according to an embodiment of the present invention;
fig. 5 is a flowchart of a method for implementing data aggregation between networlds according to an embodiment of the present invention;
fig. 6 is a flowchart of another method for implementing data aggregation between networks according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer and more complete, the technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention, and based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative efforts belong to the scope of the present invention.
As shown in fig. 1, an embodiment of the present invention provides a system for implementing data aggregation between network systems, where the system may be applied to an information system including a master network system 20 and at least one slave network system 30, and the system includes:
the data acquisition module 101 is respectively arranged on each application server 40, wherein the master network system 20 and each slave network system 30 comprise at least one application server 40;
a data transmission module 102 respectively disposed in each slave network system 30;
a data sharing module 103 disposed in the master network system 20;
each data acquisition module 101 is configured to acquire informatization data from the application server 40 according to the received index parameters;
each data transmission module 102 is configured to obtain the information data acquired by the data acquisition module 101 arranged on each application server 40 in the slave network system 30, and send the obtained information data to the data sharing module 103;
the data sharing module 103 is configured to acquire the information data acquired by the data acquisition module 101 arranged on each application server 40 in the master network system 20, receive the information data sent by each data transmission module 102, and store the acquired and received information data in a storage space in the master network system 20.
The embodiment of the invention provides a system for realizing data aggregation between network systems, which comprises a plurality of data acquisition modules respectively arranged on each application server, a plurality of data transmission modules respectively arranged in each slave network system and a data sharing module arranged in a master network system. Each data acquisition module acquires informationized data from the application server where the data acquisition module is located according to the index parameters, wherein the data acquisition module on the application server included in the slave network system transmits the acquired informationized data to the data transmission module arranged in the slave network system, and the data acquisition module on the application server included in the master network system transmits the acquired informationized data to the data sharing module arranged in the master network system. And the data sharing module stores the informationized data sent by the data acquisition module and the data transmission module into a storage space in the main network system. Therefore, after the informationized data of the application servers included in each slave network system are collected, the collected informationized data are transmitted to the data sharing module located in the master network system through the data transmission module, and finally the informationized data from the master network system and each slave network system are stored in the storage space located in the master network system through the data sharing module, so that the informationized data included in different network systems are aggregated.
Alternatively, as shown in FIG. 1,
because the data transmission module 102 is disposed in the slave network system 30 and the data sharing module 103 is disposed in the master network system 20, and because the master network system 20 is physically isolated from the slave network system 30, in order to ensure that the information data acquired by the data transmission module 102 can be transmitted to the data sharing module 103, so as to implement data transmission between the network systems, the data transmission module 102 and the data sharing module 103 can Access each other through a Simple Object Access Protocol (SOAP), and perform data transmission through a File Transfer Protocol (FTP), and a specific transmission process can be performed by using a DataBase communication proprietary interface (Java DataBase Connectivity, JDBC).
Optionally, on the basis of the system shown in fig. 1, the system for implementing data aggregation between network systems according to the embodiment of the present invention may further include a background control module, where the background control module may respectively form an index parameter corresponding to each data acquisition module according to external triggering, and send the formed index parameter to the corresponding data acquisition module.
Specifically, a user can define index parameters through the background control module, and after the background control module sends the index parameters defined by the user to the corresponding data acquisition module, the data acquisition module can acquire corresponding information data from the application server according to the received identification parameters. Therefore, different index parameters can be defined by a user according to requirements, different informationized data are collected from each network system for aggregation, different purposes are achieved, and the applicability of the system for realizing data aggregation among the network systems is improved.
In addition, different applications run on different application servers, and a user defines index data corresponding to each data acquisition module through the background control module respectively, so that the data acquisition modules can acquire informationized data concerned by the user according to index parameters, and the accuracy of the aggregated data is ensured.
In addition, when the system for realizing data aggregation between network systems is applied to an informatization system of a court, the informatization data collected by the data collection module comprises any one or more of the following data:
the system comprises a control execution assisting department coverage rate, a control execution return period, a ratio of the number of uploaded documents in the final documents, a court trial live broadcast access coverage rate, a visitor area coverage rate, performance scores (page response time, operation response time and user satisfaction), a total number of errors, a judgment execution final rate, the number of logged-in users, an operation and maintenance satisfaction degree, a quality and effect score of a science and technology court application, a plan access score, a remote access, a command execution, a approval execution final rate, a logged-in user ratio, a transaction traffic ratio and the like.
Alternatively, as shown in FIG. 2,
each data transmission module 102 may include: a relational data transfer unit 1021, a file transfer unit 1022, and a message queue unit 1023. For the relational data transmission unit 1021, the file transmission unit 1022 and the message queue unit 1023 included in the same data transmission module 102, the units may interact as follows:
the relational data transmission unit 1021 is used for acquiring the structured information data acquired by the data acquisition modules arranged on the application servers in the network system where the relational data transmission unit 1021 is located, and sending the acquired structured information data to the message queue unit 1023;
the file transmission unit 1022 is configured to obtain unstructured information data, which is obtained from a data acquisition module provided in each application server in the network system where the device is located, and send the obtained unstructured information data to the message queue unit 1023;
the message queue unit 1023 is used for sending the structured information data from the relational data transmission unit 1021 and the non-relational information data from the file transmission unit 1022 to the data sharing module 103 by way of a message queue.
Specifically, an application deployed on an application server may generate structured data and unstructured data in an operation process, and the structured data and the unstructured data have different data forms, so that the structured data and the unstructured data are conveniently transmitted, structured information data acquired by a data acquisition module are acquired through a relational data transmission unit, unstructured information data acquired by the data acquisition module are acquired through a file transmission unit, and the acquired information data are respectively sent to a data sharing module through a message queue unit. The unstructured information data comprises data files such as logs, documents, audio and video.
In the first aspect, the structured data and the unstructured data are separated, and different interfaces and protocols can be adopted to transmit the structured data and the unstructured data, so that the speed and the reliability of information data transmission are ensured. In the second aspect, the information data is transmitted to the data sharing module in a message queue mode, and the message queue can transmit simple text strings and also can transmit complex data containing embedded objects, so that the normal transmission of the structured information data and the unstructured information data is ensured; in addition, the data transmission mode based on the message queue can ensure that the information data can be successfully transmitted to the data sharing module, and ensure the reliability of the data system among the aggregation network systems.
Alternatively, as shown in FIG. 3,
the data sharing module 103 may include a data integration unit 1031 and a data storage unit 1032, wherein,
the data integration unit 1031 is configured to perform standardization and tagging on the informationized data from each data transmission module 102 and each data acquisition module 101 in the main network 20, perform data cleaning on the informationized data acquired by each data acquisition module 101, remove error data, and add a tag to each informationized data;
the data storage unit 1032 is used to store the informationized data processed by the data integration unit 1031 into the storage space 201.
Specifically, before the information data is stored in the storage space, the data integration unit performs standardization processing and labeling processing on the information data, specifically, removes error data included in the information data, and adds a label to the information data, so as to facilitate extraction and calling of the information data. The data integration unit is used for processing the informationized data, so that the quality of the informationized data is improved.
Alternatively, as shown in FIG. 3,
the storage space 201 may be a storage server in the master network system 20, and a relational database and a non-relational database are disposed in the storage server. When storing the information data in the storage space 201, the data storage unit 1032 may store the structured information data in the relational database and the unstructured information data in the non-relational database according to the type of the information data. The relational database can be a relational database based on a distributed file system, and meanwhile, the non-relational database can be a non-relational database based on the distributed file system, so that the storage space is ensured to have enough capacity and safety.
Specifically, according to the type of the informationized data, the informationized data are respectively stored in a relational database and a non-relational database, the informationized data aggregated together are reasonably stored, the informationized data are conveniently called from a storage space to be analyzed, and the informationized data stored in the storage space are conveniently managed.
Alternatively, as shown in FIG. 4,
the system for realizing data aggregation between network systems can also comprise a data analysis module 104 and a data presentation module 105;
the data analysis module 104 is configured to analyze a development trend of the information-based system by using the information-based data stored in the storage space 201 according to an external trigger, and obtain an analysis result;
the data display module 105 is configured to display the analysis result obtained by the data analysis module 104.
Specifically, after the informationized data from each network system is aggregated to the storage space in the main network system, a user can trigger the data analysis module to analyze the informationized data stored in the storage space, the data analysis module analyzes the informationized data stored in the storage space by using a predetermined defined analysis model or a machine learning method, an analysis result reflecting the development trend of the informationized system is determined, and finally the analysis result is displayed by the data display module, so that the user can update and arrange the informationized data software and hardware conveniently.
It should be noted that, in the system provided in each of the above embodiments, the data transmission module may be deployed on a virtual machine in the corresponding slave network system, and the data sharing module, the data analysis module, and the data presentation module may all be deployed on a virtual machine in the master network system.
As shown in fig. 5, an embodiment of the present invention provides a method for aggregating data between network systems, using any one of the systems for implementing data aggregation between network systems provided in the foregoing embodiments, where the method may include the following steps:
step 501: respectively utilizing each data acquisition module to acquire informationized data from an application server where the acquisition module is located according to the received index parameters;
step 502: acquiring information data acquired by a data acquisition module arranged on each application server in a network system, where each data transmission module is positioned;
step 503: and acquiring the informationized data acquired by the data acquisition modules arranged on the application servers in the main network system by using the data sharing module, receiving the informationized data sent by the data transmission modules, and storing the acquired and received informationized data in a storage space in the main network system.
Optionally, before step 501 shown in fig. 5, a background control module may be further used to respectively form an index parameter corresponding to each data acquisition module according to external trigger, and send the formed index parameter to the corresponding data acquisition module.
Alternatively, when the step 502 shown in fig. 5 uses the data transmission module to obtain the information data obtained by the data acquisition module, for each data transmission module, the following steps may be performed:
a1: acquiring structured information data acquired by a data acquisition module arranged on each application server in a network system, where the data transmission module is located, by using a relational data transmission unit included in the data transmission module;
a2: acquiring unstructured informatization data, which are acquired by data acquisition modules arranged on each application server in a network system and are located by the data transmission modules, by using a file transmission unit included by the data transmission modules;
a3: and sending the structured information data acquired by the relational data transmission unit and the unstructured information data acquired by the file transmission unit to the data sharing module in a message queue mode by using a message queue unit included in the data transmission module.
Alternatively, when storing the acquired and received information data in the storage space in step 503 shown in fig. 5, the following steps may be performed:
b1: the acquired and received information data are subjected to standardization processing and labeling processing by using a data integration unit included by the data sharing module;
b2: and storing the informationized data processed by the data integration unit into a storage space by using a data storage unit included by the data sharing module.
Alternatively, when the information data is stored in the storage space by using the data storage unit in step B2, the structured information data may be stored in a relational database included in the storage space and the unstructured information data may be stored in a non-relational database included in the storage space according to the type of the information data.
Optionally, after step 503 shown in fig. 5, the data analysis module may further be used to analyze the development trend of the information system by using the information data stored in the storage space according to an external trigger to obtain an analysis result, and the data display module may be used to display the analysis result obtained by the data analysis module.
It should be noted that, since each step included in the above method is based on the same concept as that of the embodiment of the system of the present invention, specific contents may be referred to the description in the embodiment of the system of the present invention, and are not described herein again.
The method for implementing data aggregation between network systems according to the embodiment of the present invention is further described in detail below by taking an information system of a court as an example, and as shown in fig. 6, the method may include the following steps:
step 601: the system comprises a deployment data acquisition module, a data transmission module and a data sharing module.
In an embodiment of the invention, the informatization system of the court comprises 5 network systems which are respectively a court private network, the internet, a secret private network, an external private network and a mobile private network, each network system comprises at least one application server, and each application server runs an application. In consideration of safety and related legal requirements, the court private network allows data to enter and flow out simultaneously, so that the court private network is determined as a master network system, and the other 4 network systems are determined as slave network systems. Firstly, deploying a data acquisition module on each application server included in 5 network systems; then, respectively selecting three virtual machines in machine rooms of three network systems of the internet, the secret-related private network and the external private network, and installing a data transmission module on each virtual machine; and finally, selecting one or more virtual machines in a machine room of the court private network to install the data sharing module.
Because the mobile private network is communicated with the court private network, the data on the mobile private network can be directly transmitted to the court private network, and therefore a data transmission module is not required to be deployed on the mobile private network.
Step 602: a storage server is deployed in a primary grid.
In one embodiment of the present invention, a storage server is deployed in a primary grid, wherein the storage server includes a relational database for storing structured data and a non-relational database for storing unstructured data.
For example, a storage server is deployed in a court ad hoc network.
Step 603: and collecting the informationized data by using a data collection module.
In an embodiment of the present invention, for each data acquisition module, according to the received index parameter, the data acquisition module is used to acquire the information data from the application server where the data acquisition module is located.
For example, the data acquisition module located in the external private network is used for acquiring informationized data such as coverage rate of the department performing the survey and control assistance, return rate of the survey and control, return period of the survey and control assistance, and the like; acquiring information data such as the number of uploaded documents in the final document, the live broadcast access coverage rate of court trial, the coverage rate of visitor areas, performance scores (page response time and user satisfaction) and the total number of errors by using a data acquisition module positioned in the Internet; acquiring informatization data such as trial and execution plan rate, total number of errors, performance indexes (operation response time and page response time), login user information, operation and maintenance satisfaction, quality and efficiency score of science and technology court application, plan access score, remote access, command execution and the like by using a data acquisition module in a court private network; acquiring information data such as approval execution result rate and the like by using a data acquisition module in the secret-related private network; and acquiring information data such as login user ratio, transaction traffic ratio and the like by using a data acquisition module positioned in the mobile private network.
Step 604: and transmitting the informationized data acquired by the data acquisition module.
In one embodiment of the invention, after acquiring the informationized data, the data acquisition module in the master network system transmits the acquired informationized data to the data sharing module; after the data acquisition module in the slave network system acquires the informationized data, the acquired informationized data are transmitted to the data transmission module in the same network system, and the data transmission module transmits the informationized data acquired by each data acquisition module in the same network system to the data sharing module. When the data transmission module transmits the information data to the data sharing module, the structured information data and the unstructured information data are respectively sent to the data sharing module through a SOAP protocol, an FTP protocol and a JDBC interface by using the included relational data transmission unit, the file transmission unit and the message queue.
For example, the internet includes 10 application servers, and a data acquisition module deployed on each application server acquires information data of the application server. The data transmission module deployed in the Internet comprises a relational data transmission unit, a file transmission unit and a message queue unit. The relational data transmission unit acquires the structured information data from the 10 data acquisition modules and sends the acquired structured information data to the data sharing module through the message queue unit; the file transmission unit acquires unstructured information data such as logs, documents, videos and audios from the 10 data acquisition modules and sends the acquired unstructured information data to the data sharing module through the message queue unit.
Step 605: and integrating the information data.
In an embodiment of the present invention, the data sharing module includes a data integration unit and a data storage unit, and after the data sharing module obtains the information data acquired by each data acquisition module, the data integration unit performs standardization processing and labeling processing on the information data, unifies formats of the information data, and adds a label to the information data.
Step 606: and storing the integrated information data in a storage server.
In an embodiment of the present invention, the data sharing module includes a data storage unit, and after the data integration unit completes processing of the information data, the data storage unit stores the structured information data in a relational database in the storage server according to a type of the information data, and stores the unstructured information data in a non-relational database in the storage server.
The system and the method for realizing data aggregation among network systems provided by the embodiments of the invention have at least the following beneficial effects:
1. in the embodiment of the invention, the system for realizing data aggregation between the network systems comprises a plurality of data acquisition modules respectively arranged on each application server, a plurality of data transmission modules respectively arranged in each slave network system and a data sharing module arranged in the master network system. Each data acquisition module acquires informationized data from the application server where the data acquisition module is located according to the index parameters, wherein the data acquisition module on the application server included in the slave network system transmits the acquired informationized data to the data transmission module arranged in the slave network system, and the data acquisition module on the application server included in the master network system transmits the acquired informationized data to the data sharing module arranged in the master network system. And the data sharing module stores the informationized data sent by the data acquisition module and the data transmission module into a storage space in the main network system. Therefore, after the informationized data of the application servers included in each slave network system are collected, the collected informationized data are transmitted to the data sharing module located in the master network system through the data transmission module, and finally the informationized data from the master network system and each slave network system are stored in the storage space located in the master network system through the data sharing module, so that the informationized data included in different network systems are aggregated.
2. In the embodiment of the invention, a user can define the index parameters through the background control module and send the defined index parameters to the corresponding data acquisition module through the background control module, so that the data acquisition module can acquire corresponding informationized data from the application server according to the received identification parameters. Therefore, users can define different index parameters according to requirements to collect different informationized data from each network system for aggregation, and the applicability of the data aggregation method and system is improved.
3. In the embodiment of the present invention, the data transmission module includes a relational data transmission unit, a file transmission unit, and a message queue unit, where the relational data transmission unit is configured to acquire structured information data acquired by the data acquisition module, the file transmission unit is configured to acquire unstructured information data acquired by the data acquisition unit, and the message queue unit is configured to send the information data acquired by the relational data transmission unit and the file transmission unit to the data sharing module. The structured data and the unstructured data are separated, and different interfaces and protocols can be adopted to transmit the structured data and the unstructured data, so that the speed and the reliability of information data transmission are ensured.
4. In the embodiment of the invention, the information data is transmitted to the data sharing module in a message queue mode through the message queue unit, and the message queue can transmit simple text strings and also can transmit complex data containing embedded objects, so that the structured information data and the unstructured information data can be normally transmitted; in addition, the data transmission mode based on the message queue can ensure that the information data can be successfully transmitted to the data sharing module, and ensure the reliability of the data system among the aggregation network systems.
5. In the embodiment of the invention, according to the type of the informationized data, the informationized data are respectively stored in the relational database and the non-relational database, and the informationized data aggregated together are reasonably stored, so that the informationized data are conveniently called from the storage space to be analyzed, and the informationized data stored in the storage space are conveniently managed.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it is to be noted that: the above description is only a preferred embodiment of the present invention, and is only used to illustrate the technical solutions of the present invention, and not to limit the protection scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (2)
1. A system for realizing data aggregation between network systems is characterized in that the system is applied to an information system comprising a master network system and at least one slave network system, and comprises:
the data acquisition module is respectively arranged on each application server, wherein the master network system and each slave network system comprise at least one application server;
the data transmission module is respectively arranged in each slave network system;
the data sharing module is arranged in the main network system;
each data acquisition module is used for acquiring informationized data from the application server according to the received index parameters;
each data transmission module is used for acquiring the informationized data acquired by the data acquisition module arranged on each application server in the slave network system where the data transmission module is positioned, and sending the acquired informationized data to the data sharing module;
the data sharing module is configured to acquire the informationized data acquired by the data acquisition module arranged on each application server in the master network system, receive the informationized data sent by each data transmission module, and store the acquired and received informationized data in a storage space in the master network system;
further comprising: a background control module;
the background control module is used for respectively forming the index parameters corresponding to each data acquisition module according to external triggering and sending the formed index parameters to the corresponding data acquisition modules;
each of the data transmission modules includes: the system comprises a relational data transmission unit, a file transmission unit and a message queue unit;
the relational data transmission unit is used for acquiring the structured information data acquired by the data acquisition module arranged on each application server in the slave network system, and sending the acquired structured information data to the message queue unit;
the file transmission unit is used for acquiring unstructured information data acquired by the data acquisition module arranged on each application server in the slave network system, and sending the acquired unstructured information data to the message queue unit;
the message queue unit is used for sending the structured information data from the relational data transmission unit and the unstructured information data from the file transmission unit to the data sharing module in a message queue mode;
the data sharing module comprises: the data integration unit and the data storage unit;
the data integration unit is used for carrying out standardization processing and labeling processing on the acquired and received information data;
the data storage unit is used for storing the informationized data processed by the data integration unit into the storage space;
the data storage unit is used for storing the structured information data into a relational database included in the storage space and storing the unstructured information data into a non-relational database included in the storage space according to the type of the information data;
further comprising: the data analysis module and the data display module;
the data analysis module is used for analyzing the development trend of the information system by using the information data stored in the storage space according to external triggering to obtain an analysis result;
and the data display module is used for displaying the analysis result obtained by the data analysis module.
2. A method for implementing data aggregation between networks by using the system for implementing data aggregation between networks in claim 1, comprising:
respectively utilizing each data acquisition module to acquire informationized data from the application server where the acquisition module is located according to the received index parameters;
acquiring the information data acquired by the data acquisition module arranged on each application server in the network system where each data transmission module is positioned by utilizing each data transmission module;
acquiring the informationized data acquired by the data acquisition module arranged on each application server in the main network system by using the data sharing module, receiving the informationized data sent by each data transmission module, and storing the acquired and received informationized data into a storage space in the main network system;
before the respectively using each data acquisition module to acquire the informationized data from the application server where the acquisition module is located according to the received index parameter, the method further includes:
respectively forming the index parameters corresponding to each data acquisition module according to external triggering by using the background control module, and sending the formed index parameters to the corresponding data acquisition modules;
and/or the presence of a gas in the gas,
the obtaining, by using each data transmission module, the information data acquired by the data acquisition module arranged on each application server in the network system where the data transmission module is located includes:
for each of the data transmission modules, performing:
acquiring structured information data acquired by the data acquisition module arranged on each application server in the network system where the data transmission module is positioned by using a relational data transmission unit included by the data transmission module;
acquiring unstructured information data, which are acquired by the data acquisition modules arranged on the application servers in the network system and are located by the data transmission modules, by using file transmission units included by the data transmission modules;
sending the structured information data acquired by the relational data transmission unit and the unstructured information data acquired by the file transmission unit to the data sharing module in a message queue manner by using a message queue unit included in the data transmission module;
the storing the acquired and received information data into a storage space in the master network system includes:
the acquired and received information data are subjected to standardization processing and labeling processing by using a data integration unit included by the data sharing module;
storing the informationized data processed by the data integration unit into the storage space by using a data storage unit included in the data sharing module;
after the storing the acquired and received information data into the storage space in the master network system, further comprising:
analyzing the development trend of the information system by utilizing the information data stored in the storage space according to external triggering by utilizing the data analysis module to obtain an analysis result;
and displaying the analysis result obtained by the data analysis module by using the data display module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710512510.0A CN107295087B (en) | 2017-06-29 | 2017-06-29 | System and method for realizing data aggregation between network systems |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710512510.0A CN107295087B (en) | 2017-06-29 | 2017-06-29 | System and method for realizing data aggregation between network systems |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107295087A CN107295087A (en) | 2017-10-24 |
| CN107295087B true CN107295087B (en) | 2020-06-02 |
Family
ID=60098244
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710512510.0A Active CN107295087B (en) | 2017-06-29 | 2017-06-29 | System and method for realizing data aggregation between network systems |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107295087B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111103847B (en) * | 2019-12-31 | 2023-01-24 | 中国兵器装备集团自动化研究所 | Analysis system and analysis method for real-time data stream of numerical control machine tool |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20110016749A (en) * | 2009-08-12 | 2011-02-18 | 주식회사 케이티 | Apparatus for multi-link data distribution/concentration to aggregate bandwidth in multi-connection wireless network |
| CN104361031A (en) * | 2014-10-24 | 2015-02-18 | 西安未来国际信息股份有限公司 | Big government data preprocessing system and method |
| CN105159951A (en) * | 2015-08-17 | 2015-12-16 | 成都中科大旗软件有限公司 | Open tourism multi-source heterogeneous data fusion method and system |
| CN106327396A (en) * | 2016-09-27 | 2017-01-11 | 中电科软件信息服务有限公司 | Hospital business data management platform and method |
| CN206164759U (en) * | 2016-11-18 | 2017-05-10 | 中国电力科学研究院 | Network multisource heterogeneous data collection system strides |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7539702B2 (en) * | 2004-03-12 | 2009-05-26 | Netapp, Inc. | Pre-summarization and analysis of results generated by an agent |
-
2017
- 2017-06-29 CN CN201710512510.0A patent/CN107295087B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20110016749A (en) * | 2009-08-12 | 2011-02-18 | 주식회사 케이티 | Apparatus for multi-link data distribution/concentration to aggregate bandwidth in multi-connection wireless network |
| CN104361031A (en) * | 2014-10-24 | 2015-02-18 | 西安未来国际信息股份有限公司 | Big government data preprocessing system and method |
| CN105159951A (en) * | 2015-08-17 | 2015-12-16 | 成都中科大旗软件有限公司 | Open tourism multi-source heterogeneous data fusion method and system |
| CN106327396A (en) * | 2016-09-27 | 2017-01-11 | 中电科软件信息服务有限公司 | Hospital business data management platform and method |
| CN206164759U (en) * | 2016-11-18 | 2017-05-10 | 中国电力科学研究院 | Network multisource heterogeneous data collection system strides |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107295087A (en) | 2017-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10803394B2 (en) | Integrated monitoring and communications system using knowledge graph based explanatory equipment management | |
| Wang et al. | A smart home gateway platform for data collection and awareness | |
| US20170109657A1 (en) | Machine Learning-Based Model for Identifying Executions of a Business Process | |
| CN109495533A (en) | A kind of intelligent things network management system and method | |
| US20170109667A1 (en) | Automaton-Based Identification of Executions of a Business Process | |
| CN104636421A (en) | Industrial monitoring using cloud computing | |
| CN109120719B (en) | Information pushing method, information display method, computer equipment and storage medium | |
| CN102710770A (en) | Identification method for network access equipment and implementation system for identification method | |
| US20170109639A1 (en) | General Model for Linking Between Nonconsecutively Performed Steps in Business Processes | |
| CN105184886A (en) | Cloud data center intelligence inspection system and cloud data center intelligence inspection method | |
| CN115034638A (en) | Digital twinning processing method and digital twinning system | |
| WO2020205693A1 (en) | Automatic generation of labeled data in iot systems | |
| Aboualola et al. | Edge technologies for disaster management: A survey of social media and artificial intelligence integration | |
| CN110784377A (en) | Method for uniformly managing cloud monitoring data in multi-cloud environment | |
| CN106559498A (en) | Air control data collection platform and its collection method | |
| CN109800133A (en) | A kind of method, one-stop monitoring alarm platform and the system of unified monitoring alarm | |
| US10009220B2 (en) | In-vehicle information system and information processing method thereof | |
| US20170109640A1 (en) | Generation of Candidate Sequences Using Crowd-Based Seeds of Commonly-Performed Steps of a Business Process | |
| CN107295087B (en) | System and method for realizing data aggregation between network systems | |
| CN111817865A (en) | Method for monitoring network management equipment and monitoring system | |
| CN105512318B (en) | Page generation method and device | |
| CN107342916A (en) | Monitoring method, device and the server of server info | |
| CN111917848A (en) | Data processing method based on edge computing and cloud computing cooperation and cloud server | |
| CN113750538A (en) | Big data-based hand-game security platform construction method and system | |
| CN116629802A (en) | Big data platform system for railway port station |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |