Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a visual data acquisition and display system, which can effectively solve the problems.
The technical scheme adopted by the invention is as follows:
the invention provides a visual data acquisition and display system, which comprises:
the data source definition module is used for defining different types of data sources;
the data source connection configuration module is used for setting connection configuration information of the data source defined by each data source definition module, and the data source can be accessed to the corresponding data source through the connection configuration information;
the data source connection module is used for connecting with the data source through a communication network according to the connection configuration information of the data source;
the data table directory acquisition module is used for acquiring data table directories of all data tables contained in the data source after the data table directory acquisition module is connected with the data source through a communication network; the data table catalog is composed of a plurality of data table names;
the data table previewing module is used for sending a data table previewing request carrying data table names to the data source for each data table name after the data table catalog is obtained; receiving a corresponding data sheet returned by the data source to realize a data sheet preview function;
the data model creating module is used for creating a data model; specifically, the method is used for operating the names of the multiple data tables acquired by the data table directory acquisition module and selecting at least two data tables needing to establish table association; configuring the type of the table association relation; then, acquiring a table field of each selected data table; configuring attribute information of each table field; the attribute information comprises attributes displayed or hidden by table fields; then, establishing the relevance among the table fields among the data tables, wherein the relevance among the table relevance relation and the table fields form the final relevance relation among the data tables, namely a formed data model;
the data acquisition configuration module is used for configuring data acquisition parameters; the data acquisition parameters comprise data acquisition starting time and data acquisition frequency;
the data storage mode configuration module is used for configuring a data storage mode;
the execution module is used for circularly calling the data model according to the data acquisition starting time and the data acquisition frequency configured by the data acquisition configuration module, acquiring the name of each data table with table association relation in the data model, and then acquiring the association field of each data table with the association between table fields in the data model; acquiring the type of the table association relation and the attribute information of the table field in the data model;
sending corresponding non-hidden target table field information in the query corresponding data table and table record information of the non-hidden target table field to the data source according to the data table name, the associated field, the type of the table association relation and the attribute information of the table field, namely querying the data table information after screening processing in the corresponding data table;
then, fusing the information of each screened data table returned by the data source according to the type of the table association relationship to obtain a fused data table; the data storage mode configured by the module is configured according to the data storage mode, and the fused data table is stored;
and the display module is used for analyzing the fused data table and displaying an analysis result.
Preferably, the data source definition module defines different types of data sources, including: a file type data source, a web interface type data source, a database type data source, and a data service type data source.
Preferably, the file type data source includes: an excel file type data source and a csv file type data source; the web interface type data source comprises: the device comprises a webservice interface type data source, an http interface type data source, a Splunk interface type data source and a RESTful interface type data source; the database type data source comprises: an Oracle type data source, a MySql type data source, a SQLServer type data source, a DB2 type data source, a PostgreSQL type data source, a SyBase type data source, an Informix type data source, and an H2 type data source; the data service type data source comprises a VBI data receiving service type data source.
Preferably, the connection configuration information of each data source configured by the data source connection configuration module includes:
analyzing a file corresponding to a file type data source for the file type data source, and importing the analyzed file data into a new data table of a database; the connection configuration information of the data source comprises: a name of a new data table and a database address where the new data table is stored;
for the web interface type data source, the connection configuration information comprises: connection address information of the web interface type data source;
for the database type data source, the connection configuration information comprises: database name, host address, port number, and user name and password for accessing the database;
for a data service type data source, the connection configuration information comprises: data service name and data service address information.
Preferably, the type of the table association relationship includes: left connection, right connection and inner connection.
Preferably, the data storage method includes: a real-time section storage mode, a coverage storage mode each time, an accumulation duplication-removing storage mode, an accumulation non-duplication-removing storage mode and an accumulation and updating acquisition time storage mode.
Preferably, the execution module is specifically configured to:
the method comprises the following steps of (1) respectively assuming that the names of each data table with table association relation in a data model are a data table A and a data table B; the data table A has n fields, wherein the n1 fields are non-hidden attributes, and the n-n1 fields are hidden attributes; the data table B has m fields, wherein m1 fields are of non-hidden attributes, and m-m1 fields are of hidden attributes; in the data model, association between a field C in an un-hidden attribute field in a data table A and a field D in an un-hidden attribute field in a data table B needs to be established; each field in data table a has x record values; each field in data table B has y record values; assuming that the table association of the current configuration is left connection; the data table A is a current data table; the data table B is a left data table of the data table A;
then: when the data acquisition time point is reached, the execution module acquires data from a corresponding data source, acquires m1 fields of the un-hidden attributes of the data table B and y record values of each field of the un-hidden attributes, and records the record values as a sub-table B; then acquiring x record values corresponding to the field C of the data table A; comparing y record values of a field D of the data table B with x record values of a field C of the data table A, screening to obtain record values of the field C of the data table A which are equal to the record values of the field D of the data table B, and assuming that the record values of the field C of the data table A obtained by screening are respectively a record value of the ith row and a record value of the jth row; then, the execution module sends the record values of all the non-hidden fields in the ith row and the record values of all the non-hidden fields in the jth row of the query data table A to the corresponding data source, receives the query result returned by the data source, and records the query result as a sub-table A;
and then, the execution module fuses the acquired sub-table B and the acquired sub-table A to obtain and store the fused data table.
The visual data acquisition and display system provided by the invention has the following advantages:
the visual rear-end acquisition module can directly correlate the A table and the B table to obtain correlated fusion data when data are acquired by using a java technology, correlation modeling can be performed between various types of data sources, and the visual rear-end acquisition module is wide in application range, simple to operate, small in data transmission quantity, small in consumed resource and simple and convenient to use.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects solved by the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In a traditional enterprise, if the interface and the interface are associated with the associated data or the interface data is immediately taken, the interface is customized and developed according to the interface, then the data is displayed, and if the displayed graph is changed or the attributes are changed, the code needs to be modified, so that the time and the labor are consumed. The method can abstract most types of data interfaces in an enterprise, so that the data between the interfaces can be associated on the fields, and only the display attribute of the page needs to be modified, the code does not need to be modified, the code is basically zero, and the method is suitable for the use and operation of workers.
The invention provides a visual data acquisition and presentation system, which comprises the following components with reference to fig. 1:
the data source definition module is used for defining different types of data sources; different types of data sources include, but are not limited to: a file type data source, a web interface type data source, a database type data source, and a data service type data source.
File type data sources include, but are not limited to: an excel file type data source and a csv file type data source;
the web interface type data sources include, but are not limited to: the device comprises a webservice interface type data source, an http interface type data source, a Splunk interface type data source and a RESTful interface type data source;
the database type data sources include, but are not limited to: a relational database data source and a columnar storage database; the relational database data sources include, but are not limited to, an Oracle type data source, a MySql type data source, an SQLServer type data source, a DB2 type data source, a PostgreSQL type data source, a SyBase type data source, an Informix type data source and an H2 type data source;
the data service type data source includes, but is not limited to, a VBI data receiving service type data source.
The data source connection configuration module is used for setting connection configuration information of the data source defined by each data source definition module, and the data source can be accessed to the corresponding data source through the connection configuration information;
for example, for a file type data source, analyzing a file corresponding to the file type data source, and importing the analyzed file data into a new data table of a database; the connection configuration information of the data source comprises: a name of a new data table and a database address where the new data table is stored;
for the web interface type data source, the connection configuration information comprises: connection address information of the web interface type data source;
for the database type data source, the connection configuration information comprises: database name, host address, port number, and user name and password for accessing the database;
for a data service type data source, the connection configuration information comprises: data service name and data service address information.
The data source connection module is used for connecting with the data source through a communication network according to the connection configuration information of the data source;
the data table directory acquisition module is used for acquiring data table directories of all data tables contained in the data source after the data table directory acquisition module is connected with the data source through a communication network; the data table catalog is composed of a plurality of data table names;
the data table previewing module is used for sending a data table previewing request carrying data table names to the data source for each data table name after the data table catalog is obtained; receiving a corresponding data sheet returned by the data source to realize a data sheet preview function;
the data model creating module is used for creating a data model; specifically, the method is used for operating the names of the multiple data tables acquired by the data table directory acquisition module and selecting at least two data tables needing to establish table association; configuring the type of table incidence relation, wherein the type of the table incidence relation comprises but is not limited to left connection, right connection and inner connection; then, acquiring a table field of each selected data table; configuring attribute information of each table field; the attribute information comprises attributes displayed or hidden by table fields; in addition, for the table field, the name of the field to be displayed can be selected to be modified, and the field index is specified; then, establishing the relevance among the table fields among the data tables, wherein the relevance among the table relevance relation and the table fields form the final relevance relation among the data tables, namely a formed data model;
the data acquisition configuration module is used for configuring data acquisition parameters; the data acquisition parameters comprise data acquisition starting time and data acquisition frequency;
the data storage mode configuration module is used for configuring a data storage mode; the data storage mode comprises the following steps: a real-time section storage mode, a coverage storage mode each time, an accumulation duplication-removing storage mode, an accumulation non-duplication-removing storage mode and an accumulation and updating acquisition time storage mode.
The execution module is used for circularly calling the data model according to the data acquisition starting time and the data acquisition frequency configured by the data acquisition configuration module, acquiring the name of each data table with table association relation in the data model, and then acquiring the association field of each data table with the association between table fields in the data model; acquiring the type of the table association relation and the attribute information of the table field in the data model;
sending corresponding non-hidden target table field information in the query corresponding data table and table record information of the non-hidden target table field to the data source according to the data table name, the associated field, the type of the table association relation and the attribute information of the table field, namely querying the data table information after screening processing in the corresponding data table;
then, fusing the information of each screened data table returned by the data source according to the type of the table association relationship to obtain a fused data table; the data storage mode configured by the module is configured according to the data storage mode, and the fused data table is stored;
the execution module communicates with each data source, and the manner of acquiring the query message may be: the execution module is connected with all relational databases through jdbc, generates different SQL statements to retrieve data according to different relational databases, and other types of data sources retrieve data independently and perform mutual association according to SQL ideas.
The execution module is specifically configured to:
the method comprises the following steps of (1) respectively assuming that the names of each data table with table association relation in a data model are a data table A and a data table B; the data table A has n fields, wherein the n1 fields are non-hidden attributes, and the n-n1 fields are hidden attributes; the data table B has m fields, wherein m1 fields are of non-hidden attributes, and m-m1 fields are of hidden attributes; in the data model, association between a field C in an un-hidden attribute field in a data table A and a field D in an un-hidden attribute field in a data table B needs to be established; each field in data table a has x record values; each field in data table B has y record values; assuming that the table association of the current configuration is left connection; the data table A is a current data table; the data table B is a left data table of the data table A; wherein n, n1, m and m1 are natural numbers;
then: when the data acquisition time point is reached, the execution module acquires data from a corresponding data source, acquires m1 fields of the un-hidden attributes of the data table B and y record values of each field of the un-hidden attributes, and records the record values as a sub-table B; then acquiring x record values corresponding to the field C of the data table A; comparing y record values of a field D of the data table B with x record values of a field C of the data table A, screening to obtain record values of the field C of the data table A which are equal to the record values of the field D of the data table B, and assuming that the record values of the field C of the data table A obtained by screening are respectively a record value of the ith row and a record value of the jth row; then, the execution module sends the record values of all the non-hidden fields in the ith row and the record values of all the non-hidden fields in the jth row of the query data table A to the corresponding data source, receives the query result returned by the data source, and records the query result as a sub-table A;
and then, the execution module fuses the acquired sub-table B and the acquired sub-table A to obtain and store the fused data table.
In the description of the principle of the above execution module, the left connection is taken as an example for description. Of course, in practical application, there are right connection and inner connection conditions, the implementation principle of which is basically the same as that of the left connection, if the right connection is adopted, all fields of the un-hidden attributes in the right table and the record value of each field of the un-hidden attributes are returned to form a sub table 1; for the left table, assuming that an association relationship between the field a in the left table and the field b in the right table needs to be established, comparing the record value of the field a in the left table with the record value of the field b in the right table, and screening to obtain a record value of the field a equal to the record value of the field b in the right table, for example, the 1 st record value; then, acquiring the record values of all the unhidden fields in the first row in the left table from the data source to form a sub-table 2; and fusing the sub-tables 2 and 1.
For the internal connection situation, assuming that an association relationship between a field a in the table a and a field B in the table B needs to be established, first, only the record value of the field a in the table a is obtained, then the record value of the field B in the table B is obtained, and rows with equal record values are obtained through comparison, for example, the record value of the 2 nd row of the field a in the table a is equal to the record value of the 3 rd row of the field B in the table B, then the record values of all the non-hidden fields in the 2 nd row of the table a are obtained from the data source, and a sub-table 1 is formed; then obtaining the record values of all the un-hidden fields in the 3 rd row in the table B to form a sub-table 2; and fusing the sub-tables 2 and 1.
The execution principle of the execution module can be simply described as follows: the execution module analyzes the content of the data model according to the timing configuration cycle, acquires data in a corresponding data source according to the existing rule according to the data source information corresponding to the data table in the data model, then performs correlation and data filtering according to the correlation fields of the data source and the data source (performs cycle comparison according to the correlation fields of the main table and the sub table, performs combination according to the equal correlation relation, and finally synthesizes a group of data), adds a screening condition, processes the returned data according to the field information defined in the model, and finally stores the processed data in an independent table.
And the display module is used for analyzing the fused data table and displaying an analysis result.
In fig. 1, a total of 13 steps are depicted, each of which is described below:
step 1: defining a data source and filling information of the data source;
step 2: defining connection configuration information of a data source;
and step 3: modeling data;
and 4, step 4: defining a data model;
and 5-9: the data model analyzes the content of the data model according to the timing configuration cycle, acquires data in each data source according to the existing rule according to the data source information corresponding to the data source table in the model, then performs correlation and data filtering according to the correlation fields of the data source and the data source (performs cycle comparison according to the correlation fields of the main table and the sub table, performs combination according to equal correlation relations, and finally synthesizes a group of data), adds screening conditions, processes the returned data according to the field information defined in the model, and finally stores the processed data in an independent table.
Step 10: and scheduling tasks at fixed time, wherein the execution module automatically calls the function of the data model at fixed time according to scheduling task information stored in the data model, returns the data information at fixed time and stores the data information into a database, and simultaneously informs a multidimensional analysis engine to inquire the stored table in the database and inquire the table according to the inquiry condition of a certain control in the picture.
Step 11: and inquiring data in a local database according to the data model information bound at the front end, and pushing the data to a multidimensional analysis engine, wherein the local database only supports MySql and Oracle databases at present.
Step 12: and (4) performing multi-dimensional analysis, wherein the multi-dimensional analysis refers to analysis performed by placing the multi-dimensional analysis on more than two spatial coordinates. Processing, including conversion of fields, is performed primarily when the analysis is built in the picture, and the data is pushed to the front end through ZMQ.
Step 13: ZMQ is a transport layer, a message processing queue library, which is flexible between multiple threads, cores, and host boxes.
ZMQ fall into three modes: the invention uses a subscription and publication model. The issuing end only sends data in one direction and does not care whether all information is sent to the subscribing end. If the subscriber end is not successfully connected when the publisher end starts to publish the information, the information is directly discarded. However, once the connection of the subscriber is successful, the middle part can ensure that no information is lost. Similarly, the subscriber is only responsible for receiving, but not for feedback. If the publisher and the subscriber need to interact (for example, to confirm whether the subscriber is connected), an additional socket is used to satisfy the requirement by adopting a request response model.
The visual data acquisition and display system provided by the invention can acquire data from various data formats such as real-time interfaces, relational databases, files, column databases and the like before displaying the data, preliminarily process the data, perform data modeling, data association and condition filtering on the data from different sources during modeling, and then perform multidimensional analysis or push the result to a PC large screen, a WEB page or a mobile phone APP for display in real time through mathematical statistics.
The visual data acquisition and display system provided by the invention has the following advantages:
the visual rear-end acquisition module can directly correlate the A table and the B table to obtain correlated fusion data when data are acquired by using a java technology, correlation modeling can be performed between various types of data sources, and the visual rear-end acquisition module is wide in application range, simple to operate, small in data transmission quantity, small in consumed resource and simple and convenient to use.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and improvements can be made without departing from the principle of the present invention, and such modifications and improvements should also be considered within the scope of the present invention.