CN113094385A - Data sharing fusion platform and method based on software definition open toolset - Google Patents
Data sharing fusion platform and method based on software definition open toolset Download PDFInfo
- Publication number
- CN113094385A CN113094385A CN202110261104.8A CN202110261104A CN113094385A CN 113094385 A CN113094385 A CN 113094385A CN 202110261104 A CN202110261104 A CN 202110261104A CN 113094385 A CN113094385 A CN 113094385A
- Authority
- CN
- China
- Prior art keywords
- data
- platform
- interface
- sharing
- management
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 84
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000007726 management method Methods 0.000 claims abstract description 113
- 238000012545 processing Methods 0.000 claims abstract description 44
- 238000012544 monitoring process Methods 0.000 claims abstract description 41
- 238000004458 analytical method Methods 0.000 claims abstract description 26
- 238000013480 data collection Methods 0.000 claims abstract description 19
- 238000004364 calculation method Methods 0.000 claims abstract description 17
- 238000003860 storage Methods 0.000 claims description 26
- 238000011156 evaluation Methods 0.000 claims description 22
- 238000007689 inspection Methods 0.000 claims description 16
- 238000013475 authorization Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 14
- 238000004422 calculation algorithm Methods 0.000 claims description 13
- 238000007405 data analysis Methods 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 13
- 238000011157 data evaluation Methods 0.000 claims description 11
- 238000013507 mapping Methods 0.000 claims description 10
- 238000002156 mixing Methods 0.000 claims description 8
- 238000013523 data management Methods 0.000 claims description 6
- 230000003993 interaction Effects 0.000 claims description 5
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 238000013441 quality evaluation Methods 0.000 claims description 4
- 230000000007 visual effect Effects 0.000 claims description 4
- 238000007418 data mining Methods 0.000 claims description 3
- 238000003745 diagnosis Methods 0.000 claims description 3
- 230000000977 initiatory effect Effects 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 claims description 3
- 238000013024 troubleshooting Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 40
- 230000006870 function Effects 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 9
- 238000013524 data verification Methods 0.000 description 7
- 238000011161 development Methods 0.000 description 7
- 230000010354 integration Effects 0.000 description 6
- 230000002452 interceptive effect Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000006185 dispersion Substances 0.000 description 2
- 244000035744 Hura crepitans Species 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 210000001503 joint Anatomy 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data sharing and fusing platform and method based on a software definition open toolset. The scheme comprises a data sharing and fusion platform and a data sharing and fusion tool; the data sharing and co-fusion platform comprises a sharing open data collection interface, a data calculation interface, a component service interface, a data use interface, a data warehouse, an operating environment, an extensible container engine, a metadata management module, a resource catalog management module, a data quality management module and a platform service management and operation monitoring module; the data sharing fusion tool comprises a data processing tool and a data sharing tool. According to the scheme, a data sharing fusion tool is established in a software definition mode, and the shared fusion data is supervised, fused and subjected to scene analysis by combining a data sharing fusion platform, so that the problems of low efficiency and high cost of information isolated island data are solved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data sharing and fusing platform and method based on a software defined open toolset.
Background
The fusion and integration of information resources can form wide data sharing, so that the flow of data resources and basic data is accelerated, the business innovation among all departments is further promoted, and the method has important significance for meeting public government requirements and cross-department cooperation. In recent years, the technology related to the fusion and integration of information resources has been rapidly developed. For example, information highway, visiting united states engineering, etc. all promote the continuous progress of information resource fusion and integration technology.
Before the patent technology, the existing technology is mainly used for collecting data resources, the development and utilization level is extremely low, and information islands are easy to occur. The information island multi-source data analysis sharing use process is complex, and different business systems need different shared data supports. In order to meet the requirements of different business analysis use, data in the current sharing platform are basically stored in various data sources in a dispersed manner, and the multi-source data analysis environment is complex due to different storage system interfaces, so that the data use efficiency of the sharing platform is low. Meanwhile, a unified data view is lacked in upper-layer application development, so that the use cost is high. Therefore, in the process of constructing a business system, an information isolated island occurs, so that multiple copies of data are caused, and the high cost required by the business cannot be supported.
Disclosure of Invention
In view of the above problems, the invention provides a data sharing and fusion platform and a method based on a software definition open tool set, which establish a data sharing and fusion tool through a software definition mode, and perform supervision, fusion and scene analysis on shared and fused data by combining the data sharing and fusion platform, thereby solving the problems of low efficiency and high cost of information isolated island data.
According to a first aspect of the embodiments of the present invention, a data sharing and fusion platform based on a software defined open toolset is provided. The platform comprises a data sharing and fusion platform and a data sharing and fusion tool;
the data sharing and co-fusion platform comprises a sharing open data collection interface, a data calculation interface, a component service interface, a data use interface, a data warehouse, an operating environment, an extensible container engine, a metadata management module, a resource catalog management module, a data quality management module and a platform service management and operation monitoring module;
the data sharing fusion tool comprises a data processing tool and a data sharing tool;
the shared open data collection interface is used for receiving multi-source information outside the data sharing fusion platform;
the data calculation interface is used for feeding back data obtained by calculation to the outside of the data sharing fusion platform;
the component service interface is used for carrying out data interaction between the data sharing and fusion platform and the data sharing and fusion tool;
and the data use interface carries out extended application on the data after data processing outside the data sharing fusion platform.
In one or more embodiments, preferably, the metadata management module specifically includes:
the metadata storage submodule is used for managing all interface metadata in a data source interface, a message middleware and front-end display and storing the interface metadata in a classified mode according to technical metadata and service metadata;
the metadata query submodule is used for querying and retrieving metadata basic information in the metadata storage submodule and returning query content information, wherein the query content information comprises a database table, a dimension table, indexes, processes and object information participating in input and output;
the metadata checking submodule is used for checking the integrity and the rationality of metadata information, wherein the checking content comprises middleware sql analysis success rate, table level relation integrity rate and field relation integrity rate;
and the metadata checking submodule is used for checking the validity of the parameters by using the JSR303 according to the resource directory management module.
In one or more embodiments, preferably, the resource catalog management module specifically includes:
the information resource catalog is used for inputting a department catalog, a basic catalog and a main body catalog and maintaining and managing the department catalog, the basic catalog and the main body catalog, wherein the main body catalog comprises a population, a legal person and an electronic certificate;
the government affair information resources comprise front segment codes, message middleware and rear segment codes, wherein the front segment codes are used for representing the classification of the government affair information resources, the rear segment codes are used for representing the sequence of the government affair information resources, and the message middleware is used for storing message contents;
the directory query submodule is used for querying and checking the corresponding information resources according to the directory of the government affair information resources, wherein the query of the government affair information resources needs to obtain query resource authorization;
and the directory authority submodule is used for the management personnel to authorize for different user levels, wherein the authorization content comprises viewing resource authorization, inquiring resource authorization and modifying resource authorization.
In one or more embodiments, preferably, the data quality management module specifically includes:
the data quality management submodule is used for performing quality rule management, quality rule execution, data quality monitoring, quality problem management and quality evaluation report generation;
the data evaluation management submodule is used for carrying out data use evaluation, data relation evaluation, timeliness evaluation, redundant data evaluation and importance evaluation;
and the data inspection management submodule is used for performing inspection rule management, inspection task scheduling and inspection result analysis.
In one or more embodiments, preferably, the platform service management and operation monitoring module specifically includes:
the platform service management submodule is used for registering services, discovering services, checking started services and checking service states;
the cluster management submodule is used for initiating a problem troubleshooting notice to a technical responsible person when the interface is not normally used in the processes of calling and cluster monitoring;
the collection monitoring submodule is used for providing monitoring information of the exchange task in a graphical interface mode, and the monitoring information comprises task scheduling and real-time flow;
the monitoring alarm submodule is used for carrying out state diagnosis and display on abnormal, alarming and fault conditions;
the log management submodule is used for inquiring the log content according to the operation result, the operation type, the operation time and the operation user;
and the performance monitoring submodule is used for measuring performance data at intervals of 5 minutes and inquiring historical data and real-time data according to the performance data.
In one or more embodiments, preferably, the data sharing tool specifically includes:
the operation management component is used for collecting and classifying the operation condition of the data sharing and blending platform and sending the operation condition data to a manager according to a preset operation and maintenance monitoring standard;
the storage management component comprises distributed file storage and a distributed memory database;
the retrieval management component is used for providing a retrieval entrance and carrying out full-text retrieval, sub-library statistics and record viewing according to the keywords;
the data interface component comprises an authentication interface, a data collection interface component, a data calculation interface component, a data use interface component and a platform service interface, wherein the authentication interface, the data collection interface component, the data calculation interface component, the data use interface component and the platform service interface all comprise request URLs and request methods;
the open source platform component library comprises a batch acquisition component, a distributed file system, a batch processing engine, a memory type batch processing engine, a main key query retrieval component, a full-text retrieval component, an acquisition tool, a distributed message component, a storage query component, an SQL batch processing engine, a resource scheduling engine, an open source computing framework and a stream processing engine.
In one or more embodiments, preferably, the data processing tool specifically includes:
a data exchange component for managing data exchange nodes and data exchange rules;
the data checking component is used for checking the type, the length, the null, the precision, the range and the format of the data;
the data management component is used for organizing all data, metadata and data of the data sharing and blending platform into a visual interface to be displayed to an administrator, and supporting the administrator to inquire and update resources, catalogues and metadata on line;
the data processing component is used for acquiring the service data from the data sharing platform, and the extraction mode is full extraction or incremental extraction according to the service;
and the data analysis component is used for analyzing each data task and carrying out data mining analysis according to an integrated algorithm, wherein the integrated algorithm comprises a text semantic analysis algorithm and an association rule operation algorithm.
According to a second aspect of the embodiments of the present invention, a data sharing and fusing method based on a software defined open toolset is provided. The method comprises the following steps:
the data sharing and blending platform acquires all multi-source data information;
carrying out data sharing and collection on the multi-source data information to generate shared open data, wherein the shared open data comprises non-encrypted data and encrypted data;
directly accessing the non-encrypted data to the data sharing and co-fusing platform, and storing the encrypted data in a target block chain in a preset format by using a block chain;
performing data processing of a preset format, a preset classification and a preset interface on the multi-source data information, and storing the multi-source data information in a data warehouse;
performing data collection, data fusion and authority management on the data sharing and co-fusion platform, and storing the multi-source data information into metadata, interface data and multi-source sharing data in a preset form;
establishing a structured mapping table for the multi-source data information through a data sharing and fusing tool and the data sharing and fusing platform;
establishing virtual structure data in the data sharing and co-fusion platform;
and accessing the structured mapping table by executing SQL sentences, and calling and searching the virtual structure data.
According to a third aspect of embodiments of the present invention, there is provided a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method according to any one of the first aspect of embodiments of the present invention.
According to a fourth aspect of embodiments of the present invention, there is provided an electronic device, comprising a memory and a processor, the memory being configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the steps of any one of the first aspect of embodiments of the present invention.
The technical scheme provided by the embodiment of the invention can have the following beneficial effects:
1) when the embodiment of the invention opens the data access service for the external visitor, the embodiment of the invention is in consideration of safety, and supports the establishment of a virtual structure table to shield the real metadata information and the example information of the physical data source of the remote access request.
2) The embodiment of the invention provides a unified metadata and SQL access interface, before accessing a multi-source database, a structured mapping table is created for the multi-source database through the SQL syntax provided by a data sharing and fusion tool and a platform, and the data query efficiency is improved and the application development difficulty is reduced by executing the SQL sentence to access the just created structured mapping table.
3) According to the embodiment of the invention, the external data is subjected to collection fusion analysis through the data sharing fusion platform, so that the multi-source data processing flow is closed in one platform, the data flow is more efficient, the interactive data query is realized based on the platform, and the data use efficiency is improved.
4) According to the embodiment of the invention, fusion analysis of multi-source data in one platform is realized through a data sharing and fusion platform, the same data is processed and shared by multiple data, data is prevented from being moved, the problems of data dispersion and repeated storage are solved, the data query speed is improved, unified resource scheduling is realized through platform service management, and the utilization rate of data resources is improved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a block diagram of a data sharing fusion platform based on a software defined open toolset according to an embodiment of the present invention.
Fig. 2 is an application diagram of a data sharing and fusion platform based on a software defined open toolset according to an embodiment of the present invention.
FIG. 3 is a resource data consulting flow diagram of the resource catalog management module according to one embodiment of the invention.
FIG. 4 is a diagram illustrating a query flow of the directory query submodule in an embodiment of the invention.
Fig. 5 is a schematic diagram of a rights setting flow of the directory rights submodule in an embodiment of the present invention.
Fig. 6 is a schematic diagram of an evaluation management flow of the data evaluation management submodule in an embodiment of the present invention.
Fig. 7 is a schematic diagram of a service management flow of the platform service management submodule in an embodiment of the present invention.
Fig. 8 is a schematic log management flow diagram of the log management submodule in an embodiment of the present invention.
FIG. 9 is a flow diagram illustrating the execution of a data check component in one embodiment of the invention.
FIG. 10 is a data parsing flow diagram of the data verification component in an embodiment of the invention.
FIG. 11 is a data classification flow diagram of the data verification component in one embodiment of the invention.
FIG. 12 is a flow diagram illustrating the execution of data processing components in one embodiment of the invention.
FIG. 13 is a flow diagram illustrating the execution of a data management component in one embodiment of the invention.
FIG. 14 is a flow diagram illustrating the execution of a data analysis component in one embodiment of the invention.
Fig. 15 is a multi-level technical architecture diagram of a data sharing fusion platform based on a software defined open toolset according to an embodiment of the present invention.
Fig. 16 is a flowchart of a data sharing and merging method based on the software defined open toolset according to an embodiment of the present invention.
FIG. 17 is a block diagram of an electronic device in one embodiment of the invention.
Detailed Description
In some of the flows described in the present specification and claims and in the above figures, a number of operations are included that occur in a particular order, but it should be clearly understood that these operations may be performed out of order or in parallel as they occur herein, with the order of the operations being indicated as 101, 102, etc. merely to distinguish between the various operations, and the order of the operations by themselves does not represent any order of performance. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor limit the types of "first" and "second" to be different.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The fusion and integration of information resources can form wide data sharing, so that the flow of data resources and basic data is accelerated, the business innovation among all departments is further promoted, and the method has important significance for meeting public government requirements and cross-department cooperation. In recent years, the technology related to the fusion and integration of information resources has been rapidly developed. For example, information highway, visiting united states engineering, etc. all promote the continuous progress of information resource fusion and integration technology.
Before the patent technology, the existing technology is mainly used for collecting data resources, the development and utilization level is extremely low, and information islands are easy to occur. The information island multi-source data analysis sharing use process is complex, and different business systems need different shared data supports. In order to meet the requirements of different business analysis use, data in the current sharing platform are basically stored in various data sources in a dispersed manner, and the multi-source data analysis environment is complex due to different storage system interfaces, so that the data use efficiency of the sharing platform is low. Meanwhile, a unified data view is lacked in upper-layer application development, so that the use cost is high. Therefore, in the process of constructing a business system, an information isolated island occurs, so that multiple copies of data are caused, and the high cost required by the business cannot be supported.
The embodiment of the invention provides a data sharing and fusing platform and method based on a software defined open toolset. According to the scheme, a data sharing fusion tool is established in a software definition mode, and the shared fusion data is supervised, fused and subjected to scene analysis by combining a data sharing fusion platform, so that the problems of low efficiency and high cost of information isolated island data are solved.
According to a first aspect of the embodiments of the present invention, a data sharing and fusion platform based on a software defined open toolset is provided. Fig. 1 is a block diagram of a data sharing fusion platform based on a software defined open toolset according to an embodiment of the present invention. As shown in fig. 1, the platform includes a data sharing and fusion platform 101 and a data sharing and fusion tool 102;
the data sharing and sharing platform 101 comprises a sharing open data collection interface 103, a data calculation interface 104, a component service interface 105, a data use interface 106, a data warehouse 107, a running environment 108, an extensible container engine 109, a metadata management module 110, a resource catalog management module 111, a data quality management module 112, and a platform service management and running monitoring module 113;
the data sharing fusion tool comprises a data processing tool 114 and a data sharing tool 115;
the shared open data collection interface 103 is used for receiving multi-source information outside the data sharing fusion platform;
the data calculation interface 104 is used for feeding back the data obtained by calculation to the outside of the data sharing fusion platform;
the component service interface 105 is used for performing data interaction between the data sharing and fusion platform and the data sharing and fusion tool;
the data use interface 106 performs extended application on the data after data processing outside the data sharing fusion platform.
In the embodiment of the invention, the data sharing and fusing platform and the data sharing and fusing tool realize mutual service registration and discovery through Eureka. The data sharing and fusing platform is realized by a micro-service architecture. Eureka is a representative state transition based service, which is the fundamental component of service registration and discovery. The Eureka client registers own service information to the server, maintains the information consistency of the Eureka client, brings convenience for other services to find the Eureka client, and acquires other service information depended by the Eureka client through the Eureka server to complete service calling. Specifically, registration means that the service provider registers its own metadata with the Eureka server side for service discovery. Discovery means that the Eureka client acquires registered other service information through the Eureka server, so as to find the required service to initiate remote invocation.
Therefore, if the data sharing fusion platform calls a component in the data sharing fusion tool, firstly, the component service is started, the metadata of the component service is registered to the Eureka server for service discovery, and when the application is started, the Eureka client registers the service information of the component service to the Eureka server and caches the service information of the Eureka server locally. The Eureka client will periodically perform heartbeat interaction with the Eureka server to update service leases and service information, and maintain and update the validity of the service instance metadata in the registry by sending heartbeats to the Eureka server. And within a certain time, the heartbeat information of the Eureka client is not received by the Eureka server, the Eureka server is off-line by default, and the service instance information is deleted from the registry. Service instance metadata is actively logged out of the Eureka server when the service is closed, and service provider instance data is deleted from the Eureka server's registry.
Fig. 2 is an application diagram of a data sharing and fusion platform based on a software defined open toolset according to an embodiment of the present invention. As shown in fig. 2, the data sharing and fusing platform implements functions of metadata management, resource directory management, data quality management, platform service management, platform operation monitoring, data collection interface, data use interface, data calculation interface, and component service interface. The data aggregation interface realizes data sharing and aggregation, and shared open data is divided into two types. 1) Non-encrypted data is directly imported into the platform through a shared open data attribution interface of the platform, and 2) encrypted data is stored into a block chain by using a block chain client based on an intelligent contract and is shared according to data among members of a data open authority alliance in the intelligent contract. The data computing interface mainly realizes a butt joint channel of external data and a data sharing fusion platform and realizes high-performance computing processing of the data. The component service interface mainly realizes that the registration and discovery service of the component is a channel between the data sharing and fusion platform and the data sharing and fusion tool. The data use interface mainly realizes data application of a data sharing and fusion platform, and comprises interactive application, search application, real-time report forms, real-time decision analysis, enterprise credit investigation management, institution financial risk and the like.
In one or more embodiments, preferably, the metadata management module 110 specifically includes:
the metadata storage submodule is used for managing all interface metadata in a data source interface, a message middleware and front-end display and storing the interface metadata in a classified mode according to technical metadata and service metadata;
the metadata query submodule is used for querying and retrieving metadata basic information in the metadata storage submodule and returning query content information, wherein the query content information comprises a database table, a dimension table, indexes, processes and object information participating in input and output;
the metadata checking submodule is used for checking the integrity and the rationality of metadata information, wherein the checking content comprises middleware sql analysis success rate, table level relation integrity rate and field relation integrity rate;
specifically, Structured Query Language (SQL), which is a database Query and programming Language for accessing data and querying, updating, and managing a relational database system, is used for short.
And the metadata checking submodule is used for checking the validity of the parameters by using the JSR303 according to the resource directory management module.
Specifically, JSR is an abbreviation for Java Specification Requests, and represents a Java Specification proposal. A formal request form of a standardized technical specification.
In the embodiment of the invention, the data of all the data sources are collected into the storage system of the data sharing and fusing platform. All data are stored according to the resource allocation classification. When the query is carried out, the data under the resource can be consulted through specific conditions. First, it is necessary to perform objective management to realize management of department directory, basic directory, subject directory, and each department government information resource directory. Inside this directory are all government information resources. Including basic catalog management of population, legal person and electronic certificate. The government affair information resource code structure is composed of a front segment code, a message middleware and a rear segment code. The front segment code consists of category, item, detail and item, and is used as a classification code of government affair information resources; the message middleware is actual transmission content; the back segment code is the sequence code of the government affair information resource. The directory query group is used for realizing multi-dimensional directory query and checking authorized information resources under the directory according to result sorting. The directory authority management is used for authorizing managers for different user levels and meeting the authority requirements of different users for directory browsing and retrieval.
FIG. 3 is a resource data consulting flow diagram of the resource catalog management module according to one embodiment of the invention. As shown in fig. 3, in one or more embodiments, preferably, the resource catalog management module 111 specifically includes:
the information resource catalog is used for inputting a department catalog, a basic catalog and a main body catalog and maintaining and managing the department catalog, the basic catalog and the main body catalog, wherein the main body catalog comprises a population, a legal person and an electronic certificate;
the government affair information resources comprise front segment codes, message middleware and rear segment codes, wherein the front segment codes are used for representing the classification of the government affair information resources, the rear segment codes are used for representing the sequence of the government affair information resources, and the message middleware is used for storing message contents;
the directory query submodule is used for querying and checking the corresponding information resources according to the directory of the government affair information resources, wherein the query of the government affair information resources needs to obtain query resource authorization;
and the directory authority submodule is used for the management personnel to authorize for different user levels, wherein the authorization content comprises viewing resource authorization, inquiring resource authorization and modifying resource authorization.
FIG. 4 is a diagram illustrating a query flow of the directory query submodule in an embodiment of the invention. As shown in fig. 4, by selecting different data classifications and secondary classifications of data, and further retrieving keywords, multi-dimensional directory query is realized through a query list.
Fig. 5 is a schematic diagram of a rights setting flow of the directory rights submodule in an embodiment of the present invention. As shown in fig. 5, the user performs role selection to match different types of permission groups, and then performs data permission management according to the permission group type. Finally, the management personnel can authorize different user levels, and the permission requirements of different users on directory browsing and retrieval are met.
In one or more embodiments, preferably, the data quality management module 112 specifically includes:
the data quality management submodule is used for performing quality rule management, quality rule execution, data quality monitoring, quality problem management and quality evaluation report generation;
the data evaluation management submodule is used for carrying out data use evaluation, data relation evaluation, timeliness evaluation, redundant data evaluation and importance evaluation;
and the data inspection management submodule is used for performing inspection rule management, inspection task scheduling and inspection result analysis.
In the embodiment of the invention, the implementation comprises the support of quality rule management, quality rule execution, data quality monitoring, quality problem management and quality evaluation reporting. The data evaluation management is used for realizing data usage evaluation, data relation evaluation, timeliness evaluation, redundant data evaluation and importance evaluation. The data inspection management is used for realizing inspection rule management, inspection task scheduling and inspection result analysis.
Fig. 6 is a schematic diagram of an evaluation management flow of the data evaluation management submodule in an embodiment of the present invention. And classifying legal data and marking non-legal data through data legality judgment, further selecting combination conditions of data attributes to form customized data, and further realizing data use evaluation, data relation evaluation, timeliness evaluation, redundant data evaluation and importance evaluation.
In one or more embodiments, preferably, the platform service management and operation monitoring module 113 specifically includes:
the platform service management submodule is used for registering services, discovering services, checking started services and checking service states;
the cluster management submodule is used for initiating a problem troubleshooting notice to a technical responsible person when the interface is not normally used in the processes of calling and cluster monitoring;
the collection monitoring submodule is used for providing monitoring information of the exchange task in a graphical interface mode, and the monitoring information comprises task scheduling and real-time flow;
the monitoring alarm submodule is used for carrying out state diagnosis and display on abnormal, alarming and fault conditions;
the log management submodule is used for inquiring the log content according to the operation result, the operation type, the operation time and the operation user;
and the performance monitoring submodule is used for measuring performance data at intervals of 5 minutes and inquiring historical data and real-time data according to the performance data.
In the embodiment of the invention, the platform service manages functions for managing various services, including registering/discovering services, checking all services of the system, checking started services of the current system, and checking service states of specified services. The platform operation monitor is used for cluster management, collection monitoring, monitoring alarm, log management and performance monitoring.
Specifically, the cluster management aims at a series of problems that an interface cannot be used normally or the cluster pressure is too high to provide services normally due to some reason in the interface calling and system cluster monitoring processes, and related responsible persons must be notified and processed. And node management and cluster monitoring management are realized.
Specifically, the collection monitoring is used for providing a platform administrator with a graphical interface to monitor the exchange tasks, and mainly comprises task scheduling monitoring and real-time flow monitoring. The information and the state of the front-end processor are displayed in a graphic mode, and the information and the state comprise the running state, the system performance and the resource occupation condition of each front-end processor, the log and the system abnormal condition, and the switching service running condition deployed by each front-end switching subsystem.
Fig. 7 is a schematic diagram of a service management flow of the platform service management submodule in an embodiment of the present invention. As shown in fig. 7, the creator and the manager check and apply for resources, the auditing and executing personnel create, issue and confirm, the tester judges the service issue state and the sandbox test result, and finally the observation element gives the final result, so as to support the function of managing various services, including registering/discovering services and checking all services of the system; checking the started service of the current system; the service status of the specified service is viewed.
Fig. 8 is a schematic log management flow diagram of the log management submodule in an embodiment of the present invention. As shown in fig. 8, more detailed log information is obtained by querying information and selecting a corresponding log, and in this process, a fixed query result is returned according to a query condition, and the log details are checked; finally, a user operation log can be checked, wherein the user operation log comprises the record and the display of the user login, authority change, IP, operation type, operation content, operation time and other contents; specifically, the system log viewing is supported, and the system log comprises recording and displaying of contents such as system starting, system closing, user increasing and decreasing, user login, authority change, performance abnormity, function abnormity, hardware abnormity, network abnormity, dangerous operation and the like.
In one or more embodiments, preferably, the data sharing tool 115 specifically includes:
the operation management component is used for collecting and classifying the operation condition of the data sharing and blending platform and sending the operation condition data to a manager according to a preset operation and maintenance monitoring standard;
the storage management component comprises distributed file storage and a distributed memory database;
the retrieval management component is used for providing a retrieval entrance and carrying out full-text retrieval, sub-library statistics and record viewing according to the keywords;
the data interface component comprises an authentication interface, a data collection interface component, a data calculation interface component, a data use interface component and a platform service interface, wherein the authentication interface, the data collection interface component, the data calculation interface component, the data use interface component and the platform service interface all comprise request URLs and request methods;
the open source platform component library comprises a batch acquisition component, a distributed file system, a batch processing engine, a memory type batch processing engine, a main key query retrieval component, a full-text retrieval component, an acquisition tool, a distributed message component, a storage query component, an SQL batch processing engine, a resource scheduling engine, an open source computing framework and a stream processing engine.
FIG. 9 is a flow diagram illustrating the execution of a data check component in one embodiment of the invention. As shown in fig. 9, the data verification content supports data verification of information such as type, length, whether null, precision, range, format, and the like.
FIG. 10 is a data parsing flow diagram of the data verification component in an embodiment of the invention. As shown in fig. 10, the data analysis provides a function of extracting data in a distributed heterogeneous data source, such as relational data, non-relational data, data files, FTP message middleware, and the like, to a temporary intermediate layer, and then performing a data type check function before warehousing, and a high-performance data analysis function for mass data.
FIG. 11 is a data classification flow diagram of the data verification component in one embodiment of the invention. As shown in fig. 11, when data collection occurs, data collected by each business department transmits data entry information to be checked to the data center pre-database system, the pre-database checks whether the entry information matches and is consistent, if yes, a comparison success is returned, otherwise, a comparison error is returned, and accurate information collected by each business department is returned to the caller. When the data of the business department are synchronized, the data check is used for comparing the consistency of the imported data and the data of the business department, if the data of the business department are consistent, the data do not need to be imported repeatedly, and if the data of the business department are inconsistent, the inconsistent information is covered by the latest data. And the data verification comparison tool is used for displaying the comparison result to the user in a report form convenient for reading and analyzing.
FIG. 12 is a flow diagram illustrating the execution of a data management component in one embodiment of the invention. For example, as shown in fig. 12, the data management module mainly organizes all information of data, metadata, and data attributes in the exchange platform into a visual interface to be displayed to the administrator, and the administrator can complete the query/update management of resources, directories, and metadata on line. When data exchange is carried out, the data task acquires relevant resource configuration according to the resource ID of the data to carry out detection, verification and other operations on the data.
FIG. 13 is a flow diagram illustrating the execution of data processing components in one embodiment of the invention. As shown in fig. 13, the data processing component supports obtaining business data from the data sharing platform. The data source is a service system and a file system, the extraction mode supports full extraction or incremental extraction according to specific services, and the extraction time and frequency are formulated according to the specific services to support the configuration of the parameters.
FIG. 14 is a flow diagram illustrating the execution of a data analysis component in one embodiment of the invention.
As shown in fig. 14, in one or more embodiments, preferably, the data processing tool 114 specifically includes:
a data exchange component for managing data exchange nodes and data exchange rules;
the data checking component is used for checking the type, the length, the null, the precision, the range and the format of the data;
the data management component is used for organizing all data, metadata and data of the data sharing and blending platform into a visual interface to be displayed to an administrator, and supporting the administrator to inquire and update resources, catalogues and metadata on line;
the data processing component is used for acquiring the service data from the data sharing platform, and the extraction mode is full extraction or incremental extraction according to the service;
and the data analysis component is used for analyzing each data task and carrying out data mining analysis according to an integrated algorithm, wherein the integrated algorithm comprises a text semantic analysis algorithm and an association rule operation algorithm.
Fig. 15 is a multi-level technical architecture diagram of a data sharing fusion platform based on a software defined open toolset according to an embodiment of the present invention. As shown in fig. 15, the data sharing and merging platform based on the software defined open toolset includes a plurality of technology hierarchies, which respectively include a data resource layer, a data collection layer, a data merging layer, and a data service layer. The method comprises the steps of acquiring data on a data resource layer, performing data interaction service on a data acquisition layer, preprocessing, storing and analyzing technical data on a data fusion layer, and forming an application function and coordinating fusion analysis on the data service layer.
According to a second aspect of the embodiments of the present invention, a data sharing and fusing method based on a software defined open toolset is provided. FIG. 16 is a flowchart of a data sharing and merging method based on the software defined open toolset according to the present invention. As shown in fig. 16, the method includes:
s1601, the data sharing and co-fusion platform acquires all multi-source data information;
s1602, performing data sharing and collection on the multi-source data information to generate shared open data, wherein the shared open data comprises non-encrypted data and encrypted data;
s1603, directly accessing the non-encrypted data to the data sharing and co-fusing platform, and storing the encrypted data in a target block chain in a preset format by using a block chain;
s1604, performing data processing on the multi-source data information in a preset format, a preset classification and a preset interface, and storing the multi-source data information in a data warehouse;
s1605, performing data collection, data fusion and authority management on the data sharing and co-fusion platform, and storing the multi-source data information into metadata, interface data and multi-source sharing data in a preset form;
s1606, creating a structured mapping table for the multi-source data information through a data sharing and fusing tool and the data sharing and fusing platform;
s1607, establishing virtual structure data in the data sharing and co-fusing platform;
s1608, accessing the structured mapping table by executing SQL statements, calling and updating the virtual structure data.
In the embodiment of the invention, the data sharing fusion platform solves the problem of low efficiency of multi-source data access analysis in the data sharing process, realizes that one SQL is connected with a plurality of data sources for unified access, and realizes that metadata models are consistent, authority models are consistent, access models are consistent, and management models are consistent for unified management of multi-source easily-purchased data source information. Based on an open platform and a tool set of data sharing, a platform operation unit accesses an original data interface into a data sharing platform in a data access layer. And providing a service system data interface reconstruction service to form a data interface corresponding to the service. The data sharing fusion tool solves the problem of high cost of a programmable data analysis processing environment, the platform provides a series of data processing component tools and a large number of completed algorithm components, the environment corresponding to the data sharing open platform is docked in a service mode, a data user can adopt a componentized development method, the data processing and analysis efficiency is improved, and the problem of high cost of construction of a service data analysis processing environment is solved.
According to a third aspect of embodiments of the present invention, there is provided a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method according to any one of the first aspect of embodiments of the present invention.
According to a fourth aspect of the embodiments of the present invention, there is provided an electronic apparatus. FIG. 17 is a block diagram of an electronic device in one embodiment of the invention. The electronic device shown in fig. 17 is a general data sharing fusion apparatus, which includes a general computer hardware structure including at least a processor 1701 and a memory 1702. The processor 1701 and the memory 1702 are connected by a bus 1703. The memory 1702 is adapted to store instructions or programs that are executable by the processor 1701. The processor 1701 may be a stand-alone microprocessor or a collection of one or more microprocessors. Thus, the processor 1701 implements processing of data and control of other devices by executing instructions stored by the memory 1702 to thereby execute the method flows of embodiments of the present invention as described above. The bus 1703 connects the above components together, as well as to the display controller 1704 and display device and input/output (I/O) device 1705. Input/output (I/O) device 1705 may be a mouse, keyboard, modem, network interface, touch input device, motion sensing input device, printer, and other devices known in the art. Typically, the input/output devices 1705 are connected to the system through an input/output (I/O) controller 1706.
The technical scheme provided by the embodiment of the invention can have the following beneficial effects:
1) when the embodiment of the invention opens the data access service for the external visitor, the embodiment of the invention is in consideration of safety, and supports the establishment of a virtual structure table to shield the real metadata information and the example information of the physical data source of the remote access request.
2) The embodiment of the invention provides a unified metadata and SQL access interface, before accessing a multi-source database, a structured mapping table is created for the multi-source database through the SQL syntax provided by a data sharing and fusion tool and a platform, and the data query efficiency is improved and the application development difficulty is reduced by executing the SQL sentence to access the just created structured mapping table.
3) According to the embodiment of the invention, the external data is subjected to collection fusion analysis through the data sharing fusion platform, so that the multi-source data processing flow is closed in one platform, the data flow is more efficient, the interactive data query is realized based on the platform, and the data use efficiency is improved.
4) According to the embodiment of the invention, fusion analysis of multi-source data in one platform is realized through a data sharing and fusion platform, the same data is processed and shared by multiple data, data is prevented from being moved, the problems of data dispersion and repeated storage are solved, the data query speed is improved, unified resource scheduling is realized through platform service management, and the utilization rate of data resources is improved.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A data sharing fusion platform based on a software defined open tool set is characterized by comprising a data sharing fusion platform and a data sharing fusion tool;
the data sharing and co-fusion platform comprises a sharing open data collection interface, a data calculation interface, a component service interface, a data use interface, a data warehouse, an operating environment, an extensible container engine, a metadata management module, a resource catalog management module, a data quality management module and a platform service management and operation monitoring module;
the data sharing fusion tool comprises a data processing tool and a data sharing tool;
the shared open data collection interface is used for receiving multi-source information outside the data sharing fusion platform;
the data calculation interface is used for feeding back data obtained by calculation to the outside of the data sharing fusion platform;
the component service interface is used for carrying out data interaction between the data sharing and fusion platform and the data sharing and fusion tool;
and the data use interface carries out extended application on the data after data processing outside the data sharing fusion platform.
2. The data sharing and fusion platform based on the software defined open toolset according to claim 1, wherein the metadata management module specifically includes:
the metadata storage submodule is used for managing all interface metadata in a data source interface, a message middleware and front-end display and storing the interface metadata in a classified mode according to technical metadata and service metadata;
the metadata query submodule is used for querying and retrieving metadata basic information in the metadata storage submodule and returning query content information, wherein the query content information comprises a database table, a dimension table, indexes, processes and object information participating in input and output;
the metadata checking submodule is used for checking the integrity and the rationality of metadata information, wherein the checking content comprises middleware sql analysis success rate, table level relation integrity rate and field relation integrity rate;
and the metadata checking submodule is used for checking the validity of the parameters by using the JSR303 according to the resource directory management module.
3. The data sharing fusion platform based on the software defined open toolset according to claim 1, wherein the resource directory management module specifically includes:
the information resource catalog is used for inputting a department catalog, a basic catalog and a main body catalog and maintaining and managing the department catalog, the basic catalog and the main body catalog, wherein the main body catalog comprises a population, a legal person and an electronic certificate;
the government affair information resources comprise front segment codes, message middleware and rear segment codes, wherein the front segment codes are used for representing the classification of the government affair information resources, the rear segment codes are used for representing the sequence of the government affair information resources, and the message middleware is used for storing message contents;
the directory query submodule is used for querying and checking the corresponding information resources according to the directory of the government affair information resources, wherein the query of the government affair information resources needs to obtain query resource authorization;
and the directory authority submodule is used for the management personnel to authorize for different user levels, wherein the authorization content comprises viewing resource authorization, inquiring resource authorization and modifying resource authorization.
4. The data sharing fusion platform based on the software defined open toolset according to claim 1, wherein the data quality management module specifically includes:
the data quality management submodule is used for performing quality rule management, quality rule execution, data quality monitoring, quality problem management and quality evaluation report generation;
the data evaluation management submodule is used for carrying out data use evaluation, data relation evaluation, timeliness evaluation, redundant data evaluation and importance evaluation;
and the data inspection management submodule is used for performing inspection rule management, inspection task scheduling and inspection result analysis.
5. The data sharing and fusion platform based on the software defined open toolset according to claim 1, wherein the platform service management and operation monitoring module specifically includes:
the platform service management submodule is used for registering services, discovering services, checking started services and checking service states;
the cluster management submodule is used for initiating a problem troubleshooting notice to a technical responsible person when the interface is not normally used in the processes of calling and cluster monitoring;
the collection monitoring submodule is used for providing monitoring information of the exchange task in a graphical interface mode, and the monitoring information comprises task scheduling and real-time flow;
the monitoring alarm submodule is used for carrying out state diagnosis and display on abnormal, alarming and fault conditions;
the log management submodule is used for inquiring the log content according to the operation result, the operation type, the operation time and the operation user;
and the performance monitoring submodule is used for measuring performance data at intervals of 5 minutes and inquiring historical data and real-time data according to the performance data.
6. The data sharing fusion platform based on the software defined open toolset according to claim 1, wherein the data sharing tool specifically includes:
the operation management component is used for collecting and classifying the operation condition of the data sharing and blending platform and sending the operation condition data to a manager according to a preset operation and maintenance monitoring standard;
the storage management component comprises distributed file storage and a distributed memory database;
the retrieval management component is used for providing a retrieval entrance and carrying out full-text retrieval, sub-library statistics and record viewing according to the keywords;
the data interface component comprises an authentication interface, a data collection interface component, a data calculation interface component, a data use interface component and a platform service interface, wherein the authentication interface, the data collection interface component, the data calculation interface component, the data use interface component and the platform service interface all comprise request URLs and request methods;
the open source platform component library comprises a batch acquisition component, a distributed file system, a batch processing engine, a memory type batch processing engine, a main key query retrieval component, a full-text retrieval component, an acquisition tool, a distributed message component, a storage query component, an SQL batch processing engine, a resource scheduling engine, an open source computing framework and a stream processing engine.
7. The data sharing fusion platform based on the software defined open toolset according to claim 1, wherein the data processing tool specifically includes:
a data exchange component for managing data exchange nodes and data exchange rules;
the data checking component is used for checking the type, the length, the null, the precision, the range and the format of the data;
the data management component is used for organizing all data, metadata and data of the data sharing and blending platform into a visual interface to be displayed to an administrator, and supporting the administrator to inquire and update resources, catalogues and metadata on line;
the data processing component is used for acquiring the service data from the data sharing platform, and the extraction mode is full extraction or incremental extraction according to the service;
and the data analysis component is used for analyzing each data task and carrying out data mining analysis according to an integrated algorithm, wherein the integrated algorithm comprises a text semantic analysis algorithm and an association rule operation algorithm.
8. A data sharing and fusing method based on a software defined open toolset is characterized by comprising the following steps:
the data sharing and blending platform acquires all multi-source data information;
carrying out data sharing and collection on the multi-source data information to generate shared open data, wherein the shared open data comprises non-encrypted data and encrypted data;
directly accessing the non-encrypted data to the data sharing and co-fusing platform, and storing the encrypted data in a target block chain in a preset format by using a block chain;
performing data processing of a preset format, a preset classification and a preset interface on the multi-source data information, and storing the multi-source data information in a data warehouse;
performing data collection, data fusion and authority management on the data sharing and co-fusion platform, and storing the multi-source data information into metadata, interface data and multi-source sharing data in a preset form;
establishing a structured mapping table for the multi-source data information through a data sharing and fusing tool and the data sharing and fusing platform;
establishing virtual structure data in the data sharing and co-fusion platform;
and accessing the structured mapping table by executing SQL sentences, and calling and searching the virtual structure data.
9. A computer-readable storage medium on which computer program instructions are stored, which, when executed by a processor, implement the method of any one of claims 1-7.
10. An electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the steps of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110261104.8A CN113094385B (en) | 2021-03-10 | 2021-03-10 | Data sharing fusion platform and method based on software defined open tool set |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110261104.8A CN113094385B (en) | 2021-03-10 | 2021-03-10 | Data sharing fusion platform and method based on software defined open tool set |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113094385A true CN113094385A (en) | 2021-07-09 |
| CN113094385B CN113094385B (en) | 2024-04-30 |
Family
ID=76666826
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110261104.8A Active CN113094385B (en) | 2021-03-10 | 2021-03-10 | Data sharing fusion platform and method based on software defined open tool set |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113094385B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656608A (en) * | 2021-08-18 | 2021-11-16 | 中国科学院软件研究所 | Software-defined satellite-oriented big data system and data automatic processing method |
| CN113742357A (en) * | 2021-08-25 | 2021-12-03 | 国核电力规划设计研究院有限公司 | Method and system for automatically collecting and associating cross-platform design data |
| CN114780261A (en) * | 2022-04-28 | 2022-07-22 | 四川虹魔方网络科技有限公司 | Basic data micro-service sharing system and method in intelligent equipment operation management |
| CN116720300A (en) * | 2023-08-02 | 2023-09-08 | 福州水字节科技有限公司 | Drainage pipe network model system |
| CN117033487A (en) * | 2023-08-11 | 2023-11-10 | 北京长河数智科技有限责任公司 | System and method for flexibly arranging interfaces based on data sharing |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722667A (en) * | 2012-03-07 | 2012-10-10 | 甘肃省电力公司信息通信公司 | Database security protection system and method based on virtual databases and virtual patches |
| CN103246724A (en) * | 2013-05-08 | 2013-08-14 | 昆明能讯科技有限责任公司 | Open enterprise information portal supporting component-based development and management |
| CN104123133A (en) * | 2014-07-01 | 2014-10-29 | 曾安里 | Dragon VVR-VBF modularization development framework platform tool system |
| CN105554070A (en) * | 2015-12-09 | 2016-05-04 | 北京中科云集科技有限公司 | Method based on police affair big data center service construction |
| CN106777227A (en) * | 2016-12-26 | 2017-05-31 | 河南信安通信技术股份有限公司 | Multidimensional data convergence analysis system and method based on cloud platform |
| CN106855962A (en) * | 2015-12-09 | 2017-06-16 | 星际空间（天津）科技发展有限公司 | A kind of method for building government affairs big data platform |
| CN107491510A (en) * | 2017-08-03 | 2017-12-19 | 国网江苏省电力公司信息通信分公司 | One kind mixing heterogeneous data source unified query system and distributed enquiring method |
| CN109766374A (en) * | 2018-12-26 | 2019-05-17 | 科大国创软件股份有限公司 | A kind of credit joint supervising platform |
| CN111274294A (en) * | 2020-01-09 | 2020-06-12 | 中国科学院计算机网络信息中心 | Universal distributed heterogeneous data integrated logic convergence organization, release and service method and system |
| CN111400749A (en) * | 2020-03-06 | 2020-07-10 | 江苏荣泽信息科技股份有限公司 | Government affair financial data sharing platform based on block chain and implementation method thereof |
| CN111625510A (en) * | 2020-05-25 | 2020-09-04 | 广东电网有限责任公司 | Multi-source data sharing system and method based on cloud mapping |
| CN111930355A (en) * | 2020-08-12 | 2020-11-13 | 创意信息技术股份有限公司 | Web back-end development framework and construction method thereof |
| CN112116488A (en) * | 2020-04-28 | 2020-12-22 | 刘革瑞 | Water conservancy big data comprehensive maintenance system |
| CN112307396A (en) * | 2020-10-21 | 2021-02-02 | 五凌电力有限公司 | Platform architecture based on multi-engine data modeling calculation analysis and processing method thereof |
| CN112306992A (en) * | 2020-11-04 | 2021-02-02 | 内蒙古证联信息技术有限责任公司 | Big data platform based on internet |
| CN112379653A (en) * | 2020-12-01 | 2021-02-19 | 国能信控互联技术有限公司 | Smart power plant management and control system based on micro-service architecture |
| CN112396404A (en) * | 2020-11-27 | 2021-02-23 | 广州光点信息科技有限公司 | Data center system |
-
2021
- 2021-03-10 CN CN202110261104.8A patent/CN113094385B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722667A (en) * | 2012-03-07 | 2012-10-10 | 甘肃省电力公司信息通信公司 | Database security protection system and method based on virtual databases and virtual patches |
| CN103246724A (en) * | 2013-05-08 | 2013-08-14 | 昆明能讯科技有限责任公司 | Open enterprise information portal supporting component-based development and management |
| CN104123133A (en) * | 2014-07-01 | 2014-10-29 | 曾安里 | Dragon VVR-VBF modularization development framework platform tool system |
| CN105554070A (en) * | 2015-12-09 | 2016-05-04 | 北京中科云集科技有限公司 | Method based on police affair big data center service construction |
| CN106855962A (en) * | 2015-12-09 | 2017-06-16 | 星际空间（天津）科技发展有限公司 | A kind of method for building government affairs big data platform |
| CN106777227A (en) * | 2016-12-26 | 2017-05-31 | 河南信安通信技术股份有限公司 | Multidimensional data convergence analysis system and method based on cloud platform |
| CN107491510A (en) * | 2017-08-03 | 2017-12-19 | 国网江苏省电力公司信息通信分公司 | One kind mixing heterogeneous data source unified query system and distributed enquiring method |
| CN109766374A (en) * | 2018-12-26 | 2019-05-17 | 科大国创软件股份有限公司 | A kind of credit joint supervising platform |
| CN111274294A (en) * | 2020-01-09 | 2020-06-12 | 中国科学院计算机网络信息中心 | Universal distributed heterogeneous data integrated logic convergence organization, release and service method and system |
| CN111400749A (en) * | 2020-03-06 | 2020-07-10 | 江苏荣泽信息科技股份有限公司 | Government affair financial data sharing platform based on block chain and implementation method thereof |
| CN112116488A (en) * | 2020-04-28 | 2020-12-22 | 刘革瑞 | Water conservancy big data comprehensive maintenance system |
| CN111625510A (en) * | 2020-05-25 | 2020-09-04 | 广东电网有限责任公司 | Multi-source data sharing system and method based on cloud mapping |
| CN111930355A (en) * | 2020-08-12 | 2020-11-13 | 创意信息技术股份有限公司 | Web back-end development framework and construction method thereof |
| CN112307396A (en) * | 2020-10-21 | 2021-02-02 | 五凌电力有限公司 | Platform architecture based on multi-engine data modeling calculation analysis and processing method thereof |
| CN112306992A (en) * | 2020-11-04 | 2021-02-02 | 内蒙古证联信息技术有限责任公司 | Big data platform based on internet |
| CN112396404A (en) * | 2020-11-27 | 2021-02-23 | 广州光点信息科技有限公司 | Data center system |
| CN112379653A (en) * | 2020-12-01 | 2021-02-19 | 国能信控互联技术有限公司 | Smart power plant management and control system based on micro-service architecture |
Non-Patent Citations (2)
| Title |
|---|
| MICHAEL FRANKLIN等: "From databases to dataspaces: a new abstraction for information management", ACM SIGMOD RECORD, vol. 34, no. 4, 1 December 2005 (2005-12-01), pages 27 - 33, XP002406424, DOI: 10.1145/1107499.1107502 * |
| 殷自力等: "一种面向配电网调度业务管控的混合型大数据平台研究", 供用电, vol. 37, no. 11, 5 November 2020 (2020-11-05), pages 15 - 20 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656608A (en) * | 2021-08-18 | 2021-11-16 | 中国科学院软件研究所 | Software-defined satellite-oriented big data system and data automatic processing method |
| CN113656608B (en) * | 2021-08-18 | 2023-10-24 | 中国科学院软件研究所 | Big data system and automatic data processing method for software defined satellite |
| CN113742357A (en) * | 2021-08-25 | 2021-12-03 | 国核电力规划设计研究院有限公司 | Method and system for automatically collecting and associating cross-platform design data |
| CN114780261A (en) * | 2022-04-28 | 2022-07-22 | 四川虹魔方网络科技有限公司 | Basic data micro-service sharing system and method in intelligent equipment operation management |
| CN114780261B (en) * | 2022-04-28 | 2022-11-29 | 四川虹魔方网络科技有限公司 | Basic data micro-service sharing system and method in intelligent equipment operation management |
| CN116720300A (en) * | 2023-08-02 | 2023-09-08 | 福州水字节科技有限公司 | Drainage pipe network model system |
| CN116720300B (en) * | 2023-08-02 | 2023-10-13 | 福州水字节科技有限公司 | Drainage pipe network model system |
| CN117033487A (en) * | 2023-08-11 | 2023-11-10 | 北京长河数智科技有限责任公司 | System and method for flexibly arranging interfaces based on data sharing |
| CN117033487B (en) * | 2023-08-11 | 2024-05-07 | 北京长河数智科技有限责任公司 | System and method for flexibly arranging interfaces based on data sharing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113094385B (en) | 2024-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113094385B (en) | Data sharing fusion platform and method based on software defined open tool set | |
| US11755628B2 (en) | Data relationships storage platform | |
| CN112396404A (en) | Data center system | |
| CN112650762B (en) | Data quality monitoring method and device, electronic equipment and storage medium | |
| CN110019176B (en) | Data management control system for improving success rate of data management service | |
| US9367586B2 (en) | Data validation and service | |
| US20130305224A1 (en) | Rules Engine for Architectural Governance | |
| CN110458678B (en) | Financial data verification method and system based on hadoop verification | |
| CN114925045B (en) | PaaS platform for big data integration and management | |
| US9123006B2 (en) | Techniques for parallel business intelligence evaluation and management | |
| CN111866121B (en) | Safety monitoring and management cloud platform for large crane equipment | |
| CN112527774A (en) | Data center building method and system and storage medium | |
| KR100903726B1 (en) | System for Evaluating Data Quality Management Maturity | |
| CN111538720A (en) | Method and system for cleaning basic data in power industry | |
| US20170039235A1 (en) | Air quality metrology system | |
| Fu et al. | Design and implementation of clinical LIS360 laboratory management system based on AI technology | |
| KR100796906B1 (en) | Method for Quality Control of DataBase | |
| CN116910023A (en) | Data management system | |
| CN115168474B (en) | Internet of things central station system building method based on big data model | |
| CN113706101B (en) | Intelligent system architecture and method for power grid project management | |
| KR100796905B1 (en) | System for Quality Control of DataBase | |
| KR100792322B1 (en) | Framework for Quality Control of DataBase | |
| Ribeiro et al. | A scalable data integration architecture for smart cities: implementation and evaluation | |
| CN112784129A (en) | Pump station equipment operation and maintenance data supervision platform | |
| CN111352985A (en) | Data service platform, method and storage medium based on computer system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |