CN111858731A - Data relay station system for data integration processing and service sharing based on big data virtualization technology - Google Patents
Data relay station system for data integration processing and service sharing based on big data virtualization technology Download PDFInfo
- Publication number
- CN111858731A CN111858731A CN202010671488.6A CN202010671488A CN111858731A CN 111858731 A CN111858731 A CN 111858731A CN 202010671488 A CN202010671488 A CN 202010671488A CN 111858731 A CN111858731 A CN 111858731A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- sharing
- module
- sharer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000010354 integration Effects 0.000 title claims abstract description 23
- 238000005516 engineering process Methods 0.000 title claims abstract description 17
- 238000000034 method Methods 0.000 claims abstract description 19
- 238000013499 data model Methods 0.000 claims description 15
- 238000007405 data analysis Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 9
- XEEYBQQBJWHFJM-UHFFFAOYSA-N Iron Chemical compound [Fe] XEEYBQQBJWHFJM-UHFFFAOYSA-N 0.000 claims description 6
- 230000000694 effects Effects 0.000 claims description 4
- 230000006870 function Effects 0.000 claims description 4
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 230000006399 behavior Effects 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 229910052742 iron Inorganic materials 0.000 claims description 3
- 238000010801 machine learning Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 230000001502 supplementing effect Effects 0.000 claims description 3
- 210000001503 joint Anatomy 0.000 abstract description 2
- 238000007726 management method Methods 0.000 description 6
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
- G06F16/81—Indexing, e.g. XML tags; Data structures therefor; Storage structures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to the technical field of data virtualization, and discloses a data center station system for data integration processing and service sharing based on a big data virtualization technology. This platform system in data based on big data virtualization technique is used for data integration processing and service sharing carries out unified management to big data for data can obtain fine utilization, improves the treatment effeciency of platform in the data, and in time, accurately, stably integrated processing to data has overcome supply and demand both sides information asymmetry, and the quick effective butt joint between the quick effectual realization service demander of while and the service sharer adopts mixed cloud platform to promote the treatment effeciency of platform.
Description
Technical Field
The invention relates to the technical field of data virtualization, in particular to a data center platform system for data integration processing and service sharing based on a big data virtualization technology.
Background
From the perspective of the user application, we consider that the data virtualization system architecture should contain 3 layers: the system comprises an application layer, a data virtualization layer and a source data layer, wherein the data virtualization layer comprises a data virtualization plane and a management plane which are combined with each other to execute all-round query, processing, integration and management functions.
The existing data center platform system facilitates scene, service, engine, center platform, unified technology and intelligent data management, but the existing data center platform system is not efficient enough in service, complicated in system structure and disordered in management, so that the service is difficult to be efficiently promoted, and the data center platform system based on the big data virtualization technology for data integrated processing and service sharing is provided to solve the problems.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects of the prior art, the invention provides the data center system based on the big data virtualization technology for data integration processing and service sharing, has the advantage of more efficient service, and solves the problems that the service of the existing data center system is not efficient enough, the system structure is complicated, the management is disordered, and the service is difficult to advance efficiently.
(II) technical scheme
In order to achieve the purpose of more efficient service, the invention provides the following technical scheme: the data center platform system comprises a data center platform which is built on a hybrid cloud platform based on a big data virtualization technology, third-party data are loaded to the preset data center platform through an ETL (extract transform load) process according to an information tag of the third-party data, the data center platform processes the third-party data, the data center platform provides service sharing services, and the service sharing services comprise customer matching, customer notification, customer comment and the like.
Preferably, the system adopts the hybrid cloud platform as a carrier of the data center platform, resources can be managed more flexibly and efficiently through the private cloud in the hybrid cloud platform, computing resources and storage resources of the public cloud in the hybrid cloud platform are incomparable to the private cloud system, the pressure and the demand on the internal private cloud can be reduced by transferring the non-confidential function to the public cloud, and the public cloud and the private cloud are mixed and matched to obtain the best effect.
Preferably, the data processing of iron in the data comprises: the system comprises a driving module, a data processing module, a data early warning module, a data analysis module, a data model module and a service interface module, wherein the data analysis module, the data early warning module and the data processing module are driven by the driving module.
Preferably, the service interface module: the data message is used for transmitting data messages, and the data messages comprise XML data and self-defined text data;
a data model module: the system comprises a database data model, a data configuration model and a data configuration model, wherein the database data model is used for mapping data attributes of received data messages with the database data model according to different integration modes, and the integration modes comprise one or more of WebService, intermediate library extraction and FTP file sharing;
a data analysis module: the data configuration module is used for receiving the data message transmitted by the service interface module and analyzing the data message into a data JAVA object according to the data configuration model;
the data early warning module: the early warning system is used for retrieving the data JAVA object according to early warning configuration and generating early warning reminding for the abnormal data JAVA object;
a data processing module: the method is used for removing the junk data JAVA objects and supplementing missing data JAVA objects.
Preferably, the client matches and obtains detailed information of the user according to the user registration information through the system, the user includes a service sharer and a service demander, for the service sharer, the service sharer can issue a service sharing request through a user client of the system, further, the service sharer can configure sharing preferences of the service corresponding to the service sharing request, that is, a sharing rule set, the system filters sharable services according to a preset supervision rule set and the sharing rule set provided by the service sharer, meanwhile, for the service demander, the service demander can issue a service demand request through a user client of the system, further, at S306a, the service demander can configure preferences of the service corresponding to the service demand request, that is, a service cluster, to form a second weight data set corresponding to the service demander, based on a service demand data set, a service sharing rule set and a weight configuration data set provided by a service demander, the system matches and matches the service supply and demand relationship between the service demander and the service sharer according to a preset matching algorithm, wherein the matching algorithm comprises the following steps: machine learning algorithms, deep learning algorithms, and the like.
Preferably, the notifying the client: respectively notifying a service sharer and a service demander after the completion of customer matching, and commenting by the customers: the behavior of both parties or this service sharing transaction can be evaluated so that the system can evaluate the credit of the service demander and the service sharer.
(III) advantageous effects
Compared with the prior art, the invention provides the data center platform system for data integration processing and service sharing based on the big data virtualization technology, and the data center platform system has the following beneficial effects:
this platform system in data based on big data virtualization technique is used for data integration processing and service sharing carries out unified management to big data for data can obtain fine utilization, improves the treatment effeciency of platform in the data, and in time, accurately, stably integrated processing to data has overcome supply and demand both sides information asymmetry, and the quick effective butt joint between the quick effectual realization service demander of while and the service sharer adopts mixed cloud platform to promote the treatment effeciency of platform.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is a flow chart of data processing in the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-2, a data center platform system for data integration processing and service sharing based on a big data virtualization technology includes a data center platform built on a hybrid cloud platform based on the big data virtualization technology, third-party data is loaded to a preset data center platform through an ETL process according to an information tag of the third-party data, the data center platform processes the third-party data, the data center platform provides a service sharing service, and the service sharing includes services such as customer matching, customer notification, customer comment and the like.
Preferably, the system adopts the hybrid cloud platform as a carrier of the data center platform, resources can be managed more flexibly and efficiently through the private cloud in the hybrid cloud platform, computing resources and storage resources of the public cloud in the hybrid cloud platform are incomparable to the private cloud system, the pressure and the demand on the internal private cloud can be reduced by transferring the non-confidential function to the public cloud, and the public cloud and the private cloud are mixed and matched to obtain the best effect.
Preferably, the data processing of iron in the data comprises: the system comprises a driving module, a data processing module, a data early warning module, a data analysis module, a data model module and a service interface module, wherein the data analysis module, the data early warning module and the data processing module are driven by the driving module.
Preferably, the service interface module: the data message is used for transmitting data messages, and the data messages comprise XML data and self-defined text data;
a data model module: the system comprises a database data model, a data configuration model and a data configuration model, wherein the database data model is used for mapping data attributes of received data messages with the database data model according to different integration modes, and the integration modes comprise one or more of WebService, intermediate library extraction and FTP file sharing;
a data analysis module: the data configuration module is used for receiving the data message transmitted by the service interface module and analyzing the data message into a data JAVA object according to the data configuration model;
the data early warning module: the early warning system is used for retrieving the data JAVA object according to early warning configuration and generating early warning reminding for the abnormal data JAVA object;
a data processing module: the method is used for removing the junk data JAVA objects and supplementing missing data JAVA objects.
Preferably, the client matches and obtains detailed information of the user according to the user registration information through the system, the user includes a service sharer and a service demander, for the service sharer, the service sharer can issue a service sharing request through a user client of the system, further, the service sharer can configure sharing preferences of the service corresponding to the service sharing request, that is, a sharing rule set, the system filters sharable services according to a preset supervision rule set and the sharing rule set provided by the service sharer, meanwhile, for the service demander, the service demander can issue a service demand request through a user client of the system, further, at S306a, the service demander can configure preferences of the service corresponding to the service demand request, that is, a service cluster, to form a second weight data set corresponding to the service demander, based on a service demand data set, a service sharing rule set and a weight configuration data set provided by a service demander, the system matches and matches the service supply and demand relationship between the service demander and the service sharer according to a preset matching algorithm, wherein the matching algorithm comprises the following steps: machine learning algorithms, deep learning algorithms, and the like.
Preferably, the notifying the client: respectively notifying a service sharer and a service demander after the completion of customer matching, and commenting by the customers: the behavior of both parties or this service sharing transaction can be evaluated so that the system can evaluate the credit of the service demander and the service sharer.
To sum up, the data center platform system for data integration processing and service sharing based on the big data virtualization technology performs unified management on big data, so that the data can be well utilized, the processing efficiency of the data center platform is improved, the data is integrated and processed timely, accurately and stably, the asymmetry of information of a supply party and a demand party is overcome, meanwhile, the service demander and the service sharer are quickly and effectively docked quickly, and the processing efficiency of a platform is improved by adopting a hybrid cloud platform.
It is to be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the use of the phrase "comprising a. -. said" to define an element does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (6)
1. A platform system in data that is used for data integration processing and service sharing based on big data virtualization technique includes the platform in data based on big data virtualization technique building on mixing cloud platform, its characterized in that: and loading the third-party data to a preset data center station through an ETL (extract transform and load) process according to an information tag of the third-party data, wherein the data center station processes the third-party data, the data center station provides service sharing services, and the service sharing comprises services of customer matching, customer notification, customer comment and the like.
2. The data center system for data integration processing and service sharing based on big data virtualization technology of claim 1, wherein: the system adopts the hybrid cloud platform as a carrier of the data center platform, resources can be managed more flexibly and efficiently through the private cloud in the hybrid cloud platform, computing resources and storage resources of the public cloud in the hybrid cloud platform are incomparable to the private cloud system, the pressure and the demand on the internal private cloud can be reduced by transferring the non-confidential function to the public cloud, and the public cloud and the private cloud are mixed and matched to obtain the best effect.
3. The data center system for data integration processing and service sharing based on big data virtualization technology of claim 1, wherein: the data processing of iron in the data comprises the following steps: the system comprises a driving module, a data processing module, a data early warning module, a data analysis module, a data model module and a service interface module, wherein the data analysis module, the data early warning module and the data processing module are driven by the driving module.
4. The data center system for data integration processing and service sharing based on big data virtualization technology of claim 3, wherein:
a service interface module: the data message is used for transmitting data messages, and the data messages comprise XML data and self-defined text data;
a data model module: the system comprises a database data model, a data configuration model and a data configuration model, wherein the database data model is used for mapping data attributes of received data messages with the database data model according to different integration modes, and the integration modes comprise one or more of WebService, intermediate library extraction and FTP file sharing;
a data analysis module: the data configuration module is used for receiving the data message transmitted by the service interface module and analyzing the data message into a data JAVA object according to the data configuration model;
The data early warning module: the early warning system is used for retrieving the data JAVA object according to early warning configuration and generating early warning reminding for the abnormal data JAVA object;
a data processing module: the method is used for removing the junk data JAVA objects and supplementing missing data JAVA objects.
5. The data center system for data integration processing and service sharing based on big data virtualization technology of claim 1, wherein: the client matching obtains the detailed information of the user according to the user registration information through the system, the user comprises a service sharer and a service demander, for the service sharer, the service sharer can issue a service sharing request through a user client of the system, further, the service sharer can configure the sharing preference of the service corresponding to the service sharing request, namely, a sharing rule set, the system filters shareable service according to a preset supervision rule set and the sharing rule set provided by the service sharer, meanwhile, for the service demander, the service demander can issue the service requirement request through the user client of the system, further, at S306a, the service demander can configure the preference of the service corresponding to the service requirement request, namely, a service cluster, to form a second weight data set corresponding to the service demander, based on a service demand data set, a service sharing rule set and a weight configuration data set provided by a service demander, the system matches and matches the service supply and demand relationship between the service demander and the service sharer according to a preset matching algorithm, wherein the matching algorithm comprises the following steps: machine learning algorithms, deep learning algorithms, and the like.
6. The data center system for data integration processing and service sharing based on big data virtualization technology of claim 1, wherein: the notification client: respectively notifying a service sharer and a service demander after the completion of customer matching, and commenting by the customers: the behavior of both parties or this service sharing transaction can be evaluated so that the system can evaluate the credit of the service demander and the service sharer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010671488.6A CN111858731A (en) | 2020-07-13 | 2020-07-13 | Data relay station system for data integration processing and service sharing based on big data virtualization technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010671488.6A CN111858731A (en) | 2020-07-13 | 2020-07-13 | Data relay station system for data integration processing and service sharing based on big data virtualization technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111858731A true CN111858731A (en) | 2020-10-30 |
Family
ID=72984715
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010671488.6A Pending CN111858731A (en) | 2020-07-13 | 2020-07-13 | Data relay station system for data integration processing and service sharing based on big data virtualization technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858731A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112465364A (en) * | 2020-12-03 | 2021-03-09 | 合肥天源迪科信息技术有限公司 | Management system for index library |
| CN112487333A (en) * | 2020-11-30 | 2021-03-12 | 深圳供电局有限公司 | Data center design method and system based on micro-service |
-
2020
- 2020-07-13 CN CN202010671488.6A patent/CN111858731A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112487333A (en) * | 2020-11-30 | 2021-03-12 | 深圳供电局有限公司 | Data center design method and system based on micro-service |
| CN112465364A (en) * | 2020-12-03 | 2021-03-09 | 合肥天源迪科信息技术有限公司 | Management system for index library |
| CN112465364B (en) * | 2020-12-03 | 2024-03-19 | 合肥天源迪科信息技术有限公司 | Management system for index library |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022022477A1 (en) | Management operation and maintenance platform and data processing method | |
| CN111858731A (en) | Data relay station system for data integration processing and service sharing based on big data virtualization technology | |
| EP1918829A1 (en) | Systems and methods for information exchange using object warehousing | |
| CN107071001A (en) | Intelligent transportation Web information sharing service platform framework method | |
| CN112419129A (en) | Method and system for constructing platform system infrastructure in government affair service | |
| CN116382843A (en) | Industrial AI power-calculating PaaS platform based on Kubernetes container technology | |
| Wang et al. | Research on cloud logistics-based one-stop service platform for logistics center | |
| CN108197874B (en) | Logistics management system based on big data of Internet of things | |
| CN117827441A (en) | Cloud edge end cooperative computing power resource intelligent monitoring and scheduling method and system | |
| Arora et al. | Artificial intelligence in disaster management: A survey | |
| CN116069462A (en) | Big data DAG task flow scheduling method, system and storage medium | |
| CN103970592A (en) | Single-thread multi-task processing method and server of transaction middleware | |
| CN116366692A (en) | High-performance intelligent edge terminal system | |
| CN111125046A (en) | Cross-system file sharing system and method based on unstructured platform | |
| Liu et al. | Research on cloud-edge interaction technology of power internet of things based on MQTT protocol | |
| CN103189882A (en) | System and method for collaboration, messaging and information exchange between engineering tools | |
| CN115567578A (en) | Middle platform system with big data architecture for integrated processing and service sharing | |
| CN114596046A (en) | Integrated platform based on unified digital model of business center station and data center station | |
| CN104021484A (en) | Web-based network managing method and system of market channels and customer relations | |
| CN112671867A (en) | Travel integrated cloud service system and method integrating multiple transportation modes | |
| CN112765294A (en) | Meteorological big data processing and scheduling system | |
| CN112712274A (en) | Big data based energy management system and equipment | |
| Gao | Research on intelligent logistics architecture based on edge cloud synergy | |
| CN110766362A (en) | Finished cigarette intelligent transportation scheduling system and method for multi-point warehouse coordination cooperative operation | |
| Yan et al. | A Dispatch Method of Complex Information System Based on Service-Oriented Architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20201030 |