CN105488235A - Cloud platform data management system based on industrial big data and construction method thereof - Google Patents
Cloud platform data management system based on industrial big data and construction method thereof Download PDFInfo
- Publication number
- CN105488235A CN105488235A CN201610079827.5A CN201610079827A CN105488235A CN 105488235 A CN105488235 A CN 105488235A CN 201610079827 A CN201610079827 A CN 201610079827A CN 105488235 A CN105488235 A CN 105488235A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- management system
- cloud platform
- aggregate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a cloud platform data management system based on industrial big data. The system comprises a data acquisition system, an industrial field data module, a Hadoop cluster module, a data aggregation module, a data distribution module and a data persistent storage module. The industrial field data module is located in the data acquisition system and connected with the Hadoop cluster module, the Hadoop cluster module is connected with the data aggregation module and sends processed data to the data aggregation module, the data aggregation module is connected with a data analysis module, and the data aggregation module sends the processed data to the data analysis module to be analyzed. The data distribution module is connected with the data persistent storage module, and the data distribution distributes received data into the data persistent storage module. By means of the system, the size of data blocks can be reduced, data storage efficiency is improved, and safety and reliability of data are guaranteed.
Description
Technical field
The present invention relates to a kind of data management system, particularly relate to a kind of data management system based on the large data of industry and construction method.
Background technology
Along with the continuous maturation of cloud computing technology, the features such as cloud computing is virtual with it, highly reliable, easily extensible, low cost are widely used, increasing enterprise by cloud computing technology just its data center be stored to high in the clouds, thus ensure the reliability of data, and save great amount of cost.Cloud platform architecture mainly comprises three layers from bottom to up, namely namely infrastructure serve (IaaS), platform namely serves (PaaS) and namely software serve (SaaS), the cloud computing correlation technique of current comparative maturity mainly contains openstack, Hadoop, spark etc., and the cloud platform data management system of main flow is also all build on their basis.At present based on the cloud platform data management system constructing plan comparative maturity of this application scenarios of consumer level mass data, and widely applied, compared with consumer level data, industrial data is to the real-time of platform, reliability and safety and reliability have higher requirement, therefore existing cloud platform data management system constructing plan can not well be applied in large this scene of data of industry, and the special cloud platform data management system constructing plan for the large data of industry is also fewer at present, therefore how according to the feature of industrial data itself, invent a kind of cloud platform data management system constructing plan that can adapt to industrial requirement, it is a current more urgent problem.
Current existing cloud platform data management system constructing plan is all based on this application scenarios of consumer level mass data, and industrial data cloud platform management system has higher requirement to the real-time of data, reliability and safety, current existing cloud platform data management system constructing plan can not well be applied in industrial data environment, the present invention, according to the feature of industrial data itself, invents a kind of cloud platform data management system constructing plan that can adapt to industrial requirement.
Summary of the invention
Fundamental purpose of the present invention is the feature according to the large data of industry itself, invent a kind of cloud platform data management system constructing plan that can adapt to industrial requirement, the requirement of the large data of industry to cloud platform data management system reliability, real-time and security can be met.
For solving the problem, the present invention proposes a kind of cloud platform data management system based on the large data of industry, it is characterized in that: described cloud platform data management system comprises data acquisition system (DAS), industrial field data module, Hadoop cluster module, data aggregate module, Data dissemination module and lasting data memory module, wherein, described industrial field data module is arranged in data acquisition system (DAS), the destructuring industrial data collected is transferred to industrial field data module by described data acquisition system (DAS), described industrial field data module is connected with described Hadoop cluster module, described Hadoop cluster module and described data aggregate model calling, data after process are sent to described data aggregate module by described Hadoop cluster module, described data aggregate module is connected with described data analysis module, and the data after process send to described data analysis module to analyze by described data aggregate module, described Data dissemination module is connected with described lasting data memory module, and the data received are assigned in lasting data memory module by described Data dissemination module.
Preferably, described cloud platform data management system also comprises security module, and described security module is connected with described Hadoop cluster module, described data aggregate module, described Data dissemination module, described lasting data memory module respectively.
Preferably, described Hadoop cluster module comprises data structured module, data debug module, data deduplication module, Data Integration module, is connected in series between each module.
Preferably, described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module and described data coupling model calling, described data coupling module is connected with described data compressing module.
The invention also discloses a kind of construction method according to the above-mentioned cloud platform data management system based on the large data of industry, comprise the following steps:
S1. described data acquisition system industrial field data, and the data collected are transferred to industrial field data module;
The destructuring industrial data received is carried out structuring process and generates semi-structured data by S2. described industrial field data module, then gives described Hadoop cluster module by described semi-structured data by Internet Transmission;
S3. the described semi-structured data described in the process of Hadoop cluster module, and send to described data aggregate module;
S4. the described corresponding data of data aggregate resume module, and send to described Data dissemination module;
The data received are assigned in lasting data memory module by S5. described Data dissemination module.
Preferably, described construction method also comprises step S6: described cloud platform data management system also comprises security module, and described security module communicates with each processing module, ensures the security of data.
Preferably, step S3 comprises: described Hadoop cluster module also comprises data structured module, data debug module, data deduplication module, Data Integration module, the described semi-structured data that described data structured modular structure process receives, generating structured data; Described data debug module removes the structural data of mistake, described data deduplication module row except the structural data repeated, the data after the module integrated described data debug module of described Data Integration and the process of described data deduplication module.
Preferably, step S4 comprises, and described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module by similar data gathering together, generates set of metadata of similar data class, and the module integrated described set of metadata of similar data class of described data coupling, optimizes the data of described set of metadata of similar data class; Described data compressing module calling data compression algorithm compresses the data of described set of metadata of similar data class.
Technical scheme of the present invention has following beneficial effect:
(1) distributed thought is utilized, the structuring process of situ industrial data is transferred to enterprises end, data after such process directly can consign to Hadoop cluster and use, and not only reduce the load of cloud platform data management system, and improve the real-time of cloud platform data process.
(2) after data clusters is generated set of metadata of similar data class, the basis of Data Integration is before integrated set of metadata of similar data class again, further increases the validity of data.
(3) in Data dissemination module, usage data compression algorithm is compressed a set of metadata of similar data block, thus reduces the size of data block, improves data storage efficiency.
(4) in data processing and transmitting procedure, use safety module carries out repeatedly authentication and authorization, fully ensures the reliability of data.
Accompanying drawing explanation
Fig. 1 is the schematic diagram of a kind of cloud platform data management system based on the large data of industry of the present invention.
Embodiment
For making the object of the embodiment of the present invention, technical scheme and advantage clearly, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is the present invention's part embodiment, instead of whole embodiments.
As the schematic diagram that Fig. 1 is a kind of cloud platform data management system based on the large data of industry of the present invention.Wherein, described cloud platform data management system comprises data acquisition system (DAS), industrial field data module, Hadoop cluster module, data aggregate module, Data dissemination module and lasting data memory module, wherein, described industrial field data module is arranged in data acquisition system (DAS), the destructuring industrial data collected is transferred to industrial field data module by described data acquisition system (DAS), described industrial field data module is connected with described Hadoop cluster module, described Hadoop cluster module and described data aggregate model calling, data after process are sent to described data aggregate module by described Hadoop cluster module, described data aggregate module is connected with described data analysis module, and the data after process send to described data analysis module to analyze by described data aggregate module, described Data dissemination module is connected with described lasting data memory module, and the data received are assigned in lasting data memory module by described Data dissemination module.
Described cloud platform data management system also comprises security module, and described security module is connected with described Hadoop cluster module, described data aggregate module, described Data dissemination module, described lasting data memory module respectively.
Described Hadoop cluster module comprises data structured module, data debug module, data deduplication module, Data Integration module, is connected in series between each module.
Described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module and described data coupling model calling, described data coupling module is connected with described data compressing module.
The data received are assigned in lasting data memory module by described Data dissemination module, ensure the reliability of data.Data dissemination inside modules safeguards a list, current each storage subsystem information effective of this list records, when selecting storage subsystem, Data dissemination module uses in hash algorithm from then on list and selects a storage subsystem, the set of metadata of similar data class obtained is stored in this subsystem in data aggregate module.In addition, when selecting storage subsystem, the availability of subsystems, active volume, network condition etc. to also be considered.
Described security module is connected with described Hadoop cluster module, described data aggregate module, described Data dissemination module, described lasting data memory module respectively, communicate with each processing module, ensure the security of data, especially the security of data in data processing and transmitting procedure is ensured, all to communicate with security module in each process of data processing and transmission, after only having the authentication and authorization by security module, just can carry out next step process, thus ensure the security of data.
The invention also discloses a kind of construction method according to the above-mentioned cloud platform data management system based on the large data of industry, comprise the following steps:
S1. described data acquisition system industrial field data, and the data collected are transferred to industrial field data module;
The destructuring industrial data received is carried out structuring process and generates semi-structured data by S2. described industrial field data module, then gives described Hadoop cluster module by described semi-structured data by Internet Transmission;
S3. the described semi-structured data described in the process of Hadoop cluster module, and send to described data aggregate module;
S4. the described corresponding data of data aggregate resume module, and send to described Data dissemination module;
The data received are assigned in lasting data memory module by S5. described Data dissemination module.
Described construction method also comprises step S6: described cloud platform data management system also comprises security module, and described security module communicates with each processing module, ensures the security of data.
Described Hadoop cluster module also comprises data structured module, data debug module, data deduplication module, Data Integration module, the described semi-structured data that described data structured modular structure process receives, generating structured data; Described data debug module removes the structural data of mistake, described data deduplication module row except the structural data repeated, the data after the module integrated described data debug module of described Data Integration and the process of described data deduplication module.
Described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module by similar data gathering together, generates set of metadata of similar data class, and the module integrated described set of metadata of similar data class of described data coupling, optimizes the data of described set of metadata of similar data class, ensure the validity of data; Described data compressing module calling data compression algorithm compresses the data of described set of metadata of similar data class, reduces size of data, finally data is consigned to Data dissemination module.
Finally, by carrying out structuring process at enterprises end to data, data directly can be consigned to Hadoop cluster, ensure that the real-time of system, store by data aggregate, Data dissemination, lasting data validity, the reliability that these three modules ensure that data.
Based on the embodiment in the present invention, those of ordinary skill in the art, not making other embodiments all obtained under creative work prerequisite, belong to the scope of protection of the invention.Although the present invention illustrates with regard to preferred implementation and describes, only it will be understood by those of skill in the art that otherwise exceed claim limited range of the present invention, variations and modifications can be carried out to the present invention.
Claims (8)
1. the cloud platform data management system based on the large data of industry, it is characterized in that: described cloud platform data management system comprises data acquisition system (DAS), industrial field data module, Hadoop cluster module, data aggregate module, Data dissemination module and lasting data memory module, wherein, described industrial field data module is arranged in data acquisition system (DAS), the destructuring industrial data collected is transferred to industrial field data module by described data acquisition system (DAS), described industrial field data module is connected with described Hadoop cluster module, described Hadoop cluster module and described data aggregate model calling, data after process are sent to described data aggregate module by described Hadoop cluster module, described data aggregate module is connected with described data analysis module, and the data after process send to described data analysis module to analyze by described data aggregate module, described Data dissemination module is connected with described lasting data memory module, and the data received are assigned in lasting data memory module by described Data dissemination module.
2. a kind of cloud platform data management system based on the large data of industry according to claim 1, it is characterized in that: described cloud platform data management system also comprises security module, described security module is connected with described Hadoop cluster module, described data aggregate module, described Data dissemination module, described lasting data memory module respectively.
3. a kind of cloud platform data management system based on the large data of industry according to claim 1 and 2, it is characterized in that: described Hadoop cluster module comprises data structured module, data debug module, data deduplication module, Data Integration module, is connected in series between each module.
4. a kind of cloud platform data management system based on the large data of industry according to claim 1 and 2, is characterized in that: described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module and described data coupling model calling, described data coupling module is connected with described data compressing module.
5., according to a construction method for a kind of cloud platform data management system based on the large data of industry of the claims 1-4, comprise the following steps:
S1. described data acquisition system industrial field data, and the data collected are transferred to industrial field data module;
The destructuring industrial data received is carried out structuring process and generates semi-structured data by S2. described industrial field data module, then gives described Hadoop cluster module by described semi-structured data by Internet Transmission;
S3. the described semi-structured data described in the process of Hadoop cluster module, and send to described data aggregate module;
S4. the described corresponding data of data aggregate resume module, and send to described Data dissemination module;
The data received are assigned in lasting data memory module by S5. described Data dissemination module.
6. the construction method of a kind of cloud platform data management system based on the large data of industry according to claim 5, it is characterized in that: described construction method also comprises step S6: described cloud platform data management system also comprises security module, described security module communicates with each processing module, ensures the security of data.
7. the construction method of a kind of cloud platform data management system based on the large data of industry according to claim 5, it is characterized in that: described step S3 comprises, described Hadoop cluster module also comprises data structured module, data debug module, data deduplication module, Data Integration module, first, the described semi-structured data that described data structured modular structure process receives, generating structured data; Described data debug module removes the structural data of mistake, described data deduplication module row except the structural data repeated, the data after the module integrated described data debug module of described Data Integration and the process of described data deduplication module.
8. the construction method of a kind of cloud platform data management system based on the large data of industry according to claim 5, it is characterized in that: described step S4 comprises, described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module by similar data gathering together, generates set of metadata of similar data class, and the module integrated described set of metadata of similar data class of described data coupling, optimizes the data of described set of metadata of similar data class; Described data compressing module calling data compression algorithm compresses the data of described set of metadata of similar data class.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610079827.5A CN105488235A (en) | 2016-02-03 | 2016-02-03 | Cloud platform data management system based on industrial big data and construction method thereof |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610079827.5A CN105488235A (en) | 2016-02-03 | 2016-02-03 | Cloud platform data management system based on industrial big data and construction method thereof |
Publications (1)
Publication Number | Publication Date |
---|---|
CN105488235A true CN105488235A (en) | 2016-04-13 |
Family
ID=55675210
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201610079827.5A Pending CN105488235A (en) | 2016-02-03 | 2016-02-03 | Cloud platform data management system based on industrial big data and construction method thereof |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN105488235A (en) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106885604A (en) * | 2017-02-15 | 2017-06-23 | 重庆工商职业学院 | A kind of industrial feeding vehicle based on big data feeds intake quantity collection system in real time |
CN107066551A (en) * | 2017-03-23 | 2017-08-18 | 中国科学院计算技术研究所 | The line and column storage method and system of a kind of tree shaped data |
CN107315769A (en) * | 2017-05-18 | 2017-11-03 | 北京安点科技有限责任公司 | Simplify and processing system with reference to the mass data of multifactor optimization technology and MapReduce technologies |
CN107480244A (en) * | 2017-08-10 | 2017-12-15 | 成都天衡电科科技有限公司 | A kind of industrial data collects and processing system and its processing method |
CN108021051A (en) * | 2016-10-31 | 2018-05-11 | 无锡云汇科技有限公司 | Industrial control unit (ICU) |
CN109460498A (en) * | 2018-11-07 | 2019-03-12 | 广州小天软件有限公司 | A kind of verification of data method and device |
CN110304510A (en) * | 2019-05-31 | 2019-10-08 | 安徽电梯大叔科技有限公司 | A kind of intelligent elevator supervisory systems |
CN112699108A (en) * | 2020-12-25 | 2021-04-23 | 中科恒运股份有限公司 | Data reconstruction method and device for marital registration system and terminal equipment |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102566493A (en) * | 2012-01-17 | 2012-07-11 | 上海交通大学 | Data acquiring and processing embedded adapter of numerical control machine |
CN104156810A (en) * | 2014-07-31 | 2014-11-19 | 国网山东省电力公司 | Power dispatching production management system based on cloud computing and realization method of power dispatching production management system |
CN104410662A (en) * | 2014-10-23 | 2015-03-11 | 山东大学 | Parallel mass data transmitting middleware of Internet of things and working method thereof |
US20150095384A1 (en) * | 2013-09-27 | 2015-04-02 | Tata Consultancy Services Limited | File transfer to a distributed file system |
CN104850640A (en) * | 2015-05-26 | 2015-08-19 | 华北电力大学(保定) | HBase based storage and query method and system for power equipment status monitoring data |
CN105045856A (en) * | 2015-07-09 | 2015-11-11 | 中国资源卫星应用中心 | Hadoop-based data processing system for big-data remote sensing satellite |
CN105260448A (en) * | 2015-10-10 | 2016-01-20 | 成都博元时代软件有限公司 | Big data information analysis method |
-
2016
- 2016-02-03 CN CN201610079827.5A patent/CN105488235A/en active Pending
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102566493A (en) * | 2012-01-17 | 2012-07-11 | 上海交通大学 | Data acquiring and processing embedded adapter of numerical control machine |
US20150095384A1 (en) * | 2013-09-27 | 2015-04-02 | Tata Consultancy Services Limited | File transfer to a distributed file system |
CN104156810A (en) * | 2014-07-31 | 2014-11-19 | 国网山东省电力公司 | Power dispatching production management system based on cloud computing and realization method of power dispatching production management system |
CN104410662A (en) * | 2014-10-23 | 2015-03-11 | 山东大学 | Parallel mass data transmitting middleware of Internet of things and working method thereof |
CN104850640A (en) * | 2015-05-26 | 2015-08-19 | 华北电力大学(保定) | HBase based storage and query method and system for power equipment status monitoring data |
CN105045856A (en) * | 2015-07-09 | 2015-11-11 | 中国资源卫星应用中心 | Hadoop-based data processing system for big-data remote sensing satellite |
CN105260448A (en) * | 2015-10-10 | 2016-01-20 | 成都博元时代软件有限公司 | Big data information analysis method |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108021051A (en) * | 2016-10-31 | 2018-05-11 | 无锡云汇科技有限公司 | Industrial control unit (ICU) |
CN106885604A (en) * | 2017-02-15 | 2017-06-23 | 重庆工商职业学院 | A kind of industrial feeding vehicle based on big data feeds intake quantity collection system in real time |
CN106885604B (en) * | 2017-02-15 | 2018-12-14 | 重庆工商职业学院 | A kind of industrial feeding vehicle based on big data feeds intake quantity collection system in real time |
CN107066551B (en) * | 2017-03-23 | 2020-04-03 | 中国科学院计算技术研究所 | Row-type and column-type storage method and system for tree-shaped data |
CN107066551A (en) * | 2017-03-23 | 2017-08-18 | 中国科学院计算技术研究所 | The line and column storage method and system of a kind of tree shaped data |
CN107315769A (en) * | 2017-05-18 | 2017-11-03 | 北京安点科技有限责任公司 | Simplify and processing system with reference to the mass data of multifactor optimization technology and MapReduce technologies |
CN107315769B (en) * | 2017-05-18 | 2021-03-12 | 北京安点科技有限责任公司 | Mass data simplifying and processing system combining multi-factor analysis technology and MapReduce technology |
CN107480244A (en) * | 2017-08-10 | 2017-12-15 | 成都天衡电科科技有限公司 | A kind of industrial data collects and processing system and its processing method |
CN113220776A (en) * | 2017-08-10 | 2021-08-06 | 成都天衡智造科技有限公司 | Industrial data processing system and method |
CN113220776B (en) * | 2017-08-10 | 2022-06-17 | 成都天衡智造科技有限公司 | Industrial data processing system and method |
CN109460498A (en) * | 2018-11-07 | 2019-03-12 | 广州小天软件有限公司 | A kind of verification of data method and device |
CN110304510A (en) * | 2019-05-31 | 2019-10-08 | 安徽电梯大叔科技有限公司 | A kind of intelligent elevator supervisory systems |
CN112699108A (en) * | 2020-12-25 | 2021-04-23 | 中科恒运股份有限公司 | Data reconstruction method and device for marital registration system and terminal equipment |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN105488235A (en) | Cloud platform data management system based on industrial big data and construction method thereof | |
CN109213600B (en) | GPU resource scheduling method and device based on AI cloud | |
CN108924250B (en) | Service request processing method and device based on block chain and computer equipment | |
CA2897338C (en) | Data stream splitting for low-latency data access | |
CN104331421A (en) | High-efficiency processing method and system for big data | |
CN112270833B (en) | Trajectory fitting method and device, electronic equipment and storage medium | |
CN103778034A (en) | Cloud storage-based data backup disaster recovery method and system | |
CN105338027A (en) | Method, system and device for cloud storage of video data | |
CN105045856A (en) | Hadoop-based data processing system for big-data remote sensing satellite | |
CN104239518A (en) | Repeated data deleting method and device | |
CN107612984B (en) | Big data platform based on internet | |
CN102750368B (en) | High-speed importing method of cluster data in data base | |
CN110955704A (en) | Data management method, device, equipment and storage medium | |
CN103823807A (en) | Data de-duplication method, device and system | |
CN102523410B (en) | Method for writing video data and video data storage equipment | |
CN105282045B (en) | A kind of distributed computing and storage method based on consistency hash algorithm | |
CN205540723U (en) | Information retrieval system based on cloud calculates | |
CN108306965A (en) | The data processing method and device of camera, storage medium, camera | |
CN103631804A (en) | Map cutting method and processing system of electronic map | |
CN104679905A (en) | High-speed storage system based on cloud storage | |
CN204906437U (en) | Big data storage application network framework | |
CN105162837A (en) | Method and system for improving I/O throughput rate in massive data storage environment | |
CN109597795B (en) | High-efficiency processing system for roadbed compaction construction data | |
CN104933110A (en) | MapReduce-based data pre-fetching method | |
CN105677440A (en) | Virtual machine automatic migrate system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20160413 |
|
WD01 | Invention patent application deemed withdrawn after publication |