CN110618978A - Cloud system integration and storage system and method - Google Patents
Cloud system integration and storage system and method Download PDFInfo
- Publication number
- CN110618978A CN110618978A CN201910890132.9A CN201910890132A CN110618978A CN 110618978 A CN110618978 A CN 110618978A CN 201910890132 A CN201910890132 A CN 201910890132A CN 110618978 A CN110618978 A CN 110618978A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- cloud system
- storage
- acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
- G06F16/183—Provision of network file services by network file servers, e.g. by using NFS, CIFS
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a cloud system integration and storage system and a cloud system integration and storage method, wherein the cloud system comprises an acquisition node for acquiring data and a cloud system for storing the acquired data, the cloud system comprises total data and a data processing module, the total data comprises a data storage module and a data interaction module, and the data storage module is used for storing data information acquired by the acquisition node into a total database in a classified manner. According to the cloud system integration and storage system and method, the integrity of the stored data is improved through the data cleaning module, the operation load during data storage is reduced through the data classification module, and meanwhile the data storage efficiency is improved.
Description
Technical Field
The invention relates to the technical field of cloud system integration, in particular to a cloud system integration and storage system and a cloud system integration and storage method.
Background
The existing society is a society with high-speed development, developed science and technology and information circulation, people communicate with each other more and more closely, the life is more and more convenient, and big data is a product of the high-tech era. With the advent of the cloud era, cloud system integration has attracted more and more attention. In a cloud system integration system, an important link is used for data storage, but the existing data storage cannot clean and classify the data, so that the stored data is incomplete, the data storage efficiency is low, and the operation load is large. In view of the above, a cloud system integration and storage system and method are provided.
Disclosure of Invention
The present invention provides a cloud system integration and storage system and method, so as to solve the problems of incomplete stored data, low data storage efficiency and large operation load in the background art.
In order to achieve the above object, in one aspect, the present invention provides a cloud system integration and storage system, including a collection node for collecting data and a cloud system for storing the collected data, where the cloud system includes total data and a data processing module, the total data includes a data storage module and a data interaction module, the data storage module is used for storing data information collected by the collection node into a total database in a classified manner, and the data interaction module is used for connecting a mobile terminal to the total data and performing data interaction with the total database.
Preferably, the acquisition node comprises an acquisition module, a signal conditioning module, a sample-and-hold module, an a/D conversion module, a single-chip microcomputer module and a data uploading module, wherein the acquisition module is used for acquiring front-end data; the signal conditioning module is used for respectively carrying out signal conversion on the analog output of each sensor so as to adapt to the requirement of the input end of the analog-digital converter on the input signal; the sampling and holding module is used for converting the continuous signals into discontinuous sampling signals and then converting the discontinuous sampling signals into continuous signals; the A/D conversion module is used for converting the analog quantity signal into a digital quantity signal; the single chip microcomputer module is used for processing sampled digital signals, and the data uploading module is used for uploading data values acquired by the acquisition nodes to the data processing module.
Preferably, the data processing module comprises a data receiving module, a data cleaning module, a data classifying module and a classifying and transmitting module, and the data receiving module is used for receiving data acquired by the acquisition node; the data cleaning module is used for deleting and correcting the wrong data value; the data classification module is used for classifying the cleaned data values, and the classification transmission module is used for classifying and uploading the data values according to the classified data types.
Preferably, the data cleaning module comprises an error correcting module, a repeated item deleting module, a unified specification module, a correction logic module, a conversion construction module, a data compression module, a data supplementing module and a data discarding module, wherein the error correcting module is used for correcting the form of data errors, the repeated item deleting module is used for deleting repeated records or repeated fields existing in the data, the unified specification module is used for unifying data specifications and abstracting consistent content, the correction logic module is used for determining the logic, condition and caliber of each source system and correcting the acquisition logic of an abnormal source system, the conversion construction module is used for standardizing the data, the data compression module is used for maintaining the integrity and accuracy of an original data set and reorganizing the data according to a certain algorithm and mode on the premise of not losing useful information, the data supplementing module is used for supplementing the data of the incomplete data, and the data discarding module deletes abnormal data in the data.
In another aspect, the present invention further provides a cloud system integration and storage method, including any one of the above cloud system integration and storage systems, the method steps are as follows:
s1, establishing a cloud system: building a private cloud big data center by using a cloud computing technology, and pooling physical resource data by using a virtualization technology to form a total database;
s2, data acquisition: the front end is provided with a plurality of data acquisition nodes, and the front end data is acquired in real time through the plurality of data acquisition nodes;
s3, data transmission: transmitting the acquired data into a data processing module through a data uploading module of the acquisition node;
s4, data cleaning: correcting errors, deleting repeated items, unifying specifications, correcting logics, converting structures, compressing data, supplementing data and discarding data by a data cleaning module;
s5, data storage: a large amount of data are classified through a data classification module, and the classified data are stored in a total database.
Compared with the prior art, the invention has the beneficial effects that:
1. according to the cloud system integration and storage system and method, the front end is provided with the plurality of data acquisition nodes, the front end data are acquired in real time, the acquired data are transmitted to the data processing module through the data uploading module of the acquisition nodes, and finally the data are corrected, repeated items are deleted, specifications are unified, logic is corrected, structure is converted, data compression, data supplement and data discarding are carried out on the data through the data cleaning module, so that the integrity of the stored data is improved.
2. According to the cloud system integration and storage system and method, a large amount of data are classified through the data classification module, the classified data are stored in the total database, the data can be classified and uploaded during uploading, the operation load during data storage is reduced, and meanwhile the data storage efficiency is improved.
Drawings
FIG. 1 is an overall flow diagram of the present invention;
FIG. 2 is a block diagram of a cloud system of the present invention;
FIG. 3 is a diagram of an overall database module of the present invention;
FIG. 4 is a flow diagram of a data interaction module of the present invention;
FIG. 5 is a block diagram of a collection node of the present invention;
FIG. 6 is a schematic diagram of the operation of the signal conditioning module of the present invention;
FIG. 7 is a pin diagram of the single chip microcomputer of the present invention;
FIG. 8 is a schematic diagram of an NRF401 wireless transmission chip according to the present invention;
FIG. 9 is a second schematic diagram of the NRF401 wireless transmission chip of the present invention;
FIG. 10 is a block diagram of data processing modules of the present invention;
FIG. 11 is a block diagram of a data cleansing module of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", and the like, indicate orientations and positional relationships based on those shown in the drawings, and are used only for convenience of description and simplicity of description, and do not indicate or imply that the equipment or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be considered as limiting the present invention.
Example 1
The invention provides a cloud system integration and storage system, as shown in fig. 1-3, which comprises a collection node for collecting data and a cloud system for storing the collected data, wherein the cloud system comprises total data and a data processing module, the total data comprises a data storage module and a data interaction module, the data storage module is used for storing data information collected by the collection node into a total database in a classified manner, and the data interaction module is used for connecting the total data with a mobile terminal and performing data interaction with the total database.
In this embodiment, the cloud system is implemented based on a SQLSever database, which is composed of a set of tables, where the tables include structured data and other objects defined to support operations on the data and guarantee the integrity of the data, such as views, indexes, storage procedures, user-defined functions, triggers, and the like, and they can form a logical storage structure of the database.
Further, a primary data file, a secondary data file and a transaction log 3 database file exist in the SQLSever database, the primary data file contains starting information of the database and is used for storing data, and the file extension name is as follows. mdf, each database has only one main file, and the file stores two objects, namely a user object and a system object, wherein the user object comprises a table, a view, a storage process and the like and is used for storing data input by a user; the system object has table name, user account number of database, index address, etc. to maintain the information needed by database. The system table should be saved in the primary file and the user data may be moved to the secondary data file.
Specifically, the data storage module is designed based on the HDFS system, the HDFS is composed of several interconnected node clusters, one HDFS cluster of the master/slave structure contains one node, which is a name node NameNode, i.e. a master control server, which manages the namespace of the file system and also controls the access of the clients to the files, and a stack of data nodes, typically one data node deployed on one physical node, which stores the data as blocks in files, in the HDFS, a given name node NameNode manages some file system namespace operations, such as opening, closing and renaming file and directory names, the NameNode also maps data blocks to the data node DataNode and handles read/write requests from the HDFS clients, the data nodes also create, delete and copy data blocks, etc. according to the instructions of the metadata node NameNode, HDFS architecture, as shown in fig. 4.
In addition, the data interaction module realizes data interaction based on an ftp server sharing mode, establishes an ftp server, allocates account numbers, passwords, operation authority of directories and the like to different systems, and rules of data formats, file naming modes, storage paths and the like are required to be defined for two systems needing to exchange data.
In order to facilitate data acquisition, the invention is further provided with an acquisition module, as a preferred embodiment, as shown in fig. 5, the acquisition node comprises an acquisition module, a signal conditioning module, a sample-and-hold module, an a/D conversion module, a single chip module and a data uploading module, and the acquisition module is used for acquiring front-end data; the signal conditioning module is used for respectively carrying out signal conversion on the analog output of each sensor so as to enable the analog output to be suitable for the requirement of the input end of the analog/digital converter on the input signal; the sampling and holding module is used for converting the continuous signals into intermittent sampling signals and then converting the intermittent sampling signals into continuous signals; the A/D conversion module is used for converting the analog quantity signal into a digital quantity signal; the single chip microcomputer module is used for processing the sampled digital signals, and the data uploading module is used for uploading data values acquired by the acquisition nodes to the data processing module.
In this embodiment, the signal conditioning module is configured to perform signal conversion on the analog output of each sensor, so that the signal conditioning module is adapted to the requirement of the input end of the analog-to-digital converter for the input signal, and the functions of the signal conditioning module generally include: the principle of the static processing of signal switching, signal conversion, signal amplification, calibration, linearization, compensation and the like is shown IN fig. 6, IN the diagram, a sensor signal is accessed from a J-IN port, then switches S1-S5 are selected according to the signal type, a conditioned signal is obtained from a J-OUT port and sent to an A/D conversion module, wherein: DGND, VDD denote the sensor-side digital power supply; AGND, V +5 and V-5 denote sensor-side analog power supplies.
Further, the equivalent circuit formula in the standard signal mode is as follows:
VO-＝0 (3)
in the formula, RxIs the on-resistance of MAX383, RwAQW21X, when R is7＝R8，Rw＜R2Obtained by the formulas (1) and (2):
specifically, the sample-and-hold module includes a signal sampling step and a step of converting a sampled signal into a continuous signal, wherein the sampling step is used for discretizing the continuous signal to obtain a sampled signal, and the sampled signal can be described as follows by using a unit pulse sequence function:
in the step of converting the sampling signal into the continuous signal, a zero-order retainer is adopted to convert the sampling signal into a signal which keeps a constant value between two continuous sampling moments, namely, in the interval of T e [ nT, (n +1) T ], the output value of the zero-order retainer is always kept as x (nT).
Specifically, ADC0809 is selected as the A/D conversion module, ADC0809 is a gradual approach type A/D converter, the speed is high, the precision is high, the cost is low, and the conversion time is from several microseconds to several hundred microseconds.
It should be noted that the single chip microcomputer module is an AT89C51 single chip microcomputer, a pin diagram of the single chip microcomputer module is shown in fig. 7, and some pins of the single chip microcomputer module are described as follows:
port P0: port P0 is an 8-bit drain open bidirectional I/O port, each pin can absorb 8TTL gate current, when the pin of port P1 writes 1 for the first time, it is defined as high impedance input, P0 can be used for external program data memory, it can be defined as the eighth bit of data/address;
port P1: the port P1 is an 8-bit bidirectional I/O port internally provided with a pull-up resistor, the buffer of the port P1 can receive and output 4TTL gate current, the pin of the port P1 is internally pulled up to be high after being written into 1 and can be used as input, and the port P1 outputs current when being externally pulled down to be low level;
port P2: the port P2 is an 8-bit bidirectional I/O port of an internal pull-up resistor, the port P2 buffer can receive and output 4TTL gate currents, when the port P2 is written with 1, the pin of the port P2 is pulled high by the internal pull-up resistor and is used as input, and therefore when the port P2 is used as input, the pin of the port P2 is pulled low externally to output current, and when the port P2 is used for accessing an external program memory or a 16-bit address external data memory, the port P2 outputs eight high bits of an address;
port P3: the pin of the P3 port is 8 bidirectional I/O ports with internal pull-up resistors, which can receive and output 4TTL gate currents, when the P3 port writes '1', they are internally pulled up to high level and used as input, and as input, the P3 port outputs current due to external pull-down to low level.
In addition, the data uploading module is designed based on an NRF401 wireless transmission chip, the NRF401 has only 20 pins, the pins and the volume are small, the PCB packaging is facilitated, and the working frequency is 433MHZ of the international universal data frequency band; FSK modulation is adopted, data are directly input and output, the anti-interference capability is strong, the method is particularly suitable for industrial control occasions, a DSS + PLL frequency synthesis technology is adopted, the frequency stability is excellent, and the sensitivity reaches-105 dBm; when the power consumption is small and the standby state is received, the current is only 8UA, the maximum transmitting power is 10dBm, the low working voltage (2.7V) can meet the requirements of low-power consumption equipment, a plurality of frequency channels are provided, the working frequency can be conveniently switched, the working speed can reach 20kbit/s, and the working principle is shown in fig. 8 and fig. 9.
In order to facilitate the classification of data, the present invention further improves the data processing module, as a preferred embodiment, as shown in fig. 10, the data processing module includes a data receiving module, a data cleaning module, a data classification module, and a classification transmission module, and the data receiving module is configured to receive data acquired by the acquisition node; the data cleaning module is used for deleting and correcting the wrong data value; the data classification module is used for classifying the cleaned data values, and the classification transmission module is used for classifying and uploading the data values according to the classified data types.
In this embodiment, the data cleaning module includes an error correcting module, a duplicate item deleting module, a specification unifying module, a correction logic module, a conversion constructing module, a data compressing module, a data complementing module and a data discarding module, the error correcting module is used for correcting data error forms, the duplicate item deleting module is used for deleting duplicate records or duplicate fields existing in data, the specification unifying module is used for unifying data specifications and abstracting consistent contents, the correction logic module is used for determining logic, conditions and apertures of each source system and correcting acquisition logic of an abnormal source system, the conversion constructing module is used for standardizing data, the data compressing module is used for maintaining integrity and accuracy of an original data set, and reorganizing data according to a certain algorithm and mode on the premise of not losing useful information, the data supplementing module is used for supplementing the data of the incomplete data, and the data discarding module deletes abnormal data in the data.
In this embodiment, the error correcting module is configured to correct a data error form, and the error correcting module is configured to correct a data value error, correct a data type error, correct a data coding error, correct a data format error, correct a data exception error, correct a dependency conflict, and correct a multi-value error.
Further, due to various reasons, repeated records or repeated fields (columns) may exist in the data, repeated items (rows and columns) need to be deleted and processed by a repeated item deleting module, the repeated item deleting module is used for deleting the repeated records or the repeated fields existing in the data, and for judging the repeated items, the basic idea is 'sorting and merging', the records in the database are sorted according to a certain rule, and then whether the records are repeated is detected by comparing whether adjacent records are similar.
Specifically, because the data source systems are dispersed in each service line, different service lines have different requirements, understandings and specifications for data, and the description specifications for the same data object are completely different, the data specification needs to be unified through the unified specification module and the content of consistency needs to be abstracted out in the cleaning process.
In addition, the correction logic module is used for determining the logic, conditions and caliber of each source system and correcting the acquisition logic of the abnormal source system.
In addition, the conversion construction module is used for carrying out standardization processing on the data, and comprises data type conversion, data semantic conversion, data granularity conversion, table/data splitting, row-column conversion, data discretization, data standardization, new field refinement and attribute construction.
Wherein, the data type conversion: when data come from different data sources, incompatibility of data types of the different data sources may cause error reporting of the system, and at this time, the data types of the different data sources need to be uniformly converted into a compatible data type.
Wherein, the data semantic conversion: in a conventional data warehouse, a dimension table, a fact table and the like may exist based on a third paradigm, and at this time, many fields in the fact table need to be combined with the dimension table to perform semantic parsing.
Wherein, the data granularity conversion: and aggregating the data according to different granularity requirements in the data warehouse.
Wherein, table/data splitting: some fields may store multiple data information, for example, the timestamp includes information of year, month, day, hour, minute, second, etc., and some rules need to split some or all of the time attributes to meet the data aggregation requirement at multiple granularities.
Wherein, the row-column conversion: and converting row and column data in the table.
Wherein, data discretization: the continuous attribute is discretized into a plurality of intervals to help reduce the value number of one continuous attribute.
Wherein, data standardization: different fields have different business meanings, so that the difference between values caused by different orders of magnitude among variables needs to be eliminated.
Wherein, refining the new field: in many cases, new fields, also called compound fields, need to be extracted based on business rules.
Wherein, the attribute structure is as follows: in the modeling process, new attributes are constructed according to the existing attribute set.
Furthermore, the data compression module is used for maintaining the integrity and accuracy of the original data set, and reorganizing the data according to a certain algorithm and a certain mode on the premise of not losing useful information, and complex data analysis and data calculation of large-scale data generally consume a large amount of time, so that reduction and compression of the data are required before the reorganization and the compression, the data scale is reduced, interactive data mining can be faced, and information feedback is carried out on the comparison data before and after the data mining. Thus, the data mining on the reduced data set is obviously more efficient, and the mining result is basically the same as the result obtained by using the original data set.
In addition, the data supplementing module is used for supplementing the data of the incomplete data, the data supplementation comprises supplementing a missing value and a supplementing null value, the missing value refers to the condition that the data originally and necessarily exists, but actually has no data, and the null value refers to the condition that the data actually exists and can be null.
In addition, the data discarding module deletes abnormal data in the data, the types of discarded data include whole deletion and variable deletion, the whole deletion refers to deleting samples containing missing values, the variable deletion can be considered if the invalid value and the missing value of a certain variable are many, and the variable is not particularly important for the problem under study, and this way reduces the number of variables for analysis, but does not change the sample amount.
It should be noted that the data classification module classifies data based on the characteristic information of the data, the classification algorithm is based on the frequency domain characteristic data extraction design, the frequency domain characteristic mainly uses the data extraction of the linear prediction and mel frequency cepstrum coefficient calculation method, the mel frequency cepstrum coefficient is a calculation method applied to the equidistant frequency band division data extraction characteristic, the method has high anti-interference capability, therefore, the calculation method is often used as one of the main means of data characteristic extraction, and the formula of the discrete spectrum is as follows:
in the formula, k is the number e of Fourier transform points and is the frequency.
Using W (k) to calculate discrete frequenciesIs the value of (1), i.e. W2(k) The energy of the output data at this time is:
wherein: h is the energy value of the processed output data; w2(k) Is the number of treatments. According to the sequence of data classification processing, the expression of the calculation of the mel-frequency cepstrum coefficient can be obtained as follows:
wherein: and m is the data processing sequence. The expression for the linear prediction coefficient can thus be derived as:
wherein: m is the stage of linear prediction data; t' (k) is a combination of kth sequence real numbers; i is a natural number. Through the above, the design of the system software can be completed.
On the other hand, the invention also provides a cloud system integration and storage method, which comprises the following steps:
s1, establishing a cloud system: building a private cloud big data center by using a cloud computing technology, and pooling physical resource data by using a virtualization technology to form a total database;
s2, data acquisition: the front end is provided with a plurality of data acquisition nodes, and the front end data is acquired in real time through the plurality of data acquisition nodes;
s3, data transmission: transmitting the acquired data into a data processing module through a data uploading module of the acquisition node;
s4, data cleaning: correcting errors, deleting repeated items, unifying specifications, correcting logics, converting structures, compressing data, supplementing data and discarding data by a data cleaning module;
s5, data storage: a large amount of data are classified through a data classification module, and the classified data are stored in a total database.
The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, and the preferred embodiments of the present invention are described in the above embodiments and the description, and are not intended to limit the present invention. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (5)
1. A cloud system integration and storage system comprising a collection node for collecting data and a cloud system for storing the collected data, characterized in that: the cloud system comprises a total data and data processing module, the total data comprises a data storage module and a data interaction module, the data storage module is used for storing data information collected by the collection nodes into a total database in a classified mode, and the data interaction module is used for connecting the total data with the mobile terminal and performing data interaction with the total database.
2. The cloud system integration and storage system of claim 1, wherein: the acquisition node comprises an acquisition module, a signal conditioning module, a sample-and-hold module, an A/D conversion module, a singlechip module and a data uploading module, wherein the acquisition module is used for acquiring front-end data; the signal conditioning module is used for respectively carrying out signal conversion on the analog output of each sensor so as to adapt to the requirement of the input end of the analog-digital converter on the input signal; the sampling and holding module is used for converting the continuous signals into discontinuous sampling signals and then converting the discontinuous sampling signals into continuous signals; the A/D conversion module is used for converting the analog quantity signal into a digital quantity signal; the single chip microcomputer module is used for processing sampled digital signals, and the data uploading module is used for uploading data values acquired by the acquisition nodes to the data processing module.
3. The cloud system integration and storage system of claim 1, wherein: the data processing module comprises a data receiving module, a data cleaning module, a data classifying module and a classifying and transmitting module, and the data receiving module is used for receiving data acquired by the acquisition node; the data cleaning module is used for deleting and correcting the wrong data value; the data classification module is used for classifying the cleaned data values, and the classification transmission module is used for classifying and uploading the data values according to the classified data types.
4. The cloud system integration and storage system of claim 3, wherein: the data cleaning module comprises an error correcting module, a repeated item deleting module, a unified specification module, a correction logic module, a conversion construction module, a data compression module, a data supplementing module and a data discarding module, wherein the error correcting module is used for correcting the form of data errors, the repeated item deleting module is used for deleting repeated records or repeated fields existing in the data, the unified specification module is used for unifying the data specification and abstracting the content of consistency, the correction logic module is used for determining the logic, condition and caliber of each source system and correcting the acquisition logic of an abnormal source system, the conversion construction module is used for standardizing the data, the data compression module is used for keeping the integrity and accuracy of the original data set and reorganizing the data according to a certain algorithm and mode on the premise of not losing useful information, the data supplementing module is used for supplementing the data of the incomplete data, and the data discarding module deletes abnormal data in the data.
5. A cloud system integration and storage method comprising the cloud system integration and storage system of any of claims 1-4, wherein: the method comprises the following steps:
s1, establishing a cloud system: building a private cloud big data center by using a cloud computing technology, and pooling physical resource data by using a virtualization technology to form a total database;
s2, data acquisition: the front end is provided with a plurality of data acquisition nodes, and the front end data is acquired in real time through the plurality of data acquisition nodes;
s3, data transmission: transmitting the acquired data into a data processing module through a data uploading module of the acquisition node;
s4, data cleaning: correcting errors, deleting repeated items, unifying specifications, correcting logics, converting structures, compressing data, supplementing data and discarding data by a data cleaning module;
s5, data storage: a large amount of data are classified through a data classification module, and the classified data are stored in a total database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910890132.9A CN110618978A (en) | 2019-09-20 | 2019-09-20 | Cloud system integration and storage system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910890132.9A CN110618978A (en) | 2019-09-20 | 2019-09-20 | Cloud system integration and storage system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110618978A true CN110618978A (en) | 2019-12-27 |
Family
ID=68923720
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910890132.9A Pending CN110618978A (en) | 2019-09-20 | 2019-09-20 | Cloud system integration and storage system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110618978A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110807449A (en) * | 2020-01-08 | 2020-02-18 | 杭州皓智天诚信息科技有限公司 | Science and technology project application on-line service terminal |
| CN111677658A (en) * | 2020-05-25 | 2020-09-18 | 阿勒泰正元国际矿业有限公司 | Automatic control system and method for mine water pump |
| CN111787082A (en) * | 2020-06-22 | 2020-10-16 | 珠海格力电器股份有限公司 | Method, equipment and system for reporting multi-stage service data |
| CN112866386A (en) * | 2021-01-19 | 2021-05-28 | 青岛越超传媒有限公司 | Data storage data construction method based on cloud computing |
| CN112947263A (en) * | 2021-04-20 | 2021-06-11 | 南京云玑信息科技有限公司 | Management control system based on data acquisition and coding |
| CN114760201A (en) * | 2022-03-30 | 2022-07-15 | 烽台科技（北京）有限公司 | Data acquisition method, device and equipment of industrial control equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105068459A (en) * | 2015-07-28 | 2015-11-18 | 张海丽 | GPRS-based multichannel data collection and remote transmission system |
| CN106202400A (en) * | 2016-07-11 | 2016-12-07 | 广东聚联电子商务股份有限公司 | A kind of big data collection and analysis method complaining behavior |
| CN106326331A (en) * | 2016-06-29 | 2017-01-11 | 河南许继仪表有限公司 | Intelligent power utilization data service system based on cloud computation |
| CN106682225A (en) * | 2017-01-04 | 2017-05-17 | 成都四方伟业软件股份有限公司 | Big data collecting and storing method and system |
| CN108052665A (en) * | 2017-12-29 | 2018-05-18 | 深圳市中易科技有限责任公司 | A kind of data cleaning method and device based on distributed platform |
-
2019
- 2019-09-20 CN CN201910890132.9A patent/CN110618978A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105068459A (en) * | 2015-07-28 | 2015-11-18 | 张海丽 | GPRS-based multichannel data collection and remote transmission system |
| CN106326331A (en) * | 2016-06-29 | 2017-01-11 | 河南许继仪表有限公司 | Intelligent power utilization data service system based on cloud computation |
| CN106202400A (en) * | 2016-07-11 | 2016-12-07 | 广东聚联电子商务股份有限公司 | A kind of big data collection and analysis method complaining behavior |
| CN106682225A (en) * | 2017-01-04 | 2017-05-17 | 成都四方伟业软件股份有限公司 | Big data collecting and storing method and system |
| CN108052665A (en) * | 2017-12-29 | 2018-05-18 | 深圳市中易科技有限责任公司 | A kind of data cleaning method and device based on distributed platform |
Non-Patent Citations (1)
| Title |
|---|
| 张鸿: "《基于人工智能的多媒体数据挖掘和应用实例》", 31 January 2018 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110807449A (en) * | 2020-01-08 | 2020-02-18 | 杭州皓智天诚信息科技有限公司 | Science and technology project application on-line service terminal |
| CN111677658A (en) * | 2020-05-25 | 2020-09-18 | 阿勒泰正元国际矿业有限公司 | Automatic control system and method for mine water pump |
| CN111787082A (en) * | 2020-06-22 | 2020-10-16 | 珠海格力电器股份有限公司 | Method, equipment and system for reporting multi-stage service data |
| CN111787082B (en) * | 2020-06-22 | 2021-11-23 | 珠海格力电器股份有限公司 | Method, equipment and system for reporting multi-stage service data |
| CN112866386A (en) * | 2021-01-19 | 2021-05-28 | 青岛越超传媒有限公司 | Data storage data construction method based on cloud computing |
| CN112947263A (en) * | 2021-04-20 | 2021-06-11 | 南京云玑信息科技有限公司 | Management control system based on data acquisition and coding |
| CN114760201A (en) * | 2022-03-30 | 2022-07-15 | 烽台科技（北京）有限公司 | Data acquisition method, device and equipment of industrial control equipment and storage medium |
| CN114760201B (en) * | 2022-03-30 | 2023-05-23 | 烽台科技（北京）有限公司 | Data acquisition method, device and equipment of industrial control equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110618978A (en) | Cloud system integration and storage system and method | |
| CN109684352B (en) | Data analysis system, data analysis method, storage medium, and electronic device | |
| CN106611046B (en) | Spatial data storage processing middleware system based on big data technology | |
| CN104881424B (en) | A kind of acquisition of electric power big data, storage and analysis method based on regular expression | |
| Karnitis et al. | Migration of relational database to document-oriented database: structure denormalization and data transformation | |
| Vyawahare et al. | A hybrid database approach using graph and relational database | |
| Chavan et al. | Survey paper on big data | |
| CN113535856B (en) | Data synchronization method and system | |
| CN106777027B (en) | Large-scale parallel processing row-column mixed data storage device and storage and query method | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN107301214A (en) | Data migration method, device and terminal device in HIVE | |
| CN110389950B (en) | Rapid running big data cleaning method | |
| Ramesh et al. | Data modelling for discrete time series data using Cassandra and MongoDB | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN112883125A (en) | Entity data processing method, device, equipment and storage medium | |
| CN112817958A (en) | Electric power planning data acquisition method and device and intelligent terminal | |
| CN105824892A (en) | Method for synchronizing and processing data by data pool | |
| CN115033569A (en) | Self-defined remote sensing image metadata warehousing method | |
| CN104731908A (en) | ETL-based data cleaning method | |
| CN110134690B (en) | Method and system for rapidly storing oversized data of Oracle database | |
| Jensen et al. | Time series management systems: a 2022 survey | |
| CN116089417A (en) | Information acquisition method, information acquisition device, storage medium and computer equipment | |
| CN116230220A (en) | Health assessment method, apparatus, computer device and storage medium | |
| CN117033952B (en) | Distributed multi-data source acquisition implementation method | |
| Alami et al. | Entity resolution in nosql data warehouse |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191227 |