CN110457320A - Real-time storage method and apparatus, storage medium and the computer equipment of data - Google Patents
Real-time storage method and apparatus, storage medium and the computer equipment of data Download PDFInfo
- Publication number
- CN110457320A CN110457320A CN201910722409.7A CN201910722409A CN110457320A CN 110457320 A CN110457320 A CN 110457320A CN 201910722409 A CN201910722409 A CN 201910722409A CN 110457320 A CN110457320 A CN 110457320A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- time
- storage system
- dimension
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2291—User-Defined Types; Storage management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses real-time storage method and apparatus, storage medium and the computer equipments of a kind of data, wherein the real-time storage method of the data includes: to export tool in real time by data and off-line data export tool combines, and exports target service data to target data warehouse from business datum storage system；According to preset rules, the target service data for exporting to target data warehouse are handled, dimension statistical data is obtained；The write-in of dimension statistical data is carried out to the data real-time enquiry system of business datum inquiry for data query person；Wherein, business datum storage system includes the real-time storage system of data and off-line data storage system, data export tool in real time and export the target service data from the real-time storage system of the data, and off-line data exports tool and exports the target service data from the off-line data storage system.The present invention solves in the prior art, in offline search data, the problem of cannot accomplishing real-time query business datum.
Description
Technical field
The present invention relates to communication application, more particularly to a kind of data real-time storage method and apparatus, deposit
Storage media and computer equipment.
Background technique
With the development of internet financial business, the data of each service line are more and more diversified, and data are easy to produce
Raw redundancy cannot be multiplexed well, be badly in need of a unified Data Warehouse Platform and data are carried out integration and are slightly summarized, convenient
Each service line carries out unified inquiry and uses.In addition, needing a large amount of policy engine input of real-time query for air control business
Item data facilitates business personnel to carry out the exploitation and verification of model.However, the data warehouse system of existing big data platform in the market
System, although can solve most of offline search demand, but has the following problems: first, business datum not can be carried out in real time
Inquiry；Second, for the higher tables of data of change frequency inside business datum storage system, automated import of data storehouse is not supported
Library；The data maintenance amount of third, big data platform is larger.
Summary of the invention
The embodiment of the present invention provides real-time storage method and apparatus, storage medium and the computer equipment of a kind of data, with
It solves in the prior art, in offline search data, cannot accomplish real-time query business datum, and store for business datum
The higher tables of data of change frequency inside system, the problem of not supporting automated import of data warehouse.
In order to solve the above technical problems, the first technical solution used in the embodiment of the present invention is as follows:
A kind of real-time storage method of data comprising: export tool and off-line data export tool in real time by data
It combines, exports target service data to target data warehouse from business datum storage system；According to preset rules, to exporting to
The target service data of the target data warehouse are handled, and dimension statistical data is obtained；By the dimension statistical number
The data real-time enquiry system of business datum inquiry is carried out for data query person according to write-in；Wherein, business datum storage system
System includes the real-time storage system of data and off-line data storage system, and the data export tool in real time and deposit in real time from the data
Place system exports the target service data, and the off-line data exports tool from described in off-line data storage system export
Target service data.
Optionally, it is described exported in real time by data tool and off-line data export tool combine, deposited from business datum
Place system exports target service data to target data warehouse, comprising: obtains the data by real-time import system and deposits in real time
The patch active layer data of place system, and obtain by offline import system the patch active layer data of the off-line data storage system；Its
In, the patch active layer data are the target service data.
Optionally, the patch active layer data that the real-time storage system of the data is obtained by real-time import system, comprising:
It obtains the log more new data of the business datum storage system in real time by Kafka system, and number is updated according to the log
According to corresponding renewal time point, the patch active layer data of the business datum storage system are exported into the target data storehouse
Library.
Optionally, the patch active layer data that the off-line data storage system is obtained by offline import system, comprising:
Every preset duration, the patch active layer data of the off-line data storage system are exported to by institute by the offline guiding system of Sqoop
State target data warehouse.
Optionally, described according to preset rules, to export to the target service data of the target data warehouse into
Row processing, obtains dimension statistical data, comprising: according to structured query language program, based on the patch active layer data
Layer carries out bedding storage, obtains the patch active layer data Layer, detail layer data layer, summarizes layer data layer and application layer data layer；
Further according to the structured query language program respectively by the patch active layer data, the detail layer data, described summarize
Layer data and the application layer data carry out dimension statistical disposition, obtain the corresponding dimension data of each data Layer.
Optionally, described further according to the structured query language program respectively by patch active layer data, described
Detail layer data, layer data and the application layer data of summarizing carry out dimension statistical disposition, and it is corresponding to obtain each data Layer
Dimension data, comprising: according to the structured query language program respectively by the patch active layer data, the detail number of plies
According to, described summarize layer data and the application layer data is further divided into including natural person's dimension, client's dimension, account dimension
Dimension data including degree, sub- account dimension, cell-phone number dimension and channel dimension.
Optionally, the real-time storage system includes Mysql system, and the off-line data storage system includes Mongodb
System, Redis system and Excel system.
In order to solve the above technical problems, the second technical solution used in the embodiment of the present invention is as follows:
A kind of real-time query device of big data comprising: data export module, for exporting work in real time by data
Tool and off-line data export tool combine, and export target service data to target data warehouse from business datum storage system；
Data processing module, for being carried out to the target service data for exporting to the target data warehouse according to preset rules
Processing, obtains dimension statistical data；Data write. module, for carrying out dimension statistical data write-in for data query person
The data real-time enquiry system of business datum inquiry；Wherein, the business datum storage system includes the real-time storage system of data
With off-line data storage system, the data export tool in real time and export the target service from the real-time storage system of the data
Data, the off-line data export tool and export the target service data from the off-line data storage system.
In order to solve the above technical problems, third technical solution used in the embodiment of the present invention is as follows:
A kind of storage medium is stored thereon with computer program, and it is such as above-mentioned that the computer program is performed realization
The real-time storage method of data.
In order to solve the above technical problems, the 4th technical solution used in the embodiment of the present invention is as follows:
A kind of computer equipment comprising processor, memory and be stored on the memory and can be in the processing
The computer program run on device, the processor realize the real-time storage such as above-mentioned data when executing the computer program
Method.
The beneficial effect of the embodiment of the present invention is: being in contrast to the prior art, the embodiment of the present invention passes through data reality
When export tool and off-line data export tool combine, export target service data to number of targets from business datum storage system
According to warehouse, and according to preset rules, the target service data for exporting to the target data warehouse are handled, are obtained
It is real-time that the data for carrying out business datum inquiry for data query person are finally written in the dimension statistical data by dimension statistical data
Inquiry system solves in the prior art, in offline search data, cannot accomplish real-time query business datum, and be directed to
The higher tables of data of change frequency inside business datum storage system, the problem of not supporting automated import of data warehouse.
Detailed description of the invention
Fig. 1 is the implementation flow chart of one embodiment of real-time storage method of the data of the embodiment of the present invention one；
Fig. 2 is the part-structure frame diagram of one embodiment of real-time storage device of the data of the embodiment of the present invention two；
Fig. 3 is the part-structure frame diagram of one embodiment of storage medium of the embodiment of the present invention three；
Fig. 4 is the part-structure frame diagram of one embodiment of computer equipment of the embodiment of the present invention four.
Specific embodiment
Embodiment one
Referring to Fig. 1, Fig. 1 is the implementation flow chart of the real-time storage method of the data of the embodiment of the present invention, it can in conjunction with Fig. 1
To obtain, a kind of real-time storage method of data of the embodiment of the present invention, comprising:
Step S101: exporting tool by data in real time and off-line data export tool combines, and stores from business datum
System exports target service data to target data warehouse.
Step S102: according to preset rules, the target service data for exporting to the target data warehouse are carried out
Processing, obtains dimension statistical data.
Step S103: the data that dimension statistical data write-in is carried out business datum inquiry for data query person are real-time
Inquiry system.
Wherein, the business datum storage system includes the real-time storage system of data and off-line data storage system, described
Data export tool in real time and export the target service data from the real-time storage system of the data, and the off-line data exports work
Tool exports the target service data from the off-line data storage system.
In the present embodiment, optionally, it is described exported in real time by data tool and off-line data export tool combine,
Target service data are exported to target data warehouse from business datum storage system, comprising:
The patch active layer data of the real-time storage system of the data are obtained by real-time import system, and pass through offline introgressive line
System obtains the patch active layer data of the off-line data storage system.Wherein, the patch active layer data are the target service data.
In the present embodiment, optionally, the patch that the real-time storage system of the data is obtained by real-time import system
Active layer data, comprising:
Obtain the log more new data of the business datum storage system in real time by Kafka system, and according to the day
The patch active layer data of the business datum storage system are exported to the mesh by the corresponding renewal time point of will more new data
Mark data warehouse.Kafka system be by Apache Software Foundation develop an open source stream process platform, by Scala with
Written in Java.Kafka is that a kind of distributed post of high-throughput subscribes to message system, it can handle consumer in website
Everything flow data.This movement action of search and other users (web page browsing) is many on modern network
One key factor of social function.These data are often as the requirement of handling capacity and pass through processing log and log aggregation
To solve.For as the log number Hadoop (distributed system infrastructure developed by apache foundation)
According to off-line analysis system, but require the limitation that handles in real time, this is a feasible solution.The purpose of Kafka is
Unify Message Processing on line and offline by the loaded in parallel mechanism of Hadoop, it is real-time also for being provided by cluster
Message.
In the present embodiment, optionally, the patch that the off-line data storage system is obtained by offline import system
Active layer data, comprising:
Every preset duration, by the offline guiding system of Sqoop by the patch active layer data of the off-line data storage system
Export to the target data warehouse.Sqoop is the tool of a open source, is mainly used in Hadoop (Hive) and traditional number
According between library (mysql, postgresql...) carry out data transmitting, can by a relevant database (such as: MySQL,
Oracle and Postgres etc.) in data lead in the HDFS (distributed file system of Hadoop) for entering Hadoop, can also
It is entered in relevant database with leading the data of HDFS.
In the present embodiment, optionally, described according to preset rules, to the mesh for exporting to the target data warehouse
Mark business datum is handled, and dimension statistical data is obtained, comprising:
First, according to structured query language program, layer based on the patch active layer data is subjected to bedding storage, is obtained
To the patch active layer data Layer, detail layer data layer, summarize layer data layer and application layer data layer.
Second, further according to the structured query language program respectively by the patch active layer data, the detail layer
Data, layer data and the application layer data of summarizing carry out dimension statistical disposition, obtain the corresponding dimension of each data Layer
Data.
In the present embodiment, optionally, described further according to the structured query language program respectively by the patch
Active layer data, the detail layer data, layer data and the application layer data of summarizing carry out dimension statistical disposition, obtain each
The corresponding dimension data of a data Layer, comprising:
According to the structured query language program respectively by the patch active layer data, the detail layer data, the remittance
Total layer data and the application layer data are further divided into including natural person's dimension, client's dimension, account dimension, sub- account dimension
Dimension data including degree, cell-phone number dimension and channel dimension.
In the present embodiment, specifically, first by data warehouse bedding storage, be divided into patch active layer, detail layer, summarize layer,
Application layer will export to the target service data of the target data warehouse, as the patch active layer of data warehouse, then with
Based on pasting active layer, building shows sub-layers by the big width that subject area divides, then is slightly aggregated to form and summarizes layer, eventually forms
Serve the application layer of business side's reporting system and model system etc..
During several storehouses are built, the dimension statistical number from bottom to top of sub- account, account, client and natural person are established
According to facilitating the lower brill of data.Also, it is divided into client domain, shared domain, the several subject areas of transaction domain and standing domain, each data field
For different business, facilitate inquiry.
In the present embodiment, optionally, the real-time storage system includes Mysql (relevant database) system, described
Off-line data storage system includes Mongodb system, Redis system and Excel system.Mysql is a kind of relation data depositary management
Reason system, relational database save the data in different tables, rather than all data are placed in one big warehouse, in this way
It increases speed and improves flexibility.Mongodb is a production between relational database and non-relational database
Product are that function is most abundant in non-relational database, are most like relational database.The data structure that it is supported is very loose, is
The bson format of similar json, therefore can store more complicated data type.The feature of Mongo maximum is looking into for its support
Inquiry language is very powerful, and grammer is somewhat similarly to the query language of object-oriented, and similarity relation database almost may be implemented
Most functions of single table inquiry, but also support to establish data and index.Redis is the use ANSIC language an of open source
Speech writes, support network, it is memory-based also can persistence log type, Key-Value database, and provide multilingual
API.Microsoft Excel is that Microsoft is to be compiled using the computer of Windows and Apple Macintosh
A spreadsheet software write.Intuitive interface, outstanding computing function and graph tool, along with successful market is sought
Pin makes Excel become most popular pc data processing software.
The embodiment of the present invention exports tool in real time by data and off-line data exports tool and combines, and deposits from business datum
Place system exports target service data to target data warehouse, and according to preset rules, to exporting to the target data warehouse
The target service data handled, obtain dimension statistical data, finally by the dimension statistical data be written for data
Inquiry carries out the data real-time enquiry system of business datum inquiry, solves in the prior art, in offline search data, no
It can accomplish real-time query business datum, and for the higher tables of data of change frequency inside business datum storage system, no
The problem of supporting automated import of data warehouse.
Embodiment two
Referring to Fig. 2, Fig. 2 is the part-structure frame diagram of the real-time storage device of the data of the embodiment of the present invention, in conjunction with
Fig. 2 is available, a kind of real-time storage device 100 of data of the invention, comprising:
Data export module 110, for passing through, data export tool in real time and off-line data exports tool and combines, working
Business data storage system exports target service data to target data warehouse.
Data processing module 120 is used for according to preset rules, to the target industry for exporting to the target data warehouse
Business data are handled, and dimension statistical data is obtained.
Data write. module 130 is looked into for dimension statistical data write-in to be carried out business datum for data query person
The data real-time enquiry system of inquiry.
Wherein, the business datum storage system includes the real-time storage system of data and off-line data storage system, described
Data export tool in real time and export the target service data from the real-time storage system of the data, and the off-line data exports work
Tool exports the target service data from the off-line data storage system.
The embodiment of the present invention exports tool in real time by data and off-line data exports tool and combines, and deposits from business datum
Place system exports target service data to target data warehouse, and according to preset rules, to exporting to the target data warehouse
The target service data handled, obtain dimension statistical data, finally by the dimension statistical data be written for data
Inquiry carries out the data real-time enquiry system of business datum inquiry, solves in the prior art, in offline search data, no
It can accomplish real-time query business datum, and for the higher tables of data of change frequency inside business datum storage system, no
The problem of supporting automated import of data warehouse.
Embodiment three
Referring to Fig. 3, can see with reference to Fig. 3, a kind of storage medium 10 of the embodiment of the present invention, the storage medium
10, such as: ROM/RAM, magnetic disk, CD are stored thereon with computer program 11, and the computer program 11 is performed realization
The real-time storage method of data as described in embodiment one.Since the real-time storage method of the data has been carried out in embodiment one
Detailed description, this will not be repeated here.
The real-time storage method for the data that the embodiment of the present invention is realized, exports tool by data in real time and off-line data is led
Tool combines out, from business datum storage system export target service data to target data warehouse, and according to preset rules,
The target service data for exporting to the target data warehouse are handled, dimension statistical data is obtained, finally by institute
The data real-time enquiry system that the write-in of dimension statistical data carries out business datum inquiry for data query person is stated, solves existing skill
In art, in offline search data, cannot accomplish real-time query business datum, and for business datum storage system inside
The higher tables of data of change frequency, the problem of not supporting automated import of data warehouse.
Example IV
Referring to Fig. 4, can see with reference to Fig. 4, a kind of computer equipment 20 of the embodiment of the present invention comprising processor
21, memory 22 and it is stored in the computer program 221 that can be run on the memory 22 and on the processor 21, it is described
Processor 21 realizes the real-time storage method of the data as described in embodiment one when executing the computer program 221.Due to this
The real-time storage method of data is described in detail in embodiment one, and this will not be repeated here.
The real-time storage method for the data that the embodiment of the present invention is realized, exports tool by data in real time and off-line data is led
Tool combines out, from business datum storage system export target service data to target data warehouse, and according to preset rules,
The target service data for exporting to the target data warehouse are handled, dimension statistical data is obtained, finally by institute
The data real-time enquiry system that the write-in of dimension statistical data carries out business datum inquiry for data query person is stated, solves existing skill
In art, in offline search data, cannot accomplish real-time query business datum, and for business datum storage system inside
The higher tables of data of change frequency, the problem of not supporting automated import of data warehouse.
Mode the above is only the implementation of the present invention is not intended to limit the scope of the invention, all to utilize this
Equivalent structure or equivalent flow shift made by description of the invention and accompanying drawing content, it is relevant to be applied directly or indirectly in other
Technical field is included within the scope of the present invention.
Claims (10)
1. a kind of real-time storage method of data characterized by comprising
It exports tool in real time by data and off-line data export tool combines, export target industry from business datum storage system
Data of being engaged in are to target data warehouse；
According to preset rules, the target service data for exporting to the target data warehouse are handled, dimension is obtained
Statistical data；
Dimension statistical data write-in is carried out to the data real-time enquiry system of business datum inquiry for data query person；
Wherein, the business datum storage system includes the real-time storage system of data and off-line data storage system, the data
Export tool exports the target service data from the real-time storage system of the data in real time, the off-line data export tool from
The off-line data storage system exports the target service data.
2. the real-time storage method of data according to claim 1, which is characterized in that described to export work in real time by data
Tool and off-line data export tool combine, from business datum storage system export target service data to target data warehouse,
Include:
The patch active layer data of the real-time storage system of the data are obtained by real-time import system, and are obtained by offline import system
Take the patch active layer data of the off-line data storage system；Wherein, the patch active layer data are the target service data.
3. the real-time storage method of data according to claim 2, which is characterized in that described to be obtained by real-time import system
Take the patch active layer data of the real-time storage system of the data, comprising:
Obtain the log more new data of the business datum storage system in real time by Kafka system, and more according to the log
The patch active layer data of the business datum storage system are exported to the number of targets by the corresponding renewal time point of new data
According to warehouse.
4. the real-time storage method of data according to claim 2, which is characterized in that described to be obtained by offline import system
Take the patch active layer data of the off-line data storage system, comprising:
Every preset duration, the patch active layer data of the off-line data storage system are exported by Sqoop offline guiding system
To the target data warehouse.
5. the real-time storage method of data according to claim 2, which is characterized in that it is described according to preset rules, to leading
The target service data for arriving the target data warehouse out are handled, and dimension statistical data is obtained, comprising:
According to structured query language program, layer based on the patch active layer data is subjected to bedding storage, obtains the patch
Active layer data Layer, summarizes layer data layer and application layer data layer at detail layer data layer；
Further according to the structured query language program respectively by patch active layer data, the detail layer data, described
Summarize layer data and the application layer data carries out dimension statistical disposition, obtains the corresponding dimension data of each data Layer.
6. the real-time storage method of data according to claim 5, which is characterized in that described further according to the structure
Change query language program respectively by the patch active layer data, the detail layer data, described summarize layer data and the application layer
Data carry out dimension statistical disposition, obtain the corresponding dimension data of each data Layer, comprising:
According to the structured query language program respectively by the patch active layer data, the detail layer data, described summarize layer
Data and the application layer data be further divided into including natural person's dimension, client's dimension, account dimension, sub- account dimension,
Dimension data including cell-phone number dimension and channel dimension.
7. the real-time storage method of data according to claim 1, which is characterized in that the real-time storage system includes
Mysql system, the off-line data storage system include Mongodb system, Redis system and Excel system.
8. a kind of real-time query device of big data characterized by comprising
Data export module exports tool for exporting tool and off-line data in real time by data and combines, from business datum
Storage system exports target service data to target data warehouse；
Data processing module is used for according to preset rules, to the target service data for exporting to the target data warehouse
It is handled, obtains dimension statistical data；
Data write. module, for dimension statistical data write-in to be carried out to the data of business datum inquiry for data query person
Real time inquiry system；
Wherein, the business datum storage system includes the real-time storage system of data and off-line data storage system, the data
Export tool exports the target service data from the real-time storage system of the data in real time, the off-line data export tool from
The off-line data storage system exports the target service data.
9. a kind of storage medium, which is characterized in that be stored thereon with computer program, the computer program is performed realization
The real-time storage method of the described in any item data of claim 1~7.
10. a kind of computer equipment, which is characterized in that it includes processor, memory and is stored on the memory and can
The computer program run on the processor, the processor realize claim 1~7 when executing the computer program
The real-time storage method of described in any item data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910722409.7A CN110457320A (en) | 2019-08-06 | 2019-08-06 | Real-time storage method and apparatus, storage medium and the computer equipment of data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910722409.7A CN110457320A (en) | 2019-08-06 | 2019-08-06 | Real-time storage method and apparatus, storage medium and the computer equipment of data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110457320A true CN110457320A (en) | 2019-11-15 |
Family
ID=68485131
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910722409.7A Pending CN110457320A (en) | 2019-08-06 | 2019-08-06 | Real-time storage method and apparatus, storage medium and the computer equipment of data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110457320A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110941659A (en) * | 2019-11-26 | 2020-03-31 | 上海景域文化传播股份有限公司 | Data layered splitting method for composite revenue |
| CN112365355A (en) * | 2020-12-10 | 2021-02-12 | 深圳迅策科技有限公司 | Method, device and readable medium for calculating fund valuation and risk index in real time |
| CN112380295A (en) * | 2020-11-16 | 2021-02-19 | 常州微亿智造科技有限公司 | Warehouse counting system based on industrial cloud edge service |
| CN113656370A (en) * | 2021-08-16 | 2021-11-16 | 南方电网数字电网研究院有限公司 | Data processing method and device for power measurement system and computer equipment |
| CN114168624A (en) * | 2021-12-08 | 2022-03-11 | 掌阅科技股份有限公司 | Data analysis method, computing device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108427711A (en) * | 2018-01-31 | 2018-08-21 | 北京三快在线科技有限公司 | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium |
| CN108874982A (en) * | 2018-06-11 | 2018-11-23 | 华南理工大学 | A method of based on the offline real-time processing data of Spark big data frame |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
| CN109783512A (en) * | 2018-12-13 | 2019-05-21 | 平安科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
| CN110032591A (en) * | 2018-12-28 | 2019-07-19 | 国网浙江省电力有限公司信息通信分公司 | A kind of assets big data intelligent analysis method |
-
2019
- 2019-08-06 CN CN201910722409.7A patent/CN110457320A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108427711A (en) * | 2018-01-31 | 2018-08-21 | 北京三快在线科技有限公司 | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium |
| CN108874982A (en) * | 2018-06-11 | 2018-11-23 | 华南理工大学 | A method of based on the offline real-time processing data of Spark big data frame |
| CN109783512A (en) * | 2018-12-13 | 2019-05-21 | 平安科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
| CN110032591A (en) * | 2018-12-28 | 2019-07-19 | 国网浙江省电力有限公司信息通信分公司 | A kind of assets big data intelligent analysis method |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110941659A (en) * | 2019-11-26 | 2020-03-31 | 上海景域文化传播股份有限公司 | Data layered splitting method for composite revenue |
| CN112380295A (en) * | 2020-11-16 | 2021-02-19 | 常州微亿智造科技有限公司 | Warehouse counting system based on industrial cloud edge service |
| CN112365355A (en) * | 2020-12-10 | 2021-02-12 | 深圳迅策科技有限公司 | Method, device and readable medium for calculating fund valuation and risk index in real time |
| CN112365355B (en) * | 2020-12-10 | 2023-12-26 | 深圳迅策科技有限公司 | Method, device and readable medium for calculating foundation valuation and risk index in real time |
| CN113656370A (en) * | 2021-08-16 | 2021-11-16 | 南方电网数字电网研究院有限公司 | Data processing method and device for power measurement system and computer equipment |
| CN113656370B (en) * | 2021-08-16 | 2024-04-30 | 南方电网数字电网集团有限公司 | Data processing method and device for electric power measurement system and computer equipment |
| CN114168624A (en) * | 2021-12-08 | 2022-03-11 | 掌阅科技股份有限公司 | Data analysis method, computing device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110457320A (en) | Real-time storage method and apparatus, storage medium and the computer equipment of data | |
| Phaneendra et al. | Big Data-solutions for RDBMS problems-A survey | |
| US11941016B2 (en) | Using specified performance attributes to configure machine learning pipepline stages for an ETL job | |
| CN108427711B (en) | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium | |
| JP6388655B2 (en) | Generation of multi-column index of relational database by data bit interleaving for selectivity | |
| CN104820670B (en) | A kind of acquisition of power information big data and storage method | |
| Chong et al. | Big data analytics: a literature review | |
| Aboutorabiª et al. | Performance evaluation of SQL and MongoDB databases for big e-commerce data | |
| CA2881804C (en) | High performance real-time relational database system and methods for using same | |
| US8972337B1 (en) | Efficient query processing in columnar databases using bloom filters | |
| CN106951552A (en) | A kind of user behavior data processing method based on Hadoop | |
| CN112365355B (en) | Method, device and readable medium for calculating foundation valuation and risk index in real time | |
| Caldarola et al. | Modern enterprises in the bubble: Why big data matters | |
| Caldarola et al. | Big data: A survey-the new paradigms, methodologies and tools | |
| WO2021213154A1 (en) | Blockchain data processing method, system, terminal, and computer-readable storage medium | |
| CN102819616B (en) | Cloud online real-time multi-dimensional analysis system and method | |
| CN109829003A (en) | Database backup method and device | |
| Jia et al. | Development model of enterprise green marketing based on cloud computing | |
| Delchev et al. | Big Data Analysis Architecture | |
| Dutta et al. | Big data analytics for real time systems | |
| CN106648672A (en) | Method and system for developing and running big data | |
| Chen et al. | Analysis of plant breeding on hadoop and spark | |
| Tallón-Ballesteros | The design of ERP intelligent sales management system | |
| CN114020814A (en) | Process industrial manufacturing full-chain data integration and analysis method and system | |
| Xie et al. | Explore the application of big data technology in modern enterprise logistics management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191115 |
|
| RJ01 | Rejection of invention patent application after publication |