CN112486961A - Method and device for processing big data in real time - Google Patents
Method and device for processing big data in real time Download PDFInfo
- Publication number
- CN112486961A CN112486961A CN202011293492.XA CN202011293492A CN112486961A CN 112486961 A CN112486961 A CN 112486961A CN 202011293492 A CN202011293492 A CN 202011293492A CN 112486961 A CN112486961 A CN 112486961A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- processing
- result
- processing result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a big data real-time processing method and a big data real-time processing device, wherein the processing method comprises the following steps: step 1: receiving data and transmitting the data to a database. Step 2: and sampling from the database, and obtaining the quality level of the data according to the defect and the abnormal level of the data. And step 3: and distributing computer resources according to the quality level of the data, and estimating the data processing result. And 4, step 4: and comparing the data processing estimation result with the data processing result to obtain a comparison result, and then evaluating to complete data processing. According to the invention, the computer resources are allocated according to the quality level of the data, the efficiency of big data processing is improved, and the data processing result can be evaluated, so that the utilization of the computer resources is more reasonable.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and a device for real-time processing of big data.
Background
With the continuous expansion of industrial scale and the rise of intelligent manufacturing, industrial big data processing and analysis become important methods for various manufacturers to innovate products and improve the operation level and the production efficiency.
At present, most industrial big data processing methods and devices can not perform reasonable computer resource allocation according to the quality of received data, so that the real-time processing efficiency is influenced, and can not perform estimation and judgment on the processing effect of the data and perform automatic adjustment processing strategies according to the effect, so that a big data real-time processing method and device are provided for solving the problems.
Disclosure of Invention
The application aims to at least solve the technical problems in the prior art, and provides a method and a device for processing big data in real time, so that the efficiency of big data processing is improved, and the real-time requirement of industrial big data processing is met.
The embodiment of the application provides a method for processing big data in real time, which comprises the following steps:
step 1: receiving data and transmitting the data to a database;
step 2: extracting a sample from a database, and evaluating the numerical value abnormal level according to the data loss and the numerical value abnormal level to obtain the quality level of the data;
and step 3: computer resource allocation is carried out according to the data quality level, and meanwhile, the data processing result is pre-estimated;
and 4, step 4: and comparing the data processing estimation result with the data processing result to obtain a comparison result, and then evaluating to complete data processing.
Correspondingly, the invention also provides a big data real-time processing device, which comprises: the system comprises a data receiving module, a data transmission module, a data evaluation module, a computer resource distribution module, a data processing module, a control module, a processing result estimation module and a processing result evaluation module, and is characterized in that the transmission module is connected with the data receiving module and the data evaluation module, the computer resource distribution module is connected with the data processing module, the control module and the data evaluation module, and the processing result evaluation module is connected with the processing result estimation module, the control module and the data processing module.
The data receiving module is used for receiving data and then transmitting the data into the data transmission module. The data transmission module inputs data into a database;
the data evaluation module extracts sample data from the database and evaluates the missing and numerical abnormal levels of the sample data so as to obtain the quality level of the data;
the computer resource allocation module allocates computer resources according to the data quality level obtained by the data evaluation module;
the processing result estimation module estimates the data processing result and inputs the estimated result into the processing result evaluation module;
the data processing module processes data according to the computer resources distributed by the computer resource distribution module and inputs the result into the processing result evaluation module;
the processing result evaluation module compares the result obtained by the data processing module with the estimated result obtained by the processing result estimation module and inputs the evaluation result to the control module.
Compared with the prior art, the method and the processing device based on big data real-time processing can perform quality evaluation and computer resource allocation on input data through the data receiving module, the data transmission module, the data evaluation module and the computer resource allocation module, effectively improve the speed of big data processing, simultaneously can reasonably utilize computer resources, and can further reasonably allocate computer resources through the difference between the actual processing effect and the estimated processing effect which is more obvious seen through the processing result estimation module, the data processing module, the processing result estimation module and the control module. The embodiment of the invention can reasonably distribute the computer resources according to the data quality level, maximize the utilization of the computer resources, improve the efficiency of big data processing and meet the real-time requirement of industrial big data processing.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a method for real-time processing of big data in an embodiment of the invention;
fig. 2 is a schematic structural diagram of a big data real-time processing device in an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 shows a flow chart of a method for real-time processing of big data in an embodiment of the present invention, which specifically includes the following steps:
and S101, receiving data and transmitting the data to a database.
S102, extracting a sample from the database, and evaluating the numerical value abnormal level according to the data loss and the numerical value abnormal level to obtain the quality level of the data.
S103, computer resource allocation is carried out according to the data quality level, and meanwhile, the data processing result is pre-estimated.
And S104, comparing the data processing estimation result with the data processing result to obtain a comparison result, and then evaluating to complete data processing.
Fig. 2 shows a schematic structural diagram of a big data real-time processing device in an embodiment of the present invention, which specifically includes: the system comprises a data receiving module, a data transmission module, a data evaluation module, a computer resource evaluation module, a data processing module, a control module, a processing result pre-estimation module and a processing result evaluation module. The data transmission module is connected with the data evaluation module and the data receiving module, the computer resource distribution module is connected with the data evaluation module, the data processing module, the processing result estimation module and the control module, the processing result estimation module is connected with the processing result estimation module, the data processing module and the control module, the control module is connected with the computer resource distribution module and the processing result estimation module, and the processing result estimation module is connected with the processing result estimation module.
In the embodiment, the data receiving module receives data and transmits the data to the data transmission module, the data transmission module inputs the data into the database, the data evaluation module samples from the database and obtains the quality level of the data according to the defect of sample data and the abnormal level of the output value, the computer resource allocation module allocates computer resources to the data according to the quality level of the data, the data processing module processes the data according to the allocation condition of the computer resources and inputs the result into the processing result evaluation module, and the processing result evaluation module compares the data processing result with the estimation module from the processing result estimation module and completes the data processing through the control module.
In the embodiment, the big data real-time processing device evaluates the quality of input data through the data receiving module, the data transmission module, the data evaluation module and the computer resource allocation module, reasonably allocates computer resources according to the evaluation result, effectively improves the big data processing efficiency, meets the real-time requirement of big data processing, can obviously find out that the evaluation result is compared with the actual processing result through the processing result estimation module, the data result processing module, the control module and the processing result comparison module, and further reasonably allocates computer resources through the control module. The invention can reasonably distribute the computer resources according to the quality level of the input data, not only can improve the real-time processing efficiency of the big data, but also can improve the reasonable utilization level of the computer resources.
The above embodiments of the present invention are described in detail, and the principle and the implementation manner of the present invention should be described herein by using specific embodiments, and the above description of the embodiments is only used to help understanding the method of the present invention and the core idea thereof; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (3)
1. A method for processing big data in real time is characterized by comprising the following steps:
step 1: receiving data and transmitting the data to a database;
step 2: extracting a sample from a database, and evaluating the numerical value abnormal level according to the data loss and the numerical value abnormal level to obtain the quality level of the data;
and step 3: computer resource allocation is carried out according to the data quality level, and meanwhile, the data processing result is pre-estimated;
and 4, step 4: and comparing the data processing estimation result with the data processing result to obtain a comparison result, and then evaluating to complete data processing.
2. A big data real-time processing device is characterized by comprising: the system comprises a data receiving module, a data transmission module, a data evaluation module, a computer resource distribution module, a data processing module, a control module, a processing result estimation module and a processing result evaluation module, and is characterized in that the transmission module is connected with the data receiving module and the data evaluation module, the computer resource distribution module is connected with the data processing module, the control module and the data evaluation module, and the processing result evaluation module is connected with the processing result estimation module, the control module and the data processing module.
3. The big data real-time processing device according to claim 2,
the data receiving module is used for receiving data and then transmitting the data into the data transmission module. The data transmission module inputs data into a database;
the data evaluation module extracts sample data from the database and evaluates the missing and numerical abnormal levels of the sample data so as to obtain the quality level of the data;
the computer resource allocation module allocates computer resources according to the data quality level obtained by the data evaluation module;
the processing result estimation module estimates the data processing result and inputs the estimated result into the processing result evaluation module;
the data processing module processes data according to the computer resources distributed by the computer resource distribution module and inputs the result into the processing result evaluation module;
the processing result evaluation module compares the result obtained by the data processing module with the estimated result obtained by the processing result estimation module and inputs the evaluation result to the control module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011293492.XA CN112486961A (en) | 2020-11-18 | 2020-11-18 | Method and device for processing big data in real time |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011293492.XA CN112486961A (en) | 2020-11-18 | 2020-11-18 | Method and device for processing big data in real time |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112486961A true CN112486961A (en) | 2021-03-12 |
Family
ID=74931682
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011293492.XA Pending CN112486961A (en) | 2020-11-18 | 2020-11-18 | Method and device for processing big data in real time |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112486961A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103247008A (en) * | 2013-05-07 | 2013-08-14 | 国家电网公司 | Quality evaluation method of electricity statistical index data |
| CN104503846A (en) * | 2015-01-22 | 2015-04-08 | 成都派沃特科技有限公司 | Resource management system based on cloud computing system |
| CN106649887A (en) * | 2017-01-16 | 2017-05-10 | 湖南财政经济学院 | Big data processing method and big data processing device |
| CN107103415A (en) * | 2017-04-17 | 2017-08-29 | 中国南方电网有限责任公司 | Power distribution network resource allocation method and system |
| CN107273204A (en) * | 2016-04-08 | 2017-10-20 | 华为技术有限公司 | Resource allocation methods and device for genetic analysis |
| CN110795217A (en) * | 2019-09-27 | 2020-02-14 | 广东浪潮大数据研究有限公司 | Task allocation method and system based on resource management platform |
| CN111143180A (en) * | 2019-12-26 | 2020-05-12 | 南京悠淼科技有限公司 | Performance evaluation system and method for big data analysis |
| CN111597399A (en) * | 2020-05-19 | 2020-08-28 | 莱芜职业技术学院 | Computer data processing system and method based on data fusion |
-
2020
- 2020-11-18 CN CN202011293492.XA patent/CN112486961A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103247008A (en) * | 2013-05-07 | 2013-08-14 | 国家电网公司 | Quality evaluation method of electricity statistical index data |
| CN104503846A (en) * | 2015-01-22 | 2015-04-08 | 成都派沃特科技有限公司 | Resource management system based on cloud computing system |
| CN107273204A (en) * | 2016-04-08 | 2017-10-20 | 华为技术有限公司 | Resource allocation methods and device for genetic analysis |
| CN106649887A (en) * | 2017-01-16 | 2017-05-10 | 湖南财政经济学院 | Big data processing method and big data processing device |
| CN107103415A (en) * | 2017-04-17 | 2017-08-29 | 中国南方电网有限责任公司 | Power distribution network resource allocation method and system |
| CN110795217A (en) * | 2019-09-27 | 2020-02-14 | 广东浪潮大数据研究有限公司 | Task allocation method and system based on resource management platform |
| CN111143180A (en) * | 2019-12-26 | 2020-05-12 | 南京悠淼科技有限公司 | Performance evaluation system and method for big data analysis |
| CN111597399A (en) * | 2020-05-19 | 2020-08-28 | 莱芜职业技术学院 | Computer data processing system and method based on data fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111026553B (en) | Resource scheduling method and server system for offline mixed part operation | |
| CN109743356B (en) | Industrial internet data acquisition method and device, readable storage medium and terminal | |
| ATE514994T1 (en) | METHOD AND DEVICE FOR AUTOMATICALLY BACKING UP DIGITAL DATA FROM A MEMORY OF A COMPUTER INSTALLATION AND FROM A COMPUTER WITH THE APPROPRIATE COMMANDS IN THE MEMORY OF READABLE DATA CARRIERS | |
| CN109451019A (en) | High concurrent acquisition method for industrial real-time data | |
| CN114816721B (en) | Multitask optimization scheduling method and system based on edge calculation | |
| CN112486961A (en) | Method and device for processing big data in real time | |
| CN112534399A (en) | Semantic-based Internet of things equipment data processing related application program installation method and device | |
| CN112486676A (en) | Data sharing and distributing method and device based on edge calculation | |
| CN116681350A (en) | Intelligent factory fault detection method and system | |
| CN109302723B (en) | Multi-node real-time radio monitoring control method based on Internet | |
| CN113626578B (en) | Intelligent analysis method and system for sealing material | |
| CN114760341A (en) | Intelligent control remote operation and maintenance platform and method for industrial internet equipment | |
| CN107948012A (en) | Pressure surveys method and system on a kind of cloud management platform line | |
| CN113821419A (en) | Cloud server aging prediction method based on SVR and Gaussian function | |
| CN104363280B (en) | A kind of monitor and management of cluster method and system based on dual-channel transmission | |
| CN103546341A (en) | Automatic setup method of test environment | |
| CN117472589B (en) | Park network service management method and system | |
| CN117891595A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN117171409B (en) | MES-based production visualization method | |
| CN117040141B (en) | Safety monitoring system and method for electric power intelligent gateway | |
| CN214278671U (en) | Electromechanical device installation energy-saving control system | |
| CN117522067A (en) | Mechanical processing flexible production management system and method based on MES system | |
| CN113315818B (en) | Data acquisition terminal resource adaptation method based on edge calculation | |
| CN115829166B (en) | Equipment selection optimizing system based on efficient machine room | |
| CN107528895B (en) | System and method for accessing plug-in to third-party server based on Android system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210312 |
|
| RJ01 | Rejection of invention patent application after publication |