CN111858722A - Big data application system and method based on Internet of things - Google Patents
Big data application system and method based on Internet of things Download PDFInfo
- Publication number
- CN111858722A CN111858722A CN202010776653.4A CN202010776653A CN111858722A CN 111858722 A CN111858722 A CN 111858722A CN 202010776653 A CN202010776653 A CN 202010776653A CN 111858722 A CN111858722 A CN 111858722A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- internet
- things
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2216/00—Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
- G06F2216/03—Data mining
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to the technical field of big data application, and discloses a big data application system and a method based on the Internet of things, wherein the big data application system based on the Internet of things comprises the following components: the system comprises a data acquisition module, a central control module, an internet of things communication module, a data parallel processing module, a data mining module, a data statistics module, a data retrieval module, a data analysis module, a big data storage module and a display module. The invention greatly improves the parallel processing efficiency of heterogeneous data through the data parallel processing module; meanwhile, the big data storage module is designed and used for the big data industry, can be independently used, is used as a main means for storing and collecting data, and can be matched with data service for use, so that the big data is applied in practical application, the greater effect of the data is exerted, the rapid storage and storage of mass data are realized, and the data efficiency is improved.
Description
Technical Field
The invention belongs to the technical field of big data application, and particularly relates to a big data application system and method based on the Internet of things.
Background
Big data (big data), an IT industry term, refers to a data set that cannot be captured, managed, and processed with a conventional software tool within a certain time range, and is a massive, high-growth-rate, diversified information asset that needs a new processing mode to have stronger decision-making power, insight discovery power, and process optimization capability. The big data concept is applied to data generated by the IT operation tool, and the big data can enable an IT management software provider to solve a wide range of business decisions. IT systems, applications and technology infrastructures produce data every second every day. Big data unstructured or structured data represent an absolute record of more operations like all users' behavior, service level, security, risk, fraud etc. Big data analytics are generated for IT management, where enterprises can combine real-time data flow analysis with historical relevant data, and then big data analyzes and discovers the models they need. Which in turn helps to predict and prevent future outages and performance problems. Furthermore, the big data can be used for knowing the usage model and the geographic trend, so that the insight of the big data on important users is deepened. They can also track and record network behavior, big data easily identify business impact; increasing profit growth with a deep understanding of service utilization; an IT service directory is developed across multiple systems simultaneously collecting data. However, the existing big data application system and method based on the internet of things have low efficiency in processing big data heterogeneous data; meanwhile, the efficiency of storing big data is low.
In summary, the problems of the prior art are as follows: the existing big data application system and method based on the Internet of things have low processing efficiency on big data heterogeneous data; meanwhile, the efficiency of storing big data is low.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a big data application system and method based on the Internet of things.
The invention is realized in such a way that the big data application method based on the Internet of things comprises the following steps:
acquiring network data by using Internet of things equipment through a data acquisition module; acquiring big data of multiple dimensions from a big data server of the Internet of things by using a mining program through a data mining module, and clustering all the big data of each dimension to obtain a cluster of each dimension;
extracting the characteristic information of the clustering cluster of each dimension, and determining the dimension information of the data to be mined according to the characteristic information of the clustering cluster of each dimension;
step three, respectively acquiring process data corresponding to the dimensionality of the data to be mined according to the obtained dimensionality information of the data to be mined; obtaining a big data mining result according to the acquired process data;
the central control module accesses an Internet of things network through an Internet of things communication module by using a network interface to perform Internet of things communication, and a big data mining result and a network data transmission data parallel processing module acquired by utilizing Internet of things equipment are processed;
step five, the data parallel processing module utilizes a parallel processing program to carry out parallel processing on the big data heterogeneous data; acquiring process running information of a tested process of each component acquired by a first execution object deployed in a big data cluster;
step six, scanning whether a program error exists in the tested process or not according to the process running information; if the tested process is scanned to have a program error, scanning an error log of a program error trigger point, and extracting the error type of the program error; if the data is normal, the heterogeneous data is processed in parallel;
step seven, the data parallel processing module carries out type conversion on the data through a database conversion tool, converts the data definition model acquired by mining into the data definition model of each database in the storage device, and carries out data recombination; meanwhile, the central control module controls the internet of things communication module to transmit the converted and recombined data to the big data storage module;
step eight, the big data storage module receives the transmitted data and collects the resources of each server in the server cluster of the storage device;
step nine, determining a component model corresponding to the storage device; partitioning the collected resources according to the determined component model; issuing the determined component model to each server of the storage device, and instructing the server to install each component in the component model;
step ten, receiving data to be warehoused of various data types through a unified data warehousing interface according to the storage equipment; temporarily storing the received data to be put into a warehouse to a multi-stage message queue according to priority attributes; the data needing to be processed preferentially is put into an independent queue, and the data with the same priority is put into the same message queue;
step eleven, performing dequeue operation on the temporary storage data in the message queue through polling service, and storing the data to be put into a database;
step six, counting the big data of the Internet of things stored in the database by using a statistical program through a data counting module; searching big data of the Internet of things by using a searching program through a data searching module; analyzing big data of the Internet of things by using an analysis program through a data analysis module;
and seventhly, displaying the acquired network data, the mining result, the statistical result, the retrieval result and the analysis result by using a display through a display module.
Further, in step three, obtaining a big data mining result according to the obtained process data includes the following steps:
(1) presetting data mining options including data association relations; acquiring process data, and obtaining a plurality of first mining candidate sets according to the acquired process data;
(2) determining a second mining candidate set based on the data association relation contained in the data mining option summary and the determined first mining candidate set;
(3) performing the expansion direction of the second mining candidate set based on the obtained data level of the second mining candidate set and the mining option;
(4) misshaping an expansion of the second mining candidate set based on the determined direction of expansion; and fusing the first mining candidate set and the expanded second mining candidate set to obtain the big data mining result.
Further, the receiving, according to the storage device, data to be warehoused of multiple data types through one unified data warehousing interface specifically includes:
transmitting the data to be put in storage through a uniform Json format, wherein the transmitted type parameter value corresponds to the data type of the data to be put in storage; the type parameter value is used for distinguishing the data to be put in storage in the subsequent operation.
Further, the dequeuing the temporary storage data in the message queue through the polling service, and the storing the data to be put into a database specifically includes:
the polling service is divided into different polling services according to the multi-level queues of the message queues, the queues with different priorities are processed by different polling services, and the dequeuing work of a normal message queue is not influenced under the condition that the data with high priority can be processed preferentially; the polling service stores the data to be put into storage into a corresponding database or a corresponding data table according to the data type of the data to be put into storage; and combing the input data into structured data.
Further, the data to be put in storage is stored in a relational database, a non-relational database and a memory database, and the data is updated and synchronized among the data.
Another object of the present invention is to provide an internet of things-based big data application system implementing the internet of things-based big data application method, including:
the data acquisition module is connected with the central control module and used for acquiring network data through the Internet of things equipment;
the central control module is connected with the data acquisition module, the internet of things communication module, the data parallel processing module, the data mining module, the data statistics module, the data retrieval module, the data analysis module, the big data storage module and the display module and is used for controlling each module to normally work through the main control computer;
the Internet of things communication module is connected with the central control module and is used for accessing an Internet of things network through a network interface to carry out Internet of things communication;
the data parallel processing module is connected with the central control module and is used for carrying out parallel processing on the big data heterogeneous data through a parallel processing program;
the data mining module is connected with the central control module and is used for mining the big data of the Internet of things through a mining program;
the data statistics module is connected with the central control module and used for carrying out statistics on the big data of the Internet of things through a statistical program;
the data retrieval module is connected with the central control module and used for retrieving the big data of the Internet of things through a retrieval program;
the data analysis module is connected with the central control module and used for analyzing the big data of the Internet of things through an analysis program;
the big data storage module is connected with the central control module and used for storing big data through the storage equipment;
and the display module is connected with the central control module and used for displaying the acquired network data, the mining result, the statistical result, the retrieval result and the analysis result through a display.
Further, the storage device is a storage device comprising a plurality of databases.
Further, the storage device further includes:
the storage device comprises a heterogeneous database, a distributed database and an operation and maintenance management database;
the heterogeneous database is used for merging and sharing data information, equipment resources and human resources among different databases;
the distributed database is used for carrying out data parallel processing;
and the operation and maintenance management database is used for performing operation and maintenance management on each database.
It is another object of the present invention to provide a computer program product stored on a computer readable medium, comprising a computer readable program for providing a user input interface to implement the internet of things based big data application method when executed on an electronic device.
Another object of the present invention is to provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to execute the internet-of-things-based big data application method.
The invention has the advantages and positive effects that: the parallel processing problem of heterogeneous big data is solved by adopting a distributed database and a database conversion tool through a data parallel processing module, wherein the database conversion tool converts a data model defined in one database system into a data model in another database; the distributed heterogeneous database system is used for carrying out concurrent processing on large-scale heterogeneous data, so that the parallel processing efficiency of the heterogeneous data is greatly improved; meanwhile, the big data storage module is designed and used for the big data industry, can be independently used, is used as a main means for storing and collecting data, and can be matched with data service for use, so that the big data is applied in practical application, the greater effect of the data is exerted, the rapid storage and storage of mass data are realized, and the data efficiency is improved.
Drawings
Fig. 1 is a flowchart of a big data application method based on the internet of things according to an embodiment of the present invention.
Fig. 2 is a structural block diagram of a big data application system based on the internet of things according to an embodiment of the present invention.
Fig. 3 is a flowchart of a method for mining big data of the internet of things by using a mining program according to an embodiment of the present invention.
Fig. 4 is a flowchart of a method for obtaining a big data mining result according to acquired process data according to an embodiment of the present invention.
FIG. 5 is a flowchart of a method for storing a big data storage module according to an embodiment of the present invention.
In fig. 2: 1. a data acquisition module; 2. a central control module; 3. an Internet of things communication module; 4. a data parallel processing module; 5. a data mining module; 6. a data statistics module; 7. a data retrieval module; 8. a data analysis module; 9. a big data storage module; 10. and a display module.
Detailed Description
In order to further understand the contents, features and effects of the present invention, the following embodiments are illustrated and described in detail with reference to the accompanying drawings.
The structure of the present invention will be described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the big data application method based on the internet of things provided by the embodiment of the invention includes the following steps:
s101, acquiring network data by using the Internet of things equipment through a data acquisition module; mining big data of the Internet of things by a data mining module through a mining program;
s102, the central control module accesses an Internet of things network through the Internet of things communication module by using a network interface to carry out Internet of things communication and transmits the data acquired by mining;
s103, parallel processing is carried out on the big data heterogeneous data through a data parallel processing module by using a parallel processing program;
s104, counting the big data of the Internet of things by using a counting program through a data counting module; searching big data of the Internet of things by using a searching program through a data searching module; analyzing big data of the Internet of things by using an analysis program through a data analysis module;
s105, storing the big data by using a storage device through a big data storage module;
and S106, displaying the acquired network data, the mining result, the statistical result, the retrieval result and the analysis result by using a display through a display module.
As shown in fig. 2, the big data application system based on the internet of things provided by the embodiment of the present invention includes: the system comprises a data acquisition module 1, a central control module 2, an internet of things communication module 3, a data parallel processing module 4, a data mining module 5, a data statistics module 6, a data retrieval module 7, a data analysis module 8, a big data storage module 9 and a display module 10.
The data acquisition module 1 is connected with the central control module 2 and used for acquiring network data through the Internet of things equipment;
the central control module 2 is connected with the data acquisition module 1, the internet of things communication module 3, the data parallel processing module 4, the data mining module 5, the data statistics module 6, the data retrieval module 7, the data analysis module 8, the big data storage module 9 and the display module 10 and is used for controlling each module to normally work through the main control computer;
the Internet of things communication module 3 is connected with the central control module 2 and is used for accessing an Internet of things network through a network interface to carry out Internet of things communication;
the data parallel processing module 4 is connected with the central control module 2 and is used for carrying out parallel processing on the big data heterogeneous data through a parallel processing program;
the data mining module 5 is connected with the central control module 2 and is used for mining the big data of the Internet of things through a mining program;
the data statistics module 6 is connected with the central control module 2 and used for carrying out statistics on the big data of the Internet of things through a statistics program;
the data retrieval module 7 is connected with the central control module 2 and used for retrieving the big data of the Internet of things through a retrieval program;
the data analysis module 8 is connected with the central control module 2 and used for analyzing the big data of the Internet of things through an analysis program;
the big data storage module 9 is connected with the central control module 2 and used for storing big data through storage equipment;
and the display module 10 is connected with the central control module 2 and used for displaying the acquired network data, the mining result, the statistical result, the retrieval result and the analysis result through a display.
The technical solution of the present invention is further illustrated by the following specific examples.
Example 1
The big data application method based on the internet of things provided by the embodiment of the invention is shown in fig. 1, and as a preferred embodiment, as shown in fig. 3, the mining of the big data of the internet of things by using a mining program provided by the embodiment of the invention comprises the following steps:
s201, acquiring big data of multiple dimensions from a big data server of the Internet of things by using a mining program, and clustering all the big data of each dimension to obtain a cluster of each dimension;
s202, extracting the characteristic information of the clustering cluster of each dimension, and determining the dimension information of the data to be mined according to the characteristic information of the clustering cluster of each dimension;
s203, respectively acquiring process data corresponding to the dimensionality of the data to be mined according to the obtained dimensionality information of the data to be mined; and obtaining a big data mining result according to the acquired process data.
As shown in fig. 4, in step S203, obtaining a big data mining result according to the acquired process data according to an embodiment of the present invention includes the following steps:
s301, presetting data mining options including data association relations; acquiring process data, and obtaining a plurality of first mining candidate sets according to the acquired process data;
s302, determining a second mining candidate set based on the data association relation contained in the data mining option summary and the determined first mining candidate set;
s303, performing the extension direction of the second mining candidate set based on the acquired data level of the second mining candidate set and the mining option;
s304, expanding the second mining candidate set based on the determined expansion direction deformity; and fusing the first mining candidate set and the expanded second mining candidate set to obtain the big data mining result.
Example 2
Fig. 1 shows a big data application method based on the internet of things according to an embodiment of the present invention, and as a preferred embodiment, a processing method of a data parallel processing module according to an embodiment of the present invention is as follows:
acquiring process running information of a tested process of each component acquired by a first execution object deployed in a big data cluster; scanning whether a program error exists in the tested process according to the process running information; if the tested process is scanned to have a program error, scanning an error log of a program error trigger point, and extracting the error type of the program error; and if the data is normal, the heterogeneous data is processed in parallel. Performing type conversion through a database conversion tool, accessing a source database system, converting a data definition model of a source database into a data definition model of a target database, and performing data recombination; after data conversion, on one hand, all information needing to be shared in a source database mode is converted into a destination database, and on the other hand, the conversion cannot contain redundant associated information; the heterogeneous database system constructed by adopting the database conversion tool realizes the combination and sharing of data information, equipment resources and human resources among different databases; then, a distributed database system is adopted to solve the problem of data parallel processing; and finally, adopting a multi-database management system to operate and maintain the database system.
The data reorganization provided by the embodiment of the invention is to load the data in the source database system into the destination database.
The database conversion tool provided by the embodiment of the invention is used for converting a data model defined in a database system into a data model in another database and then importing data according to the requirement.
The distributed database system provided by the embodiment of the invention is formed by integrating a plurality of stations, and can be regarded as the union of a series of centralized database systems.
The aggregated stations provided by embodiments of the present invention, also referred to as nodes, are linked together in a communications network, each node being an independent database system having its own database, central processing unit, terminal, and its own local database management system.
Example 3
As shown in fig. 1 and fig. 5, the big data application method based on the internet of things according to the embodiment of the present invention is as follows:
s401, collecting resources of each server in a server cluster of the storage device to be deployed; determining a component model corresponding to the storage device; partitioning the collected resources according to the determined component model; issuing the determined component model to each server, and instructing the server to install each component in the component model to deploy the storage device;
s402, receiving data to be warehoused of various data types through a unified data warehousing interface according to the storage equipment; temporarily storing the received data to be put into a warehouse to a message queue;
and S403, performing dequeue operation on the temporary storage data in the message queue through polling service, and storing the data to be put into a database.
The embodiment of the present invention provides a method for receiving data to be warehoused of multiple data types through a unified data warehousing interface according to a storage device, which specifically includes:
transmitting the data to be put in storage through a uniform Json format, wherein the transmitted type parameter value corresponds to the data type of the data to be put in storage; the type parameter value is used for distinguishing the data to be put in storage in the subsequent operation.
The embodiment of the present invention provides that temporarily storing the received data to be put into storage to a message queue specifically includes:
the data needing priority processing is put into an independent queue according to the priority attribute, and the data with the same priority are put into the same message queue; the message queue is a multi-level queue, and data types are not distinguished in the process of temporarily storing data.
The dequeuing operation of the temporary storage data in the message queue through the polling service provided by the embodiment of the invention, and the storing of the data to be put into a database specifically comprises:
the polling service is divided into different polling services according to the multi-level queues of the message queues, the queues with different priorities are processed by different polling services, and the dequeuing work of a normal message queue is not influenced under the condition that the data with high priority can be processed preferentially; the polling service stores the data to be put into storage into a corresponding database or a corresponding data table according to the data type of the data to be put into storage; and combing the input data into structured data.
The data to be put into a warehouse provided by the embodiment of the invention is stored in a relational database, a non-relational database and a memory database, and the data is updated and synchronized among the data.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When used in whole or in part, can be implemented in a computer program product that includes one or more computer instructions. When loaded or executed on a computer, cause the flow or functions according to embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL), or wireless (e.g., infrared, wireless, microwave, etc.)). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (ssd)), among others.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the present invention in any way, and all simple modifications, equivalent changes and modifications made to the above embodiment according to the technical spirit of the present invention are within the scope of the technical solution of the present invention.
Claims (10)
1. The big data application method based on the Internet of things is characterized by comprising the following steps:
acquiring network data by using Internet of things equipment through a data acquisition module; acquiring big data of multiple dimensions from a big data server of the Internet of things by using a mining program through a data mining module, and clustering all the big data of each dimension to obtain a cluster of each dimension;
extracting the characteristic information of the clustering cluster of each dimension, and determining the dimension information of the data to be mined according to the characteristic information of the clustering cluster of each dimension;
step three, respectively acquiring process data corresponding to the dimensionality of the data to be mined according to the obtained dimensionality information of the data to be mined; obtaining a big data mining result according to the acquired process data;
the central control module accesses an Internet of things network through an Internet of things communication module by using a network interface to perform Internet of things communication, and a big data mining result and a network data transmission data parallel processing module acquired by utilizing Internet of things equipment are processed;
step five, the data parallel processing module utilizes a parallel processing program to carry out parallel processing on the big data heterogeneous data; acquiring process running information of a tested process of each component acquired by a first execution object deployed in a big data cluster;
step six, scanning whether a program error exists in the tested process or not according to the process running information; if the tested process is scanned to have a program error, scanning an error log of a program error trigger point, and extracting the error type of the program error; if the data is normal, the heterogeneous data is processed in parallel;
step seven, the data parallel processing module carries out type conversion on the data through a database conversion tool, converts the data definition model acquired by mining into the data definition model of each database in the storage device, and carries out data recombination; meanwhile, the central control module controls the internet of things communication module to transmit the converted and recombined data to the big data storage module;
step eight, the big data storage module receives the transmitted data and collects the resources of each server in the server cluster of the storage device;
step nine, determining a component model corresponding to the storage device; partitioning the collected resources according to the determined component model; issuing the determined component model to each server of the storage device, and instructing the server to install each component in the component model;
step ten, receiving data to be warehoused of various data types through a unified data warehousing interface according to the storage equipment; temporarily storing the received data to be put into a warehouse to a multi-stage message queue according to priority attributes; the data needing to be processed preferentially is put into an independent queue, and the data with the same priority is put into the same message queue;
step eleven, performing dequeue operation on the temporary storage data in the message queue through polling service, and storing the data to be put into a database;
step six, counting the big data of the Internet of things stored in the database by using a statistical program through a data counting module; searching big data of the Internet of things by using a searching program through a data searching module; analyzing big data of the Internet of things by using an analysis program through a data analysis module;
and seventhly, displaying the acquired network data, the mining result, the statistical result, the retrieval result and the analysis result by using a display through a display module.
2. The Internet of things-based big data application method of claim 1, wherein in the third step, the obtaining of the big data mining result according to the obtained process data comprises the following steps:
(1) presetting data mining options including data association relations; acquiring process data, and obtaining a plurality of first mining candidate sets according to the acquired process data;
(2) determining a second mining candidate set based on the data association relation contained in the data mining option summary and the determined first mining candidate set;
(3) performing the expansion direction of the second mining candidate set based on the obtained data level of the second mining candidate set and the mining option;
(4) misshaping an expansion of the second mining candidate set based on the determined direction of expansion; and fusing the first mining candidate set and the expanded second mining candidate set to obtain the big data mining result.
3. The internet-of-things-based big data application method of claim 1, wherein the receiving, according to the storage device, data to be warehoused of multiple data types through a unified data warehousing interface specifically comprises:
transmitting the data to be put in storage through a uniform Json format, wherein the transmitted type parameter value corresponds to the data type of the data to be put in storage; the type parameter value is used for distinguishing the data to be put in storage in the subsequent operation.
4. The internet-of-things-based big data application method of claim 1, wherein the dequeuing the temporary storage data in the message queue through the polling service, and the storing the data to be put into storage into the database specifically comprises:
the polling service is divided into different polling services according to the multi-level queues of the message queues, the queues with different priorities are processed by different polling services, and the dequeuing work of a normal message queue is not influenced under the condition that the data with high priority can be processed preferentially; the polling service stores the data to be put into storage into a corresponding database or a corresponding data table according to the data type of the data to be put into storage; and combing the input data into structured data.
5. The big data application method based on the internet of things as claimed in claim 1, wherein the data to be put in storage is stored in a relational database, a non-relational database and an in-memory database, and data updating and synchronization are kept among the data.
6. An internet of things-based big data application system for implementing the internet of things-based big data application method according to claims 1-5, wherein the internet of things-based big data application system comprises:
the data acquisition module is connected with the central control module and used for acquiring network data through the Internet of things equipment;
the central control module is connected with the data acquisition module, the internet of things communication module, the data parallel processing module, the data mining module, the data statistics module, the data retrieval module, the data analysis module, the big data storage module and the display module and is used for controlling each module to normally work through the main control computer;
the Internet of things communication module is connected with the central control module and is used for accessing an Internet of things network through a network interface to carry out Internet of things communication;
the data parallel processing module is connected with the central control module and is used for carrying out parallel processing on the big data heterogeneous data through a parallel processing program;
the data mining module is connected with the central control module and is used for mining the big data of the Internet of things through a mining program;
the data statistics module is connected with the central control module and used for carrying out statistics on the big data of the Internet of things through a statistical program;
the data retrieval module is connected with the central control module and used for retrieving the big data of the Internet of things through a retrieval program;
the data analysis module is connected with the central control module and used for analyzing the big data of the Internet of things through an analysis program;
the big data storage module is connected with the central control module and used for storing big data through the storage equipment;
and the display module is connected with the central control module and used for displaying the acquired network data, the mining result, the statistical result, the retrieval result and the analysis result through a display.
7. The internet-of-things-based big data application system of claim 6, wherein the storage device is a storage device comprising a plurality of databases.
8. The internet-of-things-based big data application system of claim 7, wherein the storage device further comprises:
the storage device comprises a heterogeneous database, a distributed database and an operation and maintenance management database;
the heterogeneous database is used for merging and sharing data information, equipment resources and human resources among different databases;
the distributed database is used for carrying out data parallel processing;
and the operation and maintenance management database is used for performing operation and maintenance management on each database.
9. A computer program product stored on a computer readable medium, comprising a computer readable program for providing a user input interface to implement the internet of things based big data application method of any of claims 1-5 when executed on an electronic device.
10. A computer-readable storage medium storing instructions which, when executed on a computer, cause the computer to perform the internet-of-things-based big data application method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010776653.4A CN111858722A (en) | 2020-08-05 | 2020-08-05 | Big data application system and method based on Internet of things |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010776653.4A CN111858722A (en) | 2020-08-05 | 2020-08-05 | Big data application system and method based on Internet of things |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111858722A true CN111858722A (en) | 2020-10-30 |
Family
ID=72971118
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010776653.4A Withdrawn CN111858722A (en) | 2020-08-05 | 2020-08-05 | Big data application system and method based on Internet of things |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858722A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112800133A (en) * | 2021-01-22 | 2021-05-14 | 平安养老保险股份有限公司 | Product data processing method, device, equipment and medium based on database direct connection |
| CN113298686A (en) * | 2021-05-18 | 2021-08-24 | 深圳市博网科技有限公司 | Big data application system and method based on Internet of things |
| CN117076612A (en) * | 2023-08-31 | 2023-11-17 | 宁夏恒信创达数据科技有限公司 | Call center big data text mining system |
-
2020
- 2020-08-05 CN CN202010776653.4A patent/CN111858722A/en not_active Withdrawn
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112800133A (en) * | 2021-01-22 | 2021-05-14 | 平安养老保险股份有限公司 | Product data processing method, device, equipment and medium based on database direct connection |
| CN113298686A (en) * | 2021-05-18 | 2021-08-24 | 深圳市博网科技有限公司 | Big data application system and method based on Internet of things |
| CN117076612A (en) * | 2023-08-31 | 2023-11-17 | 宁夏恒信创达数据科技有限公司 | Call center big data text mining system |
| CN117076612B (en) * | 2023-08-31 | 2024-02-20 | 宁夏恒信创达数据科技有限公司 | Call center big data text mining system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11727039B2 (en) | Low-latency streaming analytics | |
| US7606804B2 (en) | System and method for information management in a distributed network | |
| CN111858722A (en) | Big data application system and method based on Internet of things | |
| Srivastava et al. | Operator placement for in-network stream query processing | |
| CN107515878B (en) | Data index management method and device | |
| CN111400393B (en) | Data processing method and device based on multi-application platform and storage medium | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| CN110955704A (en) | Data management method, device, equipment and storage medium | |
| JP2021111352A (en) | Computer program, data archive method, recording medium, and computer apparatus | |
| KR102028342B1 (en) | System and method for supporting real-time financial business provision and decision making using data process solution | |
| CN111382155A (en) | Data processing method of data warehouse, electronic equipment and medium | |
| CN114461792A (en) | Alarm event correlation method, device, electronic equipment, medium and program product | |
| CN115344207A (en) | Data processing method and device, electronic equipment and storage medium | |
| KR101686919B1 (en) | Method and apparatus for managing inference engine based on big data | |
| CN111125450A (en) | Management method of multilayer topology network resource object | |
| CN113836237A (en) | Method and device for auditing data operation of database | |
| Adzic et al. | Data warehouse population platform | |
| CN109257457B (en) | IDC information security system state monitoring method based on data analysis | |
| EP4216076A1 (en) | Method and apparatus of processing an observation information, electronic device and storage medium | |
| CN111292201A (en) | Method for pushing field operation and maintenance information of power communication network based on Apriori and RETE | |
| CN111161818A (en) | Medical data exchange sharing system and method based on big data technology | |
| CN103377286A (en) | Database history management method and system thereof | |
| CN105162837A (en) | Method and system for improving I/O throughput rate in massive data storage environment | |
| CN111368039B (en) | Data management system | |
| CN112882837A (en) | Mobile network resource management system based on cloud computing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20201030 |