CN110855768A - Data processing method and WiFi-IoT system - Google Patents
Data processing method and WiFi-IoT system Download PDFInfo
- Publication number
- CN110855768A CN110855768A CN201911080785.7A CN201911080785A CN110855768A CN 110855768 A CN110855768 A CN 110855768A CN 201911080785 A CN201911080785 A CN 201911080785A CN 110855768 A CN110855768 A CN 110855768A
- Authority
- CN
- China
- Prior art keywords
- data
- task
- analysis
- wifi
- data acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a data processing method and a WiFi-IoT system, wherein the WiFi-IoT system comprises a data collector, a message queue, a stream frame Streaming and a data warehouse, and the data processing method comprises the following steps: the data acquisition unit receives target data sent by the network equipment and sends the target data to the message queue; the stream framework Streaming acquires target data from the message queue, performs data conversion and preliminary analysis on the target data, and stores the analysis result in a data warehouse. The invention realizes the effective transmission of data between the WiFi-IoT system and the network equipment, and the WiFi-IoT system can provide powerful data analysis support for a third party.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and a WiFi-IoT system.
Background
In WiFi and internet of things systems, it is very important to integrate third party data and provide data support for the third party.
The traditional data integration mode is to define an REST API interface, and then access the REST API interface through GET, PUT, POST and DELET modes to realize the increasing, deleting, modifying and checking of third-party data.
However, the data volume in WiFi and internet of things systems is generally very large, and data transmission through the REST API interface is difficult due to the limit of the working mechanism of the REST API interface itself. Meanwhile, the third-party data is generally stored in a special data system and is difficult to convert into a specified format required by a REST API (representational state transfer application program interface). Therefore, integration of third-party data and provision of data support for a third party are difficult to achieve through the REST API interface.
Disclosure of Invention
In view of this, the present invention provides a data processing method and a WiFi-IoT system, which implement effective transmission of data between the WiFi-IoT system and a network device, and the WiFi-IoT system can provide powerful data analysis support for a third party.
In order to achieve the above purpose, the invention provides the following specific technical scheme:
a data processing method is applied to a WiFi-IoT system, wherein the WiFi-IoT system comprises a data collector, a message queue, a stream framework Streaming and a data warehouse, and the method comprises the following steps:
the data acquisition unit receives target data sent by network equipment and sends the target data to the message queue;
and the stream framework Streaming acquires the target data from the message queue, performs data conversion and preliminary analysis on the target data, and stores the analysis result in the data warehouse.
Optionally, the WiFi-IoT system further includes a browser, a data collection task generator, a data collection task manager, and a data collection database, and the method further includes:
after receiving a selection instruction of a user, the browser sends a task script and task attributes corresponding to the selection instruction to the data acquisition task generator;
the data acquisition task generator generates a data acquisition task according to the task script and the task attribute and sends the data acquisition task to the data acquisition task manager;
and the data acquisition task manager stores the relevant data of the data acquisition task in the data acquisition database.
Optionally, the WiFi-IoT system further includes an analysis module, and the method further includes:
and the data acquisition task generator generates a data analysis task and sends the data analysis task to the analysis module.
Optionally, the method further includes:
the analysis module acquires original data from the data warehouse or a third-party database according to the data analysis task;
and the analysis module performs data analysis on the original data according to the data analysis task and stores a final analysis result in the data acquisition database.
Optionally, the method further includes:
and the analysis module writes intermediate data generated in the process of performing data analysis on the original data into the data warehouse or the third-party database according to the data analysis task.
A WiFi-IoT system comprises a data collector, a message queue, a stream framework Streaming and a data warehouse;
the data acquisition unit is used for receiving target data sent by network equipment and sending the target data to the message queue;
the stream framework Streaming is used for acquiring the target data from the message queue, performing data conversion and preliminary analysis on the target data, and storing an analysis result in the data warehouse.
Optionally, the WiFi-IoT system further includes a browser, a data collection task generator, a data collection task manager, and a data collection database;
the browser is used for sending a task script and task attributes corresponding to a selection instruction to the data acquisition task generator after receiving the selection instruction of a user;
the data acquisition task generator is used for generating a data acquisition task according to the task script and the task attribute and sending the data acquisition task to the data acquisition task manager;
and the data acquisition task manager is used for storing the relevant data of the data acquisition task in the data acquisition database.
Optionally, the WiFi-IoT system further comprises an analysis module;
the data acquisition task generator is also used for generating a data analysis task and sending the data analysis task to the analysis module.
Optionally, the analysis module is configured to obtain original data from the data warehouse or a third-party database according to the data analysis task;
and the analysis module performs data analysis on the original data according to the data analysis task and stores a final analysis result in the data acquisition database.
Optionally, the analysis module is further configured to write intermediate data generated in the process of performing data analysis on the raw data into the data warehouse or the third-party database according to the data analysis task.
Compared with the prior art, the invention has the following beneficial effects:
the invention discloses a data processing method which is applied to a WiFi-IoT system, wherein the WiFi-IoT system comprises a data collector, a message queue, a stream frame Streaming and a data warehouse, the data collector can receive target data sent by network equipment, effective data transmission between the WiFi-IoT system and the network equipment is realized, the stream frame Streaming carries out data conversion and preliminary analysis on the target data by acquiring the target data from the message queue, the data conversion and analysis of the data sent by the network equipment by the WiFi-IoT system are realized, and powerful data support is provided for a third party.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic flow chart illustrating a data processing method according to an embodiment of the present invention;
FIG. 2 is an exemplary diagram of a Kafka message queue disclosed by an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a WiFi-IoT system according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a WiFi-IoT system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment discloses a data processing method, which is applied to a WiFi-IoT system, where the WiFi-IoT system is a data processing system of WiFi and IoT (Internet of things, chinese name: Internet of things), and the WiFi-IoT system includes a data collector, a message queue, a stream frame Streaming and a data warehouse, please refer to fig. 1, and the data processing method specifically includes the following steps:
s101: the data acquisition unit receives target data sent by the network equipment and sends the target data to the message queue;
the network device may be an AP (Access Point, chinese name: wireless Access Point) device, an AC (Access Point Controller, chinese name: Access Controller) device, an IoT device, a third party device, or the like.
The format of the target data sent by the network device to the data collector may be GPB, fluent, etc., and the following is a simple example of the GPB format:
the GPB is data which is subjected to coding compression when the data is sent, and the data transmission efficiency can be improved.
The message queue may be a Kafka message queue, please refer to fig. 2, and fig. 2 is a simple example of the Kafka message queue, where AP data and AC data are stored in different queues respectively for storage in partitions, and each partition corresponds to one consumer.
S102: the stream framework Streaming acquires target data from the message queue, performs data conversion and preliminary analysis on the target data, and stores the analysis result in a data warehouse.
Stream framework streaming (spark) acquires target data of each network device from the message queue, performs data conversion on the target data to obtain effective data, and performs preliminary analysis on the effective data, wherein the following are original target data of a single network device and a result part obtained after performing data conversion and preliminary analysis on the original target data:
raw object data
| STAtoAP_rx_data_bytes | Traffic sent by terminal | 921600B |
| STAtoAP_rx_data_pkts | Number of data packets transmitted by terminal | 4728 |
| APtoSTA_tx_data_bytes | Traffic received by a terminal | 460800B |
| APtoSTA_tx_data_pkts | Number of data packets received by terminal | 4825 |
| STAtoAP_rx_data_time | Total time of terminal transmission | 100s |
| APtoSTA_tx_data_time | Total time of terminal reception | 200s |
Analysis results section
| STAtoAP_rx_data_bytes | Rate of terminal transmission | 9216B/s |
| APtoSTA_tx_data_bytes | Rate of terminal reception | 2304B/s |
And the stream framework (spark) saves the analysis result to a data warehouse, so that the subsequent use and the tracing are convenient.
Therefore, the data processing method disclosed in this embodiment realizes effective data transmission between the WiFi-IoT system and the network device, and the stream framework Streaming performs data conversion and preliminary analysis on the target data by acquiring the target data from the message queue, so as to realize data conversion and analysis of the data sent by the network device by the WiFi-IoT system, and provide powerful data support for a third party.
On the basis that the WiFi-IoT system further includes a browser, a data acquisition task generator, a data acquisition task manager, a data acquisition database, and an analysis module, the data processing method disclosed in this embodiment further has a function of generating a data acquisition task and a data analysis task.
After receiving a selection instruction of a user, the browser sends a task script and task attributes corresponding to the selection instruction to the data acquisition task generator, wherein the task script describes a task execution process, including where data comes from, what processing is to be performed, what data is output, where data is output, namely a data source, a data processing process, a processing result and a processing result destination. The task attribute is used for managing the task, such as task execution time, task parameters (such as address parameters), task failure retry times and the like.
The data acquisition task generator generates a data acquisition task according to the task script and the task attribute and sends the data acquisition task to the data acquisition task manager;
and the data acquisition task manager stores the relevant data of the data acquisition task in the data acquisition database, and triggers a task execution script to execute the data acquisition task when the data acquisition task meets the task execution time.
And the data acquisition task generator generates a data analysis task and sends the data analysis task to the analysis module.
The analysis module acquires original data from the data warehouse or a third-party database according to the data analysis task;
and the analysis module performs data analysis on the original data according to the data analysis task and stores a final analysis result in the data acquisition database.
The third-party database can be a relational database or a non-relational database available in the market, such as an HDFS, and the integrated use of the third-party database by the WiFi-IoT system is realized.
In the process that the analysis module carries out data analysis on the original data according to the data analysis task, intermediate data can be generated, the intermediate data have certain storage value, and the intermediate data can be written into a data warehouse or a third-party database for subsequent use and tracing.
Therefore, the data processing method disclosed in this embodiment can also realize the generation, execution and management of data acquisition tasks and the generation, execution and management of data analysis tasks, and a user can export data in third parties, Wifi and IoT to the system, perform data conversion and preliminary analysis on the system, and store analysis result data without passing through an REST interface.
Based on the data processing method disclosed in the foregoing embodiment, the present embodiment correspondingly discloses a WiFi-IoT system, please refer to fig. 3, the WiFi-IoT system includes a data collector 100, a message queue 200, a stream frame Streaming300, and a data warehouse 400;
the data collector 100 is configured to receive target data sent by a network device, and send the target data to the message queue 200;
the stream framework Streaming300 is configured to obtain the target data from the message queue 200, perform data conversion and preliminary analysis on the target data, and store an analysis result in the data warehouse 400.
Referring to fig. 4, the WiFi-IoT system disclosed in this embodiment further includes a browser, a data collection task generator, a data collection task manager, a data collection database, and an analysis module;
the browser is used for sending a task script and task attributes corresponding to a selection instruction to the data collection task generator after receiving the selection instruction of a user;
the data acquisition task generator is used for generating a data acquisition task according to the task script and the task attribute and sending the data acquisition task to the data acquisition task manager;
and the data acquisition task manager is used for storing the relevant data of the data acquisition task in the data acquisition database.
Optionally, the WiFi-IoT system further comprises an analysis module;
the data acquisition task generator is also used for generating a data analysis task and sending the data analysis task to the analysis module.
Optionally, the analysis module is configured to obtain original data from the data warehouse or a third-party database according to the data analysis task;
and the analysis module performs data analysis on the original data according to the data analysis task and stores a final analysis result in the data acquisition database.
Optionally, the analysis module is further configured to write intermediate data generated in the process of performing data analysis on the raw data into the data warehouse or the third-party database according to the data analysis task.
The WiFi-IoT system disclosed in this embodiment includes a data collector, a message queue, a stream frame Streaming and a data warehouse, where the data collector may receive target data sent by a network device, and implement effective data transmission between the WiFi-IoT system and the network device, and the stream frame Streaming performs data conversion and preliminary analysis on the target data by acquiring the target data from the message queue, so as to implement data conversion and analysis of data sent by the network device by the WiFi-IoT system, and provide powerful data support for a third party. The data acquisition system can also realize the generation, execution and management of data acquisition tasks and the generation, execution and management of data analysis tasks, and a user can export data in third parties, Wifi and IoT to the system, perform data conversion and preliminary analysis on the data and store analysis result data without an REST interface.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A data processing method is applied to a WiFi-IoT system, wherein the WiFi-IoT system comprises a data collector, a message queue, a stream framework Streaming and a data warehouse, and the method comprises the following steps:
the data acquisition unit receives target data sent by network equipment and sends the target data to the message queue;
and the stream framework Streaming acquires the target data from the message queue, performs data conversion and preliminary analysis on the target data, and stores the analysis result in the data warehouse.
2. The method of claim 1, wherein the WiFi-IoT system further comprises a browser, a data collection task generator, a data collection task manager, and a data collection database, the method further comprising:
after receiving a selection instruction of a user, the browser sends a task script and task attributes corresponding to the selection instruction to the data acquisition task generator;
the data acquisition task generator generates a data acquisition task according to the task script and the task attribute and sends the data acquisition task to the data acquisition task manager;
and the data acquisition task manager stores the relevant data of the data acquisition task in the data acquisition database.
3. The method of claim 2, wherein the WiFi-IoT system further comprises an analytics module, the method further comprising:
and the data acquisition task generator generates a data analysis task and sends the data analysis task to the analysis module.
4. The method of claim 3, further comprising:
the analysis module acquires original data from the data warehouse or a third-party database according to the data analysis task;
and the analysis module performs data analysis on the original data according to the data analysis task and stores a final analysis result in the data acquisition database.
5. The method of claim 4, further comprising:
and the analysis module writes intermediate data generated in the process of performing data analysis on the original data into the data warehouse or the third-party database according to the data analysis task.
6. A WiFi-IoT system is characterized by comprising a data collector, a message queue, a stream framework Streaming and a data warehouse;
the data acquisition unit is used for receiving target data sent by network equipment and sending the target data to the message queue;
the stream framework Streaming is used for acquiring the target data from the message queue, performing data conversion and preliminary analysis on the target data, and storing an analysis result in the data warehouse.
7. The WiFi-IoT system in accordance with claim 6, wherein the WiFi-IoT system further comprises a browser, a data collection task generator, a data collection task manager, and a data collection database;
the browser is used for sending a task script and task attributes corresponding to a selection instruction to the data acquisition task generator after receiving the selection instruction of a user;
the data acquisition task generator is used for generating a data acquisition task according to the task script and the task attribute and sending the data acquisition task to the data acquisition task manager;
and the data acquisition task manager is used for storing the relevant data of the data acquisition task in the data acquisition database.
8. The WiFi-IoT system in accordance with claim 7, wherein the WiFi-IoT system further comprises an analytics module;
the data acquisition task generator is also used for generating a data analysis task and sending the data analysis task to the analysis module.
9. The WiFi-IoT system in accordance with claim 8, wherein the analysis module is configured to obtain raw data from the data warehouse or a third party database in accordance with the data analysis task;
and the analysis module performs data analysis on the original data according to the data analysis task and stores a final analysis result in the data acquisition database.
10. The WiFi-IoT system according to claim 9, wherein the analysis module is further configured to write intermediate data generated during data analysis on the raw data into the data warehouse or the third party database according to the data analysis task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911080785.7A CN110855768A (en) | 2019-11-07 | 2019-11-07 | Data processing method and WiFi-IoT system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911080785.7A CN110855768A (en) | 2019-11-07 | 2019-11-07 | Data processing method and WiFi-IoT system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110855768A true CN110855768A (en) | 2020-02-28 |
Family
ID=69598392
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911080785.7A Pending CN110855768A (en) | 2019-11-07 | 2019-11-07 | Data processing method and WiFi-IoT system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110855768A (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109379245A (en) * | 2018-09-21 | 2019-02-22 | 成都西加云杉科技有限公司 | A kind of wifi report form generation method and system |
| WO2019095667A1 (en) * | 2017-11-15 | 2019-05-23 | 平安科技（深圳）有限公司 | Database data collection method, application server, and computer readable storage medium |
-
2019
- 2019-11-07 CN CN201911080785.7A patent/CN110855768A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019095667A1 (en) * | 2017-11-15 | 2019-05-23 | 平安科技（深圳）有限公司 | Database data collection method, application server, and computer readable storage medium |
| CN109379245A (en) * | 2018-09-21 | 2019-02-22 | 成都西加云杉科技有限公司 | A kind of wifi report form generation method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104038908B (en) | Push message sending method and device | |
| CN102737065B (en) | Method and device for acquiring data | |
| CN103095684B (en) | Obtain and pay close attention to the method for user's aggregation information, Apparatus and system | |
| US10055134B2 (en) | Data compression method and storage system | |
| CN110620699B (en) | Message arrival rate determination method, device, equipment and computer readable storage medium | |
| CN103974057A (en) | Video quality user experience value evaluation method, device and system | |
| CN108153803A (en) | A kind of data capture method, device and electronic equipment | |
| CN105988941B (en) | Cache data processing method and device | |
| CN110209674A (en) | A kind of the cloud database statistical method and device of industrial environment dust control wechat small routine | |
| CN107562426B (en) | Method and system for collecting and analyzing Trace of browser in non-embedded cloud | |
| CN114741261A (en) | Distributed link tracking method, device, equipment and storage medium | |
| US20190370293A1 (en) | Method and apparatus for processing information | |
| JP7509886B2 (en) | Method and apparatus for pushing subscription data in the internet of things, and devices and storage media thereof | |
| CN105337850A (en) | Internet of Things data processing method and internet of Things gateway | |
| WO2021147319A1 (en) | Data processing method, apparatus, device, and medium | |
| CN104168174A (en) | Method and apparatus for information transmission | |
| WO2015021753A1 (en) | Data compression method, computer system and device | |
| CN103068052A (en) | Dynamic configuration method and system of resources and portal server | |
| CN110855768A (en) | Data processing method and WiFi-IoT system | |
| WO2016062097A1 (en) | Iptv system-based information processing method and device | |
| CN110933070A (en) | User identification method, system, equipment and computer readable storage medium | |
| US20230342369A1 (en) | Data processing method and apparatus, and electronic device and storage medium | |
| US11159989B2 (en) | Data caching method and storage medium | |
| CN105337778A (en) | Internet of Things event-driven service management system | |
| CN105897566A (en) | User acquisition method and system for network course |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200228 |
|
| RJ01 | Rejection of invention patent application after publication |