CN114996363A - Data remote processing method, system, device and electronic equipment - Google Patents
Data remote processing method, system, device and electronic equipment Download PDFInfo
- Publication number
- CN114996363A CN114996363A CN202210941818.8A CN202210941818A CN114996363A CN 114996363 A CN114996363 A CN 114996363A CN 202210941818 A CN202210941818 A CN 202210941818A CN 114996363 A CN114996363 A CN 114996363A
- Authority
- CN
- China
- Prior art keywords
- target
- remote
- data
- analysis
- servers
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 19
- 238000004458 analytical method Methods 0.000 claims abstract description 162
- 238000012545 processing Methods 0.000 claims abstract description 84
- 238000012544 monitoring process Methods 0.000 claims abstract description 67
- 238000000034 method Methods 0.000 claims abstract description 60
- 238000005516 engineering process Methods 0.000 claims abstract description 33
- 238000004891 communication Methods 0.000 claims abstract description 25
- 238000003860 storage Methods 0.000 claims description 22
- 238000009826 distribution Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 description 21
- 238000010586 diagram Methods 0.000 description 15
- 238000004590 computer program Methods 0.000 description 14
- 230000005540 biological transmission Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 230000002159 abnormal effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000007717 exclusion Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- GJWAPAVRQYYSTK-UHFFFAOYSA-N [(dimethyl-$l^{3}-silanyl)amino]-dimethylsilicon Chemical compound C[Si](C)N[Si](C)C GJWAPAVRQYYSTK-UHFFFAOYSA-N 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000002618 waking effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W28/00—Network traffic management; Network resource management
- H04W28/02—Traffic management, e.g. flow control or congestion control
- H04W28/08—Load balancing or load distribution
- H04W28/09—Management thereof
- H04W28/0992—Management thereof based on the type of application
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a data remote processing method, a system, a device and electronic equipment. Wherein, the method comprises the following steps: acquiring a target file based on a fifth generation communication technology 5G, wherein the target file stores monitoring data acquired by local monitoring terminal equipment; determining to open the remote analysis servers of the target number according to the number of the target files; equally distributing the target files to the remote analysis servers with the target number; adopting the target number of remote analysis servers to analyze the target file to obtain a target analysis result; and storing the target analysis result to a first database. The invention solves the technical problems of low data processing efficiency, poor real-time performance and high processing cost of the data remote processing method in the related technology.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a method, a system, an apparatus, and an electronic device for remote processing of data.
Background
Before the fifth generation communication technology 5G is put into practical use, a method of locally processing a large amount of high-frequency and high-precision data at a monitoring site and storing the processed data in a remote-end database is generally adopted for a large amount of high-frequency and high-precision data due to limited data transmission capability. The method of local processing requires the allocation of a relevant processing server to each monitoring site, and the processing capacity of a single monitoring site is limited, increasing the overall cost of allocating processing servers. In addition, the limited data transmission capability also causes a problem that real-time data processing cannot be realized.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a data remote processing method, a system, a device and electronic equipment, which are used for at least solving the technical problems of low data processing efficiency, poor real-time performance and high processing cost of the data remote processing method in the related technology.
According to an aspect of an embodiment of the present invention, there is provided a data remote processing method, including: acquiring a target file based on a fifth generation communication technology 5G, wherein the target file stores monitoring data acquired by local monitoring terminal equipment; determining to start the remote analysis servers of the target number according to the number of the target files; distributing the target files to the target number of remote analysis servers in an average manner; adopting the remote analysis servers with the target number to analyze the target file to obtain a target analysis result; and storing the target analysis result to a first database.
Optionally, the method further comprises: acquiring the current utilization rate of the remote analysis server; judging whether the current utilization rate reaches a preset utilization rate or not; and if the current utilization rate reaches the preset utilization rate, determining to increase the running number of the remote analysis servers.
Optionally, after the current utilization reaches the preset utilization, the method further includes: obtaining the duration of the remote analysis server reaching the preset utilization rate; judging whether the duration reaches a preset time or not; and if the duration reaches the preset time, determining to increase the running number of the remote analysis servers.
Optionally, the analyzing the target file by using the target number of remote analysis servers to obtain a target analysis result includes: performing first analysis processing on the target file to obtain a first analysis result; and calculating the number of the target files, and performing second analysis processing on the plurality of first analysis results after the target files reach a preset number to obtain the target analysis results.
Optionally, the method further comprises: determining waveform data corresponding to the target file; and storing the waveform data to a second database, and displaying the waveform data in real time.
According to another aspect of the embodiments of the present invention, there is also provided a data remote processing system, including: the load balancing server is used for acquiring a target file based on a fifth generation communication technology 5G, wherein the target file stores monitoring data acquired by local monitoring terminal equipment; determining to start the remote analysis servers of the target number according to the number of the target files; distributing the target files to the target number of remote analysis servers on average; the remote analysis server is connected with the load balancing server and used for analyzing the target file to obtain a target analysis result; and the database server is connected with the remote analysis server and used for storing the target analysis result to a first database.
According to another aspect of the embodiments of the present invention, there is also provided a data remote processing apparatus, including: the acquisition module is used for acquiring a target file based on a fifth generation communication technology 5G, wherein the target file stores monitoring data acquired by local monitoring terminal equipment; the first determining module is used for determining to start the remote analysis servers of the target number according to the number of the target files; the distribution module is used for distributing the target files to the remote analysis servers with the target number in an average manner; the analysis module is used for adopting the remote analysis servers with the target number to analyze the target file to obtain a target analysis result; and the first storage module is used for storing the target analysis result to a first database.
Optionally, the apparatus further comprises: the second determining module is used for determining waveform data corresponding to the target file; the second storage module is used for storing the waveform data to a second database; and the display module is used for displaying the waveform data in real time.
According to still another aspect of the embodiments of the present invention, there is also provided a non-volatile storage medium storing a plurality of instructions, the instructions being adapted to be loaded by a processor and to perform any one of the data remote processing methods.
According to still another aspect of the embodiments of the present invention, there is also provided an electronic device, including: one or more processors and memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement any of the data remote processing methods.
In the embodiment of the invention, a data remote processing mode is adopted, and a target file is obtained based on a fifth generation communication technology 5G, wherein the target file stores monitoring data obtained by local monitoring terminal equipment; determining to start the remote analysis servers of the target number according to the number of the target files; distributing the target files to the target number of remote analysis servers in an average manner; adopting the remote analysis servers with the target number to analyze the target file to obtain a target analysis result; and storing the target analysis result to a first database. The purposes of realizing data transmission by using a fifth generation communication technology 5G and allocating and optimizing the utilization rate of a related data processing server are achieved, the technical effects of improving the data remote processing efficiency and reducing the data processing cost are achieved, and the technical problems of low data processing efficiency, poor real-time performance and high processing cost of a data remote processing method in the related technology are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a flow chart of a method for remote processing of data provided in accordance with an embodiment of the invention;
FIG. 2 is a block diagram of a data remote processing system according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data remote processing system according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a data remote processing device according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In accordance with an embodiment of the present invention, there is provided a method embodiment of a method for remote processing of data, it being noted that the steps illustrated in the flowchart of the figure may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than that presented herein.
Fig. 1 is a data remote processing method according to an embodiment of the present invention, as shown in fig. 1, the method including the steps of:
step S102, acquiring a target file based on a fifth generation communication technology 5G, wherein the target file stores monitoring data acquired by local monitoring terminal equipment;
step S104, determining to start the remote analysis servers of the target number according to the number of the target files;
step S106, the target files are evenly distributed to the remote analysis servers with the target number;
step S108, adopting the target number of remote analysis servers to analyze the target file to obtain a target analysis result;
step S110, storing the target analysis result in a first database.
Through the steps, the purposes of realizing data transmission and allocating and optimizing the utilization rate of the related data processing server by utilizing the fifth generation communication technology 5G can be achieved, the technical effects of improving the data remote processing efficiency and reducing the data processing cost are achieved, and the technical problems of low data processing efficiency, poor real-time performance and high processing cost of the data remote processing method in the related technology are solved.
In the data remote processing method provided by the embodiment of the invention, firstly, a target file for storing the monitoring data acquired by the local monitoring terminal device is obtained based on the transmission capability of the fifth generation communication technology 5G. And determining the number of the opening targets and the remote analysis servers of the opening targets according to the number of the obtained target files for evenly distributing and processing analysis resources. And averagely obtaining the target file by the target number of remote analysis servers, so that a plurality of started remote analysis servers are in a similar running state. And analyzing and processing the target file by the target number of remote analysis servers to obtain a target analysis result, and storing the target analysis result in a first database. The term local is understood to mean that the data source monitored by the local monitoring terminal is in the same place, so as to distinguish the data source from the remote processing place.
Alternatively, the first database may be a relational database, for example, MySQL database, which is a name of a widely used relational database.
Optionally, the remote resolution servers with the target number are turned on, for example, a part of the remote resolution servers with the target number are turned on, and a part of the remote resolution servers with the target number are in a dormant state. And determining to start the remote analysis servers of the target number according to the number of the target files, so that the remote analysis servers can be flexibly scheduled, and the data processing capacity is improved.
Optionally, the target files are equally distributed to the target number of remote parsing servers, the average distribution formula may be multiple, for example, each remote parsing server is configured to process files transmitted by 10 local monitoring terminals, where the corresponding mode is based on the number of the local monitoring terminal, and the remote parsing server obtains the target files. The number of the 10 local monitoring terminals may be sequential or unordered. The 10 local monitoring terminals can be located at different monitoring places.
Alternatively, the target file may be a plurality of types, for example, the target file is a file in a binary Data Management stream (TMDS) format, the local monitoring terminal device acquires the monitoring Data at microsecond-level monitoring intervals, and the acquired monitoring Data is a large amount of high-frequency Data, for example, the monitoring frequency is 12.8k per second. Setting a time period for writing the target file, and acquiring a TDMS format file of monitoring data written in the time period as a target file.
Optionally, the monitoring data stored in the target file may be various, for example, the monitoring data is high-frequency and high-precision voltage data and current data.
In an optional embodiment, the current utilization rate of the remote resolution server is obtained; judging whether the current utilization rate reaches a preset utilization rate or not; and if the current utilization rate reaches the preset utilization rate, determining to increase the running number of the remote analysis servers.
It can be understood that the purpose of flexibly scheduling the processing capacity based on the state of the remote server is achieved by comparing whether the current utilization rate of the remote resolution server reaches the preset utilization rate and taking the current utilization rate as a basis for whether the running number of the remote resolution server is increased.
Optionally, the current utilization rate may be multiple, for example, the current utilization rate at least includes: remote resolution server CPU utilization, memory utilization, disk IO utilization, and so on.
Optionally, the manner of determining whether the current utilization rate reaches the preset utilization rate may be multiple, for example, the current utilization rate at least includes: under the conditions of the CPU utilization rate, the memory utilization rate and the disk IO utilization rate of the remote analysis server, any one of the CPU utilization rate, the memory utilization rate and the disk IO utilization rate of the remote analysis server reaches a preset utilization rate, and the current utilization rate can be regarded as reaching the preset utilization rate.
In an optional embodiment, after the current utilization rate reaches the preset utilization rate, the method further includes: acquiring the duration of the remote analysis server reaching the preset utilization rate; judging whether the duration reaches a preset time or not; and if the duration reaches the preset time, determining to increase the running number of the remote analysis server.
It can be understood that after the current utilization rate is obtained and reaches the preset utilization rate, whether the duration of the remote resolution server reaching the preset utilization rate reaches the preset time or not is judged, and if the judgment result is that the preset time is reached, the running number is increased. By adopting the method, the problem that the service life of the equipment is shortened due to switching of the equipment in a short time can be avoided because the utilization rate of the remote analysis server repeatedly jumps in a short time and the running number jumps.
Alternatively, the manner of determining to increase the number of operations of the remote resolution server may be multiple, for example, the increased number of operations may be set according to specific situations, for example, the number of operations increased at a time is 2. For another example, the manner of adding the remote resolution server may be to wake up the remote resolution server in a dormant state, and when a quantity difference between the running quantity and the maximum quantity of the remote resolution servers is smaller than a preset quantity difference threshold, send a prompt message, so that relevant staff can perform more configuration work of the remote resolution server in time.
In an optional embodiment, the parsing the target file by using the target number of remote parsing servers to obtain a target parsing result includes: performing first analysis processing on the target file to obtain a first analysis result; and calculating the number of the target files, and performing second analysis processing on a plurality of first analysis results after the target files reach a preset number to obtain the target analysis results.
It can be understood that, the target file is analyzed twice, and first, a first analysis is performed to obtain a first analysis result. And after the number of the target files reaches a preset number, reading a plurality of first analysis results, and performing second analysis processing based on the reading results to obtain the target analysis results.
Optionally, before performing the first parsing process on the target file, the target file is preprocessed, for example, Global Positioning System (GPS) state information and monitoring frequency information in the target file are read. And judging whether the GPS state information is normal or not, and if the judgment result is abnormal, setting the occurrence time of the abnormal GPS state information as the current time.
Optionally, the first parsing process may be multiple, for example, the first parsing process is performed on the monitoring data in the target file corresponding to a preset reading time period, where under the condition that the monitoring data is voltage data and current data, the first parsing process may be performed by calculating a preset index, and the first parsing result may be multiple, such as a current effective value, a voltage effective value, a harmonic, and the like. In the process of the first analysis processing, the single thread processing is ensured by adopting a mutual exclusion lock mode, and the accuracy of the data is improved.
Alternatively, the second analysis process may be a plurality of types, for example, the manner of the second analysis process may be a plurality of statistical manners, and the maximum value, the minimum value, the average value, the preset percentage, and the like of the plurality of calculated first analysis results are used as the target analysis result based on the determined statistical manner.
Optionally, the target file may be parsed by the target number of remote parsing servers, for example, after the first parsing is performed, the first parsing result is stored in a cache area, when the accumulated target file reaches a preset accumulated value, the first parsing result in the cache area is read, and based on the read result, a second parsing is performed to obtain the target parsing result.
In an optional embodiment, waveform data corresponding to the target file is determined; and storing the waveform data to a second database, and displaying the waveform data in real time.
It can be understood that based on the monitoring data in the target file, corresponding waveform data is determined, the waveform data is stored in a second database, and the waveform data is displayed in real time.
Alternatively, the second database may be various, for example, a TDengine database, which is a name of a big data engine of a time-series space.
Optionally, the real-time displayed waveform data may be various, for example, waveform data within a preset time is selected as displayed data, a real-time corresponding waveform image is determined based on the displayed data, and the waveform image is displayed in real time through a visual interface.
Based on the foregoing embodiment and alternative embodiments, the present invention provides an alternative implementation manner, and for facilitating understanding, specific numerical values are set only for convenience of description, and do not limit the embodiments of the present invention, so that the following steps are facilitated:
and S1, acquiring 20 target files based on the fifth generation communication technology 5G, wherein the target files are monitoring data stored in the monitoring data acquired by the local monitoring terminal equipment. For example: the method comprises the steps that a plurality of local monitoring terminals collect voltage data and current data every 78 microseconds and respectively write files in a TDMS format (including head data and main data, the head data are index information, and the main data are monitoring information), and when the accumulated writing time of the files reaches 2 seconds, the local monitoring terminals stop writing the files to obtain a plurality of target files. The number of the local monitoring terminal is 10 corresponding to the target files A, and the number of the local monitoring terminal is 10 corresponding to the target files B. The 20 object files are transmitted by the fifth generation communication technology 5G.
S2, it is obtained that the current utilization rate of the remote resolution server 1 is 65%, and the preset utilization rate is 60%, which indicates that the utilization rate of the remote resolution server 1 reaches the preset utilization rate. If the duration of the remote analysis server 1 reaching the preset utilization rate is 10 minutes and the preset time is 9 minutes, the number of the remote analysis servers is determined to be increased. The remote analysis server includes: remote resolution server 1 and remote resolution server 2, wherein remote resolution server 2 is in a dormant state. And if it is determined that 2 remote parsing servers are started according to the number of the target files obtained in S1 being 20, waking up the remote parsing server 2 in the sleep state.
S3, the 20 target files are equally distributed to the 2 remote analysis servers. The remote analysis server 1 corresponds to 10 target files acquired by the local monitoring terminal with the number of A, and the remote analysis server 2 corresponds to 10 target files acquired by the local monitoring terminal with the number of B.
And S4, based on the target file obtained by the local monitoring terminal with the number A, the remote analysis server 1 carries out pretreatment. Firstly, according to the TDMS format, reading the head data in the target file to obtain the GPS state information and the monitoring frequency information. And if the GPS state information is abnormal data, taking the current time as the occurrence time of the abnormal GPS state information. Thereafter, the remote analysis server 1 reads the body data in the corresponding object file, and based on the monitoring data obtained in S1, the body data are voltage data and current data, and each body data includes corresponding time data as the data generation time.
S5, the remote analysis server 1 performs a first analysis process on the target file. The first analysis process specifically includes: and reading the voltage data and the current data with the accumulated time of 1 second, and calculating to obtain a current effective value, a voltage effective value and a harmonic wave. In the calculation process, a mutual exclusion lock mode is adopted to ensure single thread processing and improve the accuracy of data. And storing the calculation result as a first analysis result in a cache region.
S6, after receiving 10 target files cumulatively, the remote parsing server 1 reads a plurality of first parsing results in the cache area, and performs a second parsing process based on the plurality of first parsing results, where the second parsing process is to calculate a maximum value, a minimum value, an average value, and a value at a 95 th% position, and obtain the calculation result as a second parsing result.
And S7, storing the second analysis result into a first database, wherein the first database is a relational database MySQL.
S8, waveform data corresponding to the plurality of target files is determined, and the waveform data is stored in the second database. And the second database is a time sequence database TDengine. The latest 2 seconds of waveform data also needs to be buffered in the process for real-time presentation.
By the optional implementation mode, the characteristics of high speed and low delay of the fifth generation communication technology 5G can be fully utilized, the acquired high-precision voltage data and high-precision current data are sent to a remote end in real time to be processed, and real-time access and analysis processing of a large amount of high-precision data are realized. Based on limited resources, the computing processing capacity of the remote end is utilized to the maximum extent.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
The embodiment of the invention also provides a data remote processing system, and the data remote processing system provided by the embodiment of the invention is introduced below.
Fig. 2 is a block diagram of a data remote processing system according to an embodiment of the present invention, as shown in fig. 2, the system includes: the system will be described with reference to a load balancing server 202, a remote resolution server 204, and a database server 206.
The load balancing server 202 is configured to obtain a target file based on a fifth generation communication technology 5G, where the target file stores monitoring data obtained by the local monitoring terminal device; determining to open the remote analysis servers of the target number according to the number of the target files; equally distributing the target files to the remote analysis servers with the target number;
a remote analysis server 204 connected 202 to the load balancing server, for analyzing the target file to obtain a target analysis result;
a database server 206 connected to the remote parsing server 204 for storing the target parsing result in a first database.
Optionally, the data remote processing system is further configured to obtain, by the load balancing server, a current utilization rate of the remote resolution server; judging whether the current utilization rate reaches a preset utilization rate or not; and if the current utilization rate reaches the preset utilization rate, determining to increase the running number of the remote analysis servers.
Optionally, the remote parsing server is further configured to perform a first parsing process on the target file to obtain a first parsing result; and calculating the number of the target files, and performing second analysis processing on a plurality of first analysis results after the target files reach a preset number to obtain the target analysis results.
Optionally, the data remote processing system is further configured to determine, by the remote parsing server, waveform data corresponding to the target file; and storing the waveform data to a second database, and displaying the waveform data in real time.
In the remote data processing system provided in the embodiment of the present invention, the load balancing server 202 is configured to obtain a target file based on a fifth generation communication technology 5G, where the target file stores monitoring data obtained by a local monitoring terminal device; determining to open the remote analysis servers of the target number according to the number of the target files; equally distributing the target files to the remote analysis servers with the target number; a remote analysis server 204 connected 202 to the load balancing server, for analyzing the target file to obtain a target analysis result; a database server 206 connected to the remote parsing server 204 for storing the target parsing result in a first database. The purposes of realizing data transmission by using a fifth generation communication technology 5G and allocating and optimizing the utilization rate of a related data processing server are achieved, the technical effects of improving the data remote processing efficiency and reducing the data processing cost are achieved, and the technical problems of low data processing efficiency, poor real-time performance and high processing cost of a data remote processing method in the related technology are solved.
Based on the above embodiments, the present invention provides an alternative implementation. FIG. 3 is a schematic diagram of a data remote processing system provided in accordance with an embodiment of the present invention. As shown in fig. 3, the data remote processing system obtains a plurality of target files acquired by the local monitoring terminal through a fifth generation communication technology 5G. And the load balancing server is connected with a plurality of remote analysis servers, and the remote analysis servers with target number are determined to be started according to the number of the target files. And the load balancing service equally distributes the target files to the target number of remote analysis servers. And the remote analysis server analyzes the target file to obtain a target analysis result. The database server is connected with the remote analysis server, and the plurality of remote analysis servers store the obtained plurality of target analysis results in the first database.
In this embodiment, a data remote processing apparatus is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and the description already made is omitted for brevity. As used hereinafter, the terms "module" and "apparatus" may refer to a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
According to an embodiment of the present invention, an embodiment of an apparatus for implementing a data remote processing method is further provided, fig. 4 is a schematic diagram of a data remote processing apparatus according to an embodiment of the present invention, and the apparatus of the data remote processing method shown in fig. 4 includes an obtaining module 402, a first determining module 404, an allocating module 406, an analyzing module 408, and a first storing module 410, and the apparatus is described as follows:
an obtaining module 402, configured to obtain a target file based on a fifth generation communication technology 5G, where the target file stores monitoring data obtained by a local monitoring terminal device;
a first determining module 404, connected to the obtaining module 402, configured to determine, according to the number of the target files, to start a target number of remote parsing servers;
a distribution module 406, connected to the first determining module 404, configured to distribute the target files to the target number of remote parsing servers on an average basis;
the analysis module 408 is connected to the allocation module 406, and is configured to perform analysis processing on the target file by using the target number of remote analysis servers to obtain a target analysis result;
the first storage module 410 is connected to the parsing module 408, and is configured to store the target parsing result in a first database.
In the remote data processing apparatus provided in the embodiment of the present invention, the obtaining module 402 is configured to obtain a target file based on a fifth generation communication technology 5G, where the target file stores monitoring data obtained by a local monitoring terminal device; a first determining module 404, connected to the obtaining module 402, configured to determine, according to the number of the target files, to start a target number of remote parsing servers; a distribution module 406, connected to the first determining module 404, configured to distribute the target files to the target number of remote parsing servers on an average basis; the analysis module 408 is connected to the allocation module 406, and is configured to perform analysis processing on the target file by using the target number of remote analysis servers to obtain a target analysis result; a first storage module 410, connected to the parsing module 408, for storing the target parsing result in a first database. The purposes of realizing data transmission by using a fifth generation communication technology 5G and allocating and optimizing the utilization rate of a related data processing server are achieved, the technical effects of improving the data remote processing efficiency and reducing the data processing cost are achieved, and the technical problems of low data processing efficiency, poor real-time performance and high processing cost of a data remote processing method in the related technology are solved.
As an optional embodiment, the apparatus for data remote processing method provided in the embodiment of the present invention further includes:
a second determining module 412, configured to determine waveform data corresponding to the target file;
a second storage module 414, connected to the second determining module 412, for storing the waveform data in a second database;
and the display module 416 is connected to the second storage module 414, and is configured to display the waveform data in real time.
It should be noted that the above modules may be implemented by software or hardware, for example, for the latter, the following may be implemented: the modules can be located in the same processor; alternatively, the modules may be located in different processors in any combination.
It should be noted that the obtaining module 402, the first determining module 404, the allocating module 406, the parsing module 408 and the first storing module 410 correspond to steps S102 to S110 in the embodiment, and the modules are the same as the corresponding steps in the implementation example and the application scenario, but are not limited to the disclosure in the embodiment. It should be noted that the modules described above may be implemented in a computer terminal as part of an apparatus.
It should be noted that, for alternative or preferred embodiments of the present embodiment, reference may be made to the relevant description in the embodiments, and details are not described herein again.
The data-based remote processing apparatus may further include a processor and a memory, where the obtaining module 402, the first determining module 404, the allocating module 406, the parsing module 408, the first storing module 410, and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. One or more cores may be provided. The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
An embodiment of the present invention provides a nonvolatile storage medium on which a program is stored, the program implementing a data remote processing method when executed by a processor.
As shown in fig. 5, an embodiment of the present invention provides an electronic device, where the electronic device 10 includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor executes the program to implement the following steps: acquiring a target file based on a fifth generation communication technology 5G, wherein the target file stores monitoring data acquired by local monitoring terminal equipment; determining to open the remote analysis servers of the target number according to the number of the target files; equally distributing the target files to the remote analysis servers with the target number; adopting the target number of remote analysis servers to analyze the target file to obtain a target analysis result; and storing the target analysis result to a first database. The device herein may be a server, a PC, etc.
The invention also provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device: acquiring a target file based on a fifth generation communication technology 5G, wherein the target file stores monitoring data acquired by local monitoring terminal equipment; determining to open the remote analysis servers of the target number according to the number of the target files; equally distributing the target files to the remote analysis servers with the target number; adopting the target number of remote analysis servers to analyze the target file to obtain a target analysis result; and storing the target analysis result to a first database.
Optionally, the computer program product is further adapted to execute a program initializing the following method steps: the method further comprises the following steps: acquiring the current utilization rate of the remote analysis server; judging whether the current utilization rate reaches a preset utilization rate or not; and if the current utilization rate reaches the preset utilization rate, determining to increase the running number of the remote analysis servers.
Optionally, the computer program product is further adapted to execute a program initializing the following method steps: after the current utilization rate reaches the preset utilization rate, the method further includes: acquiring the duration of the remote analysis server reaching the preset utilization rate; judging whether the duration reaches a preset time or not; and if the duration reaches the preset time, determining to increase the running number of the remote analysis server.
Optionally, the computer program product is further adapted to execute a program initializing the following method steps: the above-mentioned remote analysis server that adopts above-mentioned target quantity carries out analysis processing to above-mentioned target file, obtains the target analysis result, includes: performing first analysis processing on the target file to obtain a first analysis result; and calculating the number of the target files, and performing second analysis processing on a plurality of first analysis results after the target files reach a preset number to obtain the target analysis results.
Optionally, the computer program product is further adapted to execute a program initializing the following method steps: the method further comprises the following steps: determining waveform data corresponding to the target file; and storing the waveform data to a second database, and displaying the waveform data in real time.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present invention, and are not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims (10)
1. A method for remote processing of data, comprising:
acquiring a target file based on a fifth generation communication technology 5G, wherein the target file stores monitoring data acquired by local monitoring terminal equipment;
determining to start the remote analysis servers of the target number according to the number of the target files;
distributing the target files to the target number of remote analysis servers in an average manner;
adopting the remote analysis servers with the target number to analyze the target file to obtain a target analysis result;
and storing the target analysis result to a first database.
2. The method of claim 1, further comprising:
acquiring the current utilization rate of the remote resolution server;
judging whether the current utilization rate reaches a preset utilization rate or not;
and if the current utilization rate reaches the preset utilization rate, determining to increase the running number of the remote analysis servers.
3. The method of claim 2, wherein after the current utilization reaches the preset utilization, the method further comprises:
acquiring the duration of the remote analysis server reaching the preset utilization rate;
judging whether the duration reaches a preset time or not;
and if the duration reaches the preset time, determining to increase the running number of the remote analysis servers.
4. The method according to claim 1, wherein the parsing the target file by using the target number of remote parsing servers to obtain a target parsing result comprises:
performing first analysis processing on the target file to obtain a first analysis result;
and calculating the number of the target files, and performing second analysis processing on the plurality of first analysis results after the target files reach a preset number to obtain the target analysis results.
5. The method according to any one of claims 1 to 4, further comprising:
determining waveform data corresponding to the target file;
and storing the waveform data to a second database, and displaying the waveform data in real time.
6. A data remoting system, comprising:
the load balancing server is used for acquiring a target file based on a fifth generation communication technology 5G, wherein the target file stores monitoring data acquired by local monitoring terminal equipment; determining to start the remote analysis servers of the target number according to the number of the target files; distributing the target files to the target number of remote analysis servers on average;
the remote analysis server is connected with the load balancing server and used for analyzing the target file to obtain a target analysis result;
and the database server is connected with the remote analysis server and used for storing the target analysis result to a first database.
7. A data remote processing apparatus, comprising:
the acquisition module is used for acquiring a target file based on a fifth generation communication technology 5G, wherein the target file stores monitoring data acquired by local monitoring terminal equipment;
the first determining module is used for determining to start the remote analysis servers of the target number according to the number of the target files;
the distribution module is used for distributing the target files to the remote analysis servers with the target number in an average manner;
the analysis module is used for adopting the target number of remote analysis servers to analyze the target file to obtain a target analysis result;
and the first storage module is used for storing the target analysis result to a first database.
8. The apparatus of claim 7, further comprising:
the second determining module is used for determining waveform data corresponding to the target file;
the second storage module is used for storing the waveform data to a second database;
and the display module is used for displaying the waveform data in real time.
9. A non-volatile storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform a method of remotely processing data according to any of claims 1 to 5.
10. An electronic device, comprising: one or more processors and memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of remote processing of data of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210941818.8A CN114996363A (en) | 2022-08-08 | 2022-08-08 | Data remote processing method, system, device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210941818.8A CN114996363A (en) | 2022-08-08 | 2022-08-08 | Data remote processing method, system, device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114996363A true CN114996363A (en) | 2022-09-02 |
Family
ID=83023177
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210941818.8A Pending CN114996363A (en) | 2022-08-08 | 2022-08-08 | Data remote processing method, system, device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114996363A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104243537A (en) * | 2013-06-24 | 2014-12-24 | ***股份有限公司 | Automatic retractable method and system used under cloud computing environment |
| CN105245579A (en) * | 2015-09-16 | 2016-01-13 | 许继集团有限公司 | Power distribution network system massive data communication processing method and system |
| CN111447272A (en) * | 2020-03-26 | 2020-07-24 | 支付宝(杭州)信息技术有限公司 | Load balancing method and device |
-
2022
- 2022-08-08 CN CN202210941818.8A patent/CN114996363A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104243537A (en) * | 2013-06-24 | 2014-12-24 | ***股份有限公司 | Automatic retractable method and system used under cloud computing environment |
| CN105245579A (en) * | 2015-09-16 | 2016-01-13 | 许继集团有限公司 | Power distribution network system massive data communication processing method and system |
| CN111447272A (en) * | 2020-03-26 | 2020-07-24 | 支付宝(杭州)信息技术有限公司 | Load balancing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110661659A (en) | Alarm method, device and system and electronic equipment | |
| CN107451147B (en) | Method and device for dynamically switching kafka clusters | |
| CN111813545A (en) | Resource allocation method, device, medium and equipment | |
| CN106487601B (en) | Resource monitoring method, device and system | |
| CN109739627B (en) | Task scheduling method, electronic device and medium | |
| CN111984495A (en) | Big data monitoring method and device and storage medium | |
| CN112749013B (en) | Thread load detection method and device, electronic equipment and storage medium | |
| US11360872B2 (en) | Creating statistical analyses of data for transmission to servers | |
| CN113238714A (en) | Disk capacity prediction method and system based on historical monitoring data and storage medium | |
| CN105183627A (en) | Server performance prediction method and system | |
| CN110865919B (en) | Monitoring method and device based on java process and computer equipment | |
| CN112035322A (en) | JVM monitoring method and device | |
| CN114996363A (en) | Data remote processing method, system, device and electronic equipment | |
| US20170315842A1 (en) | Resource consuming tasks scheduler | |
| CN116126859A (en) | Data management method and device, electronic equipment and storage medium | |
| CN115525392A (en) | Container monitoring method and device, electronic equipment and storage medium | |
| CN113032207A (en) | Application process monitoring method and device, electronic equipment and storage medium | |
| CN109902067B (en) | File processing method and device, storage medium and computer equipment | |
| CN108805778A (en) | Electronic device, the method and storage medium for acquiring collage-credit data | |
| CN110955579A (en) | Ambari-based large data platform monitoring method | |
| CN113009407B (en) | Voltage event recording method and device for double-core intelligent ammeter and double-core intelligent ammeter | |
| US20160292077A1 (en) | Low-latency lossy processing of machine data | |
| CN110489967B (en) | Method and device for analyzing program running risk | |
| CN115220648B (en) | Data writing flow statistics method and device and electronic equipment | |
| CN115687010A (en) | Log early warning method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220902 |
|
| RJ01 | Rejection of invention patent application after publication |