CN112835931A - Method and device for determining data acquisition frequency - Google Patents
Method and device for determining data acquisition frequency Download PDFInfo
- Publication number
- CN112835931A CN112835931A CN201911159815.3A CN201911159815A CN112835931A CN 112835931 A CN112835931 A CN 112835931A CN 201911159815 A CN201911159815 A CN 201911159815A CN 112835931 A CN112835931 A CN 112835931A
- Authority
- CN
- China
- Prior art keywords
- data
- frequency
- service data
- service
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000012545 processing Methods 0.000 claims abstract description 30
- 238000004140 cleaning Methods 0.000 claims description 12
- 238000003860 storage Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 238000009826 distribution Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000002354 daily effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a method and a device for determining data acquisition frequency, which are applied to a big data platform, wherein the method comprises the following steps: acquiring the generation time of each generated service data in a service system; determining a first rule for reflecting the generation amount of the service data in different time periods according to the generation time of each service data; and determining the frequency of the big data platform for acquiring the service data from the service system according to the first rule. By the method, the frequency of acquiring the service data from the service data system can be dynamically adjusted by the big data platform, the reasonable distribution of computing resources is realized, and the data processing efficiency is improved.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a method and an apparatus for determining a data acquisition frequency.
Background
At present, with the rise of big data technology, various big data platforms have been applied to daily business systems. The service system can generate massive service data, and the big data platform can acquire the service data from the service system at a certain frequency for processing. However, in the prior art, the frequency of acquiring the service data set by the big data platform is fixed, but the amount of the service data generated by the service system in different time periods is different; acquiring traffic data at the same frequency for each time slot creates a number of problems. For example, when the amount of service data generated by the service system is large, a phenomenon that a large amount of service data is queued to be processed by a large data platform occurs, resulting in an excessively long service data processing time; when the amount of service data generated by the service system is small, the phenomenon of wasting computing resources of a large data platform occurs.
Therefore, how to dynamically adjust the frequency of acquiring the service data by the big data platform is an urgent problem to be solved.
Disclosure of Invention
The application provides a method and a device for determining data acquisition frequency, which are applied to a big data platform and are used for solving the problems that in the prior art, the big data platform acquires service data at a fixed frequency, so that the resource allocation is unreasonable and the data processing efficiency is low. By the method, the frequency of acquiring the service data from the service data system can be dynamically adjusted by the big data platform, the reasonable distribution of computing resources is realized, and the data processing efficiency is improved.
In a first aspect, an embodiment of the present application provides a method for determining a data acquisition frequency, where the method includes:
acquiring the generation time of each generated service data in a service system;
determining a first rule for reflecting the generation amount of the service data in different time periods according to the generation time of each service data;
and determining the frequency of the big data platform for acquiring the service data from the service system according to the first rule.
Optionally, the method includes:
when the first rule represents that the traffic data volume is greater than a first threshold value in a first time period, the frequency is greater than a first preset frequency; or the like, or, alternatively,
when the first rule represents that the service data volume is smaller than a second threshold value in a second time period, the frequency is smaller than a second preset frequency;
the first threshold is larger than the second threshold, and the first preset frequency is larger than the second preset frequency.
Optionally, the determining, according to the first rule, a frequency at which the big data platform acquires service data from the service system includes:
and determining the frequency of acquiring the service data from the service system by the big data platform in each time period according to the service data generation amount in each time period represented by the first rule.
Optionally, the method includes:
determining that the current time is within a third time period;
and acquiring the service data from the service system according to the frequency corresponding to the third time period.
Optionally, after the service data is acquired from the service system, the method further includes:
cleaning the service data according to a preset cleaning algorithm to obtain target data;
and storing the target data to a target database.
In a second aspect, an embodiment of the present application further provides an apparatus for determining a data acquisition frequency, including:
the acquisition module is used for acquiring the generation time of each generated service data in the service system;
the processing module is used for determining a first rule for reflecting the generation amount of the service data in different time periods according to the generation time of each service data;
the processing module is further configured to determine, according to the first rule, a frequency at which the big data platform acquires the service data from the service system.
Optionally, the method includes:
when the first rule represents that the traffic data volume is greater than a first threshold value in a first time period, the frequency is greater than a first preset frequency; or the like, or, alternatively,
when the first rule represents that the service data volume is smaller than a second threshold value in a second time period, the frequency is smaller than a second preset frequency;
the first threshold is larger than the second threshold, and the first preset frequency is larger than the second preset frequency.
Optionally, the processing module is configured to determine, according to the first rule, a frequency at which the big data platform acquires service data from the service system, and specifically, is configured to:
and determining the frequency of acquiring the service data from the service system by the big data platform in each time period according to the service data generation amount in each time period represented by the first rule.
Optionally, the processing module is further configured to:
determining that the current time is within a third time period;
and acquiring the service data from the service system according to the frequency corresponding to the third time period.
In a third aspect, an embodiment of the present application further provides an electronic device, including: a memory for storing program instructions; and the processor is used for calling the program instructions stored in the memory and executing one or more steps provided by the first aspect according to the obtained program instructions.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium storing a computer program comprising program instructions that, when executed by a computer, cause the computer to perform one or more of the steps as provided in the first aspect above.
In a fifth aspect, embodiments of the present application provide a program product comprising program instructions that, when executed by a computer, cause the computer to perform one or more of the steps as provided in the first aspect above.
In the technical scheme of the embodiment of the application, a method and a device for determining data acquisition frequency are provided, and are applied to a big data platform, wherein the method comprises the following steps: acquiring the generation time of each generated service data in a service system; determining a first rule for reflecting the generation amount of the service data in different time periods according to the generation time of each service data; and determining the frequency of the big data platform for acquiring the service data from the service system according to the first rule. By the method, the frequency of acquiring the service data from the service data system can be dynamically adjusted by the big data platform, the reasonable distribution of computing resources is realized, and the data processing efficiency is improved.
Drawings
Fig. 1 is a schematic flowchart of a method for determining a data acquisition frequency according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of an apparatus for determining a data acquisition frequency according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the accompanying drawings, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms "first" and "second" in the description and claims of the present application and the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the term "comprises" and any variations thereof, which are intended to cover non-exclusive protection. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
In the embodiments of the present application, "at least one" may mean one or at least two, for example, one, two, three, or more, and the embodiments of the present application are not limited.
In addition, the term "and/or" herein is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" in this document generally indicates that the preceding and following related objects are in an "or" relationship unless otherwise specified.
The background of the present application is described below.
At present, with the rise of big data technology, various big data platforms have been applied to daily business systems. The service system can generate massive service data, and the big data platform can acquire the service data from the service system at a certain frequency for processing. However, in the prior art, the frequency of acquiring the service data set by the big data platform is fixed, but the amount of the service data generated by the service system in different time periods is different; acquiring traffic data at the same frequency for each time slot creates a number of problems. For example, when the amount of service data generated by the service system is large, a phenomenon that a large amount of service data is queued to be processed by a large data platform occurs, resulting in an excessively long service data processing time; when the amount of service data generated by the service system is small, the phenomenon of wasting computing resources of a large data platform occurs.
In order to solve the above technical problem, the present application provides a method for determining a data acquisition frequency, which is applied to a big data platform, and the method includes: acquiring the generation time of each generated service data in a service system; determining a first rule for reflecting the generation amount of the service data in different time periods according to the generation time of each service data; and determining the frequency of the big data platform for acquiring the service data from the service system according to the first rule. By the method, the frequency of acquiring the service data from the service data system can be dynamically adjusted by the big data platform, the reasonable distribution of computing resources is realized, and the data processing efficiency is improved.
In order to better understand the technical solutions, the technical solutions of the present application are described in detail below through the drawings and the specific embodiments of the specification, and it should be understood that the specific features of the embodiments and examples of the present application are detailed descriptions of the technical solutions of the present application, and are not limitations of the technical solutions of the present application, and the technical features of the embodiments and examples of the present application may be combined with each other without conflict.
The shapes and sizes of the various elements in the drawings are not to be considered as true proportions, but rather are merely intended to illustrate the context of the application.
A method for determining a data acquisition frequency provided by an embodiment of the present application is described below. Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a method for determining a data acquisition frequency according to an embodiment of the present disclosure, where the method may be applied to a big data platform. The method comprises the following steps:
101: the generation time of each service data generated in the service system is acquired.
It should be understood that the big data platform obtains the generation time of each service data from the service data interface, i.e. obtains the creation time field in each service data. The big data platform obtains the generation time of each service data from the service data interface, and the efficiency of the big data platform for acquiring the service data is improved.
102: and determining a first rule for reflecting the generation amount of the service data in different time periods according to the generation time of each service data.
Illustratively, suppose that the big data platform acquires the generation time of all the service data in 2018 from the service data interface, wherein the big data platform analyzes the generation time of all the service data in 2018 according to a built-in clustering algorithm to determine a data generation rule in each hour of one day, that is, the amount of the service data generated in each hour from 8 am to 11 am in one day is 200, and the amount of the service data generated in each hour from 2 pm to 6 pm is 100.
103: and determining the frequency of the big data platform for acquiring the service data from the service system according to the first rule.
In a possible implementation manner, when the first rule indicates that the traffic data volume is greater than a first threshold value in a first time period, the frequency is greater than a first preset frequency; or the like, or, alternatively,
when the first rule represents that the service data volume is smaller than a second threshold value in a second time period, the frequency is smaller than a second preset frequency; the first threshold is larger than the second threshold, and the first preset frequency is larger than the second preset frequency.
It should be understood that the first time period and the second time period may be 15 minutes, half an hour, one hour, three hours, one day, etc., depending on the data samples collected by the big data platform. The examples of the present application are not particularly limited.
Illustratively, assuming that the first threshold is 250, the first preset frequency is 5 times per hour; the second threshold is 100, the second preset frequency is 2 times per hour, and the data volume acquired from the service system by the big data platform each time is 50; if the first time period is from 8 am to 9 am, the service data volume is 350, then from 8 am to 9 am, and the data acquisition frequency of the large data platform is 7 times per hour; if the first time period is from 8 am to 9 am, the service data volume is 50, and the data acquisition frequency of the large data platform is 1 time per hour from 8 am to 9 am. By the aid of the method, the data acquisition frequency of the big data platform is improved in a time period with large service data production amount, the phenomenon that a large amount of service data waits to be processed in a queuing mode in a message queue is avoided, the big data platform can timely clean the acquired service data according to a subsequent cleaning algorithm, and the requirement that a user interface displays the data in real time can be met.
In a possible implementation manner, the determining, by the big data platform, a frequency of the big data platform acquiring the service data from the service system according to the first rule specifically includes: and determining the frequency of acquiring the service data from the service system by the big data platform in each time period according to the service data generation amount in each time period represented by the first rule.
It should be understood that the frequency of the big data platform for acquiring the service data from the service system provided by the application is dynamically changed, the data generation rule of the service system in each time period in the future is predicted by analyzing the generation rule of the historical service data, and the frequency of the big data platform for acquiring the service data from the service system is dynamically adjusted according to the data generation rule in each time period.
For example, assume that the first rule is that the amount of traffic data generated per hour is 200 from 8 am to 11 am and 100 from 2 pm to 6 pm; assuming that a big data platform can acquire 50 business data from a business system once, the frequency of data acquisition generated every hour is 4 times in 8 am to 11 am and 2 times in 2 pm to 6 pm.
In one possible implementation, the big data platform determines that the current time is within a third time period; and acquiring the service data from the service system according to the frequency corresponding to the third time period.
It should be understood that the big data platform analyzes the historical service data to obtain the service data generation rule of each time period in the future, and further analyzes to obtain the appropriate data acquisition frequency. Before a big data platform acquires new data from a service system for processing, a current time period and a data acquisition frequency corresponding to the current time period need to be determined, and the operation cycle of a data acquisition program is further adjusted according to the current data acquisition frequency, wherein the specific adjustment can be executed by the big data platform, or prompt information for prompting a user to adjust the operation cycle of the data acquisition program can be output by the big data platform, and the embodiment of the application is not specifically limited.
Optionally, after acquiring the service data from the service system, the method further includes:
cleaning the service data according to a preset cleaning algorithm to obtain target data;
and storing the target data to a target database. It should be understood that after the big data platform acquires a plurality of service data from the service system, the plurality of service data needs to be cleaned in order to reduce redundant parts in the data. The preset cleaning algorithm comprises a data table source, a data table acquisition mode, a field extraction or deletion realization rule and a target output database.
Illustratively, the big data platform cleans the acquired business data by adopting kafka + spark lines, and the kafka + spark lines have high real-time performance compared with other data cleaning technologies and can simultaneously process more data. Specifically, the big data platform runs a built-in data acquisition program, sends the acquired service data to Kafka (message queue), then runs a built-in data cleaning program (spark sequencing technology) to perform data cleaning on the message content of Kafka (message queue), obtains target data, and stores the target data in a target database for the front end or the back end of the big data platform to use.
The method for determining the data acquisition frequency provided by the present application will be explained with reference to specific embodiments.
Example 1
Assuming that 100 ten thousand service data are generated in 2018, the big data platform acquires the generation time of all the service data in 2018 from the service data interface. The big data platform analyzes the generation time of the 100 ten thousand service data according to a clustering algorithm stored in the big data platform, and predicts the generation rule of the service data in each hour every day in 2019. The expected traffic data production amount per hour from 8 am to 11 am on 11/7/11/year in 2019 is 250 pieces of traffic data. The big data platform further analyzes the service data generation rule from 8 am to 11 am on 11.7.7.2019 through a preset deep learning algorithm, the data acquisition frequency of the big data platform in each hour from 8 am to 11 am on 11.7.7.2019 is determined, the big data platform sets the data acquisition frequency parameter in the operation period of a data acquisition program of the big data platform, the big data platform operates the program for acquiring the service data according to the data acquisition frequency parameter to acquire the data of the service system, and corresponding cleaning processing is carried out, so that the acquired service data are displayed on a user interface.
Example 2
Suppose that 100 ten thousand service data are generated in 2017, 80 ten thousand service data are generated in 2018, and the big data platform acquires the generation time of all the service data in 2017 and 2018 from the service data interface. The big data platform analyzes the generation time of the 180 ten thousand service data according to a clustering algorithm stored in the big data platform, and predicts the service data generation rule in each hour of the corresponding date of 2019 by taking the average value of the service data generated on the same day in the same month. The analysis result shows that the predicted service data generation amount of each hour from 8 am to 11 am of 7 th of 9 th and 9 th of 2019 is 300 service data, the predicted service data generation amount of each hour from 2 am to 6 pm is 110, the big data platform analyzes the predicted service data generation rule of the big data platform during the operation period of the big data platform from 7 am of 9 th and 7 th of 2019 through a preset deep learning algorithm, and determines that the data acquisition frequency of the big data platform from 8 am to 11 am of 7 th of 9 th and 9 th of 2019 th is 6 times, and the data acquisition frequency of each hour from 2 pm to 6 pm of 7 th of 9 th and 9 th of 2019 is 2 times. The big data platform outputs a parameter value for prompting a user to modify the frequency of obtaining the service data by the big data platform (namely, the operation period of the modified data obtaining program), and the big data platform operates the program for obtaining the service data according to the data obtaining frequency parameter to obtain the data of the service system and carries out corresponding cleaning treatment so as to display the obtained service data on a user interface.
Based on the same inventive concept, the embodiment of the invention provides a device for determining the data acquisition frequency. Referring to fig. 2, fig. 2 is a schematic structural diagram of an apparatus for determining a data acquisition frequency according to an embodiment of the present invention, the apparatus includes an acquisition module 201 and a processing module 202, wherein,
an obtaining module 201, configured to obtain a generation time of each service data generated in a service system;
the processing module 202 is configured to determine, according to the generation time of each piece of service data, a first rule for reflecting the generation amount of the service data in different time periods;
the processing module 202 is further configured to determine, according to the first rule, a frequency at which the big data platform obtains the service data from the service system.
In one possible embodiment, the apparatus comprises:
when the first rule represents that the traffic data volume is greater than a first threshold value in a first time period, the frequency is greater than a first preset frequency; or the like, or, alternatively,
when the first rule represents that the service data volume is smaller than a second threshold value in a second time period, the frequency is smaller than a second preset frequency;
the first threshold is larger than the second threshold, and the first preset frequency is larger than the second preset frequency.
In a possible implementation manner, the processing module 202 is configured to determine, according to the first rule, a frequency at which the big data platform acquires service data from the service system, and specifically is configured to:
and determining the frequency of acquiring the service data from the service system by the big data platform in each time period according to the service data generation amount in each time period represented by the first rule.
In a possible implementation, the processing module 202 is further configured to:
determining that the current time is within a third time period; and acquiring the service data from the service system according to the frequency corresponding to the third time period.
Based on the same inventive concept, an electronic device provided with a function of determining a data acquisition frequency according to an embodiment of the present application is provided, and please refer to fig. 3, where the electronic device includes at least one processor 302 and a memory 301 connected to the at least one processor, a specific connection medium between the processor 302 and the memory 301 is not limited in this embodiment of the present application, fig. 3 illustrates an example in which the processor 302 and the memory 301 are connected by a bus 300, the bus 300 is represented by a thick line in fig. 3, and a connection manner between other components is merely a schematic illustration and not limited thereto. The bus 300 may be divided into an address bus, a data bus, a control bus, etc., and is shown with only one thick line in fig. 3 for ease of illustration, but does not represent only one bus or type of bus.
In the embodiment of the present application, the memory 301 stores instructions executable by the at least one processor 302, and the at least one processor 302 may execute the steps included in the foregoing method for determining the data acquisition frequency by calling the instructions stored in the memory 301.
The processor 302 is a control center of the electronic device provided with the function of determining the data acquisition frequency, and can connect various parts of the electronic device provided with the function of determining the data acquisition frequency by using various interfaces and lines, and implement various functions of the electronic device provided with the function of determining the data acquisition frequency by executing the instructions stored in the memory 301. Optionally, the processor 302 may include one or more processing units, and the processor 302 may integrate an application processor and a modem processor, wherein the application processor mainly handles an operating system, a user interface, application programs, and the like, and the modem processor mainly handles wireless communication. It will be appreciated that the modem processor described above may not be integrated into the processor 302. In some embodiments, processor 302 and memory 301 may be implemented on the same chip, or in some embodiments, they may be implemented separately on separate chips.
The processor 302 may be a general-purpose processor, such as a Central Processing Unit (CPU), digital signal processor, application specific integrated circuit, field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like, that may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method for determining the data acquisition frequency disclosed in the embodiments of the present application may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.
By programming the processor 302, the code corresponding to the method for determining the data acquisition frequency described in the foregoing embodiment may be fixed in the chip, so that the chip can execute the steps of the method for determining the data acquisition frequency when running.
Based on the same inventive concept, the present application also provides a storage medium storing computer instructions, which when run on a computer, cause the computer to perform the steps of the method for determining data acquisition frequency as described above.
In some possible embodiments, the various aspects of the method for determining a data acquisition frequency provided by the present application may also be implemented in the form of a program product, which includes program code for a big data platform to perform the steps of the method for determining a data acquisition frequency according to various exemplary embodiments of the present application described above in this specification when the program product is run on the big data platform.
Based on the above embodiments, in the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the method for determining the data acquisition frequency in any of the above method embodiments.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (11)
1. A method for determining data acquisition frequency, applied to a big data platform, the method comprising:
acquiring the generation time of each generated service data in a service system;
determining a first rule for reflecting the generation amount of the service data in different time periods according to the generation time of each service data;
and determining the frequency of the big data platform for acquiring the service data from the service system according to the first rule.
2. The method of claim 1, comprising:
when the first rule represents that the traffic data volume is greater than a first threshold value in a first time period, the frequency is greater than a first preset frequency; or the like, or, alternatively,
when the first rule represents that the service data volume is smaller than a second threshold value in a second time period, the frequency is smaller than a second preset frequency;
the first threshold is larger than the second threshold, and the first preset frequency is larger than the second preset frequency.
3. The method of claim 1, wherein said determining a frequency at which the big data platform obtains business data from the business system according to the first law comprises:
and determining the frequency of acquiring the service data from the service system by the big data platform in each time period according to the service data generation amount in each time period represented by the first rule.
4. The method of claim 3, comprising:
determining that the current time is within a third time period;
and acquiring the service data from the service system according to the frequency corresponding to the third time period.
5. The method of claim 4, wherein after the obtaining the service data from the service system, the method further comprises:
cleaning the service data according to a preset cleaning algorithm to obtain target data;
and storing the target data to a target database.
6. An apparatus for determining a data acquisition frequency, comprising:
the acquisition module is used for acquiring the generation time of each generated service data in the service system;
the processing module is used for determining a first rule for reflecting the generation amount of the service data in different time periods according to the generation time of each service data;
the processing module is further configured to determine, according to the first rule, a frequency at which the big data platform acquires the service data from the service system.
7. The apparatus of claim 6, comprising:
when the first rule represents that the traffic data volume is greater than a first threshold value in a first time period, the frequency is greater than a first preset frequency; or the like, or, alternatively,
when the first rule represents that the service data volume is smaller than a second threshold value in a second time period, the frequency is smaller than a second preset frequency;
the first threshold is larger than the second threshold, and the first preset frequency is larger than the second preset frequency.
8. The apparatus of claim 6, wherein the processing module, when configured to determine, according to the first rule, a frequency at which the big data platform obtains the service data from the service system, is specifically configured to:
and determining the frequency of acquiring the service data from the service system by the big data platform in each time period according to the service data generation amount in each time period represented by the first rule.
9. The apparatus of claim 8, wherein the processing module is further to:
determining that the current time is within a third time period;
and acquiring the service data from the service system according to the frequency corresponding to the third time period.
10. An electronic device, comprising:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory and for executing the steps comprised by the method of any one of claims 1 to 5 in accordance with the obtained program instructions.
11. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911159815.3A CN112835931A (en) | 2019-11-22 | 2019-11-22 | Method and device for determining data acquisition frequency |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911159815.3A CN112835931A (en) | 2019-11-22 | 2019-11-22 | Method and device for determining data acquisition frequency |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112835931A true CN112835931A (en) | 2021-05-25 |
Family
ID=75922144
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911159815.3A Pending CN112835931A (en) | 2019-11-22 | 2019-11-22 | Method and device for determining data acquisition frequency |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112835931A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114071770A (en) * | 2021-12-20 | 2022-02-18 | 广东新电电力科技有限公司 | Communication control method and device based on intelligent communication management gateway |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150161257A1 (en) * | 2013-12-11 | 2015-06-11 | Ebay Inc. | Web crawler optimization system |
| CN106611005A (en) * | 2015-10-26 | 2017-05-03 | 北京国双科技有限公司 | Method and device for setting crawl time interval of crawler |
| CN109388736A (en) * | 2018-09-21 | 2019-02-26 | 真相网络科技（北京）有限公司 | Response scheduling method in crawler system |
| CN110069692A (en) * | 2019-05-06 | 2019-07-30 | 中国船舶工业综合技术经济研究院 | A kind of resource updates time forecasting methods and system towards open source information collection |
-
2019
- 2019-11-22 CN CN201911159815.3A patent/CN112835931A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150161257A1 (en) * | 2013-12-11 | 2015-06-11 | Ebay Inc. | Web crawler optimization system |
| CN106611005A (en) * | 2015-10-26 | 2017-05-03 | 北京国双科技有限公司 | Method and device for setting crawl time interval of crawler |
| CN109388736A (en) * | 2018-09-21 | 2019-02-26 | 真相网络科技（北京）有限公司 | Response scheduling method in crawler system |
| CN110069692A (en) * | 2019-05-06 | 2019-07-30 | 中国船舶工业综合技术经济研究院 | A kind of resource updates time forecasting methods and system towards open source information collection |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114071770A (en) * | 2021-12-20 | 2022-02-18 | 广东新电电力科技有限公司 | Communication control method and device based on intelligent communication management gateway |
| CN114071770B (en) * | 2021-12-20 | 2023-09-05 | 广东新电电力科技有限公司 | Communication control method and device based on intelligent communication management gateway |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108415952B (en) | User data storage method, label calculation method and calculation equipment | |
| CN106815254A (en) | A kind of data processing method and device | |
| CN105005585A (en) | Log data processing method and device | |
| CN112835682A (en) | Data processing method and device, computer equipment and readable storage medium | |
| CN114995791A (en) | API (application program interface) polymerization method and device, electronic equipment and storage medium | |
| CN110221914B (en) | File processing method and device | |
| CN112835931A (en) | Method and device for determining data acquisition frequency | |
| CN107688959B (en) | Breakpoint list processing method, storage medium and server | |
| CN115576973B (en) | Service deployment method, device, computer equipment and readable storage medium | |
| CN111651522A (en) | Data synchronization method and device | |
| CN110865877A (en) | Task request response method and device | |
| CN116185545A (en) | Page rendering method and device | |
| CN107748711B (en) | Method for automatically optimizing Storm parallelism, terminal equipment and storage medium | |
| CN115543317A (en) | Front-end page development method and device | |
| CN111858542B (en) | Data processing method, device, equipment and computer readable storage medium | |
| CN116342087A (en) | Event processing method and device and electronic equipment | |
| CN105677652B (en) | A kind of data managing method and device | |
| CN113655958A (en) | Application data storage method | |
| CN113190381A (en) | Data backup method, system, device and storage medium | |
| CN113010311A (en) | Data self-adaptive adjusting method and device | |
| CN112612583A (en) | Data synchronization method and device, computer equipment and readable storage medium | |
| CN111258676A (en) | Module preloading method, device, equipment and readable storage medium | |
| CN112148470A (en) | Parameter synchronization method, computer device and readable storage medium | |
| CN112650971B (en) | Method, device, equipment and storage medium for realizing formula calculation | |
| CN109947469B (en) | Address division method, device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |