CN113434787B - Network data acquisition method and device and terminal equipment - Google Patents
Network data acquisition method and device and terminal equipment Download PDFInfo
- Publication number
- CN113434787B CN113434787B CN202110527696.3A CN202110527696A CN113434787B CN 113434787 B CN113434787 B CN 113434787B CN 202110527696 A CN202110527696 A CN 202110527696A CN 113434787 B CN113434787 B CN 113434787B
- Authority
- CN
- China
- Prior art keywords
- parameter
- url
- data acquisition
- time
- time parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000005540 biological transmission Effects 0.000 claims abstract description 82
- 238000004590 computer program Methods 0.000 claims description 17
- 230000015654 memory Effects 0.000 claims description 14
- 230000015572 biosynthetic process Effects 0.000 claims description 8
- 238000012795 verification Methods 0.000 claims description 8
- 238000012937 correction Methods 0.000 claims description 5
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9537—Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
- G06F16/9566—URL specific, e.g. using aliases, detecting broken or misspelled links
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
- G06F16/986—Document structures and storage, e.g. HTML extensions
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application provides a network data acquisition method, a device and terminal equipment, wherein the method comprises the following steps: obtaining html data generated by a target database website; the html data comprises a database address and a first data acquisition time period which are input by a user in a target database website; generating a first url transmission parameter according to html data; the first url parameter comprises an address parameter corresponding to the address of the database and a first time parameter corresponding to the first data acquisition time period; correcting the first time parameter in the first url transmission based on a preset second time parameter to obtain a second url transmission; the second url parameter includes an address parameter and a second time parameter; and acquiring data corresponding to the second time parameter from a database corresponding to the address parameter based on the second url parameter. The network data acquisition method, the device and the terminal equipment provided by the application can reduce the network data acquisition cost of the user and improve the network data acquisition efficiency of the user.
Description
Technical Field
The present application belongs to the field of network technology, and in particular, relates to a method, an apparatus and a terminal device for acquiring network data.
Background
In the prior art, a Web interface of a part of database websites is provided with a time control, and the time control can only input a set time period, so that only data in the set time period can be queried at a time. When the set time period range is narrower, the database website only provides information display for users for a short time, for example, data in a time period corresponding to 4 hours can be displayed at one time, if the data corresponding to 8:00 yesterday to 8:00 so far are required to be obtained, the data in 24 hours can be obtained through six times of operations, and the operation is complicated.
Therefore, how to improve the network data acquisition efficiency of the user is a problem to be solved in the art.
Disclosure of Invention
The application aims to provide a network data acquisition method, a network data acquisition device and terminal equipment, so as to improve the network data acquisition efficiency of a user.
In a first aspect of an embodiment of the present application, there is provided a network data acquisition method, including:
obtaining html data generated by a target database website; the html data comprises a database address input by a user in a target database website and a first data acquisition time period;
generating a first url transmission parameter according to the html data; the first url parameter comprises an address parameter corresponding to the database address and a first time parameter corresponding to the first data acquisition time period;
correcting the first time parameter in the first url transmission based on a preset second time parameter to obtain a second url transmission; the second url parameter includes the address parameter and a second time parameter;
and acquiring data corresponding to the second time parameter from a database corresponding to the address parameter based on the second url parameter.
In a second aspect of an embodiment of the present application, there is provided a network data acquisition apparatus, including:
the first data acquisition module is used for acquiring html data generated by a target database website; the html data comprises a database address input by a user in a target database website and a first data acquisition time period;
the url parameter generation module is used for generating a first url parameter according to the html data; the first url parameter comprises an address parameter corresponding to the database address and a first time parameter corresponding to the first data acquisition time period;
the url parameter correction module is used for correcting the first time parameter in the first url transmission based on a preset second time parameter to obtain a second url transmission; the second url parameter includes the address parameter and a second time parameter;
and the second data acquisition module is used for acquiring the data corresponding to the second time parameter from the database corresponding to the address parameter based on the second url parameter.
In a third aspect of the embodiments of the present application, there is provided a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the network data acquisition method described above when executing the computer program.
In a fourth aspect of the embodiments of the present application, there is provided a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the network data acquisition method described above.
The network data acquisition method, the device and the terminal equipment provided by the application have the beneficial effects that:
the method and the system generate the first url transmission by utilizing the html data generated by the target database website according to the user input information, and are different from the scheme of directly acquiring the data based on the first url transmission in the prior art. Compared with the prior art, the data acquisition mode is more flexible, and the second time parameter is supported to be adjusted according to the user requirement, so that the adjustment of the data acquisition time period is realized, the operation time of the user is saved, the operation cost of the user is reduced, the user can be ensured to acquire the required data, and the network data acquisition efficiency of the user is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a network data acquisition method according to an embodiment of the present application;
fig. 2 is a block diagram of a network data acquisition device according to an embodiment of the present application;
fig. 3 is a schematic block diagram of a terminal device according to an embodiment of the present application.
Detailed Description
In order to make the technical problems, technical schemes and beneficial effects to be solved more clear, the application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
Referring to fig. 1, fig. 1 is a flowchart of a network data acquisition method according to an embodiment of the application, where the method includes:
s101: and obtaining html (HyperText Markup Language ) data generated by the target database website, wherein the html data comprises a database address input by a user in the target database website and a first data acquisition time period.
In this embodiment, after the user clicks the database to be accessed on the web interface of the target database website (or other terminal device capable of accessing the database) and inputs the time control in the target database website for a specified time, the target database website generates html data, where the html data includes the database address to be accessed by the user and the data acquisition time period (i.e., the first data acquisition time period) input by the user.
S102: a first url (Uniform Resource Locator ) argument is generated from the html data, the first url argument comprising an address parameter corresponding to the database address and a first time parameter corresponding to the first data acquisition time period.
In this embodiment, html data may be converted into url data, that is, the first url transmission parameter, by means of a concatenation string, route rule, lambda expression, or the like.
S103: and correcting the first time parameter in the first url transmission based on a preset second time parameter to obtain a second url transmission, wherein the second url transmission comprises an address parameter and a second time parameter.
In this embodiment, the time parameter in the url transmission corresponds to the data acquisition time period of the network data, and the first time parameter in the first url transmission is corrected, that is, the first data acquisition time period is corrected.
S104: and acquiring data corresponding to the second time parameter from a database corresponding to the address parameter based on the second url parameter.
In this embodiment, the local machine (corresponding to the execution subject of the method steps S101-S102) may acquire data from the corresponding database based on the second url transmission, or may send the second url transmission to the web server corresponding to the target database website (in this case, the local machine is used as a third party device and does not belong to the target database website), and the web server corresponding to the target database website acquires the corresponding data.
In this embodiment, the execution subject of steps S101 to S103 may be a third party device, or may be a web server of the target database website, which is not limited herein.
It can be obtained from the above that, in the embodiment of the present application, the first url transmission is generated by using the html data generated by the target database website according to the user input information, unlike the scheme of directly performing data acquisition based on the first url transmission in the prior art, the embodiment of the present application further corrects the first time parameter in the first url transmission based on the preset second time parameter, so as to obtain the second url transmission, and finally performs data acquisition according to the second url transmission. Compared with the prior art, the data acquisition mode of the embodiment of the application is more flexible, and the second time parameter is supported to be adjusted according to the user requirement, so that the adjustment of the data acquisition time period is realized, the operation time of the user is saved, the operation cost of the user is reduced, the user can be ensured to acquire the required data, and the network data acquisition efficiency of the user is improved.
Optionally, as a specific implementation manner of the network data acquisition method provided by the embodiment of the present application, before correcting the first time parameter in the first url transmission based on the preset second time parameter to obtain the second url transmission, the method further includes a step of determining the second time parameter. The determining step of the second time parameter includes:
a second data acquisition period entered by the user is acquired.
A second time parameter is determined based on the second data acquisition time period.
In this embodiment, the second time parameter may be preset by the system, or may be determined by user input. When the system is preset, the time period for acquiring the network data is fixed, and when the system is input by a user, the time period for acquiring the network data is customized by the user.
Optionally, as a specific implementation manner of the network data acquisition method provided by the embodiment of the present application, determining the second time parameter according to the second data acquisition time period includes:
and if the second data acquisition time period is empty, taking the preset time parameter as a second time parameter.
And if the second data acquisition time period is not empty, performing program calculation on the second data acquisition time period to obtain a second time parameter.
In this embodiment, if the second data acquisition time period is empty, it is indicated that the user does not input the required data acquisition time period (i.e., the second data acquisition time period), and the default time parameter is just required to be the second time parameter. If the second data acquisition time period is not empty, it indicates that the user has input the required data acquisition time period, and at this time, the second time parameter may be determined according to the data acquisition time period input by the user.
In the present embodiment, the procedure of the program calculation is also a procedure of converting the second data acquisition period into a time parameter in url.
Optionally, as a specific implementation manner of the network data acquisition method provided by the embodiment of the present application, correcting the first time parameter in the first url transmission based on the preset second time parameter to obtain the second url transmission includes:
and replacing the first time parameter in the first url transmission with a preset second time parameter to obtain a second url transmission.
In this embodiment, if there is no bandwidth/acquisition permission limitation between the local and database addresses, or between the target database website and the database address, the first time parameter in the first url transmission may be directly replaced by the preset second time parameter, so as to obtain the second url transmission.
Optionally, as a specific implementation manner of the network data acquisition method provided by the embodiment of the present application, correcting the first time parameter in the first url transmission based on the preset second time parameter to obtain the second url transmission includes:
and acquiring the maximum acquisition time period corresponding to the target database website.
The second time parameter is divided into a plurality of sub-time parameters based on a multiple relationship of the second time parameter and the maximum acquisition time period. Wherein, the time period corresponding to each sub-time parameter is smaller than the maximum acquisition time period.
And sequentially replacing the first time parameter in the first url transmission parameters with each sub-time parameter to obtain a plurality of replaced first url transmission parameters.
And performing program synthesis on the plurality of replaced first url transmission parameters to obtain a second url transmission parameter.
In this embodiment, if there are limitations such as bandwidth/acquisition rights between the local and database addresses or between the target database website and the database address, the second time parameter is divided into a plurality of sub-time parameters based on the concept of multiple acquisition, the first time parameter in the first url transmission is replaced by each sub-time parameter in turn, so as to obtain a plurality of replaced first url transmission, and finally, program synthesis is performed on the plurality of replaced first url transmission, so as to obtain the second url transmission.
In this embodiment, the program synthesis may be a sequential connection of steps on the program or a cyclic implementation of the steps on the program, which is not limited herein.
Optionally, as a specific implementation manner of the network data acquisition method provided by the embodiment of the present application, acquiring data corresponding to a second time parameter from a database corresponding to an address parameter based on a second url parameter includes:
and acquiring authority setting information of a database corresponding to the address parameters.
If the permission setting information display database needs permission verification, the second url transmission parameters are sent to the web server corresponding to the target database website, so that the web server is instructed to acquire data corresponding to the second time parameters from the database corresponding to the address parameters based on the second url transmission parameters.
If the permission setting information display database does not need permission verification, the database is positioned based on the address parameters, and data corresponding to the second time parameters are obtained from the database.
In this embodiment, if the authority setting information display database needs the authority verification, the web server of the target database website with the authority is utilized to acquire data, and if the authority setting information display database does not need the authority verification, the local server directly acquires the data.
Fig. 2 is a block diagram of a network data acquisition device according to an embodiment of the present application, corresponding to the network data acquisition method of the above embodiment. For convenience of explanation, only portions relevant to the embodiments of the present application are shown. Referring to fig. 2, the network data acquisition device 20 includes: a first data acquisition module 21, a url parameter generation module 22, a url parameter modification module 23, a second data acquisition module 24.
The first data obtaining module 21 is configured to obtain html data generated by a target database website. The html data includes a database address input by a user at a target database website and a first data acquisition time period.
The url parameter generation module 22 is configured to generate a first url parameter according to html data. The first url parameter includes an address parameter corresponding to the database address, and a first time parameter corresponding to the first data acquisition time period.
The url parameter correction module 23 is configured to correct the first time parameter in the first url transmission based on the preset second time parameter, so as to obtain the second url transmission. The second url parameter includes an address parameter and a second time parameter.
The second data obtaining module 24 is configured to obtain data corresponding to the second time parameter from the database corresponding to the address parameter based on the second url parameter.
Optionally, as a specific implementation manner of the network data acquisition device provided by the embodiment of the present application, the network data acquisition device may further include:
the second parameter determining module 25 is configured to determine a second time parameter before correcting the first time parameter in the first url transmission based on a preset second time parameter to obtain the second url transmission.
Wherein the second parameter determining module 25 is specifically configured to obtain a second data acquisition period input by the user. A second time parameter is determined based on the second data acquisition time period.
Optionally, as a specific implementation manner of the network data acquisition device provided by the embodiment of the present application, the second parameter determining module 25 is specifically configured to: and when the second data acquisition time period is empty, taking the preset time parameter as a second time parameter. And when the second data acquisition time period is not empty, performing program calculation on the second data acquisition time period to obtain a second time parameter.
Optionally, as a specific implementation manner of the network data acquisition device provided by the embodiment of the present application, the url parameter correction module 23 is specifically configured to:
and replacing the first time parameter in the first url transmission with a preset second time parameter to obtain a second url transmission.
Optionally, as a specific implementation manner of the network data acquisition device provided by the embodiment of the present application, the url parameter correction module 23 is specifically configured to:
and acquiring the maximum acquisition time period corresponding to the target database website.
The second time parameter is divided into a plurality of sub-time parameters based on a multiple relationship of the second time parameter and the maximum acquisition time period. Wherein, the time period corresponding to each sub-time parameter is smaller than the maximum acquisition time period.
And sequentially replacing the first time parameter in the first url transmission parameters with each sub-time parameter to obtain a plurality of replaced first url transmission parameters.
And performing program synthesis on the plurality of replaced first url transmission parameters to obtain a second url transmission parameter.
Optionally, as a specific implementation manner of the network data acquisition device provided by the embodiment of the present application, the second data acquisition module 24 is specifically configured to:
and acquiring authority setting information of a database corresponding to the address parameters.
If the permission setting information display database needs permission verification, the second url transmission parameters are sent to the web server corresponding to the target database website, so that the web server is instructed to acquire data corresponding to the second time parameters from the database corresponding to the address parameters based on the second url transmission parameters.
If the permission setting information display database does not need permission verification, the database is positioned based on the address parameters, and data corresponding to the second time parameters are obtained from the database.
Referring to fig. 3, fig. 3 is a schematic block diagram of a terminal device according to an embodiment of the present application. The terminal 300 in the present embodiment as shown in fig. 3 may include: one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processor 301, the input device 302, the output device 303, and the memory 304 communicate with each other via a communication bus 305. The memory 304 is used to store a computer program comprising program instructions. The processor 301 is configured to execute program instructions stored in the memory 304. Wherein the processor 301 is configured to invoke program instructions to perform the following functions of the modules/units in the above described device embodiments, such as the functions of the modules 21 to 25 shown in fig. 2.
It should be appreciated that in embodiments of the present application, the processor 301 may be a central processing unit (Central Processing Unit, CPU), which may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSPs), application specific integrated circuits (Application Specific Integrated Circuit, ASICs), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The input device 302 may include a touch pad, a fingerprint sensor (for collecting fingerprint information of a user and direction information of a fingerprint), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.
The memory 304 may include read only memory and random access memory and provides instructions and data to the processor 301. A portion of memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store information of device type.
In a specific implementation, the processor 301, the input device 302, and the output device 303 described in the embodiments of the present application may execute the implementation described in the first embodiment and the second embodiment of the network data acquisition method provided in the embodiments of the present application, and may also execute the implementation of the terminal described in the embodiments of the present application, which is not described herein again.
In another embodiment of the present application, a computer readable storage medium is provided, where the computer readable storage medium stores a computer program, where the computer program includes program instructions, where the program instructions, when executed by a processor, implement all or part of the procedures in the method embodiments described above, or may be implemented by instructing related hardware by the computer program, where the computer program may be stored in a computer readable storage medium, where the computer program, when executed by the processor, implements the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, executable files or in some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the content of the computer readable medium can be appropriately increased or decreased according to the requirements of the jurisdiction's jurisdiction and the patent practice, for example, in some jurisdictions, the computer readable medium does not include electrical carrier signals and telecommunication signals according to the jurisdiction and the patent practice.
The computer readable storage medium may be an internal storage unit of the terminal of any of the foregoing embodiments, such as a hard disk or a memory of the terminal. The computer readable storage medium may also be an external storage device of the terminal, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the terminal. Further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device. The computer-readable storage medium is used to store a computer program and other programs and data required for the terminal. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working procedures of the terminal and the unit described above may refer to the corresponding procedures in the foregoing method embodiments, which are not repeated herein.
In several embodiments provided by the present application, it should be understood that the disclosed terminal and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of elements is merely a logical functional division, and there may be additional divisions of actual implementation, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted, or not performed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via some interfaces or units, or may be an electrical, mechanical, or other form of connection.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The present application is not limited to the above embodiments, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the present application, and these modifications and substitutions are intended to be included in the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
Claims (6)
1. A method for acquiring network data, comprising:
obtaining html data generated by a target database website; the html data comprises a database address input by a user in a target database website and a first data acquisition time period;
generating a first url transmission parameter according to the html data; the first url parameter comprises an address parameter corresponding to the database address and a first time parameter corresponding to the first data acquisition time period;
correcting the first time parameter in the first url transmission based on a preset second time parameter to obtain a second url transmission; the second url parameter includes the address parameter and a second time parameter;
the correcting the first time parameter in the first url transmission parameter based on the preset second time parameter to obtain a second url transmission parameter includes:
acquiring a maximum acquisition time period corresponding to a target database website;
dividing the second time parameter into a plurality of sub-time parameters based on a multiple relation between the second time parameter and the maximum acquisition time period; wherein, the time period corresponding to each sub-time parameter is smaller than the maximum acquisition time period;
sequentially replacing the first time parameter in the first url transmission parameters with each sub-time parameter to obtain a plurality of replaced first url transmission parameters;
program synthesis is carried out on the plurality of replaced first url transmission parameters to obtain a second url transmission parameter; the program synthesis is a sequential connection of the steps on the program or a cyclic implementation of the steps on the program;
the time parameter in the url transmission corresponds to the data acquisition time period of the network data, and the first time parameter in the first url transmission is corrected, namely the first data acquisition time period is corrected;
acquiring data corresponding to the second time parameter from a database corresponding to the address parameter based on the second url parameter;
before correcting the first time parameter in the first url transmission based on a preset second time parameter to obtain a second url transmission, the method further comprises a step of determining the second time parameter; the determining of the second time parameter includes:
acquiring a second data acquisition time period input by a user;
and determining a second time parameter according to the second data acquisition time period.
2. The network data acquisition method of claim 1, wherein the determining a second time parameter from the second data acquisition time period comprises:
if the second data acquisition time period is empty, taking a preset time parameter as a second time parameter;
and if the second data acquisition time period is not empty, performing program calculation on the second data acquisition time period to obtain a second time parameter.
3. The network data acquisition method according to claim 1 or 2, wherein the acquiring the data corresponding to the second time parameter from the database corresponding to the address parameter based on the second url parameter includes:
acquiring authority setting information of a database corresponding to the address parameters;
if the authority setting information shows that the database needs authority verification, the second url transmission is sent to a web server corresponding to a target database website, so that the web server is instructed to acquire data corresponding to the second time parameter from the database corresponding to the address parameter based on the second url transmission;
if the authority setting information shows that the database does not need authority verification, positioning the database based on the address parameter, and acquiring data corresponding to the second time parameter from the database.
4. A network data acquisition device, comprising:
the first data acquisition module is used for acquiring html data generated by a target database website; the html data comprises a database address input by a user in a target database website and a first data acquisition time period;
the url parameter generation module is used for generating a first url parameter according to the html data; the first url parameter comprises an address parameter corresponding to the database address and a first time parameter corresponding to the first data acquisition time period;
the url parameter correction module is used for correcting the first time parameter in the first url transmission based on a preset second time parameter to obtain a second url transmission; the second url parameter includes the address parameter and a second time parameter; the time parameter in the url transmission corresponds to the data acquisition time period of the network data, and the first time parameter in the first url transmission is corrected, namely the first data acquisition time period is corrected; the correcting the first time parameter in the first url transmission parameter based on the preset second time parameter to obtain a second url transmission parameter includes:
acquiring a maximum acquisition time period corresponding to a target database website;
dividing the second time parameter into a plurality of sub-time parameters based on a multiple relation between the second time parameter and the maximum acquisition time period; wherein, the time period corresponding to each sub-time parameter is smaller than the maximum acquisition time period;
sequentially replacing the first time parameter in the first url transmission parameters with each sub-time parameter to obtain a plurality of replaced first url transmission parameters;
program synthesis is carried out on the plurality of replaced first url transmission parameters to obtain a second url transmission parameter; the program synthesis is a sequential connection of the steps on the program or a cyclic implementation of the steps on the program;
the second data acquisition module is used for acquiring data corresponding to the second time parameter from the database corresponding to the address parameter based on the second url parameter;
the second parameter determining module is used for determining a second time parameter before correcting the first time parameter in the first url transmission based on a preset second time parameter to obtain a second url transmission;
the second parameter determining module is specifically configured to obtain a second data acquisition time period input by the user, and determine a second time parameter according to the second data acquisition time period.
5. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 3 when the computer program is executed.
6. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110527696.3A CN113434787B (en) | 2021-05-14 | 2021-05-14 | Network data acquisition method and device and terminal equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110527696.3A CN113434787B (en) | 2021-05-14 | 2021-05-14 | Network data acquisition method and device and terminal equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113434787A CN113434787A (en) | 2021-09-24 |
| CN113434787B true CN113434787B (en) | 2023-11-07 |
Family
ID=77802343
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110527696.3A Active CN113434787B (en) | 2021-05-14 | 2021-05-14 | Network data acquisition method and device and terminal equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113434787B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020313A (en) * | 2013-01-08 | 2013-04-03 | 北京航空航天大学 | Capturing method based on detection of webpage refreshing period |
| CN103399933A (en) * | 2013-08-08 | 2013-11-20 | 人民搜索网络股份公司 | Method and system for grabbing webpage contents of network print media |
| CN107277274A (en) * | 2017-08-01 | 2017-10-20 | 北京宝福万通科技有限公司 | A kind of development history record terminal and development history recording method |
| CN111198641A (en) * | 2018-11-19 | 2020-05-26 | 珠海格力电器股份有限公司 | Message pushing method and terminal equipment |
| WO2021022689A1 (en) * | 2019-08-05 | 2021-02-11 | 苏州闻道网络科技股份有限公司 | Information collection method and apparatus |
| CN112685270A (en) * | 2020-12-24 | 2021-04-20 | 恒安嘉新（北京）科技股份公司 | System monitoring log acquisition method and device, electronic equipment and medium |
-
2021
- 2021-05-14 CN CN202110527696.3A patent/CN113434787B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020313A (en) * | 2013-01-08 | 2013-04-03 | 北京航空航天大学 | Capturing method based on detection of webpage refreshing period |
| CN103399933A (en) * | 2013-08-08 | 2013-11-20 | 人民搜索网络股份公司 | Method and system for grabbing webpage contents of network print media |
| CN107277274A (en) * | 2017-08-01 | 2017-10-20 | 北京宝福万通科技有限公司 | A kind of development history record terminal and development history recording method |
| CN111198641A (en) * | 2018-11-19 | 2020-05-26 | 珠海格力电器股份有限公司 | Message pushing method and terminal equipment |
| WO2021022689A1 (en) * | 2019-08-05 | 2021-02-11 | 苏州闻道网络科技股份有限公司 | Information collection method and apparatus |
| CN112685270A (en) * | 2020-12-24 | 2021-04-20 | 恒安嘉新（北京）科技股份公司 | System monitoring log acquisition method and device, electronic equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113434787A (en) | 2021-09-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110597511B (en) | Page automatic generation method, system, terminal equipment and storage medium | |
| CN107463348B (en) | Method and system for realizing Web end custom format printing based on B/S architecture | |
| CN107766040A (en) | A kind of method, apparatus and computer-readable recording medium for generating interface document | |
| CN109493048B (en) | Financial accounting method, device, equipment and storage medium based on block chain | |
| CN111694589B (en) | Upgrade package generation method, device, server and computer readable storage medium | |
| CN110493336A (en) | The method and apparatus at target user end are determined in a kind of gray scale issuing process | |
| CN112187921A (en) | Object file downloading method, device, system, server and storage medium | |
| CN109345401B (en) | Data synchronization method and device for product data, computer equipment and storage medium | |
| CN113434787B (en) | Network data acquisition method and device and terminal equipment | |
| CN113360300B (en) | Interface call link generation method, device, equipment and readable storage medium | |
| CN114510262A (en) | Interface request parameter verification method and device | |
| CN109582345A (en) | Report automatic generation method, device, storage medium and computer equipment | |
| CN109814916B (en) | IVR flow configuration method, device, storage medium and server | |
| CN115357319A (en) | User interface rendering method and device | |
| CN114185871A (en) | Data migration method, data migration device, medium, and electronic device | |
| CN113626392A (en) | Method and device for updating document data, electronic equipment and storage medium | |
| CN113239297A (en) | Message pushing method, system and storage medium | |
| CN112311716A (en) | Data access control method and device based on openstack and server | |
| CN110096266A (en) | A kind of characteristic processing method and device | |
| CN110400046A (en) | Monitor control index method of adjustment, device, equipment and computer readable storage medium | |
| CN111460340B (en) | Method and device for generating TOM system embedded page address | |
| CN113688586B (en) | Dynamic logic flow diagram generation method and device of equipment system and terminal equipment | |
| CN112835974B (en) | Method and device for credibly acquiring and storing evidence of data outside link | |
| CN117221049B (en) | Gateway data intelligent acquisition method and system | |
| CN116795905A (en) | Data visualization method, device, equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |