CN112417022B - Method, device, terminal and storage medium for acquiring data - Google Patents
Method, device, terminal and storage medium for acquiring data Download PDFInfo
- Publication number
- CN112417022B CN112417022B CN202011362965.7A CN202011362965A CN112417022B CN 112417022 B CN112417022 B CN 112417022B CN 202011362965 A CN202011362965 A CN 202011362965A CN 112417022 B CN112417022 B CN 112417022B
- Authority
- CN
- China
- Prior art keywords
- data
- data packet
- target
- packet
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000004590 computer program Methods 0.000 claims description 19
- 238000012545 processing Methods 0.000 abstract description 12
- 238000012544 monitoring process Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000001914 filtration Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Multimedia (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides a method, a device, a terminal and a storage medium for acquiring data, relates to the technical field of data processing, and can effectively improve the efficiency of data acquisition. The method for acquiring data comprises the following steps: if at least one interface is monitored to acquire first data, acquiring at least one first data packet corresponding to a target interface from the first data; and obtaining target transaction data according to the at least one first data packet.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method, an apparatus, a terminal, and a storage medium for acquiring data.
Background
With the development of information technology, data is becoming an indispensable part of the information field. Typically, a business system, such as a transaction system, is configured to obtain the required data in order to update the current data in the target system based on the data. In order to acquire data, the database in which related service data is stored is generally polled continuously to acquire the data, and then corresponding data processing is completed based on the acquired data.
However, when data is acquired by polling the interface library, in order to reduce the pressure of the database, the database cannot be frequently accessed, but when the time interval for accessing the interface library is too large, the data cannot be returned to the target system in time, so that the corresponding processing of the data is completed, and the experience of a user on the service system is greatly reduced. Therefore, the efficiency is not high when the data acquisition is performed by using the current data acquisition mode.
Disclosure of Invention
The embodiment of the application provides a method, a device, a terminal and a storage medium for acquiring data, which are used for solving the problem of low efficiency when the data is acquired by utilizing the current data acquisition mode.
In a first aspect, an embodiment of the present application provides a method for acquiring data, including:
if at least one interface is monitored to acquire first data, acquiring at least one first data packet corresponding to a target interface from the first data;
And obtaining target transaction data according to the at least one first data packet.
By adopting the method for acquiring the data, provided by the application, whether the interface acquires the first data or not is monitored in real time, and if at least one interface acquires the first data, at least one first data packet corresponding to the target interface is acquired from the first data, so that the condition that the designated object sends the data to the interface can be known conveniently. Further, according to the at least one first data packet, target transaction data is obtained, so that the data can be timely obtained, the data obtaining efficiency is improved, and the requirements of a data obtaining scene can be better met.
Optionally, the interface is identified by a port number and/or an IP address;
if the at least one interface is monitored to acquire the first data, acquiring at least one first data packet corresponding to the target interface from the first data, including:
and if the at least one interface is monitored to acquire first data, determining the at least one first data packet corresponding to the target interface from the first data according to the port number and/or the IP address of the target interface.
Optionally, the obtaining the target transaction data according to the at least one first data packet includes:
Determining at least one target data packet from the at least one first data packet, wherein the target data packet comprises the target transaction data;
and analyzing the at least one target data packet to obtain the target transaction data.
Optionally, the determining at least one target data packet from the at least one first data packet includes:
for each first data packet, acquiring a first data length of the first data packet, wherein the first data length is the length of data about the first data packet sent by a specified object to the target interface;
If the first data length is equal to the length of the first data packet, acquiring a second data length of the first data packet, wherein the second data length is the data length of a packet body in the first data packet;
And if the second data length is equal to the data length corresponding to the packet body in the first data packet, taking the first data packet as a target data packet.
Optionally, after determining that the first data length is equal to the length of the first data packet and the second data length is acquired, the method further includes:
If the second data length is greater than the data length corresponding to the packet body in the acquired first data packet and the data length of the packet body of the first data packet is greater than the preset data length, confirming that the first data packet lacks part of packet body data;
if at least one interface is monitored to acquire second data, acquiring at least one second data packet corresponding to the target interface from the second data;
Obtaining a residual data packet corresponding to the stored first data packet from the at least one second data packet, wherein the residual data packet is a data packet corresponding to part of inclusion data which is missing in inclusion data of the first data packet;
Combining the body of the residual data packet with the body of the stored first data packet to form a new data packet;
and taking the new data packet as the target data packet.
Optionally, the parsing the at least one target data packet to obtain the target transaction data includes:
aiming at each target data packet, acquiring the file type of the target data packet;
If the file type accords with the preset file type, acquiring a first identification parameter in transaction data included in the target data packet;
And if the first identification parameter is consistent with the preset first identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
Optionally, if the first identification parameter is consistent with the preset first identification parameter, obtaining the target transaction data in the target data packet according to the transaction parameter in the target data packet, including:
if the first identification parameter of the target data packet is consistent with the preset first identification parameter, acquiring a second identification parameter in transaction data included in the target data packet;
And if the second identification parameter is consistent with the preset second identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
In a second aspect, an embodiment of the present application provides an apparatus for acquiring data, including:
the first acquisition module is used for acquiring at least one first data packet corresponding to a target interface from the first data if at least one interface is monitored to acquire the first data;
And the second acquisition module is used for acquiring target transaction data according to the at least one first data packet.
In a third aspect, an embodiment of the present application provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the method when executing the computer program.
In a fourth aspect, embodiments of the present application provide a computer readable storage medium storing a computer program which, when executed by a processor, implements the method.
In a fifth aspect, an embodiment of the application provides a computer program product for, when run on a terminal, causing the terminal to perform the method of any of the first aspects above.
It will be appreciated that the advantages of the second to fifth aspects may be found in the relevant description of the first aspect, and are not described here again.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for acquiring data according to an embodiment of the present application;
FIG. 2 is an application scenario diagram of a method for acquiring data according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating a specific implementation of the method step S12 for acquiring data according to an embodiment of the present application;
FIG. 4 is a flowchart illustrating a specific implementation of the method step S21 for acquiring data according to an embodiment of the present application;
FIG. 5 is a flow chart of a method for acquiring data according to another embodiment of the present application;
FIG. 6 is a flowchart showing a specific implementation of the method step S22 for acquiring data according to another embodiment of the present application;
FIG. 7 is a schematic diagram of an apparatus for acquiring data according to an embodiment of the present application;
Fig. 8 is a schematic structural diagram of a terminal according to another embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
As used in the present specification and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".
Furthermore, the terms "first," "second," "third," and the like in the description of the present specification and in the appended claims, are used for distinguishing between descriptions and not necessarily for indicating or implying a relative importance.
Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.
In order to illustrate the technical scheme of the application, the following description is made by specific examples.
Referring to fig. 1, fig. 1 is a flowchart of a method for acquiring data according to an embodiment of the present application. In this embodiment, the method for acquiring data is used to acquire target transaction data in a process that the service system needs to acquire data for processing, where an execution body is a terminal. The terminal comprises, but is not limited to, a mobile terminal such as a smart phone, a tablet personal computer, a wearable device and the like, and can also be a desktop computer, a robot, a server and the like.
The method for acquiring data as shown in fig. 1 comprises the steps of:
S11: and if the at least one interface is monitored to acquire the first data, acquiring at least one first data packet corresponding to the target interface from the first data.
In step S11, the interface is a data specification for realizing data acquisition and transmission. It will be appreciated that data may be retrieved based on the interface and stored in a designated database or sent out through the interface.
The target interface is used to describe the interface that receives the target transaction data.
The first data is data which is returned by the data source and is used for sending a data acquisition request to the data source through the interfaces based on the data requirements corresponding to the interfaces. It will be appreciated that the data acquisition requirements of the respective interfaces may be described by the first data.
The first data packet is used for describing the data requirement corresponding to the target interface. It will be appreciated that transaction data describing the data acquisition requirements corresponding to the target interface is recorded within the data packet. In some embodiments, the first data packet is a data packet corresponding to a table data flow protocol (Tabular DATA STREAM, TDS protocol).
It should be noted that, in order to confirm whether the target interface acquires the first data, so as to acquire at least one first data packet corresponding to the target interface from the first data corresponding to the target interface, before monitoring the at least one interface to acquire the first data, the environment for monitoring the at least one interface to acquire the first data in the terminal is configured based on the code script, and further, based on controlling the running of the code script, the first data acquired by the at least one interface is monitored. Wherein, the network equipment used for obtaining the data packet based on the code script configures the Internet protocol address (Internet Protocol Address, IP address), name, working mode of the interface. It can be understood that when the code script is controlled to run, the first data returned by each data source can be captured based on the code script, and further, each interface is confirmed to acquire the first data, so that at least one first data packet corresponding to the target interface is acquired from the first data to be acquired by the target interface.
For example, before the at least one interface is monitored to obtain the first data, an environment for monitoring the at least one interface to obtain the first data in the terminal is configured based on the code script, that is, the network device interface corresponding to the terminal is obtained based on the code script, and the working mode of the network device interface is configured as a target mode, for example, the network device interface is configured as a promiscuous mode, so that in the working mode, the network device interface can capture all data in a communication network connected with the terminal, that is, can monitor the first data transmitted back to the at least one interface by the data source.
In this embodiment, when at least one interface is monitored to obtain the first data, at least one first data packet corresponding to the target interface is obtained from the first data, and because the first data can be used to describe the data obtaining requirement of the target interface, at least one first data packet corresponding to the target interface is obtained from the first data, so that the first data packet describing the data requirement of the target system can be better obtained. In addition, in this embodiment, when at least one interface is monitored to obtain the first data, the first data packet corresponding to the target interface is directly obtained from the first data corresponding to the interface, so that the corresponding data obtaining requirement is met, the database corresponding to the interface is not required to be accessed again, the first data packet corresponding to the data obtaining requirement is obtained, the pressure of the database can be effectively reduced, and the data meeting the target system can be obtained without accessing the database, so that the data obtaining efficiency is higher.
As to when to obtain at least one first data packet corresponding to the target interface from the first data, the following two scenarios may be included, but not limited to.
Scene 1: when a data acquisition request of a target system is received, starting to monitor whether at least one interface acquires first data, and if the at least one interface acquires the first data, acquiring at least one first data packet corresponding to the target interface from the first data.
For example, when a user performs a transaction operation in the transaction system, that is, the transaction system needs to acquire transaction data corresponding to the transaction operation of the user from a transaction program corresponding to the transaction exchange. Therefore, in order to efficiently acquire the transaction data, the first data acquired by at least one interface is monitored, so that at least one first data packet corresponding to the target interface is acquired from the first data.
Scene 2: in order to update data of the trading system, data of the trading market needs to be periodically acquired from a trading program corresponding to the trading exchange. Therefore, in order to efficiently acquire the transaction data, the first data acquired by at least one interface is monitored, so that at least one first data packet corresponding to the target interface is acquired from the first data.
It should be understood that in practical application, the method may be based on the target system acquiring at least one first data packet corresponding to the target interface from the first data, or return to the target system after the terminal connected to the target system acquires at least one first data packet corresponding to the target interface from the first data.
In an embodiment, the first data is acquired by periodically monitoring at least one interface, and when the first data is acquired by monitoring at least one interface, at least one first data packet corresponding to a target interface is acquired from the first data.
In this embodiment, the first data is acquired by cyclically monitoring at least one interface, that is, when at least one first data packet corresponding to the target interface is acquired from the first data, a new round of monitoring is started, that is, monitoring is restarted to acquire the first data by at least one interface.
As a possible implementation manner of this embodiment, the interface is identified by a port number and/or an IP address;
if the at least one interface is monitored to acquire the first data, acquiring at least one first data packet corresponding to the target interface from the first data, including:
and if the at least one interface is monitored to acquire first data, determining the at least one first data packet corresponding to the target interface from the first data according to the port number and/or the IP address of the target interface.
In the present embodiment, the port number is an identification configured to distinguish the respective interfaces, the identification including a number. The service content provided by the interface can be known through the port number and the definition of the port number. The IP address is a logical address of a terminal corresponding to the interface on the internet.
In this embodiment, each interface has a unique port number and/or IP address, and in order to accurately send data to each interface, the data packet includes the port number and/or IP address of the interface. Therefore, in order to determine the at least one first data packet corresponding to the target interface from the first data corresponding to the target interface, the at least one first data packet may be matched with the port number and/or the IP address included in the data packet according to the port number and/or the IP address of the target interface, if the matching is successful, the data packet is a data packet to be received by the target interface, and the data packet may include transaction data of a data requirement of the target system.
Specifically, the port number and/or the IP address of each data packet in the first data are obtained, the port number and/or the IP address of each data packet are respectively matched with the port number and/or the IP address of the target interface, and the first data packet corresponding to the target interface is obtained according to the matching result. It can be understood that, because the first data corresponding to the target interface is still stored in the database corresponding to the target interface through the target interface, the first data packet corresponding to the target interface is obtained as a copied version.
For example, as shown in fig. 2, when the specified object is a newspaper disc program of the exchange, the newspaper disc program needs to write back the transaction data to the interface library based on the port number and the IP address of the interface, so as to store the transaction data into the database through the interface in the interface library. Meanwhile, the terminal is configured with a code script for acquiring the transaction data packet so as to return the target transaction data in the acquired transaction data packet to the target system through the code script, and the target system can finish relevant data processing according to the target transaction data. The network interface can be configured into a preset mode through a configured code script so that a terminal can capture all data in a communication network based on the network interface in order to acquire the transaction data packet of the newspaper-disc program write-back interface library, and a corresponding data packet filtering expression is configured in the code script so that the transaction data packet of the non-newspaper-disc program write-back to the interface library is filtered through the filtering expression in order to accurately acquire the transaction data packet of the newspaper-disc program write-back interface library. For example, the filtering expression configured in the code script is "tcp & dst port number & dst host port & IP address and len >60", based on the filtering expression, the data packet of the non-disc-reporting program write-back interface library can be filtered, but only the transaction data packet of the disc-reporting program write-back interface library is reserved, that is, when at least one interface is monitored to acquire network data, the at least one transaction data packet corresponding to the target interface in the interface library can be acquired from the network data according to the port number and the IP address of the target interface, and the purpose is to acquire the transaction data through the code script configured in the terminal before the disc-reporting program writes the transaction data into the interface library, that is, by bypassing the time-consuming operation of the write-interface library, the transaction data can be directly acquired, so that the efficiency of acquiring the transaction data by the target system is greatly improved, and meanwhile, the pressure of the data corresponding to the interface library can be effectively reduced because the target system does not need to poll the interface library frequently.
S12: and obtaining target transaction data according to the at least one data packet.
In step S12, the target transaction data is transaction data satisfying the data acquisition requirement. For example, the target transaction data is transaction data for updating a target transaction unit in the transaction system.
In this embodiment, because the acquired at least one data packet is a data packet meeting the data acquisition requirement corresponding to the target interface, the target transaction data may be obtained by analyzing the at least one data packet according to a preset policy after acquiring the at least one first data packet corresponding to the target interface from the first data. The preset strategy is used for describing a logic process of analyzing at least one data packet to obtain target transaction data.
In one embodiment, a transaction group package is formed based on the target transaction data, and the transaction group package is returned to the target system so that the target system can perform corresponding data processing according to the transaction group package.
As shown in fig. 3, as a possible implementation manner of this embodiment, the obtaining, according to the at least one first data packet, target transaction data includes:
S21: and determining at least one target data packet from the at least one first data packet, wherein the target data packet comprises the target transaction data.
S22: and analyzing the at least one target data packet to obtain the target transaction data.
In this embodiment, the target data packet records a data packet satisfying the target data acquisition requirement.
And determining at least one target data packet from the at least one first data packet, namely determining the data packet recorded with target transaction data in the at least one first data packet describing various data acquisition requirements corresponding to the target interface so as to be convenient for directly acquiring the target data packet to return to a target system or a terminal connected with the target system, so as to return the target data packet to the target system through the terminal, and finally finishing data processing, such as updating of the transaction data or display of the data, through the target system according to the target transaction data recorded in the target data packet.
As shown in fig. 4, as a possible implementation manner of this embodiment, the determining at least one target data packet from the at least one first data packet includes:
S31: and acquiring a first data length of each first data packet, wherein the first data length is the length of data about the first data packet, which is sent to the target interface by a specified object.
S32: and if the first data length is equal to the length of the first data packet, acquiring a second data length of the first data packet, wherein the second data length is the data length of a packet body in the first data packet.
S33: and if the second data length is equal to the data length corresponding to the packet body in the first data packet, taking the first data packet as a target data packet.
In this embodiment, the specified object is a terminal that directly returns the first data to the interface or an intermediate terminal that forwards the first data sent by the data source. Wherein the designated object is a different terminal than the terminal that acquired the first data packet.
It will be appreciated that the data packet has a particular data format and that the identification field in the data packet is predefined as the body in which the transaction data describing the data acquisition requirements is recorded.
The first data packet is a data packet of a TDS protocol, and the data fields included in the data format of the data packet include a data packet header and a data field, where the data field is used as a packet body.
In order to confirm whether the data corresponding to the obtained target interface is complete, when at least one target data packet is determined from at least one first data packet, a first data length of the first data packet is obtained for each first data packet, so that the length of data about the first data packet, which is sent to the target interface by a specified object, is known through the first data length. If the first data length is equal to the length of the first data packet, the acquisition of a complete data packet is indicated. And further acquiring a second data length used for describing the data length of the packet body in the first data packet so as to determine whether the packet body in the acquired first data packet is complete or not through the second data length, and when the second data length is equal to the data length corresponding to the packet body in the first data packet, indicating that the packet body of the acquired first data packet is complete, and taking the first data packet as a target data packet.
In an embodiment, the data packet has a specific data format, the first data length is recorded in a specific field of the data format, and the first data length of the data packet is extracted according to a preset data length acquisition strategy.
The preset data length obtaining strategy is used for describing a logic process for obtaining the first data length. For example, the data format of the data packet is identified, and when the data format is consistent with the data format recorded in the preset data length acquisition strategy, the data recorded in the preset field is extracted, and the data is used as the first data length.
As shown in fig. 5, as a possible implementation manner of this embodiment, after determining that the first data length is equal to the length of the first data packet and obtaining the second data length, the method further includes:
S41: and if the second data length is greater than the data length corresponding to the packet body in the acquired first data packet and the data length of the packet body of the first data packet is greater than the preset data length, confirming that the first data packet lacks part of packet body data.
S42: and if the at least one interface is monitored to acquire the second data, acquiring at least one second data packet corresponding to the target interface from the second data.
S43: and acquiring a residual data packet corresponding to the stored first data packet from the at least one second data packet, wherein the residual data packet is a data packet corresponding to part of inclusion data which is missing in inclusion data of the first data packet.
S44: and combining the inclusion of the residual data packet with the inclusion of the stored first data packet to form a new data packet.
S45: and taking the new data packet as the target data packet.
In this embodiment, the first data is data that sends a data acquisition request to the data source based on the data requirements corresponding to the respective interfaces, and is returned by the data source. It will be appreciated that the data requirements of the respective interfaces may be described by the first data.
The second data packet is used for describing the data requirement corresponding to the target interface. It will be appreciated that transaction data describing the data acquisition requirements corresponding to the target interface is recorded within the data packet.
The preset data length is used for describing the minimum packet body length meeting the data retention requirement. For example, the preset data length is defined to be greater than 0, otherwise, when the data length of the packet body is equal to 0, the first data packet corresponding to the packet body is directly discarded.
The remaining data packets are data packets corresponding to part of the packet body data which are missing in the packet body data of the first data packet.
It can be understood that, because the second data length is the data length of the packet body in the first data packet, when the second data length is greater than the acquired data length corresponding to the packet body in the first data packet, that is, the first data packet acquired from the first data corresponding to the target interface, the packet body that completely records one piece of information is not acquired, that is, the first data packet lacks partial packet body data. Therefore, it is necessary to continue to monitor the second data acquired by the interface, so as to acquire at least one second data packet corresponding to the target interface from the second data, and acquire the remaining data packets corresponding to the stored first data packets from the at least one second data packet. Further, combining the body of the residual data packet with the stored body of the first data packet to form a new data packet; and taking the new data packet as the target data packet.
In this embodiment, since it is not known when a packet corresponding to a portion of packet body data lacking the packet body data of the first packet will be acquired, the first packet may be stored in a preset storage space, so that when a remaining packet is acquired, the first packet is acquired from the preset storage space, and the packet body of the remaining packet and the packet body of the stored first packet are combined to form a new packet.
As shown in fig. 6, as a possible implementation manner of this embodiment, the parsing the at least one target data packet to obtain the target transaction data includes:
s51: aiming at each target data packet, acquiring the file type of the target data packet;
s52: if the file type accords with the preset file type, acquiring a first identification parameter in transaction data included in the target data packet;
s53: and if the first identification parameter is consistent with the preset first identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
In the present embodiment, the file type is a coding scheme of information used to describe stored information.
The first identification parameter is a distinguishing parameter describing whether the data packet satisfies the target data acquisition request. For example, the first identification parameter is "INSERT INTO transaction table name".
The transaction data is used for describing data containing identification parameters and transaction parameters, and whether the data packet is a target data packet meeting certain data acquisition requirements can be determined based on the identification parameters in the transaction data.
The transaction parameters are used to describe the data acquisition requirements. It will be appreciated that the target transaction parameters in the target data packet may be derived based on the transaction parameters, and that the target transaction parameters are part of the transaction parameters that do not include the identification parameters in the transaction parameters, since the target transaction parameters are data describing only the transaction changes.
It may be understood that the preset first identification parameter is data for identifying a specific data acquisition request, so that the first identification parameter is matched with the preset first identification parameter, if the matching result indicates that the matching is consistent, the target data packet is a data packet corresponding to the specific data acquisition request, and the transaction parameter in the target data packet may satisfy the corresponding data acquisition request, so that the target transaction data in the target data packet may be obtained according to the transaction parameter in the target data packet, that is, the target transaction data is extracted from the transaction parameter of the target data packet.
As a possible implementation manner of this embodiment, if the first identification parameter is consistent with a preset first identification parameter, obtaining, according to a transaction parameter in the target data packet, target transaction data in the target data packet includes:
if the first identification parameter of the target data packet is consistent with the preset first identification parameter, acquiring a second identification parameter in transaction data included in the target data packet;
And if the second identification parameter is consistent with the preset second identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
In this embodiment, the second identification parameter is used to describe a parameter describing whether the data packet satisfies the target data acquisition request. For example, whether the transaction data in the target data packet is the data corresponding to the target transaction unit is determined through the second identification parameter. It may be understood that when the first identification parameter of the target data packet is consistent with the preset first identification parameter, it indicates that the transaction data in the target data packet is the required transaction data, but in order to confirm whether the transaction data is the data corresponding to the target transaction unit in the target system, it is also necessary to acquire the second identification parameter in the transaction data included in the target data packet, so as to determine whether the transaction data is the transaction data required by the target transaction unit of the target system through the second identification parameter. If the second identification parameter is consistent with the preset second identification parameter, the target data packet is a data packet meeting the data acquisition requirement, so that the target transaction data in the target data packet can be obtained according to the transaction parameter in the target data packet, and the target system can conveniently complete corresponding data processing according to the target transaction data.
By adopting the method for acquiring data provided by the embodiment of the application, whether the interface acquires the first data or not is monitored in real time, and if at least one interface acquires the first data, at least one first data packet corresponding to the target interface is acquired from the first data, so that the condition that the designated object sends data to the interface can be known conveniently. Further, according to the at least one first data packet, target transaction data is obtained, so that the data can be timely obtained, the data obtaining efficiency is improved, and the requirements of a data obtaining scene can be better met.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application.
Corresponding to the method for acquiring data described in the above embodiments, fig. 7 shows a block diagram of an apparatus for acquiring data provided in an embodiment of the present application, and for convenience of explanation, only a portion related to the embodiment of the present application is shown.
Referring to fig. 7, the apparatus 100 includes:
the first obtaining module 101 is configured to obtain, if at least one interface is monitored to obtain first data, at least one first data packet corresponding to a target interface from the first data;
the second obtaining module 102 is configured to obtain target transaction data according to the at least one first data packet.
Optionally, the interface is identified by a port number and/or an IP address.
The first obtaining module 101 is further configured to determine, if the at least one interface is monitored to obtain first data, the at least one first data packet corresponding to the target interface from the first data according to the port number and/or the IP address of the target interface.
Optionally, the second obtaining module 102 further includes a determining unit and an analyzing unit.
The determining unit is configured to determine at least one target data packet from the at least one first data packet, where the target data packet includes the target transaction data.
The analyzing unit is configured to analyze the at least one target data packet to obtain the target transaction data.
Optionally, the determining unit is further configured to obtain, for each first data packet, a first data length of the first data packet, where the first data length is a length of data about the first data packet sent by a specified object to the target interface; if the first data length is equal to the length of the first data packet, acquiring a second data length of the first data packet, wherein the second data length is the data length of a packet body in the first data packet; and if the second data length is equal to the data length corresponding to the packet body in the first data packet, taking the first data packet as a target data packet.
Optionally, the determining unit is further configured to, after determining that the first data length is equal to the length of the first data packet and the second data length is acquired, further include: and if the second data length is greater than the data length corresponding to the packet body in the acquired first data packet and the data length of the packet body of the first data packet is greater than the preset data length, confirming that the first data packet lacks part of packet body data.
The first obtaining module 101 is further configured to obtain, if at least one interface is monitored to obtain second data, at least one second data packet corresponding to the target interface from the second data.
The determining unit is further configured to obtain a remaining data packet corresponding to the stored first data packet from the at least one second data packet, where the remaining data packet is a data packet corresponding to a part of inclusion data that is missing in inclusion data of the first data packet; combining the body of the residual data packet with the body of the stored first data packet to form a new data packet; and taking the new data packet as the target data packet.
Optionally, the parsing unit is further configured to obtain, for each target data packet, a file type of the target data packet; if the file type accords with the preset file type, acquiring a first identification parameter in transaction data included in the target data packet; and if the first identification parameter is consistent with the preset first identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
Optionally, the parsing unit is further configured to obtain a second identification parameter in the transaction data included in the target data packet if the first identification parameter of the target data packet is consistent with a preset first identification parameter; and if the second identification parameter is consistent with the preset second identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
Fig. 8 is a schematic structural diagram of a terminal according to an embodiment of the present application. As shown in fig. 8, the terminal 7 of this embodiment includes: at least one processor 70 (only one processor is shown in fig. 8), a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, the processor 70 implementing the steps in any of the various method embodiments described above when executing the computer program 72.
The terminal 7 may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The terminal may include, but is not limited to, a processor 70, a memory 71. It will be appreciated by those skilled in the art that fig. 8 is merely an example of the terminal 7 and is not limiting of the terminal 7, and may include more or fewer components than shown, or may combine some components, or may include different components, such as input-output devices, network access devices, etc.
The Processor 70 may be a central processing unit (Central Processing Unit, CPU), and the Processor 70 may be another general purpose Processor, a digital signal Processor (DIGITAL SIGNAL Processor, DSP), an Application SPECIFIC INTEGRATED Circuit (ASIC), an off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 71 may in some embodiments be an internal storage unit of the terminal 7, such as a hard disk or a memory of the terminal 7. The memory 71 may in other embodiments also be an external storage device of the terminal 7, such as a plug-in hard disk provided on the terminal 7, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like. Further, the memory 71 may also include both an internal storage unit and an external storage device of the terminal 7. The memory 71 is used for storing an operating system, application programs, boot loader (BootLoader), data, other programs, etc., such as program codes of the computer program. The memory 71 may also be used for temporarily storing data that has been output or is to be output.
It should be noted that, because the content of information interaction and execution process between the above devices/units is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method embodiment section, and will not be described herein.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
Embodiments of the present application also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements steps for implementing the various method embodiments described above.
Embodiments of the present application provide a computer program product which, when run on a terminal, causes the terminal to perform steps that enable the implementation of the method embodiments described above.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiments, and may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to an apparatus/terminal, a recording medium, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Such as a U-disk, removable hard disk, magnetic or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal and method may be implemented in other manners. For example, the apparatus/terminal embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (8)
1. A method for acquiring data, comprising:
if at least one interface is monitored to acquire first data, acquiring at least one first data packet corresponding to a target interface from the first data;
Obtaining target transaction data according to the at least one first data packet, wherein the target transaction data comprises: determining at least one target data packet from the at least one first data packet, wherein the target data packet comprises the target transaction data; analyzing the at least one target data packet to obtain the target transaction data;
The analyzing the at least one target data packet to obtain the target transaction data includes:
aiming at each target data packet, acquiring the file type of the target data packet;
If the file type accords with the preset file type, acquiring a first identification parameter in transaction data included in the target data packet;
And if the first identification parameter is consistent with the preset first identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
2. The method of claim 1, wherein the interface is identified by a port number and/or an IP address;
if the at least one interface is monitored to acquire the first data, acquiring at least one first data packet corresponding to the target interface from the first data, including:
and if the at least one interface is monitored to acquire first data, determining the at least one first data packet corresponding to the target interface from the first data according to the port number and/or the IP address of the target interface.
3. The method of claim 1, wherein said determining at least one target data packet from said at least one first data packet comprises:
for each first data packet, acquiring a first data length of the first data packet, wherein the first data length is the length of data about the first data packet sent by a specified object to the target interface;
If the first data length is equal to the length of the first data packet, acquiring a second data length of the first data packet, wherein the second data length is the data length of a packet body in the first data packet;
And if the second data length is equal to the data length corresponding to the packet body in the first data packet, taking the first data packet as a target data packet.
4. The method of claim 3, further comprising, after determining that the first data length is equal to the length of the first data packet and obtaining the second data length:
If the second data length is greater than the data length corresponding to the packet body in the acquired first data packet and the data length of the packet body of the first data packet is greater than the preset data length, confirming that the first data packet lacks part of packet body data;
if at least one interface is monitored to acquire second data, acquiring at least one second data packet corresponding to the target interface from the second data;
Obtaining a residual data packet corresponding to the stored first data packet from the at least one second data packet, wherein the residual data packet is a data packet corresponding to part of inclusion data which is missing in inclusion data of the first data packet;
Combining the body of the residual data packet with the body of the stored first data packet to form a new data packet;
and taking the new data packet as the target data packet.
5. The method of claim 1, wherein if the first identification parameter is consistent with a preset first identification parameter, obtaining the target transaction data in the target data packet according to the transaction parameter in the target data packet, comprises:
if the first identification parameter of the target data packet is consistent with the preset first identification parameter, acquiring a second identification parameter in transaction data included in the target data packet;
And if the second identification parameter is consistent with the preset second identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
6. An apparatus for acquiring data, comprising:
the first acquisition module is used for acquiring at least one first data packet corresponding to a target interface from the first data if at least one interface is monitored to acquire the first data;
The second acquisition module is used for acquiring target transaction data according to the at least one first data packet;
a second acquisition module comprising: a determining unit, configured to determine at least one target data packet from the at least one first data packet, where the target data packet includes the target transaction data; the analysis unit is used for analyzing the at least one target data packet to obtain the target transaction data;
The analyzing unit is further configured to obtain, for each target data packet, a file type of the target data packet; if the file type accords with the preset file type, acquiring a first identification parameter in transaction data included in the target data packet; and if the first identification parameter is consistent with the preset first identification parameter, obtaining target transaction data in the target data packet according to the transaction parameter in the target data packet.
7. A terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the method of any one of claims 1 to 5 when the computer program is executed.
8. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011362965.7A CN112417022B (en) | 2020-11-27 | 2020-11-27 | Method, device, terminal and storage medium for acquiring data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011362965.7A CN112417022B (en) | 2020-11-27 | 2020-11-27 | Method, device, terminal and storage medium for acquiring data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112417022A CN112417022A (en) | 2021-02-26 |
| CN112417022B true CN112417022B (en) | 2024-06-07 |
Family
ID=74842738
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011362965.7A Active CN112417022B (en) | 2020-11-27 | 2020-11-27 | Method, device, terminal and storage medium for acquiring data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112417022B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101136812A (en) * | 2007-09-28 | 2008-03-05 | 北京北方烽火科技有限公司 | Independent WIMAX base station network performance testing method and system |
| CN107682125A (en) * | 2017-11-13 | 2018-02-09 | 平安科技（深圳）有限公司 | A kind of data capture method, device, computer equipment and storage medium |
| WO2019072103A1 (en) * | 2017-10-11 | 2019-04-18 | 阿里巴巴集团控股有限公司 | Method and apparatus for obtaining target data |
| CN109800259A (en) * | 2018-12-11 | 2019-05-24 | 深圳市金证科技股份有限公司 | Collecting method, device and terminal device |
| KR102009336B1 (en) * | 2018-04-25 | 2019-08-12 | 주식회사쿠콘 | Apparatus, method and computer program for cloud scrapping using pre-scrapped bigdata |
| CN111813573A (en) * | 2020-06-29 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Communication method of management platform and robot software and related equipment thereof |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10445335B2 (en) * | 2015-07-31 | 2019-10-15 | Aetna Inc. | Computing environment connectivity system |
| US10067002B2 (en) * | 2016-06-08 | 2018-09-04 | Mark Smith | Retrofit gas kiln atmospheric monitoring system |
| CN110007952B (en) * | 2018-01-02 | 2022-07-12 | 斑马智行网络(香港)有限公司 | Data processing method, device, equipment and machine readable medium |
-
2020
- 2020-11-27 CN CN202011362965.7A patent/CN112417022B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101136812A (en) * | 2007-09-28 | 2008-03-05 | 北京北方烽火科技有限公司 | Independent WIMAX base station network performance testing method and system |
| WO2019072103A1 (en) * | 2017-10-11 | 2019-04-18 | 阿里巴巴集团控股有限公司 | Method and apparatus for obtaining target data |
| CN109947795A (en) * | 2017-10-11 | 2019-06-28 | 阿里巴巴集团控股有限公司 | The acquisition methods and device of target data |
| CN107682125A (en) * | 2017-11-13 | 2018-02-09 | 平安科技（深圳）有限公司 | A kind of data capture method, device, computer equipment and storage medium |
| WO2019090997A1 (en) * | 2017-11-13 | 2019-05-16 | 平安科技（深圳）有限公司 | Data acquisition method and device, computer device and storage medium |
| KR102009336B1 (en) * | 2018-04-25 | 2019-08-12 | 주식회사쿠콘 | Apparatus, method and computer program for cloud scrapping using pre-scrapped bigdata |
| CN109800259A (en) * | 2018-12-11 | 2019-05-24 | 深圳市金证科技股份有限公司 | Collecting method, device and terminal device |
| CN111813573A (en) * | 2020-06-29 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Communication method of management platform and robot software and related equipment thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112417022A (en) | 2021-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109582655B (en) | Method and device for positioning system log and computer readable storage medium | |
| CN109656487B (en) | Data processing method, device, equipment and storage medium | |
| CN109905292B (en) | Terminal equipment identification method, system and storage medium | |
| CN112135185A (en) | Screen projection method, device, equipment and storage medium based on NFC technology | |
| CN110851334B (en) | Flow statistics method, electronic equipment, system and medium | |
| CN114598512A (en) | Honeypot-based network security guarantee method and device and terminal equipment | |
| CN112671878B (en) | Block chain information subscription method, device, server and storage medium | |
| CN106790380A (en) | Data reporting method and device | |
| CN109787963A (en) | Business handling method, apparatus, computer equipment and storage medium | |
| CN112417022B (en) | Method, device, terminal and storage medium for acquiring data | |
| CN112416557A (en) | Method and device for determining call relation, storage medium and electronic device | |
| CN111212153A (en) | IP address checking method, device, terminal equipment and storage medium | |
| CN114298699B (en) | Method for generating and acquiring non-homogeneous general evidence and device | |
| CN115952398A (en) | Data uploading statistical calculation method and system based on Internet of things and storage medium | |
| CN113923190B (en) | Equipment identification jump identification method and device, server and storage medium | |
| CN115510012A (en) | Service data processing method and device, computer equipment and storage medium | |
| CN110868360B (en) | Flow statistics method, electronic equipment, system and medium | |
| CN108683716B (en) | business logic learning and protecting method and learning and protecting device based on big data | |
| US20210120037A1 (en) | Monitoring apparatus, monitoring method, and program | |
| CN111367750A (en) | Exception handling method, device and equipment | |
| WO2014173129A1 (en) | Mobile terminal flow identification method and apparatus | |
| CN115038089B (en) | Multi-terminal data monitoring and collecting method based on information extraction | |
| CN116645182A (en) | Banking system, data acquisition method thereof, server and medium | |
| CN117667604B (en) | Data monitoring method, device, electronic equipment and storage medium for tracking event | |
| CN112395600B (en) | Misinformation removing method, device and equipment for malicious behaviors |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |