CN111198859B - Data processing method, device, electronic equipment and computer readable storage medium - Google Patents
Data processing method, device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111198859B CN111198859B CN201811369129.4A CN201811369129A CN111198859B CN 111198859 B CN111198859 B CN 111198859B CN 201811369129 A CN201811369129 A CN 201811369129A CN 111198859 B CN111198859 B CN 111198859B
- Authority
- CN
- China
- Prior art keywords
- data
- format
- service parameters
- log
- converted
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title abstract description 18
- 238000006243 chemical reaction Methods 0.000 claims abstract description 52
- 238000012545 processing Methods 0.000 claims abstract description 32
- 238000000034 method Methods 0.000 claims abstract description 31
- 238000004590 computer program Methods 0.000 claims description 8
- 238000011022 operating instruction Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 description 12
- 238000013500 data storage Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000011156 evaluation Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure provides a data processing method, apparatus, electronic device, and computer readable storage medium, the method comprising: acquiring log data from a plurality of service servers, wherein the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different; and carrying out format conversion on the log data according to a preset first data format to obtain converted data. In the embodiment of the disclosure, the data format of the log data acquired from the plurality of service servers can be converted into the preconfigured data format, so that the data with uniform format is obtained and stored, and the log data acquired from the plurality of service servers are log data with different formats, so that subsequent data processing is inconvenient.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method, a data processing device, an electronic apparatus, and a computer readable storage medium.
Background
For online products, different service parameters are required to be monitored according to different service requirements, so that service performance or user service conditions of the products are mastered based on data corresponding to the service parameters obtained through monitoring.
In the prior art, when index data corresponding to different service parameters are monitored, the index data corresponding to each service parameter needs to be obtained from one or more servers for monitoring the index data of the different service parameters, and when the index data of each service scene is monitored by a plurality of servers, the data amount of the index data corresponding to the service parameters obtained from each server is large, and the data needs to be obtained from different servers, so that the data obtaining efficiency is low.
In the prior art, obtaining index data corresponding to service parameters from different servers is slow.
Disclosure of Invention
The present disclosure aims to solve at least one of the above technical drawbacks and improve data processing efficiency. The technical scheme adopted by the present disclosure is as follows:
in a first aspect, the present disclosure provides a data processing method, the method comprising:
acquiring log data from a plurality of service servers, wherein the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different;
and carrying out format conversion on the log data according to a preset first data format to obtain converted data.
In an embodiment of the present disclosure, format converting log data according to a preconfigured first data format includes:
determining a data conversion rule for converting the second data format into the first data format according to the first data format and the second data format of the log data;
and converting the log data according to the data conversion rule.
In an embodiment of the present disclosure, the log data includes index data of a monitored service parameter, and converting the log data according to a data conversion rule includes:
extracting index data corresponding to the target service parameters from the log data according to the parameter configuration information, wherein the parameter configuration information comprises the target service parameters, and the monitored service parameters comprise the target service parameters;
and converting the target service parameters and index data corresponding to the target service parameters according to the data conversion rules.
In the embodiment of the disclosure, the parameter configuration information further includes an indirect service parameter, and index data corresponding to the indirect service parameter is determined by index data corresponding to at least two monitored service parameters;
the method further comprises the steps of:
and obtaining index data corresponding to the indirect service parameters according to the index data corresponding to the at least two monitored service parameters corresponding to the indirect service parameters.
In an embodiment of the present disclosure, the method further comprises:
and storing the converted data according to the data quantity of the converted data.
In an embodiment of the present disclosure, storing converted data according to a data amount of the converted data includes:
if the data quantity of the converted data is larger than a preset data quantity threshold value, storing the converted data according to a first storage mode;
if the data quantity of the converted data is not greater than the data quantity threshold value, storing the converted data according to a second storage mode;
the storage space occupied by the data of the data quantity threshold value stored in the second storage mode is larger than the storage space occupied by the data of the data quantity threshold value stored in the first storage mode.
In an embodiment of the present disclosure, the method further comprises:
receiving a data query request, wherein the data query request comprises target service parameters, and a request statement format of the data query request corresponds to a first data format;
and obtaining index data corresponding to the target service parameters from the converted data according to the data query request, and displaying the index data corresponding to the target service parameters.
In a second aspect, the present disclosure provides a data processing apparatus comprising:
the system comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring log data from a plurality of service servers, and the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different;
the data conversion module is used for carrying out format conversion on the log data according to a preset first data format to obtain converted data.
In an embodiment of the disclosure, when the data conversion module performs format conversion on log data according to a preconfigured first data format, the data conversion module is specifically configured to:
determining a data conversion rule for converting the second data format into the first data format according to the first data format and the second data format of the log data;
and converting the log data according to the data conversion rule.
In the embodiment of the disclosure, the log data includes index data of the monitored service parameter, and the data conversion module is specifically configured to:
extracting index data corresponding to the target service parameters from the log data according to the parameter configuration information, wherein the parameter configuration information comprises the target service parameters, and the monitored service parameters comprise the target service parameters;
and converting the target service parameters and index data corresponding to the target service parameters according to the data conversion rules.
In the embodiment of the disclosure, the parameter configuration information further includes an indirect service parameter, and index data corresponding to the indirect service parameter is determined by index data corresponding to at least two monitored service parameters;
the apparatus further comprises:
and the indirect parameter determining module is used for obtaining index data corresponding to the indirect service parameters according to the index data corresponding to the at least two monitored service parameters corresponding to the indirect service parameters.
In an embodiment of the disclosure, the apparatus further comprises:
and the data storage module is used for storing the converted data according to the data quantity of the converted data.
In an embodiment of the disclosure, the data storage module is specifically configured to:
if the data quantity of the converted data is larger than a preset data quantity threshold value, storing the converted data according to a first storage mode;
if the data quantity of the converted data is not greater than the data quantity threshold value, storing the converted data according to a second storage mode;
the storage space occupied by the data of the data quantity threshold value stored in the second storage mode is larger than the storage space occupied by the data of the data quantity threshold value stored in the first storage mode.
In an embodiment of the disclosure, the apparatus further comprises:
the data processing module is used for receiving a data query request, wherein the data query request comprises target service parameters, and the request statement format of the data query request corresponds to the first data format; and obtaining index data corresponding to the target service parameters from the converted data according to the data query request, and displaying the index data corresponding to the target service parameters.
In a third aspect, the present disclosure provides an electronic device comprising:
a processor and a memory;
a memory for storing computer operating instructions;
a processor for performing the method as shown in any of the embodiments of the first aspect of the present disclosure by invoking computer operating instructions.
In a fourth aspect, the present disclosure provides a computer readable storage medium having at least one instruction, at least one program, code set, or instruction set stored thereon, the at least one instruction, at least one program, code set, or instruction set being loaded and executed by a processor to implement a method as shown in any embodiment of the first aspect of the present disclosure.
The technical scheme provided by the embodiment of the disclosure has the beneficial effects that:
according to the data processing method, the device, the electronic equipment and the computer readable storage medium, the data format of the log data acquired from the plurality of service servers can be converted into the preconfigured data format, the data with the uniform format is obtained and stored, and the log data acquired from the plurality of service servers are log data with different formats, so that subsequent data processing is inconvenient.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings that are required to be used in the description of the embodiments of the present disclosure will be briefly introduced below.
FIG. 1 is a flow chart of a data processing method according to an embodiment of the disclosure;
FIG. 2 is a flow chart of another data processing method according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
Embodiments of the present disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are exemplary only for explaining the technical aspects of the present disclosure and should not be construed as limiting the present disclosure.
As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification of this disclosure, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein includes all or any element and all combination of one or more of the associated listed items.
The following describes the technical solutions of the present disclosure and how the technical solutions of the present disclosure solve the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present disclosure will be described below with reference to the accompanying drawings.
FIG. 1 is a flow chart of a data processing method according to an embodiment of the disclosure; as shown in fig. 1, the method may include:
step S110, obtaining log data from a plurality of service servers, wherein the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different.
In which, the network device, system, service program, etc. generate a log record, i.e. log file, in which log data usually describes the date, time, user and action of the first-pass operation. The specific content described in the log file varies depending on the service that the system is started.
It can be seen that the service server is configured to monitor service parameters, where in practical application, it is configured according to actual service requirements, which service parameters need to be monitored, that is, which monitored service parameters are specifically configurable. In addition, based on the actual application scenario and different service requirements, one service server may correspondingly monitor one or more service parameters, or one service parameter may also be monitored by one or more service servers.
In an example, for an application program, for example, a user access amount UV of a certain day period a needs to be acquired, that is, a Unique identifier, and one client accessing a certain application program is a Visitor, where in this example, the service requirement is the access amount UV of the period a, the service parameter is the access amount UV, and since the access amount UV can be specifically obtained by accessing a user ID of the user, in an actual application, the service server can monitor the user ID to obtain the number of user IDs of the application program accessed by the period a, and after performing processing such as deduplication on the obtained user ID, the number of user IDs corresponding to the period a is finally obtained, that is, the access amount UV of the user.
Step S120, format conversion is carried out on the log data according to a preset first data format, and converted data are obtained.
The first data format may be configured according to actual service requirements, and the specific format form of the first data format is not limited in the disclosure.
According to the scheme, the data format of the log data acquired from the plurality of service servers can be converted into the preconfigured data format, the data with the uniform format is obtained and stored, and the log data acquired from the plurality of service servers are log data with different formats, so that subsequent data processing is inconvenient.
In the embodiment of the present disclosure, in step S120, performing format conversion on log data according to a preconfigured first data format may include:
determining a data conversion rule for converting the second data format into the first data format according to the first data format and the second data format of the log data;
and converting the log data according to the data conversion rule.
The data conversion rule indicates a conversion mode between two different data formats, the data conversion rule can be preconfigured, and can also be configured in a data format conversion process, and different data formats can correspond to different data conversion rules, for example, a data conversion rule a corresponding to the data format A is used for converting the data format A into the data format B; the data conversion rule b corresponding to the data format a is used for converting the data format a into the data format C.
In an example, the second data format is an apache format, the first data format is a key value format, and the data conversion rule corresponding to the apache format is c, where the data conversion rule c is used to convert log data in the apache format into data in the key value format.
In yet another example, the second data format is json format, the first data format is key value format, and the data conversion rule corresponding to the json format is d, where the data conversion rule d is used to convert the log data in the json format into data in the key value format.
In an embodiment of the present disclosure, the log data includes index data of the monitored service parameter, and converting the log data according to the data conversion rule may include:
extracting index data corresponding to the target service parameters from the log data according to the parameter configuration information, wherein the parameter configuration information comprises the target service parameters, and the monitored service parameters comprise the target service parameters;
and converting the target service parameters and index data corresponding to the target service parameters according to the data conversion rules.
The parameter configuration information can be configured based on actual service requirements so as to meet different service requirements; specifically, index data corresponding to one or more target service parameters in the log data to be extracted may be configured, and similarly, index data corresponding to all target service parameters in the log data to be extracted may be configured. The monitored service parameters in the log data can be configured according to actual service requirements, specifically, one or more service parameters in the log data can be selected as the monitored service parameters according to the service requirements, and similarly, all the service parameters in the log data can be used as the monitored service parameters.
In an embodiment of the present disclosure, the parameter configuration information may further include an indirect service parameter, and index data corresponding to the indirect service parameter is determined by index data corresponding to at least two monitored service parameters, where the method may further include:
and obtaining index data corresponding to the indirect service parameters according to the index data corresponding to the at least two monitored service parameters corresponding to the indirect service parameters.
In practical application, a service parameter can be correspondingly obtained based on different monitored service parameters, for example, a service parameter obtained by calculating index data of different monitored service parameters, namely an indirect service parameter, can be preconfigured with the indirect service parameter in parameter configuration information, so that index data corresponding to the indirect service parameter can be obtained according to index data corresponding to at least two monitored service parameters corresponding to the indirect service parameter.
In an example, for instance, the indirect service parameter m may be determined by averaging the index data of the monitored service parameter a and the index data of the monitored service parameter b, and then in the parameter configuration information, the index data corresponding to the monitored service parameter a and the monitored service parameter b may be determined as the index data corresponding to the indirect service parameter m.
It may be understood that the data format of the index data corresponding to the obtained indirect service parameter may be a first data format or may be other data formats, and in order to unify the data format of the index data corresponding to the indirect service parameter with the first data format, the data format of the index data corresponding to at least two monitored service parameters corresponding to the indirect service parameter may be converted into the first data format before the data format of the index data corresponding to the indirect service parameter is obtained; the data format of the index data corresponding to the indirect service parameter may be converted into the first data format after the data format of the index data corresponding to the indirect service parameter is obtained.
In an embodiment of the present disclosure, the method may further include:
and storing the converted data according to the data quantity of the converted data.
In order to facilitate subsequent processing, such as data display, data analysis, etc., on the converted data, the converted data may be stored, where the converted data refers to data with a data format being a first data format, and includes index data corresponding to indirect service parameters with the data format being the first data format, and index data corresponding to target service parameters; in practical applications, the converted data may be stored in a local server or a cloud server, and the storage location of the data is not limited in the present disclosure.
In an embodiment of the present disclosure, storing the converted data according to the data amount of the converted data may include:
if the data quantity of the converted data is larger than a preset data quantity threshold value, storing the converted data according to a first storage mode;
if the data quantity of the converted data is not greater than the data quantity threshold value, storing the converted data according to a second storage mode;
the storage space occupied by the data of the data quantity threshold value stored in the second storage mode is larger than the storage space occupied by the data of the data quantity threshold value stored in the first storage mode.
In practical application, in order to effectively save storage space when the data amount of the data to be stored is large on the premise of ensuring the data storage efficiency, different storage modes can be configured according to the data amount of the converted data, so that in the data storage process, the data storage mode is dynamically adjusted based on the data amount on the premise of ensuring the data storage efficiency.
In an example, the first storage mode is a bitmap structure storage mode, a bloom filter storage mode and a Hyper Log counter storage mode, the second storage mode is a set structure storage mode, when the data size of the data to be stored is 1000 thousands of users, the storage space occupied by the first storage mode for storing the 1000 thousands of users is 1.5MB, the storage space occupied by the second storage mode for storing the 1000 thousands of users is 100MB, the data size threshold is 2000 tens of thousands, when the data size of the data to be stored is 3000 tens of thousands, the first storage mode may only need 30MB of storage space, and when the second storage mode needs 300MB of storage space, the first storage mode for storing the 3000 thousands of users can save the storage space.
In an embodiment of the present disclosure, the method may further include:
receiving a data query request, wherein the data query request comprises target service parameters, and a request statement format of the data query request corresponds to a first data format;
and obtaining index data corresponding to the target service parameters from the converted data according to the data query request, and displaying the index data corresponding to the target service parameters.
When the subsequent processing is performed on the converted data based on the uniform format, for example, when a data query request is received, index data corresponding to the target service parameter can be obtained from the converted data based on the data query request, and in order to meet the industry requirements of different users, the queried data can be displayed to the users, in practical application, different display modes, such as a histogram, a graph or a pie chart, can be configured based on different service parameters, and the display modes of the data are not limited in the disclosure.
In combination with the foregoing embodiments, the present embodiment provides a data processing method, as shown in fig. 2, where the data processing method may include:
step S210, obtaining log data from a plurality of service servers, wherein the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different.
Step S220, determining a data conversion rule for converting the second data format into the first data format according to the preconfigured first data format and the second data format of the log data.
Step S230, converting the log data according to the data conversion rule to obtain converted data.
Step S240, if the data volume of the converted data is larger than the preset data volume threshold, storing the converted data according to the first storage mode;
if the data quantity of the converted data is not greater than the data quantity threshold value, storing the converted data according to a second storage mode;
the storage space occupied by the data of the data quantity threshold value stored in the second storage mode is larger than the storage space occupied by the data of the data quantity threshold value stored in the first storage mode.
In an example, the second data format is an apache format, the first data format is a key value format, a data conversion rule corresponding to the apache format is c, log data is converted into data in the key value format based on the data conversion rule c, the data amount of the converted log data is 3000 thousands of users and is larger than a preset data amount threshold value of 2000 thousands, the first storage mode is a bitmap structure storage mode, the second storage mode is a set structure storage mode, the first storage mode needs 30MB of storage space for storing the converted log data, the second storage mode needs 300MB of storage space for storing the converted log data, and therefore, when the data with larger data amount is stored, the first storage mode saves the storage space compared with the second storage mode.
In order to embody that the processing efficiency of the data can be improved in the subsequent processing process of the converted data based on the uniform format, the following examples are used for illustration:
in an example, monitoring an application program in real time through different service servers to obtain monitoring data, wherein the data format of the monitoring data corresponding to at least two service servers in the plurality of service servers is different, the index data of the monitored service parameter in the monitoring data can reflect the running state of the application program, the monitoring data are data obtained from different service servers after being converted according to a first data format, the converted data are data with uniform format, when the index data corresponding to the monitored service parameter A in the application program are abnormal, the index data corresponding to the monitored service parameter A need to be checked to find the reason of the data abnormality, so that the application program is updated in time, namely, the service requirement is the index data corresponding to the service parameter A at the current moment of inquiry, and the corresponding target service parameter is the monitored service parameter A; and searching index data corresponding to the target service parameter A at the current moment in the monitoring data based on the target service parameter A in the data query request, and timely finding out the reason of abnormality of the index data corresponding to the target service parameter A. Because the monitoring data are uniform-format data, the index data of the corresponding target service parameters can be found in the monitoring data only by providing the data query request corresponding to the first data format, and format conversion or other processing of the monitoring data in different formats is not needed during data query, so that the uniform-format monitoring data provide a data basis for data query, and the data query efficiency is improved.
Based on the same principle as the method shown in fig. 1, there is also provided in an embodiment of the present disclosure a data processing apparatus 20, as shown in fig. 3, the data processing apparatus 20 may include: a data acquisition module 210, and a data conversion module 220, wherein,
a data obtaining module 210, configured to obtain log data from a plurality of service servers, where data formats of log data corresponding to at least two service servers in the plurality of service servers are different;
the data conversion module 220 is configured to perform format conversion on the log data according to a preconfigured first data format, so as to obtain converted data.
According to the scheme, the data format of the log data acquired from the plurality of service servers can be converted into the preconfigured data format, the data with the uniform format is obtained and stored, and the log data acquired from the plurality of service servers are log data with different formats, so that subsequent data processing is inconvenient.
In the embodiment of the present disclosure, the data conversion module 220 is specifically configured to, when performing format conversion on log data according to a preconfigured first data format:
determining a data conversion rule for converting the second data format into the first data format according to the first data format and the second data format of the log data;
and converting the log data according to the data conversion rule.
In the embodiment of the present disclosure, the log data includes index data of the monitored service parameter, and the data conversion module 220 is specifically configured to, when converting the log data according to the data conversion rule:
extracting index data corresponding to the target service parameters from the log data according to the parameter configuration information, wherein the parameter configuration information comprises the target service parameters, and the monitored service parameters comprise the target service parameters;
and converting the target service parameters and index data corresponding to the target service parameters according to the data conversion rules.
In an embodiment of the present disclosure, the parameter configuration information further includes an indirect service parameter, and index data corresponding to the indirect service parameter is determined by index data corresponding to at least two monitored service parameters, where the apparatus may further include:
and the indirect parameter determining module is used for obtaining index data corresponding to the indirect service parameters according to the index data corresponding to the at least two monitored service parameters corresponding to the indirect service parameters.
In an embodiment of the present disclosure, the apparatus may further include:
and the data storage module is used for storing the converted data according to the data quantity of the converted data.
In an embodiment of the present disclosure, storing the converted data according to the data amount of the converted data may include:
if the data quantity of the converted data is larger than a preset data quantity threshold value, storing the converted data according to a first storage mode;
if the data quantity of the converted data is not greater than the data quantity threshold value, storing the converted data according to a second storage mode;
the storage space occupied by the data of the data quantity threshold value stored in the second storage mode is larger than the storage space occupied by the data of the data quantity threshold value stored in the first storage mode.
In an embodiment of the present disclosure, the apparatus may further include:
the data processing module is used for receiving a data query request, wherein the data query request comprises target service parameters, and the request statement format of the data query request corresponds to the first data format; and obtaining index data corresponding to the target service parameters from the converted data according to the data query request, and displaying the index data corresponding to the target service parameters.
The data processing apparatus according to the embodiments of the present disclosure may perform a data processing method provided by the embodiments of the present disclosure, and the implementation principle is similar, and actions performed by each module in the data processing apparatus according to each embodiment of the present disclosure correspond to steps in the data processing method according to each embodiment of the present disclosure, and detailed functional descriptions of each module in the data processing apparatus may be specifically referred to descriptions in the corresponding data processing method shown in the foregoing, which are not repeated herein.
Based on the same principle as the data processing method in the embodiments of the present disclosure, the present disclosure provides an electronic device including a processor and a memory; a memory for storing operation instructions; a processor for executing the method as shown in any one of the embodiments of the data processing methods of the present disclosure by invoking the operational instructions.
Based on the same principles as the data processing methods in the embodiments of the present disclosure, the present disclosure provides a computer readable storage medium storing at least one instruction, at least one program, code set, or instruction set that is loaded and executed by a processor to implement a method as shown in any of the embodiments of the data processing methods of the present disclosure.
In an embodiment of the present disclosure, as shown in fig. 4, a schematic diagram of a structure of an electronic device 30 (e.g., a terminal device or server implementing the method shown in fig. 1) suitable for use in implementing an embodiment of the present disclosure is shown. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 4 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
As shown in fig. 4, the electronic device 30 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 301 that may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 302 or a program loaded from a storage means 308 into a Random Access Memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 30 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.
In general, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 307 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 308 including, for example, magnetic tape, hard disk, etc.; and communication means 309. The communication means 309 may allow the electronic device 30 to communicate with other devices wirelessly or by wire to exchange data. While fig. 4 shows the electronic device 30 with various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing means 301.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring at least two internet protocol addresses; sending a node evaluation request comprising at least two internet protocol addresses to node evaluation equipment, wherein the node evaluation equipment selects an internet protocol address from the at least two internet protocol addresses and returns the internet protocol address; receiving an Internet protocol address returned by node evaluation equipment; wherein the acquired internet protocol address indicates an edge node in the content distribution network.
Alternatively, the computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: receiving a node evaluation request comprising at least two internet protocol addresses; selecting an internet protocol address from at least two internet protocol addresses; returning the selected internet protocol address; wherein the received internet protocol address indicates an edge node in the content distribution network.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The name of the unit does not in any way constitute a limitation of the unit itself, for example the first acquisition unit may also be described as "unit acquiring at least two internet protocol addresses".
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).
Claims (7)
1. A method of data processing, comprising:
acquiring log data from a plurality of service servers, wherein the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different;
according to a first preset data format, carrying out format conversion on the log data to obtain converted data;
receiving a data query request, wherein the data query request comprises target service parameters, and the request statement format of the data query request corresponds to the first data format;
according to the data query request, index data corresponding to the target service parameter is obtained from the converted data, and the index data corresponding to the target service parameter is displayed;
wherein said format converting said log data comprises:
extracting index data corresponding to the target service parameters from the log data according to the parameter configuration information, and obtaining index data corresponding to the indirect service parameters; the log data comprises index data of monitored service parameters, the monitored service parameters comprise the target service parameters, the parameter configuration information comprises the target service parameters and the indirect service parameters, and the index data corresponding to the indirect service parameters is determined by index data corresponding to at least two monitored service parameters;
and converting the target service parameters and index data corresponding to the target service parameters.
2. The method of claim 1, wherein said formatting said log data according to a preconfigured first data format comprises:
determining a data conversion rule for converting the second data format into the first data format according to the first data format and the second data format of the log data;
and converting the log data according to the data conversion rule.
3. The method according to claim 1 or 2, further comprising:
and storing the converted data according to the data quantity of the converted data.
4. A method according to claim 3, wherein said storing said converted data according to the data amount of said converted data comprises:
if the data quantity of the converted data is larger than a preset data quantity threshold value, storing the converted data according to a first storage mode;
if the data quantity of the converted data is not greater than the data quantity threshold value, storing the converted data according to a second storage mode;
and the storage space occupied by the second storage mode for storing the data of the data quantity threshold value is larger than the storage space occupied by the first storage mode for storing the data of the data quantity threshold value.
5. A data processing apparatus, comprising:
the system comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring log data from a plurality of service servers, and the data formats of the log data corresponding to at least two service servers in the plurality of service servers are different;
the data conversion module is used for carrying out format conversion on the log data according to a preset first data format to obtain converted data; the method is specifically used for extracting index data corresponding to the target service parameters from the log data according to the parameter configuration information, and obtaining index data corresponding to the indirect service parameters; the log data comprises index data of monitored service parameters, the monitored service parameters comprise the target service parameters, the parameter configuration information comprises the target service parameters and the indirect service parameters, and the index data corresponding to the indirect service parameters is determined by index data corresponding to at least two monitored service parameters; converting the target service parameters and index data corresponding to the target service parameters;
the data query module is used for receiving a data query request, wherein the data query request comprises the target service parameters, and the request statement format of the data query request corresponds to the first data format; and obtaining index data corresponding to the target service parameter from the converted data according to the data query request, and displaying the index data corresponding to the target service parameter.
6. An electronic device, comprising:
a processor and a memory;
the memory is used for storing computer operation instructions;
the processor is configured to execute the method of any one of the preceding claims 1 to 4 by invoking the computer operating instructions.
7. A computer readable storage medium having stored thereon a computer program, the storage medium having stored thereon at least one instruction, at least one program, a set of codes or a set of instructions, the at least one instruction, the at least one program, the set of codes or the set of instructions being loaded and executed by a processor to implement the method of any of the preceding claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811369129.4A CN111198859B (en) | 2018-11-16 | 2018-11-16 | Data processing method, device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811369129.4A CN111198859B (en) | 2018-11-16 | 2018-11-16 | Data processing method, device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111198859A CN111198859A (en) | 2020-05-26 |
| CN111198859B true CN111198859B (en) | 2023-11-03 |
Family
ID=70744179
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811369129.4A Active CN111198859B (en) | 2018-11-16 | 2018-11-16 | Data processing method, device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111198859B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112052247B (en) * | 2020-09-29 | 2024-05-07 | 微医云(杭州)控股有限公司 | Index updating system, method and device for search engine, electronic equipment and storage medium |
| CN113159970A (en) * | 2021-05-14 | 2021-07-23 | 中国工商银行股份有限公司 | Data format conversion method, device, system, electronic equipment and storage medium |
| CN113297148B (en) * | 2021-06-18 | 2023-06-06 | 杭州安恒信息安全技术有限公司 | Method, device and equipment for collecting service log data and readable storage medium |
| CN113504896B (en) * | 2021-07-12 | 2023-08-18 | 云南腾云信息产业有限公司 | Service data processing method and device of application program and mobile terminal |
| CN113656445A (en) * | 2021-08-26 | 2021-11-16 | 五八同城信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114241815B (en) * | 2021-11-08 | 2023-02-28 | 深圳市瑞达飞行科技有限公司 | Computing method and device, electronic equipment and computer readable storage medium |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2009009448A (en) * | 2007-06-29 | 2009-01-15 | Mitsubishi Electric Corp | Data transmission device, data transmission method, and program |
| CN103235811A (en) * | 2013-04-24 | 2013-08-07 | 微梦创科网络科技（中国）有限公司 | Data storage method and device |
| CN103823811A (en) * | 2012-11-19 | 2014-05-28 | 北京百度网讯科技有限公司 | Method and system for processing journals |
| CN104239538A (en) * | 2014-09-22 | 2014-12-24 | 北京国双科技有限公司 | Method, system and device for compressing snapshot log |
| CN105095518A (en) * | 2015-09-18 | 2015-11-25 | 中国农业银行股份有限公司 | Data conversion method and device |
| CN105262812A (en) * | 2015-10-16 | 2016-01-20 | 浪潮(北京)电子信息产业有限公司 | Log data processing method based on cloud computing platform, log data processing device and log data processing system |
| CN105378716A (en) * | 2014-03-18 | 2016-03-02 | 华为技术有限公司 | Method and apparatus for conversion of data storage formats |
| CN106202235A (en) * | 2016-06-28 | 2016-12-07 | 微梦创科网络科技（中国）有限公司 | A kind of data processing method and device |
| CN107092683A (en) * | 2017-04-21 | 2017-08-25 | 北京奇艺世纪科技有限公司 | A kind of log processing system and method |
| CN107249019A (en) * | 2017-05-26 | 2017-10-13 | 北京小米移动软件有限公司 | Data handling system, method, device and server based on business |
| CN107861859A (en) * | 2017-11-22 | 2018-03-30 | 北京汇通金财信息科技有限公司 | A kind of blog management method and system based on micro services framework |
| CN108345528A (en) * | 2018-02-13 | 2018-07-31 | 北京天元创新科技有限公司 | A kind of background server monitoring method, device and equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7062551B2 (en) * | 2001-05-24 | 2006-06-13 | International Business Machines Corporation | Method and apparatus to solve compatibility between heterogeneous web server access logs formats |
| CA2419305C (en) * | 2003-02-20 | 2006-03-21 | Ibm Canada Limited - Ibm Canada Limitee | Unified logging service for distributed applications |
-
2018
- 2018-11-16 CN CN201811369129.4A patent/CN111198859B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2009009448A (en) * | 2007-06-29 | 2009-01-15 | Mitsubishi Electric Corp | Data transmission device, data transmission method, and program |
| CN103823811A (en) * | 2012-11-19 | 2014-05-28 | 北京百度网讯科技有限公司 | Method and system for processing journals |
| CN103235811A (en) * | 2013-04-24 | 2013-08-07 | 微梦创科网络科技（中国）有限公司 | Data storage method and device |
| CN105378716A (en) * | 2014-03-18 | 2016-03-02 | 华为技术有限公司 | Method and apparatus for conversion of data storage formats |
| CN104239538A (en) * | 2014-09-22 | 2014-12-24 | 北京国双科技有限公司 | Method, system and device for compressing snapshot log |
| CN105095518A (en) * | 2015-09-18 | 2015-11-25 | 中国农业银行股份有限公司 | Data conversion method and device |
| CN105262812A (en) * | 2015-10-16 | 2016-01-20 | 浪潮(北京)电子信息产业有限公司 | Log data processing method based on cloud computing platform, log data processing device and log data processing system |
| CN106202235A (en) * | 2016-06-28 | 2016-12-07 | 微梦创科网络科技（中国）有限公司 | A kind of data processing method and device |
| CN107092683A (en) * | 2017-04-21 | 2017-08-25 | 北京奇艺世纪科技有限公司 | A kind of log processing system and method |
| CN107249019A (en) * | 2017-05-26 | 2017-10-13 | 北京小米移动软件有限公司 | Data handling system, method, device and server based on business |
| CN107861859A (en) * | 2017-11-22 | 2018-03-30 | 北京汇通金财信息科技有限公司 | A kind of blog management method and system based on micro services framework |
| CN108345528A (en) * | 2018-02-13 | 2018-07-31 | 北京天元创新科技有限公司 | A kind of background server monitoring method, device and equipment |
Non-Patent Citations (1)
| Title |
|---|
| 一个开放环境中Client/Server结构的日志审计***;吴涛, 丁伟, 高毓航;计算机工程(第S1期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111198859A (en) | 2020-05-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111198859B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN109299348B (en) | Data query method and device, electronic equipment and storage medium | |
| CN111857720B (en) | User interface state information generation method and device, electronic equipment and medium | |
| CN111813465B (en) | Information acquisition method, device, medium and equipment | |
| CN111241137B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN111274104B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN109902726B (en) | Resume information processing method and device | |
| CN111694629A (en) | Information display method and device and electronic equipment | |
| CN112291121B (en) | Data processing method and related equipment | |
| CN108011936B (en) | Method and device for pushing information | |
| CN111198853B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN111596992A (en) | Navigation bar display method and device and electronic equipment | |
| CN111460020B (en) | Method, device, electronic equipment and medium for resolving message | |
| CN112379967B (en) | Simulator detection method, device, equipment and medium | |
| CN111143355B (en) | Data processing method and device | |
| CN111241368B (en) | Data processing method, device, medium and equipment | |
| CN112688793B (en) | Data packet obtaining method and device and electronic equipment | |
| CN110377654B (en) | Data request processing method and device, electronic equipment and computer-readable storage medium | |
| CN109614137B (en) | Software version control method, device, equipment and medium | |
| CN109522133B (en) | Data splicing method and device, electronic equipment and storage medium | |
| CN113282455A (en) | Monitoring processing method and device | |
| CN112100159A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| CN111222048A (en) | User number query calculation method and device, electronic equipment and storage medium | |
| US10516767B2 (en) | Unifying realtime and static data for presenting over a web service | |
| CN114040014B (en) | Content pushing method, device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |