CN112131221A - Tracking data processing method, device, equipment and storage medium - Google Patents
Tracking data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112131221A CN112131221A CN202010992198.1A CN202010992198A CN112131221A CN 112131221 A CN112131221 A CN 112131221A CN 202010992198 A CN202010992198 A CN 202010992198A CN 112131221 A CN112131221 A CN 112131221A
- Authority
- CN
- China
- Prior art keywords
- data
- tracking data
- target tracking
- target
- original text
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method for processing tracking data, which comprises the following steps: acquiring target tracking data sent to a tracking interface by a target function module; calculating a target hash value of the target tracking data; determining whether repeated data of the target tracking data exists in the hash table or not based on the target hash value; if the target tracking data exists, determining original text replacing data of the target tracking data based on the sequence number of the tracking data, wherein the data volume of the original text replacing data of the target tracking data is smaller than that of the original text of the target tracking data; and replacing the original text of the target tracking data with the original text of the target tracking data, and sending the original text of the target tracking data to the tracking data summarizing module. By applying the technical scheme provided by the application, the data volume of the trace data is reduced, the load of the trace data is reduced, and the risks of overflow and loss of the trace data in the on-chip transmission are reduced. The application also discloses a tracking data processing device, equipment and a storage medium, and the tracking data processing device, the equipment and the storage medium have corresponding technical effects.
Description
Technical Field
The present application relates to the field of computer application technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing trace data.
Background
With the rapid development of computer technology, the application requirements become more and more complex, and the design of the chip becomes more and more complex. For example, an SoC (System on Chip) Chip is designed by integrating a complete computer System on a Chip according to application requirements, and the integrated design usually includes a plurality of functional modules, and an on-Chip interconnection network is used to connect all the functional modules for data transmission.
In order to observe the operation state of some functional modules and analyze and repair errors when errors occur, a Trace interface, namely a Trace interface, is usually added in the design of some key functional modules and complex functional modules. Each functional module with the Trace interface can write some values or statements representing the operating state of the functional module into the Trace interface to form Trace data, then the Trace data is sent to the Trace data summarizing module through an interconnection network on a Trace chip to be summarized, and finally the Trace data is sent to the outside of the chip through an Input/Output (IO) interface in a unified manner so as to be received and used by an external receiver. For example, an external receiver can analyze the received trace data to know the working state of each functional module inside the chip, and analyze and eliminate errors in the system.
In the normal operation process of the chip, the data volume of the trace data generated by each functional module is large, the problem of overlarge load of the trace data is easy to occur, and if the peak value of the data load exceeds the transmission bandwidth of the trace data, the risk of overflow and loss of the trace data in transmission is caused. In summary, how to process the trace data, reduce the data amount of the trace data, reduce the load of the trace data, and reduce the risks of overflow and loss of the trace data during transmission is a technical problem that needs to be solved urgently by those skilled in the art.
Disclosure of Invention
The application aims to provide a method, a device, equipment and a storage medium for processing trace data, so that the data volume of the trace data is reduced, the load of the trace data is reduced, the risks of overflow and loss of the trace data in transmission are reduced, and the trace data can be effectively utilized.
In order to solve the technical problem, the application provides the following technical scheme:
a method of trace data processing, comprising:
acquiring target tracking data sent to a tracking interface by a target function module;
calculating a target hash value of the target tracking data;
determining whether duplicate data of the target tracking data exists in a hash table based on the target hash value;
if the repeated data of the target tracking data exists in the hash table, determining original text substitute data of the target tracking data based on the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data, wherein the data volume of the original text substitute data of the target tracking data is smaller than that of the original text of the target tracking data;
and replacing the original text of the target tracking data with the original text substitute data of the target tracking data, and sending the original text substitute data to the tracking data summarizing module so that the tracking data summarizing module summarizes the multi-path tracking data and then sends the multi-path tracking data to the outside of the chip.
In a specific embodiment of the present application, each row of the hash table holds two elements: the sequence number of the trace data and the original text of the trace data; the determining whether duplicate data of the target tracking data exists in a hash table based on the target hash value includes:
determining the position to be stored of the target tracking data in the hash table according to the target hash value and the total line number of the hash table;
reading the content of the position to be stored in the hash table;
if the read content is valid, comparing the read original text of the existing tracking data with the original text of the target tracking data;
if the two data are the same, determining that the repeated data of the target tracking data exist in the hash table;
and if the contents are not the same or the read contents are invalid, determining that the repeated data of the target tracking data does not exist in the hash table.
In one embodiment of the present application, the method further includes:
replacing the sequence number of the repeated data of the target tracking data in the hash table with the sequence number of the target tracking data if the repeated data of the target tracking data exists in the hash table;
if the repeated data of the target tracking data does not exist in the hash table, recording the sequence number and the original text of the target tracking data at the position to be stored in the hash table.
In a specific embodiment of the present application, the determining, based on the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data, original text replacement data of the target tracking data includes:
obtaining the sequence number of the repeated data of the target tracking data in the hash table;
determining a number offset of the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data;
and determining the number offset as original text replacing data of the target tracking data.
In one embodiment of the present application, the method further includes:
and if the repeated data of the target tracking data does not exist in the hash table, directly sending the original text of the target tracking data to the tracking data summarizing module.
In one embodiment of the present application, the method further includes:
if repeated data of the target tracking data exists in the hash table, replacing the original text of the target tracking data with original text replacement data of the target tracking data, sending the replaced data to a tracking data summarizing module, and meanwhile outputting a replaced mark so as to determine that the original text of the target tracking data is replaced based on the replaced mark when the target tracking data is used;
if the repeated data of the target tracking data does not exist in the hash table, outputting an un-replaced flag while sending the original text of the target tracking data to the tracking data summarizing module, so as to determine that the original text of the target tracking data is not replaced based on the un-replaced flag when the target tracking data is used.
In a specific embodiment of the present application, before determining whether duplicate data of the target trace data exists in a hash table, the method further includes:
determining whether the sequence number of the target tracking data is 0;
and if so, emptying the hash table.
A trace data processing apparatus comprising:
the tracking data acquisition unit is used for acquiring target tracking data sent to the tracking interface by the target function module;
a hash value calculation unit for calculating a target hash value of the target trace data;
a duplicate data determination unit, configured to determine whether duplicate data of the target tracking data exists in a hash table based on the target hash value;
a substitute data determination unit, configured to determine, if duplicate data of the target tracking data exists in the hash table, original text substitute data of the target tracking data based on the sequence number of the target tracking data and the sequence number of the duplicate data of the target tracking data, where a data amount of the original text substitute data of the target tracking data is smaller than a data amount of an original text of the target tracking data;
and the data sending unit is used for replacing the original text of the target tracking data with the original text substitute data of the target tracking data and sending the original text substitute data to the tracking data summarizing module so that the tracking data summarizing module summarizes the multi-path tracking data and then sends the multi-path tracking data to the outside of the chip.
A trace data processing apparatus comprising:
a memory for storing a computer program;
a processor for implementing the steps of any of the above-described trace data processing methods when executing the computer program.
A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of processing trace data according to any of the preceding claims.
By applying the technical scheme provided by the embodiment of the application, after the target tracking data sent to the tracking interface by the target function module is obtained, the target hash value of the target tracking data is calculated, on the basis of the target hash value, under the condition that the repeated data of the target tracking data exist in the hash table, the original text replacing data of the target tracking data is determined on the basis of the sequence number of the target tracking data and the sequence number of the repeated data, and after the original text is replaced by the original text replacing data, the target tracking data is sent to the tracking data summarizing module. The tracking data is repeatedly judged at the interface for generating the tracking data, namely the source, so that the characteristic that the tracking data of a single functional module has periodic repetition can be better utilized, the original text of the tracking data with repeated data uses data with smaller data volume to replace the original text of the tracking data with repeated data, the data volume of the tracking data is reduced, the load of the tracking data is reduced, the risks of overflow and loss of the tracking data in the transmission of the tracking data in the chip are reduced, the tracking data can be effectively utilized, and the tracking function is more stable.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic structural diagram of an SoC chip in the related art;
fig. 2 is a schematic structural diagram of an SoC chip in the embodiment of the present application;
FIG. 3 is a flowchart illustrating an implementation of a trace data processing method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of recovery of trace data when used in an embodiment of the present application;
FIG. 5 is a diagram illustrating a process of processing trace data according to an embodiment of the present application;
FIG. 6 is a schematic structural diagram of a trace data processing apparatus according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a trace data processing device in an embodiment of the present application.
Detailed Description
The core of the application is to provide a tracking data processing method. For convenience of understanding, before describing the technical solution of the embodiment of the present application, a structure of an SoC chip in the related art is explained.
As shown in fig. 1, the chip integrates a CPU (central processing Unit), a DMAC (Direct Memory Access Control, a common function module in SoC chip design), a PCIe (Peripheral Component Interconnect Express, a high-speed serial computer extended bus standard, a common high-speed IO interface function module in SoC chip design), an algorithm module a, an algorithm module B, LMU (Local Memory Unit, a common function module in SoC chip design), a ROM (Read Only Memory, a common function module in SoC chip design), a DDR (Double Data Rate Synchronous Dynamic Random Access Memory, a common function module in SoC chip design), an I2C (interleaved Circuit, a two-wire Synchronous serial bus, a common low-speed u interface function module in SoC chip design), and a single IO System, a system control Unit, a common function module in SoC Chip design), and MON (Monitor Signal Unit, a Signal observation Unit, a common debugging function module in SoC Chip design), which are connected via a NoC (Network on Chip, an interconnection Network module for connecting each function module in SoC Chip design). Wherein, tracing interfaces, namely Trace interfaces (Trace ports), are added in key functional modules and complex functional modules, such as a CPU, a DMAC, an algorithm module A, an algorithm module B and a MON. And the tracking data is transmitted to a Trace on-chip internet (TNoC) from the tracking interface and then transmitted to a tracking data summarizing module. The trace data summarization module summarizes and processes the multi-path trace data from different functional modules to form a path of data, and the path of data is sent to the outside of the chip through an IO interface, such as PCIe.
In fig. 1, AXI is called an Advanced eXtensible Interface, an ultra-High Performance chip internal Bus, AHB is called an Advanced High Performance Bus, a High Performance chip internal Bus, and APB is called an Advanced Performance Bus.
The design idea has the problems that the data volume of the trace data generated by each functional module is large and the trace data load is easy to be overlarge, so that the trace data is easy to overflow and lose in transmission.
In view of this, in the chip design of the embodiment of the present application, a Trace data processing module is added at the Trace interface, as shown in fig. 2, so that the Trace data sent to the Trace interface by the corresponding functional module, that is, the Trace data of the Trace interface, is processed by the Trace data processing module first and then sent to the Trace data summarizing module. After the processing of the Trace data processing module, the Trace data load can be reduced, so that the risks of Trace data overflow and loss between an interconnection network on a Trace chip and the Trace data summarizing module can be reduced.
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 3, a flowchart for implementing a trace data processing method provided in an embodiment of the present application may include the following steps:
s310: and acquiring target tracking data sent to the tracking interface by the target function module.
In the embodiment of the present application, a trace data processing module is disposed at a trace interface of each functional module having a trace interface. Each tracking data processing module can process the tracking data sent to the tracking interface by the corresponding functional module and then send the processed tracking data to the tracking data summarizing module. The method provided by the embodiment of the application can be applied to any tracking data processing module.
The tracking data processing module can obtain the target tracking data sent to the tracking interface by the target function module. The tracking data processing module corresponds to the target function module and is a tracking data processing module arranged at a tracking interface of the target function module. The target tracking data is any piece of tracking data which is sent to the tracking interface by the target function module.
For a functional module, each trace data sent by the functional module to the trace interface includes a sequence number and a text.
S320: and calculating a target hash value of the target tracking data.
After the target tracking data sent to the tracking interface by the target function module is obtained, the hash value of the target tracking data can be calculated based on the original text of the target tracking data, and the calculated hash value is the target hash value of the target tracking module.
The trace data is in a character string form in most cases, and an open-source ELF hash algorithm can be selected to perform hash calculation on the trace data, for example, after each character in an input character string is operated, a hash value is a hash value corresponding to the input character string.
S330: and determining whether repeated data of the target tracking data exists in the hash table or not based on the target hash value.
After the target hash value of the target tracking data is obtained through calculation, whether repeated data of the target tracking data exists in the hash table can be determined based on the target hash value.
Generally, a functional module is often used to periodically perform some tasks during operation, for example, a data compression module is used to continuously repeat four steps: (1) read data to be compressed, (2) read compression configuration, (3) generate compressed data, and (4) write back compressed data. Therefore, the trace data generated by the functional module may also have a periodically repeating characteristic, and there may be repeated data in the hash table.
In this embodiment of the present application, the hash table may be a tabular storage space having two columns and a plurality of rows, the number of rows may be determined according to the data amount of trace data of a certain specific function module, and the larger the data amount of the trace data is, the larger the number of rows of the hash table is, the better the reduction effect on the data load is. Such as determining the total number of rows to be 500. Two columns of the hash table record the sequence number and the original text of the tracking data respectively.
Since the length of the original text of the trace data is not determined, in practical applications, only the first few characters, such as the first 50 characters, of the original text of the trace data may be saved in the hash table. The more the number of characters of the original text stored in the hash table is, the more effectively the hash collision can be prevented. However, the larger the number of characters, the more memory space is required, and the higher the corresponding chip cost. Therefore, the number of characters to be saved can be determined according to the size of the data volume of the trace data of the specific function module, and the larger the data volume of the trace data is, the more characters should be saved.
The sequence number of the trace data is the number of each piece of received trace data in sequence by the trace data processing module, starting from zero, the maximum value can be 32 times of 2, and after the sequence number counts to the maximum value, the number is counted again from zero.
In one embodiment of the present application, each row of the hash table holds two elements: the sequential numbering of trace data and the original text of the trace data, which may include the steps of:
the method comprises the following steps: determining the position of target tracking data to be stored in the hash table according to the target hash value and the total row number of the hash table;
step two: reading the content of the position to be stored in the hash table;
step three: if the read content is valid, comparing the read original text of the existing tracking data with the original text of the target tracking data;
step four: if the two data are the same, determining that repeated data of the target tracking data exist in the hash table;
step five: and if the contents are not the same or the read contents are invalid, determining that the repeated data of the target tracking data does not exist in the hash table.
The above steps are combined for convenience of description.
And after the target tracking data sent to the tracking interface by the target function module is obtained and the target hash value of the target tracking data is calculated, determining the position of the target tracking data to be stored in the hash table according to the target hash value and the total row number of the hash table.
In the embodiment of the present application, each piece of trace data has a corresponding storage location in the hash table. The hash value of the piece of trace data may be determined by modulo operation of the hash value and the total number of rows of the hash table. For example, if the hash value of a piece of trace data is calculated to be 123654 and the total number of rows in the hash table is 500, then according to 123654MOD 500 ═ 154, it can be determined that the piece of trace data should be stored in row 154 of the hash table.
After the position of the target tracking data to be saved in the hash table is determined, the content of the position to be saved in the hash table can be further read. If the trace data is recorded at the position to be saved in the hash table, the read content can be considered to be valid, and if the trace data is not recorded at the position to be saved in the hash table, the read content can be considered to be invalid.
If the read content is valid, the read original of the existing trace data may be compared with the original of the target trace data. If the two are the same, the comparison is hit, and it can be determined that the hash table contains the repeated data of the target tracking data, that is, the tracking data at the position to be stored is the repeated data of the target tracking data.
If the two are not the same, or the read content is invalid, the comparison is not hit, and it can be determined that the duplicate data of the target tracking data does not exist in the hash table.
The text to be compared here may be the first several characters of the text held in the hash table.
And meanwhile, the hash value and the original text of the tracking data are used for determining the searching hit and miss of the hash table, and compared with the method for searching the repeated data by independently using the hash value, the method can better avoid the condition of hash collision.
S340: and if the repeated data of the target tracking data exists in the hash table, determining the original text substitute data of the target tracking data based on the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data.
The data amount of the original text substitute data of the target trace data is smaller than the data amount of the original text of the target trace data.
After determining that the repeated data of the target tracking data exists in the hash table, the original text substitute data of the target tracking data may be determined based on the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data. The data amount of the original text substitute data of the target trace data is smaller than the data amount of the original text of the target trace data.
In a specific embodiment of the present application, the sequence number of the repeated data of the target tracking data in the hash table may be obtained first, then the number offset between the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data is determined, and then the number offset is determined as the original text substitute data of the target tracking data.
And each line in the hash table records the sequence number and the original text of one piece of tracking data. After determining that the repeated data of the target tracking data exists in the hash table, the sequence number of the repeated data may be obtained first, and then the number offset between the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data is determined. Specifically, the number offset may be obtained by subtracting the sequence number of the target tracking data from the sequence number of the repeated data of the target tracking data. If the sequence number of the repeated data of the target tracking data is A, and the sequence number of the target tracking data is B, B-A can obtain the number offset. And after the number offset is obtained, determining the number offset as original text substitute data of the target tracking data. Thus, the data amount of the original text replacement data is smaller than that of the original text.
The number offset is used as the original text to replace data to replace the original text, so that the space of 4 bytes is occupied at most, and the storage space can be further saved.
S350: and replacing the original text of the target tracking data with the original text of the target tracking data, and sending the original text of the target tracking data to the tracking data summarizing module so that the tracking data summarizing module summarizes the multi-path tracking data and then sends the multi-path tracking data to the outside of the chip.
Under the condition that repeated data of the target tracking data exist in the hash table, after the original text replacing data of the target tracking data is determined, the original text replacing data can replace the original text of the target tracking data, and the target tracking data after the original text replacement is sent to the tracking data summarizing module.
And if the repeated data of the target tracking data does not exist in the hash table, directly sending the original text of the target tracking data to the tracking data summarizing module.
The tracking data processing module arranged at each tracking interface can execute the above operations, and the tracking data sent to the tracking interface by the corresponding functional module is processed and then sent to the tracking data summarizing module, so that the multiple paths of tracking data received by the tracking data summarizing module are all correspondingly processed under the condition that repeated data exists, the data quantity of the tracking data transmitted from the tracking interface to the tracking data summarizing module is reduced, and the load of the tracking data is lightened. And the tracking data summarizing module summarizes the multi-path tracking data and then sends the multi-path tracking data to the outside of the chip so as to be received and used by an external receiver.
The tracking data summarizing module can also compress the summarized data and then send the compressed data to the outside of the chip, so that the load of the tracking data is further reduced.
By applying the method provided by the embodiment of the application, after the target tracking data sent to the tracking interface by the target function module is obtained, the target hash value of the target tracking data is calculated, on the basis of the target hash value, under the condition that the repeated data of the target tracking data exist in the hash table, the original text replacing data of the target tracking data is determined on the basis of the sequence number of the target tracking data and the sequence number of the repeated data, and after the original text is replaced by the original text replacing data, the target tracking data is sent to the tracking data summarizing module. The tracking data is repeatedly judged at the interface for generating the tracking data, namely the source, so that the characteristic that the tracking data of a single functional module has periodic repetition can be better utilized, the original text of the tracking data with repeated data uses data with smaller data volume to replace the original text of the tracking data with repeated data, the data volume of the tracking data is reduced, the load of the tracking data is reduced, the risks of overflow and loss of the tracking data in the transmission of the tracking data in the chip are reduced, the tracking data can be effectively utilized, and the tracking function is more stable.
In one embodiment of the present application, the method may further comprise the steps of:
if the repeated data of the target tracking data exists in the hash table, replacing the sequence number of the repeated data of the target tracking data in the hash table by using the sequence number of the target tracking data;
and if the hash table does not contain repeated data of the target tracking data, recording the sequence number and the original text of the target tracking data at the position to be stored in the hash table.
In the embodiment of the application, the hash table can be updated based on the obtained target tracking data, so that the accurate determination of the repeated data can be performed based on the updated hash table in the following.
If the hash table has the repeated data of the target tracking data, the sequence number of the repeated data of the target tracking data in the hash table can be replaced by the sequence number of the target tracking data, and the recorded original text is kept unchanged. And if the hash table does not contain repeated data of the target tracking data, recording the sequence number and the original text of the target tracking data at the position to be stored in the hash table. The text here may be the first several characters of the text.
In one embodiment of the present application, the method may further comprise the steps of:
if repeated data of the target tracking data exist in the hash table, replacing the original text of the target tracking data with original text replacement data of the target tracking data, sending the replaced mark to the tracking data summarizing module, and meanwhile, determining that the original text of the target tracking data is replaced based on the replaced mark when the target tracking data is used;
if the repeated data of the target tracking data does not exist in the hash table, the original text of the target tracking data is sent to the tracking data summarizing module, and meanwhile the non-replacement mark is output, so that the original text of the target tracking data is determined to be not replaced based on the non-replacement mark when the target tracking data is used.
In the embodiment of the application, if repeated data of the target tracking data exists in the hash table, the original text of the target tracking data is replaced by the original text replacement data of the target tracking data and sent to the tracking data summarizing module, and meanwhile, a replaced mark can be output to indicate that the original text of the target tracking data is replaced. Thus, when the target tracking data needs to be used, the original text of the target tracking data can be determined to be replaced based on the replaced mark. Similarly, if the hash table does not contain the repeated data of the target tracking data, the original text of the target tracking data can be sent to the tracking data summarizing module, and meanwhile, the non-replacement flag can be output to indicate that the original text of the target tracking data is not replaced. In this way, it may be determined that the original text of the target trace data is not replaced based on the non-replaced flag when the target trace data needs to be used.
The trace data processed by the trace data processing module provided at the trace interface is transmitted to the trace data summarization module together with the replaced/non-replaced flag. The trace data summarization module needs to process and encode the trace data input from different functional modules together with the replaced mark/the un-replaced mark, and generate a final trace data stream to be sent to the outside of the chip for use by an external receiver.
When the external receiver uses the trace data, the trace data can be processed and recovered piece by piece. Each piece of trace data received by the external receiver has a corresponding processing sequence number. For a piece of trace data, if it corresponds to replaced flag 1, it indicates that the number offset is not the original text of the trace data but is stored in the piece of trace data. In this case, the number offset may be read first, and then the processing sequence number indicated by the number offset may be backed back, the original text of the trace data corresponding to the processing sequence number may be read, and the original text of the current trace data may be recovered by using the original text. If it corresponds to the non-replaced flag 0, it indicates that the original text of the trace data is stored in the piece of trace data, not the number offset, and the piece of trace data can be directly used. As shown in fig. 4, assuming that Trace data with the processing sequence number L +200, that is, Trace data corresponds to the replaced flag 1, it can be determined that the number offset 198 is stored in the piece of Trace data, and by (L +200) -198 ═ L +2, the original text of the Trace data with the processing sequence number L +2 can be used to restore the original text of the Trace data with the processing sequence number L +200, that is, the original text of the Trace data with the processing sequence number L +200 can be restored to XXXX 2.
Fig. 5 is a schematic diagram illustrating the overall flow of processing trace data. The Trace data processing module obtains a piece of Trace data, namely Trace data, carries out hash calculation on the Trace data to obtain a hash value, carries out table lookup comparison on the basis of the hash value to determine whether repeated data of the Trace data exist in a hash table, carries out offset substitution processing if the repeated data exist, namely, original text substitution data determined on the basis of sequence numbering replaces an original text to obtain the Trace data after hash substitution, and does not carry out offset substitution processing if the repeated data do not exist, and can simultaneously output a substituted mark or an unsubstituted mark when the Trace data are sent to the Trace data summarizing module. After the table lookup comparison, the hash table may be updated according to the table lookup result. The hash table may be in a format of N rows and 2 columns.
In one embodiment of the present application, before determining whether duplicate data of the target trace data exists in the hash table, the method may further include the following steps:
determining whether the sequence number of the target tracking data is 0;
if so, the hash table is cleared.
In the embodiment of the application, after the target tracking data is obtained, it may be determined whether the sequence number of the target tracking data is 0, and if the sequence number of the target tracking data is 0, it indicates that the target tracking data is a new round of first data, and in this case, the hash table may be cleared. After the target hash value of the target tracking data is calculated, the fact that the repeated data of the target tracking data does not exist in the hash table is determined, the target tracking data can be directly sent to the tracking data summarizing module, and the target tracking data is recorded at the corresponding position in the hash table. If not, after calculating the target hash value of the target tracking data and determining that the repeated data of the target tracking data exists in the hash table, determining original text substitution data of the target tracking data, performing original text substitution, and sending the original text substitution data to the tracking data summarizing module.
It should be noted that, in the trace data processing module in the present application, duplicate data is looked up in the hash table based on the hash value of the original text of the trace data, so as to complete offset substitution. Therefore, the contents stored in the hash table, and how to dynamically update the contents of the hash table, have a direct impact on the final effect of reducing the trace data load. In the embodiment of the present application, the initial content of the hash table is empty, and during the running process of the chip, the content of the hash table is continuously updated by using each piece of current trace data. When the content of the hash table is updated, further optimization can be performed in combination with a specific application scenario, for example, if the frequency of occurrence of a plurality of specific trace data in the trace data of some functional modules is high, the original text of the trace data with high frequency of occurrence can be considered to be fixedly retained in the hash table, and is not dynamically updated, so that the hit rate when searching for repeated data is increased, and the effect of reducing the load of the trace data is improved.
Corresponding to the above method embodiments, the present application further provides a trace data processing apparatus, and the trace data processing apparatus described below and the trace data processing method described above may be referred to in correspondence with each other.
Referring to fig. 6, the apparatus may include the following modules:
a trace data obtaining unit 610, configured to obtain target trace data sent to the trace interface by the target function module;
a hash value calculation unit 620 for calculating a target hash value of the target trace data;
a duplicate data determination unit 630, configured to determine whether duplicate data of the target tracking data exists in the hash table based on the target hash value;
a substitute data determining unit 640, configured to determine, if there is duplicate data of the target tracking data in the hash table, original text substitute data of the target tracking data based on the sequence number of the target tracking data and the sequence number of the duplicate data of the target tracking data, where a data amount of the original text substitute data of the target tracking data is smaller than a data amount of the original text of the target tracking data;
and the data sending unit 650 is configured to send the original text of the target tracking data to the tracking data summarizing module instead of the original text of the target tracking data, so that the tracking data summarizing module summarizes the multiple paths of tracking data and sends the multiple paths of tracking data to the outside of the chip.
By applying the device provided by the embodiment of the application, after the target tracking data sent to the tracking interface by the target function module is obtained, the target hash value of the target tracking data is calculated, on the basis of the target hash value, under the condition that the repeated data of the target tracking data exist in the hash table, the original text replacing data of the target tracking data is determined on the basis of the sequence number of the target tracking data and the sequence number of the repeated data, and after the original text is replaced by the original text replacing data, the target tracking data is sent to the tracking data summarizing module. The tracking data is repeatedly judged at the interface for generating the tracking data, namely the source, so that the characteristic that the tracking data of a single functional module has periodic repetition can be better utilized, the original text of the tracking data with repeated data uses data with smaller data volume to replace the original text of the tracking data with repeated data, the data volume of the tracking data is reduced, the load of the tracking data is reduced, the risks of overflow and loss of the tracking data in the transmission of the tracking data in the chip are reduced, the tracking data can be effectively utilized, and the tracking function is more stable.
In one embodiment of the present application, each row of the hash table holds two elements: the sequence number of the trace data and the original text of the trace data; a duplicate data determination unit 630, configured to:
determining the position of target tracking data to be stored in the hash table according to the target hash value and the total row number of the hash table;
reading the content of the position to be stored in the hash table;
if the read content is valid, comparing the read original text of the existing tracking data with the original text of the target tracking data;
if the two data are the same, determining that repeated data of the target tracking data exist in the hash table;
and if the contents are not the same or the read contents are invalid, determining that the repeated data of the target tracking data does not exist in the hash table.
In a specific embodiment of the present application, the hash table updating apparatus further includes a hash table updating unit, configured to:
if the repeated data of the target tracking data exists in the hash table, replacing the sequence number of the repeated data of the target tracking data in the hash table by using the sequence number of the target tracking data;
and if the hash table does not contain repeated data of the target tracking data, recording the sequence number and the original text of the target tracking data at the position to be stored in the hash table.
In one embodiment of the present application, the substitute data determining unit 640 is configured to:
obtaining the sequence number of repeated data of the target tracking data in the hash table;
determining the number offset of the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data;
and determining the number offset as original text replacing data of the target tracking data.
In an embodiment of the present application, the data sending unit 650 is further configured to:
and if the repeated data of the target tracking data does not exist in the hash table, directly sending the original text of the target tracking data to the tracking data summarizing module.
In an embodiment of the present application, the system further includes a replacement flag output unit, configured to:
if repeated data of the target tracking data exist in the hash table, replacing the original text of the target tracking data with original text replacement data of the target tracking data, sending the replaced mark to the tracking data summarizing module, and meanwhile, determining that the original text of the target tracking data is replaced based on the replaced mark when the target tracking data is used;
if the repeated data of the target tracking data does not exist in the hash table, the original text of the target tracking data is sent to the tracking data summarizing module, and meanwhile the non-replacement mark is output, so that the original text of the target tracking data is determined to be not replaced based on the non-replacement mark when the target tracking data is used.
In a specific embodiment of the present application, the hash table clearing unit is further configured to:
before determining whether repeated data of the target tracking data exists in the hash table, determining whether the sequence number of the target tracking data is 0;
if so, the hash table is cleared.
Corresponding to the above method embodiment, an embodiment of the present application further provides a trace data processing device, including:
a memory for storing a computer program;
and the processor is used for realizing the steps of the tracking data processing method when executing the computer program.
As shown in fig. 7, in order to illustrate the structure of the trace data processing apparatus, the trace data processing apparatus may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through a communication bus 13.
In the embodiment of the present application, the processor 10 may be a Central Processing Unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, etc.
The processor 10 may call a program stored in the memory 11, and in particular, the processor 10 may perform operations in embodiments of the trace data processing method.
The memory 11 is used for storing one or more programs, the program may include program codes, the program codes include computer operation instructions, in this embodiment, the memory 11 stores at least the program for implementing the following functions:
acquiring target tracking data sent to a tracking interface by a target function module;
calculating a target hash value of the target tracking data;
determining whether repeated data of the target tracking data exists in the hash table or not based on the target hash value;
if repeated data of the target tracking data exist in the hash table, determining original text substitute data of the target tracking data based on the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data, wherein the data volume of the original text substitute data of the target tracking data is smaller than that of the original text of the target tracking data;
and replacing the original text of the target tracking data with the original text of the target tracking data, and sending the original text of the target tracking data to the tracking data summarizing module so that the tracking data summarizing module summarizes the multi-path tracking data and then sends the multi-path tracking data to the outside of the chip.
In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a hash calculation function, a lookup table function), and the like; the storage data area can store data created in the using process, such as tracking data, hash table recording data and the like.
Further, the memory 11 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device or other volatile solid state storage device.
The communication interface 13 may be an interface of a communication module for connecting with other devices or systems.
Of course, it should be noted that the structure shown in fig. 7 does not constitute a limitation of the tracking data processing device in the embodiment of the present application, and the tracking data processing device may include more or less components than those shown in fig. 7 or some components in combination in practical applications.
Corresponding to the above method embodiments, the present application further provides a computer readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the above trace data processing method.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The principle and the implementation of the present application are explained in the present application by using specific examples, and the above description of the embodiments is only used to help understanding the technical solution and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
Claims (10)
1. A method of processing trace data, comprising:
acquiring target tracking data sent to a tracking interface by a target function module;
calculating a target hash value of the target tracking data;
determining whether duplicate data of the target tracking data exists in a hash table based on the target hash value;
if the repeated data of the target tracking data exists in the hash table, determining original text substitute data of the target tracking data based on the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data, wherein the data volume of the original text substitute data of the target tracking data is smaller than that of the original text of the target tracking data;
and replacing the original text of the target tracking data with the original text substitute data of the target tracking data, and sending the original text substitute data to the tracking data summarizing module so that the tracking data summarizing module summarizes the multi-path tracking data and then sends the multi-path tracking data to the outside of the chip.
2. The method of claim 1, wherein each row of the hash table holds two elements: the sequence number of the trace data and the original text of the trace data; the determining whether duplicate data of the target tracking data exists in a hash table based on the target hash value includes:
determining the position to be stored of the target tracking data in the hash table according to the target hash value and the total line number of the hash table;
reading the content of the position to be stored in the hash table;
if the read content is valid, comparing the read original text of the existing tracking data with the original text of the target tracking data;
if the two data are the same, determining that the repeated data of the target tracking data exist in the hash table;
and if the contents are not the same or the read contents are invalid, determining that the repeated data of the target tracking data does not exist in the hash table.
3. The method of claim 2, further comprising:
replacing the sequence number of the repeated data of the target tracking data in the hash table with the sequence number of the target tracking data if the repeated data of the target tracking data exists in the hash table;
if the repeated data of the target tracking data does not exist in the hash table, recording the sequence number and the original text of the target tracking data at the position to be stored in the hash table.
4. The method according to claim 1, wherein the determining textual replacement data for the target tracking data based on the sequential number of the target tracking data and the sequential number of duplicate data of the target tracking data comprises:
obtaining the sequence number of the repeated data of the target tracking data in the hash table;
determining a number offset of the sequence number of the target tracking data and the sequence number of the repeated data of the target tracking data;
and determining the number offset as original text replacing data of the target tracking data.
5. The method of claim 1, further comprising:
and if the repeated data of the target tracking data does not exist in the hash table, directly sending the original text of the target tracking data to the tracking data summarizing module.
6. The method of claim 5, further comprising:
if repeated data of the target tracking data exists in the hash table, replacing the original text of the target tracking data with original text replacement data of the target tracking data, sending the replaced data to a tracking data summarizing module, and meanwhile outputting a replaced mark so as to determine that the original text of the target tracking data is replaced based on the replaced mark when the target tracking data is used;
if the repeated data of the target tracking data does not exist in the hash table, outputting an un-replaced flag while sending the original text of the target tracking data to the tracking data summarizing module, so as to determine that the original text of the target tracking data is not replaced based on the un-replaced flag when the target tracking data is used.
7. The method according to any one of claims 1 to 6, further comprising, prior to determining whether duplicate data of the target trace data exists in a hash table:
determining whether the sequence number of the target tracking data is 0;
and if so, emptying the hash table.
8. A trace data processing apparatus, comprising:
the tracking data acquisition unit is used for acquiring target tracking data sent to the tracking interface by the target function module;
a hash value calculation unit for calculating a target hash value of the target trace data;
a duplicate data determination unit, configured to determine whether duplicate data of the target tracking data exists in a hash table based on the target hash value;
a substitute data determination unit, configured to determine, if duplicate data of the target tracking data exists in the hash table, original text substitute data of the target tracking data based on the sequence number of the target tracking data and the sequence number of the duplicate data of the target tracking data, where a data amount of the original text substitute data of the target tracking data is smaller than a data amount of an original text of the target tracking data;
and the data sending unit is used for replacing the original text of the target tracking data with the original text substitute data of the target tracking data and sending the original text substitute data to the tracking data summarizing module so that the tracking data summarizing module summarizes the multi-path tracking data and then sends the multi-path tracking data to the outside of the chip.
9. A trace data processing apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the trace data processing method according to any one of claims 1 to 7 when executing said computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the tracking data processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010992198.1A CN112131221A (en) | 2020-09-18 | 2020-09-18 | Tracking data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010992198.1A CN112131221A (en) | 2020-09-18 | 2020-09-18 | Tracking data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112131221A true CN112131221A (en) | 2020-12-25 |
Family
ID=73841687
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010992198.1A Withdrawn CN112131221A (en) | 2020-09-18 | 2020-09-18 | Tracking data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112131221A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006121215A (en) * | 2004-10-19 | 2006-05-11 | Fuji Electric Holdings Co Ltd | Key exchange method and key exchange processing apparatus |
| US20100211987A1 (en) * | 2009-02-19 | 2010-08-19 | Pixel8 Networks, Inc. | Video deduplication, cache, and virtual private content delivery network |
| CN103095843A (en) * | 2013-01-28 | 2013-05-08 | 刘海峰 | Method and client of data backup based on version vectors |
| CN103116615A (en) * | 2013-01-28 | 2013-05-22 | 袁华强 | Data index method and server based edition vector |
| CN103118104A (en) * | 2013-01-28 | 2013-05-22 | 刘海峰 | Data restoration method based on version vector, and server |
| CN103403717A (en) * | 2013-01-09 | 2013-11-20 | 华为技术有限公司 | Data processing method and data processing device |
| CN103902410A (en) * | 2014-03-28 | 2014-07-02 | 西北工业大学 | Data backup acceleration method for cloud storage system |
| CN106503250A (en) * | 2016-11-10 | 2017-03-15 | 广东浪潮大数据研究有限公司 | A kind of log tracks method and apparatus |
-
2020
- 2020-09-18 CN CN202010992198.1A patent/CN112131221A/en not_active Withdrawn
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006121215A (en) * | 2004-10-19 | 2006-05-11 | Fuji Electric Holdings Co Ltd | Key exchange method and key exchange processing apparatus |
| US20100211987A1 (en) * | 2009-02-19 | 2010-08-19 | Pixel8 Networks, Inc. | Video deduplication, cache, and virtual private content delivery network |
| CN103403717A (en) * | 2013-01-09 | 2013-11-20 | 华为技术有限公司 | Data processing method and data processing device |
| WO2014107845A1 (en) * | 2013-01-09 | 2014-07-17 | 华为技术有限公司 | Data processing method and device |
| CN103095843A (en) * | 2013-01-28 | 2013-05-08 | 刘海峰 | Method and client of data backup based on version vectors |
| CN103116615A (en) * | 2013-01-28 | 2013-05-22 | 袁华强 | Data index method and server based edition vector |
| CN103118104A (en) * | 2013-01-28 | 2013-05-22 | 刘海峰 | Data restoration method based on version vector, and server |
| CN103902410A (en) * | 2014-03-28 | 2014-07-02 | 西北工业大学 | Data backup acceleration method for cloud storage system |
| CN106503250A (en) * | 2016-11-10 | 2017-03-15 | 广东浪潮大数据研究有限公司 | A kind of log tracks method and apparatus |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA1191273A (en) | Memory module selection and reconfiguration apparatus in a data processing system | |
| CA2433750A1 (en) | Automatic collection of trace detail and history data | |
| CN107204875B (en) | Data reporting link monitoring method and device, electronic equipment and storage medium | |
| CN101478481B (en) | Buffer management method and apparatus, data forwarding system | |
| CN112052010B (en) | ICD byte order automatic conversion method | |
| CN116089343A (en) | AXI-based data storage method, device, storage medium and equipment | |
| CN109918221B (en) | Hard disk error reporting analysis method, system, terminal and storage medium | |
| CN102193787B (en) | Methods for serialization and de-serialization, device and system | |
| CN110209605A (en) | The register reading/writing method and calculating equipment of PCIE Cardbus NIC Cardbus | |
| CN108170571B (en) | Chip tracking debugging device and method | |
| CN112131221A (en) | Tracking data processing method, device, equipment and storage medium | |
| US8751990B2 (en) | System for determining median values of video data | |
| US11847037B2 (en) | Device and method for high performance memory debug record generation and management | |
| US9948587B2 (en) | Data deduplication at the network interfaces | |
| CN116126763A (en) | Bus interconnection system, data processing method and device, electronic equipment and medium | |
| CN112312396B (en) | NoC tracking data generation method and related device | |
| CN113268395A (en) | Service data processing method, processing device and terminal | |
| JP2019125360A (en) | Data processing system and operating method thereof | |
| US20060256877A1 (en) | Rapid I/O Compliant Message Mapper | |
| US20060282719A1 (en) | Unique Addressable Memory Data Path | |
| CN112035391B (en) | Interface device and method for serial communication and electronic equipment | |
| CN113360498A (en) | Streaming data unloading processing method and device and server | |
| CN111857604A (en) | Method, apparatus, device and medium for quickly reconstructing packet management mapping reverse lookup table | |
| CN112132583A (en) | Transaction processing method and device of block chain, electronic equipment and readable storage medium | |
| CN109408431B (en) | Semiconductor device and method for profiling events in a semiconductor device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20201225 |