CN104954280B - A kind of data message processing method and device - Google Patents
A kind of data message processing method and device Download PDFInfo
- Publication number
- CN104954280B CN104954280B CN201510196654.0A CN201510196654A CN104954280B CN 104954280 B CN104954280 B CN 104954280B CN 201510196654 A CN201510196654 A CN 201510196654A CN 104954280 B CN104954280 B CN 104954280B
- Authority
- CN
- China
- Prior art keywords
- data message
- domain
- information
- sequence
- sequencing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 238000012163 sequencing technique Methods 0.000 claims abstract description 81
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000005457 optimization Methods 0.000 claims description 19
- 238000007906 compression Methods 0.000 claims description 12
- 230000006835 compression Effects 0.000 claims description 12
- 238000007789 sealing Methods 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 12
- 238000011282 treatment Methods 0.000 description 6
- 238000003860 storage Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000005538 encapsulation Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/38—Flow control; Congestion control by adapting coding or compression rate
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
- H04L67/5651—Reducing the amount or size of exchanged application data
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the invention discloses a kind of data message processing method and device, this method may include：A point domain is carried out to the data packet of former data message to handle, wherein described point of domain processing records the size information each gathered for the identical domain in the former data message to be divided in identity set；Key field is selected from the data packet of the data message, and the set where the key field is ranked up, and record ordering information；The set where foreign lands according to the sequencing information to removing the key field in the data message is ranked up；Acquisition includes the new data message of each set after the size information, the sequencing information and sequence.The embodiment of the present invention can improve the efficiency of data message processing.
Description
Technical field
The present invention relates to the communications field more particularly to a kind of data message processing methods and device.
Background technology
Currently, in data message treatment technology, data message may include multiple data packets, and the structure of data message
In be to be arranged as unit of data packet, i.e., multiple domains of each data packet are all arranged together.In this way to data message into
Each domain respectively to each data packet is just needed to handle when row processing, however, data message in many cases, is more
The content in identical domain may be identical or there are certain regularities in a data packet.Such as：Data message includes data packet
1, data packet 2 and data packet 3, wherein each data packet all includes domain A, domain B and domain C, wherein data packet 1, data packet 2 and data
The content of the domain A of packet 3 is identical.Each domain is handled respectively in this way, it will be to the content of repetition or to there are a set patterns
The content of rule property carries out separating processing, so as to cause the relatively low problem of the efficiency of data message processing.As it can be seen that current data
The efficiency that message structure is easy to cause data message processing is low.
Invention content
The present invention provides a kind of data message processing method and devices, can improve the efficiency of data message processing.
In a first aspect, a kind of data message processing method provided by the invention, including：
A point domain is carried out to the data packet of former data message to handle, wherein described to divide domain processing for by the former datagram
Identical domain in text is divided in identity set, and records the size information each gathered；
Key field is selected from the data packet of the data message, and the set where the key field is ranked up,
And record ordering information；
The set where foreign lands according to the sequencing information to removing the key field in the data message is ranked up；
Acquisition includes the new data message of each set after the size information, the sequencing information and sequence.
In the first possible realization method of first aspect, the method further includes：
The domain of each data packet of the former data message is adjusted, wherein the adjustment includes at least one of following：
One domain is split at least two domains, the sequence between domain is adjusted and at least two domains are merged into one
A domain；
The data packet to former data message carries out a point domain and handles, including：
A point domain is carried out to the data packet of the former data message after the adjustment to handle.
With reference to first aspect or the first possible realization method of first aspect, second in first aspect are possible
Realization method in, it is described according to the sequencing information in the data message remove the key field foreign lands where set
After being ranked up, the method further includes：
Code optimization processing is carried out to the set after sequence；
The acquisition includes the new data message of each set after the size information, the sequencing information and sequence,
Including：
Acquisition includes the new data of each set after the size information, the sequencing information and the code optimization are handled
Message.
With reference to first aspect or the first possible realization method of first aspect, the third in first aspect are possible
Realization method in, it is described acquisition include the size information, the sequencing information and sort after each set new data
Message, including：
Each set after the size information, the sequencing information and sequence is packaged, to obtain new datagrams
Text.
With reference to first aspect or the first possible realization method of first aspect, in the 4th kind of possibility of first aspect
Realization method in, the method further includes：
Each set after sequence is compressed；
The acquisition includes the new data message of each set after the size information, the sequencing information and sequence,
Including：
The size information, the sequencing information and compressed each set are packaged, to obtain new number
According to message.
Second aspect, the present invention provide a kind of data message processing unit, including：Point domain unit, the first sequencing unit, the
Two sequencing units and acquiring unit, wherein：
Described point of domain unit carries out a point domain for the data packet to former data message and handles, wherein described that domain processing is divided to use
It is divided in identity set in by the identical domain in the former data message, and records the size information each gathered；
First sequencing unit, for selecting key field from the data packet of the data message, and to the key
Set where domain is ranked up, and record ordering information；
Second sequencing unit, for according to the sequencing information in the data message in addition to the key field
Set where domain is ranked up；
The acquiring unit, for obtains include the size information, the sequencing information and sort after each set
New data message.
In the first possible realization method of second aspect, described device further includes：
Adjustment unit, the domain for each data packet to the former data message are adjusted, wherein the adjustment includes
It is at least one of following：
One domain is split at least two domains, the sequence between domain is adjusted and at least two domains are merged into one
A domain；
Described point of domain unit is handled for carrying out a point domain to the data packet of the former data message after the adjustment.
In conjunction with the possible realization method of the first of second aspect or second aspect, second in second aspect may
Realization method in, described device further includes：
Coding unit, for carrying out code optimization processing to the set after sequence；
The acquiring unit includes after the size information, the sequencing information and the code optimization are handled for obtaining
The new data message of each set.
In conjunction with the possible realization method of the first of second aspect or second aspect, the third in second aspect may
Realization method in, the acquiring unit be used for by the size information, the sequencing information and sequence after it is each gather into
Row encapsulation, to obtain new data message.
In conjunction with the possible realization method of the first of second aspect or second aspect, in the 4th kind of possibility of second aspect
Realization method in, described device further includes：
Compression unit, for being compressed to each set after sequence；
The acquiring unit is used to carry out the size information, the sequencing information and compressed each set
Encapsulation, to obtain new data message.
In above-mentioned technical proposal, a point domain is carried out to the data packet of former data message and is handled, wherein described point of domain processing is used for
Identical domain in the former data message is divided in identity set, and records the size information each gathered；From described
Key field is selected in the data packet of data message, and the set where the key field is ranked up, and record ordering information；
The set where foreign lands according to the sequencing information to removing the key field in the data message is ranked up；Acquisition includes
The new data message of each set after the size information, the sequencing information and sequence.The data of new data message in this way
In structure repeat content or to there are the content of certain regularity be divided to set in, in this way handle the datagram
Treatment effeciency can be improved when literary.
Description of the drawings
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
Obtain other attached drawings according to these attached drawings.
Fig. 1 is a kind of flow diagram of data message processing method provided in an embodiment of the present invention；
Fig. 2 is a kind of example schematic of data message provided in an embodiment of the present invention；
Fig. 3 is a kind of example schematic of data sorting provided in an embodiment of the present invention；
Fig. 4 is a kind of example schematic of data message sequence provided in an embodiment of the present invention；
Fig. 5 is the flow diagram of another data message processing method provided in an embodiment of the present invention；
Fig. 6 is a kind of example schematic of data message adjustment provided in an embodiment of the present invention；
Fig. 7 is the example schematic of another data message adjustment provided in an embodiment of the present invention；
Fig. 8 is a kind of parsing schematic diagram for being not fixed length field of data message provided in an embodiment of the present invention；
Fig. 9 is a kind of example schematic of data encoding optimization processing provided in an embodiment of the present invention；
Figure 10 is a kind of example schematic of data message encapsulation provided in an embodiment of the present invention；
Figure 11 is a kind of example schematic of data message compression provided in an embodiment of the present invention；
Figure 12 is the flow diagram of another data message processing method provided in an embodiment of the present invention；
Figure 13 is a kind of structural schematic diagram of data message processing unit provided in an embodiment of the present invention；
Figure 14 is the structural schematic diagram of another data message processing unit provided in an embodiment of the present invention；
Figure 15 is the structural schematic diagram of another data message processing unit provided in an embodiment of the present invention；
Figure 16 is the structural schematic diagram of another data message processing unit provided in an embodiment of the present invention；
Figure 17 is the structural schematic diagram of another data message processing unit provided in an embodiment of the present invention.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other without creative efforts
Embodiment shall fall within the protection scope of the present invention.
Referring to Fig. 1, Fig. 1 is a kind of flow diagram of data message processing method provided in an embodiment of the present invention, such as
Shown in Fig. 1, include the following steps：
101, it carries out a point domain to the data packet of former data message to handle, wherein described to divide domain processing for by the former number
It is divided in identity set according to the identical domain in message, and records the size information each gathered.
Former data message in the present embodiment may include multiple data packets, in addition, data packet can have a fixed number
According to the data packet of structure, such as：Packet.In addition, each data packet may include multiple domains, such as：Each data packet can
To include domain A, domain B, domain C and domain D, as shown in Fig. 2, former data packet may include data packet 0, data packet 1,2 sum number of data packet
According to packet 3, data packet 0 includes：Domain A0, domain B0, domain C0 and domain D0, data packet 1 include：Domain A1, domain B1, domain C1 and domain D1, data
Wrapping 2 includes：Domain A2, domain B2, domain C2 and domain D2, data packet 3 include：Domain A3, domain B3, domain C3 and domain D3.In this way by step 101
Domain A0, domain A1, domain A2 and domain A3 can be divided in a set, and domain B0, domain B1, domain B2 and domain B3 are divided in separately
One set etc., thus may be implemented identical domain in former data message being divided in identity set.
In addition, above-mentioned size information can record each domain in the size information each gathered, or each set of record
Size information etc..
102, key field is selected from the data packet of the data message, and the set where the key field is arranged
Sequence, and record ordering information.
Wherein, above-mentioned key field can be one or more key field, in addition, when being multiple key fields, Ke Yixian
One layer of sequence is carried out by one of, if identical carry out two layers of sequence according still further to second, such as：As shown in figure 3, key field
Including IP address domain and Traffic Domain, first layer sequence first can be carried out according to IP address domain by step 102, to it is identical again
Second layer sequence is carried out according to Traffic Domain.
In addition, above-mentioned selection key field can preset a kind of selection rule, such as：It selects most important in data packet
Domain preassign important domain as key field etc. as key field or user, this is not construed as limiting.In addition, sequence rule
Can be then to be sorted using merger, be either incremented by rule or support part be incremented by either part successively decrease equal ordering rules or
It can also be user-defined ordering rule etc..
Such as：As shown in figure 4, selecting domain B as key field, the sequence of sequence is B3, domain B1, domain B0 and domain B2.
103, the set where the foreign lands according to the sequencing information to removing the key field in the data message carries out
Sequence.
The sequence of step 103 can be the sequence using above-mentioned steps 102, such as：As shown in figure 4, what step 102 sorted
Sequence is B3, domain B1, domain B0 and domain B2, and such step 103 can be ranked up other three set shown in Fig. 4, from
And sequence as shown in Figure 4 is obtained, each set is all ranked up with 3,1,0 to 2 sequence.
104, acquisition includes the new data message of each set after the size information, the sequencing information and sequence.
It can realize through the above steps and generate new data message as shown in Figure 4, identical domain in the data message
It is arranged on together, and be ranked up according to certain ordering rule, former data message thus may be implemented can be by some
The content repeated is arranged together, such as：As shown in figure 3, identical IP address is arranged together, or there will be certain rules
Content be arranged together, as shown in figure 3, similar IP address is arranged together.In this way to the new data message
When reason, so that it may handled together the content repeated or the content that there is certain rule, so as to improve at data message
The efficiency of reason.
Optionally, the above method can be applied to the equipment for arbitrarily having data message processing, such as：Server, people's meter
Calculation machine (Personal Computer, PC), tablet computer, mobile phone, electronic reader, remote controler, vehicle-mounted is set laptop
Standby, Web TV, wearable device etc. have the smart machine of data storage and processing ability.
In addition, above-mentioned data message can be the message for arbitrarily having specific data structure, wherein the specific data structure
Message can be to data structure message known to developer or user, such as：List message or network message etc..
In above-mentioned technical proposal, a point domain is carried out to the data packet of former data message and is handled, wherein described point of domain processing is used for
Identical domain in the former data message is divided in identity set, and records the size information each gathered；From described
Key field is selected in the data packet of data message, and the set where the key field is ranked up, and record ordering information；
The set where foreign lands according to the sequencing information to removing the key field in the data message is ranked up；Acquisition includes
The new data message of each set after the size information, the sequencing information and sequence.The data of new data message in this way
In structure repeat content or to there are the content of certain regularity be divided to set in, in this way handle the datagram
Treatment effeciency can be improved when literary.
Referring to Fig. 5, Fig. 5 is the flow diagram of another data message processing method provided in an embodiment of the present invention,
As shown in figure 5, including the following steps：
501, the domain of each data packet of former data message is adjusted.
Wherein, the adjustment may include at least one of following：
One domain is split at least two domains, the sequence between domain is adjusted and at least two domains are merged into one
A domain.
Such as：In the former data message of identification after each data packet and each packet structure and the information in each domain,
The domain of each data packet of the former data message can be adjusted.Wherein, can be to be adjusted to part domain here,
Such as：As shown in fig. 6, former data message includes timestamp domain, and often immobilize in the presence of a part in timestamp domain
Feature, 2E 85C93A as shown in FIG. 6, in addition, the random nature of another part of the timestamp is stronger, such as Fig. 6 institutes
Rear 2 bytes shown.Above-mentioned steps 501 can be as shown in fig. 6, be divided into a list by the front in the timestamp domain in this way
Only domain, the content of all data packets is all identical in the domain, and another part is divided into another individual domain.Pass through this in this way
Changeless domain as shown in FIG. 6 can be arranged together by method, small thereby using the Entropy Changes of data, then, locating
It, will be very high to the efficiency of the contents processing repeated when managing the data message.Such as：Compression when these identical domains more
Matching is easy tod produce, to promote the compression efficiency and compression ratio of data.
Such as：The domain of identical rule can also be put together, to excavate the regularity and relevance of data.Such as：
As shown in fig. 7, each data packet includes field of constants, then the two domains can be adjusted to to be formed together bigger
Field of constants, or as shown in fig. 7, the position of one of field of constants is adjusted, to increase the relevance of data.
Such as：Multiple domains can also be merged into the domain of a bigger, this multiple domain can have High relevancy in itself
The characteristics of, longer matching length can be obtained after merging in dictionary compression, to improve compression ratio.Such as：For including association
The data packet of view number and two domains of port numbers, can merge into a domain by protocol number and the two domains of port numbers.In another example：
In data message shown in Fig. 7, name and the two domains of IP address can be merged into a domain, can also by activity and website this
A domain is merged into two domains.
The repeatability and systematicness that data message can be improved through the above steps, keep the Entropy Changes of data small, so more
It is more efficient when being suitble to compression, and handling.
502, it carries out a point domain to the data packet of the former data message after the adjustment to handle, wherein described that domain processing is divided to use
It is divided in identity set in by the identical domain in the former data message, and records the size information each gathered.
It is above-mentioned that identical domain is divided in same set it is to be understood that using identical domain as one in the present embodiment
Subfile.
In addition, the domain of data packet can be the domain of regular length in above-mentioned original data message, or it is not fixed the domain of length,
Wherein, be not fixed length domain can by obtaining its length information to the parsing of the domain of regular length, such as：As shown in figure 8, number
According to including IP1 length fields and the domains IP1, wherein the length of IP1 length fields is 1 byte, and function works as the domains IP1 for indicating
For be IPV4 when the content in the domains IP1 IP address when, IP1 length of field is 4 bytes, when the domains IP1 are when the content in the domains IP1 is IPV6
IP address when, IP1 length of field be 16 bytes.
503, key field is selected from the data packet of the data message, and the set where the key field is arranged
Sequence, and record ordering information.
In the present embodiment, above-mentioned key field can be have the domain of information separation characteristic, such as：Protocol number, port numbers, IP
The domains such as address.
In addition, above-mentioned sequencing information can be sequence dictionary.
504, the set where the foreign lands according to the sequencing information to removing the key field in the data message carries out
Sequence.
505, acquisition includes the new data message of each set after the size information, the sequencing information and sequence.
In the present embodiment, after step 504, the method can also include the following steps：
Code optimization processing is carried out to the set after sequence；
Step 505 may include：
Acquisition includes the new data of each set after the size information, the sequencing information and the code optimization are handled
Message.
Wherein, above-mentioned code optimization processing can carry out code optimization processing to some or all of set, wherein compile
LRU and delta codings may be used in code optimization processing.
Wherein, delta coded treatments can be that the next data of user subtract a data, be suitble to the number of local order
According to, such as the sort key field used or timestamp etc..
Wherein, LRU codings standby numeric distribution can be suitble to sparse or local value be distributed sparse feature domain collection
It closes, such as：The set in the domains such as protocol number, IP address.LRU uses a LRU record sheet, and the table is for recording most recently used number
According to value, next numerical value is detected every time, if there are identical numerical value in LRU record sheets, replaces with LRU record sheets
In index (index), otherwise increase by one indicate be inserted into index before numerical value.In addition, LRU record sheets can be brushed in real time
Newly, fixed empirical value can also only be recorded.Such as：As shown in figure 9, original mono- domain size of 16bit, after LRU is encoded, such as
Fruit is found in LRU record sheets, has reformed into 2bit.
In the present embodiment, step 505 may include：
Each set after the size information, the sequencing information and sequence is packaged, to obtain new datagrams
Text.
Wherein, above-mentioned size information and sequencing information can be continuous the stem for being placed on new data message, such as：Such as figure
Shown in 10a, it should be noted that in the present embodiment, data message can be understood as a file, and gathers and can be understood as son
File, the subfile size in Figure 10 mean that the size information of the subfile.In addition, in the present embodiment, it can be with discontinuous
Placement, such as：As shown in fig. lob.In addition, the packaged type of new data message is not construed as limiting in the present embodiment, as long as
The packaged type that can be parsed can.
In the present embodiment, the above method can also include the following steps：
Each set after sequence is compressed；
Step 505 may include：
The size information, the sequencing information and compressed each set are packaged, to obtain new number
According to message.
The embodiment, may be implemented first to compress and encapsulates again, specifically can as shown in fig. 11a, the data that domain will be divided to sort
It is compressed, wherein be compressed in Figure 11 and indicated with C.
In addition, in the present embodiment, it can also as shown in figure 11b, first by the size information, the sequencing information and sequence
After each set afterwards is packaged, then compressed.
It should be noted that in the present embodiment, the above method can be applied to support lossless data compression or damage number
According to the scene of compression, in addition, the above method can also be applied to any required scene for promoting data redundancy and regularity, example
Such as：In the scene that database scene etc. needs to arrange data.
In the present embodiment, the embodiment of plurality of optional is increased on the basis of embodiment shown in Fig. 1, and can
Realize the efficiency for improving data message processing.
Please refer to Fig.1 the flow signal that 2, Figure 12 is another data message processing method provided in an embodiment of the present invention
Figure, as shown in figure 12, includes the following steps：
1201, the information in each domain in each data packet and packet structure in list message is identified.
1202, part domain is adjusted.
Wherein, the adjustment may include at least one of following：
One domain is split at least two domains, the sequence between domain is adjusted and at least two domains are merged into one
A domain；
1203, a point domain is carried out to the data after adjustment, obtains multiple subfiles.
Wherein, the identical domain in data message is divided in same subfile, and records the size letter of each subfile
Breath.
1204, the corresponding subfile of key field is ranked up.
1205, record ordering dictionary.
Wherein, above-mentioned sequence dictionary may include sequencing information.
1206, all subfiles are reset according to sequence dictionary.
Wherein, the sequence of all subfiles can be ranked up according to above-mentioned sequence dictionary.
1207, code optimization processing is carried out to part subfile.
1208, all subfiles, all subfile size informations, sequence dictionary information are packaged.
Can be specifically to all subfile size informations of all subfiles after coding priority processing, sequence dictionary information
It is packaged.
1209, the file after encapsulation is compressed.
The embodiment can improve the compression ratio of data.
The following is an embodiment of the apparatus of the present invention, and apparatus of the present invention embodiment is for executing the method for the present invention embodiment one to three
The method of realization is illustrated only and is not disclosed with the relevant part of the embodiment of the present invention, particular technique details for convenience of description
, please refer to the embodiment of the present invention one, embodiment two and embodiment three.
The structural schematic diagram that 3, Figure 13 is a kind of data message processing unit provided in an embodiment of the present invention is please referred to Fig.1,
Including：Divide domain unit 131, the first sequencing unit 132, the second sequencing unit 133 and acquiring unit 134, wherein：
Divide domain unit 131, carry out a point domain for the data packet to former data message and handle, wherein is described that domain processing is divided to use
It is divided in identity set in by the identical domain in the former data message, and records the size information each gathered.
First sequencing unit 132, for selecting key field from the data packet of the data message, and to the key field
The set at place is ranked up, and record ordering information.
Second sequencing unit 133, for according to the sequencing information in the data message in addition to the key field
Set where domain is ranked up.
Acquiring unit 134, for obtains include the size information, the sequencing information and sort after each set
New data message.
Optionally, above-mentioned apparatus can be applied to the equipment for arbitrarily having data message processing, such as：Server, PC, pen
Remember that this computer, tablet computer, mobile phone, electronic reader, remote controler, mobile unit, Web TV, wearable device etc. have number
According to the smart machine of storage and processing ability.
In addition, above-mentioned data message can be the message for arbitrarily having specific data structure, wherein the specific data structure
Message can to data structure message known to developer or user, such as：List message or network message etc..
In above-mentioned technical proposal, a point domain is carried out to the data packet of former data message and is handled, wherein described point of domain processing is used for
Identical domain in the former data message is divided in identity set, and records the size information each gathered；From described
Key field is selected in the data packet of data message, and the set where the key field is ranked up, and record ordering information；
The set where foreign lands according to the sequencing information to removing the key field in the data message is ranked up；Acquisition includes
The new data message of each set after the size information, the sequencing information and sequence.The data of new data message in this way
In structure repeat content or to there are the content of certain regularity be divided to set in, in this way handle the datagram
Treatment effeciency can be improved when literary.
Please refer to Fig.1 the structural representation that 4, Figure 14 is another data message processing unit provided in an embodiment of the present invention
Figure, including：Adjustment unit 141 divides domain unit 142, the first sequencing unit 143, the second sequencing unit 144 and acquiring unit 145,
Wherein：
Adjustment unit 141, the domain for each data packet to the former data message are adjusted.
Wherein, the adjustment includes at least one of following：
One domain is split at least two domains, the sequence between domain is adjusted and at least two domains are merged into one
A domain；
Divide domain unit 142, carries out a point domain for the data packet to the former data message after the adjustment and handle, wherein institute
State the processing of point domain for the identical domain in the former data message is divided in identity set, and is recorded each gather it is big
Small information.
First sequencing unit 143, for selecting key field from the data packet of the data message, and to the key field
The set at place is ranked up, and record ordering information.
Second sequencing unit 144, for according to the sequencing information in the data message in addition to the key field
Set where domain is ranked up.
Acquiring unit 145, for obtains include the size information, the sequencing information and sort after each set
New data message.
Optionally, as shown in figure 15, above-mentioned apparatus can also include：
Coding unit 146, for carrying out code optimization processing to the set after sequence；
It includes at the size information, the sequencing information and the code optimization that acquiring unit 145, which can be used for obtaining,
The new data message of each set after reason.
Optionally, acquiring unit 145 can be used for each collection after the size information, the sequencing information and sequence
Conjunction is packaged, to obtain new data message.
Optionally, as shown in figure 16, above-mentioned apparatus can also include：
Compression unit 147, for being compressed to each set after sequence；
Acquiring unit 145 can be used for the size information, the sequencing information and compressed each set
It is packaged, to obtain new data message.
In the present embodiment, the embodiment of plurality of optional is increased on the basis of embodiment shown in Figure 13, and all may be used
To realize the efficiency for improving data message processing.
Please refer to Fig.1 the structural representation that 7, Figure 17 is another data message processing unit provided in an embodiment of the present invention
Figure, as shown in figure 17, which includes：At least one processor 171, such as CPU, at least one network interface 174 or its
His user interface 173, memory 175, at least one communication bus 172.Communication bus 172 is for realizing between these components
Connection communication.The calculate node 170 includes optionally user interface 173, including display, keyboard or pointing device (for example,
Mouse, trace ball (trackball), touch-sensitive plate or touch sensitive display screen).Memory 175 may include high-speed RAM memory,
May also further include non-labile memory (non-volatile memory), for example, at least a magnetic disk storage.Storage
Device 175 can include optionally at least one storage device for being located remotely from aforementioned processor 171.
In some embodiments, memory 175 stores following element, executable modules or data structures, or
Their subset of person or their superset:
Operating system 1751, including various system programs, hardware based for realizing various basic businesses and processing
Task；
Application program module 1752, including various application programs, for realizing various applied business.
In embodiments of the present invention, it is used for by the program or instruction, processor 171 of calling memory 175 to store：
A point domain is carried out to the data packet of former data message to handle, wherein described to divide domain processing for by the former datagram
Identical domain in text is divided in identity set, and records the size information each gathered；
Key field is selected from the data packet of the data message, and the set where the key field is ranked up,
And record ordering information；
The set where foreign lands according to the sequencing information to removing the key field in the data message is ranked up；
Acquisition includes the new data message of each set after the size information, the sequencing information and sequence.
Optionally, processor 171 can be also used for：
The domain of each data packet of the former data message is adjusted, wherein the adjustment includes at least one of following：
One domain is split at least two domains, the sequence between domain is adjusted and at least two domains are merged into one
A domain；
Optionally, the data packet to former data message that processor 171 executes carries out a point domain processing routine, may include：
A point domain is carried out to the data packet of the former data message after the adjustment to handle.
Optionally, processor 171 execute according to the sequencing information to the data message in except the key field it
After set where foreign lands is ranked up, it can be also used for：
Code optimization processing is carried out to the set after sequence；
Optionally, processor 171 execute acquisitions include the size information, the sequencing information and sort after it is each
The new data message procedure of set may include：
Acquisition includes the new data of each set after the size information, the sequencing information and the code optimization are handled
Message.
Optionally, processor 171 execute acquisitions include the size information, the sequencing information and sort after it is each
The program of the new data message of set may include：
Each set after the size information, the sequencing information and sequence is packaged, to obtain new datagrams
Text.
Optionally, processor 171 can be also used for：
Each set after sequence is compressed；
Optionally, processor 171 execute acquisitions include the size information, the sequencing information and sort after it is each
The program of the new data message of set may include：
The size information, the sequencing information and compressed each set are packaged, to obtain new number
According to message.
In the present embodiment, a point domain is carried out to the data packet of former data message and is handled, wherein is described to divide domain processing for by institute
The identical domain stated in former data message is divided in identity set, and records the size information each gathered；From the data
Key field is selected in the data packet of message, and the set where the key field is ranked up, and record ordering information；Foundation
Set where foreign lands of the sequencing information to removing the key field in the data message is ranked up；It includes described to obtain
The new data message of each set after size information, the sequencing information and sequence.The data structure of new data message in this way
The content of middle repetition or to there are the content of certain regularity be divided to set in, in this way when handling the data message
Treatment effeciency can be improved.
One of ordinary skill in the art will appreciate that realizing all or part of flow in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the program can be stored in a computer read/write memory medium
In, the program is when being executed, it may include such as the flow of the embodiment of above-mentioned each method.Wherein, the storage medium can be magnetic
Dish, CD, read-only memory (Read-Only Memory, ROM) or random access memory (Random Access
Memory, abbreviation RAM) etc..
The above disclosure is only the preferred embodiments of the present invention, cannot limit the right model of the present invention with this certainly
It encloses, therefore equivalent changes made in accordance with the claims of the present invention, is still within the scope of the present invention.
Claims (10)
1. a kind of data message processing method, which is characterized in that including：
A point domain is carried out to the data packet of former data message to handle, wherein described to divide domain processing for be in the former data message
Identical domain be divided in identity set, and record the size information each gathered；
Key field is selected from the data packet of the data message, and the set where the key field is ranked up, and is remembered
Record sequencing information；
The set where foreign lands according to the sequencing information to removing the key field in the data message is ranked up；
Acquisition includes the new data message of each set after the size information, the sequencing information and sequence.
2. the method as described in claim 1, which is characterized in that the method further includes：
The domain of each data packet of the former data message is adjusted, wherein the adjustment includes at least one of following：
One domain is split at least two domains, the sequence between domain is adjusted and at least two domains are merged into one
Domain；
The data packet to former data message carries out a point domain and handles, including：
A point domain is carried out to the data packet of the former data message after the adjustment to handle.
3. method as claimed in claim 1 or 2, which is characterized in that it is described according to the sequencing information to the data message
In except the key field foreign lands where set be ranked up after, the method further includes：
Code optimization processing is carried out to the set after sequence；
The acquisition includes the new data message of each set after the size information, the sequencing information and sequence, including：
Acquisition includes the new datagrams of each set after the size information, the sequencing information and the code optimization are handled
Text.
4. method as claimed in claim 1 or 2, which is characterized in that the acquisition includes the size information, sequence letter
The new data message of each set after breath and sequence, including：
Each set after the size information, the sequencing information and sequence is packaged, to obtain new data message.
5. method as claimed in claim 1 or 2, which is characterized in that the method further includes：
Each set after sequence is compressed；
The acquisition includes the new data message of each set after the size information, the sequencing information and sequence, including：
The size information, the sequencing information and compressed each set are packaged, to obtain new datagrams
Text.
6. a kind of data message processing unit, which is characterized in that including：Divide domain unit, the first sequencing unit, the second sequencing unit
And acquiring unit, wherein：
Described point of domain unit carries out a point domain for the data packet to former data message and handles, wherein described to divide domain processing for inciting somebody to action
Identical domain in the original data message is divided in identity set, and records the size information each gathered；
First sequencing unit, for selecting key field from the data packet of the data message, and to the key field institute
Set be ranked up, and record ordering information；
Second sequencing unit is used for according to the sequencing information to the foreign lands institute except the key field in the data message
Set be ranked up；
The acquiring unit, for obtains include the size information, the sequencing information and sort after each set it is new
Data message.
7. device as claimed in claim 6, which is characterized in that described device further includes：
Adjustment unit, the domain for each data packet to the former data message are adjusted, wherein the adjustment includes as follows
At least one of：
One domain is split at least two domains, the sequence between domain is adjusted and at least two domains are merged into one
Domain；
Described point of domain unit is handled for carrying out a point domain to the data packet of the former data message after the adjustment.
8. device as claimed in claims 6 or 7, which is characterized in that described device further includes：
Coding unit, for carrying out code optimization processing to the set after sequence；
It includes each after the size information, the sequencing information and the code optimization are handled that the acquiring unit, which is used to obtain,
The new data message of set.
9. device as claimed in claims 6 or 7, which is characterized in that the acquiring unit is used for the size information, described
Each set after sequencing information and sequence is packaged, to obtain new data message.
10. device as claimed in claims 6 or 7, which is characterized in that described device further includes：
Compression unit, for being compressed to each set after sequence；
The acquiring unit is for sealing the size information, the sequencing information and compressed each set
Dress, to obtain new data message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510196654.0A CN104954280B (en) | 2015-04-23 | 2015-04-23 | A kind of data message processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510196654.0A CN104954280B (en) | 2015-04-23 | 2015-04-23 | A kind of data message processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104954280A CN104954280A (en) | 2015-09-30 |
| CN104954280B true CN104954280B (en) | 2018-10-09 |
Family
ID=54168642
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510196654.0A Active CN104954280B (en) | 2015-04-23 | 2015-04-23 | A kind of data message processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104954280B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107800653B (en) * | 2016-09-06 | 2021-09-03 | 中兴通讯股份有限公司 | Message compression method and device |
| CN106789695B (en) * | 2016-11-24 | 2020-04-03 | 杭州迪普科技股份有限公司 | Message processing method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101286895A (en) * | 2008-05-22 | 2008-10-15 | 上海交通大学 | Dynamic configurable data monitoring system and method for distributed network |
| CN103401850A (en) * | 2013-07-19 | 2013-11-20 | 北京星网锐捷网络技术有限公司 | Message filtering method and device |
| CN104267937A (en) * | 2014-09-16 | 2015-01-07 | 中国工商银行股份有限公司 | Online small-batch data processing method and online small-batch data processing system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2304922A1 (en) * | 2008-06-17 | 2011-04-06 | Attivio, Inc. | Ordered message processing |
-
2015
- 2015-04-23 CN CN201510196654.0A patent/CN104954280B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101286895A (en) * | 2008-05-22 | 2008-10-15 | 上海交通大学 | Dynamic configurable data monitoring system and method for distributed network |
| CN103401850A (en) * | 2013-07-19 | 2013-11-20 | 北京星网锐捷网络技术有限公司 | Message filtering method and device |
| CN104267937A (en) * | 2014-09-16 | 2015-01-07 | 中国工商银行股份有限公司 | Online small-batch data processing method and online small-batch data processing system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104954280A (en) | 2015-09-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107102941B (en) | Test case generation method and device | |
| CN106980669B (en) | A kind of storage of data, acquisition methods and device | |
| CN107895011B (en) | Session information processing method, system, storage medium and electronic equipment | |
| US20100253556A1 (en) | Method of constructing an approximated dynamic huffman table for use in data compression | |
| CN104205035B (en) | File Mapping compresses | |
| CN110347716A (en) | Daily record data processing method, device, terminal and storage medium | |
| CN109951538A (en) | Data processing method, device, system, electronic equipment and storage medium | |
| CN104426713A (en) | Method and device for monitoring network site access effect data | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN106407442B (en) | A kind of mass text data processing method and device | |
| CN108710670A (en) | A kind of log analysis method, device, electronic equipment and readable storage medium storing program for executing | |
| CN104765746A (en) | Data processing method and device for mobile communication terminal browser | |
| CN105227425A (en) | The method of syndication message, equipment and network social intercourse system | |
| CN106599120A (en) | Stream processing framework-based data processing method and apparatus | |
| US10140377B2 (en) | Data processing, data collection | |
| CN104954280B (en) | A kind of data message processing method and device | |
| CN109408502A (en) | A kind of data standard processing method, device and its storage medium | |
| CN105550179A (en) | Webpage collection method and browser plug-in | |
| CN106559498A (en) | Air control data collection platform and its collection method | |
| CN105653550B (en) | Webpage filtering method and device | |
| WO2021189766A1 (en) | Data visualization method and related device | |
| CN110851317A (en) | Method, device, equipment and storage medium for predicting IOPS performance data of storage equipment | |
| CN107888663A (en) | A kind of method of distribution of document, equipment and computer-readable medium | |
| CN107562426A (en) | Without the method and system for burying point type high in the clouds collection and analysis browser Trace | |
| CN109559121A (en) | Transaction path calls exception analysis method, device, equipment and readable storage medium storing program for executing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20211227 Address after: 450046 Floor 9, building 1, Zhengshang Boya Plaza, Longzihu wisdom Island, Zhengdong New Area, Zhengzhou City, Henan Province Patentee after: xFusion Digital Technologies Co., Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Patentee before: HUAWEI TECHNOLOGIES Co.,Ltd. |