CN105071897B - A kind of network real-time audio conversation media data multi-path redundancy transmission method - Google Patents
A kind of network real-time audio conversation media data multi-path redundancy transmission method Download PDFInfo
- Publication number
- CN105071897B CN105071897B CN201510388523.2A CN201510388523A CN105071897B CN 105071897 B CN105071897 B CN 105071897B CN 201510388523 A CN201510388523 A CN 201510388523A CN 105071897 B CN105071897 B CN 105071897B
- Authority
- CN
- China
- Prior art keywords
- media data
- audio
- audio media
- jitter elimination
- fsn
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/22—Arrangements for detecting or preventing errors in the information received using redundant apparatus to increase reliability
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Telephonic Communication Services (AREA)
Abstract
The present invention provides a kind of network real-time audio conversation media data multi-path redundancy transmission method, including：The audio coding decoding form and the network transmission protocol that the transmitting terminal of audio media data consults the audio media data of capture according to real-time audio session both sides are packaged, it is sent to simultaneously using redundant fashion in a plurality of not exclusively intersecting transmission path, carry out the transmission of real-time audio media data multi-path redundancy, the receiving terminal of audio media data carries out redundant eliminating and reorganization operation to the grouping of the audio media data from different transmission path of reception, and the audio coding decoding form and the network transmission protocol consulted according to real-time audio session both sides reduce original audio data.Thus, packet loss, time delay and shake are to the entire effect of the end-to-end transmission of media data caused by the variation of single-pathway condition can effectively be reduced, and then the reliability of real-time audio media data transmission is promoted, promote the user experience quality (QoE) of network audio session service.
Description
Technical field：
The present invention relates to network communication technology field more particularly to a kind of network real-time audio conversation media data multipath are superfluous
Remaining transmission method.
Background technology：
Internet can not provide service quality using the transmission mechanism of " doing one's best " for end-to-end media transmission
(QoS) ensure.Real-time audio session is a kind of typical network traffic, and media transmission generally takes up several kbps to hundreds of
The transmission bandwidth of kbps is a kind of narrow band service, higher to transmission requirement of real-time although occupied bandwidth is small.IP leads at present
Letter terminal audio frequency media transmission generally use tradition RTP and udp protocol are transmitted control, could be used without service quality (QoS)
The end-to-end default route path (single-pathway) ensured is transmitted, and number caused by congestion is not known in end-to-end path link
Often resulting in audio media data according to packet loss and delay variation can not recombinate and decode, and seriously affect the product of real-time audio session
Matter.Internet has become the important bearer network of network communication, improves real-time audio conversation media transmission mode, promotes business
Quality of experience is major issue urgently to be resolved hurrily.
The content of the invention：
The defects of for the prior art, the present invention provide a kind of network real-time audio conversation media data multi-path redundancy transmission
Method, this method combination real-time audio media occupy the relatively low feature of transmission bandwidth, not exclusively intersecting end to end by building
Transmission path, and using multi-path redundancy transmission controlling mechanism and agreement, realize real-time audio media data multipath redundant fashion
Transmission.Packet loss, time delay and shake are to matchmaker caused by this multi-path redundancy transmission method can effectively reduce the variation of single-pathway condition
The entire effect of the end-to-end transmission of volume data, and then the reliability of real-time audio media data transmission is promoted, promote network audio
The user experience quality of session service.
The present invention provides a kind of network real-time audio conversation media data multi-path redundancy transmission method, including：
The sound that the transmitting terminal of audio media data consults the audio media data of capture according to real-time audio session both sides
Frequency code/decode format and the network transmission protocol are packaged, and are sent to a plurality of not exclusively intersecting transmission path simultaneously using redundant fashion
On, carry out the transmission of real-time audio media data multi-path redundancy；
The receiving terminal of audio media data carries out the grouping of the audio media data from different transmission path of reception superfluous
Remaining rejecting and reorganization operation, and the audio coding decoding form and the network transmission protocol consulted according to real-time audio session both sides reduce
Original audio data.
Optionally, a plurality of not exclusively intersecting transmission path includes：One default path based on default route and
One or more not exclusively intersecting redundant transmission paths.
Optionally, redundancy coefficient, a plurality of not exclusively intersecting transmission path are set according to the real-time audio session
Item number be more than the real-time audio session set redundancy coefficient.
Optionally, the redundant eliminating and reorganization operation, including：
It sets and initializes subflow buffering area, reassembly buffer area and jitter elimination window；
The receiving terminal of the audio media data carries out redundant eliminating behaviour to the audio media data grouping received
Make；
Audio media data grouping after the receiving terminal of the audio media data operates redundant eliminating carries out weight
Group operation.
Optionally, it is described to set and initialize subflow buffering area, reassembly buffer area and jitter elimination window, including：
It sets and is used for the subflow buffering area of the transmission path item number same number not exclusively intersected, the subflow buffering area
The audio media data grouping of different transmission path is received, each storage location for initializing each subflow buffering area is
It is empty；
Reassembly buffer area is set, and the reassembly buffer area is used to storing the sequence of the audio media data grouping after redundant eliminating
Number, the sequence number of the subflow buffering area of storage audio media data grouping and the audio media data be grouped in the son of storage
Storage location in streambuf initializes reassembly buffer area size as Nj, for storage in the reassembly buffer area
The sequence number of audio media data grouping after redundant eliminating operation is all initialized as -1, the storage audio media data
The storage location that the sequence number of the subflow buffering area of grouping and the audio media data are grouped in the subflow buffering area of storage is complete
Portion is initialized as sky；
Jitter elimination window is set, and the shake that the jitter elimination window is used to implement the audio media data grouping disappears
It removes, the size for initializing the jitter elimination window is W, W ∈ [Wmin,Wmax]。
Optionally, the redundant eliminating operation, including：
S1, each subflow buffering area is inquired about by the way of poll, obtains the recently received audio frequency media
Packet i extracts the sequence number of the audio media data grouping i, is denoted as FSNi；
S2, the sequence number FSN for being grouped the audio media dataiWith the reassembly buffer area size NjModular arithmetic is carried out, is obtained
To numerical value m, i.e. m=FSNi modNj, the sequence number that the audio media data that m positions store in the reassembly buffer area is grouped is inquired about,
It is denoted as J_FSNm；
If S3, J_FSNm=-1, then by FSNiValue be assigned to J_FSNm, by the son where audio media data grouping i
The sequence number and storage location of streambuf are stored into reassembly buffer area m positions, perform step S1；
If S4, J_FSNm≠ -1, FSNi＞ J_FSNm, then by FSNiValue be assigned to J_FSNm, by the audio media data
The sequence number and storage location of subflow buffering area where grouping i are stored into reassembly buffer area m positions, perform step S1；
If S5, J_FSNm≠ -1, FSNi≤J_FSNm, then step S1 is performed.
Optionally, it is described that each subflow buffering area is inquired about by the way of poll, including：
The each audio media data in the subflow buffering area point of the receiving terminal periodic statistical of the audio media data
The reception of group and redundant eliminating situation, obtain the polling priority sequence of the corresponding subflow buffering area of different transmission path；
The receiving terminal of the audio media data according to each subflow buffering area of the polling priority sequence queries,
Carry out redundant eliminating operation.
Optionally, the reorganization operation, including：
Dynamic regulation jitter elimination window size；
According to the jitter elimination window size, the audio media data grouping callback operation is carried out.
Optionally, the dynamic regulation jitter elimination window size, including：
The time R (i) that audio media data grouping i reaches the reassembly buffer area is recorded, according to the audio frequency media number
According to timestamp in grouping i or other flag bits for being used to record audio media data packet delivery time, the audio matchmaker is obtained
Volume data is grouped the sending time S (i) of i, calculates delay variation J (i)=R (i)-S (i) of the audio media data grouping i；
Audio media data according to the reassembly buffer area has been reached is grouped, and is newly reached the reassembly buffer area
Audio media data is grouped the expection delay variation of iWherein, N is fixed value, and i >=N, P (k) are to add
Weight coefficient, and
Calculate the class standard error of the audio media data grouping i delay variations
Threshold value g is set1、g2For judging that jitter elimination window size changes scope, and g1＜ g2；
WhenIncrease jitter elimination window size, if JARMSE(i)∈[g1,g2], jitter elimination window increases
AddIf JARMSE(i) ＞ g2, the increase of jitter elimination windowIts
In, k ∈ [0.5,1]；
WhenReduce jitter elimination window size, if JARMSE(i)∈[g1,g2], the contracting of jitter elimination window
SubtractIf JARMSE(i) ＞ g2, the reduction of jitter elimination windowIts
In, k ∈ [0.5,1]；
If the size W of the jitter elimination window after adjusting is less than jitter elimination window minimum Wmin, then by jitter elimination window
Mouth is dimensioned to jitter elimination window minimum WminIf the size W of the jitter elimination window after adjusting is more than jitter elimination
Window maximum Wmax, then jitter elimination window is dimensioned to jitter elimination window maximum Wmax；
Work as JARMSE(i) ＜ g1, the size of jitter elimination window is without changing；
If being modified to jitter elimination window size, the expection time delay of the audio media data grouping i changed is trembled
It is dynamicFor any integer in section [J (i) -4, J (i)+4], otherwise, without change.
Optionally, it is described according to the jitter elimination window size, the audio media data grouping callback operation is carried out,
Including：
The audio media data that audio media data grouping serial number is FSN is found in jitter elimination window to be grouped, and is pressed
The audio coding decoding form and the network transmission protocol that audio session both sides consult when according to the facts carry out readjustment decoding, in jitter elimination window
The audio media data that audio media data grouping serial number is FSN+1 is searched in mouthful to be grouped；
If finding, the audio media data grouping readjustment that audio media data grouping serial number is FSN+1 is decoded, after
It is continuous to search the audio media data grouping that audio media data grouping serial number is FSN+2, otherwise, whether judge jitter elimination window
There is residue, if without residue, the audio media data grouping that audio media data grouping serial number is FSN+1 is performed again and is once returned
Decoding operate is adjusted, the audio media data that audio media data grouping serial number is FSN+2 is continued to search for and is grouped, otherwise, wait pre-
If it is continued to search for after time t；
If continuous search the size that jitter elimination window is successfully reduced to a data frame three times, judgement is shaken at this time
Eliminate whether window is less than jitter elimination window minimum Wmin, if so, the shake that is dimensioned to of jitter elimination window is disappeared
Except window minimum Wmin, continue to execute search operation；
It searches three times unsuccessful if continuous, jitter elimination window is increased to the size of a data frame, judgement is trembled at this time
It is dynamic to eliminate whether window is more than jitter elimination window maximum Wmax, if so, jitter elimination window is dimensioned to shake
Eliminate window maximum Wmax, continue to execute search operation.
As shown from the above technical solution, network real-time audio conversation media data multi-path redundancy transmission method of the invention,
Including：The transmitting terminal of audio media data compiles the audio that the audio media data of capture is consulted according to real-time audio session both sides
Codec format and the network transmission protocol are packaged, and are sent to simultaneously using redundant fashion in a plurality of not exclusively intersecting transmission path,
The transmission of real-time audio media data multi-path redundancy is carried out, the receiving terminal of audio media data comes from different transmission path to reception
Audio media data grouping carry out redundant eliminating and reorganization operation, and the audio consulted according to real-time audio session both sides compiles solution
Code form and network transmission protocol reduction original audio data.As a result, by using multipath for real-time audio media data
Redundant transmission can effectively improve the Discarded Packets compensation probability in data transmission, improve and transmit relevant packet loss, time delay and shake
Index, and then the reliability of data transmission is promoted, improve business experience quality (QoE).
Description of the drawings：
Fig. 1 is the network real-time audio conversation media data multi-path redundancy transmission method stream that first embodiment of the invention provides
Journey schematic diagram；
Fig. 2 is the multipath relaying biography that the SIP Proxy/IMS CSCF that second embodiment of the invention provides participate in session negotiation
The multi-path redundancy Transmission system structure chart of defeated service system；
Fig. 3 is the network real-time audio conversation media data multi-path redundancy transmission method stream that second embodiment of the invention provides
Journey schematic diagram；
Fig. 4 is the multipath relaying biography that the SIP Proxy/IMS CSCF that second embodiment of the invention provides participate in session negotiation
The transmission schematic diagram of the audio media data grouping of defeated service system；
Fig. 5 is the multipath relaying biography that the SIP Proxy/IMS CSCF that second embodiment of the invention provides participate in session negotiation
The message flow chart of the multi-path redundancy transmission process of defeated service system；
Fig. 6 is the network real-time audio conversation media data multi-path redundancy transmission method that second embodiment of the invention provides
Audio media data grouping rx-side buffering area sets block diagram；
Fig. 7 is the network real-time audio conversation media data multi-path redundancy transmission method that second embodiment of the invention provides
The buffering area of audio media data grouping receiving terminal implements design drawing；
Fig. 8 is the multipath relaying biography that the SIP Proxy/IMS CSCF that second embodiment of the invention provides participate in session negotiation
The encapsulation format schematic diagram of the audio media data grouping of the multi-path redundancy transmission of defeated service system；
Fig. 9 is the network real-time audio conversation media data multi-path redundancy transmission method that second embodiment of the invention provides
The redundant eliminating flow chart of audio media data grouping.
Specific embodiment：
With reference to the accompanying drawings and examples, the specific embodiment of the present invention is described in further detail.Implement below
Example is not limited to the scope of the present invention for illustrating the present invention.
Fig. 1 shows the network real-time audio conversation media data multi-path redundancy transmission side that first embodiment of the invention provides
Method flow diagram, as shown in Figure 1, the method for the present embodiment is as described below.
101st, the transmitting terminal of audio media data consults the audio media data of capture according to real-time audio session both sides
Audio coding decoding form and the network transmission protocol be packaged, be sent to a plurality of not exclusively intersecting transmission simultaneously using redundant fashion
On path, the transmission of real-time audio media data multi-path redundancy is carried out.
In this step, during audio session foundation, the calling party of audio session and callee's acquisition are a plurality of not exclusively
Intersecting transmission path, it should be noted that a plurality of not exclusively intersecting transmission path includes：One based on default route
Default path and one or more not exclusively intersecting redundant transmission paths.
Specifically, during audio session progress, redundancy coefficient is set according to the real-time audio session, it is described a plurality of
Not exclusively the item number of intersecting transmission path is more than the redundancy coefficient that the real-time audio session is set.
102nd, the receiving terminal of audio media data the audio media data from different transmission path of reception is grouped into
Row redundant eliminating and reorganization operation, and according to the audio coding decoding form and the network transmission protocol of real-time audio session both sides negotiation
Reduce original audio data.
In this step, the redundant eliminating and reorganization operation, including：
It sets and initializes subflow buffering area, reassembly buffer area and jitter elimination window；
The receiving terminal of the audio media data carries out redundant eliminating to the audio media data grouping received；
Audio media data grouping after the receiving terminal of the audio media data operates redundant eliminating carries out weight
Group operation.
Further, it is described to set and initialize subflow buffering area, reassembly buffer area and jitter elimination window, including：
It sets and is used for the subflow buffering area of the transmission path item number same number not exclusively intersected, the subflow buffering area
The audio media data grouping of different transmission path is received, each storage location for initializing each subflow buffering area is
It is empty；
Reassembly buffer area is set, and the reassembly buffer area is used to storing the sequence of the audio media data grouping after redundant eliminating
Number FSN, the storage audio media data grouping subflow buffering area sequence number and the audio media data be grouped in storage
Subflow buffering area in storage location, initialize reassembly buffer area size as Nj, for depositing in the reassembly buffer area
The sequence number FSN of audio media data grouping after the redundant eliminating operation of storage is all initialized as -1, the storage audio
The subflow buffering area sequence number of media data packet and the audio media data are grouped in the storage in the subflow buffering area of storage
Position is all initialized as sky；
Jitter elimination window is set, and the shake that the jitter elimination window is used to implement the audio media data grouping disappears
It removes, the size for initializing the jitter elimination window is W, W ∈ [Wmin,Wmax]。
Further, the redundant eliminating operation of the audio media data grouping, including：
S1, each subflow buffering area is inquired about by the way of poll, obtains the recently received audio frequency media
Packet i extracts the sequence number of the audio media data grouping i, is denoted as FSNi；
S2, the sequence number FSN for being grouped the audio media dataiWith the reassembly buffer area size NjModular arithmetic is carried out, is obtained
To numerical value m, i.e. m=FSNi modNj, the sequence number of the audio media data grouping of the reassembly buffer area m positions storage is inquired about, is remembered
For J_FSNm；
If S3, J_FSNm=-1, then by FSNiValue be assigned to J_FSNm, by the son where audio media data grouping i
The sequence number and storage location of streambuf are stored into reassembly buffer area m positions, perform step S1；
If S4, J_FSNm≠ -1, FSNi＞ J_FSNm, then by FSNiValue be assigned to J_FSNm, by the audio media data
The sequence number and storage location of subflow buffering area where grouping i are stored into reassembly buffer area m positions, perform step S1；
If S5, J_FSNm≠ -1, FSNi≤J_FSNm, then step S1 is performed.
It should be noted that described inquire about each subflow buffering area by the way of poll, including：
The each audio media data in the subflow buffering area point of the receiving terminal periodic statistical of the audio media data
The reception of group obtains the polling priority sequence of the corresponding subflow buffering area of different transmission path with rejecting situation；
The receiving terminal of the audio media data according to each subflow buffering area of the polling priority sequence queries,
Carry out redundant eliminating operation.
Further, the reorganization operation, including：
Dynamic regulation jitter elimination window size；
According to the jitter elimination window size, the audio media data grouping callback operation is carried out.
Specifically, during multipath real-time audio session progress, the receiving terminal of voice data packet receives institute
The audio media data grouping of a plurality of not exclusively intersecting transmission path is stated, in being grouped according to the audio media data altogether
The mark that some sequence numbers or timestamp etc. can distinguish the audio media data grouping carries out the superfluous of audio media data grouping
It is remaining to reject operation；By searching for the audio media data grouping in dynamic buffering area or dynamic buffering window, obtain to be decoded
Audio media data is grouped, and the audio coding decoding form and the network transmission protocol consulted according to real-time audio session both sides are to described
Audio media data grouping carries out readjustment decoding, reduces original voice data.
Fig. 2 shows that the SIP Proxy/IMS CSCF that second embodiment of the invention provides are participated in the multipath of session negotiation
After the multi-path redundancy Transmission system structure chart of transmission service system, as shown in Fig. 2, calling party 210 and callee 220 are located at respectively
The Customer Premises Network at both ends, SIP Proxy/IMS CSCF 230, media relays controller 240, media relay servers 250 are equal
It is deployed in network.
The SIP Proxy/IMS CSCF 230 are the SIP Proxy/IMS for having multi-path transmission session negotiation ability
CSCF.By extending signaling negotiation of the SIP and SDP message realization for multipath session transmissions, in SIP Proxy/IMS
One is added in CSCF 230 for handling the module of multipath session request, can effectively realize SIP Proxy/IMS CSCF
230 multipath session requests of the processing from both ends Customer Premises Network.
The media relays controller 240 is used to manage network topology and the behavior of media relay servers 250, in being responsible for
After the distribution of transmission path, the media relay servers 250 participate in the structure of relay transmission path, are responsible for connecing for packet
It receives and forwards.The media relays controller 240 forms the service of multipath relay transmission with the media relay servers 250
System, relay transmission service is provided for multipath session, for example, two media relay servers is included in the present embodiment, i.e.,
Media relay servers 250-1 and media relay servers 250-2 shown in Fig. 2.
Specifically, the SIP Proxy/IMS CSCF 230 with multi-path transmission session negotiation ability are received from two
The multipath session establishment request of the Customer Premises Network at end asks distribution relay transmission path to media relays controller 240.Institute
Media relays controller 240 and the media relay servers 250 are stated to consult to complete the distribution and foundation of relay route, to
The SIP Proxy/IMS CSCF230 of multi-path transmission session negotiation ability return to the distribution information of relay transmission path.
The SIP Proxy/IMS CSCF of above-mentioned Fig. 2 participate in the multipath relay transmission service system of session negotiation, with reference to Fig. 3
The network real-time audio conversation media data multi-path redundancy transmission method flow signal that shown second embodiment of the invention provides
Figure, the method that the present embodiment is expanded on further.
301st, real-time audio session is established between calling party and callee, the transmitting terminal of audio media data is by the sound of capture
The audio coding decoding form and the network transmission protocol that frequency media data is consulted according to real-time audio session both sides are packaged, using redundancy
Mode is sent to simultaneously in three not exclusively intersecting transmission paths, carries out the transmission of real-time audio media data multi-path redundancy.
In this step, as shown in Fig. 2, redundancy coefficient is set according to audio session, the audio media data in the present embodiment
The redundancy coefficient of transmission is 2, three transmission paths 260 is obtained between calling party 210 and callee 220, including one
Default path 260-D based on default route, via the media relay servers 250-1 relay transmission path 260-R1 and
Via the relay transmission path 260-R2 of the media relay servers 250-2.
Fig. 4 shows that the SIP Proxy/IMS CSCF that second embodiment of the invention provides are participated in the multipath of session negotiation
After the transmission schematic diagram of the audio media data grouping of transmission service system, as shown in figure 4, being transmitted per transmission paths identical
Audio media data is grouped.
Fig. 5 shows that the SIP Proxy/IMS CSCF that second embodiment of the invention provides are participated in the multipath of session negotiation
After the message flow chart of the multi-path redundancy transmission process of transmission service system, the signaling procedure of specific multipath redundant transmission is such as
It is lower described.
501st, calling party 210 asks to the SIP Proxy/IMS CSCF 230 with multi-path transmission session negotiation
Multipath real-time audio session is established between calling party 210 and callee 220, the redundancy coefficient of audio session is 2；
502nd, with Multipath Transmission session negotiation ability SIP Proxy/IMS CSCF 230 check calling party 210 and by
Whether the side of crying 220 specifically has multipath session ability, if the no multipath session ability of two sides or a side do not have multipath
Session ability, the SIP Proxy/IMS CSCF 230 with Multipath Transmission session negotiation ability just refuse multipath audio session
Request.If calling party 210 and callee 220 have multipath session ability, there is the SIP of Multipath Transmission session negotiation ability
Just to the request of media relays controller 240, multipath is real between calling party 210 and callee 220 by Proxy/IMS CSCF 230
When audio session distribute a plurality of relay transmission path；
503rd, since the redundancy coefficient of session is 2, media relays controller 240 is between calling party 210 and callee 220
Multipath real-time audio session is assigned with two relay transmission paths, one via media relay servers 250-1, one via
Media relay servers 250-2.Media relays controller 240 is respectively to media relay servers 250-1 and media relays service
Device 250-2 sends relay route addition request 503-1 and 503-2；
504th, media relay servers 250-1 and media relay servers 250-2 complete the addition of relay route, respectively to
Media relays controller 240 returns to relay route addition success response 504-1 and 504-2；
505th, media relays controller 240 is to the SIP Proxy/IMS CSCF with multi-path transmission session negotiation ability
230 return relay routes are allocated successfully response, and the information of two relay routes is included in response message；
506th, the SIP Proxy/IMS CSCF 230 with multi-path transmission session negotiation ability notify callee 220 to pass
The distribution condition in defeated path；
507th, callee 220 distributes for multipath audio session and initializes buffering area, and to multi-path transmission meeting
The SIP Proxy/IMS CSCF 230 for talking about negotiation ability return to 506 notice success response；
508th, the SIP Proxy/IMS CSCF230 with Multipath Transmission session negotiation ability are returned more to calling party 210
Footpath session establishment success response, and notify 210 liang of transmission paths distribution conditions of calling party；
509th, there are three transmission paths between calling party 210 and callee 220：One default path based on default route
260-D, via the relay transmission path 260-R1 of media relay servers 250-1 and via media relay servers 250-2's
Relay transmission path 260-R2.Using redundant transmission, the transmission mode of media data packet is as shown in Figure 4.
302nd, the receiving terminal of audio media data set and initialize subflow buffering area, reassembly buffer area and jitter elimination window
Mouthful, receive the audio media data grouping transmitted in three not exclusively intersecting transmission paths.
It is described to set and initialize subflow buffering area, reassembly buffer area and jitter elimination window in this step, including：
It sets and is used for the subflow buffering area of the transmission path item number same number not exclusively intersected, the subflow buffering area
The audio media data grouping of different transmission path is received, each storage location for initializing each subflow buffering area is
It is empty；
Reassembly buffer area is set, and the reassembly buffer area is used to storing the sequence of the audio media data grouping after redundant eliminating
Number, the sequence number of the subflow buffering area of storage audio media data grouping and the audio media data be grouped in the son of storage
Storage location in streambuf initializes reassembly buffer area size as Nj, for storage in the reassembly buffer area
The sequence number of audio media data grouping after redundant eliminating operation is all initialized as -1, the storage audio media data
The storage location that the sequence number of the subflow buffering area of grouping and the audio media data are grouped in the subflow buffering area of storage is complete
Portion is initialized as sky；
Jitter elimination window is set, and the shake that the jitter elimination window is used to implement the audio media data grouping disappears
It removes, the size for initializing the jitter elimination window is W=100ms, W ∈ [Wmin,Wmax], Wmin=40ms, Wmax=160ms.
Specifically, Fig. 6 shows that the network real-time audio conversation media data multipath that second embodiment of the invention provides is superfluous
The audio media data grouping rx-side buffering area of remaining transmission method sets block diagram, as shown in fig. 6, Fig. 6 includes subflow buffering
Area 610, reassembly buffer area 620 and jitter elimination window 630, the network provided with reference to second embodiment of the invention shown in Fig. 7
The buffering area of the audio media data receiving terminal of real-time audio conversation media data multi-path redundancy transmission method implements design drawing, figure
Reassembly buffer area is fixed in 7, and audio media data grouping is corresponded to by a plurality of not exclusively intersecting paths into each transmission path
Subflow buffering area 710, wherein, subflow buffering area 710-D corresponds to transmission path 260-D, and subflow buffering area 710-R1, which is corresponded to, to be passed
Defeated path 260-R1, subflow buffering area 710-R2 correspond to transmission path 260-R2, reassembly buffer are operated by redundant eliminating
Area 720, and then by 730 size of jitter elimination window in dynamic regulation reassembly buffer area 720, realize audio media data point
Group jitter elimination, finally to audio media data be grouped according to real-time audio session both sides consult audio coding decoding form and
The network transmission protocol carries out readjustment decoding, reduces original voice data.
Fig. 8 shows that the SIP Proxy/IMS CSCF that second embodiment of the invention provides are participated in the multipath of session negotiation
After the encapsulation format schematic diagram of the audio media data grouping of the multi-path redundancy transmission of transmission service system, form encapsulation uses
It is Multipath Transmission agreement (MPTP), the meaning of each flag bit is as follows in Fig. 8：
801：Version number, 2bit are currently version 1；
802：Type, 1bit, for illustrating the type of audio media data grouping (media data packet or control data
Grouping)；
803：Filler, 1bit indicate whether the filling data of non-effective load；
804：Towards the MPTP types of special applications, 4bit indicates the specific application of such packet；
805：Type of service, 4bit indicate the transmission demand of different classs of business；
806：Reserved field, 4bit are set to 0；
807：Subflow sequence number, 16bit are superfluous for identifying the transmission sequence number of audio media data grouping in the path
In remaining transmission, set subflow sequence number identical with audio media data packet sequence number；
808：For identifying a transmission paths, receiving terminal is grouped for audio media data by path identifier, 32bit
By this storage audio media data grouping；
809：Sequence number, 32bit, for identifying unique audio media data grouping in real-time audio session, for sound
Frequency media data packet receiving terminal recombinates original audio data by this sequence number；
810：Load is real-time audio media data packet to be transmitted.
In concrete application, subflow buffering area 610 is responsible for receiving the audio media data from a plurality of incomplete intersecting paths
Grouping carries out redundant eliminating by being grouped to audio media data, and the storage location storage that audio media data is grouped is arrived
Reassembly buffer area 620, while picked according to what the statistics and redundancy audio media data of the grouping of the audio media data of reception were grouped
Except situation, the polling priority of the corresponding subflow buffering area of different transmission path of periodic adjustment subflow buffering area 610.By dynamic
State adjusts the size of jitter elimination window 630, can realize the jitter elimination of audio media data grouping, while audio frequency media number
The size of jitter elimination window will be also adjusted during the fructufy according to readjustment according to the readjustment decoding process of grouping.
303rd, the receiving terminal of audio media data carries out redundant eliminating to the audio media data grouping received.
In this step, Fig. 9 shows the network real-time audio conversation media data multipath that second embodiment of the invention provides
The redundant eliminating flow chart of the audio media data grouping of redundant transmission method, specific steps include：
901st, reassembly buffer area, size N are initializedj；
902nd, audio media data grouping i is obtained from each path subflow buffering area by the way of poll；
903rd, judge whether the audio media data grouping i payloads are empty, if so, step 911 is performed, it is no
Then, step 904 is performed；
904th, the sequence number of audio media data grouping i is extracted, sequence number 809 as shown in Figure 8 is denoted as FSNi；
905th, m=FSN is calculatedi modNj；
906th, the sequence for the audio media data grouping stored in query reformulation buffering area on m positions, equally such as institute in Fig. 8
The sequence number 809 shown, is denoted as J_FSNm；
907th, J_FSN is judgedmWhether -1 is equal to, if so, illustrating that audio media data is grouped into new arrival reassembly buffer
Area, step 909 is performed, otherwise, performs step 908；
908th, J_FSN is judgedmWhether FSN is less thani, if so, illustrating that audio media data grouping i is newly arrived audio
Media data packet performs step 909, otherwise, performs step 911；
909th, by FSNiValue be assigned to J_FSNm；
910th, storage locations of the audio media data grouping i in subflow buffering area is stored in reassembly buffer area m
In putting；
911st, judge whether to terminate to receive, if so, terminating the reception of audio media data packet, otherwise, perform step
902。
304th, the audio media data grouping after the receiving terminal of audio media data operates redundant eliminating carries out weight
Group operation, and the audio coding decoding form and the network transmission protocol consulted according to real-time audio session both sides reduce original audio number
According to.
In this step, the reorganization operation, including：
Dynamic regulation jitter elimination window size；
According to the jitter elimination window size, the audio media data grouping callback operation is carried out.
Further, the adjusting jitter elimination window size, including：
The time R (i) that audio media data grouping i reaches the reassembly buffer area is recorded, according to the audio frequency media number
According to timestamp in grouping i or other flag bits for being used to record audio media data packet delivery time, the audio matchmaker is obtained
Volume data is grouped the sending time S (i) of i, calculates delay variation J (i)=R (i)-S (i) of the audio media data grouping i；
Audio media data according to the reassembly buffer area has been reached is grouped, and is newly reached the reassembly buffer area
Audio media data is grouped the expection delay variation of iWherein, N is fixed value, takes N=4, i >=N, P
(k) it is weighting coefficient, general value P (i-1)=0.5, P (i-2)=0.3, P (i-3)=0.125, P (i-4)=0.075；
Calculate the class standard error of the audio media data grouping i delay variations
Threshold value g is set1=5ms, g2=10ms is used to judge that jitter elimination window size changes scope；
WhenIncrease jitter elimination window size, if JARMSE(i) ∈ [5,10], jitter elimination window
IncreaseIf JARMSE(i) ＞ 10, jitter elimination window increaseIts
In, k=1；
WhenReduce jitter elimination window size, if JARMSE(i) ∈ [5,10], jitter elimination window
ReductionIf JARMSE(i) ＞ 10, the reduction of jitter elimination windowWherein,
K=0.6；
If the size W of the jitter elimination window after adjusting is less than jitter elimination window minimum Wmin, then by jitter elimination window
Mouth is sized to jitter elimination window minimum WminIf the size W of the jitter elimination window after adjusting is more than jitter elimination window
Mouth maximum Wmax, then jitter elimination window is sized to jitter elimination window maximum Wmax；
Work as JARMSE(i) ＜ 5, the size of jitter elimination window is without changing；
It should be noted that present video media data packet i is to the pre- of next audio media data grouping i+1 in order to prevent
The influence of accuracy is surveyed, if being modified to jitter elimination window size, the time delay for changing audio media data grouping i is trembled
Dynamic J (i) is any integer in section [J (i) -4, J (i)+4], otherwise, without change.
Further, it is described according to the jitter elimination window size, carry out the audio media data grouping readjustment behaviour
Make, including：
The audio media data that audio media data grouping serial number is FSN is found in jitter elimination window to be grouped, and is pressed
The audio coding solution form and the network transmission protocol that audio session both sides consult when according to the facts carry out readjustment decoding, in jitter elimination window
The audio media data that audio media data grouping serial number is FSN+1 is searched in mouthful to be grouped；
If finding, the audio media data grouping readjustment that audio media data grouping serial number is FSN+1 is decoded, after
It is continuous to search the audio media data grouping that audio media data grouping serial number is FSN+2；
Otherwise, judge whether jitter elimination window has residue, if without residue, be by audio media data grouping serial number
The audio data packet of FSN+1 performs again once adjusts back decoding operate, continues to search for audio media data grouping serial number as FSN+2
Audio media data grouping, otherwise, wait preset time t after continue to search for, t can use 10ms；
If continuous search the size that jitter elimination window is successfully reduced to a data frame three times, judgement is shaken at this time
Eliminate whether window is less than jitter elimination window minimum Wmin, if so, the shake that is dimensioned to of jitter elimination window is disappeared
Except window minimum Wmin, continue to execute search operation；
It searches three times unsuccessful if continuous, jitter elimination window is increased to the size of a data frame, judgement is trembled at this time
It is dynamic to eliminate whether window is more than jitter elimination window maximum Wmax, if so, jitter elimination window is dimensioned to shake
Eliminate window maximum Wmax, continue to execute search operation.
In conclusion the network real-time audio conversation media data multi-path redundancy transmission method of the present embodiment passes through conversation class
Voice data can effectively improve the Discarded Packets compensation probability in data transmission using multipath redundant transmission, ensure data transmission
Reliability.And the receiving terminal by being grouped in audio media data carries out redundant eliminating and the shake based on dynamic buffering window disappears
The reorganization operation removed significantly reduces the influence that delay variation receives audio media data grouping, and then is effectively promoted
Customer service Quality of experience.
Finally it should be noted that：The above embodiments are only used to illustrate the technical solution of the present invention., rather than its limitations；To the greatest extent
Pipe is described in detail the present invention with reference to foregoing embodiments, it will be understood by those of ordinary skill in the art that：Its according to
Can so modify to the technical solution recorded in foregoing embodiments either to which part or all technical characteristic into
Row equivalent substitution；And these modifications or replacement, the essence of appropriate technical solution is not made to depart from the claims in the present invention and is limited
Fixed scope.
Claims (8)
1. a kind of network real-time audio conversation media data multi-path redundancy transmission method, including：
The transmitting terminal of audio media data compiles the audio that the audio media data of capture is consulted according to real-time audio session both sides
Codec format and the network transmission protocol are packaged, and are sent to simultaneously using redundant fashion in a plurality of not exclusively intersecting transmission path,
Carry out realaudio data redundant transmission；
The receiving terminal of audio media data carries out redundancy to the grouping of the audio media data from different transmission path of reception and picks
It removes and reorganization operation, and the audio coding decoding form consulted according to real-time audio session both sides and network transmission protocol reduction are original
Voice data；
The redundant eliminating and reorganization operation, including：
It sets and initializes subflow buffering area, reassembly buffer area and jitter elimination window；
The receiving terminal of the audio media data carries out redundant eliminating operation to the audio media data grouping received；
Audio media data grouping after the receiving terminal of the audio media data operates redundant eliminating carries out restructuring behaviour
Make；
It is characterized in that, described set and initialize subflow buffering area, reassembly buffer area and jitter elimination window, including：
It sets and is used to receive with the subflow buffering area of the transmission path item number same number not exclusively intersected, the subflow buffering area
The audio media data grouping of different transmission path initializes each storage location of each subflow buffering area as sky；
Reassembly buffer area is set, the reassembly buffer area be used to storing the audio media data grouping after redundant eliminating sequence number,
It stores the sequence number of the subflow buffering area of the audio media data grouping and the audio media data is grouped in the subflow of storage
Storage location in buffering area initializes reassembly buffer area size as Nj, for the superfluous of storage in the reassembly buffer area
The sequence number of the remaining audio media data grouping rejected after operating all is initialized as -1, the storage audio media data point
The storage location that the sequence number of the subflow buffering area of group and the audio media data are grouped in the subflow buffering area of storage is whole
It is initialized as sky；
Jitter elimination window is set, the jitter elimination window is used to implement the jitter elimination of the audio media data grouping,
The size for initializing the jitter elimination window is W, W ∈ [Wmin,Wmax], WminFor jitter elimination window minimum, WmaxTo tremble
It is dynamic to eliminate window maximum.
2. network real-time audio conversation media data multi-path redundancy transmission method according to claim 1, which is characterized in that
The a plurality of not exclusively intersecting transmission path includes：One default path and one or more based on default route is not exclusively
Intersecting redundant transmission paths.
3. network real-time audio conversation media data multi-path redundancy transmission method according to claim 1, which is characterized in that
Redundancy coefficient is set according to the real-time audio session, the item number of a plurality of not exclusively intersecting transmission path is more than the reality
When audio session set redundancy coefficient.
4. network real-time audio conversation media data multi-path redundancy transmission method according to claim 1, which is characterized in that
The redundant eliminating operation, including：
S1, each subflow buffering area is inquired about by the way of poll, obtains the recently received audio media data
I is grouped, the sequence number of the audio media data grouping i is extracted, is denoted as FSNi；
S2, the sequence number FSN by audio media data grouping iiWith the reassembly buffer area size NjModular arithmetic is carried out, is obtained
Numerical value m, i.e. m=FSNimodNj, the sequence number for the audio media data grouping that m positions store in the reassembly buffer area is inquired about, is remembered
For J_FSNm；
If S3, J_FSNm=-1, then by FSNiValue be assigned to J_FSNm, the subflow where audio media data grouping i is delayed
The sequence number and storage location for rushing area are stored into reassembly buffer area m positions, perform step S1；
If S4, J_FSNm≠ -1, FSNi>J_FSNm, then by FSNiValue be assigned to J_FSNm, the audio media data is grouped i
The sequence number and storage location of the subflow buffering area at place are stored into reassembly buffer area m positions, perform step S1；
If S5, J_FSNm≠ -1, FSNi≤J_FSNm, then step S1 is performed.
5. network real-time audio conversation media data multi-path redundancy transmission method according to claim 4, which is characterized in that
It is described that each subflow buffering area is inquired about by the way of poll, including：
The each audio media data grouping in the subflow buffering area of the receiving terminal periodic statistical of the audio media data
Reception and redundant eliminating situation, obtain the polling priority sequence of the corresponding subflow buffering area of different transmission path；
The receiving terminal of the audio media data is carried out according to each subflow buffering area of the polling priority sequence queries
Redundant eliminating operates.
6. network real-time audio conversation media data multi-path redundancy transmission method according to claim 1, which is characterized in that
The reorganization operation, including：
Dynamic regulation jitter elimination window size；
According to the jitter elimination window size, the audio media data grouping callback operation is carried out.
7. network real-time audio conversation media data multi-path redundancy transmission method according to claim 6, which is characterized in that
The dynamic regulation jitter elimination window size, including：
The time R (i) that audio media data grouping i reaches the reassembly buffer area is recorded, according to the audio media data point
Timestamp or other flag bits for being used to record audio media data packet delivery time, obtain the audio frequency media number in group i
According to the sending time S (i) of grouping i, delay variation J (i)=R (i)-S (i) that the audio media data is grouped i is calculated；
Audio media data according to the reassembly buffer area has been reached is grouped, and is newly reached the audio in the reassembly buffer area
The expection delay variation of media data packet iWherein, N is fixed value, and i >=N, P (k) are for weighting
Number, and
Calculate the class standard error of the audio media data grouping i delay variations
Threshold values g is set1、g2For judging that jitter elimination window size changes scope, and g1<g2；
WhenIncrease jitter elimination window size, if JARMSE(i)∈[g1,g2], jitter elimination window increasesIf JARMSE(i)>g2, the increase of jitter elimination windowWherein, k
∈[0.5,1]；
WhenReduce jitter elimination window size, if JARMSE(i)∈[g1,g2], the reduction of jitter elimination windowIf JARMSE(i)>g2, the reduction of jitter elimination windowWherein, k
∈[0.5,1]；
If the size W of the jitter elimination window after adjusting is less than jitter elimination window minimum Wmin, then by jitter elimination window
It is dimensioned to jitter elimination window minimum WminIf the size W of the jitter elimination window after adjusting is more than jitter elimination window
Maximum Wmax, then jitter elimination window is dimensioned to jitter elimination window maximum Wmax；
Work as JARMSE(i)<g1, the size of jitter elimination window is without changing；
If being modified to jitter elimination window size, the expection delay variation of the audio media data grouping i changed
For any integer in section [J (i) -4, J (i)+4], otherwise, without change.
8. network real-time audio conversation media data multi-path redundancy transmission method according to claim 6, which is characterized in that
It is described that the audio media data grouping callback operation is carried out according to the jitter elimination window size, including：
The audio data packet that audio media data grouping serial number is FSN is found in jitter elimination window, according to real-time sound
The audio coding decoding form and the network transmission protocol that frequency session both sides consult carry out readjustment decoding, are searched in jitter elimination window
The audio media data that audio media data grouping serial number is FSN+1 is grouped；
If finding, the audio media data grouping readjustment decoding that audio media data grouping serial number is FSN+1 continues to look into
The audio media data that audio media data grouping serial number is FSN+2 is looked for be grouped, otherwise, it is surplus to judge whether jitter elimination window has
It is remaining, if without residue, the audio media data grouping that audio media data grouping serial number is FSN+1 is performed into once readjustment again and is solved
Code operation continues to search for the audio media data that audio media data grouping serial number is FSN+2 and is grouped, otherwise, when waiting default
Between continue to search for after t；
If continuous search the size that jitter elimination window is successfully reduced to a data frame three times, jitter elimination at this time is judged
Whether window is less than jitter elimination window minimum Wmin, if so, jitter elimination window is dimensioned to jitter elimination window
Mouth minimum value Wmin, continue to execute search operation；
It searches three times unsuccessful if continuous, jitter elimination window is increased to the size of a data frame, judges that shake disappears at this time
Except whether window is more than jitter elimination window maximum Wmax, if so, jitter elimination window is dimensioned to jitter elimination
Window maximum Wmax, continue to execute search operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510388523.2A CN105071897B (en) | 2015-07-03 | 2015-07-03 | A kind of network real-time audio conversation media data multi-path redundancy transmission method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510388523.2A CN105071897B (en) | 2015-07-03 | 2015-07-03 | A kind of network real-time audio conversation media data multi-path redundancy transmission method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105071897A CN105071897A (en) | 2015-11-18 |
| CN105071897B true CN105071897B (en) | 2018-05-29 |
Family
ID=54501195
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510388523.2A Active CN105071897B (en) | 2015-07-03 | 2015-07-03 | A kind of network real-time audio conversation media data multi-path redundancy transmission method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105071897B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105656774B (en) * | 2016-01-12 | 2018-10-23 | 东北大学 | A kind of network real-time video conversation media data multipath hybrid redundancy transmission method |
| US20190379597A1 (en) * | 2018-06-06 | 2019-12-12 | Nokia Solutions And Networks Oy | Selective duplication of data in hybrid access networks |
| CN108965624A (en) * | 2018-07-09 | 2018-12-07 | 南京莱斯电子设备有限公司 | A kind of redundancy structure and method for VoIP voice communication system |
| CN108540213B (en) * | 2018-07-20 | 2019-02-26 | 北京邮电大学 | A kind of space method for routing and device |
| CN109768845B (en) * | 2018-12-28 | 2021-03-09 | 北京诺亦腾科技有限公司 | Data processing method, device and storage medium |
| CN112740708B (en) * | 2020-05-21 | 2022-07-22 | 华为技术有限公司 | Audio data transmission method and related device |
| CN115883527A (en) * | 2021-09-28 | 2023-03-31 | 中兴通讯股份有限公司 | Audio control method, device, equipment and computer readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1578289A (en) * | 2003-06-27 | 2005-02-09 | 三星电子株式会社 | Receiving device with time-domain equalizer and receiving method thereof |
| CN101076989A (en) * | 2004-10-13 | 2007-11-21 | 高通股份有限公司 | De-jitter buffer adjustments based on estimated delay |
| CN102546418A (en) * | 2012-01-16 | 2012-07-04 | 东北大学 | Overlay-network-multipath-transmission-based Internet protocol multimedia subsystem (IMS) client and media exchange method |
| CN104023006A (en) * | 2014-05-09 | 2014-09-03 | 东北大学 | Multi-path transmission system and method based on application layer relaying |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9531503B2 (en) * | 2012-04-18 | 2016-12-27 | Acme Packet, Inc. | Redundancy for real time communications |
-
2015
- 2015-07-03 CN CN201510388523.2A patent/CN105071897B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1578289A (en) * | 2003-06-27 | 2005-02-09 | 三星电子株式会社 | Receiving device with time-domain equalizer and receiving method thereof |
| CN101076989A (en) * | 2004-10-13 | 2007-11-21 | 高通股份有限公司 | De-jitter buffer adjustments based on estimated delay |
| CN102546418A (en) * | 2012-01-16 | 2012-07-04 | 东北大学 | Overlay-network-multipath-transmission-based Internet protocol multimedia subsystem (IMS) client and media exchange method |
| CN104023006A (en) * | 2014-05-09 | 2014-09-03 | 东北大学 | Multi-path transmission system and method based on application layer relaying |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105071897A (en) | 2015-11-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105071897B (en) | A kind of network real-time audio conversation media data multi-path redundancy transmission method | |
| AU2003248437B2 (en) | Packet Transmission System and Packet Reception System | |
| KR100551859B1 (en) | Priority handling of voice over data in a voice-over-internet protocol processor | |
| FI108692B (en) | Method and apparatus for scheduling processing of data packets | |
| US7397819B2 (en) | Packet compression system, packet restoration system, packet compression method, and packet restoration method | |
| US6584110B1 (en) | Voice gateway and route selection | |
| CN1316797C (en) | Method and apparatus for creating a packet using a digital signal processor | |
| US9350784B2 (en) | Method and communication system for selecting a transmission mode for transmitting payload data | |
| US20080101338A1 (en) | METHODS AND APPARATUS TO IMPLEMENT HIGHER DATA RATE VOICE OVER INTERNET PROTOCOL (VoIP) SERVICES | |
| US7379466B2 (en) | In band signal detection and presentation for IP phone | |
| CN101039254B (en) | Media data regrouping method and packet server | |
| CN1860801B (en) | Intelligent multimedia calls | |
| CN105656774B (en) | A kind of network real-time video conversation media data multipath hybrid redundancy transmission method | |
| US20170318062A1 (en) | Method for setting up a communication link | |
| JP3994946B2 (en) | Quality report server and system | |
| US7191370B2 (en) | Data transmitter device, repeater device, data transmission/reception device, and data communication method | |
| CN101026545B (en) | Real-time multimedia transmitting system and method | |
| Toral-Cruz et al. | An introduction to VoIP: End-to-end elements and QoS parameters | |
| CN101515974B (en) | Method, equipment and VG for detecting VoIP channel and switching PSTN backup | |
| JP4995637B2 (en) | Communication apparatus and wireless communication method | |
| US20070274316A1 (en) | Method For Switching A Communication Connection From A First Connection Path To A Second Connection Path | |
| EP1145509A2 (en) | Method and apparatus for providing efficient multiplexing between gateways using dynamic timers | |
| KR101384125B1 (en) | Apparatus and method for generating quality of service parameter about mac layer in communication system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |