Disclosure of Invention
In order to make up for the defects of the prior art and solve the problems that when the fingerprints of an audio clip or the whole audio file are required to be matched, the expression is not stable enough and the risk of large data amount of calculation and storage is high, the invention provides an audio long-time fingerprint extraction and matching method.
The technical scheme adopted by the invention for solving the technical problems is as follows: an audio long-time fingerprint extraction method comprises the following steps:
s1: inputting an audio signal (PCM) and resampling the audio signal;
s2: performing framing, windowing and DFT (discrete Fourier transform) change on the resampled audio signal to obtain a frame frequency spectrum;
s3: performing interframe smoothing processing on the frame frequency spectrum to obtain an updated frame frequency spectrum;
s4: carrying out frame-level short-time feature extraction on the updated frame frequency spectrum;
s5: and processing the frame-level short-time features and extracting the frame group long-time features.
Preferably, in S1, the resampling specifically includes extracting a frequency range of 110Hz to 7KHz as an analysis frequency band, and setting the resampling frequency of the input signal to 16KHz according to the nyquist sampling theorem, so as to avoid signal sampling distortion.
Preferably, in S2, the specific operations of framing, windowing and DFT changing are to frame the resampled signal by 4096 samples (256ms) and 50% overlap; after framing, a Hamming window is added frame by frame and DFT frequency domain transformation is carried out to obtain a frame frequency spectrum, wherein the frame frequency spectrum is the relation between frequency and energy.
Preferably, in S3, the specific operation of inter-frame smoothing is to add adjacent 5 frames of spectral data by using a sliding windowWeight averaging, which aims to increase the spectral stationarity and obtain an updated frame spectrum: m is 0.25M1+0.75M2+M3+0.75M4+0.25M5(ii) a Where the sliding window is stepped one frame at a time.
Preferably, in S4, the specific operation steps of frame-level short-time feature extraction are as follows:
a1: dividing a frame frequency spectrum by a logarithmic frequency domain sub-band;
a2: calculating the average spectral energy of the sub-bands;
a3: and carrying out regularization processing on the subband spectral energy L2 to obtain frame-level short-time features.
Preferably, in a1, since the human ear feels logarithmic to sound, dividing the frame spectrum into logarithmic frequency domain sub-bands, that is, converting the frequency F in the frame spectrum into logarithmic frequency F-log2(f) In the logarithmic frequency domain, the target frequency range log2(110)~log2(7000) The division into 16 sub-bands of equal width.
Preferably, in the a2, the subband average spectral energy is calculated, that is, the average spectral energy is calculated over 16 frequency subbands for each audio frame, thereby forming a 16-dimensional vector.
Preferably, in a3, the sub-band spectral energy L2 regularization processing obtains frame-level short-time features, that is, L2 regularization is performed on the obtained 16-dimensional vector, that is, the obtained short-time features of the audio frame are denoted as V.
Preferably, in S5, the specific operation of frame group long-term feature extraction is to combine a continuous fixed number of audios into a frame group, perform DFT change on the frame-level short-term features again in the time axis direction, and retain the low-frequency stable components to form the frame group long-term features.
An audio long-time fingerprint matching method comprises the following steps:
b1: extracting long-term features from 2 audio files or segments to be matched according to frame groups;
b2: and carrying out frame group level matching on the long-term characteristics of the 2 frame groups, and determining a matching relation.
The invention has the technical effects and advantages that:
1. the invention provides an audio long-time fingerprint extraction and matching method, which is characterized in that after audio signals are subjected to resampling, framing, windowing and DFT change, interframe smoothing operation is carried out on obtained frame spectrums, then frame-level short-time characteristic extraction is carried out, frequency spectrum sub-band characteristics of audio frames are extracted, and then the change attribute of the time axis direction is calculated to form long-time characteristics, so that the rapid extraction of the audio fingerprints is realized, the matching of two or more subsequent sets of audio fingerprints is facilitated, and the defects of short time and instability of the traditional audio fingerprints are overcome.
2. The invention provides an audio long-time fingerprint extraction and matching method, which is characterized in that different audio signals are subjected to fingerprint extraction, after frame group long-time characteristics of audio are rapidly extracted, the similarity between two or more groups of audio signals is calculated by utilizing a similarity meter, then whether the two or more groups of audio signals are matched or not is obtained, and the optimal similarity and offset are obtained simultaneously during matching.
Detailed Description
In order to make the technical means, the creation characteristics, the achievement purposes and the effects of the invention easy to understand, the invention is further described with the specific embodiments.
As shown in fig. 1 to 5, the method for extracting long-term fingerprints in audio according to the present invention includes the following steps:
s1: inputting an audio signal (PCM) and resampling the audio signal;
s2: performing framing, windowing and DFT (discrete Fourier transform) change on the resampled audio signal to obtain a frame frequency spectrum;
s3: performing interframe smoothing processing on the frame frequency spectrum to obtain an updated frame frequency spectrum;
s4: carrying out frame-level short-time feature extraction on the updated frame frequency spectrum;
s5: and processing the frame-level short-time features and extracting the frame group long-time features.
In an embodiment of the present invention, in S1, the resampling specifically extracts a frequency range from 110Hz to 7KHz as an analysis frequency band, and sets the resampling frequency of the input signal to 16KHz according to the nyquist sampling theorem, so as to avoid signal sampling distortion.
As an embodiment of the present invention, in S2, the specific operations of framing, windowing, and DFT changing are to frame the resampled signal by 4096 samples (256ms) and 50% overlap; after framing, a Hamming window is added frame by frame and DFT frequency domain transformation is carried out to obtain a frame frequency spectrum, wherein the frame frequency spectrum is the relation between frequency and energy.
In an embodiment of the present invention, in S3, the specific operation of inter-frame smoothing is to perform weighted average on the adjacent 5 frames of spectrum data by using a sliding window, which aims to increase the spectral stationarity and obtain an updated frame spectrum: m is 0.25M1+0.75M2+M3+0.75M4+0.25M5(ii) a Where the sliding window is stepped one frame at a time.
As an embodiment of the present invention, in S4, the specific operation steps of the frame-level short-time feature extraction are as follows:
a1: dividing a frame frequency spectrum by a logarithmic frequency domain sub-band;
a2: calculating the average spectral energy of the sub-bands;
a3: and carrying out regularization processing on the subband spectral energy L2 to obtain frame-level short-time features.
In the embodiment of the invention, in the a1, since the feeling of the human ear to the sound is logarithmic,therefore, dividing the logarithmic frequency domain sub-band into frame spectrums, i.e. converting the frequency F in the frame spectrums into the logarithmic frequency F-log2(f) In the logarithmic frequency domain, the target frequency range log2(110)~log2(7000) The division into 16 sub-bands of equal width.
In a2, a 16-dimensional vector is formed by calculating the subband average spectral energy, i.e., calculating the average spectral energy in 16 frequency subbands for each audio frame.
In an embodiment of the present invention, in a3, the subband spectral energy L2 regularization processing obtains frame-level short-time features, i.e., performs L2 regularization on the obtained 16-dimensional vectors, i.e., the obtained short-time features of the audio frame are denoted as V.
As an embodiment of the present invention, in S5, the specific operation of frame group long-term feature extraction is to group a continuous fixed number of audios into a frame group, perform DFT change on the frame-level short-term features again in the time axis direction, and retain the low-frequency stable components to form the frame group long-term features.
Specifically, the specific process of extracting the long-term features of the frame group is as follows:
c1: if a frame group is formed from consecutive T audio frames (for example, T ═ 32, i.e., a frame group duration of 4096ms), the frame group is characterized as: [ V ]0,V1,…,VT-1]Where V is a frame-level short-time feature (16-dimensional vector).
C2: for each dimension [ V ] of frame group characteristics0d,V1d,…,V(T-1)d],d∈[0,15]And performing DFT conversion.
C3: taking the former m-level coefficient (for example, m is 12) theta after DFT conversion0,θ1c,θ1s,…,θmc,θmsWherein: theta0Is a coefficient of the first term, θ1c,…,θmcIs a coefficient of a cosine term, theta1s,…,θmsAre coefficients of sinusoidal terms.
C4: new feature [ A ] of frame group formed by 16-dimensional m-level DFT coefficients0,A1c,A1s,…,Amc,Ams]Then the characteristic size is [16 x (2m +1)]。
C5: the 2m +1 DFT coefficients (vectors) are regularized by L2.
C6: the 2m +1 DFT coefficients (vectors) are multiplied by the weight, which is calculated as follows:
wherein,
b is a Bessel function; sinh is a hyperbolic sine function; the calculation result of the above process is the long-term characteristic [ A ] of the frame group
0',A
1c',A
1s',…,A
mc',A
ms']。
An audio long-time fingerprint matching method comprises the following steps:
b1: extracting long-term features from two audio files or segments to be matched according to frame groups;
b2: and carrying out frame group level matching on the long-term characteristics of the two frame groups, and determining a matching relation.
Specifically, the specific flow of frame group level matching is as follows:
d1: the long-term characteristics of two frame groups are marked as [ A ] respectively0',A1c',A1s',…,Amc',Ams']And [ B0',B1c',B1s',…,Bmc',Bms']The possible time offset between two frame groups (i.e. the number of audio frames) is denoted t.
D2: the similarity s of the two frame groups at the time offset t is calculated according to the following formula:
wherein:
t is the number of audio frames in the frame group;<A'|B'>represents the inner product of vector a 'and vector B'.
D3: according to step D2, for all possible offsets T, T e [ - (T-1), (T-1)]Calculating corresponding similarity s, and counting the maximum s of all sbestObtained sbestI.e. the best similarity in the two frame groups, corresponding to tbestTo optimize the similar offset.
D4: setting a similarity threshold s according to the application requirementsthrdIf the optimal similarity obtained in step D3 is greater than or equal to sthrdI.e. the two frame groups are considered to match, whereas the two frame groups are considered to not match.
The invention solves the defects of short-term and unstable traditional audio fingerprints, extracts the frequency spectrum sub-band characteristics of an audio frame, then calculates the change attribute of the time axis direction to form long-term characteristics, and obtains the optimal similarity and offset during matching, and the invention can be applied to the matching of audio streams segment by segment and the integral matching of audio files;
compared with the prior art of the same type, the method is characterized in that:
the foregoing illustrates and describes the principles, general features, and advantages of the present invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the specification and illustrated only to illustrate the principle of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the present invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.