WO2023231787A1 - 音频处理方法和装置 - Google Patents
音频处理方法和装置 Download PDFInfo
- Publication number
- WO2023231787A1 WO2023231787A1 PCT/CN2023/094896 CN2023094896W WO2023231787A1 WO 2023231787 A1 WO2023231787 A1 WO 2023231787A1 CN 2023094896 W CN2023094896 W CN 2023094896W WO 2023231787 A1 WO2023231787 A1 WO 2023231787A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- terminal device
- audio signal
- sound source
- audio signals
- audio
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 230000005236 sound signal Effects 0.000 claims abstract description 362
- 238000000926 separation method Methods 0.000 claims abstract description 90
- 238000000034 method Methods 0.000 claims abstract description 48
- 238000012545 processing Methods 0.000 claims description 53
- 230000006870 function Effects 0.000 claims description 35
- 230000015654 memory Effects 0.000 claims description 24
- 239000011159 matrix material Substances 0.000 claims description 20
- 238000012546 transfer Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 10
- 230000000875 corresponding effect Effects 0.000 description 76
- 238000004891 communication Methods 0.000 description 20
- 238000010586 diagram Methods 0.000 description 14
- 238000004458 analytical method Methods 0.000 description 13
- 230000001629 suppression Effects 0.000 description 10
- 230000003287 optical effect Effects 0.000 description 8
- 238000012360 testing method Methods 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 5
- 238000010295 mobile communication Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 239000000284 extract Substances 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000012880 independent component analysis Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
- H04M1/7243—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages
- H04M1/72433—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages for voice messaging, e.g. dictaphones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
- H04M1/7243—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages
- H04M1/72439—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages for image or video messaging
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
Definitions
- the present application relates to the field of terminal technology, and in particular, to an audio processing method and device.
- the terminal device can obtain N-channel audio signals by separating the sound sources in the video shooting scene, so that the N-channel audio signals can be used for speech enhancement, speech suppression, speech recognition, or human speech processing during video processing. Computer interaction and other functions.
- Embodiments of the present application provide an audio processing method and device so that a terminal device can acquire a first audio signal, perform sound source separation on the first audio signal to obtain N audio signals; and perform sound source separation on the N audio signals.
- Accuracy judgment when it is determined that the N-channel audio signal meets the preset conditions, a video that meets the user's needs is generated based on the N-channel audio signal, and the accuracy of the sound when generating the video based on the N-channel audio signal is improved.
- inventions of the present application provide an audio processing method.
- the method includes: a terminal device displays a first interface; the first interface includes: a control for starting recording; and the terminal device receives a control for starting recording.
- the terminal device displays a second interface and obtains a first audio signal; wherein, the second interface includes a first picture and a second picture, the second picture covers the first picture, and the first picture includes a third The content in the second picture; the second picture includes the target object; the terminal device performs sound source separation on the first audio signal to obtain N audio signals; N is an integer greater than or equal to 2; the terminal device generates the first video and the second Video; wherein, the first video is obtained based on the first audio signal and the first picture; when the N audio signals meet the preset conditions, the second video is obtained based on the N audio signals and the second picture; The second audio signal corresponding to the second video is obtained by processing the target audio signal in the N audio signals and/or the signals other than the target audio signal in the N audio
- the terminal device can obtain the first audio signal, perform sound source separation on the first audio signal to obtain N-channel audio signals; and judge the accuracy of the N-channel audio signal separation, and determine that the N-channel audio signal meets the predetermined requirements.
- a video that meets user needs is generated based on N-channel audio signals, and the accuracy of the sound when generating a video based on N-channel audio signals is improved.
- the first interface may be an interface corresponding to the video recording function; the first picture may be a video recording interface, and the second picture may be a picture-in-picture overlaid on the first picture.
- the terminal device can use the first audio signal without sound source separation to generate the second video when the N-channel audio signal separation is inaccurate, so as to avoid using the N-channel audio signal with inaccurate separation to affect the audio effect in the second video.
- the N-channel audio signals not meeting the preset conditions include: the energy of any audio signal among the N-channel audio signals is greater than the energy threshold, and the angle of any audio signal is within the corresponding angle within the time threshold.
- the variance is greater than the variance threshold; and/or the energy of any audio signal is greater than the energy threshold, and the correlation between any audio signal and other audio signals in the N audio signals is greater than or equal to the correlation threshold.
- the terminal device can be based on the direction of the N-channel audio signal and the energy of the N-channel audio signal; and/or, the energy of the N-channel audio signal and the correlation between any audio signal and other audio signals in the N-channel audio signal, Perform accuracy analysis on the N-channel audio signals obtained after sound source separation.
- the angle of any audio signal is obtained based on the column data corresponding to any audio signal in the unmixing matrix and the transfer function of the terminal device at each preset angle; where, the unmixing The matrix is obtained when the terminal device performs sound source separation on the first audio signal.
- the terminal device can calculate the angle of any audio signal based on the unmixing matrix and the preset transfer function, so that the terminal device can perform an accuracy analysis of sound source separation based on the angle of any audio signal.
- the range of the preset angle satisfies: 0°-180°, or 180°-360°. In this way, it can be avoided that the angle of any audio signal cannot be determined because the two microphones in the terminal device cannot identify the forward and backward directions of the connection between the two microphones.
- N audio signals satisfy preset conditions, including: the energy of any audio signal is greater than the energy threshold, and the angle variance of any audio signal corresponding to the time threshold is less than or equal to the variance threshold; And/or, the energy of any audio signal is greater than the energy threshold, and the correlation between any audio signal and other audio signals is less than the correlation threshold.
- the terminal device can be based on the direction of the N-channel audio signal and the energy of the N-channel audio signal; and/or, the energy of the N-channel audio signal and the correlation between any audio signal and other audio signals in the N-channel audio signal, Perform accuracy analysis on the N-channel audio signals obtained after sound source separation.
- the terminal device when the terminal device receives an operation for the control for starting recording, the terminal device displays the second interface, including: when the terminal device receives an operation for the control for starting recording, The terminal device displays a third interface; the third interface includes a first screen, and the first screen includes a target object; when the terminal device receives an operation for the target object, the terminal device displays the second interface.
- the terminal device can flexibly select and highlight the target object according to the user's triggering operation on the target object, and process the sound of the target object, thereby improving the three-dimensional sense of the target object's sound.
- the second interface includes: a control for ending recording, and the terminal device performs sound source separation on the first audio signal to obtain N audio signals, including: when the terminal device receives a signal for When the operation of the recorded control ends, the terminal device performs sound source separation on the first audio signal to obtain N audio signals.
- the terminal device can determine when to end the recording according to the user's needs, and perform sound source separation processing on the first audio signal, thereby increasing the user's experience of the video recording function.
- inventions of the present application provide an audio processing device.
- the device includes: a display unit and a processing unit. unit; a display unit, used to display the first interface; the first interface includes: a control for starting recording; when the terminal device receives an operation for the control used to start recording, the display unit is also used for terminal device display second interface, and obtains the first audio signal; wherein, the second interface includes a first picture and a second picture, the second picture covers the first picture, and the first picture includes the content in the second picture;
- the second picture includes the target object;
- the processing unit is used to separate the sound source of the first audio signal to obtain N audio signals; N is an integer greater than or equal to 2; the processing unit is also used to generate the first video and the second Video; wherein, the first video is obtained based on the first audio signal and the first picture; when the N audio signals meet the preset conditions, the second video is obtained based on the N audio signals and the second picture;
- the second audio signal corresponding to the second video is obtained by processing the target
- the second video is obtained based on the first audio signal and the second picture.
- the N-channel audio signals not meeting the preset conditions include: the energy of any audio signal among the N-channel audio signals is greater than the energy threshold, and the angle of any audio signal is within the corresponding angle within the time threshold.
- the variance is greater than the variance threshold; and/or the energy of any audio signal is greater than the energy threshold, and the correlation between any audio signal and other audio signals in the N audio signals is greater than or equal to the correlation threshold.
- the angle of any audio signal is obtained based on the column data corresponding to any audio signal in the unmixing matrix and the transfer function of the terminal device at each preset angle; where, the unmixing The matrix is obtained when the terminal device performs sound source separation on the first audio signal.
- the range of the preset angle satisfies: 0°-180°, or 180°-360°.
- N audio signals satisfy preset conditions, including: the energy of any audio signal is greater than the energy threshold, and the angle variance of any audio signal corresponding to the time threshold is less than or equal to the variance threshold; And/or, the energy of any audio signal is greater than the energy threshold, and the correlation between any audio signal and other audio signals is less than the correlation threshold.
- the terminal device display unit when the terminal device receives an operation for a control for starting recording, the terminal device display unit is specifically configured to display a third interface; wherein the third interface includes a first screen, a third screen, and a third screen.
- a screen includes a target object; when the terminal device receives an operation for the target object, the display unit is also specifically used to display the second interface.
- the second interface includes: a control for ending recording.
- the processing unit is specifically configured to process the first audio signal.
- the sound source is separated to obtain N audio signals.
- embodiments of the present application provide a terminal device, including a processor and a memory.
- the memory is used to store code instructions; the processor is used to run the code instructions, so that the terminal device can execute the first aspect or any one of the first aspects. method described in Implementation Methods.
- embodiments of the present application provide a computer-readable storage medium.
- the computer-readable storage medium stores instructions. When the instructions are executed, the computer executes as in the first aspect or any implementation of the first aspect. described method.
- a computer program product includes a computer program that, when executed, causes the computer to The computer executes the method described in the first aspect or any implementation manner of the first aspect.
- Figure 1 is a schematic interface diagram of a video recording scene provided by an embodiment of the present application.
- Figure 2 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.
- Figure 3 is a schematic interface diagram of another recording scene provided by an embodiment of the present application.
- Figure 4 is a schematic flow chart of an audio processing method provided by an embodiment of the present application.
- Figure 5 is a schematic diagram of a spatial angle division provided by an embodiment of the present application.
- Figure 6 is a schematic flow chart of a sound source accuracy analysis provided by an embodiment of the present application.
- Figure 7 is a schematic diagram of an interface for selecting a target object provided by an embodiment of the present application.
- Figure 8 is a schematic flow chart of an audio processing method provided by an embodiment of the present application.
- Figure 9 is a schematic flow chart of another sound source accuracy analysis provided by an embodiment of the present application.
- Figure 10 is a schematic structural diagram of an audio processing device provided by an embodiment of the present application.
- Figure 11 is a schematic diagram of the hardware structure of another terminal device provided by an embodiment of the present application.
- blind source separation may also be called blind signal separation or sound source separation. It may be a method of separating the source signal from the received mixed signal without knowing the source signal and signal mixing parameters. .
- the sound source separation methods may include: independent vector analysis (IVA) method, independent component analysis (ICA) method, non-negative matrix factorization (NMF) method, Or multi-channel non-negative matrix factorization method (multichannel NMF, MNMF), etc.
- IVA independent vector analysis
- ICA independent component analysis
- NMF non-negative matrix factorization
- MNMF multi-channel non-negative matrix factorization method
- Suppression refers to reducing the energy of an audio signal so that the audio signal sounds smaller or even inaudible. Suppression describes the degree to which an audio signal is reduced. The greater the degree of suppression, the more the energy of the audio signal is reduced.
- the degree of suppression can be expressed by the gain coefficient.
- the gain coefficient When the gain coefficient is 0.0, it means that the audio signal is completely removed; when the gain coefficient is 1.0, it means no suppression is performed; when the gain coefficient is greater than 1.0, it means the audio signal is enhanced.
- Picture-in-picture divides the display screen of the terminal device into two display interfaces, one large and one small, with the smaller display interface included in the larger display interface.
- the larger display area generally covers the screen of the device, and the image in the smaller display area can be overlaid on the image in the larger display area.
- the smaller display area also supports zooming, and its position on the terminal device screen can also be changed.
- words such as “first” and “second” are used to distinguish the same or similar items with basically the same functions and effects.
- the first value and the second value are only used to distinguish different values, and their order is not limited.
- words such as “first” and “second” do not limit the number and execution order, and words such as “first” and “second” do not limit the number and execution order.
- At least one refers to one or more, and “plurality” refers to two or more.
- “And/or” describes the association of associated objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A, B can be singular or plural.
- the character “/” generally indicates that the related objects are in an “or” relationship.
- “At least one of the following” or similar expressions thereof refers to any combination of these items, including any combination of a single item (items) or a plurality of items (items).
- At least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, c can be single or multiple.
- FIG. 1 is a schematic interface diagram of a video recording scene provided by an embodiment of the present application.
- the recording scene can be a conference scene.
- the terminal device is a mobile phone as an example for illustration. This example does not constitute a limitation on the embodiment of the present application.
- the terminal device When the terminal device receives the user's operation to open the camera application and trigger the operation of entering the video recording function, the terminal device can display an interface as shown in Figure 1.
- the interface may include: a preview screen 100, a control for turning on or off the flash, a setting control for setting the camera application, a control for adjusting the shooting magnification, a control for flipping the camera, As well as a control for opening the gallery, a control 106 for starting recording corresponding to the video recording function, and multiple function controls in the first-level menu of the camera application.
- the multiple function controls in the first-level menu of the camera application may include: portrait function controls, camera function controls, video recording function controls, short video function controls, and controls for enabling more functions.
- the preview screen 100 shown in FIG. 1 may include: target 101, target 102, target 103, target 104, and target 105 surrounding the conference table.
- the terminal device can record the video of the conference scene and extract the sound source for a certain target in the scene through audio processing methods. Since there may be many sound sources in this scene, and/or the energy of the sound sources may be large or small, etc., it may have an impact on the accuracy of sound source separation when the terminal device extracts the sound source of a certain target, thereby affecting the accuracy of the sound source separation. Other audio processing processes after sound source separation.
- inventions of the present application provide an audio processing method.
- the terminal device displays a first interface; the first interface includes: a control for starting recording; when the terminal device receives an operation for the control for starting recording , the terminal device displays the second interface and obtains the first audio signal; wherein, the second interface includes the first picture and the second picture, the second picture covers the first picture, and the first picture includes the second picture
- the first video is obtained based on the first audio signal and the first picture; when the N audio signals meet the preset conditions, the second video is obtained based on the N audio signals. and the second screen obtained.
- the terminal device can use N-channel audio signals with relatively accurate sound source separation for generating the second video.
- the second audio signal corresponding to the second video is obtained by processing the target audio signal in the N audio signals and/or the signals other than the target audio signal in the Nth audio signal, so that the terminal device can pass the audio signal. Processing of different audio signals in N-channel audio signals with accurate source separation, enhancing the three-dimensional sense of the sound source of the target object in the second video.
- the audio processing method provided by the embodiment of the present application can not only be applied to the video recording scene as shown in Figure 1, but can also be used in scenes involving sound source separation such as live broadcast scenes. In the embodiment of the present application, This is not limited.
- terminal equipment can also be called a terminal (terminal), user equipment (user equipment, UE), mobile station (mobile station, MS), mobile terminal (mobile terminal, MT), etc.
- Terminal devices can be mobile phones with MICs, smart TVs, wearable devices, tablets (Pads), computers with wireless transceiver functions, virtual reality (VR) terminal devices, augmented reality, AR) terminal equipment, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grid Terminals, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc.
- the embodiments of this application do not limit the specific technology and specific equipment form used by the terminal equipment.
- FIG. 2 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.
- the terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, and a mobile communication module.
- a processor 110 an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, and a mobile communication module.
- Wireless communication module 160 audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone interface 170D, sensor module 180, button 190, indicator 192, camera 193, and display screen 194, etc.
- the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device.
- the terminal device may include more or less components than shown in the figures, or some components may be combined, or some components may be separated, or may be arranged differently.
- the components illustrated may be implemented in hardware, software, or a combination of software and hardware.
- Processor 110 may include one or more processing units. Among them, different processing units can be independent devices or integrated in one or more processors.
- the processor 110 may also be provided with a memory for storing instructions and data.
- the USB interface 130 is an interface that complies with the USB standard specification, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.
- the USB interface 130 can be used to connect a charger to charge the terminal device, and can also be used to transmit data between the terminal device and peripheral devices. It can also be used to connect headphones to play audio through them. This interface can also be used to connect other terminal devices, such as AR devices.
- the charging management module 140 is used to receive charging input from the charger.
- the charger can be a wireless charger or a wired charger.
- the power management module 141 is used to connect the charging management module 140 and the processor 110 .
- the wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.
- Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
- Antennas in end devices can be used to cover single or multiple communication bands. Different antennas can also be reused to improve antenna utilization.
- the mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied to terminal devices.
- the mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.
- the mobile communication module 150 can receive electromagnetic waves through the antenna 1, perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation.
- the wireless communication module 160 can provide applications including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), and global navigation satellite systems (WLAN) applied to terminal devices.
- WLAN wireless local area networks
- BT Bluetooth
- WLAN global navigation satellite systems
- GNSS Global navigation satellite system
- FM frequency modulation
- the terminal device implements display functions through the GPU, the display screen 194, and the application processor.
- the GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
- the display screen 194 is used to display images, videos, etc.
- Display 194 includes a display panel.
- the terminal device may include 1 or N display screens 194, where N is a positive integer greater than 1.
- the terminal device can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor.
- Camera 193 is used to capture still images or video.
- the terminal device may include 1 or N cameras 193, where N is a positive integer greater than 1.
- the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device.
- the external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. Such as saving music, videos, etc. files in external memory card.
- Internal memory 121 may be used to store computer executable program code, which includes instructions.
- the internal memory 121 may include a program storage area and a data storage area.
- the terminal device can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. Such as music playback, recording, etc.
- the audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals.
- Speaker 170A also called “speaker”
- Receiver 170B also called “earpiece”
- the headphone interface 170D is used to connect wired headphones.
- Microphone (MIC) 170C also known as “microphone” or “microphone” is used to convert sound signals into electrical signals.
- the terminal device can receive a sound signal based on the microphone 170C and convert the sound signal into an electrical signal that can be subsequently processed.
- the terminal device can have at least two microphones 170C.
- the microphone 170C can acquire audio signals within a 360° range centered on the terminal device.
- the sensor module 180 may include one or more of the following sensors, such as: pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light Sensor, or bone conduction sensor, etc. (not shown in Figure 2).
- sensors such as: pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light Sensor, or bone conduction sensor, etc. (not shown in Figure 2).
- the buttons 190 include a power button, a volume button, etc.
- Key 190 may be a mechanical key. It can also be a touch button.
- the terminal device can receive key input and generate key signal input related to user settings and function control of the terminal device.
- the indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, or may be used to indicate messages, missed calls, notifications, etc.
- the software system of the terminal device can adopt a layered architecture, event-driven architecture, micro-kernel architecture, micro-service architecture, or cloud architecture, etc., which will not be described again here.
- the terminal device may support audio and video processing methods in picture-in-picture mode.
- FIG. 3 is a schematic interface diagram of another recording scene provided by an embodiment of the present application.
- the terminal device may display the interface shown in a in FIG. 3 .
- the interface may include: controls for pausing recording, controls 301 for ending recording, controls for capturing images during video recording, and controls for zooming in on images. , controls to zoom out, logos to indicate the video recording duration, and more.
- the triggering operation may include a single-click operation, a double-click operation, a long-press operation, etc., which are not limited in the embodiments of the present application.
- the terminal device when the terminal device supports the picture-in-picture mode, when the terminal device receives the user's trigger operation for the target 103 in the interface shown in a in Figure 3, the terminal device extracts the preview screen containing the target 103. , and the interface shown in b in Figure 3 is displayed. As shown in b in Figure 3, the interface may include a picture-in-picture 302, and the picture-in-picture 302 may include a target 103.
- the picture-in-picture 302 can be superimposed on the preview screen 100, and the picture-in-picture 302 can be located in the upper left corner of the preview screen 100, or can be located elsewhere. In terms of location, the location of the picture-in-picture 302 is not limited in the embodiment of the present application.
- the terminal device when the terminal device supports the picture-in-picture mode, the terminal device can detect the sound source with the highest energy in the current scene. When the sound source with the highest energy is the sound source of the target 103, the terminal device can extract the preview image containing the target 103 and display the interface shown in b in Figure 3.
- the terminal device when the terminal device receives the user's trigger operation for the control 301 for ending recording, the terminal device can obtain the corresponding preview image 100 based on the audio signal obtained by the microphone and the preview image 100. video; and, perform sound source separation on the audio signal obtained based on the microphone to obtain N-channel audio signals, and obtain the audio signal at the angle corresponding to the target 103 in the N-channel audio signal, and remove the target 103 from the N-channel audio signal.
- Other sound sources are processed, and then the video corresponding to the picture-in-picture 302 is obtained based on the processed N-channel audio signals and the picture-in-picture 302.
- the embodiment of the present application provides two audio processing methods, for example: Method 1, an audio processing method based on sound source orientation estimation; and Method 2, based on sound source correlation discrimination. Audio processing methods.
- Method 1 Audio processing method based on sound source orientation estimation.
- FIG. 4 is a schematic flowchart of an audio processing method provided by an embodiment of the present application. As shown in Figure 4, The audio processing method may include the following steps:
- the terminal device uses M MICs to obtain initial audio signals.
- M is greater than or equal to 2
- the M MICs may include: at least two MICs in the terminal device; or, when the terminal device establishes a connection with a true wireless stereo (TWS) headset , the M MICs may also include: at least two MICs in TWS earphones, and/or at least two MICs in terminal equipment, etc.
- TWS true wireless stereo
- the specific form of the MIC for collecting the initial audio signal is not specified. limited.
- the user's operation to start recording may be: the user's trigger operation on the control for starting recording, voice operation, or other gesture operation, etc., which is not limited in the embodiment of the present application.
- the terminal device when the terminal device receives the user's trigger operation on the control 106 for starting recording, the terminal device can use the camera to obtain the image signal display as shown in a in Figure 3 Recording interface, and use M MICs to obtain initial audio signals.
- the terminal device When the terminal device receives the user's operation to end recording, the terminal device performs sound source separation on the initial audio signal to obtain a demixing matrix and N-channel audio signals.
- the operation to end recording may be: a user's trigger operation on the control for ending recording, a voice operation, or other gesture operations, etc. This is not limited in the embodiments of this application.
- the terminal device when the terminal device receives the user's trigger operation on the control 301 for ending recording, the terminal device can perform sound source separation on the initial audio signal to obtain the solution. Mixing matrix and N-channel audio signals.
- the terminal device can perform Fourier transform on the initial audio signal in the time domain obtained based on S401 and convert it into an initial audio signal in the frequency domain; further, the initial audio signal in the frequency domain can be separated from the sound source to obtain Unmixing matrix and N-channel audio signals.
- the sound source separation method may be IVA or the like.
- the time domain sequence of the sound sources can be expressed as: s 1(t) , s 2(t) ,..., s N(t) ; s represents the source signal (source), t represents the time sampling sequence; assuming that there are M MICs collecting initial audio signals, the audio signals corresponding to each sound source obtained by the terminal device can be expressed as x 1(t) , x 2(t) , ..., x M(t) .
- s n(t) is the n-th sound source
- x m(t) is the audio signal picked up by the m-th MIC
- ⁇ is the time delay
- L is the maximum time delay
- h can be understood as the source signal s n(t ) and the signal x m(t) collected by the MIC.
- N is the number of sound sources and M is the number of MICs
- x is a vector of length M
- s is a vector of length N
- ⁇ is the frequency
- t is the number of frames
- A is an M row and N column
- the matrix can be understood as the transmission path between the source signal s and the signal x collected by the MIC.
- W can be a matrix with N rows and M columns, or it can also be called an unmixing matrix; for other parameters in formula (3), please refer to the description in formula (2), and will not be described again here.
- the terminal device can make the actual number of sound sources N and the MIC mathematical M the same during the sound source processing method.
- the terminal device obtains the sound source angles corresponding to the N audio signals and obtains N sound source angles.
- the terminal device can obtain the transfer function corresponding to each preset angle around the terminal device, and use the column vector corresponding to any sound source in the unmixing matrix and the transfer function corresponding to the preset angle to obtain the sound source at any
- the dimensionless value at an angle is then calculated as the minimum value of the dimensionless value at each angle of the terminal device as the direction (or angle) of the sound source.
- the transfer function corresponding to each preset angle of the terminal device can be used to indicate the relative relationship between the audio signals received at different angles.
- the transfer function corresponding to each preset angle may be stored in the terminal device.
- FIG. 5 is a schematic diagram of spatial angle division provided by an embodiment of the present application.
- the preset angle may be 0°, 10°, 20°,...350°.
- test audio signals are collected every 10° in the direction of 360° around the terminal device, for example, test audio signals are collected in the directions of 0°, 10°, 20°,..., 350°. Then 36 test audio signals corresponding to 36 directions are obtained.
- Y ( ⁇ , q) is the audio signal of MIC at frame q and frequency point ⁇ .
- the terminal equipment can use the test audio signal obtained by any MIC, such as the test audio signal X 1 ( ⁇ , q) at the first MIC (or reference MIC) as a reference audio signal, to obtain the test audio collected by other MICs.
- H ( ⁇ , ⁇ ) [1, ⁇ 1,2 ,..., ⁇ 1,m ]
- ⁇ 1,m is the transfer function relationship between the audio signal X m ( ⁇ , q) of the m-th MIC and the audio signal X 1 ( ⁇ , q) of the reference MIC.
- * represents the conjugate calculation
- E represents the expectation calculation
- the terminal device can store the obtained H ( ⁇ , ⁇ ) corresponding to each preset angle in the device, so that the terminal device can estimate the subsequent sound source orientation based on the H ( ⁇ , ⁇ ) .
- the device for obtaining H ( ⁇ , ⁇ ) may not be limited to a terminal device, and may be a server, for example, which is not limited in the embodiments of the present application.
- the method of using H ( ⁇ , ⁇ ) to estimate the sound source direction can be: dot multiply the transfer function H 1 ( ⁇ ) corresponding to any preset angle with a certain column w H of the unmixing matrix, to obtain ⁇ .
- each column in the unmixing matrix can be Responds to a sound source.
- ⁇ 2 > ⁇ 1 , and ⁇ 1 - ⁇ 2 can be any frequency band range.
- the terminal device can obtain N sound source angles respectively corresponding to the N audio signals.
- the terminal device can determine the angle of the target 103 based on image recognition, and the angles corresponding to the N sound source angles in the step shown in S404, Then, the sound source of the target 103 is determined when the angle at which the target 103 is located coincides with any of the N sound source angles.
- the terminal device determines the accuracy of the sound source in the current scene based on the N sound source angles and the energy of the N audio signals.
- the steps shown in S405 can be executed; when the terminal device determines that the sound sources in the current scene are partially accurately separated (or understood as partial sound source separation) (accurate, partial sound source separation is inaccurate), the step shown in S406 can be performed; or, when the terminal device determines that the separation of each sound source in the current scene is inaccurate, the step shown in S406 can be performed.
- the terminal device can determine the accuracy of separation of any of the N audio signals based on the embodiment corresponding to Figure 6, and then output the accuracy of the current scene sound source.
- FIG. 6 is a schematic flowchart of a sound source accuracy analysis provided by an embodiment of the present application. As shown in Figure 6, the sound source accuracy analysis method may include the following steps:
- the terminal device calculates the variance corresponding to the sound source angle within a time period.
- the period of time can be a time period of 5 seconds, 10 seconds, or 20 seconds; the sound source angle can be any sound source angle among N sound source angles.
- the terminal device can use the sound source to change the direction of the sound source.
- the variance of the source angle determines the accuracy of the sound source.
- the terminal device determines whether the variance is greater than the variance threshold.
- the terminal device when the terminal device determines that the variance is greater than the variance threshold, the terminal device can perform the steps shown in S603; or when the terminal device determines that the variance is less than or equal to the variance threshold, the terminal device can perform the steps shown in S606.
- the terminal device calculates the energy corresponding to the sound source.
- the sound source can be any audio signal among the N audio signals; the energy corresponding to the sound source can be the energy corresponding to any audio signal among the N audio signals.
- the terminal device can also count the average energy of the sound source within a period of time, and based on this energy The average value is used to accurately identify the sound source.
- the terminal device determines whether the energy is greater than the first energy threshold.
- the step shown in S605 may be performed; or, when the terminal device determines that the energy is less than or equal to the first energy threshold, the step shown in S606 may be performed.
- the terminal device may also perform the steps shown in S605.
- the variance corresponding to a certain sound source angle within a period of time is greater than the variance threshold, and the energy of the sound source is less than the first energy threshold, it can be determined that the sound source can be scatter noise, and scatter noise does not affect the accuracy. Therefore, it can be determined that the sound source separation of this road is accurate; and when the variance corresponding to a certain sound source angle within a period of time is less than or equal to the variance threshold, and the energy of the sound source is greater than the first energy threshold, the sound source can be determined Separation is not accurate.
- the terminal device can also only use the energy of the sound source to determine whether the sound source separation is accurate.
- the terminal device determines that the current sound source separation is inaccurate.
- the terminal device determines that the current sound source separation is accurate.
- the terminal device counts the accuracy of all sound sources and determines the accuracy of the current scene sound sources.
- the terminal device determines that the sound source separation of the current scene is accurate, and then performs the steps shown in S405; when it is determined that some of the N-channel audio signals in the scene are accurately separated and some When the signal separation is inaccurate, the terminal device determines that the current scene sound source separation is partially accurate, and then performs the steps shown in S406; or when it is determined that the N audio signals in the scene are all inaccurately separated, the terminal device determines that the current scene sound source separation is inaccurate. Inaccurate, and then perform the steps shown in S406.
- the terminal device can accurately analyze the sound source separation based on the sound source direction of the N audio signals and the energy of the N audio signals.
- the terminal device When it is determined that the current scene sound source separation is accurate, the terminal device outputs N audio signals.
- the terminal device when the terminal device determines that the scene sound source separation is accurate, when generating the video corresponding to the picture-in-picture 302, the terminal device can process the N-channel audio signals. Suppress the sound sources other than the sound source of the target 103 (or enhance the sound source of the target 103), so that the sound of the target 103 can be highlighted in the video corresponding to the picture-in-picture 302, and the sound quality in the picture-in-picture scene can be improved. Three-dimensional feeling.
- the method for the terminal device to suppress the sound sources other than the sound source of the target 103 in the scene may include: the terminal device enhances the sound source of the target 103, such as The sound source is multiplied by the first gain coefficient, for example, the first gain coefficient can be a value such as 1.5; the sound source other than the sound source of the target 103 is enhanced, such as multiplying by the second gain coefficient, for example, the second gain coefficient can be Values such as 1.2, where the second gain coefficient is smaller than the first gain coefficient.
- the terminal device suppresses sound sources other than the sound source of the target 103, such as multiplying by a third gain coefficient.
- the third gain coefficient can be a value such as 0.7.
- the suppression described in the embodiments of the present application is the concept of relative suppression
- the enhancement is the concept of relative enhancement
- the method of suppressing the sound source in the embodiments of the present application is not specifically limited.
- the terminal device When it is determined that the current scene sound source separation is partially accurate or the sound source separation is inaccurate, the terminal device outputs the audio signal before the sound source separation.
- the audio signal before sound source separation may be the initial audio signal in the frequency domain described in the step shown in S402.
- the terminal device when the terminal device determines that scene sound source separation is inaccurate In this case, when generating the video corresponding to picture-in-picture 302, the terminal device can use the audio signal before sound source separation.
- the audio signal output by the terminal device may not be limited to the audio signal before the sound source separation, and the audio signal output by it may also be the audio signal before the sound source separation.
- the N-channel audio signal is a signal output after being processed by other algorithms. This is not limited in the embodiments of the present application.
- the terminal device can perform accuracy analysis on the N audio signals obtained after sound source separation based on the direction of the N audio signals and the energy of the N audio signals, so that the terminal device can perform subsequent audio based on the accuracy analysis results.
- Signal processing
- the terminal device can display a picture-in-picture including the target object, And when generating the video corresponding to the picture-in-picture, the sound source corresponding to the target object is enhanced (or the sound source other than the target object is suppressed).
- FIG. 7 is a schematic diagram of an interface for selecting a target object provided by an embodiment of the present application.
- the target object is taken as the target 103 as an example for illustration.
- a coordinate system is established with the terminal device as the center, and the terminal device can identify 360° audio signals around it. Since the terminal device cannot distinguish the forward direction of the terminal device and the backward direction of the terminal device, the terminal device is used as the center.
- the forward direction is 0-180° as an example to explain the sound source recognized by the terminal device.
- the terminal device can display an interface including a picture-in-picture 302 as shown in a in Figure 7, and determine where the target 103 is located based on image recognition (such as face recognition). Angle (or coordinate). Further, when the terminal device receives the user's operation to end the recording, the terminal device can determine the accuracy of the sound source of the current scene based on the steps shown in S402-S406, output the corresponding audio signal, and generate the preview screen 100 The corresponding video, and the video corresponding to picture-in-picture 302.
- the terminal device can determine the accuracy of the sound source based on the steps shown in S402 and S406, output the corresponding audio signal, and combine it with the multiple frame images corresponding to the captured preview screen 100 to obtain Preview the video corresponding to screen 100.
- the terminal device can determine the accuracy of the sound source based on the steps shown in S402 and S406, and then combine the angle of the target 103 obtained in the image recognition and the steps shown in S404.
- the sound source angle corresponding to the sound source is determined to determine the sound source of target 103.
- the terminal device suppresses sound sources in the scene other than the sound source of the target 103 , and combines the captured multi-frame images including the target 103 to obtain a video corresponding to the picture-in-picture 302 .
- the terminal device when the terminal device receives the user's operation of selecting the target object, when the terminal device combines the angle of the target 103 obtained in the image recognition, but does not detect the sound source from the step shown in S404 At the corresponding sound source angle, the terminal device can determine that the target 103 may not emit sound, so it is difficult to separate the sound source of the target 103 during the sound source separation process. At this time, the terminal device can determine the sound source around the target 103 as the sound source of the target 103 , for example, the sound source at the smallest angle with the target 103 as the sound source of the target 103 . Furthermore, the terminal device can enhance the sound sources around the target 103 in the scene.
- the terminal device not only generates a video containing the target object based on the user's operation of selecting the target object, but can also suppress sound sources other than the target object in the video containing the target object to enhance the stereoscopic sound of the target object. feel.
- the terminal device after the terminal device selects the target object, it can also support functions such as real-time tracking of the target object and automatic switching of the target object.
- the terminal device can support real-time tracking of target objects. For example, after the user selects the target object, the terminal device can track the location of the target object in real time.
- the terminal device can track the target 103 in real time based on image recognition, and determine the angle (or coordinates) of the target 103 after movement.
- the terminal device can continue to suppress the sound source of the target 103 being tracked in real time.
- the terminal device can switch the tracking target object.
- the terminal device can stop tracking the target 103 and start tracking the target 104.
- the terminal device can display the picture corresponding to the target 104 in the picture-in-picture 302 .
- the terminal device can enhance the sound source of the target 104 at the moment when the user switches the target 104.
- the method of determining the sound source angle of the target 104 and the method of enhancing the sound source of the target 104 please refer to the corresponding description of the target 103, and will not be described again here.
- the terminal device may support automatic switching of target objects. For example, after the user selects the target object, the terminal device can track the location of the sound source with the highest energy based on the energy of the sound source in the scene.
- the terminal device when the user selects target 103, when the terminal device detects that the energy of target 104 in the preview screen 100 is the highest, the terminal device can display as shown in Figure 7 The interface shown in c. As shown in the interface shown as c in Figure 7 , the terminal device can display the picture corresponding to the target 104 in the picture-in-picture 302 .
- the terminal device can enhance the sound source of the target 104 when it detects that the energy of the target 104 is the highest.
- the terminal device when the terminal device tracks the sound source with the highest energy based on the energy of the sound source in the scene, when there are multiple sound sources with the same energy, the terminal device automatically tracks the sound source located in the middle of the scene (for example, the angle is at 90 °), or the terminal device automatically tracks the target that is the most frequently selected by the user, or the terminal device automatically tracks the target that is the closest to the last selected target object, etc.
- the terminal device when the terminal device tracks the sound source with the highest energy based on the energy of the sound source in the scene, when the terminal device receives the user's operation to switch the target object, the terminal device can stop automatically tracking the sound source with the highest energy. source, and track the target object selected by the user.
- Method 2 Audio processing method based on sound source correlation discrimination.
- FIG. 8 is a schematic flowchart of an audio processing method provided by an embodiment of the present application.
- the terminal device uses M MICs to obtain initial audio signals.
- S802 The terminal device performs sound source separation on the initial audio signal to obtain N audio signals.
- the terminal device determines the accuracy of the sound source of the current scene based on the energy of the N audio signals and the correlation between any audio signal and other audio signals in the N audio signals.
- the correlation is used to indicate the degree of similarity between the two signals. It can be understood that since any two of the N audio signals must be uncorrelated, the accuracy of sound source separation can be determined through correlation.
- the terminal device can determine the accuracy of any of the N audio signals based on the embodiment shown in Figure 9, and then output the accuracy of the current scene sound source.
- FIG. 9 is a schematic flowchart of another sound source accuracy analysis provided by an embodiment of the present application.
- the terminal device determines the energy corresponding to the audio signal.
- the audio signal may be any audio signal among the N audio signals.
- the terminal device determines whether the energy is greater than the second energy threshold.
- the second energy threshold may be the same as the first energy threshold; when it is determined that the energy is greater than the second energy threshold, the terminal device may perform the steps shown in S903; or, when it is determined that the energy is less than or equal to the second energy threshold, it may be considered that If the current audio signal is scatter noise, it does not need to participate in the calculation of the correlation between subsequent audios, for example, perform S908.
- the terminal device determines the correlation between any one of the N audio signals and other audio signals among the N audio signals.
- the correlation calculation formula can be:
- ⁇ represents correlation
- a represents any audio signal among N audio signals
- b represents other audio signals among N audio signals
- t represents the number of frames of Fourier transform, which is equivalent to time
- ⁇ represents frequency
- * represents conjugate calculation
- E represents mathematical expectation
- the N-channel audio signals after the sound sources are separated are independent of each other, and the audio signals that are independent of each other are not correlated, it can be determined by the correlation between any audio signal and other audio signals in the N-channel audio signals. performance calculation to determine the accuracy of audio signal separation.
- the terminal device determines whether the correlation between any audio signal and other audio signals in the N audio signals is less than the correlation threshold.
- the terminal device determines that the correlation between any audio signal and other audio signals in the N audio signals is less than the correlation threshold, the terminal device can perform the steps shown in S905; or when the terminal device determines that the correlation between any audio signal and the N audio signals is less than the correlation threshold.
- the terminal device can perform the steps shown in S906.
- the correlation between any audio signal and other audio signals among N audio signals is not less than the correlation threshold, it can be understood as: the correlation between any audio signal and other audio signals among N audio signals is greater than Or equal to the correlation threshold; or, there is a small correlation between any audio signal and part of the N audio signals. With respect to the correlation threshold, and the correlation between any audio signal and the partial separation of the N audio signals is greater than or equal to the correlation threshold.
- the terminal device can also only use the correlation between any audio signal and other audio signals in the N audio signals to determine whether the sound source separation is accurate.
- the terminal device determines that the current sound source separation is accurate.
- the terminal device determines that the current sound source separation is inaccurate.
- the terminal device counts the accuracy of all sound sources and determines the accuracy of the current scene sound sources.
- the terminal device determines that the sound source separation of the current scene is accurate, and then performs the steps shown in S804; when it is determined that some of the N-channel audio signals in the scene are accurately separated and some When the signal separation is inaccurate, the terminal device determines that the current scene sound source separation is partially accurate, and then performs the steps shown in S805; or when it is determined that the N audio signals in the scene are all inaccurately separated, the terminal device determines that the current scene sound source separation is inaccurate. Inaccurate, then perform the steps shown in S805.
- S908 The terminal device ends the judgment of the current sound source separation accuracy.
- the terminal device can perform accuracy analysis on the audio signal based on the energy of the N audio signals and the correlation between any audio signal and other audio signals in the N audio signals.
- the terminal device When it is determined that the current scene sound source separation is accurate, the terminal device outputs N audio signals.
- the terminal device When it is determined that the sound source separation part of the current scene is accurate or the sound source separation is inaccurate, the terminal device outputs the audio signal before the sound source separation.
- the terminal device can accurately analyze the audio signal obtained after the sound source is separated based on the energy of the N audio signals and the correlation between any audio signal and other audio signals in the N audio signals, so that the terminal device Subsequent audio signal processing can be performed based on the accuracy analysis result.
- the terminal device can also judge the accuracy of sound source separation based on method one and method two, for example, when the terminal device determines the sound source separation in the current scene.
- the terminal device can determine that the current scene sound source separation is accurate and output the corresponding audio signal.
- Figure 10 is a schematic structural diagram of an audio processing device provided by an embodiment of the present application.
- the audio processing device may be a terminal device in an embodiment of the present application, or may be a chip or chip system in the terminal device. .
- the audio processing device 1000 can be used in communication equipment, circuits, hardware components or chips.
- the audio processing device includes: a display unit 1001 and a processing unit 1002 .
- the display unit 1001 is used to support the display steps performed by the audio processing device 1000; the processing unit 1002 is used to support the information processing steps performed by the audio processing device 1000.
- An embodiment of the present application provides an audio processing device 1000.
- the device includes: a display unit 1001 and a processing unit 1001.
- the second video is obtained based on the first audio signal and the second picture.
- the N-channel audio signals not meeting the preset conditions include: the energy of any audio signal among the N-channel audio signals is greater than the energy threshold, and the angle of any audio signal is within the corresponding angle within the time threshold.
- the variance is greater than the variance threshold; and/or the energy of any audio signal is greater than the energy threshold, and the correlation between any audio signal and other audio signals in the N audio signals is greater than or equal to the correlation threshold.
- the angle of any audio signal is obtained based on the column data corresponding to any audio signal in the unmixing matrix and the transfer function of the terminal device at each preset angle; where, the unmixing The matrix is obtained when the terminal device performs sound source separation on the first audio signal.
- the range of the preset angle satisfies: 0°-180°, or 180°-360°.
- N audio signals satisfy preset conditions, including: the energy of any audio signal is greater than the energy threshold, and the angle variance of any audio signal corresponding to the time threshold is less than or equal to the variance threshold; And/or, the energy of any audio signal is greater than the energy threshold, and the correlation between any audio signal and other audio signals is less than the correlation threshold.
- the terminal device display unit 1001 when the terminal device receives an operation for the control for starting recording, the terminal device display unit 1001 is specifically configured to display a third interface; wherein the third interface includes the first screen, The first screen includes the target object; when the terminal device receives an operation for the target object, the display unit 1001 is also specifically configured to display the second interface.
- the second interface includes: a control for ending recording.
- the processing unit 1002 is specifically configured to process the first audio signal. Perform sound source separation to obtain N-channel audio signals.
- the audio processing device 1000 may also include a communication unit 1003. Specifically, the communication unit is used to support the audio processing device 1000 in performing the steps of sending data and receiving data.
- the communication unit 1003 may be an input or output interface, a pin or a circuit, etc.
- the audio processing device may also include: a storage unit 1004.
- the processing unit 1002 and the storage unit 1004 are connected through lines.
- the storage unit 1004 may include one or more memories, which may be devices used to store programs or data in one or more devices or circuits.
- the storage unit 1004 may exist independently and be connected to the processing unit 1002 of the audio processing device through a communication line.
- Storage unit 1004 may also be Yuan 1002 integrated together.
- the storage unit 1004 may store computer execution instructions for the method in the terminal device, so that the processing unit 1002 executes the method in the above embodiment.
- the storage unit 1004 may be a register, cache, RAM, etc., and the storage unit 1004 may be integrated with the processing unit 1002.
- the storage unit 1004 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, and the storage unit 1004 may be independent from the processing unit 1002.
- Figure 11 is a schematic diagram of the hardware structure of another terminal device provided by an embodiment of the present application.
- the terminal device includes a processor 1101, a communication line 1104 and at least one communication interface (example in Figure 11 is a communication interface).
- Interface 1103 is taken as an example for explanation).
- the processor 1101 can be a general central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more processors used to control the execution of the program of the present application. integrated circuit.
- CPU central processing unit
- ASIC application-specific integrated circuit
- Communication lines 1104 may include circuitry that communicates information between the components described above.
- the communication interface 1103 uses any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.
- a transceiver to communicate with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.
- WLAN wireless local area networks
- the terminal device may also include a memory 1102.
- Memory 1102 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (random access memory (RAM)) or other type that can store information and instructions.
- a dynamic storage device can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), disk storage media or other magnetic storage devices, or can be used to carry or store desired program code in the form of instructions or data structures and can be used by a computer Any other medium for access, but not limited to this.
- the memory may exist independently and be connected to the processor through a communication line 1104 . Memory can also be integrated with the processor.
- the memory 1102 is used to store computer execution instructions for executing the solution of the present application, and is controlled by the processor 1101 for execution.
- the processor 1101 is used to execute computer execution instructions stored in the memory 1102, thereby implementing the audio processing method provided by the embodiment of the present application.
- the computer execution instructions in the embodiments of the present application may also be called application codes, which are not specifically limited in the embodiments of the present application.
- the processor 1101 may include one or more CPUs, such as CPU0 and CPU1 in FIG. 11 .
- the terminal device may include multiple processors, such as the processor 1101 and the processor 1105 in FIG. 11 .
- processors may be a single-CPU processor or a multi-CPU processor.
- a processor here may refer to one or more devices, circuits, and/or processing cores for processing data (eg, computer program instructions).
- a computer program product includes one or more computer instructions. When computer program instructions are loaded and executed on a computer, processes or functions according to embodiments of the present application are generated in whole or in part.
- the computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
- Computer instructions may be stored in a computer-readable storage medium, Or from one computer-readable storage medium to another computer-readable storage medium, for example, computer instructions may be transmitted from a website, computer, server or data center via wires (such as coaxial cables, optical fiber, digital subscriber lines). line, DSL) or wireless (such as infrared, wireless, microwave, etc.) to another website site, computer, server or data center.
- wires such as coaxial cables, optical fiber, digital subscriber lines. line, DSL
- wireless such as infrared, wireless, microwave, etc.
- the computer-readable storage medium can be any available medium that the computer can store or include a or Data storage devices such as servers and data centers integrating multiple available media.
- the available media may include magnetic media (such as floppy disks, hard disks or tapes), optical media (such as digital versatile discs (DVD)), Or semiconductor media (eg, solid state disk (SSD)), etc.
- Computer-readable media may include computer storage media and communication media and may include any medium that can transfer a computer program from one place to another.
- the storage media can be any target media that can be accessed by the computer.
- the computer-readable medium may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disk storage; the computer-readable medium may include a magnetic disk memory or other disk storage device.
- any connection line is also properly termed a computer-readable medium.
- coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave
- coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave
- Disk and optical disk include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Reproduce data.
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Circuit For Audible Band Transducer (AREA)
- Stereophonic System (AREA)
Abstract
本申请实施例提供一种音频处理方法和装置,方法包括:终端设备显示第一界面;在终端设备接收到针对用于开启录制的控件的操作时,终端设备显示第二界面,并获取第一音频信号;终端设备对第一音频信号进行声源分离,得到N路音频信号;N为大于或等于2的整数;终端设备生成第一视频以及第二视频;其中,在N路音频信号满足预设条件的情况下,第二视频是基于N路音频信号以及第二画面得到的;目标音频信号为目标对象的音频信号。这样,使得终端设备对第一音频信号进行声源分离得到的N路音频信号;并对N路音频信号分离的准确性进行判断,在确定该N路音频信号满足预设条件的情况下,基于N路音频信号生成满足用户需求的视频。
Description
本申请要求于2022年05月30日提交中国国家知识产权局、申请号为202210603382.1、申请名称为“音频处理方法和装置”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及终端技术领域,尤其涉及一种音频处理方法和装置。
随着终端设备的普及和发展,视频的拍摄和制作已经成为我们日常生活和娱乐中不可或缺的一环。在视频的拍摄以及播放场景中,用户对视频画面的观感以及视频音频的听感的要求日渐提高。
通常情况下,终端设备可以通过对视频拍摄场景中声音的声源分离,得到N路音频信号,使得该N路音频信号可以用于视频处理过程中的语音增强、语音抑制、语音识别、或人机交互等功能。
然而,由于视频拍摄场景的声源较为复杂且声源变化较大,使得音频处理方法的准确性较低,进而影响由该N路音频信号构成的视频。
发明内容
本申请实施例提供一种音频处理方法和装置,使得终端设备可以获取第一音频信号,对第一音频信号进行声源分离得到的N路音频信号;并对N路音频信号进行声源分离的准确性判断,在确定该N路音频信号满足预设条件的情况下,基于N路音频信号生成满足用户需求的视频,提高基于N路音频信号生成视频时的声音的准确性。
第一方面,本申请实施例提供一种音频处理方法,方法包括:终端设备显示第一界面;第一界面中包括:用于开启录制的控件;在终端设备接收到针对用于开启录制的控件的操作时,终端设备显示第二界面,并获取第一音频信号;其中,第二界面中包括第一画面以及第二画面,第二画面覆盖在第一画面之上,第一画面中包括第二画面中的内容;第二画面中包括目标对象;终端设备对第一音频信号进行声源分离,得到N路音频信号;N为大于或等于2的整数;终端设备生成第一视频以及第二视频;其中,第一视频是基于第一音频信号以及第一画面得到的;在N路音频信号满足预设条件的情况下,第二视频是基于N路音频信号以及第二画面得到的;第二视频对应的第二音频信号是对N路音频信号中的目标音频信号、和/或N路音频信号中的除目标音频信号以外的信号进行处理后得到的;目标音频信号为目标对象的音频信号。这样,使得终端设备可以获取第一音频信号,对第一音频信号进行声源分离得到的N路音频信号;并对N路音频信号分离的准确性进行判断,在确定该N路音频信号满足预设条件的情况下,基于N路音频信号生成满足用户需求的视频,提高基于N路音频信号生成视频时的声音的准确性。
其中,第一界面可以为录像功能对应的界面;第一画面可以为录像界面,第二画面可以为覆盖在第一画面上的画中画。
在一种可能的实现方式中,在N路音频信号不满足预设条件的情况下,第二视频是基于第一音频信号以及第二画面得到的。这样,终端设备可以实现在N路音频信号分离不准确时,采用未进行声源分离的第一音频信号生成第二视频,避免使用分离不准确的N路音频信号影响第二视频中的音频效果。
在一种可能的实现方式中,N路音频信号不满足预设条件包括:N路音频信号中的任一音频信号的能量大于能量阈值,且任一音频信号的角度在时间阈值内对应的角度方差大于方差阈值;和/或,任一音频信号的能量大于能量阈值,且任一音频信号与N路音频信号中的其他音频信号的相关性大于或等于相关性阈值。这样,终端设备可以基于N路音频信号所处方向以及N路音频信号的能量;和/或,N路音频信号的能量以及任一路音频信号与N路音频信号中的其他音频信号的相关性,对声源分离后得到的N路音频信号进行准确性分析。
在一种可能的实现方式中,任一音频信号的角度是基于解混矩阵中的任一音频信号对应的列数据、以及终端设备在各预设角度处的传递函数得到的;其中,解混矩阵是基于终端设备对第一音频信号进行声源分离时得到的。这样,终端设备可以基于解混矩阵以及预设的传递函数,对任一音频信号的角度进行计算,使得终端设备可以基于该任一音频信号的角度进行声源分离的准确性分析。
在一种可能的实现方式中,当终端设备的麦克风数量为2时,预设角度所在的范围满足:0°-180°,或者180°-360°。这样,可以避免由于终端设备中的2个麦克风无法识别2个麦克风连线的前向和后向,而无法确定任一音频信号的角度的情况。
在一种可能的实现方式中,N路音频信号满足预设条件,包括:任一音频信号的能量大于能量阈值,且任一音频信号的角度在时间阈值对应的角度方差小于或等于方差阈值;和/或,任一音频信号的能量大于能量阈值,且任一音频信号与其他音频信号的相关性均小于相关性阈值。这样,终端设备可以基于N路音频信号所处方向以及N路音频信号的能量;和/或,N路音频信号的能量以及任一路音频信号与N路音频信号中的其他音频信号的相关性,对声源分离后得到的N路音频信号进行准确性分析。
在一种可能的实现方式中,在终端设备接收到针对用于开启录制的控件的操作时,终端设备显示第二界面,包括:在终端设备接收到针对用于开启录制的控件的操作时,终端设备显示第三界面;其中,第三界面中包括第一画面,第一画面中包括目标对象;在终端设备接收到针对目标对象的操作时,终端设备显示第二界面。这样,使得终端设备可以灵活的根据用户针对目标对象的触发操作,选择突出显示目标对象,并对目标对象的声音进行处理,进而提高目标对象的声音的立体感。
在一种可能的实现方式中,第二界面中包括:用于结束录制的控件,终端设备对第一音频信号进行声源分离,得到N路音频信号,包括:在终端设备接收到针对用于结束录制的控件的操作时,终端设备对第一音频信号进行声源分离,得到N路音频信号。这样,使得终端设备可以根据用户用户的需求确定何时结束录制,并对第一音频信号进行声源分离处理,增加用户对于录像功能的使用体验。
第二方面,本申请实施例提供一种音频处理装置,装置包括:显示单元以及处理
单元;显示单元,用于显示第一界面;第一界面中包括:用于开启录制的控件;在终端设备接收到针对用于开启录制的控件的操作时,显示单元,还用于终端设备显示第二界面,并获取第一音频信号;其中,第二界面中包括第一画面以及第二画面,第二画面覆盖在第一画面之上,第一画面中包括第二画面中的内容;第二画面中包括目标对象;处理单元,用于对第一音频信号进行声源分离,得到N路音频信号;N为大于或等于2的整数;处理单元,还用于生成第一视频以及第二视频;其中,第一视频是基于第一音频信号以及第一画面得到的;在N路音频信号满足预设条件的情况下,第二视频是基于N路音频信号以及第二画面得到的;第二视频对应的第二音频信号是对N路音频信号中的目标音频信号、和/或N路音频信号中的除目标音频信号以外的信号进行处理后得到的;目标音频信号为目标对象的音频信号。
在一种可能的实现方式中,在N路音频信号不满足预设条件的情况下,第二视频是基于第一音频信号以及第二画面得到的。
在一种可能的实现方式中,N路音频信号不满足预设条件包括:N路音频信号中的任一音频信号的能量大于能量阈值,且任一音频信号的角度在时间阈值内对应的角度方差大于方差阈值;和/或,任一音频信号的能量大于能量阈值,且任一音频信号与N路音频信号中的其他音频信号的相关性大于或等于相关性阈值。
在一种可能的实现方式中,任一音频信号的角度是基于解混矩阵中的任一音频信号对应的列数据、以及终端设备在各预设角度处的传递函数得到的;其中,解混矩阵是基于终端设备对第一音频信号进行声源分离时得到的。
在一种可能的实现方式中,当终端设备的麦克风数量为2时,预设角度所在的范围满足:0°-180°,或者180°-360°。
在一种可能的实现方式中,N路音频信号满足预设条件,包括:任一音频信号的能量大于能量阈值,且任一音频信号的角度在时间阈值对应的角度方差小于或等于方差阈值;和/或,任一音频信号的能量大于能量阈值,且任一音频信号与其他音频信号的相关性均小于相关性阈值。
在一种可能的实现方式中,在终端设备接收到针对用于开启录制的控件的操作时,终端设备显示单元,具体用于显示第三界面;其中,第三界面中包括第一画面,第一画面中包括目标对象;在终端设备接收到针对目标对象的操作时,显示单元,还具体用于显示第二界面。
在一种可能的实现方式中,第二界面中包括:用于结束录制的控件,在终端设备接收到针对用于结束录制的控件的操作时,处理单元,具体用于对第一音频信号进行声源分离,得到N路音频信号。
第三方面,本申请实施例提供一种终端设备,包括处理器和存储器,存储器用于存储代码指令;处理器用于运行代码指令,使得终端设备以执行如第一方面或第一方面的任一种实现方式中描述的方法。
第四方面,本申请实施例提供一种计算机可读存储介质,计算机可读存储介质存储有指令,当指令被执行时,使得计算机执行如第一方面或第一方面的任一种实现方式中描述的方法。
第五方面,一种计算机程序产品,包括计算机程序,当计算机程序被运行时,使得计
算机执行如第一方面或第一方面的任一种实现方式中描述的方法。
应当理解的是,本申请的第二方面至第五方面与本申请的第一方面的技术方案相对应,各方面及对应的可行实施方式所取得的有益效果相似,不再赘述。
图1为本申请实施例提供的一种录像场景的界面示意图;
图2为本申请实施例提供的一种终端设备的结构示意图;
图3为本申请实施例提供的另一种录像场景的界面示意图;
图4为本申请实施例提供的一种音频处理方法的流程示意图;
图5为本申请实施例提供的一种空间角度划分的示意图;
图6为本申请实施例提供的一种声源准确性分析的流程示意图;
图7为本申请实施例提供的一种选中目标对象的界面示意图;
图8为本申请实施例提供的一种音频处理方法的流程示意图;
图9为本申请实施例提供的另一种声源准确性分析的流程示意图;
图10为本申请实施例提供的一种音频处理装置的结构示意图;
图11为本申请实施例提供的另一种终端设备的硬件结构示意图。
下面对本申请实施例中所描述的词汇进行说明。可以理解,该说明是为更加清楚的解释本申请实施例,并不必然构成对本申请实施例的限定。
(1)盲源分离(blind source separation,BSS)
本申请实施例中,盲源分离也可以称为盲信号分离、或声源分离,可以为在不知道源信号以及信号混合参数的情况下,从接收到的混合信号中分离出源信号的方法。
其中,该声源分离的方法可以包括:独立向量分析(Independent vector analysis,IVA)方法、独立分量分析(Independent Component Analysis,ICA)方法、非负矩阵分解(non-negative matrix factorization,NMF)方法、或者多通道非负矩阵分解方法(multichannel NMF,MNMF)等。
(2)抑制
抑制是指将音频信号的能量减小使得该音频信号听起来变小甚至听不见。抑制程度用于描述对音频信号进行减小的程度。抑制程度越大,则音频信号的能量减小越多。
其中,抑制程度可以用增益系数表示。当增益系数为0.0时,则表示完全除去该音频信号;当增益系数为1.0时,表示不进行抑制;当增益系数大于1.0时,则表示增强音频信号。其中,越接近于0.0则抑制程度越大,越接近与1.0则抑制程度越小。
(3)画中画
画中画为将终端设备设备的显示屏分为一大一小的两个显示界面,较小的显示界面包含于较大的显示界面中。
其中,较大的显示区域一般铺满设备的屏幕,较小显示区域中的图像可以覆盖于较大的显示区域中的图像之上。在一些情况中,较小的显示区域还支持进行缩放,其在终端设备屏幕中的位置还可以变换。
为了便于清楚描述本申请实施例的技术方案,在本申请的实施例中,采用了“第一”、“第二”等字样对功能和作用基本相同的相同项或相似项进行区分。例如,第一值和第二值仅仅是为了区分不同的值,并不对其先后顺序进行限定。本领域技术人员可以理解“第一”、“第二”等字样并不对数量和执行次序进行限定,并且“第一”、“第二”等字样也并不限定一定不同。
需要说明的是,本申请中,“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其他实施例或设计方案更优选或更具优势。确切而言,使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
本申请中,“至少一个”是指一个或者多个,“多个”是指两个或两个以上。“和/或”,描述关联对象的关联关系,表示可以存在三种关系,例如,A和/或B,可以表示:单独存在A,同时存在A和B,单独存在B的情况,其中A,B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项(个)”或其类似表达,是指的这些项中的任意组合,包括单项(个)或复数项(个)的任意组合。例如,a,b,或c中的至少一项(个),可以表示:a,b,c,a和b,a和c,b和c,或a、b和c,其中a,b,c可以是单个,也可以是多个。
示例性的,图1为本申请实施例提供的一种录像场景的界面示意图。如图1所示,该录像场景可以为一种会议场景。例如,当用户利用终端设备对会议场景进行录像时,则可以拍摄得到如图1所示的界面。在图1对应的实施例中,以终端设备为手机为例进行示例说明,该示例并不构成对本申请实施例的限定。
当终端设备接收到用户打开相机应用,并触发进入录像功能的操作时,终端设备可以显示如图1所示的界面。如图1所示,该界面中可以包括:预览画面100、用于开启或关闭闪光灯控件、用于对相机应用进行设置的设置控件,用于调整拍摄倍数的控件、用于翻转摄像头的控件、以及用于打开图库的控件、录像功能对应的用于开启录制的控件106、以及相机应用的一级菜单中的多个功能控件。其中,该相机应用的一级菜单中的多个功能控件中可以包括:人像功能控件、拍照功能控件、录像功能控件、短视频功能控件、以及用于开启更多功能的控件等。
如图1所示的预览画面100中可以包括:围绕在会议桌周围的目标101、目标102、目标103、目标104以及目标105。在如图1所示的会议场景,终端设备可以对该会议场景进行视频录制,并通过音频处理方法提取给该场景中的某个目标的声源。由于该场景中可能存在声源较多、和/或声源的能量时大时小等情况,可能对终端设备提取某个目标的声源时的声源分离的准确性带来影响,进而影响声源分离处理后的其他音频处理流程。
有鉴于此,本申请实施例提供一种音频处理方法,终端设备显示第一界面;第一界面中包括:用于开启录制的控件;在终端设备接收到针对用于开启录制的控件的操作时,终端设备显示第二界面,并获取第一音频信号;其中,第二界面中包括第一画面以及第二画面,第二画面覆盖在第一画面之上,第一画面中包括第二画面中的内容;第二画面中包括目标对象;终端设备对第一音频信号进行声源分离,得到N路音频信号;N为大于或等于2的整数;终端设备生成第一视频以及第二视频。其中,第一视频是基于第一音频信号以及第一画面得到的;在N路音频信号满足预设条件的情况下,第二视频是基于N路音频信号以
及第二画面得到的。这样,终端设备可以将声源分离较为准确的N路音频信号用于第二视频的生成。第二视频对应的第二音频信号是对N路音频信号中的目标音频信号、和/或第N路音频信号中的除目标音频信号以外的信号进行处理后得到的,使得终端设备可以通过声源分离准确的N路音频信号中的不同音频信号的处理,增强第二视频中目标对象的声源的立体感。
可以理解的是,本申请实施例提供的音频处理方法,不仅可以应用于如图1所示的录像场景,还可以用于如直播场景等涉及声源分离的场景中,本申请实施例中对此不做限定。
可以理解的是,上述终端设备也可以称为终端,(terminal)、用户设备(user equipment,UE)、移动台(mobile station,MS)、移动终端(mobile terminal,MT)等。终端设备可以为拥有MIC的手机(mobile phone)、智能电视、穿戴式设备、平板电脑(Pad)、带无线收发功能的电脑、虚拟现实(virtual reality,VR)终端设备、增强现实(augmented reality,AR)终端设备、工业控制(industrial control)中的无线终端、无人驾驶(self-driving)中的无线终端、远程手术(remote medical surgery)中的无线终端、智能电网(smart grid)中的无线终端、运输安全(transportation safety)中的无线终端、智慧城市(smart city)中的无线终端、智慧家庭(smart home)中的无线终端等等。本申请的实施例对终端设备所采用的具体技术和具体设备形态不做限定。
因此,为了能够更好地理解本申请实施例,下面对本申请实施例的终端设备的结构进行介绍。示例性的,图2为本申请实施例提供的一种终端设备的结构示意图。
终端设备可以包括处理器110,外部存储器接口120,内部存储器121,通用串行总线(universal serial bus,USB)接口130,充电管理模块140,电源管理模块141,天线1,天线2,移动通信模块150,无线通信模块160,音频模块170,扬声器170A,受话器170B,麦克风170C,耳机接口170D,传感器模块180,按键190,指示器192,摄像头193,以及显示屏194等。
可以理解的是,本申请实施例示意的结构并不构成对终端设备的具体限定。在本申请另一些实施例中,终端设备可以包括比图示更多或更少的部件,或者组合某些部件,或者拆分某些部件,或者不同的部件布置。图示的部件可以以硬件,软件或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元。其中,不同的处理单元可以是独立的器件,也可以集成在一个或多个处理器中。处理器110中还可以设置存储器,用于存储指令和数据。
USB接口130是符合USB标准规范的接口,具体可以是Mini USB接口,Micro USB接口,USB Type C接口等。USB接口130可以用于连接充电器为终端设备充电,也可以用于终端设备与***设备之间传输数据。也可以用于连接耳机,通过耳机播放音频。该接口还可以用于连接其他终端设备,例如AR设备等。
充电管理模块140用于从充电器接收充电输入。其中,充电器可以是无线充电器,也可以是有线充电器。电源管理模块141用于连接充电管理模块140与处理器110。
终端设备的无线通信功能可以通过天线1,天线2,移动通信模块150,无线通信模块160,调制解调处理器以及基带处理器等实现。
天线1和天线2用于发射和接收电磁波信号。终端设备中的天线可用于覆盖单个或多个通信频带。不同的天线还可以复用,以提高天线的利用率。
移动通信模块150可以提供应用在终端设备上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器,开关,功率放大器,低噪声放大器(low noise amplifier,LNA)等。移动通信模块150可以由天线1接收电磁波,并对接收的电磁波进行滤波,放大等处理,传送至调制解调处理器进行解调。
无线通信模块160可以提供应用在终端设备上的包括无线局域网(wirelesslocal area networks,WLAN)(如无线保真(wireless fidelity,Wi-Fi)网络),蓝牙(bluetooth,BT),全球导航卫星***(global navigation satellite system,GNSS),调频(frequency modulation,FM)等无线通信的解决方案。
终端设备通过GPU,显示屏194,以及应用处理器等实现显示功能。GPU为图像处理的微处理器,连接显示屏194和应用处理器。GPU用于执行数学和几何计算,用于图形渲染。
显示屏194用于显示图像,视频等。显示屏194包括显示面板。在一些实施例中,终端设备可以包括1个或N个显示屏194,N为大于1的正整数。
终端设备可以通过ISP,摄像头193,视频编解码器,GPU,显示屏194以及应用处理器等实现拍摄功能。
摄像头193用于捕获静态图像或视频。在一些实施例中,终端设备可以包括1个或N个摄像头193,N为大于1的正整数。
外部存储器接口120可以用于连接外部存储卡,例如Micro SD卡,实现扩展终端设备的存储能力。外部存储卡通过外部存储器接口120与处理器110通信,实现数据存储功能。例如将音乐,视频等文件保存在外部存储卡中。
内部存储器121可以用于存储计算机可执行程序代码,可执行程序代码包括指令。内部存储器121可以包括存储程序区和存储数据区。
终端设备可以通过音频模块170,扬声器170A,受话器170B,麦克风170C,耳机接口170D,以及应用处理器等实现音频功能。例如音乐播放,录音等。
音频模块170用于将数字音频信息转换成模拟音频信号输出,也用于将模拟音频输入转换为数字音频信号。扬声器170A,也称“喇叭”,用于将音频电信号转换为声音信号。终端设备可以通过扬声器170A收听音乐,或收听免提通话。受话器170B,也称“听筒”,用于将音频电信号转换成声音信号。当终端设备接听电话或语音信息时,可以通过将受话器170B靠近人耳接听语音。耳机接口170D用于连接有线耳机。
麦克风(microphone,MIC)170C,也称“话筒”,“传声器”,用于将声音信号转换为电信号。本申请实施例中,终端设备可以基于麦克风170C接收声音信号,并将声音信号转换为可以进行后续处理的电信号,该终端设备可以拥有至少两个麦克风170C。
本申请实施例中,麦克风170C可以获取以终端设备为中心的360°范围内的音频信号。
传感器模块180可以包括下述一种或多种传感器,例如:压力传感器,陀螺仪传感器,气压传感器,磁传感器,加速度传感器,距离传感器,接近光传感器,指纹传感器,温度传感器,触摸传感器,环境光传感器,或骨传导传感器等(图2中未示出)。
按键190包括开机键,音量键等。按键190可以是机械按键。也可以是触摸式按键。
终端设备可以接收按键输入,产生与终端设备的用户设置以及功能控制有关的键信号输入。指示器192可以是指示灯,可以用于指示充电状态,电量变化,也可以用于指示消息,未接来电,通知等。
终端设备的软件***可以采用分层架构,事件驱动架构,微核架构,微服务架构,或云架构等,在此不再赘述。
下面以具体地实施例对本申请的技术方案以及本申请的技术方案如何解决上述技术问题进行详细说明。下面这几个具体的实施例可以独立实现,也可以相互结合,对于相同或相似的概念或过程可能在某些实施例中不再赘述。
本申请实施例中,终端设备可以支持在画中画模式下的音频以及视频的处理方法。示例性的,图3为本申请实施例提供的另一种录像场景的界面示意图。
当终端设备接收到用户针对图1所示的界面中的用于开启录制的控件106的触发操作时,终端设备可以显示如图3中的a所示的界面。如图3中的a所示的界面,该界面中可以包括:用于暂停录制的控件、用于结束录制的控件301、用于在视频录制过程中拍摄画面的控件、用于放大画面的控件、用于缩小画面的控件、以及用于指示视频录制时长的标识等内容。其中,触发操作可以包括:单击操作、双击操作、或长按操作等,本申请实施例中对此不做限定。
示例性的,在终端设备支持画中画模式的情况下,当终端设备接收到用户针对图3中的a所示的界面中的目标103的触发操作时,终端设备提取包含目标103的预览画面,并显示如图3中的b所示的界面。如图3中的b所示的界面,该界面中可以包括画中画302,该画中画302中可以包括目标103。
可以理解的是,在如图3中的b所示的界面中,预览画面100的上方可以叠加画中画302,且该画中画302可以位于预览画面100的左上角,或者也可以位于其他位置上,本申请实施例中对该画中画302所在的位置不做限定。
可能的实现方式中,在终端设备支持画中画模式的情况下,终端设备可以对当前场景中能量最高的声源进行检测。当能量最高的声源为目标103的声源时,终端设备可以提取包含目标103的预览画面,并显示如图3中的b所示的界面。
在如图3中的b所示的界面中,当终端设备接收到用户针对用于结束录制的控件301的触发操作时,终端设备可以基于麦克风获取的音频信号以及预览画面100得到预览画面100对应的视频;并且,对基于麦克风获取的音频信号进行声源分离,得到N路音频信号,对N路音频信号中的目标103所对应的角度处的音频信号、以及N路音频信号中除目标103以外的声源进行处理,进而基于经过处理的N路音频信号以及画中画302得到画中画302对应的视频。
在图3对应的实施例的基础上,本申请实施例提供两种音频处理方法的方法,例如:方法一、基于声源方位估计的音频处理方法;以及方法二、基于声源相关性判别的音频处理方法。
方法一、基于声源方位估计的音频处理方法。
示例性的,图4为本申请实施例提供的一种音频处理方法的流程示意图。如图4所示,
该音频处理方法可以包括如下步骤:
S401、当终端设备接收到用户开启录像的操作时,终端设备利用M个MIC获取初始音频信号。
本申请实施例中,M大于或等于2,该M个MIC中可以包括:终端设备中的至少两个MIC;或者,在终端设备与真无线(true wireless stereo,TWS)耳机建立连接的情况下,该M个MIC中也可以包括:TWS耳机中的至少两个MIC,和/或,终端设备中的至少两个MIC等,本申请实施例中对采集初始音频信号的MIC的具体形态不做限定。
本申请实施例中,该用户开启录像的操作可以为:用户针对用于开启录制的控件的触发操作,语音操作,或其他手势操作等,本申请实施例中对此不做限定。
示例性的,在如图1所示的界面中,当终端设备接收到用户针对用于开启录制的控件106的触发操作时,终端设备可以利用摄像头获取图像信号显示如图3中的a所示的录像界面,并利用M个MIC获取初始音频信号。
S402、当终端设备接收到用户结束录像的操作时,终端设备对初始音频信号进行声源分离,得到解混矩阵以及N路音频信号。
本申请实施例汇总,该结束录制的操作可以为:用户针对用于结束录制的控件的触发操作,语音操作,或其他手势操作等,本申请实施例中对此不做限定。
示例性的,在如图3中的b所示的界面中,当终端设备接收到用户针对用于结束录制的控件301的触发操作时,终端设备可以对初始音频信号进行声源分离,得到解混矩阵以及N路音频信号。
本申请实施例中,终端设备可以对基于S401获取的时域的初始音频信号进行傅里叶变换,转化为频域的初始音频信号;进而,对频域的初始音频信号进行声源分离,得到解混矩阵以及N路音频信号。其中,该声源分离方法可以为IVA等。
示例性的,若当前场景中声源的个数为N,声源的时域序列可以表示为:s1(t),s2(t),…,sN(t);s表示源信号(source),t表示时间的采样序列;假设有M个MIC在采集初始音频信号,则终端设备得到各声源所对应的音频信号可以分别表示为x1(t),x2(t),…,xM(t)。
可以理解的是,声波从声源到MIC需要经过一个传输路径(比如时间延迟、反射、以及不同声源进入到同一MIC产生的混合等),因此MIC采集的音频信号xm(t)与源信号sn(t)的关系在时域的表述如下:
其中,sn(t)为第n个声源,xm(t)为第m个MIC拾取的音频信号,τ为时间延迟,L为最大时间延迟,h可以理解为源信号sn(t)与MIC采集的信号xm(t)之间的传输路径。
进一步的,终端设备可以对公式(1)进行傅里叶变换,则可以得到源信号s和MIC采集的信号x的频域关系:
x(ω,t)=A(ω,t)s(ω,t) 公式(2)
x(ω,t)=A(ω,t)s(ω,t) 公式(2)
其中,若N为声源个数,M为MIC的数量,则x为长度为M的向量,s为长度为N的向量,ω为频率,t为帧数,A为一个M行N列的矩阵,或可以理解为源信号s与MIC采集的信号x之间的传输路径。
公式(2)也可以为:
W(ω,t)x(ω,t)=s(ω,t) 公式(3)
W(ω,t)x(ω,t)=s(ω,t) 公式(3)
其中,W可以为一个N行M列的矩阵,或也可以称为解混矩阵;该公式(3)中的其他参数可以参见公式(2)中的描述,在此不再赘述。
可以理解的是,在MIC数量M大于实际声源数量N时,M-N路声源可以为噪声;在MIC数量小于实际声源数量N时,可以按照M个声源进行分离,但分离质量难以保证,因此本申请实施例中终端设备可以在进行声源处理方法的过程中令实际声源数量N与MIC数理M相同。
S403、终端设备获取N路音频信号分别对应的声源角度,得到N个声源角度。
示例性的,终端设备可以获取终端设备周围各预设角度对应的传递函数,并利用解混矩阵中的任一声源对应的列向量与预设角度对应的传递函数,得到该声源在该任一角度处的无量纲值,进而统计终端设备各角度处的无量纲值的最小值作为该声源的方向(或角度)。
可以理解的是,由于不同终端设备中的MIC的布局可能存在差异,使得各预设角度上接收到的音频信号存在差异。因此,可以利用终端设备的各预设角度分别对应的传递函数指示不同角度上接收到的音频信号之间的相对关系。
本申请实施例中,终端设备中可以存储有各预设角度对应的传递函数。示例性的,图5为本申请实施例提供的一种空间角度划分的示意图。在图5对应的实施例中,预设角度可以为0°,10°,20°,…350°。例如,以终端设备为中心,在终端设备周围360°方向上,每10°采集一次测试音频信号,例如在0°,10°,20°,…,350°方向上分别采集一次测试音频信号,进而得到36个方向上对应的36个测试音频信号。
获取终端设备各预设角度对应的传递函数的方法可以为:当终端设备中的MIC数量为M个,对基于M个MIC获取到的时域的测试音频信号进行傅里叶变换,得到对应的频域的测试音频信号:
Y(ω,q)=[X1(ω,q),…,XM(ω,q)] 公式(4)
Y(ω,q)=[X1(ω,q),…,XM(ω,q)] 公式(4)
其中,Y(ω,q)为MIC在q帧、频点ω处的音频信号。
终端设备可以将任一MIC获取的测试音频信号,如第一个MIC(或称为参考MIC)处的测试音频信号X1(ω,q)作为参考音频信号,获取其他MIC采集到的测试音频信号相对该参考MIC处的参考音频信号的关系H(θ,ω)。
Y(ω,q)=[X1(ω,q),…,XM(ω,q)]=H(θ,ω)X1(ω,q) 公式(5)
H(θ,ω)=[1,ρ1,2,…,ρ1,m] 公式(6)
Y(ω,q)=[X1(ω,q),…,XM(ω,q)]=H(θ,ω)X1(ω,q) 公式(5)
H(θ,ω)=[1,ρ1,2,…,ρ1,m] 公式(6)
其中,ρ1,m为第m个MIC的音频信号Xm(ω,q)与参考MIC的音频信号X1(ω,q)之间的传递函数关系。
其中,*表示为共轭计算、E表示为期望计算。
进而得到角度为θ,频点为ω处的传递函数:
H(θ,ω)=[1,ρ1,2,…,ρ1,m],θ=0,10,20,…,350 公式(9)
H(θ,ω)=[1,ρ1,2,…,ρ1,m],θ=0,10,20,…,350 公式(9)
可以理解的是,终端设备可以将获取的各预设角度对应的H(θ,ω)存储在本设备中,使得终端设备可以基于该H(θ,ω)进行后续声源方位的估计。并且,获取H(θ,ω)的设备可以不限于终端设备,例如可以为服务器等,本申请实施例中对此不做限定。
示例性的,利用该H(θ,ω)进行声源方位估计的方法可以为:将任一预设角度对应的传递函数H1(ω)与解混矩阵的某一列wH点乘,得到γ。其中,解混矩阵中的每一列可以对
应于一个声源。
其中,ω2>ω1,且ω1-ω2可以为任一频带范围。
进一步的,在固定声源以及固定方向处获取ω在所有频带范围内的γ的累加值,得到无量纲值vθ;
θ=argminvθ 公式(12)
θ=argminvθ 公式(12)
遍历0-350°中的所有角度对应的vθ,选取vθ最小时对应的角度,作为声源所在方向θ。
可以理解的是,当MIC数量为2时,由于无法区分该两个MIC连线的前向以及后向,因此可以令θ满足:0°≤θ≤180°或者180°≤θ≤360°;当MIC数量大于2时,终端设备对θ则没有要求。
基于此,终端设备可以得到N个音频信号分别对应的N个声源角度。
可能的实现方式中,在如图3中的b所示的界面,终端设备可以基于图像识别确定的目标103所处的角度,以及S404所示的步骤中N个声源角度分别对应的角度,进而确定目标103所处的角度与N个声源角度中的任一角度符合时,目标103的声源。
S404、终端设备基于N个声源角度以及N路音频信号的能量,确定当前场景声源的准确性。
本申请实施例中,当终端设备确定当前场景中的各声源均分离准确时,可以执行S405所示的步骤;当终端设备确定当前场景中的声源部分准确(或理解为部分声源分离准确、部分声源分离不准确)时,可以执行S406所示的步骤;或者,当终端设备确定当前场景中的各声源均分离不准确时,可以执行S406所示的步骤。
终端设备可以基于图6对应的实施例,对N路音频信号中的任一音频信号分离的准确性进行判别,进而输出当前场景声源的准确性情况。
图6为本申请实施例提供的一种声源准确性分析的流程示意图。如图6所示,该声源准确性分析方法可以包括如下步骤:
S601、终端设备计算一时间段内声源角度对应的方差。
其中,该一段时间可以为5秒、10秒或20秒等时间段;该声源角度可以为N个声源角度中的任一声源角度。
可以理解的是,由于声源角度的方差可以反映一段时间内声源方向的变化程度,而通常情况下声源的方向很难在较小的时间内发生剧烈的变化,因此终端设备可以通过声源角度的方差对声源的准确性进行判别。
S602、终端设备判断方差是否大于方差阈值。
其中,当终端设备确定方差大于方差阈值时,终端设备可以执行S603所示的步骤;或者,当终端设备确定方差小于或等于方差阈值时,终端设备可以执行S606所示的步骤。
S603、终端设备计算声源对应的能量。
其中,该声源可以为N路音频信号中的任一音频信号;声源对应的能量可以为N路音频信号中的任一音频信号对应的能量。
可能的实现方式中,终端设备也可以统计该一段时间内声源的能量的均值,基于该能
量均值进行声源的准确性判别。
S604、终端设备判断能量是否大于第一能量阈值。
其中,当终端设备确定能量大于第一能量阈值时,可以执行S605所示的步骤;或者,当终端设备确定能量小于或等于第一能量阈值时,可以执行S606所示的步骤。
可能的实现方式中,当能量大于第一能量阈值,且方差小于方差阈值时,终端设备也可以执行S605所示的步骤。
可以理解的是,当一段时间内某个声源角度对应的方差大于方差阈值,且该声源的能量小于第一能量阈值时,则可以确定该声源可以为散场噪声,散场噪声不影响准确性的判别,因此可以确定该路声源分离准确;而当一段时间内某个声源角度对应的方差小于或等于方差阈值,且该声源的能量大于第一能量阈值时,可以确定声源分离不准确。
可能的实现方式中,终端设备也可以只利用声源的能量,确定声源分离是否准确。
S605、终端设备确定当前声源分离不准确。
S606、终端设备确定当前声源分离准确。
S607、终端设备统计所有声源的准确性情况,并确定当前场景声源的准确性。
其中,当确定场景中的N路音频信号均分离准确时,终端设备确定当前场景声源分离准确,进而执行S405所示的步骤;当确定场景中的N路音频信号中部分信号分离准确以及部分信号分离不准确时,终端设备确定当前场景声源分离部分准确,进而执行S406所示的步骤;或者,当确定场景中的N路音频信号均分离不准确时,终端设备确定当前场景声源分离不准确,进而执行S406所示的步骤。
可以理解的是,终端设备可以基于N路音频信号所在的声源方向以及N路音频信号的能量,对声源分离情况进行准确性分析。
S405、在确定当前场景声源分离准确的情况下,终端设备输出N路音频信号。
可能的实现方式中,如图3中的b所示的界面,当终端设备确定场景声源分离准确的情况下,在生成画中画302对应的视频时,终端设备可以对N路音频信号中的除目标103的声源以外的声源进行抑制(或者对目标103的声源进行增强),使得画中画302对应的视频中可以突出目标103的声音,提高画中画场景中的声音的立体感。
其中,终端设备对场景中除目标103的声源以外的声源进行抑制(或者对目标103的声源进行增强)的方法可以包括:终端设备对目标103的声源进行增强,如将目标103的声源乘以第一增益系数,如第一增益系数可以为1.5等数值;对除目标103的声源以外的声源进行增强,如乘以第二增益系数,如第二增益系数可以为1.2等数值,其中第二增益系数小于第一增益系数。或者,终端设备保持103对应的声源不变的情况下,对除目标103的声源以外的声源进行抑制,如乘以第三增益系数,如第三增益系数可以为0.7等数值。
可以理解的是,本申请实施例中描述的抑制为相对抑制的概念,增强为相对增强的概念,且本申请实施例中对声源进行抑制的方法不做具体限定。
S406、在确定当前场景声源分离部分准确或者声源分离不准确的情况下,终端设备输出声源分离前的音频信号。
其中,该声源分离前的音频信号可以为S402所示的步骤中描述的频域的初始音频信号。
可能的实现方式中,如图3中的b所示的界面,当终端设备确定场景声源分离不准确
的情况下,在生成画中画302对应的视频时,终端设备可以采用声源分离前的音频信号。
可以理解的是,在确定当前场景声源分离部分准确或者声源分离不准确的情况下,终端设备输出的音频信号可以不限于声源分离前的音频信号,其输出的音频信号也可以为对N路音频信号进行其他算法处理后输出的信号,本申请实施例中对此不做限定。
基于此,终端设备可以基于N路音频信号所在方向以及N路音频信号的能量,对声源分离后得到的N路音频信号进行准确性分析,使得终端设备可以基于该准确性分析结果进行后续音频信号的处理。
在上述图4对应的实施例的基础上,可能的实现方式中,S401之后,当终端设备接收到用户在录像界面中选中目标对象的操作时,终端设备可以显示包含目标对象的画中画,并在生成画中画对应的视频时对目标对象对应的声源进行增强(或者对除目标对象以外的声源进行抑制)。
示例性的,图7为本申请实施例提供的一种选中目标对象的界面示意图。在图7对应的实施例中,以目标对象为目标103为例进行示例说明。在图7对应的实施例中,以终端设备为中心建立坐标系,终端设备可以识别周围360°的音频信号,由于终端设备无法区分终端设备的前向以及终端设备的后向,因此以终端设备的前向为0-180°为例,对终端设备识别的声源进行说明。
当终端设备接收到针对目标103的触发操作时,终端设备可以显示如图7中的a所示的包含画中画302的界面,并基于图像识别(例如人脸识别)确定目标103所处的角度(或者坐标)。进一步的,在终端设备接收到用户结束录像的操作时,终端设备可以基于S402-S406所示的步骤,对当前场景的声源准确性进行判别,输出相应的音频信号,并生成得到预览画面100对应的视频、以及画中画302对应的视频。
在预览画面100对应的视频中,终端设备可以在基于S402以及S406所示的步骤对声源准确性进行判别后,输出相应的音频信号,并结合拍摄得到的预览画面100对应的多帧图像得到预览画面100对应的视频。
在画中画302对应的视频中,终端设备可以在基于S402以及S406所示的步骤对声源准确性进行判别后,结合图像识别中得到的目标103所处的角度,以及S404所示的步骤中声源对应的声源角度,确定目标103的声源。进而,终端设备对场景中除目标103的声源以外的声源进行抑制,并结合拍摄得到的包含目标103的多帧图像得到画中画302对应的视频。
可能的实现方式中,在终端设备接收到用户选中目标对象的操作的情况下,当终端设备结合图像识别中得到的目标103所处的角度,但未从S404所示的步骤中检测到声源对应的声源角度时,终端设备可以确定目标103可能未发出声音,因此声源分离过程中难以分离出目标103的声源。此时,终端设备可以将目标103周围的声源确定为目标103的声源,例如将与目标103之间的角度最小处的声源作为目标103的声源。进而,终端设备可以对场景中对目标103周围的声源进行增强。
基于此,终端设备不仅基于用户选中目标对象的操作,生成包含目标对象的视频,还可以在该包含目标对象的视频中,对除目标对象以外的声源进行抑制,增强目标对象的声音的立体感。
在如图7中的a对应的实施例中,终端设备选中目标对象的基础上,也可以支持目标对象的实时追踪、目标对象的自动切换等功能。
一种实现中,终端设备可以支持目标对象的实时追踪。例如,在用户选中目标对象后,终端设备可以实时追踪目标对象的位置。
示例性的,在图7中的a所示的界面中,当目标103向左侧移动方向,例如由图7中的a所示的位置移动至如图7中的b所示的位置时,终端设备可以基于图像识别,实时追踪目标103,确定目标103移动后的所处的角度(或坐标)。
可以理解的是,在后续画中画302对应的视频中,终端设备可以继续实现对实时追踪的目标103的声源进行抑制。
进一步的,在终端设备可以实时追踪目标对象的位置的情况下,当终端设备接收到用户切换目标对象的操作时,终端设备可以切换追踪的目标对象。例如,在如图7中的b所示的界面中,终端设备追踪目标103的情况下,当终端设备接收到用户针对目标104的触发操作时,终端设备可以停止追踪目标103并开始追踪目标104,并显示如图7中的c所示的界面。如图7中的c所示的界面,终端设备可以在画中画302中显示目标104对应的画面。
可以理解的是,在后续画中画302对应的视频中,终端设备可以在用户切换目标104的时刻对目标104的声源进行增强。其中,该目标104的声源角度的确定方法、以及对目标104的声源进行增强的方法均可以参见目标103对应的描述,在此不再赘述。
另一种实现中,终端设备可以支持目标对象的自动切换。例如,在用户选中目标对象后,终端设备可以基于场景中声源的能量,追踪能量最高的声源的位置。
示例性的,在图7中的b所示的界面中,在用户选中目标103的情况下,当终端设备检测到预览画面100中目标104的能量最高时,终端设备可以显示如图7中的c所示的界面。如图7中的c所示的界面,终端设备可以在画中画302中显示目标104对应的画面。
可以理解的是,在后续画中画302对应的视频中,终端设备可以在检测到目标104的能量最高时,对目标104的声源进行增强。
可能的实现方式中,在终端设备基于场景中声源的能量,追踪能量最高的声源的情况下,当存在多个能量相同的声源时,终端设备自动追踪位于场景中间(如角度位于90°附近)位置的目标、或者终端设备自动追踪距离用户选中次数最多的目标、或者终端设备自动追踪距离上一次选中的目标对象的距离最近处的目标等。
可能的实现方式中,在终端设备基于场景中声源的能量,追踪能量最高的声源的情况下,当终端设备接收到用户切换目标对象的操作时,终端设备可以停止自动追踪能量最高的声源,并追踪用户选中的目标对象。
方法二、基于声源相关性判别的音频处理方法。
示例性的,图8为本申请实施例提供的一种音频处理方法的流程示意图。
S801、当终端设备接收到用户开启录像的操作时,终端设备利用M个MIC获取初始音频信号。
S802、终端设备对初始音频信号进行声源分离,得到N路音频信号。
其中,该S801-S802所示的步骤可以参见图4对应的实施例中的S401-S402所示的步骤,在此不再赘述。
S803、终端设备基于N路音频信号的能量以及任一路音频信号与N路音频信号中的其他音频信号的相关性,确定当前场景声源的准确性。
其中,该相关性用于指示两路信号之间的相似程度。可以理解的是,由于N路音频信号中的任两路信号之间一定是不相关的,因此可以通过相关性确定声源分离的准确性。
示例性的,终端设备可以基于如图9对应的实施例,对N路音频信号中的任一音频信号的准确性进行判别,进而输出当前场景声源的准确性情况。示例性的,图9为本申请实施例提供的另一种声源准确性分析的流程示意图。
S901、终端设备确定音频信号对应的能量。
其中,该音频信号可以为N路音频信号中的任一音频信号。
S902、终端设备判断能量是否大于第二能量阈值。
其中,第二能量阈值与第一能量阈值可以相同;在确定能量大于第二能量阈值时,终端设备可以执行S903所示的步骤;或者,在确定能量小于或等于第二能量阈值时,可以认为当前音频信号为散场噪声,则可以不参与后续音频之间相关性的计算,例如执行S908。
S903、终端设备确定N路音频信号中的任一路音频信号与N路音频信号中的其他音频信号的相关性。
具体的,相关性计算公式可以为:
其中,γ表示相关性,a表示N路音频信号中的任一路音频信号,b表示N路音频信号中的其他音频信号;t表示傅里叶变换的帧数,相当于时间,ω表示频率。
其中,*表示共轭计算,E表示数学期望。
可以理解的是,由于声源分离后的N路音频信号彼此独立,而彼此独立的音频信号之间是不相关的,因此可以通过任一路音频信号与N路音频信号中的其他音频信号的相关性计算,确定音频信号分离的准确性。
S904、终端设备判断任一路音频信号与N路音频信号中的其他音频信号的相关性是否小于相关性阈值。
当终端设备确定任一路音频信号与N路音频信号中的其他音频信号的相关性均小于相关性阈值时,终端设备可以执行S905所示的步骤;或者,当终端设备确定任一路音频信号与N路音频信号中的其他音频信号的相关性不满足均小于相关性阈值时,终端设备可以执行S906所示的步骤。
其中,该任一路音频信号与N路音频信号中的其他音频信号的相关性不满足均小于相关性阈值可以理解为:任一路音频信号与N路音频信号中的其他音频信号的相关性均大于或等于相关性阈值;或者,存在任一路音频信号与N路音频信号中的部分分离的相关性小
于相关性阈值,以及任一路音频信号与N路音频信号中的部分分离的相关性大于或等于相关性阈值的情况。
可能的实现方式中,终端设备也可以只利用任一路音频信号与N路音频信号中的其他音频信号的相关性,确定声源分离是否准确。
S905、终端设备确定当前声源分离准确。
S906、终端设备确定当前声源分离不准确。
S907、终端设备统计所有声源的准确性情况,并确定当前场景声源的准确性。
其中,当确定场景中的N路音频信号均分离准确时,终端设备确定当前场景声源分离准确,进而执行S804所示的步骤;当确定场景中的N路音频信号中部分信号分离准确以及部分信号分离不准确时,终端设备确定当前场景声源分离部分准确,进而执行S805所示的步骤;或者,当确定场景中的N路音频信号均分离不准确时,终端设备确定当前场景声源分离不准确,进而执行S805所示的步骤。
S908、终端设备结束当前声源分离准确性的判断。
可以理解的是,终端设备可以基于N路音频信号的能量以及任一路音频信号与N路音频信号中的其他音频信号的相关性,对音频信号进行准确性分析。
S804、在确定当前场景声源分离准确的情况下,终端设备输出N路音频信号。
S805、在确定当前场景声源分离部分准确或者声源分离不准确的情况下,终端设备输出声源分离前的音频信号。
其中,该S804-S805所示的步骤可以参见S405-S406所示的步骤中的描述,在此不再赘述。
基于此,终端设备可以基于N路音频信号的能量、以及任一路音频信号与N路音频信号中的其他音频信号的相关性,对声源分离后得到的音频信号进行准确性分析,使得终端设备可以基于该准确性分析结果进行后续音频信号的处理。
在上述图4-图8对应的实施例的基础上,可能的实现方式中,终端设备也可以基于方法一以及方法二进行声源分离的准确性判断,例如在终端设备确定当前场景下的声源满足方法一中的准确性判别、以及方法二中的准确性判别的情况下,终端设备可以确定当前场景声源分离准确,并输出相应的音频信号。
可以理解的是,本申请实施例提供的界面仅作为一种示例,并不能构成对本申请实施例的限定。
上面结合图3-图9,对本申请实施例提供的方法进行了说明,下面对本申请实施例提供的执行上述方法的装置进行描述。如图10所示,图10为本申请实施例提供的一种音频处理装置的结构示意图,该音频处理装置可以是本申请实施例中的终端设备,也可以是终端设备内的芯片或芯片***。
如图10所示,音频处理装置1000可以用于通信设备、电路、硬件组件或者芯片中,该音频处理装置包括:显示单元1001、以及处理单元1002。其中,显示单元1001用于支持音频处理装置1000执行的显示的步骤;处理单元1002用于支持音频处理装置1000执行信息处理的步骤。
本申请实施例提供一种音频处理装置1000,装置包括:显示单元1001以及处理
单元1002;显示单元1001,用于显示第一界面;第一界面中包括:用于开启录制的控件;在终端设备接收到针对用于开启录制的控件的操作时,显示单元1001,还用于终端设备显示第二界面,并获取第一音频信号;其中,第二界面中包括第一画面以及第二画面,第二画面覆盖在第一画面之上,第一画面中包括第二画面中的内容;第二画面中包括目标对象;处理单元1002,用于对第一音频信号进行声源分离,得到N路音频信号;N为大于或等于2的整数;处理单元1002,还用于生成第一视频以及第二视频;其中,第一视频是基于第一音频信号以及第一画面得到的;在N路音频信号满足预设条件的情况下,第二视频是基于N路音频信号以及第二画面得到的;第二视频对应的第二音频信号是对N路音频信号中的目标音频信号、和/或N路音频信号中的除目标音频信号以外的信号进行处理后得到的;目标音频信号为目标对象的音频信号。
在一种可能的实现方式中,在N路音频信号不满足预设条件的情况下,第二视频是基于第一音频信号以及第二画面得到的。
在一种可能的实现方式中,N路音频信号不满足预设条件包括:N路音频信号中的任一音频信号的能量大于能量阈值,且任一音频信号的角度在时间阈值内对应的角度方差大于方差阈值;和/或,任一音频信号的能量大于能量阈值,且任一音频信号与N路音频信号中的其他音频信号的相关性大于或等于相关性阈值。
在一种可能的实现方式中,任一音频信号的角度是基于解混矩阵中的任一音频信号对应的列数据、以及终端设备在各预设角度处的传递函数得到的;其中,解混矩阵是基于终端设备对第一音频信号进行声源分离时得到的。
在一种可能的实现方式中,当终端设备的麦克风数量为2时,预设角度所在的范围满足:0°-180°,或者180°-360°。
在一种可能的实现方式中,N路音频信号满足预设条件,包括:任一音频信号的能量大于能量阈值,且任一音频信号的角度在时间阈值对应的角度方差小于或等于方差阈值;和/或,任一音频信号的能量大于能量阈值,且任一音频信号与其他音频信号的相关性均小于相关性阈值。
在一种可能的实现方式中,在终端设备接收到针对用于开启录制的控件的操作时,终端设备显示单元1001,具体用于显示第三界面;其中,第三界面中包括第一画面,第一画面中包括目标对象;在终端设备接收到针对目标对象的操作时,显示单元1001,还具体用于显示第二界面。
在一种可能的实现方式中,第二界面中包括:用于结束录制的控件,在终端设备接收到针对用于结束录制的控件的操作时,处理单元1002,具体用于对第一音频信号进行声源分离,得到N路音频信号。
可能的实现方式中,该音频处理装置1000中也可以包括通信单元1003。具体的,通信单元用于支持音频处理装置1000执行数据的发送以及数据的接收的步骤。其中,该通信单元1003可以是输入或者输出接口、管脚或者电路等。
可能的实施例中,音频处理装置还可以包括:存储单元1004。处理单元1002、存储单元1004通过线路相连。存储单元1004可以包括一个或者多个存储器,存储器可以是一个或者多个设备、电路中用于存储程序或者数据的器件。存储单元1004可以独立存在,通过通信线路与音频处理装置具有的处理单元1002相连。存储单元1004也可以和处理单
元1002集成在一起。
存储单元1004可以存储终端设备中的方法的计算机执行指令,以使处理单元1002执行上述实施例中的方法。存储单元1004可以是寄存器、缓存或者RAM等,存储单元1004可以和处理单元1002集成在一起。存储单元1004可以是只读存储器(read-only memory,ROM)或者可存储静态信息和指令的其他类型的静态存储设备,存储单元1004可以与处理单元1002相独立。
图11为本申请实施例提供的另一种终端设备的硬件结构示意图,如图11所示,该终端设备包括处理器1101,通信线路1104以及至少一个通信接口(图11中示例性的以通信接口1103为例进行说明)。
处理器1101可以是一个通用中央处理器(central processing unit,CPU),微处理器,特定应用集成电路(application-specific integrated circuit,ASIC),或一个或多个用于控制本申请方案程序执行的集成电路。
通信线路1104可包括在上述组件之间传送信息的电路。
通信接口1103,使用任何收发器一类的装置,用于与其他设备或通信网络通信,如以太网,无线局域网(wireless local area networks,WLAN)等。
可能的,该终端设备还可以包括存储器1102。
存储器1102可以是只读存储器(read-only memory,ROM)或可存储静态信息和指令的其他类型的静态存储设备,随机存取存储器(random access memory,RAM)或者可存储信息和指令的其他类型的动态存储设备,也可以是电可擦可编程只读存储器(electrically erasable programmable read-only memory,EEPROM)、只读光盘(compact disc read-only memory,CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质,但不限于此。存储器可以是独立存在,通过通信线路1104与处理器相连接。存储器也可以和处理器集成在一起。
其中,存储器1102用于存储执行本申请方案的计算机执行指令,并由处理器1101来控制执行。处理器1101用于执行存储器1102中存储的计算机执行指令,从而实现本申请实施例所提供的音频处理方法。
可能的,本申请实施例中的计算机执行指令也可以称之为应用程序代码,本申请实施例对此不作具体限定。
在具体实现中,作为一种实施例,处理器1101可以包括一个或多个CPU,例如图11中的CPU0和CPU1。
在具体实现中,作为一种实施例,终端设备可以包括多个处理器,例如图11中的处理器1101和处理器1105。这些处理器中的每一个可以是一个单核(single-CPU)处理器,也可以是一个多核(multi-CPU)处理器。这里的处理器可以指一个或多个设备、电路、和/或用于处理数据(例如计算机程序指令)的处理核。
计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行计算机程序指令时,全部或部分地产生按照本申请实施例的流程或功能。计算机可以是通用计算机、专用计算机、计算机网络或者其他可编程装置。计算机指令可以存储在计算机可读存储介质中,
或者从一个计算机可读存储介质向另一计算机可读存储介质传输,例如,计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(digital subscriber line,DSL)或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。计算机可读存储介质可以是计算机能够存储的任何可用介质或者是包括一个或多个可用介质集成的服务器、数据中心等数据存储设备。例如,可用介质可以包括磁性介质(例如,软盘、硬盘或磁带)、光介质(例如,数字通用光盘(digital versatile disc,DVD))、或者半导体介质(例如,固态硬盘(solid state disk,SSD))等。
本申请实施例还提供了一种计算机可读存储介质。上述实施例中描述的方法可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。计算机可读介质可以包括计算机存储介质和通信介质,还可以包括任何可以将计算机程序从一个地方传送到另一个地方的介质。存储介质可以是可由计算机访问的任何目标介质。
作为一种可能的设计,计算机可读介质可以包括紧凑型光盘只读储存器(compact disc read-only memory,CD-ROM)、RAM、ROM、EEPROM或其它光盘存储器;计算机可读介质可以包括磁盘存储器或其它磁盘存储设备。而且,任何连接线也可以被适当地称为计算机可读介质。例如,如果使用同轴电缆,光纤电缆,双绞线,DSL或无线技术(如红外,无线电和微波)从网站,服务器或其它远程源传输软件,则同轴电缆,光纤电缆,双绞线,DSL或诸如红外,无线电和微波之类的无线技术包括在介质的定义中。如本文所使用的磁盘和光盘包括光盘(CD),激光盘,光盘,数字通用光盘(digital versatile disc,DVD),软盘和蓝光盘,其中磁盘通常以磁性方式再现数据,而光盘利用激光光学地再现数据。
上述的组合也应包括在计算机可读介质的范围内。以上,仅为本发明的具体实施方式,但本发明的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本发明揭露的技术范围内,可轻易想到变化或替换,都应涵盖在本发明的保护范围之内。因此,本发明的保护范围应以权利要求的保护范围为准。
Claims (11)
- 一种音频处理方法,其特征在于,所述方法包括:终端设备显示第一界面;所述第一界面中包括:用于开启录制的控件;在所述终端设备接收到针对所述用于开启录制的控件的操作时,所述终端设备显示第二界面,并获取第一音频信号;其中,所述第二界面中包括第一画面以及第二画面,所述第二画面覆盖在所述第一画面之上,所述第一画面中包括所述第二画面中的内容;所述第二画面中包括目标对象;所述终端设备对所述第一音频信号进行声源分离,得到N路音频信号;N为大于或等于2的整数;所述终端设备生成第一视频以及第二视频;其中,所述第一视频是基于所述第一音频信号以及所述第一画面得到的;在所述N路音频信号满足预设条件的情况下,所述第二视频是基于所述N路音频信号以及所述第二画面得到的;所述第二视频对应的第二音频信号是对所述N路音频信号中的目标音频信号、和/或所述N路音频信号中的除所述目标音频信号以外的信号进行处理后得到的;所述目标音频信号为所述目标对象的音频信号。
- 根据权利要求1所述的方法,其特征在于,在所述N路音频信号不满足所述预设条件的情况下,所述第二视频是基于所述第一音频信号以及所述第二画面得到的。
- 根据权利要求2所述的方法,其特征在于,所述N路音频信号不满足所述预设条件包括:所述N路音频信号中的任一音频信号的能量大于能量阈值,且所述任一音频信号的角度在时间阈值内对应的角度方差大于方差阈值;和/或,所述任一音频信号的能量大于所述能量阈值,且所述任一音频信号与所述N路音频信号中的其他音频信号的相关性大于或等于相关性阈值。
- 根据权利要求3所述的方法,其特征在于,所述任一音频信号的角度是基于解混矩阵中的所述任一音频信号对应的列数据、以及所述终端设备在各预设角度处的传递函数得到的;其中,所述解混矩阵是基于所述终端设备对所述第一音频信号进行声源分离时得到的。
- 根据权利要求4所述的方法,其特征在于,当所述终端设备的麦克风数量为2时,所述预设角度所在的范围满足:0°-180°,或者180°-360°。
- 根据权利要求3所述的方法,其特征在于,所述N路音频信号满足预设条件,包括:所述任一音频信号的能量大于所述能量阈值,且所述任一音频信号的角度在所述时间阈值对应的角度方差小于或等于所述方差阈值;和/或,所述任一音频信号的能量大于所述能量阈值,且所述任一音频信号与所述其他音频信号的相关性均小于所述相关性阈值。
- 根据权利要求5所述的方法,其特征在于,所述在所述终端设备接收到针对所述用于开启录制的控件的操作时,所述终端设备显示第二界面,包括:在所述终端设备接收到针对所述用于开启录制的控件的操作时,所述终端设备显示第三界面;其中,所述第三界面中包括所述第一画面,所述第一画面中包括所述目标对象;在所述终端设备接收到针对所述目标对象的操作时,所述终端设备显示所述第二界面。
- 根据权利要求7所述的方法,其特征在于,所述第二界面中包括:用于结束录制的控件,所述终端设备对所述第一音频信号进行声源分离,得到N路音频信号,包括:在所述终端设备接收到针对所述用于结束录制的控件的操作时,所述终端设备对所述第一音频信号进行声源分离,得到所述N路音频信号。
- 一种音频处理装置,其特征在于,包括显示单元以及处理单元,所述显示单元,用于显示第一界面;所述第一界面中包括:用于开启录制的控件;在终端设备接收到针对所述用于开启录制的控件的操作时,所述显示单元,还用于显示第二界面,所述处理单元,用于获取第一音频信号;其中,所述第二界面中包括第一画面以及第二画面,所述第二画面覆盖在所述第一画面之上,所述第一画面中包括所述第二画面中的内容;所述处理单元,还用于对所述第一音频信号进行声源分离,得到N路音频信号;N为大于或等于2的整数;所述处理单元,还用于生成第一视频以及第二视频;其中,所述第一视频是基于所述第一音频信号以及所述第一画面得到的;在所述N路音频信号满足预设条件的情况下,所述第二视频是基于所述N路音频信号以及所述第二画面得到的;所述第二视频对应的第二音频信号是对所述N路音频信号中的目标音频信号、和/或所述N路音频信号中的除所述目标音频信号以外的信号进行处理后得到的。
- 一种终端设备,包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序,其特征在于,所述处理器执行所述计算机程序时,使得所述终端设备执行如权利要求1至8任一项所述的方法。
- 一种计算机可读存储介质,所述计算机可读存储介质存储有计算机程序,其特征在于,所述计算机程序被处理器执行时,使得计算机执行如权利要求1至8任一项所述的方法。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP23814972.8A EP4383697A1 (en) | 2022-05-30 | 2023-05-17 | Audio processing method and apparatus |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210603382.1A CN116095254B (zh) | 2022-05-30 | 2022-05-30 | 音频处理方法和装置 |
CN202210603382.1 | 2022-05-30 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2023231787A1 true WO2023231787A1 (zh) | 2023-12-07 |
WO2023231787A9 WO2023231787A9 (zh) | 2024-05-10 |
Family
ID=86185558
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2023/094896 WO2023231787A1 (zh) | 2022-05-30 | 2023-05-17 | 音频处理方法和装置 |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP4383697A1 (zh) |
CN (1) | CN116095254B (zh) |
WO (1) | WO2023231787A1 (zh) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116095254B (zh) * | 2022-05-30 | 2023-10-20 | 荣耀终端有限公司 | 音频处理方法和装置 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113365012A (zh) * | 2020-03-06 | 2021-09-07 | 华为技术有限公司 | 一种音频处理方法及设备 |
US20210337331A1 (en) * | 2020-04-28 | 2021-10-28 | Beijing Xiaomi Pinecone Electronics Co., Ltd. | Method and device for detecting audio input module, and storage medium |
CN113573120A (zh) * | 2021-06-16 | 2021-10-29 | 荣耀终端有限公司 | 音频的处理方法及电子设备 |
CN114038452A (zh) * | 2021-11-17 | 2022-02-11 | 杭州海康威视数字技术股份有限公司 | 一种语音分离方法和设备 |
CN116095254A (zh) * | 2022-05-30 | 2023-05-09 | 荣耀终端有限公司 | 音频处理方法和装置 |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8538749B2 (en) * | 2008-07-18 | 2013-09-17 | Qualcomm Incorporated | Systems, methods, apparatus, and computer program products for enhanced intelligibility |
US8577677B2 (en) * | 2008-07-21 | 2013-11-05 | Samsung Electronics Co., Ltd. | Sound source separation method and system using beamforming technique |
CN106653008B (zh) * | 2015-10-28 | 2021-02-02 | 中兴通讯股份有限公司 | 一种语音控制方法、装置及*** |
ES2713685T3 (es) * | 2016-04-26 | 2019-05-23 | Nokia Technologies Oy | Métodos, aparatos y programas informáticos relativos a la modificación de una característica asociada a una señal de audio separada |
KR20180067920A (ko) * | 2016-12-13 | 2018-06-21 | 한국전자통신연구원 | 화성 성분에 기반한 음성 끝점 검출 장치 및 방법 |
CN111081269B (zh) * | 2018-10-19 | 2022-06-14 | ***通信集团浙江有限公司 | 通话过程中的噪声检测方法及*** |
CN110505403A (zh) * | 2019-08-20 | 2019-11-26 | 维沃移动通信有限公司 | 一种录像处理方法及装置 |
US11107492B1 (en) * | 2019-09-18 | 2021-08-31 | Amazon Technologies, Inc. | Omni-directional speech separation |
CN113747047B (zh) * | 2020-05-30 | 2023-10-13 | 华为技术有限公司 | 一种视频播放的方法及设备 |
CN113380267B (zh) * | 2021-04-30 | 2024-04-19 | 深圳地平线机器人科技有限公司 | 对音区进行定位的方法、装置、存储介质及电子设备 |
CN113203987A (zh) * | 2021-07-05 | 2021-08-03 | 成都启英泰伦科技有限公司 | 一种基于k均值聚类的多声源方位估计方法 |
CN114363512B (zh) * | 2021-09-30 | 2023-10-24 | 北京荣耀终端有限公司 | 一种视频处理的方法及相关电子设备 |
-
2022
- 2022-05-30 CN CN202210603382.1A patent/CN116095254B/zh active Active
-
2023
- 2023-05-17 EP EP23814972.8A patent/EP4383697A1/en active Pending
- 2023-05-17 WO PCT/CN2023/094896 patent/WO2023231787A1/zh unknown
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113365012A (zh) * | 2020-03-06 | 2021-09-07 | 华为技术有限公司 | 一种音频处理方法及设备 |
US20210337331A1 (en) * | 2020-04-28 | 2021-10-28 | Beijing Xiaomi Pinecone Electronics Co., Ltd. | Method and device for detecting audio input module, and storage medium |
CN113573120A (zh) * | 2021-06-16 | 2021-10-29 | 荣耀终端有限公司 | 音频的处理方法及电子设备 |
CN114038452A (zh) * | 2021-11-17 | 2022-02-11 | 杭州海康威视数字技术股份有限公司 | 一种语音分离方法和设备 |
CN116095254A (zh) * | 2022-05-30 | 2023-05-09 | 荣耀终端有限公司 | 音频处理方法和装置 |
Also Published As
Publication number | Publication date |
---|---|
CN116095254B (zh) | 2023-10-20 |
CN116095254A (zh) | 2023-05-09 |
EP4383697A1 (en) | 2024-06-12 |
WO2023231787A9 (zh) | 2024-05-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6039111B2 (ja) | マルチチャネルオーディオデータのビデオ解析支援生成 | |
US20230116044A1 (en) | Audio processing method and device | |
US9966084B2 (en) | Method and device for achieving object audio recording and electronic apparatus | |
US10924877B2 (en) | Audio signal processing method, terminal and storage medium thereof | |
US11039261B2 (en) | Audio signal processing method, terminal and storage medium thereof | |
CN113014983B (zh) | 视频播放方法、装置、存储介质及电子设备 | |
US20130106997A1 (en) | Apparatus and method for generating three-dimension data in portable terminal | |
JP2023519422A (ja) | オーディオ処理方法、装置、可読媒体及び電子機器 | |
WO2023231787A1 (zh) | 音频处理方法和装置 | |
US20210037336A1 (en) | An apparatus and associated methods for telecommunications | |
JP2022546542A (ja) | 通話方法、通話装置、通話システム、サーバ及びコンピュータプログラム | |
US20230209255A1 (en) | Electronic device and control method | |
WO2023151526A1 (zh) | 音频采集方法、装置、电子设备及外设组件 | |
CN112637529A (zh) | 一种录像处理方法、装置、存储介质及电子设备 | |
WO2022262416A1 (zh) | 音频的处理方法及电子设备 | |
JP6364130B2 (ja) | レコーディング方法、装置、プログラム及び記録媒体 | |
US11902754B2 (en) | Audio processing method, apparatus, electronic device and storage medium | |
US20240144948A1 (en) | Sound signal processing method and electronic device | |
CN114339582B (zh) | 双通道音频处理、方向感滤波器生成方法、装置以及介质 | |
US11368611B2 (en) | Control method for camera device, camera device, camera system, and storage medium | |
US10993064B2 (en) | Apparatus and associated methods for presentation of audio content | |
CN115942108A (zh) | 一种视频处理方法及电子设备 | |
WO2022161146A1 (zh) | 视频录制方法及电子设备 | |
WO2022218271A1 (zh) | 一种视频录制方法和电子设备 | |
JP2024041721A (ja) | ビデオ電話会議 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23814972 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2023814972 Country of ref document: EP Effective date: 20240305 |