CN112185406A

CN112185406A - Sound processing method, sound processing device, electronic equipment and readable storage medium

Info

Publication number: CN112185406A
Application number: CN202010990014.8A
Authority: CN
Inventors: 陈昌儒; 包英泽
Original assignee: Beijing Dami Technology Co Ltd
Current assignee: Beijing Dami Technology Co Ltd
Priority date: 2020-09-18
Filing date: 2020-09-18
Publication date: 2021-01-05

Abstract

The embodiment of the invention provides a sound processing method, a sound processing device, electronic equipment and a readable storage medium, and relates to the technical field of computers.

Description

Sound processing method, sound processing device, electronic equipment and readable storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a sound processing method and apparatus, an electronic device, and a readable storage medium.

Background

At present, with the development of computer technology, terminal devices can be linked with each other, for example, a user can put a picture displayed by a smart phone on a display screen (a display screen of a personal computer or a smart television).

For example, in a case of a video call, the user a may perform a video call with the user B through the mobile terminal a, and the user a may launch the display screen of the terminal a onto the smart tv a, at this time, the terminal a is responsible for recording the sound emitted by the user a, and the smart tv a is responsible for playing the display screen and the sound of the mobile terminal a.

In this process, the user a may set aside the mobile terminal a, however, due to the existence of the environmental noise (the sound emitted by the smart television a or the environmental sound which cannot be eliminated, etc.), and the user a is far away from the mobile terminal a, the sound recorded by the mobile terminal a is too small or unclear.

Disclosure of Invention

Embodiments of the present invention provide a sound processing method, an apparatus, an electronic device, and a readable storage medium, so as to perform directional sound pickup for a specific sound source and clearly capture sound emitted by the specific sound source.

In a first aspect, an electronic device is provided, which includes:

an apparatus main body;

a microphone array including a plurality of microphones respectively arranged in different planes of the device body; and

a controller configured to perform the steps of:

determining audio data collected by each microphone;

determining a pickup direction characteristic of the microphone array based on the audio data, wherein the pickup direction characteristic is used for representing a sound source direction characteristic acquired by the microphone array; and performing noise reduction processing based on the sound pickup direction characteristic.

Optionally, the controller is configured to perform the following steps:

determining audio features of the audio data based on a voice activity detection algorithm, the audio features including at least a speech intensity, a signal-to-noise ratio, a speech segment, and a non-speech segment; and

based on the audio features, a pickup direction characteristic of the microphone array is determined.

Optionally, the controller is configured to determine a direction of a noise source or a non-noise source based on the sound pickup direction characteristic; and adjusting the sound pickup direction according to the direction of the noise source or the non-noise source.

Optionally, the controller is configured to adjust the sound pickup direction by adjusting gains of the plurality of microphones.

Optionally, the method is configured to perform the following steps:

based on a preset deep neural network model, carrying out noise reduction and suppression on sound signals from other than the pickup direction, and determining a first output characteristic;

enhancing the sound signals from within the pickup direction, and determining a second output characteristic;

determining output data based on the first output characteristic and the second output characteristic; and

and sending the output data.

Optionally, the electronic device further includes: a plurality of speakers;

each loudspeaker is correspondingly arranged at the adjacent position of each specific microphone.

Optionally, the controller is further configured to control the at least one speaker to play the noise reduction audio to cancel the noise.

Optionally, the controller is configured to turn off at least one microphone according to the sound pickup direction characteristic, and noise in audio data picked up by the turned-off microphone satisfies a predetermined condition.

Optionally, the microphone array includes: a first microphone, a second microphone, a third microphone, and a fourth microphone; the equipment main body comprises a first surface, a second surface, a third surface and a fourth surface, wherein the first surface and the second surface are oppositely arranged, and the third surface and the fourth surface are oppositely arranged;

the first microphone and the second microphone are respectively arranged at the corresponding positions of the first surface and the second surface; the third microphone and the fourth microphone are respectively arranged at the corresponding positions of the third surface and the fourth surface.

Optionally, the controller is configured to perform the following steps:

determining a sound signal collected by each microphone;

performing analog-to-digital conversion on each sound signal, and determining a digital signal corresponding to the sound signal; and

and carrying out normalization processing on the digital signal to determine the audio data.

In a second aspect, a sound processing method is provided, which is applied to an electronic device, and includes:

determining audio data collected by each microphone;

determining a sound pickup direction characteristic of a microphone array based on the audio data, wherein the sound pickup direction characteristic is used for representing sound source direction characteristics acquired by the microphone array; and

and performing noise reduction processing based on the sound pickup direction characteristic.

Optionally, the determining, based on the audio data, a pickup direction characteristic of the microphone array includes:

Optionally, the performing noise reduction processing based on the sound pickup direction characteristic includes:

determining the direction of a noise source or a non-noise source based on the pickup direction characteristic; and

and adjusting the sound pickup direction according to the direction of the noise source or the non-noise source.

Optionally, the adjusting the pickup direction includes:

and adjusting the gain of each microphone in the microphone array to adjust the sound pickup direction.

and sending the output data.

controlling at least one speaker to play noise reduction audio to cancel the noise.

and at least one microphone is turned off according to the sound pickup direction characteristic, and noise in audio data picked up by the turned-off microphone meets a preset condition.

Optionally, the determining the audio data collected by each microphone includes:

determining a sound signal collected by each microphone;

In a third aspect, a sound processing apparatus is provided, the apparatus being applied to an electronic device, the apparatus including:

a first determining module for determining the audio data collected by each microphone;

the second determining module is used for determining a sound pickup direction characteristic of a microphone array based on the audio data, wherein the sound pickup direction characteristic is used for representing the sound source direction characteristic acquired by the microphone array; and

and the noise reduction module is used for carrying out noise reduction processing based on the pickup direction characteristic.

Optionally, the second determining module is specifically configured to:

Optionally, the noise reduction module is specifically configured to:

Optionally, the noise reduction module is further specifically configured to:

and sending the output data.

Optionally, the noise reduction module is further specifically configured to:

Optionally, the first determining module is specifically configured to:

determining a sound signal collected by each microphone;

In a fourth aspect, embodiments of the present invention provide a computer-readable storage medium on which computer program instructions are stored, which when executed by a processor implement the method according to the second aspect.

According to the embodiment of the invention, the electronic equipment can determine the sound pickup direction characteristic of the microphone array based on the audio data collected by each microphone in the microphone array, namely the electronic equipment can accurately determine the sound pickup direction according to the collected audio data, and further, the electronic equipment can perform noise reduction processing based on the sound pickup direction and the plurality of microphones, so that directional sound pickup aiming at a specific sound source is realized, and the electronic equipment can clearly capture the sound emitted by the specific sound source even if noise exists in the environment or the electronic equipment is far away from the specific sound source.

Drawings

The above and other objects, features and advantages of the embodiments of the present invention will become more apparent from the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

fig. 1 is a schematic diagram of an application scenario provided in an embodiment of the present invention;

fig. 2 is a schematic diagram of another application scenario provided in the embodiment of the present invention;

fig. 3 is a schematic diagram of an electronic device according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of another electronic device provided in an embodiment of the invention;

fig. 5 is a flowchart of a sound processing method according to an embodiment of the present invention;

FIG. 6 is a flow chart of another sound processing method according to an embodiment of the present invention;

FIG. 7 is a flow chart of another sound processing method according to an embodiment of the present invention;

fig. 8 is a schematic diagram illustrating an effect of beam forming according to an embodiment of the present invention;

fig. 9 is a schematic diagram of a pickup direction according to an embodiment of the present invention;

fig. 10 is a schematic diagram of an electronic device playing noise-reduced audio according to an embodiment of the present invention;

FIG. 11 is a flow chart of another sound processing method according to an embodiment of the present invention;

fig. 12 is a schematic structural diagram of a sound processing apparatus according to an embodiment of the present invention.

Detailed Description

The present invention will be described below based on examples, but the present invention is not limited to only these examples. In the following detailed description of the present invention, certain specific details are set forth. It will be apparent to one skilled in the art that the present invention may be practiced without these specific details. Well-known methods, procedures, components and circuits have not been described in detail so as not to obscure the present invention.

Further, those of ordinary skill in the art will appreciate that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale.

Unless the context clearly requires otherwise, throughout the description, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, what is meant is "including, but not limited to".

In the description of the present invention, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified.

As shown in fig. 1, fig. 1 is a schematic diagram of an application scenario provided in an embodiment of the present invention, where the application scenario includes: user a, electronic device 11, and television 12.

The scene may be an online education classroom, for example, the user a is a student in the online education classroom, which may make a video call with the teacher through the electronic device 11 to learn the knowledge taught by the teacher.

On the teacher side, the teacher may have a video call with user a (students) through the teacher side (e.g., smartphone, tablet, etc.).

It should be noted that the electronic device 11 may be a smart phone, a tablet computer, or the like, and the television 12 may be an intelligent screen with a screen projection function, and the embodiment of the present invention does not limit the parameters, such as the type, the model, and the like, of the electronic device 11 and the television 12.

When the user a makes a video call with the teacher through the electronic device 11, the electronic device 11 may collect sound emitted by the user a (the electronic device 11 may also shoot the user a at the same time, which is not limited in the embodiment of the present invention), and then send the sound to the teacher end through the network as student audio data.

Meanwhile, the electronic device 11 may also receive teacher video data and teacher audio data sent by the teacher end, and further, the electronic device 11 may play the teacher video data through the display and play the teacher audio data through the speaker.

In this process, the user a may project the data received by the electronic device 11 to the television 12, so that the television 12 plays the teacher video data and the teacher audio data received by the electronic device 11 through the network.

After the user a projects the data received by the electronic device 11 to the television 12, the electronic device 11 may be placed aside (as shown in fig. 1) to watch the teacher video data and the teacher audio data played in the television 12 to obtain a better experience in class, but the distance between the electronic device 11 and the user a is increased, so that the electronic device 11 needs to pick up sound from the user a in a long distance.

In this scenario as well, as shown in fig. 2, fig. 2 is a schematic diagram of an application scenario of an electronic device provided in an embodiment of the present invention, where the application scenario includes: user a, electronic device 11, and television 12.

When the user a places the electronic device 11 aside, the electronic device 11 needs to collect the sound emitted by the user a, but the television 12 also emits the sound through the speaker (as shown in fig. 2), at this time, the electronic device 11 collects the sound emitted by the television 12 and the sound emitted by the user a at the same time, and therefore, in this scenario, the sound emitted by the television 12 interferes with the sound emitted by the user a, that is, the sound emitted by the television 12 is an interfering noise, and the sound emitted by the user a is a non-interfering noise.

In order to solve the problem of noise interference in the above scenario, an embodiment of the present invention provides an electronic device, including: the microphone array includes a plurality of microphones respectively arranged on different planes of the device body.

Specifically, as shown in fig. 3, fig. 3 is a schematic view of an electronic device according to an embodiment of the present invention, where the electronic device includes: a device body 3 and a microphone array.

The electronic device may be a mobile terminal, such as a smart phone, a tablet computer, etc., which can be used for video call.

The microphone array includes a first microphone 31, a second microphone 32, a third microphone 33, and a fourth microphone 34, and the device body 3 includes first and second faces disposed oppositely and third and fourth faces disposed oppositely.

The first microphone 31 is disposed on the first surface of the device body 3, the second microphone 32 is disposed on the second surface of the device body 3, the third microphone 33 is disposed on the third surface of the device body 3, and the fourth microphone 34 is disposed on the fourth surface of the device body 3.

The first microphone 31 and the second microphone 32 are respectively arranged at the corresponding positions of the first surface and the second surface; the third microphone 33 and the fourth microphone 34 are disposed at positions corresponding to the third face and the fourth face, respectively.

It should be noted that, in the embodiment of the present invention, in order to indicate the position of each microphone, the external shape of the electronic device in fig. 3 is simplified, and the external shape of the electronic device is not limited in the embodiment of the present invention.

Moreover, the microphone installation position shown in fig. 3 is a preferred implementation manner provided by the embodiment of the present invention, and the installation of the microphone array as shown in fig. 3 can make the electronic device have a better sound pickup effect, and in practical applications, the installation positions of the microphones are in different planes of the device main body 3.

Because the installation positions of the microphones are on different planes of the device main body 3, the orientations of the microphones are different, and the electronic device has better sound pickup effect.

As shown in fig. 4, fig. 4 is a schematic diagram of an electronic device according to an embodiment of the present invention, where the electronic device shown in fig. 4 is the electronic device shown in fig. 3, and includes a computer hardware structure, which includes at least a processor 41 and a memory 42.

The processor 41 and the memory 42 are connected by a bus 43, the memory 42 is adapted to store instructions or programs executable by the processor 41, and the processor 41 may be a stand-alone microprocessor or a collection of one or more microprocessors.

Thus, the processor 41 implements processing of the sound signal and control of other devices by executing instructions stored in the memory 42.

A bus 43 couples the various components together and to a display controller 44 and at least one microphone 45, the microphone 45 typically being coupled to the system through an input/output (I/O) controller 46.

In order to solve the problem of noise interference during the process of picking up sound remotely by an electronic device, an embodiment of the present invention provides a sound processing method, which may be configured in a controller of the electronic device, so that the controller executes the sound processing method, specifically, as shown in fig. 5, the specific steps are as follows:

at step 100, audio data collected by each microphone is determined.

At step 200, based on the audio data, a sound pickup directional characteristic of the microphone array is determined.

The pickup direction characteristic is used for representing the sound source direction characteristic acquired by the microphone array.

In step 300, noise reduction processing is performed based on the sound pickup direction characteristic.

Further, as shown in fig. 6, in step 100, the controller may be configured to perform the steps of:

at step 110, the sound signals collected by each microphone are determined.

In practical application, when sound waves are transmitted to the microphones, vibration of the sound waves is transmitted to vibrating diaphragms of the microphones, then vibration of the vibrating diaphragms can push magnets in the microphones to form variable currents, and further variable voltages are formed, then the electronic equipment can amplify the variable voltages based on a sound processing circuit, and further analog signals are obtained, namely the sound signals in the embodiment of the invention.

At step 120, analog-to-digital conversion is performed on each sound signal to determine a digital signal corresponding to the sound signal.

In practical applications, the electronic device needs to convert the analog signal into a digital signal for further operation.

In the embodiment of the present invention, the digital signal may exist in a form of a standard digital audio file, and the format of the digital signal may be a WaveForm sound file (WaveForm, WAV) or other format.

Specifically, the electronic device may quantize the analog signal into an integrated value represented by a binary number based on an analog-to-digital converter (ADC), and then encode and store the integrated value as audio stream data, i.e., convert the sound signal into a digital signal in the embodiment of the present invention.

In step 130, the digital signal is normalized to determine audio data.

In an implementation manner, the digital signal may be mapped into the interval (0, 1) through normalization processing, so that the electronic device may process data more conveniently and quickly.

After the electronic device preprocesses the sound signal and determines the audio data collected by each microphone, the electronic device may further determine the sound pickup direction characteristics of the microphone array according to the audio data, i.e. step 200.

Specifically, as shown in fig. 7, in step 200, the controller may be configured to perform the following steps:

at step 210, audio characteristics of the audio data are determined based on a Voice Activity Detection (VAD) algorithm.

Wherein the audio features include at least speech intensity, signal-to-noise ratio, speech segments, and non-speech segments.

The speech strength can be used to characterize the signal strength of the non-noise signal, the signal-to-noise ratio can be used to characterize the ratio of the non-noise signal to the noise signal, the speech segments can be used to characterize segments containing non-noise signals, and the non-speech segments can be used to characterize segments not containing non-noise signals.

In an implementation manner, the VAD algorithm may determine whether the audio data includes voice data through a feature extraction module, a threshold calculation module, a threshold decision module, and the like, that is, determine an input signal, and distinguish the voice data from noise data.

In another implementation manner, the purpose of determining the audio characteristics of the audio data may also be achieved by constructing a VAD model, and specifically, a frame-by-frame (where each audio frame may be a preset length) recognition may be performed on a sound sample through a pre-trained acoustic model, so as to determine a recognition result output by the acoustic model. Wherein the recognition result can be represented by 0 or 1 (for example, 0 can be used to characterize that the corresponding audio frame does not include the non-noise signal, and 1 can be used to characterize that the corresponding audio frame includes the non-noise signal).

Then, the voice samples and the recognition result output by the acoustic model can be used as a training set to train the VAD model, so that the trained VAD model can distinguish voice data from noise data.

Specifically, the process of determining the audio feature of the audio data by the electronic device may specifically be executed as: preprocessing the audio data; and performing feature extraction and VAD detection on the preprocessed audio data to determine audio features.

The preprocessing of the audio data may include Fast Fourier Transform (FFT), and the like, and the number of multiplications required by the electronic device to calculate the discrete Fourier Transform may be reduced by using the FFT, thereby saving the computational power of the electronic device.

At step 220, based on the audio features, a pickup directional characteristic of the microphone array is determined.

Further, the electronic device may determine a sound pickup direction of the microphone array based on the sound pickup direction characteristic.

Specifically, the controller may be configured to determine the direction of the noise source or the non-noise source based on the sound pickup direction characteristic; and adjusting the sound pickup direction according to the direction of the noise source or the non-noise source.

In the embodiment of the present invention, the sound pickup direction characteristic may be an angle of a noise source or a non-noise source with respect to the electronic device, and the electronic device may determine the sound pickup direction according to the angle.

Further, the controller may be configured to adjust the sound pickup direction by adjusting gains of the plurality of microphones.

Specifically, the controller may adjust the gains of the plurality of microphones to enhance the sound signals from a specific direction, and weaken the sound signals from other directions to achieve directional sound pickup of the microphone array, that is, directional sound pickup of the microphone array in the electronic device is achieved through beamforming (beamforming).

The beamforming is a technique for processing a received wave signal by using a digital signal processing technique to realize directional signal transmission or directional signal reception. The principle is that the desired ideal signal is formed by performing weighted synthesis on a plurality of paths of signals, and in practical application, the beam forming technology has wider application in the field of antennas. Meanwhile, the propagation of the sound signals is also carried out in a sound wave mode, so that the beam forming technology can also be applied to directional sound pickup.

As shown in fig. 8, fig. 8 is a schematic diagram of the effect of beam forming provided by the embodiment of the present invention, and the schematic diagram includes a microphone array 81, a circular scale for representing angles, and a sector area for representing a sound pickup range.

Specifically, a direction having a larger sound collection range indicates a larger gain of the microphone array 81 in that direction (i.e., a larger weight of the directional sound signal), and a direction having a smaller sound collection range indicates a larger suppression of the microphone array 81 in that direction (i.e., a smaller weight of the directional sound signal).

In fig. 8, including 2 gain directions and 4 suppression directions, when a sound wave propagates from the gain direction to the microphone array 81, a sound signal corresponding to the sound wave is enhanced, whereas when the sound wave propagates from the suppression direction to the microphone array 81, the sound signal corresponding to the sound wave is suppressed.

The 4 suppression directions shown in fig. 8 are directions in which the electronic device mainly suppresses, and in regions other than the gain direction and the suppression direction, the electronic device can perform relative suppression to reduce noise interference.

It should be noted that the number of gain directions and the number of suppression directions are not fixed, and may be adjusted according to the actual environment, for example, in the embodiment of the present invention, the main object of the sound collected by the electronic device is the sound emitted by the user, and the main noise source is the sound emitted by other devices (for example, the television 12 in fig. 1), so that in the embodiment of the present invention, there may be 1 gain direction and 1 suppression direction, and in an implementation manner, the region other than the 1 gain direction and 1 suppression direction may neither gain nor suppress.

In another possible implementation, the sound signals in the 1 gain direction may be gained, and the sound signals except for the 1 gain direction may be suppressed.

As shown in fig. 9, fig. 9 is a schematic diagram of a sound pickup direction according to an embodiment of the present invention, where the schematic diagram includes a user a, an electronic device 11, an area a, and an area b.

In the embodiment of the present invention, the user a is a non-noise source, and when the microphone array of the electronic device 11 collects a sound signal, the electronic device may determine the sound pickup direction characteristic of the microphone array based on the

above steps

210 and 220, and further, the electronic device may determine the sound pickup direction of the microphone array according to the sound pickup direction characteristic.

In practical applications, the sound pickup direction may be represented by an angle range, as shown in fig. 9, an area a and an area b represent the angle range, and the area a and the area b are combined together to form a complete circle around the electronic device 11, where the area a is a sound pickup area, and the area b is a noise reduction area, that is, after the microphone array collects the sound signal, the controller may perform speech enhancement on the sound signal collected in the area a, and perform noise reduction processing on the sound signal collected in the area b.

It should be noted that, in the principle of beam forming described in conjunction with fig. 8, in fig. 9, the region a may represent the gain direction, and in practical applications, there may be a plurality of non-noise sources, and in this case, the sound pickup direction may be represented by a plurality of discrete regions a (i.e., there are a plurality of gain directions).

According to the embodiment of the invention, the electronic equipment can determine the sound pickup direction characteristic based on the microphone array, and further determine the sound pickup direction according to the sound pickup direction characteristic, and the sound pickup direction points to the non-noise source needing to be collected by the electronic equipment, so that the sound signal of the non-noise source can be clearer.

In the embodiment of the invention, after the electronic equipment determines the sound pickup direction, the sound signals within the sound pickup direction can be enhanced, and the sound signals outside the sound pickup direction can be subjected to noise reduction and suppression.

Specifically, in one possible implementation, the controller may be configured to perform noise reduction and suppression on sound signals from directions other than the pickup direction based on a preset deep neural network model, and determine the first output characteristic; enhancing the sound signal from the pickup direction to determine a second output characteristic; determining output data based on the first output characteristic and the second output characteristic; and transmitting the output data.

The deep neural network model is a model constructed based on a machine learning technology, and the model can analyze the characteristics of the sound signal through a convolutional neural network.

Specifically, the deep neural network model can be trained through a training set collected in advance, so that the deep neural network model can identify the characteristics of noise signals and the characteristics of non-noise signals, in practical application, the deep neural network model can perform characteristic extraction and convolution operation on sound signals, then perform noise reduction and suppression on the noise signals, and enhance the non-noise signals.

After the electronic device determines the first output characteristic and the second output characteristic, format conversion may be performed on the first output characteristic and the second output characteristic to determine output data, where a format of the output data may be a format such as WAV.

According to the embodiment of the invention, the electronic equipment realizes noise reduction of sound signals except for the pickup direction based on the deep neural network model, so that non-noise signals are clearer from a digital level, and the final pickup effect is better.

In another possible implementation, the electronic device may further include a plurality of speakers, wherein each speaker is disposed adjacent to each specific microphone.

The specific microphone may be a partially designated microphone or all microphones.

Specifically, in conjunction with the content shown in fig. 3, the embodiment of the present invention provides the following two examples of speaker installation that can be implemented, specifically as follows:

in an example one, the plane where the third microphone 33 is located is a plane where the electronic device is installed with a display screen, and the first microphone 31, the second microphone 32 and the fourth microphone 34 are specific microphones, wherein one speaker may be installed at a position adjacent to each of the first microphone 31 and the fourth microphone 34, two speakers may be installed at a position adjacent to the second microphone 32, and no speaker may be installed at a position adjacent to the third microphone 33.

In this case, for two speakers installed at adjacent positions of the second microphone 32, the two speakers may be located as far away from the second microphone 32 as possible to reduce the influence of the two speakers on the second microphone 32.

Meanwhile, the distance between two speakers installed at adjacent positions of the second microphone 32 may be a preset distance, for example, 1 cm, and the embodiment of the present invention is not limited thereto.

In addition, since the third microphone 33 is located on a plane with a display screen, additional process cost is added when the speaker is installed on the plane, and therefore, in practical applications, the speaker may not be installed adjacent to the third microphone 33, so as to save process cost.

In example two, the first microphone 31, the second microphone 32, the third microphone 33, and the fourth microphone 34 are each a specific microphone, wherein at least one speaker may be installed at each of adjacent positions of the four microphones.

Further, the controller may be configured to control the at least one speaker to play the noise reduction audio to cancel the noise.

Specifically, after the electronic device determines the sound pickup direction characteristic, it may determine not only the non-noise source, but also the noise source, and therefore, after the electronic device determines the main noise source, it may control at least one speaker closest to the main noise source to play the noise reduction audio to cancel the noise.

As shown in fig. 10, fig. 10 is a schematic diagram of playing noise reduction audio by an electronic device according to an embodiment of the present invention, where the schematic diagram includes: the apparatus main body 3, the first microphone 31, the second microphone 32, the third microphone 33, the fourth microphone 34, the noise source X, the non-noise source Y, and the noise reduction audio Z.

In fig. 10, one speaker may be disposed at each of adjacent positions of the first microphone 31 and the fourth microphone 34, two speakers may be disposed at adjacent positions of the second microphone 32, and no speaker may be disposed at adjacent positions of the third microphone 33.

After the electronic device determines the sound pickup direction characteristic, it may determine that the noise source is X and the non-noise source is Y, and meanwhile, since the first microphone 31 is closest to the noise source X, the controller in the electronic device may control the speaker in the position adjacent to the first microphone 31 to play the noise reduction audio Z to cancel the noise.

The noise reduction audio is a sound wave with the same amplitude and the opposite phase of the noise, namely the noise reduction audio Z is the same in amplitude and the opposite phase of the sound wave sent by the noise source X, when the two sound waves with the same amplitude and the opposite phase meet, the two sound waves can mutually offset vibration, and then the purpose of offsetting the noise through the noise reduction audio is achieved.

According to the embodiment of the invention, the electronic equipment can offset part of noise from the outside through the noise reduction audio, so that the microphone array can collect the sound signal after noise sound waves are filtered, and the final sound pickup effect is better.

In another possible embodiment, the controller of the electronic device may be further configured to turn off the at least one microphone according to the sound pickup direction characteristic, and noise in the audio data picked up by the turned-off microphone satisfies a predetermined condition.

The predetermined condition may be at least one of a condition that the voice intensity is less than the voice intensity threshold, a condition that the signal-to-noise ratio is less than the signal-to-noise ratio threshold, and a condition that no voice segment is included in the audio data.

In practical application, if audio data picked up by a certain microphone in the microphone array meets a predetermined condition, it is characterized that the noise interference received by the microphone is too large, and further, the controller in the electronic device can control the microphone to be closed, so as to reduce the noise interference received by the whole microphone array.

In practical applications, the electronic device may use any of the above embodiments alone, or may combine each of the above embodiments to perform noise reduction.

For example, as shown in fig. 10, when the electronic device is in a noisy environment and speakers are disposed at adjacent positions of the first to fourth microphones 31 to 34, the electronic device may determine sound pickup directional characteristics according to audio data collected by the first to fourth microphones 31 to 34, and further determine the noise source X and the non-noise source Y according to the sound pickup directional characteristics.

Then, the electronic device may determine the microphone closest to the noise source X as the first microphone 31, and the controller of the electronic device may control the speaker adjacent to the first microphone 31 to play the noise reduction audio Z to filter the noise sound wave.

Meanwhile, the controller of the electronic device may control the first microphone 31 to be turned off to reduce noise received by the microphone array.

Meanwhile, the controller of the electronic equipment can also reduce and inhibit the noise of sound signals from the outside of the sound pickup direction and enhance the sound signals from the inside of the sound pickup direction based on the deep neural network model so as to further weaken the noise and enhance the non-noise.

In the embodiment of the invention, the electronic device can combine a plurality of noise reduction means (at least including sound wave suppression, microphone turning off and digital noise reduction), so as to reduce the influence of noise to the maximum extent, so that the electronic device can clearly capture the sound of a user, and the user has good conversation experience.

In summary, as shown in fig. 11, fig. 11 is a flowchart of a sound processing method according to an embodiment of the present invention, where the method can be applied to the electronic device shown in fig. 3, and specifically includes the following steps:

in step 1111, the sound signal 1 collected by the first microphone 31 is determined.

In step 1112, the sound signal 2 collected by the second microphone 32 is determined.

In step 1113, the sound signal 3 collected by the third microphone 33 is determined.

In step 1114, the sound signal 4 collected by the fourth microphone 34 is determined.

At step 112, the sound signals collected by each microphone are analog-to-digital converted and normalized to determine audio data.

Specifically, the electronic device may determine the audio data according to the above steps 110 to 130, which is not described herein again in this embodiment of the present invention.

In step 113, based on the audio data, a sound pickup direction characteristic of the microphone array is determined.

The sound pickup direction characteristic can be used for representing the direction of a noise source and the direction of a non-noise source.

In step 114, noise reduction processing is performed based on the sound pickup direction characteristic.

Specifically, the electronic device may implement the noise reduction processing by one or a combination of sound wave suppression, turning off a microphone, and digital noise reduction.

In step 115, digital-to-analog conversion is performed on the noise-reduced data to determine output data.

At step 116, the output data is transmitted.

With reference to the application scenarios shown in fig. 1 and fig. 2, after determining the output data, the electronic device may send the output data to the teacher end through the network, so that the teacher end plays the output data.

Based on the same technical concept, an embodiment of the present invention further provides a sound processing apparatus, as shown in fig. 12, the apparatus including: a first determination module 121, a second determination module 122, and a noise reduction module 123.

A first determining module 121, configured to determine audio data collected by each microphone;

a second determining module 122, configured to determine, based on the audio data, a sound pickup directional characteristic of a microphone array, where the sound pickup directional characteristic is used to characterize a sound source directional characteristic acquired by the microphone array; and

and the noise reduction module 123 is configured to perform noise reduction processing based on the sound pickup direction characteristic.

Optionally, the second determining module 122 is specifically configured to:

Optionally, the noise reduction module 123 is specifically configured to:

Optionally, the noise reduction module 123 is further specifically configured to:

and sending the output data.

Optionally, the first determining module 121 is specifically configured to:

determining a sound signal collected by each microphone;

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus (device) or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations of methods, apparatus (devices) and computer program products according to embodiments of the invention. It will be understood that each flow in the flow diagrams can be implemented by computer program instructions.

These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows.

These computer program instructions may also be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows.

Another embodiment of the invention is directed to a non-transitory storage medium storing a computer-readable program for causing a computer to perform some or all of the above-described method embodiments.

That is, as can be understood by those skilled in the art, all or part of the steps in the method of the above embodiments may be accomplished by specifying related hardware through a program, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps in the method of the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An electronic device, characterized in that the electronic device comprises:

an apparatus main body (3);

a microphone array including a plurality of microphones respectively arranged in different planes of the device body (3); and

a controller configured to perform the steps of:

determining audio data collected by each microphone;

determining a pickup direction characteristic of the microphone array based on the audio data, wherein the pickup direction characteristic is used for representing a sound source direction characteristic acquired by the microphone array; and

2. The electronic device of claim 1, wherein the controller is configured to perform the steps of:

3. The electronic device of claim 2, wherein the controller is configured to determine a direction of a noise source or a non-noise source based on the pickup direction characteristic; and adjusting the sound pickup direction according to the direction of the noise source or the non-noise source.

4. The electronic device of claim 3, wherein the controller is configured to adjust the pickup direction by adjusting gains of the plurality of microphones.

5. The electronic device of claim 4, wherein the electronic device is configured to perform the steps of:

and sending the output data.

6. The electronic device of claim 1, further comprising: a plurality of speakers;

7. The electronic device of claim 6, wherein the controller is further configured to control at least one speaker to play noise-reducing audio to cancel noise.

8. The electronic device according to claim 1, wherein the controller is configured to turn off at least one microphone according to the sound pickup direction characteristic, and noise in audio data picked up by the turned-off microphone satisfies a predetermined condition.

9. The electronic device of claim 1, wherein the microphone array comprises: a first microphone (31), a second microphone (32), a third microphone (33) and a fourth microphone (34); the device body (3) comprises a first surface and a second surface which are oppositely arranged, and a third surface and a fourth surface which are oppositely arranged;

the first microphone (31) and the second microphone (32) are respectively arranged at the corresponding positions of the first surface and the second surface; the third microphone (33) and the fourth microphone (34) are disposed at positions corresponding to the third surface and the fourth surface, respectively.

10. The electronic device of claim 1, wherein the controller is configured to perform the steps of:

determining a sound signal collected by each microphone;

11. A method of sound processing, the method comprising:

determining audio data collected by each microphone;

12. The method of claim 11, wherein determining a pickup direction characteristic of the microphone array based on the audio data comprises:

13. The method according to claim 12, wherein the performing noise reduction processing based on the sound pickup direction characteristic includes:

14. The method of claim 13, wherein the adjusting the pickup direction comprises:

15. The method according to claim 14, wherein the performing noise reduction processing based on the sound pickup direction characteristic includes:

and sending the output data.

16. The method according to claim 11, wherein the performing noise reduction processing based on the sound pickup direction characteristic includes:

17. The method according to claim 11, wherein the performing noise reduction processing based on the sound pickup direction characteristic includes:

18. The method of claim 11, wherein the determining the audio data collected by each microphone comprises:

determining a sound signal collected by each microphone;

19. A sound processing apparatus, characterized in that the apparatus comprises:

20. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of any one of claims 11-18.