WO2021084905A1 - Sound pickup device and sound pickup method - Google Patents
Sound pickup device and sound pickup method Download PDFInfo
- Publication number
- WO2021084905A1 WO2021084905A1 PCT/JP2020/033469 JP2020033469W WO2021084905A1 WO 2021084905 A1 WO2021084905 A1 WO 2021084905A1 JP 2020033469 W JP2020033469 W JP 2020033469W WO 2021084905 A1 WO2021084905 A1 WO 2021084905A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- sound
- state
- image
- control unit
- instruction
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 66
- 230000005236 sound signal Effects 0.000 claims abstract description 11
- 238000001514 detection method Methods 0.000 claims description 41
- 230000008569 process Effects 0.000 description 49
- 230000006870 function Effects 0.000 description 17
- 238000010586 diagram Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000007792 addition Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0272—Voice signal separating
- G10L21/028—Voice signal separating using properties of sound source
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
- H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
- H04R1/40—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers
- H04R1/406—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers microphones
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10K—SOUND-PRODUCING DEVICES; METHODS OR DEVICES FOR PROTECTING AGAINST, OR FOR DAMPING, NOISE OR OTHER ACOUSTIC WAVES IN GENERAL; ACOUSTICS NOT OTHERWISE PROVIDED FOR
- G10K11/00—Methods or devices for transmitting, conducting or directing sound in general; Methods or devices for protecting against, or for damping, noise or other acoustic waves in general
- G10K11/18—Methods or devices for transmitting, conducting or directing sound
- G10K11/26—Sound-focusing or directing, e.g. scanning
- G10K11/34—Sound-focusing or directing, e.g. scanning using electrical steering of transducer arrays, e.g. beam steering
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/24—Speech recognition using non-acoustical features
- G10L15/25—Speech recognition using non-acoustical features using position of the lips, movement of the lips or face analysis
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L21/0216—Noise filtering characterised by the method used for estimating noise
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0272—Voice signal separating
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
- H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
- H04R1/40—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R3/00—Circuits for transducers, loudspeakers or microphones
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/027—Concept to speech synthesisers; Generation of natural phrases from machine-based concepts
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
- G10L21/0208—Noise filtering
- G10L21/0216—Noise filtering characterised by the method used for estimating noise
- G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
- G10L2021/02166—Microphone arrays; Beamforming
Definitions
- This disclosure relates to a sound collecting device and a sound collecting method for collecting a target sound.
- Patent Document 1 discloses an estimation device for estimating which participant spoke when when from data recording discourse by one or more participants. This estimation device calculates the probability that the discourse participant has spoken by integrating the information obtained from the acoustic signal output from the microphone and the information obtained from the video signal output from the camera. This makes it possible to track the position of the discourse participant even when the discourse participant moves in a situation where there is no utterance.
- the present disclosure provides a sound collecting device and a sound collecting method capable of facilitating sound collection of a target sound.
- the sound collecting device of one aspect of the present disclosure is a sound collecting device for collecting a target sound output by a user from a sound source, and includes a first input unit, a second input unit, a control unit, and the like. It has an output unit.
- the first input unit inputs image data generated by the camera.
- the second input unit inputs the acoustic signal output from the microphone array.
- the control unit determines the sound collecting direction for collecting the target sound based on the image data and the acoustic signal.
- the output unit outputs the information presented to the user.
- the control unit determines whether or not the image indicated by the input image data and the sound indicated by the input acoustic signal are in a predetermined state. When the control unit determines that the image and sound are in a predetermined state, the control unit outputs an instruction to the user to the output unit, and after the instruction is output, inputs further image data and sound signal to adjust the sound collection direction. decide.
- the target sound can be easily picked up by using the instruction to the user.
- Block diagram illustrating the configuration of a sound collecting device The block diagram which shows the function of the control part in the sound collecting device of FIG.
- Flow chart exemplifying the overall operation of the sound collector Diagram for explaining the outline of the operation of the sound collecting device
- the figure for demonstrating the area of the detection unit in image data A flowchart illustrating the process of determining the acoustic state in the sound collecting device.
- the figure which illustrates the state table in the sound collecting device Flow chart exemplifying sound collection processing via instructions in the sound collection device
- the figure which illustrates the instruction table in the sound collecting device Flow chart exemplifying sound collection processing without instructions in the sound collection device
- FIG. 1 is a diagram for explaining an outline of the sound collecting device 10 according to the present embodiment.
- the sound collecting device 10 of the present embodiment performs multimodal speech enhancement using the camera 11 and the microphone array 12.
- the camera 11 captures a detection range in an environment that is the target of sound collection.
- the microphone array 12 receives sound waves from the same detection range as the camera 11. In the figure, the horizontal angle ⁇ corresponding to the horizontal direction and the vertical angle ⁇ corresponding to the vertical direction of the detection range are illustrated.
- FIG. 1 shows an example in which a plurality of persons 21 and 22 and a noise source 25 are included in the detection range of the sound collecting device 10.
- the direction toward the sound source of the target sound such as the person 21 speaking from the sound collecting device 10 is estimated based on the detection results of the camera 11 and the microphone array 12.
- speech enhancement can be performed so as to avoid the non-speaking person 22 and the noise source 25 and pick up only the sound from the direction of the speaking person 21 with high sensitivity.
- the sound collecting device 10 of the present embodiment timely outputs an instruction to a user such as a person 21 so as to enable sound collecting of a target sound even in a situation where such speech enhancement does not function effectively. ..
- a user such as a person 21
- the configuration of the sound collecting device 10 of the present embodiment will be described.
- FIG. 2 illustrates the configuration of the sound collection device 10 according to the present embodiment.
- the sound collecting device 10 includes a camera 11, a microphone array 12, a control unit 13, a storage unit 14, an input / output interface unit 15, an audio output unit 16, a display unit 17, and a bus 18.
- the sound collecting device 10 is a dedicated sound collecting device in which the camera 11 and the microphone array 12 are built and integrated.
- the sound collecting device 1 does not have to incorporate either one or both of the camera 11 and the microphone array 12.
- the sound collecting device 1 is electrically connected to the external camera 11 or the microphone array 12.
- the sound collecting device 1 may be an electronic device such as a smartphone, a tablet terminal, or a PC equipped with a camera 11, and may be electrically and mechanically connected to an external device including the microphone array 12.
- the camera 11 includes an image sensor such as a CCD image sensor, a CMOS image sensor, or an NMOS image sensor.
- the camera 11 generates and outputs image data which is a video signal.
- the microphone array 12 includes a plurality of microphones.
- the microphone array 12 receives a sound wave in each microphone, converts it into an acoustic signal which is an electric signal, and outputs the sound wave.
- the microphone array 12 can be applied with a known beamforming technique that controls directivity such as the direction and range of sound collection by adjusting the amount of delay between a plurality of microphones.
- the control unit 13 determines the sound collecting direction based on the image data obtained from the camera 11 and the acoustic signal obtained from the microphone array 12.
- the sound collecting direction is the direction in which the target sound is picked up from the microphone array 12 because it is considered that the target sound source exists.
- the target sound source is a sound source that outputs the target sound, for example, the mouth of a person.
- the control unit 13 extracts a target sound from the acoustic signal output by the microphone array 12 by performing signal processing that emphasizes the sound coming from the sound collecting direction.
- the control unit 13 can be realized by a semiconductor element or the like.
- the control unit 13 can be composed of, for example, a microcomputer, a CPU, an MPU, a DSP, an FPGA, or an ASIC.
- FIG. 3 shows the functional configuration of the control unit 13 in the sound collecting device 10 of FIG.
- the function of the control unit 13 may be configured only by hardware, or may be realized by combining hardware and software.
- the control unit 13 includes an image state determination unit 31, an acoustic state determination unit 32, an instruction determination unit 33, an output processing unit 34, a state difference detection unit 35, a direction estimation unit 36, and a signal processing unit.
- includes functional units 31 to 37 such as 37. The operation of each function unit 31 to 37 will be described later.
- the storage unit 14 is a storage medium that stores programs and data necessary for realizing the functions of the sound collecting device 10.
- the storage unit 14 can be realized by, for example, a hard disk (HDD), an SSD, a RAM, a DRAM, a ferroelectric memory, a flash memory, a magnetic disk, or a combination thereof.
- the image data obtained from the camera 11 and the acoustic signal obtained from the microphone array 12 may be stored in the storage unit 14. Further, the storage unit 14 stores, for example, a state table and an instruction table, which will be described later.
- the storage unit 14 may store the target sound data of the sound collection result.
- the input / output interface unit 15 includes a circuit that communicates with an external device in accordance with a predetermined communication standard.
- Predetermined communication standards include, for example, LAN, Wi-Fi®, Bluetooth®, USB, and HDMI®.
- the input / output interface unit 15 is an example of an input unit of the sound collecting device 10 for an external device, and is also an example of an output unit.
- the voice output unit 16 is a speaker device that converts an acoustic signal into sound waves.
- the audio output unit 16 is an example of an output unit that outputs audio and presents various types of information.
- the display unit 17 is composed of, for example, a liquid crystal display or an organic EL display.
- the display unit 17 is an example of an output unit that displays various types of information.
- the display unit 17 may be various projectors or the like.
- the bus 18 is a signal line that electrically connects the camera 11, the microphone array 12, the control unit 13, the storage unit 14, the input / output interface unit 15, the audio output unit 16, and the display unit 17.
- control unit 13 When the control unit 13 acquires image data from the camera 11 or retrieves it from the storage unit 14, the control unit 13 corresponds to an image data input unit. When the control unit 13 acquires the acoustic signal from the microphone array 12 or takes it out from the storage unit 14, the control unit 13 corresponds to the input unit of the acoustic signal.
- the control unit 13 or the bus 18 corresponds to the image data input unit.
- the control unit 13 extracts the acoustic signal from the storage unit 14, the control unit 13 or the bus 18 corresponds to the input unit of the acoustic signal.
- the input / output interface unit 15 inputs image data from the camera 11 externally attached to the sound collecting device 10
- the input / output interface unit 15 corresponds to the image data input unit.
- the input / output interface unit 15 inputs an acoustic signal from the microphone array 12 externally attached to the sound collecting device 10
- the input / output interface unit 15 corresponds to the audio signal input unit.
- the sound collecting device 10 includes a plurality of output units such as an audio output unit 16, a display unit 17, and an input / output interface unit 15 has been described, but the sound collecting device 10 has such an output unit. Any one of the above may be provided. Further, the output unit is not limited to the above example, and may be, for example, a blinking lamp, a notification buzzer, or the like.
- FIG. 4 is a flowchart illustrating the overall operation of the sound collecting device 10 according to the present embodiment.
- FIG. 5 is a diagram for explaining an outline of the operation of the sound collecting device 10. Each process shown in the flowchart of FIG. 4 is executed by the control unit 13 of the sound collecting device 10.
- control unit 13 executes an image state determination process that functions as an image state determination unit 31 based on the image data from the camera 11 (S1). Further, the control unit 13 executes an acoustic state determination process that functions as the acoustic state determination unit 32 based on the acoustic signal from the microphone array 12 (S2).
- steps S1 and S2 the current state of the user, environment, etc. included in the image indicated by the image data and the acoustic indicated by the acoustic signal is determined. Details of the processing in steps S1 and S2 will be described later.
- the order in which the processes of steps S1 and S2 are executed is not particularly limited, and may be in the reverse order of the illustrated order or may be executed in parallel.
- control unit 13 determines whether or not the determined current state of the image and the sound is a predetermined state to be instructed to the user (S3).
- An example of the state of the referent is illustrated in FIG. 5 (A).
- FIG. 5A illustrates a state in which the speaking person 21 (that is, the user) faces a different direction from the camera 11 and the microphone array 12 of the sound collecting device 10.
- the speaking person 21 that is, the user
- the sound signal is intended. It becomes difficult to collect the sound clearly. Further, it is conceivable that such a situation occurs not only due to the orientation of the face of the person 21, but also due to various factors that hinder sound collection, such as noise from the noise source 25.
- the sound collecting device 10 of the present embodiment makes a determination in step S3 based on the current state of the image and sound in order to instruct the user to improve the state when the above situation occurs. Since it is difficult to collect the target sound, the state of the referent in step S3 can be set to various states that should be instructed to the user. The details of step S3 will be described later.
- control unit 13 determines that the current state of the image and the sound is the state to be instructed (YES in S3), the control unit 13 executes the sound collection process via the instruction to the user (S4).
- the outline of the process of step S4 will be described with reference to FIG. 5 (B).
- FIG. 5 (B) illustrates a case where an instruction is output from the state of FIG. 5 (A).
- the voice output unit 16 of the sound collecting device 10 outputs the voice of the instruction "Please face the camera and speak.” According to the output of such an instruction, the user can grasp that there is a problem in the sound collection of the utterance and the reason.
- the person 21 who wants to pick up the utterance in this example improves the direction of the face and speaks again according to the output instruction. Therefore, in the sound collection process (S4) via the instruction of the present embodiment, the accuracy of collecting the re-uttered utterance is improved based on the difference between the states before and after the instruction. Details of the process in step S4 will be described later.
- step S5 when the control unit 13 determines that the current state of the image and the sound is not the state of the referent (NO in S3), the control unit 13 executes the sound collection process without any particular instruction (S5). According to the process of step S5, for example, if the sound can be picked up without instructing the user, the target sound can be picked up without giving unnecessary instructions. Details of the process in step S5 will be described later.
- control unit 13 After performing the sound collection processing in step S4 or step S5, the control unit 13 periodically repeats the processing from step S1 onward, for example.
- the sound collecting device 10 determines the image state and the sound state at any time (S1, S2), and when the state to be resolved is detected (YES in S3), the sound collecting device 10 picks up the sound via an instruction to the user.
- the process (S4) is performed.
- the sound collecting device 10 performs sound collecting processing without instructing (S5) unless it is in a state to be particularly resolved (NO in S3). As a result, it is possible to avoid a situation in which unnecessary instructions occur frequently and become annoying to the user.
- the details of the operation of the above sound collecting device 10 will be described below.
- FIG. 6 is a flowchart illustrating the image state determination process (S1) in the sound collecting device 10.
- the control unit 13 functions as an image state determination unit 31 and executes each process of the flowchart illustrated in FIG.
- control unit 13 as the image state determination unit 31 inputs image data from the camera 11 via the bus 18 (S11).
- the camera 11 generates image data indicating the captured image, for example, at a predetermined frame rate.
- Image data is input sequentially for each frame, for example.
- the control unit 13 detects various states of the image related to the utterance based on the input image data (S12). For example, the control unit 13 first detects a person in the image indicated by the image data, and when the person is detected, detects a face in the area of the person in the image. Further, when the face is detected, the control unit 13 detects the orientation of the face in the image and parts such as the mouth. Further, the control unit 13 detects the movement of an object such as the mouth based on the detection results for a plurality of frames. Each of these various detection targets is an example of information indicating the arrangement of the target sound source in the image.
- the various detections in step S12 as described above can be performed, for example, by using a classifier that identifies the feature amount of each part.
- a classifier that identifies the feature amount of each part.
- a weak discriminator that has been machine-learned in advance can be used (see, for example, Non-Patent Document 1).
- the detected target area can be represented by, for example, horizontal coordinates and vertical coordinates in the image indicated by the image data.
- FIG. 7 shows the region r ( ⁇ , ⁇ ) of the detection unit in the image data v.
- step S12 for example, at the coordinates corresponding to the horizontal angle ⁇ and the vertical angle ⁇ , the image indicated by the image data v is divided into a plurality of regions r ( ⁇ , ⁇ ), and the detection result is obtained for each region r ( ⁇ , ⁇ ). Recorded.
- the grid is divided and the respective axes are associated with ⁇ and ⁇ .
- ⁇ and ⁇ may be associated with each other in a circumferential shape.
- control unit 13 classifies the current image state in, for example, a preset classification class based on various information detected from the image (S13).
- classification class in step S13 for example, A class, B class, and C class, which are set in three stages from a good image state in terms of the detection result such as the mouth of the face on the image, are used (see FIG. 9).
- the classification of the image state does not have to be in three stages.
- Class A regarding the image state includes an image state in which the face and its mouth are detected and the face is detected in the forward direction, that is, in the direction facing the camera 11. According to the A-class image state, it is considered that the presence or absence of an utterance can be determined based on the movement of the mouth in the image, and when there is an utterance, the direction in which the utterance is picked up can be determined only by the image.
- the B class of the image includes an image state in which the face and its mouth are detected, but the direction of the face is different from the forward direction (for example, sideways). According to the B-class image state, it is possible to determine the presence or absence of an utterance based on the movement of the mouth, but it is difficult to determine the sound collecting direction of the utterance from the image alone, or it is difficult to determine whether the utterance can be collected. Can be considered.
- the C class of the image is a state in which the face and mouth were not detected. According to the C-class image state, it is not feasible to determine the presence or absence of utterance based on the movement of the mouth. As for the C-class image state, for example, even if a person is detected, the person wears a mask on the mouth, the face is facing backward, the distance between the person and the camera 11 is too long, or the image is dark. It may pass.
- the C class is not necessarily limited to the case where a person is detected, and may include the case where a person is not detected.
- the control unit 13 determines whether or not there is an utterance based on the detection result of the image state (S15). ). Specifically, the control unit 13 calculates an utterance score indicating the likelihood of whether or not the movement of the mouth in the image state is an utterance, and compares the utterance score in the image state with a predetermined threshold value.
- the threshold indicates a criterion for an utterance score that is large enough to be confident that the movement of the mouth is an utterance.
- the control unit 13 sets the utterance flag on the image side to "ON" as a result of determining that there is an utterance when the utterance score in the image state is equal to or higher than the threshold value (YES in S15) (S16).
- the utterance flag on the image side is a flag that manages the determination result of the presence or absence of utterance depending on the image state, and is stored in the storage unit 14, for example.
- the control unit 13 sets the utterance flag on the image side to "OFF" as a result of the determination that there is no utterance (NO in S15) when the utterance score in the image state is less than the threshold value (S17). After setting the utterance flag on the image side (S16, S17), the control unit 13 ends the process of step S1 of FIG. 4, and proceeds to, for example, step S2.
- the control unit 13 does not determine the utterance based on the image state (S15 to S17), and ends the process of step S1.
- FIG. 8 is a flowchart illustrating the acoustic state determination process (S2).
- the control unit 13 functions as an acoustic state determination unit 32, and executes each process of the flowchart illustrated in FIG.
- the control unit 13 as the acoustic state determination unit 32 acquires an acoustic signal from the microphone array 12 via the bus 18 (S21).
- the microphone array 12 sequentially generates an acoustic signal according to a sound wave received from the outside.
- the acoustic signal is acquired in synchronization with the image data acquired in step S11 of FIG.
- control unit 13 detects various acoustic states related to utterance based on the acquired acoustic signal (S22).
- the detection in step S22 is performed, for example, in each direction ( ⁇ , ⁇ ) corresponding to the region r ( ⁇ , ⁇ ) of each detection unit in FIG. 7.
- step S22 the control unit 13 performs an operation of estimating the position of the sound source or the direction in which the sound arrives from the sound source by the CSP (Cross-Power Spectrum Phase Analysis) method or the MUSIC (Multiple Signal Classification) method. Further, the control unit 13 detects a voice in the frequency band of the human voice, which is the target sound, by using, for example, a voice GMM (Gausian Mixture Model) and a non-voice GMM trained in advance, or is different from the target sound. Detects noise.
- CSP Cross-Power Spectrum Phase Analysis
- MUSIC Multiple Signal Classification
- the control unit 13 classifies the current acoustic state in a predetermined classification class, for example, in the same manner as the image state classification (S13) (S23).
- a predetermined classification class for example, in the same manner as the image state classification (S13) (S23).
- the classification class in step S23 for example, the A, B, and C classes that are set in three stages from the good acoustic state from the viewpoint of the noise detection result are used (see FIG. 9).
- Acoustic A class includes, for example, an acoustic state in which the signal level at which noise is detected is lower than the lower limit level of noise that can interfere with the detection of a target sound such as utterance.
- the A class may include an acoustic state in which noise and target sound can be separated from the viewpoint of frequency, phase (or arrival direction), and the like. According to the A-class acoustic state, it is considered that the presence or absence of an utterance can be determined from the detection result of the target sound, and when there is an utterance, the sound collection direction of the utterance can be determined without using the image state.
- the acoustic B class includes, for example, an acoustic state in which the noise signal level is detected between the lower limit level and the upper limit level.
- the upper limit level of noise indicates an upper limit in which noise and target sound can be separated from the viewpoint of signal level, and is set larger than the lower limit level.
- the B class may include an acoustic state in which the noise and the target sound are difficult to separate from the viewpoint of frequency, phase, etc., for example, the frequency band of the noise is near the frequency band of the human voice.
- the B-class acoustic state it may be possible to determine the presence or absence of utterance sound, but it may be difficult to determine the direction of utterance sound collection without using the image state, or it may be difficult to determine whether or not utterance sound collection is possible. Be done.
- Acoustic C class includes, for example, an acoustic state detected when the signal level of noise exceeds the upper limit level.
- the C class may include an acoustic state in which noise and target sound cannot be separated from the viewpoint of frequency, phase, and the like. According to the C-class acoustic state, it is not possible to determine the presence or absence of spoken voice.
- the control unit 13 determines whether or not there is an utterance based on the detection result of the acoustic state (S25). Specifically, the control unit 13 calculates an utterance score indicating the likelihood of whether or not the detected voice is a human voice, and compares the utterance score in the acoustic state with a predetermined threshold value.
- the threshold indicates a measure of speech score that is large enough to be confident that the voice is human voice.
- the control unit 13 sets the utterance flag on the acoustic side to "ON" as a result of the determination that there is utterance (YES in S25) when the utterance score in the acoustic state is equal to or higher than the threshold value (S26).
- the utterance flag on the acoustic side is a flag that manages the determination result of the presence or absence of utterance depending on the acoustic state, and is stored in the storage unit 14 separately from the utterance flag on the image side, for example.
- the control unit 13 sets the utterance flag on the acoustic side to "OFF" as a result of the determination that there is no utterance (NO in S25) when the utterance score in the acoustic state is less than the threshold value (S27). After setting the utterance flag on the acoustic side (S26, S27), the control unit 13 ends the process of step S2 of FIG. 4, and proceeds to, for example, step S3.
- the control unit 13 does not determine the utterance based on the acoustic state (S25 to S27), and ends the process of step S2.
- the target sound and noise are detected (S22) based on the acoustic signals sequentially input from the microphone array 12, and the current acoustic state is determined (S23 to S23). S27).
- FIG. 9 is a diagram illustrating a state table D1 in the sound collecting device 10.
- the state table D1 is a data table that manages the state of the referent and the state of the non-target in association with the image state and the acoustic state, and is stored in advance in, for example, the storage unit 14.
- "-" for each utterance flag indicates that the presence or absence of utterance has not been determined.
- the determination in step S3 is performed by combining the image state and the acoustic state, for example, as shown in FIG.
- the control unit 13 refers to the state table D1 and determines whether or not the current state based on the image state determination result (S1) and the acoustic state determination result (S2) is the referent state (S3). ..
- the image state A, B, C class and the utterance flag on the image side are turned ON / OFF, and the acoustic state A, B, C class and the utterance flag on the acoustic side are turned ON / OFF.
- the five-stage states 1 to 5 are defined.
- the state to be instructed to improve the state is state 3 (YES in S3), and the other states 1, 2, 4 and 5 are not the target of the instruction (NO in S3).
- the state does not have to be in five stages.
- State 1 is a state in which both the image state and the acoustic state are A class. According to the state 1, since the utterance can be determined in both the image state and the acoustic state, it is considered that the utterance can be picked up without using an instruction for improving the state.
- State 2 is a state in which one of the image state and the acoustic state is A class and the other is B class or C class. According to the state 2, since the utterance can be determined in either the image state or the acoustic state, it is considered that the utterance can be picked up without using any particular instruction.
- both the image state and the acoustic state are B class or C class, at least one is B class, and at least one of the utterance flags on the image side and the acoustic side is "ON".
- state 3 although it is determined that there is an utterance, it is difficult to determine the sound collecting direction of the utterance in both the image state and the acoustic state. Therefore, it is considered useful to output an instruction to eliminate such a state in which there is an obstacle in which the sound collection direction cannot be determined.
- State 4 is a state in which both the utterance flags on the image side and the sound side are not "ON” but at least one is "OFF” in the same classification class as state 3. According to the state 4, it is difficult to determine the sound collecting direction and the like in both the image state and the acoustic state, but since it is determined that there is no utterance, it is considered that no particular instruction needs to be output.
- State 5 is a C-class state in both the image state and the acoustic state. According to the state 5, it is impossible to determine whether or not there is an utterance, the sound collecting direction, and the like. For this reason, it is possible that no particular instruction is required because no utterance has been made, and it is considered difficult to output an instruction that specifies appropriate content in order to improve the state.
- the sound collection direction cannot be determined and the target sound is collected even though it is known that the utterance is in progress. It is possible to avoid a situation where no sound is produced.
- the referent and non-referent states described above are examples.
- the state of the referent is not limited to the above state 3, and may include various obstacles that make it difficult to determine the sound collecting direction. It can be set.
- FIG. 10 is a flowchart illustrating the sound collection process (S4) via instructions. The process shown in this flowchart is executed when the current state is the state 3 of FIG. 9 (YES in S3).
- control unit 13 functions as an instruction determination unit 33, and determines the content of the information to be included in the instruction based on the current state obtained in steps S1 to S3 (S41).
- the process of step S41 is performed with reference to the instruction table D2 as shown in FIG. 11, for example.
- the instruction table D2 is a data table that manages instruction contents indicating information to be included in the instruction, and is stored in the storage unit 14 in advance.
- the instruction table D2 illustrated in FIG. 11 records the ID number that identifies the instruction content and the instruction content in association with each other.
- the control unit 13 as the instruction determination unit 33 selects an ID number to be determined as the instruction content from the instruction table D2, for example, based on the current state determined in step S3 of FIG.
- the instruction determination unit 33 may appropriately use the detection results of various image states in step S1 (S12 in FIG. 6) and the detection results of various acoustic states in step S2 (S22 in FIG. 8). Further, the instruction content may be determined from the combination of each image state and acoustic state classified into the state 3 by using the state table D1 (FIG. 9).
- the instruction determination unit 33 selects the ID number N1. According to the instruction content of the ID number N1, it is possible to realize an instruction for resolving a state in which there is an obstacle such as the face not facing forward (see FIG. 5B).
- the instruction determination unit 33 selects the ID number N2. Further, when the acoustic state is C class and remarkable noise is generated, the instruction determination unit 33 selects the ID number N3. When the image state is C class and the speaking person is too far away to detect the movement of the mouth, the instruction determination unit 33 selects the ID number N4. In this way, by selecting from the instruction table D2, the instruction content can be set so as to eliminate the state in which the utterance could not be picked up due to each obstacle. The instruction determination unit 33 can detect various obstacles based on the detection results of the image state and the acoustic state.
- the control unit 13 detects, for example, the image state and the acoustic state before outputting the instruction (S42).
- the control unit 13 functions as, for example, the image state determination unit 31 and the acoustic state determination unit 32, and performs the same processing as in steps S11 and S12 of FIG. 6 and steps S21 and S22 of FIG.
- step S42 for example, as an image state, an image state such as a person's position, a face orientation, a mouth position, and a distance between the person and the camera 11 is detected. Further, as the acoustic state, the probability of the arrival direction of the voice due to the phase difference, the likelihood of the human voice for each direction, and the like are detected.
- control unit 13 functions as an output processing unit 34, and transmits instruction signals for controlling various output units so as to output the determined instruction via the bus 18 (S43).
- control unit 13 as the output processing unit 34 controls the voice output unit 16 to output the instruction content by voice.
- the control unit 13 may display the instruction content on the display unit 17. Further, when presenting an instruction to the user using an external device such as an external monitor, the control unit 13 transmits an instruction signal to the external device to the input / output interface unit 15 as an output unit.
- step S44 the control unit 13 detects the image state and the acoustic state after the instruction (S44).
- step S44 for example, the control unit 13 performs the same detection as the detection before the instruction (S42) as the image state determination unit 31 and the acoustic state determination unit 32 after a predetermined period from the output of the instruction.
- the predetermined period is a period in which the user is expected to start an action according to an instruction, for example, 1 second.
- the process of step S44 is the same as, for example, steps S11 and S12 of FIG. 6 and steps S21 and S22 of FIG.
- control unit 13 functions as a state difference detecting unit 35, and detects a difference between the image state and the acoustic state (S42) before the instruction and the image state and the acoustic state (S44) after the instruction (S44). S45).
- the difference between the states before and after the instruction is detected from the part where the user who responds to the instruction for improving the state speaks, while the difference between the states before and after the instruction is detected from the external part that does not respond to the instruction. It is assumed that the difference between the states is not detected.
- step S45 the control unit 13 as the state difference detecting unit 35 detects, for example, the position of the person, the orientation of the face, the mouth, etc. that have changed before and after the instruction as the difference in the image state.
- the control unit 13 calculates, for example, a difference score indicating the degree of difference in the detected image states for each region r ( ⁇ , ⁇ ) of the detection unit in the image. Further, the control unit 13 detects, for example, the probability of the arrival direction of the voice changed before and after the instruction and the likelihood of the human voice in each direction ( ⁇ , ⁇ ) as the difference in the acoustic state.
- the control unit 13 calculates a difference score indicating the degree of difference in the detected acoustic state for each direction.
- control unit 13 functions as a direction estimation unit 36, and performs an operation of estimating the direction in which the target sound due to the utterance after the instruction arrives based on the difference between the detected states (S46). For example, the control unit 13 calculates the linear sum of the difference score of the image state and the difference score of the acoustic state in each direction within the detection range, and sets the direction in which the score obtained as the calculation result is the highest as the target sound. Determine the sound collection direction of.
- the linear sum of the difference scores may be a weighted sum that weights the image state and the acoustic state, respectively. Further, instead of the linear sum, multiplication or non-linear calculation may be appropriately adopted.
- control unit 13 functions as a signal processing unit 37 and executes a process of emphasizing the sound from the sound collecting direction of the calculation result (S47).
- the signal processing unit 37 performs signal processing so as to direct the beam in the sound collecting direction by, for example, beamforming, and generates target sound data indicating the sound collecting result of the target sound.
- the signal processing unit 37 may perform various signal processing such as a frequency filter to emphasize the target sound.
- the target sound data is stored in, for example, the storage unit 14.
- control unit 13 ends the process of step S4 in FIG. 4 after executing the speech enhancement (S47). After that, the control unit 13 returns to, for example, step S1 and performs the subsequent processing again.
- the instruction content of the ID number N1 is output as shown in FIG. 5 (B) (S41 to S43).
- the person 21 who made the utterance improves the direction of the face according to the instruction, so that the next utterance can be picked up.
- the non-speaking person 22 and the noise source 25 do not respond to the instruction. Therefore, for example, the person 21 whose face orientation is improved can be easily detected as the difference between the states before and after the instruction (S44, S45), and the sound collecting direction can be accurately determined (S46, S47).
- step S46 for example, the likelihood of the human voice increases in the voice state by the instruction of ID numbers N2 and N3, and the human and the camera in the image state by the instruction of ID number N4. It is conceivable that the distance between 11 will be shorter. By using the difference between these states, the accuracy of collecting the target sound can be improved.
- step S42 is not limited to before the output of the instruction (S43), and may be performed at the same time as step S43, for example.
- the process of step S42 can be executed within the range before the state detection (S44) after the instruction, and may be, for example, before step S41.
- the detection result of step S42 may be used to determine the instruction content (S41).
- the state detection before the instruction may be performed before step S3, the processing of step S42 may be omitted and the detection results of steps S12 and S22 may be used instead.
- the instruction may be uniform.
- a content such as "Please face the camera and speak in a loud voice” may be adopted. Even with such uniform instructions, it is considered possible to improve the condition so that sound can be picked up.
- FIG. 12 is a flowchart illustrating a sound collection process (S5) that does not involve instructions. The process shown in this flowchart is executed when the current state is any of the states 1, 2, 4, and 5 in FIG. 9 (NO in S3).
- control unit 13 first determines whether or not the current state is determined to have an utterance (S51). For example, the control unit 13 proceeds to YES in step S51 when at least one of the utterance flag on the image side and the utterance flag on the acoustic side is “ON” in the states 1 and 2. On the other hand, the control unit 13 proceeds to NO in step S51 in other cases.
- step S42 is performed using, for example, the detection result of the current image state (S12 in FIG. 6) and the detection result of the acoustic state (S22 in FIG. 8).
- control unit 13 functions as a signal processing unit 37 and executes a process of emphasizing the sound from the sound collecting direction of the calculation result (S53).
- the process of step S53 is performed in the same manner as in step S47 of FIG. 10, for example.
- the target sound is picked up without using an instruction.
- the control unit 13 ends the process of step S5 in FIG. 4 after executing the speech enhancement (S53). After that, the control unit 13 returns to, for example, step S1 and performs the subsequent processing again.
- control unit 13 determines whether or not the current state is the state 5 of FIG. 9, that is, the state in which the utterance cannot be determined (S54). ..
- control unit 13 When the current state is the state 5 (YES in S54), the control unit 13 outputs, for example, an error notification to various output units (S55), and ends step S5. For example, the control unit 13 causes the display unit 17 to display an error message or the like in order to notify the sound that cannot be picked up.
- steps S54 and S55 may be omitted as appropriate.
- step S5 when the current state is not the state 5 (NO in S54), the control unit 13 ends step S5 without performing the process of step S55 in particular.
- NO in S51 and S54 since it is determined that there is no utterance in the states 1, 2, and 4, the processing in steps S52 and S53 can also be omitted.
- control unit 13 may determine whether or not the current state is the same as the state in the previous control cycle, and only when it is determined that the current state is not the same as the previous time.
- the processing after step S51 may be performed. If it is determined that the current state is the same as the previous time, the same processing as the previous time may be applied to the subsequent processing, for example, the same sound collecting direction may be used.
- the sound collecting device 10 collects a target sound such as a voice of an utterance output from a mouth by a user such as a person 21 as an example of a sound source.
- the sound collecting device 10 includes a first input unit, a second input unit, a control unit 13, and an output unit such as an audio output unit 16, a display unit 17, and an input / output interface unit 15.
- the control unit 13 as the first input unit inputs the image data generated by the camera 11 (S11).
- the control unit 13 as the second input unit inputs the acoustic signal output from the microphone array 12 (S21).
- the control unit 13 determines the sound collecting direction for collecting the target sound based on the image data and the acoustic signal (S3, S4).
- the output unit outputs the information presented to the user.
- the control unit 13 determines whether or not the image indicated by the input image data and the sound indicated by the input acoustic signal are in a predetermined state such as the state 3 of FIG. 9 (S3).
- the control unit 13 determines that the image and sound are in a predetermined state (YES in S3), the control unit 13 outputs an instruction to the user to the output unit (S43), and after the instruction is output, further image data and sound signal. Is input (S44) to determine the sound collecting direction (S46, S47).
- the target sound when it becomes difficult to collect the target sound (YES in S3), an instruction is output to the user (S43), and the subsequent image data and acoustic signal are output.
- the sound collecting direction is determined by using (S46, S47).
- control unit 13 determines the difference between the image data input after the instruction and the image data input before the image data, and the acoustic signal input after the instruction and before the acoustic signal.
- the difference between the input acoustic signals is detected (S45), and the sound collection direction is determined (S46, S47).
- control unit 13 determines the difference between the image data and the acoustic signal for each direction in a plurality of directions for each region r ( ⁇ , ⁇ ) of the detection unit as a candidate for the sound collection direction (see FIG. 7). A score according to the magnitude of the difference between the two is calculated, and the direction in which the calculated score is the largest is determined as the sound collection direction (S46). By such a simple arithmetic process, it is possible to easily collect the target sound.
- the sound collecting device 10 further includes a storage unit 14 that stores a state table D1 that classifies image and sound states.
- the control unit 13 refers to the state table D1 and determines the information to be included in the instruction according to the image and sound when it is determined that the state is in a predetermined state (S41). For example, the control unit 13 changes the information included in the instruction so as to eliminate the predetermined state, as illustrated in FIG. Instructions to be illustrated to the user can be appropriately set according to the state determined from the image and sound, whereby the current state can be easily improved.
- control unit 13 determines that the image and sound are not in a predetermined state (NO in S3), the control unit 13 determines the sound collecting direction without outputting an instruction (S5). This makes it possible for the user to easily use the sound collecting device 10 by avoiding the frequent occurrence of unnecessary instructions.
- the control unit 13 detects information indicating the arrangement of the sound source in the image based on the image data (S12), and detects the target sound and noise in the sound based on the acoustic signal (S22).
- the sound collecting direction is determined according to the detection result of the image data and the acoustic signal (S3, S4).
- the predetermined state is a state including an obstacle in determining the sound collecting direction as in the state 3. According to the same state, the control unit 13 does not determine the sound collecting direction according to the above detection result. When such a predetermined state occurs, an instruction to the user is output to improve the state of the obstacle, so that the target sound can be easily picked up.
- the sound collecting device 10 may include at least one of the camera 11 and the microphone array 12.
- the sound collecting device 10 does not have to include both the camera 11 and the microphone array 12.
- the sound collecting method is a method for collecting the target sound output from the sound source by the user, from the step (S11) of inputting the image data generated by the camera 11 and the microphone array 12.
- a step (S43) of outputting an instruction to the user and further image data and sound signal are input after the instruction is output to collect sound.
- the steps (S44 to S47) for determining the direction are included. According to this method, the target sound can be easily collected.
- a program for causing a computer to execute the above sound collecting method is provided.
- Such a program can be stored and provided in various storage media including, for example, the storage unit 14, or can be provided via a communication network.
- the case where the horizontal angle ⁇ and the vertical angle ⁇ are determined as the sound collection direction has been described, but the case where the direction of the target sound source can be specified only by at least one of the horizontal angle ⁇ and the vertical angle ⁇ . May only determine at least one of a horizontal angle ⁇ and a vertical angle ⁇ .
- the sound uttered by a person is picked up as the target sound
- the target sound is not limited to the voice uttered by the person and may be various sounds uttered by the person.
- the target sound may be the sound of a car, the bark of an animal, or the operating sound of equipment. Similar to the first embodiment, by presenting an instruction to a user who wants to collect the target sound when it is difficult to collect the target sound, it is possible to facilitate the collection of the target sound.
- the sound collecting device of the present disclosure is useful as, for example, a device for collecting sound spoken by a person.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computational Linguistics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Otolaryngology (AREA)
- Studio Devices (AREA)
- Circuit For Audible Band Transducer (AREA)
Abstract
A sound pickup device (10) for picking up a target sound outputted from a sound source by a user is provided with: a first input unit to which image data generated by a camera (11) is inputted; a second input unit to which a sound signal outputted from a microphone array (12) is inputted; a control unit (13) which, on the basis of the image data and the sound signal, determines a sound pickup direction in which the target sound is picked up; and an output unit (16) which outputs information to be presented to the user. The control unit assesses whether or not an image indicated by the inputted image data and a sound indicated by the inputted sound signal are in a predetermined state (S3). When assessing that the image and the sound are in the predetermined state (YES in S3), the control unit causes the output unit to output an instruction to the user, inputs additional image data and sound signal after the instruction is outputted, and determines the sound pickup direction (S4).
Description
本開示は、目的音を収音する収音装置および収音方法に関する。
This disclosure relates to a sound collecting device and a sound collecting method for collecting a target sound.
特許文献1は、一人以上の参加者による談話を記録したデータから、どの参加者がいつ発話したかを推定するための推定装置を開示している。この推定装置は、マイクロホンから出力される音響信号から得られる情報と、カメラから出力される映像信号から得られる情報とを統合することによって、談話参加者が発話した確率を算出している。これにより、談話参加者が発話のない状況で移動した場合であっても、談話参加者の位置を追跡することを可能にしている。
Patent Document 1 discloses an estimation device for estimating which participant spoke when when from data recording discourse by one or more participants. This estimation device calculates the probability that the discourse participant has spoken by integrating the information obtained from the acoustic signal output from the microphone and the information obtained from the video signal output from the camera. This makes it possible to track the position of the discourse participant even when the discourse participant moves in a situation where there is no utterance.
本開示は、目的音が収音され易くすることができる収音装置および収音方法を提供する。
The present disclosure provides a sound collecting device and a sound collecting method capable of facilitating sound collection of a target sound.
本開示の一態様の収音装置は、ユーザが音源から出力させる目的音を収音するための収音装置であって、第1の入力部と、第2の入力部と、制御部と、出力部とを備える。第1の入力部は、カメラによって生成された画像データを入力する。第2の入力部は、マイクアレイから出力された音響信号を入力する。制御部は、画像データ及び音響信号に基づいて、目的音を収音する収音方向を決定する。出力部は、ユーザに提示する情報を出力する。制御部は、入力された画像データが示す画像及び入力された音響信号が示す音響が、所定の状態であるか否かを判断する。制御部は、画像及び音響が所定の状態であると判断したとき、ユーザに対する指示を出力部に出力させ、指示が出力された後に更なる画像データ及び音響信号を入力して、収音方向を決定する。
The sound collecting device of one aspect of the present disclosure is a sound collecting device for collecting a target sound output by a user from a sound source, and includes a first input unit, a second input unit, a control unit, and the like. It has an output unit. The first input unit inputs image data generated by the camera. The second input unit inputs the acoustic signal output from the microphone array. The control unit determines the sound collecting direction for collecting the target sound based on the image data and the acoustic signal. The output unit outputs the information presented to the user. The control unit determines whether or not the image indicated by the input image data and the sound indicated by the input acoustic signal are in a predetermined state. When the control unit determines that the image and sound are in a predetermined state, the control unit outputs an instruction to the user to the output unit, and after the instruction is output, inputs further image data and sound signal to adjust the sound collection direction. decide.
これらの概括的かつ特定の態様は、システム、方法、及びコンピュータプログラム、並びに、それらの組み合わせにより、実現されてもよい。
These general and specific aspects may be realized by systems, methods, and computer programs, and combinations thereof.
本開示の収音装置および収音方法によると、ユーザに対する指示を用いて、目的音が収音され易くすることができる。
According to the sound collecting device and the sound collecting method of the present disclosure, the target sound can be easily picked up by using the instruction to the user.
以下、適宜図面を参照しながら、実施の形態を詳細に説明する。但し、必要以上に詳細な説明は省略する場合がある。例えば、既によく知られた事項の詳細説明や実質的に同一の構成に対する重複説明を省略する場合がある。これは、以下の説明が不必要に冗長になるのを避け、当業者の理解を容易にするためである。
Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art.
なお、出願人は、当業者が本開示を十分に理解するために添付図面および以下の説明を提供するのであって、これらによって特許請求の範囲に記載の主題を限定することを意図するものではない。
It should be noted that the applicant is not intended to limit the subject matter described in the claims by those skilled in the art by providing the accompanying drawings and the following description in order to fully understand the present disclosure. Absent.
(実施形態1)
以下、本開示の実施形態1について、図面を参照しながら説明する。本実施形態では、目的音の一例として人の声を収音する例について説明する。 (Embodiment 1)
Hereinafter, the first embodiment of the present disclosure will be described with reference to the drawings. In this embodiment, an example of collecting a human voice as an example of a target sound will be described.
以下、本開示の実施形態1について、図面を参照しながら説明する。本実施形態では、目的音の一例として人の声を収音する例について説明する。 (Embodiment 1)
Hereinafter, the first embodiment of the present disclosure will be described with reference to the drawings. In this embodiment, an example of collecting a human voice as an example of a target sound will be described.
1.構成
1-1.概要
図1は、本実施形態に係る収音装置10の概要を説明するための図である。本実施形態の収音装置10は、カメラ11とマイクアレイ12を用いたマルチモーダルの音声強調を行う。カメラ11は、収音の対象となる環境における検出範囲を撮影する。マイクアレイ12は、カメラ11と同じ検出範囲からの音波を受信する。図中では、検出範囲の水平方向に対応する水平角度θ及び垂直方向に対応する垂直角度φを例示している。 1. 1. Configuration 1-1. Outline FIG. 1 is a diagram for explaining an outline of thesound collecting device 10 according to the present embodiment. The sound collecting device 10 of the present embodiment performs multimodal speech enhancement using the camera 11 and the microphone array 12. The camera 11 captures a detection range in an environment that is the target of sound collection. The microphone array 12 receives sound waves from the same detection range as the camera 11. In the figure, the horizontal angle θ corresponding to the horizontal direction and the vertical angle φ corresponding to the vertical direction of the detection range are illustrated.
1-1.概要
図1は、本実施形態に係る収音装置10の概要を説明するための図である。本実施形態の収音装置10は、カメラ11とマイクアレイ12を用いたマルチモーダルの音声強調を行う。カメラ11は、収音の対象となる環境における検出範囲を撮影する。マイクアレイ12は、カメラ11と同じ検出範囲からの音波を受信する。図中では、検出範囲の水平方向に対応する水平角度θ及び垂直方向に対応する垂直角度φを例示している。 1. 1. Configuration 1-1. Outline FIG. 1 is a diagram for explaining an outline of the
図1は、複数の人物21,22及び騒音源25が収音装置10の検出範囲に含まれた例を示している。収音装置10では、カメラ11とマイクアレイ12による検出結果に基づいて、例えば、収音装置10から発話している人物21などの目的音の音源に向かう方向が推定される。これにより、発話していない人物22及び騒音源25などを避けて、発話中の人物21の方向からの音声のみを高感度に収音するように音声強調を行うことができる。
FIG. 1 shows an example in which a plurality of persons 21 and 22 and a noise source 25 are included in the detection range of the sound collecting device 10. In the sound collecting device 10, the direction toward the sound source of the target sound such as the person 21 speaking from the sound collecting device 10 is estimated based on the detection results of the camera 11 and the microphone array 12. As a result, speech enhancement can be performed so as to avoid the non-speaking person 22 and the noise source 25 and pick up only the sound from the direction of the speaking person 21 with high sensitivity.
本実施形態の収音装置10は、こうした音声強調が有効に機能しない状況に陥った場合であっても目的音の収音を可能にするべく、人物21等のユーザに対する指示を適時に出力する。以下、本実施形態の収音装置10の構成について説明する。
The sound collecting device 10 of the present embodiment timely outputs an instruction to a user such as a person 21 so as to enable sound collecting of a target sound even in a situation where such speech enhancement does not function effectively. .. Hereinafter, the configuration of the sound collecting device 10 of the present embodiment will be described.
1-2.収音装置の構成
図2は、本実施形態に係る収音装置10の構成を例示している。収音装置10は、カメラ11、マイクアレイ12、制御部13、記憶部14、入出力インタフェース部15、音声出力部16、表示部17及びバス18を備える。本実施形態において、収音装置10は、カメラ11及びマイクアレイ12を内蔵して一体化された専用の収音機である。 1-2. Configuration of Sound Collection Device FIG. 2 illustrates the configuration of thesound collection device 10 according to the present embodiment. The sound collecting device 10 includes a camera 11, a microphone array 12, a control unit 13, a storage unit 14, an input / output interface unit 15, an audio output unit 16, a display unit 17, and a bus 18. In the present embodiment, the sound collecting device 10 is a dedicated sound collecting device in which the camera 11 and the microphone array 12 are built and integrated.
図2は、本実施形態に係る収音装置10の構成を例示している。収音装置10は、カメラ11、マイクアレイ12、制御部13、記憶部14、入出力インタフェース部15、音声出力部16、表示部17及びバス18を備える。本実施形態において、収音装置10は、カメラ11及びマイクアレイ12を内蔵して一体化された専用の収音機である。 1-2. Configuration of Sound Collection Device FIG. 2 illustrates the configuration of the
なお、別の構成例では、収音装置1は、カメラ11とマイクアレイ12のいずれか一方又は両方を内蔵していなくてもよい。この場合、収音装置1は、外付けのカメラ11又はマイクアレイ12と電気的に接続される。例えば、収音装置1は、カメラ11を備えたスマートフォン、タブレット端末或いはPCなどの電子機器であって、マイクアレイ12を備えた外部機器と電気的及び機械的に接続されてもよい。
In another configuration example, the sound collecting device 1 does not have to incorporate either one or both of the camera 11 and the microphone array 12. In this case, the sound collecting device 1 is electrically connected to the external camera 11 or the microphone array 12. For example, the sound collecting device 1 may be an electronic device such as a smartphone, a tablet terminal, or a PC equipped with a camera 11, and may be electrically and mechanically connected to an external device including the microphone array 12.
カメラ11は、CCDイメージセンサ、CMOSイメージセンサ、又はNMOSイメージセンサなどのイメージセンサを備える。カメラ11は、映像信号である画像データを生成して、出力する。
The camera 11 includes an image sensor such as a CCD image sensor, a CMOS image sensor, or an NMOS image sensor. The camera 11 generates and outputs image data which is a video signal.
マイクアレイ12は、複数のマイクロホンを備える。マイクアレイ12は、各マイクロホンにおいて、音波を受信して、電気信号である音響信号に変換して出力する。マイクアレイ12は、複数のマイクロホン間の遅延量等を調整することにより収音の方向及び範囲といった指向性を制御する公知のビームフォーミング技術を適用可能である。
The microphone array 12 includes a plurality of microphones. The microphone array 12 receives a sound wave in each microphone, converts it into an acoustic signal which is an electric signal, and outputs the sound wave. The microphone array 12 can be applied with a known beamforming technique that controls directivity such as the direction and range of sound collection by adjusting the amount of delay between a plurality of microphones.
制御部13は、カメラ11から得られた画像データと、マイクアレイ12から得られた音響信号に基づいて、収音方向を決定する。収音方向は、目的音源が存在すると考えられることから、マイクアレイ12から目的音を収音する方向である。目的音源は、目的音を出力する音源であり、例えば人の口元である。制御部13は、収音方向から到来する音を強調するような信号処理を行うことによって、マイクアレイ12が出力した音響信号から目的音を取り出す。制御部13は、半導体素子などで実現可能である。制御部13は、例えば、マイコン、CPU、MPU、DSP、FPGA、又はASICで構成することができる。
The control unit 13 determines the sound collecting direction based on the image data obtained from the camera 11 and the acoustic signal obtained from the microphone array 12. The sound collecting direction is the direction in which the target sound is picked up from the microphone array 12 because it is considered that the target sound source exists. The target sound source is a sound source that outputs the target sound, for example, the mouth of a person. The control unit 13 extracts a target sound from the acoustic signal output by the microphone array 12 by performing signal processing that emphasizes the sound coming from the sound collecting direction. The control unit 13 can be realized by a semiconductor element or the like. The control unit 13 can be composed of, for example, a microcomputer, a CPU, an MPU, a DSP, an FPGA, or an ASIC.
図3は、図2の収音装置10における制御部13の機能的構成を示している。制御部13の機能は、ハードウェアのみで構成してもよいし、ハードウェアとソフトウェアとを組み合わせることにより実現してもよい。制御部13は、例えば図3に示ように、画像状態判定部31、音響状態判定部32、指示決定部33、出力処理部34、状態差分検出部35、方向推定部36、及び信号処理部37といった機能部31~37を含む。各機能部31~37の動作については後述する。
FIG. 3 shows the functional configuration of the control unit 13 in the sound collecting device 10 of FIG. The function of the control unit 13 may be configured only by hardware, or may be realized by combining hardware and software. As shown in FIG. 3, for example, the control unit 13 includes an image state determination unit 31, an acoustic state determination unit 32, an instruction determination unit 33, an output processing unit 34, a state difference detection unit 35, a direction estimation unit 36, and a signal processing unit. Includes functional units 31 to 37 such as 37. The operation of each function unit 31 to 37 will be described later.
記憶部14は、収音装置10の機能を実現するために必要なプログラム及びデータを記憶する記憶媒体である。記憶部14は、例えば、ハードディスク(HDD)、SSD、RAM、DRAM、強誘電体メモリ、フラッシュメモリ、磁気ディスク、又はこれらの組み合わせによって実現できる。カメラ11から得られた画像データ及びマイクアレイ12から得られた音響信号が記憶部14に格納されてもよい。又、記憶部14は、例えば後述する状態テーブル及び指示テーブルを格納する。記憶部14は、収音結果の目的音データを格納してもよい。
The storage unit 14 is a storage medium that stores programs and data necessary for realizing the functions of the sound collecting device 10. The storage unit 14 can be realized by, for example, a hard disk (HDD), an SSD, a RAM, a DRAM, a ferroelectric memory, a flash memory, a magnetic disk, or a combination thereof. The image data obtained from the camera 11 and the acoustic signal obtained from the microphone array 12 may be stored in the storage unit 14. Further, the storage unit 14 stores, for example, a state table and an instruction table, which will be described later. The storage unit 14 may store the target sound data of the sound collection result.
入出力インタフェース部15は、所定の通信規格に準拠して外部機器との通信を行う回路を含む。所定の通信規格は、例えば、LAN、Wi-Fi(登録商標)、Bluetooth(登録商標)、USB、及びHDMI(登録商標)を含む。入出力インタフェース部15は、外部機器に対する収音装置10の入力部の一例であり、また出力部の一例でもある。
The input / output interface unit 15 includes a circuit that communicates with an external device in accordance with a predetermined communication standard. Predetermined communication standards include, for example, LAN, Wi-Fi®, Bluetooth®, USB, and HDMI®. The input / output interface unit 15 is an example of an input unit of the sound collecting device 10 for an external device, and is also an example of an output unit.
音声出力部16は、音響信号を音波に変換するスピーカ装置である。音声出力部16は、音声出力を行って各種の情報を提示する出力部の一例である。
The voice output unit 16 is a speaker device that converts an acoustic signal into sound waves. The audio output unit 16 is an example of an output unit that outputs audio and presents various types of information.
表示部17は、例えば液晶ディスプレイ又は有機ELディスプレイで構成される。表示部17は、各種の情報を表示する出力部の一例である。表示部17は、各種プロジェクタ等であってもよい。
The display unit 17 is composed of, for example, a liquid crystal display or an organic EL display. The display unit 17 is an example of an output unit that displays various types of information. The display unit 17 may be various projectors or the like.
バス18は、カメラ11、マイクアレイ12、制御部13、記憶部14、入出力インタフェース部15、音声出力部16、及び表示部17を電気的に接続する信号線である。
The bus 18 is a signal line that electrically connects the camera 11, the microphone array 12, the control unit 13, the storage unit 14, the input / output interface unit 15, the audio output unit 16, and the display unit 17.
制御部13が画像データをカメラ11から取得又は記憶部14から取り出すときは、制御部13が画像データの入力部に相当する。制御部13が音響信号をマイクアレイ12から取得又は記憶部14から取り出すときは、制御部13が音響信号の入力部に相当する。
When the control unit 13 acquires image data from the camera 11 or retrieves it from the storage unit 14, the control unit 13 corresponds to an image data input unit. When the control unit 13 acquires the acoustic signal from the microphone array 12 or takes it out from the storage unit 14, the control unit 13 corresponds to the input unit of the acoustic signal.
以上の各構成例の収音装置10において、制御部13が画像データを記憶部14から取り出すときは、制御部13又はバス18が画像データの入力部に相当する。制御部13が音響信号を記憶部14から取り出すときは、制御部13又はバス18が音響信号の入力部に相当する。入出力インタフェース部15が収音装置10に外付けされたカメラ11から画像データを入力するときは、入出力インタフェース部15が画像データの入力部に相当する。入出力インタフェース部15が収音装置10に外付けされたマイクアレイ12から音響信号を入力するときは、入出力インタフェース部15が音響信号の入力部に相当する。
In the sound collecting device 10 of each of the above configuration examples, when the control unit 13 takes out the image data from the storage unit 14, the control unit 13 or the bus 18 corresponds to the image data input unit. When the control unit 13 extracts the acoustic signal from the storage unit 14, the control unit 13 or the bus 18 corresponds to the input unit of the acoustic signal. When the input / output interface unit 15 inputs image data from the camera 11 externally attached to the sound collecting device 10, the input / output interface unit 15 corresponds to the image data input unit. When the input / output interface unit 15 inputs an acoustic signal from the microphone array 12 externally attached to the sound collecting device 10, the input / output interface unit 15 corresponds to the audio signal input unit.
また、以上の構成例では、収音装置10が、音声出力部16、表示部17及び入出力インタフェース部15といった複数の出力部を備える例を説明したが、収音装置10は、こうした出力部のいずれか1つを備えてもよい。また、出力部は上記の例に限らず、例えば点滅ランプ、或いは報知ブザー等であってもよい。
Further, in the above configuration example, an example in which the sound collecting device 10 includes a plurality of output units such as an audio output unit 16, a display unit 17, and an input / output interface unit 15 has been described, but the sound collecting device 10 has such an output unit. Any one of the above may be provided. Further, the output unit is not limited to the above example, and may be, for example, a blinking lamp, a notification buzzer, or the like.
2.動作
以上のように構成される収音装置10の動作について、以下説明する。 2. Operation The operation of thesound collecting device 10 configured as described above will be described below.
以上のように構成される収音装置10の動作について、以下説明する。 2. Operation The operation of the
図4は、本実施形態に係る収音装置10の全体動作を例示するフローチャートである。図5は、収音装置10の動作の概要を説明するための図である。図4のフローチャートに示す各処理は、収音装置10の制御部13によって実行される。
FIG. 4 is a flowchart illustrating the overall operation of the sound collecting device 10 according to the present embodiment. FIG. 5 is a diagram for explaining an outline of the operation of the sound collecting device 10. Each process shown in the flowchart of FIG. 4 is executed by the control unit 13 of the sound collecting device 10.
本実施形態の収音装置10において、制御部13は、カメラ11からの画像データに基づいて、画像状態判定部31として機能する画像状態の判定処理を実行する(S1)。また、制御部13は、マイクアレイ12からの音響信号に基づいて、音響状態判定部32として機能する音響状態の判定処理を実行する(S2)。
In the sound collecting device 10 of the present embodiment, the control unit 13 executes an image state determination process that functions as an image state determination unit 31 based on the image data from the camera 11 (S1). Further, the control unit 13 executes an acoustic state determination process that functions as the acoustic state determination unit 32 based on the acoustic signal from the microphone array 12 (S2).
ステップS1,S2では、画像データが示す画像と音響信号が示す音響とに含まれるユーザ及び環境等の現在の状態が判定される。ステップS1,S2の処理の詳細は後述する。なお、ステップS1,S2の処理を実行する順番は特に限定されず、図示した順番とは逆順であってもよいし、並列的に実行されてもよい。
In steps S1 and S2, the current state of the user, environment, etc. included in the image indicated by the image data and the acoustic indicated by the acoustic signal is determined. Details of the processing in steps S1 and S2 will be described later. The order in which the processes of steps S1 and S2 are executed is not particularly limited, and may be in the reverse order of the illustrated order or may be executed in parallel.
次に、制御部13は、判定した画像と音響の現在状態が、ユーザに対する指示の対象とする所定の状態であるか否かを判断する(S3)。指示対象の状態の一例を、図5(A)に例示する。
Next, the control unit 13 determines whether or not the determined current state of the image and the sound is a predetermined state to be instructed to the user (S3). An example of the state of the referent is illustrated in FIG. 5 (A).
図5(A)では、発話中の人物21(即ちユーザ)が、収音装置10のカメラ11及びマイクアレイ12とは異なる向きに顔を向けた状態を例示している。このような状態においては、画像データと音響信号に基づき発話を収音するために最適な方向を求められない、或いは最適な方向からの音声を強調する処理を音響信号に施したとしても、目的音の明瞭な収音が困難になるといった事態に陥ってしまう。また、こうした事態は、人物21の顔の向きだけでなく、例えば騒音源25からの雑音など、収音の障害となる様々な要因で生じることが考えられる。
FIG. 5A illustrates a state in which the speaking person 21 (that is, the user) faces a different direction from the camera 11 and the microphone array 12 of the sound collecting device 10. In such a state, even if the optimum direction cannot be obtained in order to pick up the utterance based on the image data and the sound signal, or the sound from the optimum direction is emphasized, the sound signal is intended. It becomes difficult to collect the sound clearly. Further, it is conceivable that such a situation occurs not only due to the orientation of the face of the person 21, but also due to various factors that hinder sound collection, such as noise from the noise source 25.
そこで、本実施形態の収音装置10は、上記のような事態に陥っているときには状態の改善をユーザに指示するべく、画像と音響の現在状態に基づきステップS3の判断を行う。ステップS3の指示対象の状態は、目的音の収音が困難であることから、ユーザに指示すべきと考えられる種々の状態に設定できる。ステップS3の詳細は後述する。
Therefore, the sound collecting device 10 of the present embodiment makes a determination in step S3 based on the current state of the image and sound in order to instruct the user to improve the state when the above situation occurs. Since it is difficult to collect the target sound, the state of the referent in step S3 can be set to various states that should be instructed to the user. The details of step S3 will be described later.
制御部13は、画像と音響の現在状態が、指示対象の状態であると判断した場合(S3でYES)、ユーザに対する指示を介した収音処理を実行する(S4)。ステップS4の処理の概要を、図5(B)を用いて説明する。
When the control unit 13 determines that the current state of the image and the sound is the state to be instructed (YES in S3), the control unit 13 executes the sound collection process via the instruction to the user (S4). The outline of the process of step S4 will be described with reference to FIG. 5 (B).
図5(B)は、図5(A)の状態から指示が出力された場合を例示する。本例では、収音装置10の音声出力部16が、「カメラの方を向いて発話してください」という指示の音声を出力している。こうした指示の出力によると、ユーザは、発話の収音に不具合があったこと及びその理由を把握できる。
FIG. 5 (B) illustrates a case where an instruction is output from the state of FIG. 5 (A). In this example, the voice output unit 16 of the sound collecting device 10 outputs the voice of the instruction "Please face the camera and speak." According to the output of such an instruction, the user can grasp that there is a problem in the sound collection of the utterance and the reason.
さらに、本例において発話を収音させたい人物21は、出力された指示に従って、顔の向きを改善して再び、発話すると考えられる。そこで、本実施形態の指示を介した収音処理(S4)では、指示前後の状態の差分に基づいて、再度の発話を収音する精度を向上する。ステップS4の処理の詳細は後述する。
Furthermore, it is considered that the person 21 who wants to pick up the utterance in this example improves the direction of the face and speaks again according to the output instruction. Therefore, in the sound collection process (S4) via the instruction of the present embodiment, the accuracy of collecting the re-uttered utterance is improved based on the difference between the states before and after the instruction. Details of the process in step S4 will be described later.
図4に戻り、制御部13は、画像と音響の現在状態が、指示対象の状態でないと判断した場合(S3でNO)、特に指示を介さない収音処理を実行する(S5)。ステップS5の処理によると、例えばユーザに指示しなくても収音可能な状態であれば、不要な指示を行わずに目的音を収音することができる。ステップS5の処理の詳細は後述する。
Returning to FIG. 4, when the control unit 13 determines that the current state of the image and the sound is not the state of the referent (NO in S3), the control unit 13 executes the sound collection process without any particular instruction (S5). According to the process of step S5, for example, if the sound can be picked up without instructing the user, the target sound can be picked up without giving unnecessary instructions. Details of the process in step S5 will be described later.
制御部13は、ステップS4又はステップS5の収音処理を行った後に、例えば周期的に、ステップS1以降の処理を繰り返す。
After performing the sound collection processing in step S4 or step S5, the control unit 13 periodically repeats the processing from step S1 onward, for example.
以上の動作によると、収音装置10は、画像状態および音響状態を随時判定し(S1,S2)、解消すべき状態が検知されると(S3でYES)、ユーザに対する指示を介した収音処理(S4)を行う。これにより、ユーザに対する指示を用いて、目的音を収音し難い状態を解消させてから、収音が行えて、時間応答の良い音声強調を実現することできる。
According to the above operation, the sound collecting device 10 determines the image state and the sound state at any time (S1, S2), and when the state to be resolved is detected (YES in S3), the sound collecting device 10 picks up the sound via an instruction to the user. The process (S4) is performed. As a result, it is possible to eliminate the state in which it is difficult to collect the target sound by using the instruction to the user, and then collect the sound, and realize speech enhancement with a good time response.
また、収音装置10は、特に解消すべき状態でなければ(S3でNO)、指示を介さない収音処理を行う(S5)。これにより、不要な指示が頻発してユーザにとって煩わしくなるような事態も回避できる。以上の収音装置10の動作の詳細について、以下説明する。
Further, the sound collecting device 10 performs sound collecting processing without instructing (S5) unless it is in a state to be particularly resolved (NO in S3). As a result, it is possible to avoid a situation in which unnecessary instructions occur frequently and become annoying to the user. The details of the operation of the above sound collecting device 10 will be described below.
2-1.画像状態の判定処理(S1)
図4のステップS1の処理の詳細を、図6,7を用いて説明する。 2-1. Image state determination process (S1)
The details of the process of step S1 of FIG. 4 will be described with reference to FIGS. 6 and 7.
図4のステップS1の処理の詳細を、図6,7を用いて説明する。 2-1. Image state determination process (S1)
The details of the process of step S1 of FIG. 4 will be described with reference to FIGS. 6 and 7.
図6は、収音装置10における画像状態の判定処理(S1)を例示するフローチャートである。制御部13は、画像状態判定部31として機能し、図6に例示するフローチャートの各処理を実行する。
FIG. 6 is a flowchart illustrating the image state determination process (S1) in the sound collecting device 10. The control unit 13 functions as an image state determination unit 31 and executes each process of the flowchart illustrated in FIG.
まず、画像状態判定部31としての制御部13は、カメラ11からバス18を介して画像データを入力する(S11)。カメラ11は、例えば所定のフレームレートにおいて、撮像された画像を示す画像データを生成する。画像データは、例えばフレーム毎に順次、入力される。
First, the control unit 13 as the image state determination unit 31 inputs image data from the camera 11 via the bus 18 (S11). The camera 11 generates image data indicating the captured image, for example, at a predetermined frame rate. Image data is input sequentially for each frame, for example.
次に、制御部13は、入力した画像データに基づき、発話に関する画像の各種状態を検出する(S12)。例えば、制御部13は、画像データが示す画像において、まず人物を検出し、人物が検出された場合には画像における人物の領域において顔を検出する。また、制御部13は、顔が検出された場合には画像における顔の向き、及び口元などのパーツを検出する。更に、複数フレーム分の検出結果に基づき、制御部13は、例えば口元などの対象物の動きを検出する。こうした各種の検出対象は、それぞれ画像における目的音源の配置を示す情報の一例である。
Next, the control unit 13 detects various states of the image related to the utterance based on the input image data (S12). For example, the control unit 13 first detects a person in the image indicated by the image data, and when the person is detected, detects a face in the area of the person in the image. Further, when the face is detected, the control unit 13 detects the orientation of the face in the image and parts such as the mouth. Further, the control unit 13 detects the movement of an object such as the mouth based on the detection results for a plurality of frames. Each of these various detection targets is an example of information indicating the arrangement of the target sound source in the image.
上記のようなステップS12の各種検出は、例えば各部の特徴量を識別する識別器を用いて行える。例えば、機械学習が予め為された弱識別器を用いることができる(例えば非特許文献1参照)。検出された対象物の領域は、例えば画像データが示す画像における水平座標及び垂直座標で表すことができる。
The various detections in step S12 as described above can be performed, for example, by using a classifier that identifies the feature amount of each part. For example, a weak discriminator that has been machine-learned in advance can be used (see, for example, Non-Patent Document 1). The detected target area can be represented by, for example, horizontal coordinates and vertical coordinates in the image indicated by the image data.
図7は、画像データvにおける検出単位の領域r(θ,φ)を示している。ステップS12では、例えば水平角度θ及び垂直角度φに応じた座標において、画像データvが示す画像を複数の領域r(θ,φ)に分割し、領域r(θ,φ)毎に検出結果が記録される。なお、図7では、格子状に分割し、それぞれの軸をθ,φに対応付けたが、全周カメラを使用した場合は、円周状にθ,φを対応付けるようにしてもよい。
FIG. 7 shows the region r (θ, φ) of the detection unit in the image data v. In step S12, for example, at the coordinates corresponding to the horizontal angle θ and the vertical angle φ, the image indicated by the image data v is divided into a plurality of regions r (θ, φ), and the detection result is obtained for each region r (θ, φ). Recorded. In FIG. 7, the grid is divided and the respective axes are associated with θ and φ. However, when an all-around camera is used, θ and φ may be associated with each other in a circumferential shape.
図6に戻り、制御部13は、画像から検出された各種情報に基づいて、例えば予め設定された分類クラスの中で現在の画像状態を分類する(S13)。ステップS13の分類クラスは、例えば、画像上の顔の口元などの検出結果の観点において良好な画像状態から三段階に設定されるAクラス、Bクラス及びCクラスを用いる(図9参照)。なお、画像状態の分類は三段階でなくてもよい。
Returning to FIG. 6, the control unit 13 classifies the current image state in, for example, a preset classification class based on various information detected from the image (S13). As the classification class in step S13, for example, A class, B class, and C class, which are set in three stages from a good image state in terms of the detection result such as the mouth of the face on the image, are used (see FIG. 9). The classification of the image state does not have to be in three stages.
画像状態に関するAクラスは、顔とその口元が検出され、且つ、顔の向きが前向き、即ちカメラ11に向かって正対した向きに検出された画像状態を含む。Aクラスの画像状態によると、画像における口元の動きに基づいて発話の有無を判定可能であり、且つ、発話があったときには画像のみで発話を収音する方向を決定できると考えられる。
Class A regarding the image state includes an image state in which the face and its mouth are detected and the face is detected in the forward direction, that is, in the direction facing the camera 11. According to the A-class image state, it is considered that the presence or absence of an utterance can be determined based on the movement of the mouth in the image, and when there is an utterance, the direction in which the utterance is picked up can be determined only by the image.
画像のBクラスは、顔とその口元は検出されたが、顔の向きが前向きとは異なる向き(例えば横向き)に検出された画像状態を含む。Bクラスの画像状態によると、口元の動きに基づき発話の有無の判定は実行できても、画像のみでは発話の収音方向を決定し難かったり、発話が収音可能か判定し難かったりすることが考えられる。
The B class of the image includes an image state in which the face and its mouth are detected, but the direction of the face is different from the forward direction (for example, sideways). According to the B-class image state, it is possible to determine the presence or absence of an utterance based on the movement of the mouth, but it is difficult to determine the sound collecting direction of the utterance from the image alone, or it is difficult to determine whether the utterance can be collected. Can be considered.
画像のCクラスは、顔や口元が検出されなかった状態である。Cクラスの画像状態によると、口元の動きに基づく発話の有無の判定は、実行不可である。Cクラスの画像状態としては、例えば、人物が検出されたとしても、人物が口元にマスクを掛けていたり、顔が後向きであったり、人物とカメラ11間の距離が遠過ぎたり、画像が暗過ぎたりする場合が考えられる。Cクラスは、必ずしも人物が検出された場合に限らず、人物が検出されなかった場合を含んでもよい。
The C class of the image is a state in which the face and mouth were not detected. According to the C-class image state, it is not feasible to determine the presence or absence of utterance based on the movement of the mouth. As for the C-class image state, for example, even if a person is detected, the person wears a mask on the mouth, the face is facing backward, the distance between the person and the camera 11 is too long, or the image is dark. It may pass. The C class is not necessarily limited to the case where a person is detected, and may include the case where a person is not detected.
例えば分類された画像状態がAクラス又はBクラスである場合、即ちCクラスではない場合(S14でNO)、制御部13は、画像状態の検出結果に基づいて、発話の有無を判定する(S15)。具体的に、制御部13は、画像状態の口元の動きが発話であるかどうかの尤度を示す発話スコアを算出し、画像状態の発話スコアを所定のしきい値と比較する。当該しきい値は、口元の動きが発話であると確信できる程度に大きい発話スコアの基準を示す。
For example, when the classified image state is A class or B class, that is, when it is not C class (NO in S14), the control unit 13 determines whether or not there is an utterance based on the detection result of the image state (S15). ). Specifically, the control unit 13 calculates an utterance score indicating the likelihood of whether or not the movement of the mouth in the image state is an utterance, and compares the utterance score in the image state with a predetermined threshold value. The threshold indicates a criterion for an utterance score that is large enough to be confident that the movement of the mouth is an utterance.
制御部13は、画像状態の発話スコアがしきい値以上である場合に発話が有るという判定(S15でYES)の結果として、画像側の発話フラグを「ON」に設定する(S16)。画像側の発話フラグは、画像状態による発話の有無の判定結果を管理するフラグであり、例えば記憶部14に記憶される。
The control unit 13 sets the utterance flag on the image side to "ON" as a result of determining that there is an utterance when the utterance score in the image state is equal to or higher than the threshold value (YES in S15) (S16). The utterance flag on the image side is a flag that manages the determination result of the presence or absence of utterance depending on the image state, and is stored in the storage unit 14, for example.
制御部13は、画像状態の発話スコアがしきい値未満である場合に発話が無いという判定(S15でNO)の結果として、画像側の発話フラグを「OFF」に設定する(S17)。画像側の発話フラグの設定(S16,S17)後、制御部13は、図4のステップS1の処理を終了し、例えばステップS2に進む。
The control unit 13 sets the utterance flag on the image side to "OFF" as a result of the determination that there is no utterance (NO in S15) when the utterance score in the image state is less than the threshold value (S17). After setting the utterance flag on the image side (S16, S17), the control unit 13 ends the process of step S1 of FIG. 4, and proceeds to, for example, step S2.
一方、分類された画像状態がCクラスである場合(S14でYES)、制御部13は、画像状態に基づく発話の判定(S15~S17)を行わず、ステップS1の処理を終了する。
On the other hand, when the classified image state is C class (YES in S14), the control unit 13 does not determine the utterance based on the image state (S15 to S17), and ends the process of step S1.
以上の画像状態の判定処理(S1)によると、カメラ11から順次、入力される画像データに基づいて(S11)、画像において人物の口元などの音源の配置を示す各種情報が検出され(S12)、現在の画像状態が判定される(S13~S17)。
According to the above image state determination process (S1), various information indicating the arrangement of sound sources such as the mouth of a person is detected in the image based on the image data sequentially input from the camera 11 (S12). , The current image state is determined (S13 to S17).
2-2.音響状態の判定処理(S2)
図4のステップS2の処理の詳細を、図8を用いて説明する。 2-2. Acoustic state determination process (S2)
The details of the process of step S2 of FIG. 4 will be described with reference to FIG.
図4のステップS2の処理の詳細を、図8を用いて説明する。 2-2. Acoustic state determination process (S2)
The details of the process of step S2 of FIG. 4 will be described with reference to FIG.
図8は、音響状態の判定処理(S2)を例示するフローチャートである。制御部13は、音響状態判定部32として機能し、図8に例示するフローチャートの各処理を実行する。
FIG. 8 is a flowchart illustrating the acoustic state determination process (S2). The control unit 13 functions as an acoustic state determination unit 32, and executes each process of the flowchart illustrated in FIG.
まず、音響状態判定部32としての制御部13は、マイクアレイ12からバス18を介して音響信号を取得する(S21)。マイクアレイ12は順次、外部から受信される音波に応じて音響信号を生成する。ステップS21では、例えば、図6のステップS11で取得される画像データと同期をとって音響信号が取得される。
First, the control unit 13 as the acoustic state determination unit 32 acquires an acoustic signal from the microphone array 12 via the bus 18 (S21). The microphone array 12 sequentially generates an acoustic signal according to a sound wave received from the outside. In step S21, for example, the acoustic signal is acquired in synchronization with the image data acquired in step S11 of FIG.
次に、制御部13は、取得した音響信号に基づき、発話に関する音響の各種状態を検出する(S22)。ステップS22の検出は、例えば図7の各検出単位の領域r(θ,φ)に対応する方向(θ,φ)毎に行われる。
Next, the control unit 13 detects various acoustic states related to utterance based on the acquired acoustic signal (S22). The detection in step S22 is performed, for example, in each direction (θ, φ) corresponding to the region r (θ, φ) of each detection unit in FIG. 7.
例えば、制御部13はステップS22において、CSP(Cross-Power Spectrum Phase Analysis)法又はMUSIC(Multiple Signal Classification)法などにより、音源の位置或いは音源から音が到来する方向を推定する演算を行う。また、制御部13は、例えば予め学習させた音声GMM(Gausian Mixture Model)及び非音声GMMを用いて、目的音である人の声が有する周波数帯の音声を検出したり、目的音とは異なる雑音を検出したりする。
For example, in step S22, the control unit 13 performs an operation of estimating the position of the sound source or the direction in which the sound arrives from the sound source by the CSP (Cross-Power Spectrum Phase Analysis) method or the MUSIC (Multiple Signal Classification) method. Further, the control unit 13 detects a voice in the frequency band of the human voice, which is the target sound, by using, for example, a voice GMM (Gausian Mixture Model) and a non-voice GMM trained in advance, or is different from the target sound. Detects noise.
音響の各種検出結果に基づいて、制御部13は、例えば画像状態の分類(S13)と同様に、所定の分類クラスの中で現在の音響状態を分類する(S23)。ステップS23の分類クラスは、例えば、雑音の検出結果の観点において良好な音響状態から三段階に設定されるA,B,Cクラスを用いる(図9参照)。
Based on various acoustic detection results, the control unit 13 classifies the current acoustic state in a predetermined classification class, for example, in the same manner as the image state classification (S13) (S23). As the classification class in step S23, for example, the A, B, and C classes that are set in three stages from the good acoustic state from the viewpoint of the noise detection result are used (see FIG. 9).
音響のAクラスは、例えば雑音が検出された信号レベルが、発話などの目的音の検出に干渉し得る雑音の下限レベルを下回る音響状態を含む。Aクラスは、周波数および位相(或いは到来方向)等の観点から、雑音と目的音とが分離可能な音響状態を含んでもよい。Aクラスの音響状態によると、目的音の検出結果から発話の有無を判定可能であり、且つ、発話があったときには画像状態を用いなくても発話の収音方向を決定できると考えられる。
Acoustic A class includes, for example, an acoustic state in which the signal level at which noise is detected is lower than the lower limit level of noise that can interfere with the detection of a target sound such as utterance. The A class may include an acoustic state in which noise and target sound can be separated from the viewpoint of frequency, phase (or arrival direction), and the like. According to the A-class acoustic state, it is considered that the presence or absence of an utterance can be determined from the detection result of the target sound, and when there is an utterance, the sound collection direction of the utterance can be determined without using the image state.
音響のBクラスは、例えば雑音の信号レベルが、下限レベルと上限レベルとの間で検出された音響状態を含む。雑音の上限レベルは、信号レベルの観点から雑音と目的音とが分離可能な上限を示し、下限レベルよりも大きく設定される。Bクラスは、例えば雑音の周波数帯が人声の周波数帯の近傍であるなど、周波数および位相等の観点から雑音と目的音とが分離し難い音響状態を含んでもよい。Bクラスの音響状態によると、発話の音声の有無は判定できても、画像状態を用いなければ発話の収音方向を決定し難かったり、発話の収音可否を判定し難かったりすることが考えられる。
The acoustic B class includes, for example, an acoustic state in which the noise signal level is detected between the lower limit level and the upper limit level. The upper limit level of noise indicates an upper limit in which noise and target sound can be separated from the viewpoint of signal level, and is set larger than the lower limit level. The B class may include an acoustic state in which the noise and the target sound are difficult to separate from the viewpoint of frequency, phase, etc., for example, the frequency band of the noise is near the frequency band of the human voice. According to the B-class acoustic state, it may be possible to determine the presence or absence of utterance sound, but it may be difficult to determine the direction of utterance sound collection without using the image state, or it may be difficult to determine whether or not utterance sound collection is possible. Be done.
音響のCクラスは、例えば雑音の信号レベルが、上限レベルを上回って検出された音響状態を含む。Cクラスは、周波数および位相等の観点からも雑音と目的音とが分離不可の音響状態を含んでもよい。Cクラスの音響状態によると、発話の音声の有無は判定不可である。
Acoustic C class includes, for example, an acoustic state detected when the signal level of noise exceeds the upper limit level. The C class may include an acoustic state in which noise and target sound cannot be separated from the viewpoint of frequency, phase, and the like. According to the C-class acoustic state, it is not possible to determine the presence or absence of spoken voice.
例えば分類された音響状態がAクラス又はBクラスであり、Cクラスではない場合(S24でNO)、制御部13は、音響状態の検出結果に基づいて、発話の有無を判定する(S25)。具体的に、制御部13は、検出された音声が人声であるかどうかの尤度を示す発話スコアを算出し、音響状態の発話スコアを所定のしきい値と比較する。当該しきい値は、音声が人声であると確信できる程度に大きい発話スコアの基準を示す。
For example, when the classified acoustic state is A class or B class and not C class (NO in S24), the control unit 13 determines whether or not there is an utterance based on the detection result of the acoustic state (S25). Specifically, the control unit 13 calculates an utterance score indicating the likelihood of whether or not the detected voice is a human voice, and compares the utterance score in the acoustic state with a predetermined threshold value. The threshold indicates a measure of speech score that is large enough to be confident that the voice is human voice.
制御部13は、音響状態の発話スコアがしきい値以上である場合には発話が有るという判定(S25でYES)の結果として、音響側の発話フラグを「ON」に設定する(S26)。音響側の発話フラグは、音響状態による発話の有無の判定結果を管理するフラグであり、例えば画像側の発話フラグとは別途、記憶部14に記憶される。
The control unit 13 sets the utterance flag on the acoustic side to "ON" as a result of the determination that there is utterance (YES in S25) when the utterance score in the acoustic state is equal to or higher than the threshold value (S26). The utterance flag on the acoustic side is a flag that manages the determination result of the presence or absence of utterance depending on the acoustic state, and is stored in the storage unit 14 separately from the utterance flag on the image side, for example.
制御部13は、音響状態の発話スコアがしきい値未満である場合には発話が無いという判定(S25でNO)の結果として、音響側の発話フラグを「OFF」に設定する(S27)。音響側の発話フラグの設定(S26,S27)後、制御部13は、図4のステップS2の処理を終了し、例えばステップS3に進む。
The control unit 13 sets the utterance flag on the acoustic side to "OFF" as a result of the determination that there is no utterance (NO in S25) when the utterance score in the acoustic state is less than the threshold value (S27). After setting the utterance flag on the acoustic side (S26, S27), the control unit 13 ends the process of step S2 of FIG. 4, and proceeds to, for example, step S3.
一方、分類された音響状態がCクラスである場合(S24でYES)、制御部13は、音響状態に基づく発話の判定(S25~S27)を行わず、ステップS2の処理を終了する。
On the other hand, when the classified acoustic state is C class (YES in S24), the control unit 13 does not determine the utterance based on the acoustic state (S25 to S27), and ends the process of step S2.
以上の音響状態の判定処理によると、マイクアレイ12から順次、入力される音響信号に基づいて(S21)、目的音および雑音が検出され(S22)、現在の音響状態が判定される(S23~S27)。
According to the above acoustic state determination process, the target sound and noise are detected (S22) based on the acoustic signals sequentially input from the microphone array 12, and the current acoustic state is determined (S23 to S23). S27).
2-3.指示対象の状態について(S3)
図4のステップS3の判断の詳細と指示対象の状態について、図9を用いて説明する。 2-3. About the state of the referent (S3)
The details of the determination in step S3 of FIG. 4 and the state of the referent will be described with reference to FIG.
図4のステップS3の判断の詳細と指示対象の状態について、図9を用いて説明する。 2-3. About the state of the referent (S3)
The details of the determination in step S3 of FIG. 4 and the state of the referent will be described with reference to FIG.
図9は、収音装置10における状態テーブルD1を例示する図である。状態テーブルD1は、画像状態と音響状態とに関連付けて、指示対象の状態及び対象外の状態を管理するデータテーブルであり、例えば記憶部14に予め格納されている。図中で各発話フラグに関する「-」は、発話の有無が判定されていないことを示す。
FIG. 9 is a diagram illustrating a state table D1 in the sound collecting device 10. The state table D1 is a data table that manages the state of the referent and the state of the non-target in association with the image state and the acoustic state, and is stored in advance in, for example, the storage unit 14. In the figure, "-" for each utterance flag indicates that the presence or absence of utterance has not been determined.
本実施形態において、ステップS3の判断は、例えば図9に示すように、画像状態と、音響状態とを組み合わせることによって行われる。制御部13は、状態テーブルD1を参照して、画像状態の判定結果(S1)及び音響状態の判定結果(S2)による現在状態が、指示対象の状態であるか否かを判断する(S3)。
In the present embodiment, the determination in step S3 is performed by combining the image state and the acoustic state, for example, as shown in FIG. The control unit 13 refers to the state table D1 and determines whether or not the current state based on the image state determination result (S1) and the acoustic state determination result (S2) is the referent state (S3). ..
図9に例示する状態テーブルD1は、画像状態のA,B,Cクラス及び画像側の発話フラグのON/OFFと、音響状態のA,B,Cクラス及び音響側の発話フラグのON/OFFとの組み合わせにより、5段階の状態1~5を規定している。本実施形態において、状態改善の指示対象とする状態は状態3であり(S3でYES)、その他の状態1,2,4,5は指示の対象外である(S3でNO)。なお、状態は5段階でなくてもよい。
In the state table D1 illustrated in FIG. 9, the image state A, B, C class and the utterance flag on the image side are turned ON / OFF, and the acoustic state A, B, C class and the utterance flag on the acoustic side are turned ON / OFF. In combination with, the five-stage states 1 to 5 are defined. In the present embodiment, the state to be instructed to improve the state is state 3 (YES in S3), and the other states 1, 2, 4 and 5 are not the target of the instruction (NO in S3). The state does not have to be in five stages.
状態1は、画像状態と音響状態の双方がAクラスの状態である。状態1によると、画像状態と音響状態との双方において発話を判定可能であることから、特に状態を改善するための指示を用いなくても、発話を収音することができると考えられる。
State 1 is a state in which both the image state and the acoustic state are A class. According to the state 1, since the utterance can be determined in both the image state and the acoustic state, it is considered that the utterance can be picked up without using an instruction for improving the state.
状態2は、画像状態と音響状態の一方がAクラスであり、他方がBクラス又はCクラスの状態である。状態2によると、画像状態と音響状態の一方において発話を判定可能であることから、特に指示を用いなくても、発話を収音可能と考えられる。
State 2 is a state in which one of the image state and the acoustic state is A class and the other is B class or C class. According to the state 2, since the utterance can be determined in either the image state or the acoustic state, it is considered that the utterance can be picked up without using any particular instruction.
状態3は、画像状態と音響状態の双方がBクラス又はCクラスであり、少なくとも一方がBクラスの状態において、画像側と音響側の発話フラグの少なくとも一方が「ON」の状態である。状態3によると、発話が有ると判定されているにも拘らず、画像状態と音響状態の双方において、発話の収音方向の決定等が困難になっている。よって、このような収音方向を決定できない障害がある状態は解消するように指示を出力することが、有用であると考えられる。
In state 3, both the image state and the acoustic state are B class or C class, at least one is B class, and at least one of the utterance flags on the image side and the acoustic side is "ON". According to the state 3, although it is determined that there is an utterance, it is difficult to determine the sound collecting direction of the utterance in both the image state and the acoustic state. Therefore, it is considered useful to output an instruction to eliminate such a state in which there is an obstacle in which the sound collection direction cannot be determined.
状態4は、状態3と同様の分類クラスにおいて、画像側と音響側の発話フラグの双方が「ON」ではなく、少なくとも一方が「OFF」の状態である。状態4によると、画像状態と音響状態の双方が収音方向等を判定し難いものの、発話が無いと判定されていることから、特に指示は出力しなくてもよいと考えられる。
State 4 is a state in which both the utterance flags on the image side and the sound side are not "ON" but at least one is "OFF" in the same classification class as state 3. According to the state 4, it is difficult to determine the sound collecting direction and the like in both the image state and the acoustic state, but since it is determined that there is no utterance, it is considered that no particular instruction needs to be output.
状態5は、画像状態と音響状態の双方がCクラスの状態である。状態5によると、発話の有無も収音方向等も判定不可である。このため、発話が為されておらず特に指示が要らない場合もあり得ると共に、状態を改善するために適切な内容を指定するような指示は出力し難いことが考えられる。
State 5 is a C-class state in both the image state and the acoustic state. According to the state 5, it is impossible to determine whether or not there is an utterance, the sound collecting direction, and the like. For this reason, it is possible that no particular instruction is required because no utterance has been made, and it is considered difficult to output an instruction that specifies appropriate content in order to improve the state.
以上のような状態1~5のうちの状態3を指示対象の状態とすることにより、発話中であることが判明しているにも拘わらず、収音方向を決定できずに目的音の収音が為されないような事態を回避することができる。なお、以上に説明した指示対象及び対象外の状態は一例である。指示対象の状態は、特に上記の状態3に限らず、収音方向の決定を困難にする各種の障害を含む状態であってもよく、収音装置10を適用する場面等に応じて適宜、設定可能である。
By setting the state 3 of the above states 1 to 5 as the referent state, the sound collection direction cannot be determined and the target sound is collected even though it is known that the utterance is in progress. It is possible to avoid a situation where no sound is produced. The referent and non-referent states described above are examples. The state of the referent is not limited to the above state 3, and may include various obstacles that make it difficult to determine the sound collecting direction. It can be set.
2-4.指示を介した収音処理(S4)
図4のステップS4の処理の詳細を、図10,11を用いて説明する。 2-4. Sound collection processing via instructions (S4)
The details of the process of step S4 of FIG. 4 will be described with reference to FIGS. 10 and 11.
図4のステップS4の処理の詳細を、図10,11を用いて説明する。 2-4. Sound collection processing via instructions (S4)
The details of the process of step S4 of FIG. 4 will be described with reference to FIGS. 10 and 11.
図10は、指示を介した収音処理(S4)を例示するフローチャートである。本フローチャートに示す処理は、現在状態が、図9の状態3である場合(S3でYES)に実行される。
FIG. 10 is a flowchart illustrating the sound collection process (S4) via instructions. The process shown in this flowchart is executed when the current state is the state 3 of FIG. 9 (YES in S3).
まず、制御部13は、指示決定部33として機能し、ステップS1~S3で得られた現在状態に基づいて、指示に含める情報の内容を決定する(S41)。ステップS41の処理は、例えば図11に示すような指示テーブルD2を参照して行われる。指示テーブルD2は、指示に含める情報を示す指示内容を管理するデータテーブルであり、予め記憶部14に格納されている。
First, the control unit 13 functions as an instruction determination unit 33, and determines the content of the information to be included in the instruction based on the current state obtained in steps S1 to S3 (S41). The process of step S41 is performed with reference to the instruction table D2 as shown in FIG. 11, for example. The instruction table D2 is a data table that manages instruction contents indicating information to be included in the instruction, and is stored in the storage unit 14 in advance.
図11に例示する指示テーブルD2は、指示内容を識別するID番号と、指示内容とを関連付けて記録している。ステップS41において、指示決定部33としての制御部13は、例えば図4のステップS3で判定された現在状態に基づいて、指示テーブルD2の中から、指示内容として決定するID番号を選択する。指示決定部33は、ステップS1における各種の画像状態の検出結果(図6のS12)、及びステップS2における各種の音響状態の検出結果(図8のS22)を適宜、用いてもよい。また、状態テーブルD1(図9)を用いて、状態3に分類される各々の画像状態及び音響状態の組み合わせから指示内容が決定されてもよい。
The instruction table D2 illustrated in FIG. 11 records the ID number that identifies the instruction content and the instruction content in association with each other. In step S41, the control unit 13 as the instruction determination unit 33 selects an ID number to be determined as the instruction content from the instruction table D2, for example, based on the current state determined in step S3 of FIG. The instruction determination unit 33 may appropriately use the detection results of various image states in step S1 (S12 in FIG. 6) and the detection results of various acoustic states in step S2 (S22 in FIG. 8). Further, the instruction content may be determined from the combination of each image state and acoustic state classified into the state 3 by using the state table D1 (FIG. 9).
例えば、画像状態及び音響状態の双方がBクラスの場合に、画像上で顔の向きが前向きに映っていないことから発話を収音し難い場合が考えられる(図5(A)参照)。こうした場合、指示決定部33は、ID番号N1を選択する。ID番号N1の指示内容によると、顔の向きが前向きでないといった障害がある状態を解消させる指示を実現することができる(図5(B)参照)。
For example, when both the image state and the acoustic state are B class, it may be difficult to pick up the utterance because the direction of the face is not reflected positively on the image (see FIG. 5 (A)). In such a case, the instruction determination unit 33 selects the ID number N1. According to the instruction content of the ID number N1, it is possible to realize an instruction for resolving a state in which there is an obstacle such as the face not facing forward (see FIG. 5B).
又、画像側の発話フラグがONである一方、音響側の発話フラグがOFFである場合、口元の動きが発話と検出されたが、発話の音声が充分に得られなかったことが想定される。この場合、指示決定部33は、ID番号N2を選択する。また、音響状態がCクラスであり、顕著な雑音が生じている場合に、指示決定部33は、ID番号N3を選択する。画像状態がCクラスであり、発話する人物が遠過ぎて口元の動きが検出できない場合に、指示決定部33は、ID番号N4を選択する。このように、指示テーブルD2からの選択により、各々の障害で発話が収音できなかった状態を解消させるように、指示内容を設定できる。なお、指示決定部33は、画像状態及び音響状態の検出結果に基づいて、各種の障害を検知可能である。
Further, when the utterance flag on the image side is ON and the utterance flag on the acoustic side is OFF, it is assumed that the movement of the mouth is detected as utterance, but the utterance voice is not sufficiently obtained. .. In this case, the instruction determination unit 33 selects the ID number N2. Further, when the acoustic state is C class and remarkable noise is generated, the instruction determination unit 33 selects the ID number N3. When the image state is C class and the speaking person is too far away to detect the movement of the mouth, the instruction determination unit 33 selects the ID number N4. In this way, by selecting from the instruction table D2, the instruction content can be set so as to eliminate the state in which the utterance could not be picked up due to each obstacle. The instruction determination unit 33 can detect various obstacles based on the detection results of the image state and the acoustic state.
図10に戻り、制御部13は、例えば指示を出力する前の画像状態および音響状態を検出する(S42)。ステップS42において、制御部13は、例えば、画像状態判定部31及び音響状態判定部32として機能して図6のステップS11,S12及び図8のステップS21,S22と同様の処理を行う。ステップS42では、例えば画像状態として、人の位置、顔の向き、口元の位置、人とカメラ11間の距離などの画像状態が検出される。また、音響状態としては位相差による音声の到来方向の確率、及び方向毎の人声の尤度などが検出される。
Returning to FIG. 10, the control unit 13 detects, for example, the image state and the acoustic state before outputting the instruction (S42). In step S42, the control unit 13 functions as, for example, the image state determination unit 31 and the acoustic state determination unit 32, and performs the same processing as in steps S11 and S12 of FIG. 6 and steps S21 and S22 of FIG. In step S42, for example, as an image state, an image state such as a person's position, a face orientation, a mouth position, and a distance between the person and the camera 11 is detected. Further, as the acoustic state, the probability of the arrival direction of the voice due to the phase difference, the likelihood of the human voice for each direction, and the like are detected.
又、制御部13は、出力処理部34として機能し、決定した指示を出力するように各種の出力部を制御する指示信号を、バス18を介して送信する(S43)。例えば、出力処理部34としての制御部13は、音声出力部16を制御して指示内容を音声出力させる。制御部13は、表示部17に指示内容を表示させてもよい。また、例えば外部モニタなどの外部機器を用いてユーザに指示を提示する場合、制御部13は、外部機器に対する指示信号を、出力部としての入出力インタフェース部15に送信する。
Further, the control unit 13 functions as an output processing unit 34, and transmits instruction signals for controlling various output units so as to output the determined instruction via the bus 18 (S43). For example, the control unit 13 as the output processing unit 34 controls the voice output unit 16 to output the instruction content by voice. The control unit 13 may display the instruction content on the display unit 17. Further, when presenting an instruction to the user using an external device such as an external monitor, the control unit 13 transmits an instruction signal to the external device to the input / output interface unit 15 as an output unit.
次に、制御部13は、指示後の画像状態および音響状態を検出する(S44)。ステップS44において、制御部13は、例えば指示前の検出(S42)と同様の検出を、指示を出力してから所定期間後に、画像状態判定部31及び音響状態判定部32として行う。所定期間は、ユーザが、指示に従って行動を始めることが想定される期間であり、例えば1秒である。ステップS44の処理は、例えば図6のステップS11,S12及び図8のステップS21,S22と同様である。
Next, the control unit 13 detects the image state and the acoustic state after the instruction (S44). In step S44, for example, the control unit 13 performs the same detection as the detection before the instruction (S42) as the image state determination unit 31 and the acoustic state determination unit 32 after a predetermined period from the output of the instruction. The predetermined period is a period in which the user is expected to start an action according to an instruction, for example, 1 second. The process of step S44 is the same as, for example, steps S11 and S12 of FIG. 6 and steps S21 and S22 of FIG.
次に、制御部13は、状態差分検出部35として機能し、指示前の画像状態および音響状態(S42)と、指示後の画像状態および音響状態(S44)との間の差分を検出する(S45)。この際、収音装置10の検出範囲内において、状態改善の指示に反応したユーザが発話する箇所からは指示前後の状態間の差分が検出される一方、指示に反応しない外的な箇所からは状態間の差分が検出されないことが想定される。
Next, the control unit 13 functions as a state difference detecting unit 35, and detects a difference between the image state and the acoustic state (S42) before the instruction and the image state and the acoustic state (S44) after the instruction (S44). S45). At this time, within the detection range of the sound collecting device 10, the difference between the states before and after the instruction is detected from the part where the user who responds to the instruction for improving the state speaks, while the difference between the states before and after the instruction is detected from the external part that does not respond to the instruction. It is assumed that the difference between the states is not detected.
ステップS45において、状態差分検出部35としての制御部13は、例えば画像状態の差分として、指示前後に変化した人の位置、顔の向き及び口元などを検出する。制御部13は、例えば画像における検出単位の領域r(θ,φ)毎に、検出された画像状態の差分の程度を示す差分スコアを算出する。又、制御部13は、音響状態の差分として、例えば指示前後に変化した音声の到来方向の確率、及び各方向(θ,φ)における人声の尤度を検出する。制御部13は、検出された音響状態の差分の程度を示す差分スコアを方向毎に算出する。
In step S45, the control unit 13 as the state difference detecting unit 35 detects, for example, the position of the person, the orientation of the face, the mouth, etc. that have changed before and after the instruction as the difference in the image state. The control unit 13 calculates, for example, a difference score indicating the degree of difference in the detected image states for each region r (θ, φ) of the detection unit in the image. Further, the control unit 13 detects, for example, the probability of the arrival direction of the voice changed before and after the instruction and the likelihood of the human voice in each direction (θ, φ) as the difference in the acoustic state. The control unit 13 calculates a difference score indicating the degree of difference in the detected acoustic state for each direction.
次に、制御部13は、方向推定部36として機能し、検出した各状態の差分に基づいて、指示後の発話による目的音が到来する方向を推定する演算を行う(S46)。例えば、制御部13は、検出範囲内における各々の方向において、画像状態の差分スコアと、音響状態の差分スコアの線形和を演算し、演算結果として得られたスコアが最も高い方向を、目的音の収音方向に決定する。差分スコアの線形和は、画像状態と音響状態とにそれぞれ重み付けした重み付け和であってもよい。又、線形和の代わりに乗算或いは非線形の演算が適宜、採用されてもよい。
Next, the control unit 13 functions as a direction estimation unit 36, and performs an operation of estimating the direction in which the target sound due to the utterance after the instruction arrives based on the difference between the detected states (S46). For example, the control unit 13 calculates the linear sum of the difference score of the image state and the difference score of the acoustic state in each direction within the detection range, and sets the direction in which the score obtained as the calculation result is the highest as the target sound. Determine the sound collection direction of. The linear sum of the difference scores may be a weighted sum that weights the image state and the acoustic state, respectively. Further, instead of the linear sum, multiplication or non-linear calculation may be appropriately adopted.
次に、制御部13は、信号処理部37として機能し、演算結果の収音方向からの音を強調する処理を実行する(S47)。信号処理部37は、例えばビームフォーミングにより、収音方向にビームを向けるように信号処理を行って、目的音の収音結果を示す目的音データを生成する。信号処理部37は、周波数フィルタ等の目的音を強調する各種の信号処理を行ってもよい。目的音データは、例えば記憶部14に格納される。
Next, the control unit 13 functions as a signal processing unit 37 and executes a process of emphasizing the sound from the sound collecting direction of the calculation result (S47). The signal processing unit 37 performs signal processing so as to direct the beam in the sound collecting direction by, for example, beamforming, and generates target sound data indicating the sound collecting result of the target sound. The signal processing unit 37 may perform various signal processing such as a frequency filter to emphasize the target sound. The target sound data is stored in, for example, the storage unit 14.
制御部13は、音声強調の実行(S47)後に、図4のステップS4の処理を終了する。その後、制御部13は、例えばステップS1に戻り、その後の処理を再度、行う。
The control unit 13 ends the process of step S4 in FIG. 4 after executing the speech enhancement (S47). After that, the control unit 13 returns to, for example, step S1 and performs the subsequent processing again.
以上の指示を介した収音処理(S4)によると、例えば図5(A)の状態において図5(B)参照に示すようにID番号N1の指示内容が出力される(S41~S43)。この際、発話した人物21が指示に従って顔の向きを改善することで、次の発話が収音可能になる。一方、発話していない人物22および騒音源25などは、指示に反応しない。よって、例えば指示の前後の状態間の差分として顔の向きが改善された人物21を容易に検出でき(S44,S45)、収音方向を精度良く決定できる(S46,S47)。
According to the sound collection process (S4) via the above instructions, for example, in the state of FIG. 5 (A), the instruction content of the ID number N1 is output as shown in FIG. 5 (B) (S41 to S43). At this time, the person 21 who made the utterance improves the direction of the face according to the instruction, so that the next utterance can be picked up. On the other hand, the non-speaking person 22 and the noise source 25 do not respond to the instruction. Therefore, for example, the person 21 whose face orientation is improved can be easily detected as the difference between the states before and after the instruction (S44, S45), and the sound collecting direction can be accurately determined (S46, S47).
ステップS46の状態間の差分としては、上記以外にも例えば、ID番号N2,N3の指示により音声状態において人声の尤度が大きくなったりする、ID番号N4の指示により画像状態において人とカメラ11間の距離が近くなったりことが考えられる。こうした状態間の差分を用いることにより、目的音を収音する精度を良くすることができる。
In addition to the above, as the difference between the states of step S46, for example, the likelihood of the human voice increases in the voice state by the instruction of ID numbers N2 and N3, and the human and the camera in the image state by the instruction of ID number N4. It is conceivable that the distance between 11 will be shorter. By using the difference between these states, the accuracy of collecting the target sound can be improved.
上記のステップS42の処理は、指示の出力(S43)前に限らず、例えばステップS43と同時に行われてもよい。ステップS42の処理は、指示後の状態検出(S44)よりも前の範囲内で実行可能であり、例えばステップS41前であってもよい。この場合、指示内容の決定(S41)に、ステップS42の検出結果が用いられてもよい。又、指示前の状態検出はステップS3前であってもよいことから、ステップS42の処理を省略して、その代わりにステップS12,S22の検出結果が用いられてもよい。
The process of step S42 described above is not limited to before the output of the instruction (S43), and may be performed at the same time as step S43, for example. The process of step S42 can be executed within the range before the state detection (S44) after the instruction, and may be, for example, before step S41. In this case, the detection result of step S42 may be used to determine the instruction content (S41). Further, since the state detection before the instruction may be performed before step S3, the processing of step S42 may be omitted and the detection results of steps S12 and S22 may be used instead.
また、以上の説明では、収音し難い状態を解消するように指示を変化させる例を説明したが、指示は画一的であってもよい。例えば、指示内容として「カメラの方に向いて、大きな声で発話してください。」といった内容が採用されてもよい。こうした画一的な指示であっても、収音可能に状態を改善可能と考えられる。
Further, in the above explanation, an example of changing the instruction so as to solve the state where it is difficult to collect sound has been explained, but the instruction may be uniform. For example, as an instruction content, a content such as "Please face the camera and speak in a loud voice" may be adopted. Even with such uniform instructions, it is considered possible to improve the condition so that sound can be picked up.
2-5.指示を介さない収音処理(S5)
図4のステップS5の処理の詳細を、図12を用いて説明する。 2-5. Sound collection processing without instructions (S5)
The details of the process of step S5 of FIG. 4 will be described with reference to FIG.
図4のステップS5の処理の詳細を、図12を用いて説明する。 2-5. Sound collection processing without instructions (S5)
The details of the process of step S5 of FIG. 4 will be described with reference to FIG.
図12は、指示を介さない収音処理(S5)を例示するフローチャートである。本フローチャートに示す処理は、現在状態が、図9の状態1,2,4,5のいずれかである場合(S3でNO)に実行される。
FIG. 12 is a flowchart illustrating a sound collection process (S5) that does not involve instructions. The process shown in this flowchart is executed when the current state is any of the states 1, 2, 4, and 5 in FIG. 9 (NO in S3).
例えば、制御部13は、まず、現在状態が、発話が有ると判定されたか否かを判断する(S51)。例えば、制御部13は、状態1,2において画像側の発話フラグと音響側の発話フラグの少なくとも一方が「ON」である場合に、ステップS51でYESに進む。一方、制御部13は、その他の場合にはステップS51でNOに進む。
For example, the control unit 13 first determines whether or not the current state is determined to have an utterance (S51). For example, the control unit 13 proceeds to YES in step S51 when at least one of the utterance flag on the image side and the utterance flag on the acoustic side is “ON” in the states 1 and 2. On the other hand, the control unit 13 proceeds to NO in step S51 in other cases.
現在状態が、発話が有ると判定された場合(S51でYES)、制御部13は、方向推定部36として機能し、収音方向の推定の演算を行う(S52)。ステップS42の処理は、例えば現在の画像状態の検出結果(図6のS12)及び音響状態の検出結果(図8のS22)を用いて行われる。
When it is determined that there is an utterance in the current state (YES in S51), the control unit 13 functions as the direction estimation unit 36 and performs an calculation for estimating the sound collection direction (S52). The process of step S42 is performed using, for example, the detection result of the current image state (S12 in FIG. 6) and the detection result of the acoustic state (S22 in FIG. 8).
次に、制御部13は、信号処理部37として機能し、演算結果の収音方向からの音を強調する処理を実行する(S53)。ステップS53の処理は、例えば図10のステップS47と同様に行われる。ステップS53により、指示を用いず目的音が収音される。
Next, the control unit 13 functions as a signal processing unit 37 and executes a process of emphasizing the sound from the sound collecting direction of the calculation result (S53). The process of step S53 is performed in the same manner as in step S47 of FIG. 10, for example. In step S53, the target sound is picked up without using an instruction.
制御部13は、音声強調の実行(S53)後に、図4のステップS5の処理を終了する。その後、制御部13は、例えばステップS1に戻り、その後の処理を再度、行う。
The control unit 13 ends the process of step S5 in FIG. 4 after executing the speech enhancement (S53). After that, the control unit 13 returns to, for example, step S1 and performs the subsequent processing again.
一方、発話が有ると判定されなかった場合(S51でNO)、例えば制御部13は、現在状態が、図9の状態5すなわち発話の判定不可の状態であるか否かを判断する(S54)。
On the other hand, when it is not determined that there is an utterance (NO in S51), for example, the control unit 13 determines whether or not the current state is the state 5 of FIG. 9, that is, the state in which the utterance cannot be determined (S54). ..
制御部13は、現在状態が状態5の場合(S54でYES)、例えばエラーの報知を各種出力部に出力させ(S55)、ステップS5を終了する。例えば、制御部13は、収音できていない音を知らせるために、エラーメッセージ等を表示部17に表示させる。ステップS54,S55の処理は適宜、省略されてもよい。
When the current state is the state 5 (YES in S54), the control unit 13 outputs, for example, an error notification to various output units (S55), and ends step S5. For example, the control unit 13 causes the display unit 17 to display an error message or the like in order to notify the sound that cannot be picked up. The processes of steps S54 and S55 may be omitted as appropriate.
一方、現在状態が状態5ではない場合(S54でNO)、制御部13は、特にステップS55の処理を行わずにステップS5を終了する。この場合(S51とS54でNO)は、状態1,2,4において発話が無いと判定された場合であることから、ステップS52,S53の処理も省略できる。
On the other hand, when the current state is not the state 5 (NO in S54), the control unit 13 ends step S5 without performing the process of step S55 in particular. In this case (NO in S51 and S54), since it is determined that there is no utterance in the states 1, 2, and 4, the processing in steps S52 and S53 can also be omitted.
以上の指示を介さない収音処理(S5)によると、例えば状態1,2において発話が有り、指示を用いなくても収音できる場合(S51でYES)、余計な指示を出力せずに収音を達成できる(S53)。また、発話が無いことが判明している場合(S51でNO)、不要な指示が出力されないようにすることができる。
According to the sound collection process (S5) that does not go through the above instructions, for example, if there is an utterance in states 1 and 2 and the sound can be collected without using the instruction (YES in S51), the sound is collected without outputting an extra instruction. Sound can be achieved (S53). Further, when it is known that there is no utterance (NO in S51), unnecessary instructions can be prevented from being output.
以上の処理において、制御部13は、例えばステップS51の前に、現在状態が、前回の制御周期における状態と同じであるか否かを判断してもよく、前回と同じでないと判断した場合のみステップS51以降の処理を行うようにしてもよい。現在状態が前回と同じと判断された場合、その後の処理も前回と同じ処理を適用する、例えば同じ収音方向を用いるようにしてもよい。
In the above processing, for example, before step S51, the control unit 13 may determine whether or not the current state is the same as the state in the previous control cycle, and only when it is determined that the current state is not the same as the previous time. The processing after step S51 may be performed. If it is determined that the current state is the same as the previous time, the same processing as the previous time may be applied to the subsequent processing, for example, the same sound collecting direction may be used.
3.まとめ
本開示に係る収音装置10は、人物21等のユーザが音源の一例として口元から出力させる発話の音声などの目的音を収音する。収音装置10は、第1の入力部と、第2の入力部と、制御部13と、音声出力部16、表示部17および入出力インタフェース部15等の出力部とを備える。第1の入力部としての制御部13は、カメラ11によって生成された画像データを入力する(S11)。第2の入力部としての制御部13は、マイクアレイ12から出力された音響信号を入力する(S21)。制御部13は、画像データ及び音響信号に基づいて、目的音を収音する収音方向を決定する(S3、S4)。出力部は、ユーザに提示する情報を出力する。制御部13は、入力された画像データが示す画像及び入力された音響信号が示す音響が、図9の状態3等の所定の状態であるか否かを判断する(S3)。制御部13は、画像及び音響が所定の状態であると判断したとき(S3でYES)、ユーザに対する指示を出力部に出力させ(S43)、指示が出力された後に更なる画像データ及び音響信号を入力して(S44)、収音方向を決定する(S46,S47)。 3. 3. Summary Thesound collecting device 10 according to the present disclosure collects a target sound such as a voice of an utterance output from a mouth by a user such as a person 21 as an example of a sound source. The sound collecting device 10 includes a first input unit, a second input unit, a control unit 13, and an output unit such as an audio output unit 16, a display unit 17, and an input / output interface unit 15. The control unit 13 as the first input unit inputs the image data generated by the camera 11 (S11). The control unit 13 as the second input unit inputs the acoustic signal output from the microphone array 12 (S21). The control unit 13 determines the sound collecting direction for collecting the target sound based on the image data and the acoustic signal (S3, S4). The output unit outputs the information presented to the user. The control unit 13 determines whether or not the image indicated by the input image data and the sound indicated by the input acoustic signal are in a predetermined state such as the state 3 of FIG. 9 (S3). When the control unit 13 determines that the image and sound are in a predetermined state (YES in S3), the control unit 13 outputs an instruction to the user to the output unit (S43), and after the instruction is output, further image data and sound signal. Is input (S44) to determine the sound collecting direction (S46, S47).
本開示に係る収音装置10は、人物21等のユーザが音源の一例として口元から出力させる発話の音声などの目的音を収音する。収音装置10は、第1の入力部と、第2の入力部と、制御部13と、音声出力部16、表示部17および入出力インタフェース部15等の出力部とを備える。第1の入力部としての制御部13は、カメラ11によって生成された画像データを入力する(S11)。第2の入力部としての制御部13は、マイクアレイ12から出力された音響信号を入力する(S21)。制御部13は、画像データ及び音響信号に基づいて、目的音を収音する収音方向を決定する(S3、S4)。出力部は、ユーザに提示する情報を出力する。制御部13は、入力された画像データが示す画像及び入力された音響信号が示す音響が、図9の状態3等の所定の状態であるか否かを判断する(S3)。制御部13は、画像及び音響が所定の状態であると判断したとき(S3でYES)、ユーザに対する指示を出力部に出力させ(S43)、指示が出力された後に更なる画像データ及び音響信号を入力して(S44)、収音方向を決定する(S46,S47)。 3. 3. Summary The
以上の収音装置10によると、目的音の収音が困難と想定される状態になったとき(S3でYES)、ユーザに指示を出力して(S43)、その後の画像データ及び音響信号を用いて収音方向が決定される(S46,S47)。これにより、目的音が収音し難くなることがある状況であっても、目的音が収音され易くすることができる。
According to the above sound collecting device 10, when it becomes difficult to collect the target sound (YES in S3), an instruction is output to the user (S43), and the subsequent image data and acoustic signal are output. The sound collecting direction is determined by using (S46, S47). As a result, even in a situation where it may be difficult to collect the target sound, the target sound can be easily collected.
本実施形態において、制御部13は、指示後に入力された画像データと当該画像データよりも前に入力された画像データ間の差分、および指示後に入力された音響信号と当該音響信号よりも前に入力された音響信号間の差分を検出して(S45)、収音方向を決定する(S46,S47)。これにより、ユーザが指示に従って改善した状態の変化を効率良く得て、目的音を収音する精度を良くすることができる。
In the present embodiment, the control unit 13 determines the difference between the image data input after the instruction and the image data input before the image data, and the acoustic signal input after the instruction and before the acoustic signal. The difference between the input acoustic signals is detected (S45), and the sound collection direction is determined (S46, S47). As a result, it is possible to efficiently obtain a change in the improved state according to the instruction of the user and improve the accuracy of collecting the target sound.
本実施形態において、制御部13は、収音方向の候補とする検出単位の領域r(θ,φ)毎の複数の方向における各方向について(図7参照)、画像データ間の差分及び音響信号間の差分の大きさに応じたスコアを算出し、算出されたスコアが最も大きい方向を、収音方向として決定する(S46)。こうした簡単な演算処理により、目的音の収音を容易に行うことができる。
In the present embodiment, the control unit 13 determines the difference between the image data and the acoustic signal for each direction in a plurality of directions for each region r (θ, φ) of the detection unit as a candidate for the sound collection direction (see FIG. 7). A score according to the magnitude of the difference between the two is calculated, and the direction in which the calculated score is the largest is determined as the sound collection direction (S46). By such a simple arithmetic process, it is possible to easily collect the target sound.
本実施形態において、収音装置10は、画像および音響の状態を分類する状態テーブルD1を格納する記憶部14をさらに備える。制御部13は、状態テーブルD1を参照し、所定の状態であると判断したときの画像及び音響に応じて指示に含める情報を決定する(S41)。例えば、制御部13は、図11に例示するように、所定の状態を解消するように指示に含める情報を変化させる。画像及び音響から判定される状態に応じてユーザに例示する指示を適切に設定でき、これによって現在状態を改善し易くすることができる。
In the present embodiment, the sound collecting device 10 further includes a storage unit 14 that stores a state table D1 that classifies image and sound states. The control unit 13 refers to the state table D1 and determines the information to be included in the instruction according to the image and sound when it is determined that the state is in a predetermined state (S41). For example, the control unit 13 changes the information included in the instruction so as to eliminate the predetermined state, as illustrated in FIG. Instructions to be illustrated to the user can be appropriately set according to the state determined from the image and sound, whereby the current state can be easily improved.
本実施形態において、制御部13は、画像及び音響が所定の状態でないと判断したとき(S3でNO)、指示を出力せずに、収音方向を決定する(S5)。これにより、不要な指示が頻発することを回避して、ユーザにとって収音装置10を利用し易くすることができる。
In the present embodiment, when the control unit 13 determines that the image and sound are not in a predetermined state (NO in S3), the control unit 13 determines the sound collecting direction without outputting an instruction (S5). This makes it possible for the user to easily use the sound collecting device 10 by avoiding the frequent occurrence of unnecessary instructions.
本実施形態において、制御部13は、画像データに基づいて、画像において音源の配置を示す情報を検出し(S12)、音響信号に基づいて、音響において目的音、及び雑音を検出し(S22)、画像データと音響信号の検出結果に応じて、収音方向を決定する(S3,S4)。所定の状態は、状態3のように、収音方向を決定する際の障害を含む状態である。同状態によると、制御部13において上記の検出結果に応じて収音方向が決定されない。こうした所定の状態が生じたときに、ユーザに対する指示を出力して、障害がある状態を改善させることにより、目的音が収音され易くすることができる。
In the present embodiment, the control unit 13 detects information indicating the arrangement of the sound source in the image based on the image data (S12), and detects the target sound and noise in the sound based on the acoustic signal (S22). , The sound collecting direction is determined according to the detection result of the image data and the acoustic signal (S3, S4). The predetermined state is a state including an obstacle in determining the sound collecting direction as in the state 3. According to the same state, the control unit 13 does not determine the sound collecting direction according to the above detection result. When such a predetermined state occurs, an instruction to the user is output to improve the state of the obstacle, so that the target sound can be easily picked up.
本実施形態において、収音装置10は、カメラ11及びマイクアレイ12のうちの少なくとも一方を備えてもよい。収音装置10は、カメラ11及びマイクアレイ12の双方を備えなくてもよい。
In the present embodiment, the sound collecting device 10 may include at least one of the camera 11 and the microphone array 12. The sound collecting device 10 does not have to include both the camera 11 and the microphone array 12.
本実施形態に係る収音方法は、ユーザが音源から出力させる目的音を収音するための方法であって、カメラ11によって生成された画像データを入力するステップ(S11)と、マイクアレイ12から出力された音響信号を入力するステップ(S21)と、入力された画像データが示す画像及び入力された音響信号が示す音響が、所定の状態であるか否かを判断するステップ(S3)とを含む。本方法は、画像及び音響が所定の状態であると判断したとき、ユーザに対する指示を出力するステップ(S43)と、指示が出力された後に更なる画像データ及び音響信号を入力して、収音方向を決定するステップ(S44~S47)とを含む。本方法によると、目的音が収音され易くすることができる。
The sound collecting method according to the present embodiment is a method for collecting the target sound output from the sound source by the user, from the step (S11) of inputting the image data generated by the camera 11 and the microphone array 12. A step of inputting the output acoustic signal (S21) and a step of determining whether or not the image indicated by the input image data and the sound indicated by the input acoustic signal are in a predetermined state (S3). Including. In this method, when it is determined that the image and sound are in a predetermined state, a step (S43) of outputting an instruction to the user and further image data and sound signal are input after the instruction is output to collect sound. The steps (S44 to S47) for determining the direction are included. According to this method, the target sound can be easily collected.
本実施形態において、上記のような収音方法をコンピュータに実行させるためのプログラムが提供される。こうしたプログラムは、例えば記憶部14を含む各種の記憶媒体に格納して提供したり、通信ネットワークを介して提供したりすることができる。
In the present embodiment, a program for causing a computer to execute the above sound collecting method is provided. Such a program can be stored and provided in various storage media including, for example, the storage unit 14, or can be provided via a communication network.
(他の実施形態)
以上のように、本出願において開示する技術の例示として、実施形態1を説明した。しかしながら、本開示における技術は、これに限定されず、適宜、変更、置換、付加、省略などを行った実施の形態にも適用可能である。また、上記各実施形態で説明した各構成要素を組み合わせて、新たな実施の形態とすることも可能である。そこで、以下、他の実施形態を例示する。 (Other embodiments)
As described above, the first embodiment has been described as an example of the technology disclosed in the present application. However, the technique in the present disclosure is not limited to this, and can be applied to embodiments in which changes, substitutions, additions, omissions, etc. are appropriately made. It is also possible to combine the components described in each of the above embodiments into a new embodiment. Therefore, other embodiments will be illustrated below.
以上のように、本出願において開示する技術の例示として、実施形態1を説明した。しかしながら、本開示における技術は、これに限定されず、適宜、変更、置換、付加、省略などを行った実施の形態にも適用可能である。また、上記各実施形態で説明した各構成要素を組み合わせて、新たな実施の形態とすることも可能である。そこで、以下、他の実施形態を例示する。 (Other embodiments)
As described above, the first embodiment has been described as an example of the technology disclosed in the present application. However, the technique in the present disclosure is not limited to this, and can be applied to embodiments in which changes, substitutions, additions, omissions, etc. are appropriately made. It is also possible to combine the components described in each of the above embodiments into a new embodiment. Therefore, other embodiments will be illustrated below.
上記実施形態では、収音方向として、水平角θ及び垂直角φをそれぞれ決定する場合について説明したが、水平角θ及び垂直角φの少なくともいずれか一方のみで、目的音源の方向を特定できる場合は、水平角θ及び垂直角φの少なくともいずれか一方を決定するだけであってもよい。
In the above embodiment, the case where the horizontal angle θ and the vertical angle φ are determined as the sound collection direction has been described, but the case where the direction of the target sound source can be specified only by at least one of the horizontal angle θ and the vertical angle φ. May only determine at least one of a horizontal angle θ and a vertical angle φ.
上記実施形態においては、人が発話する音声を目的音として収音したが、目的音は人が発話する音声に限らず、人が発する種々の音であってもよい。目的音は、車の音、動物の鳴き声、又は設備の動作音であってもよい。こうした目的音を収音したいユーザに対して、実施形態1と同様に収音困難なときに指示を提示することにより、目的音の収音を行い易くすることができる。
In the above embodiment, the sound uttered by a person is picked up as the target sound, but the target sound is not limited to the voice uttered by the person and may be various sounds uttered by the person. The target sound may be the sound of a car, the bark of an animal, or the operating sound of equipment. Similar to the first embodiment, by presenting an instruction to a user who wants to collect the target sound when it is difficult to collect the target sound, it is possible to facilitate the collection of the target sound.
以上のように、本開示における技術の例示として、実施の形態を説明した。そのために、添付図面および詳細な説明を提供した。
As described above, an embodiment has been described as an example of the technology in the present disclosure. To that end, the accompanying drawings and detailed description are provided.
したがって、添付図面および詳細な説明に記載された構成要素の中には、課題解決のために必須な構成要素だけでなく、上記技術を例示するために、課題解決のためには必須でない構成要素も含まれ得る。そのため、それらの必須ではない構成要素が添付図面や詳細な説明に記載されていることをもって、直ちに、それらの必須ではない構成要素が必須であるとの認定をするべきではない。
Therefore, among the components described in the attached drawings and the detailed description, not only the components essential for problem solving but also the components not essential for problem solving in order to exemplify the above technology. Can also be included. Therefore, the fact that those non-essential components are described in the accompanying drawings and detailed description should not immediately determine that those non-essential components are essential.
また、上述の実施の形態は、本開示における技術を例示するためのものであるから、特許請求の範囲またはその均等の範囲において、種々の変更、置換、付加、省略などを行うことができる。
Further, since the above-described embodiment is for exemplifying the technology in the present disclosure, various changes, substitutions, additions, omissions, etc. can be made within the scope of claims or the equivalent thereof.
本開示の収音装置は、例えば、人が発話する音声を収音する装置として、有用である。
The sound collecting device of the present disclosure is useful as, for example, a device for collecting sound spoken by a person.
Claims (9)
- ユーザが音源から出力させる目的音を収音するための収音装置であって、
カメラによって生成された画像データを入力する第1の入力部と、
マイクアレイから出力された音響信号を入力する第2の入力部と、
前記画像データ及び前記音響信号に基づいて、前記目的音を収音する収音方向を決定する制御部と、
前記ユーザに提示する情報を出力する出力部とを備え、
前記制御部は、
入力された画像データが示す画像及び入力された音響信号が示す音響が、所定の状態であるか否かを判断し、
前記画像及び前記音響が前記所定の状態であると判断したとき、前記ユーザに対する指示を前記出力部に出力させ、
前記指示が出力された後に更なる画像データ及び音響信号を入力して、前記収音方向を決定する
収音装置。 It is a sound collecting device for collecting the target sound that the user outputs from the sound source.
A first input unit for inputting image data generated by the camera,
A second input unit that inputs the acoustic signal output from the microphone array,
A control unit that determines a sound collecting direction for collecting the target sound based on the image data and the acoustic signal, and a control unit.
It is provided with an output unit that outputs information to be presented to the user.
The control unit
It is determined whether or not the image indicated by the input image data and the sound indicated by the input acoustic signal are in a predetermined state.
When it is determined that the image and the sound are in the predetermined state, an instruction to the user is output to the output unit.
A sound collecting device that determines the sound collecting direction by inputting further image data and an acoustic signal after the instruction is output. - 前記制御部は、前記指示の後に入力された画像データと当該画像データよりも前に入力された画像データ間の差分、および前記指示の後に入力された音響信号と当該音響信号よりも前に入力された音響信号間の差分を検出して、前記収音方向を決定する
請求項1に記載の収音装置。 The control unit inputs the difference between the image data input after the instruction and the image data input before the image data, and the acoustic signal input after the instruction and before the acoustic signal. The sound collecting device according to claim 1, wherein the difference between the sound signals is detected and the sound collecting direction is determined. - 前記制御部は、前記収音方向の候補とする複数の方向における各方向について、前記画像データ間の差分および前記音響信号間の差分の大きさに応じたスコアを算出し、算出されたスコアが最も大きい方向を、前記収音方向として決定する
請求項2に記載の収音装置。 The control unit calculates a score according to the magnitude of the difference between the image data and the difference between the acoustic signals in each direction in a plurality of directions as candidates for the sound collecting direction, and the calculated score is obtained. The sound collecting device according to claim 2, wherein the largest direction is determined as the sound collecting direction. - 画像および音響の状態を分類する状態テーブルを格納する記憶部をさらに備え、
前記制御部は、前記状態テーブルを参照し、前記所定の状態であると判断したときの前記画像及び前記音響に応じて前記指示に含める情報を決定する
請求項1から3のいずれか1項に記載の収音装置。 It also has a storage unit that stores a state table that classifies image and acoustic states.
The control unit refers to the state table, and determines the information to be included in the instruction according to the image and the sound when it is determined that the state is the predetermined state, according to any one of claims 1 to 3. The described sound collector. - 前記制御部は、前記画像及び前記音響が前記所定の状態でないと判断したとき、前記指示を出力せずに、前記収音方向を決定する
請求項1から4のいずれか1項に記載の収音装置。 The collection according to any one of claims 1 to 4, wherein when the control unit determines that the image and the sound are not in the predetermined state, the control unit determines the sound collection direction without outputting the instruction. Sound device. - 前記制御部は、
前記画像データに基づいて、前記画像において前記音源の配置を示す情報を検出し、
前記音響信号に基づいて、前記音響において前記目的音、及び雑音を検出し、
前記画像と前記音響における検出結果に応じて、前記収音方向を決定し、
前記所定の状態は、前記画像と前記音響における検出結果が、前記収音方向を決定する際の障害を含む状態である
請求項1から5のいずれか1項に記載の収音装置。 The control unit
Based on the image data, information indicating the arrangement of the sound source is detected in the image, and
Based on the acoustic signal, the target sound and noise are detected in the acoustic, and the sound is detected.
The sound collecting direction is determined according to the detection result in the image and the sound.
The sound collecting device according to any one of claims 1 to 5, wherein the predetermined state is a state in which the detection result in the image and the sound includes an obstacle in determining the sound collecting direction. - 前記カメラ及び前記マイクアレイのうちの少なくとも一方を備える
請求項1から6のいずれか1項に記載の収音装置。 The sound collecting device according to any one of claims 1 to 6, further comprising at least one of the camera and the microphone array. - ユーザが音源から出力させる目的音を収音するための収音方法であって、
カメラによって生成された画像データを入力するステップと、
マイクアレイから出力された音響信号を入力するステップと、
入力された画像データが示す画像及び入力された音響信号が示す音響が、所定の状態であるか否かを判断するステップと、
前記画像及び前記音響が前記所定の状態であると判断したとき、前記ユーザに対する指示を出力するステップと、
前記指示が出力された後に更なる画像データ及び音響信号を入力して、前記目的音を収音する収音方向を決定するステップと
を含む収音方法。 It is a sound collection method for collecting the target sound that the user outputs from the sound source.
Steps to enter the image data generated by the camera,
Steps to input the acoustic signal output from the microphone array,
A step of determining whether or not the image indicated by the input image data and the sound indicated by the input acoustic signal are in a predetermined state, and
A step of outputting an instruction to the user when it is determined that the image and the sound are in the predetermined state, and
A sound collection method including a step of inputting further image data and an acoustic signal after the instruction is output to determine a sound collection direction for collecting the target sound. - 請求項8に記載の収音方法をコンピュータに実行させるためのプログラム。 A program for causing a computer to execute the sound collecting method according to claim 8.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2021554131A JPWO2021084905A1 (en) | 2019-10-30 | 2020-09-03 | |
CN202080073270.9A CN114586374A (en) | 2019-10-30 | 2020-09-03 | Sound collecting device and sound collecting method |
US17/705,687 US12033654B2 (en) | 2019-10-30 | 2022-03-28 | Sound pickup device and sound pickup method |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2019197723 | 2019-10-30 | ||
JP2019-197723 | 2019-10-30 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/705,687 Continuation US12033654B2 (en) | 2019-10-30 | 2022-03-28 | Sound pickup device and sound pickup method |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021084905A1 true WO2021084905A1 (en) | 2021-05-06 |
Family
ID=75716173
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2020/033469 WO2021084905A1 (en) | 2019-10-30 | 2020-09-03 | Sound pickup device and sound pickup method |
Country Status (4)
Country | Link |
---|---|
US (1) | US12033654B2 (en) |
JP (1) | JPWO2021084905A1 (en) |
CN (1) | CN114586374A (en) |
WO (1) | WO2021084905A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024075434A1 (en) * | 2022-10-07 | 2024-04-11 | ソニーグループ株式会社 | Information processing system, device, information processing method, and program |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2017034645A (en) * | 2015-08-03 | 2017-02-09 | 株式会社リコー | Imaging apparatus, program, and imaging method |
JP2019008134A (en) * | 2017-06-23 | 2019-01-17 | カシオ計算機株式会社 | Sound source separation information detection device, robot, sound source separation information detection method and program |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS5215826B1 (en) | 1971-04-13 | 1977-05-04 | ||
JP5075664B2 (en) * | 2008-02-15 | 2012-11-21 | 株式会社東芝 | Spoken dialogue apparatus and support method |
JP5215826B2 (en) | 2008-11-28 | 2013-06-19 | 日本電信電話株式会社 | Multiple signal section estimation apparatus, method and program |
US8665312B2 (en) * | 2010-12-06 | 2014-03-04 | Ricoh Company, Ltd. | Apparatus, system, and method of managing data transmission, and transmission management program |
US8886530B2 (en) | 2011-06-24 | 2014-11-11 | Honda Motor Co., Ltd. | Displaying text and direction of an utterance combined with an image of a sound source |
CN106067996B (en) | 2015-04-24 | 2019-09-17 | 松下知识产权经营株式会社 | Voice reproduction method, voice dialogue device |
JP6678315B2 (en) | 2015-04-24 | 2020-04-08 | パナソニックIpマネジメント株式会社 | Voice reproduction method, voice interaction device, and voice interaction program |
US11391944B2 (en) * | 2017-03-21 | 2022-07-19 | Sony Corporation | Information processing device, information processing method, and program |
US10665250B2 (en) * | 2018-09-28 | 2020-05-26 | Apple Inc. | Real-time feedback during audio recording, and related devices and systems |
-
2020
- 2020-09-03 WO PCT/JP2020/033469 patent/WO2021084905A1/en active Application Filing
- 2020-09-03 JP JP2021554131A patent/JPWO2021084905A1/ja active Pending
- 2020-09-03 CN CN202080073270.9A patent/CN114586374A/en active Pending
-
2022
- 2022-03-28 US US17/705,687 patent/US12033654B2/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2017034645A (en) * | 2015-08-03 | 2017-02-09 | 株式会社リコー | Imaging apparatus, program, and imaging method |
JP2019008134A (en) * | 2017-06-23 | 2019-01-17 | カシオ計算機株式会社 | Sound source separation information detection device, robot, sound source separation information detection method and program |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024075434A1 (en) * | 2022-10-07 | 2024-04-11 | ソニーグループ株式会社 | Information processing system, device, information processing method, and program |
Also Published As
Publication number | Publication date |
---|---|
US20220215852A1 (en) | 2022-07-07 |
US12033654B2 (en) | 2024-07-09 |
CN114586374A (en) | 2022-06-03 |
JPWO2021084905A1 (en) | 2021-05-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6464449B2 (en) | Sound source separation apparatus and sound source separation method | |
US11043231B2 (en) | Speech enhancement method and apparatus for same | |
JP6230726B2 (en) | Speech recognition apparatus and speech recognition method | |
EP2541543B1 (en) | Signal processing apparatus and signal processing method | |
US9491553B2 (en) | Method of audio signal processing and hearing aid system for implementing the same | |
US6707921B2 (en) | Use of mouth position and mouth movement to filter noise from speech in a hearing aid | |
WO2015172630A1 (en) | Camera shooting device and focusing method therefor | |
US20100302401A1 (en) | Image Audio Processing Apparatus And Image Sensing Apparatus | |
JP6844608B2 (en) | Voice processing device and voice processing method | |
JP2010187363A (en) | Acoustic signal processing apparatus and reproducing device | |
US10089980B2 (en) | Sound reproduction method, speech dialogue device, and recording medium | |
JPWO2012001928A1 (en) | Conversation detection device, hearing aid, and conversation detection method | |
JP2012003326A (en) | Information processing device, information processing method, and program | |
JP2020156107A (en) | Voice processor, voice processing method and voice processing program | |
TW201801069A (en) | Method and system for receiving voice message and electronic device using the method | |
CN113676592A (en) | Recording method, recording device, electronic equipment and computer readable medium | |
WO2021084905A1 (en) | Sound pickup device and sound pickup method | |
JP3838159B2 (en) | Speech recognition dialogue apparatus and program | |
Berghi et al. | Visually supervised speaker detection and localization via microphone array | |
JP2023008029A (en) | Customer service support method and program | |
US20230238001A1 (en) | Eyeglass augmented reality speech to text device and method | |
JP7217471B2 (en) | Imaging device | |
US10812898B2 (en) | Sound collection apparatus, method of controlling sound collection apparatus, and non-transitory computer-readable storage medium | |
JP2006304125A (en) | Apparatus and method for correcting sound signal | |
JP2020178150A (en) | Voice processing device and voice processing method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20883033 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2021554131 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20883033 Country of ref document: EP Kind code of ref document: A1 |