CN110557554B - Image switching device and system - Google Patents
Image switching device and system Download PDFInfo
- Publication number
- CN110557554B CN110557554B CN201810557099.3A CN201810557099A CN110557554B CN 110557554 B CN110557554 B CN 110557554B CN 201810557099 A CN201810557099 A CN 201810557099A CN 110557554 B CN110557554 B CN 110557554B
- Authority
- CN
- China
- Prior art keywords
- image
- module
- sound source
- panoramic
- spherical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004364 calculation method Methods 0.000 claims abstract description 44
- 238000012545 processing Methods 0.000 claims abstract description 13
- 230000005236 sound signal Effects 0.000 claims description 25
- 230000005540 biological transmission Effects 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 10
- 238000000034 method Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
- G06T3/047—Fisheye or wide-angle transformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Studio Devices (AREA)
Abstract
The invention is suitable for the technical field of image processing, and provides an image switching device and an image switching system, wherein the image switching device is applied to a sound box and comprises an audio acquisition module, an image acquisition module, a network transceiving module, a calculation module and a display module; the computing module is respectively connected with the audio acquisition module, the image acquisition module, the network transceiver module and the display module, and the network transceiver module is also communicated with external equipment. According to the invention, when a multi-person video conference is carried out, even if a sound source changes, the image switching device can directly acquire and switch the sound source image corresponding to the new sound source object, so that the real-time performance of sending the sound source image and the panoramic image to the external equipment and the efficiency of switching the sound source image are improved.
Description
Technical Field
The present invention relates to image processing technologies, and in particular, to an image switching apparatus and system.
Background
The multi-person video conference system is a multimedia communication technology which enables people in different places to realize real-time, visual and interactive through a certain transmission medium, so that users scattered geographically can be communicated in a plurality of modes such as graphs and sounds.
In a video conference, in order to highlight who a speaker is, show facial expressions, body movements and the like of the speaker, images of the speaker need to be displayed on a screen of an opposite party performing the video conference, however, a current video conference system usually uses a camera tracking technology to capture a human face first, then determine the speaker, and finally position and shoot the speaker, steps are complex, and when the speaker changes, image switching efficiency is low.
Disclosure of Invention
In view of this, embodiments of the present invention provide an image switching apparatus and system to solve the problem that the images of speakers cannot be switched quickly when a multi-person video conference is performed in the prior art.
The first aspect of the embodiment of the invention provides an image switching device, which is applied to a sound box and comprises an audio acquisition module, an image acquisition module, a network transceiving module, a calculation module and a display module;
the computing module is respectively connected with the audio acquisition module, the image acquisition module, the network transceiving module and the display module, and the network transceiving module is also communicated with external equipment;
the audio acquisition module acquires a sound signal and sends the sound signal to the calculation module;
the image acquisition module acquires a panoramic camera, and the panoramic camera acquires a panoramic image and sends the panoramic image to the calculation module;
the computing module computes a sound source position according to the sound signal, generates a spherical panoramic image according to the panoramic image, intercepts a sound source image corresponding to the sound source position in the spherical panoramic image, and sends the spherical panoramic image and the sound source image to the display module and the network transceiver module;
and the network transceiver module sends the spherical panoramic image and the sound source image to the external equipment and receives the image sent by the external equipment.
Optionally, the calculation module comprises:
the spherical coordinate system establishing unit is used for establishing a spherical coordinate system by taking the sound box as an origin of coordinates;
the panoramic mapping unit is used for mapping the panoramic image acquired by the image acquisition module into a spherical coordinate system when the panoramic image is sent to the calculation module, so as to obtain a spherical panoramic image;
and the coordinate conversion unit is used for converting the sound source position into a spherical coordinate to obtain a sound source position coordinate.
Optionally, the computing module further comprises:
the image intercepting unit is used for intercepting an image according to the sound source position coordinate in the spherical panoramic image to obtain the sound source image;
an image processing unit for processing the spherical panoramic image into a rectangular expanded image;
and the image transmission unit is used for sending the sound source image and the spherical panoramic image to the display module and the network transceiver module.
Optionally, the audio acquisition module comprises a first microphone array disposed at one end of the sound box;
each microphone in the first microphone array is arranged on a first plane of the sound box;
the calculation module calculates a first angle of a sound source on a coordinate plane passing through the origin of the spherical coordinate system and parallel to the first plane according to the intensity difference or the time difference of the sound signals collected by the microphones in the first microphone array, and obtains a first position parameter of the sound source position.
Optionally, the audio capture module further comprises a second microphone array disposed on a second plane parallel to the first plane;
and the calculation module calculates a second angle of a sound source on a horizontal plane which passes through the origin of the spherical coordinate system and is parallel to the second plane according to the intensity difference or the time difference of the sound signals collected by all the microphones in the second microphone array, so as to obtain a second position parameter of the sound source position.
Optionally, the calculation module further includes a preset parameter obtaining unit;
the preset parameter acquiring unit is used for acquiring preset parameters;
and the calculation module determines the position of the sound source according to the preset parameter and the first position parameter or according to the preset parameter and the second position parameter.
Optionally, the calculation module calculates a third angle of the sound source relative to a direction perpendicular to an end surface of the sound box according to an intensity difference or a time difference of sound signals collected by the first microphone array and the second microphone array, so as to obtain a third position parameter of the sound source position.
Optionally, the calculation module determines the sound source position according to the third position parameter and the first position parameter;
or alternatively
And the calculation module determines the sound source position according to the third position parameter and the second position parameter.
Optionally, the panoramic camera is arranged at the center of one end of the sound box;
the panoramic camera shoots the panoramic image and sends the panoramic image to the computing module.
Optionally, the panoramic camera comprises at least two cameras;
the sum of the shooting angles of all the cameras is greater than or equal to 360 degrees;
the camera shoots images and sends the images to the computing module, and the computing module splices all the images sent by the camera into the panoramic image.
Optionally, the camera is a fisheye camera.
A second aspect of embodiments of the present invention provides an image switching system including an external device and the image switching apparatus described above.
Optionally, the external device is the same as the image switching apparatus;
the display module of the external device receives the spherical panoramic image and the sound source image which are sent by the computing module in the image switching device through a network transceiving module of the external device;
and the display module of the image switching device receives the spherical panoramic image and the sound source image which are sent by the computing module in the external equipment through the network transceiving module of the image switching device.
Compared with the prior art, the embodiment of the invention has the following beneficial effects: in the image switching apparatus provided in this embodiment, the computing module receives the sound signal collected by the audio collecting module and the panoramic image collected by the image collecting module, and computes the sound source position according to the sound signal, and generates a spherical panoramic image according to the panoramic image collected by the image collecting module, and then intercepts the sound source image from the sound source position in the spherical panoramic image, so that when the sound source changes, algorithms or operations for capturing a new sound source, such as face capture, identity authentication, localization tracking, and the like performed by an object by a camera are not required, and a sound source image corresponding to a new sound source object is directly obtained and switched, thereby improving the real-time performance of sending the sound source image and the panoramic image to an external device by the image switching apparatus and the efficiency of switching the sound source image.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a schematic structural diagram of an image switching apparatus according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a computing module according to a second embodiment of the present invention;
fig. 3 is a schematic diagram of an external device displaying an image according to a second embodiment of the present invention;
fig. 4 is a schematic structural diagram of a spherical panoramic image according to a second embodiment of the present invention;
fig. 5 is a schematic structural diagram of an audio acquisition module according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of another audio acquisition module according to a third embodiment of the present invention;
fig. 7 is a schematic structural diagram of an image acquisition module according to a fourth embodiment of the present invention;
fig. 8 is a schematic structural diagram of another image capturing module according to a fourth embodiment of the present invention;
fig. 9 is a schematic structural diagram of an image switching system according to a fifth embodiment of the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, modules, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
The terms "comprises" and "comprising," and any variations thereof, in the description and claims of this invention and the above-described drawings are intended to cover non-exclusive inclusions. For example, a process, method, or system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus. Furthermore, the terms "first", "second", and "third", etc. are used to distinguish different objects, and are not used to describe a particular order.
In order to explain the technical means of the present invention, the following description will be given by way of specific examples.
Example one
As shown in fig. 1, the present embodiment provides an image switching apparatus 10 applied to a sound box, which includes an audio acquisition module 11, an image acquisition module 12, a network transceiver module 13, a calculation module 14, and a display module 15.
The connection relationship of the devices in the image switching apparatus 10 provided in the present embodiment is as follows:
the computing module 14 is respectively connected with the audio acquisition module 11, the image acquisition module 12, the network transceiver module 13 and the display module 15, and the network transceiver module 13 is further connected with an external device.
In this embodiment, the audio acquisition module is configured to acquire a sound signal and send the sound signal to the computing module; the image acquisition module comprises a panoramic camera (not shown in the figure), and the panoramic camera acquires a panoramic image and sends the panoramic image to the calculation module; the computing module is used for computing the position of a sound source according to the sound signal collected in the audio collecting module, generating a spherical panoramic image according to the panoramic image, intercepting a sound source image corresponding to the position of the sound source in the spherical panoramic image generated in the computing module according to the position of the sound source, and sending the spherical panoramic image and a sound source image to the display module and the network transceiving module; the network transceiver module is used for sending the spherical panoramic image and the acoustic source image to external equipment and receiving the image sent by the external equipment; the display module is used for displaying the sound source image and/or the spherical panoramic image.
In specific application, the sound signal collected by the audio collection module can be the sound emitted by a person or the sound emitted by a tool and is used for prompting the position of the sound source of the calculation module; in this embodiment, the audio collection module is used to collect the voice of the human speaking.
In specific application, the panoramic image acquired by the image acquisition module refers to an image of all angles which can be acquired by taking the image acquisition module as a center, wherein the image can be a static image or a video image; in this embodiment, the image capturing module is configured to capture an image in a current environment in real time, that is, capture a video and send the video to an external device for display through a network or a data line in real time.
In specific applications, the network transceiver module may be any network module capable of implementing wireless data transmission between two end devices, for example, a bluetooth module, a Wi-Fi module, an ethernet transceiver module, an optical fiber transceiver, and a 485 isolation transceiver module.
In Specific applications, the computing module may be any Processing module or Processor capable of implementing data analysis and transmission, such as a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA), a general purpose Processor or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, and so on. Wherein a general purpose processor may be a microprocessor or any conventional processor or the like.
In a specific application, the display module may be any display device or several display devices capable of displaying multiple pictures, such as a television display screen, a computer display, a projector, a multi-screen display, and the like.
In the image switching apparatus provided in this embodiment, the computing module receives the sound signal collected by the audio collecting module and the panoramic image collected by the image collecting module, and computes the sound source position according to the sound signal, and generates a spherical panoramic image according to the panoramic image collected by the image collecting module, and then intercepts the sound source image from the sound source position in the spherical panoramic image, so that when the sound source changes, algorithms or operations for capturing a new sound source, such as face capture, identity authentication, localization tracking, and the like performed by an object by a camera are not required, and a sound source image corresponding to a new sound source object is directly obtained and switched, thereby improving the real-time performance of sending the sound source image and the panoramic image to an external device by the image switching apparatus and the efficiency of switching the sound source image.
Example two
As shown in fig. 2, the calculation module 14 in the first embodiment includes:
a spherical coordinate system establishing unit 141, configured to establish a spherical coordinate system with the sound box as an origin of coordinates;
the panoramic mapping unit 142 is used for mapping the panoramic image acquired by the image acquisition module into a spherical coordinate system when the panoramic image is sent to the calculation module, so as to obtain a spherical panoramic image;
and a coordinate conversion unit 143 configured to convert the sound source position into spherical coordinates, and obtain sound source position coordinates.
As shown in fig. 2, the calculating module 14 in this embodiment further includes:
an image capturing unit 144, configured to capture an image according to the sound source position coordinates in the spherical panoramic image to obtain a sound source image;
an image processing unit 145 for processing the spherical panoramic image and the sound source image into a rectangular expanded image;
an image transmission unit 146, configured to send the sound source image and the processed spherical panoramic image to the display module and the network transceiver module.
In a specific application, when the spherical coordinate system is established, the central point of the sound box may be used as the origin of coordinates, or the device disposed on the sound box may be used as the origin of coordinates, for example, the position of the computing module on the sound box is used as the origin of coordinates, which is not limited in this embodiment.
In specific application, the external device may simultaneously display the rectangular expanded view of the sound source image and/or the spherical panoramic image, and the display module may also select to display the rectangular expanded view of the sound source image and/or the spherical panoramic image as needed.
As shown in fig. 3, the present embodiment exemplarily provides a schematic diagram in which an external device simultaneously displays a sound source image and a rectangular expansion of a spherical panoramic image, where T1 is the sound source image and T2 is the rectangular expansion of the spherical panoramic image.
As shown in fig. 4, in the present embodiment, the computing module maps the panoramic image into a spherical coordinate system, and a spherical panoramic image is simulated in the computing module. When an image is captured according to the sound source position coordinates, an image of an area near the sound source position coordinates is captured. Since the distance of the sound source position coordinates relative to the origin of the coordinate system does not affect the interception of the sound source image, the image can be intercepted according to the angle of the sound source position relative to the origin of the spherical coordinates. For example, a circle, rectangle, or other shape of a certain area size may be truncated with the sound source position coordinates as the center.
In a specific application, the sound source position coordinates may be a continuous coordinate point set or a dispersed set of several coordinate points, and the size of the area of the intercepted sound source image is adjusted according to the calculated coordinates.
In practical application, the sound source is assumed to be the voice of a person speaking, sound source position information is the position of the person speaking, panoramic images around the image switching device can be collected by the image collection module, the panoramic images are mapped into a spherical panoramic image in a spherical coordinate system by the calculation module, the position of the person speaking can be obtained by the audio collection module, so that the position of the person speaking is given by the calculation module according to the audio collection module, the person speaking is found in the spherical panoramic image, the image of the person speaking is captured and an image of a certain area is intercepted, the image is sent to the display through the calculation module, the image is sent to external equipment through the network transceiving module, the image of the person speaking is displayed by the external equipment in real time, and the person around the image switching device can also know whether the image of the person speaking is accurately intercepted through the display module.
The spherical panoramic image provided by the embodiment is an intermediate result image in the process of intercepting a sound source image by a computing module, and because a person observes a spherical scene actually observed from the surrounding scene, the computing module in the embodiment simulates the way of observing the surrounding visual angle by the person, and maps the panoramic image in a spherical coordinate system, so that the computing of the position of the sound source is more accurate and simpler, the sound source image is cut more quickly when being cut, and the sound source image cut from the spherical panoramic image is a part of the spherical surface and can reduce the possible distortion of the upper part and the lower part of the image when being unfolded and displayed on a plane display device.
The computing module provided by this embodiment can determine the position of the sound source, acquire the sound source image, and can also generate the spherical panoramic image according to the panoramic image acquired by the image acquisition module, and when the sound source changes, the computing module captures a new sound source position in the panoramic image in real time, acquires a new sound source image, and sends the new sound source image to the external device for display, thereby improving the real-time performance of sending the sound source image and the spherical panoramic image to the external device and the efficiency of switching the sound source image by the image switching device.
EXAMPLE III
As shown in fig. 5, the audio acquisition module 11 in the first and second embodiments includes a first microphone array 112 disposed at one end of a sound box 111; each microphone of the first microphone array 112 is disposed on a first plane of the sound box 111; the calculating module 14 collects the sound signals emitted by the same sound source according to the microphones in the first microphone array 112 to obtain the intensity difference or the time difference of the sound signals, so as to calculate a first angle of the sound source passing through the origin of the spherical coordinate system and on the coordinate plane parallel to the first plane, and obtain a first position parameter of the sound source position.
As shown in fig. 6, in the present embodiment, the audio acquisition module 11 may further include a second microphone array 113 disposed on a second plane parallel to the first plane; the calculating module 14 collects sound source signals emitted by the same sound source according to each microphone in the second microphone array 113, obtains intensity difference or time difference of the sound signals, calculates a second angle of the sound source on a horizontal plane passing through the origin of the spherical coordinate system and parallel to the second plane, and obtains a second position parameter of the sound source position.
In this embodiment, the calculating module 14 further collects sound source signals emitted by the same sound source according to the first microphone array 112 and the second microphone array 113, obtains an intensity difference or a time difference of the sound signals, calculates a third angle of the sound source relative to the origin perpendicular to the spherical coordinates, and obtains a third position parameter of the sound source position.
In a specific application, the arrangement of the microphones in the first microphone array may be arbitrary, and the arrangement of the sound sources with respect to the horizontal direction of the sound box can be calculated, for example, a uniform arrangement on the same plane, or a non-uniform arrangement on different planes. In this embodiment, by placing each microphone in the first microphone array at one end of the sound box, if one end of the sound box is a plane, the microphone arrays can be considered to be placed on the same horizontal plane and arranged along the outline of one end of the sound box, so that the first angle of the sound source relative to the origin of the spherical coordinate system, that is, the azimuth angle of the sound source relative to the horizontal plane passing through the origin of the spherical coordinate system, can be calculated and expressed as the first position parameter.
In a specific application, the arrangement manner of the second microphone array at the other end of the sound box is the same as that of the first microphone array, and may be any, the arrangement manner of the sound source relative to the horizontal direction angle of the sound box can be calculated, and the obtained second position parameter is also the second angle of the sound source relative to the origin of the spherical coordinate system, that is, the angle of the sound source relative to the horizontal plane direction of the origin of the spherical coordinate system, and is expressed by the second position parameter.
In practical applications, the first angle and the second angle are less than or equal to 180 ° in combination with the positional relationship between the sound source and the microphone array.
In this embodiment, the calculation module may further include a preset parameter obtaining unit; the preset parameter acquisition unit is used for acquiring preset parameters; and the calculation module determines the position of the sound source according to the preset parameter and the first position parameter or according to the preset parameter and the second position parameter.
In a specific application, the preset parameter represents the height of the speaker relative to the loudspeaker box, and can be expressed as an angle of the sound source in the vertical direction relative to the origin of the spherical coordinate system in the spherical coordinate system.
In practical application, if there is only one microphone array, an angle of the sound source in the horizontal plane direction with respect to the origin of the spherical coordinate system can be obtained, and the angle is less than or equal to 180 °, and at this time, the calculation module needs to determine the position of the sound source according to one of the first position parameter or the second position parameter. However, when the sound source position is determined according to one of the first position parameter or the second position parameter in the spherical coordinate system (also the spherical panoramic image), a half plane passing through the Z axis is obtained first, the sound source position is the junction of the half plane and the spherical panoramic image, the range is large, but the height of the speaker relative to the loudspeaker box is a parameter with a small variation range, so that the sound source position can be determined more accurately by presetting a height parameter of the sound source relative to the loudspeaker box in the calculation module.
In one embodiment, a third angle of the sound source relative to the origin of the spherical coordinate system, i.e. an angle in a vertical direction of the sound source relative to the origin of the spherical coordinate system, may also be calculated by an arrangement of the first microphone array and the second microphone array, in particular, by an arrangement of the first microphone array and the second microphone array which are non-coincident with each other, and expressed as a third position parameter.
In this embodiment, the calculation module determines the sound source position according to the third position parameter and the first position parameter; or, the calculation module determines the sound source position according to the third position parameter and the second position parameter.
As will be understood by those skilled in the art, the spherical coordinate system is established based on a spatial coordinate system, and in a standard spatial coordinate system, the XOY plane is taken as a horizontal plane, and the Z axis is taken as a coordinate axis perpendicular to the XOY plane, so that the horizontal plane of the spherical coordinate system is the XOY plane, and a direction angle on the horizontal plane of the sound source relative to the origin of the spherical coordinate system, that is, an angle relative to the X axis after the sound source position is projected on the XOY plane; i.e. the angle of the sound source in the vertical direction with respect to the origin of the spherical coordinate system, i.e. the angle with respect to the Z-axis after the line between the sound source position and the origin.
In practical application, the calculation module can calculate the angle of the sound source relative to the sound box in the horizontal direction or the angle in the vertical direction, and because the distance from the position of the sound box to any point in the spherical panoramic image does not affect the interception of the sound source image, the positioning can be performed only according to the angle in the horizontal direction or the angle in the vertical direction, and meanwhile, the accurate positioning can be performed according to the angle in the horizontal direction and the angle in the vertical direction.
The audio acquisition module provided by the embodiment can acquire sound signals through different microphone arrays, so that the position of a sound source is positioned.
Example four
As shown in fig. 7, the image capturing module 12 (not shown in the figure) in the first and second embodiments includes a panoramic camera 121; the panoramic camera 121 is arranged at the center of one end of the sound box 111; the panoramic camera 121 may capture a panoramic image and send the panoramic image to the calculation module 14.
As shown in fig. 8, in this embodiment, the panoramic camera 121 includes at least two cameras 122, the sum of the shooting angles of all the cameras 122 is greater than or equal to 360 ° and the images shot by the cameras 122 are sent to the calculation module 14, and the calculation module 14 splices the images sent by all the cameras 122 into a panoramic image.
In a specific application, the number of the cameras is at least two, and is not limited herein, and the taken images can be spliced into a panoramic image, the types of the cameras can be arbitrary, and the sum of the shooting angles of the cameras is larger than or equal to 360 degrees, such as a spherical camera and a fisheye camera. In this embodiment, two fisheye cameras are provided, images shot by the two fisheye cameras can be overlapped to some extent, and after the images shot by the two fisheye cameras are spliced together according to the overlapped parts, a panoramic image is obtained.
As can be understood by those skilled in the art, the shooting device in the image acquisition module in this embodiment is a panoramic camera, and the panoramic camera is a name that defines a camera device according to a shooting effect, and is not limited to a specific model and a specific category thereof. In practical application, the panoramic camera can be a camera with a panoramic shooting function, and directly outputs a panoramic image after processing a shot image inside; or a plurality of general cameras, and a panoramic image may be obtained by taking an image of a whole angle with the plurality of general cameras and processing the image with an image processing device, a computer program, or computer software.
The panoramic image can be collected by the image collection module provided by the embodiment, images at different angles can be obtained without ceaselessly rotating the camera for splicing, and when the sound source position is prevented from changing, the calculation module intercepts the sound source image according to a new sound source position, and due to the time difference of the rotation camera, the image switching efficiency of the image switching device and the real-time performance of the image switching of the sound source are improved.
EXAMPLE five
The present embodiment provides an image switching system including an external device and all the image switching apparatuses in the above embodiments.
As shown in fig. 9, in the present embodiment, the external device 20 is the same as the image switching apparatus 10, and includes an audio capture module 21, an image capture module 22, a network transceiver module 23, a computing module 24, and a display module 25 which are the same as the audio capture module 11, the image capture module 12, the network transceiver module 13, the computing module 14, and the display module 15 in the image switching apparatus 10, and the network transceiver module 23 of the external device 20 and the network transceiver module 13 of the image switching apparatus 10 can communicate with each other.
In a specific application, the display module 25 of the external device 20 receives the spherical panoramic image and the acoustic source image sent by the computing module 14 in the image switching apparatus 10 through the network transceiving module 23 of the external device 20;
the display module 15 of the image switching apparatus 10 receives the spherical panoramic image and the acoustic source image transmitted by the computing module 24 in the external device 20 through the network transceiver module 13 of the image switching apparatus 10.
In this embodiment, the signal transmission relationship between the image switching device and the external device is as follows: in the image switching device, a computing module intercepts images from panoramic images acquired by an image acquisition module according to audio data acquired by an audio acquisition module, sends the intercepted images to a network transceiver module of external equipment through the network transceiver module, and sends the intercepted images to a display module of the image switching device through the computing module; the display module in the external equipment displays the image sent by the image switching device, meanwhile, the computing module in the external equipment intercepts the image from the panoramic image collected by the image collecting module according to the audio data collected by the audio collecting module, sends the intercepted image to the network transceiving module of the image switching device in real time through the network transceiving module of the external equipment, and sends the intercepted image to the display module of the external equipment through the computing module. Thereby, the display module in the image switching device displays the sound source image at the side of the external device, the panoramic image, and the sound source image at the side of the image switching device; a display module in the external device displays the sound source image at one end of the image switching device, the panoramic image, and the sound source image at one end of the external device. The display module of the image switching device may also display the panoramic image at one end of the image switching device, and similarly, the display module of the external device may also display the panoramic image at one end of the external device.
It will be understood by those skilled in the art that the above-described truncated image represents a spherical panoramic image and/or a sound source image.
In a specific application, the use scene of the image switching device and the external device can be a video conference scene, when a video conference is carried out, the image switching device and the external device can intercept the image of the speaker according to the position of the speaker and send the image to the display device of the other party in real time for displaying, and when the video conference is carried out, the image of the speaker of the other party can be seen by the two parties.
The image switching system provided by this embodiment can realize real-time sound source image and panoramic image display at the image switching device end and the external device end, and switch the sound source image in real time when the sound source is changed.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (12)
1. An image switching device is applied to a sound box and is characterized by comprising an audio acquisition module, an image acquisition module, a network transceiving module, a calculation module and a display module;
the computing module is respectively connected with the audio acquisition module, the image acquisition module, the network transceiver module and the display module, and the network transceiver module is also communicated with external equipment;
the audio acquisition module acquires a sound signal and sends the sound signal to the calculation module;
the image acquisition module acquires a panoramic camera, and the panoramic camera acquires a panoramic image and sends the panoramic image to the computing module;
the computing module computes a sound source position according to the sound signal, generates a spherical panoramic image according to the panoramic image, intercepts a sound source image corresponding to the sound source position in the spherical panoramic image, and sends the spherical panoramic image and the sound source image to the display module and the network transceiver module; the computing module comprises a spherical coordinate system establishing unit, a voice box and a control unit, wherein the spherical coordinate system establishing unit is used for establishing a spherical coordinate system by taking the voice box as an origin of coordinates;
the audio acquisition module comprises a first microphone array arranged at one end of the sound box;
each microphone in the first microphone array is arranged on a first plane of the loudspeaker box;
the calculation module calculates a first angle of a sound source on a coordinate plane which passes through the origin of the spherical coordinate system and is parallel to the first plane according to the intensity difference or the time difference of the sound signals collected by the microphones in the first microphone array, and obtains a first position parameter of the sound source position;
the calculation module also comprises a preset parameter acquisition unit;
the preset parameter acquiring unit is used for acquiring preset parameters; the preset parameters represent the angle of the sound source in the vertical direction relative to the origin of the spherical coordinate system;
the calculation module determines the sound source position according to the preset parameter and the first position parameter;
and the network transceiver module sends the spherical panoramic image and the sound source image to the external equipment and receives the image sent by the external equipment.
2. The image switching apparatus according to claim 1, wherein the calculation module includes:
the panoramic mapping unit is used for mapping the panoramic image acquired by the image acquisition module into a spherical coordinate system when the panoramic image is sent to the calculation module, so as to obtain a spherical panoramic image;
and the coordinate conversion unit is used for converting the sound source position into a spherical coordinate to obtain a sound source position coordinate.
3. The image switching apparatus according to claim 2, wherein the calculation module further comprises:
the image intercepting unit is used for intercepting an image according to the sound source position coordinate in the spherical panoramic image to obtain the sound source image;
an image processing unit for processing the spherical panoramic image into a rectangular expanded image;
and the image transmission unit is used for sending the sound source image and the rectangular unfolded image to the display module and the network transceiving module.
4. The image switching apparatus of claim 1, wherein the audio acquisition module further comprises a second array of microphones disposed on a second plane parallel to the first plane;
and the calculation module calculates a second angle of a sound source on a horizontal plane which passes through the origin of the spherical coordinate system and is parallel to the second plane according to the intensity difference or the time difference of the sound signals collected by the microphones in the second microphone array, and obtains a second position parameter of the sound source position.
5. The image switching apparatus according to claim 4, wherein said calculation module determines said sound source position based on said preset parameter and said second position parameter.
6. The image switching apparatus according to claim 4, wherein the calculating module calculates a third angle of the sound source on a vertical plane passing through the origin of the spherical coordinate system and perpendicular to the horizontal plane according to an intensity difference or a time difference of the sound signals collected by the first microphone array and the second microphone array, and obtains a third position parameter of the sound source position.
7. The image switching apparatus according to claim 6, wherein said calculation module determines said sound source position based on said third position parameter and said first position parameter;
or
The calculation module determines the sound source position according to the third position parameter and the second position parameter.
8. The image switching apparatus according to any one of claims 1 to 3, wherein the panoramic camera is disposed at a center of one end of the sound box;
the panoramic camera shoots the panoramic image and sends the panoramic image to the computing module.
9. The image switching apparatus according to claim 8, wherein said panoramic camera includes at least two cameras;
the sum of the shooting angles of all the cameras is greater than or equal to 360 degrees;
the camera shoots images and sends the images to the computing module, and the computing module splices all the images sent by the camera into the panoramic image.
10. The image switching apparatus according to claim 9, wherein said camera is a fisheye camera.
11. An image switching system characterized by comprising an external device and the image switching apparatus according to any one of claims 1 to 10.
12. The image switching system according to claim 11, wherein the external device is the same as the image switching apparatus;
the display module of the external device receives the spherical panoramic image and the sound source image which are sent by the computing module in the image switching device through a network transceiving module of the external device;
and the display module of the image switching device receives the spherical panoramic image and the sound source image which are sent by the computing module in the external equipment through the network transceiving module of the image switching device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810557099.3A CN110557554B (en) | 2018-06-01 | 2018-06-01 | Image switching device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810557099.3A CN110557554B (en) | 2018-06-01 | 2018-06-01 | Image switching device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110557554A CN110557554A (en) | 2019-12-10 |
| CN110557554B true CN110557554B (en) | 2023-04-18 |
Family
ID=68734581
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810557099.3A Active CN110557554B (en) | 2018-06-01 | 2018-06-01 | Image switching device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110557554B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101442654A (en) * | 2008-12-26 | 2009-05-27 | 深圳华为通信技术有限公司 | Method, apparatus and system for switching video object of video communication |
| CN103973944A (en) * | 2013-02-06 | 2014-08-06 | 深圳市振华微电子有限公司 | Hemispherical omnidirectional imaging device and hemispherical omnidirectional imaging method |
| CN105592268A (en) * | 2016-03-03 | 2016-05-18 | 苏州科达科技股份有限公司 | Video conferencing system, processing device and video conferencing method |
| CN106341665A (en) * | 2016-09-30 | 2017-01-18 | 浙江宇视科技有限公司 | Tracking monitoring method and device |
| CN106357976A (en) * | 2016-08-30 | 2017-01-25 | 深圳市保千里电子有限公司 | Omni-directional panoramic image generating method and device |
| CN206559473U (en) * | 2017-02-20 | 2017-10-13 | 北京光年无限科技有限公司 | A kind of image collecting device and intelligent robot |
| CN107613243A (en) * | 2017-11-02 | 2018-01-19 | 深圳市裂石影音科技有限公司 | A kind of panoramic video recording arrangement and method for recording based on tone tracking |
| CN107948577A (en) * | 2017-12-26 | 2018-04-20 | 深圳市保千里电子有限公司 | A kind of method and its system of panorama video conference |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9906720B2 (en) * | 2014-12-05 | 2018-02-27 | Ricoh Company, Ltd. | Service system, information processing apparatus, and service providing method |
-
2018
- 2018-06-01 CN CN201810557099.3A patent/CN110557554B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101442654A (en) * | 2008-12-26 | 2009-05-27 | 深圳华为通信技术有限公司 | Method, apparatus and system for switching video object of video communication |
| CN103973944A (en) * | 2013-02-06 | 2014-08-06 | 深圳市振华微电子有限公司 | Hemispherical omnidirectional imaging device and hemispherical omnidirectional imaging method |
| CN105592268A (en) * | 2016-03-03 | 2016-05-18 | 苏州科达科技股份有限公司 | Video conferencing system, processing device and video conferencing method |
| CN106357976A (en) * | 2016-08-30 | 2017-01-25 | 深圳市保千里电子有限公司 | Omni-directional panoramic image generating method and device |
| CN106341665A (en) * | 2016-09-30 | 2017-01-18 | 浙江宇视科技有限公司 | Tracking monitoring method and device |
| CN206559473U (en) * | 2017-02-20 | 2017-10-13 | 北京光年无限科技有限公司 | A kind of image collecting device and intelligent robot |
| CN107613243A (en) * | 2017-11-02 | 2018-01-19 | 深圳市裂石影音科技有限公司 | A kind of panoramic video recording arrangement and method for recording based on tone tracking |
| CN107948577A (en) * | 2017-12-26 | 2018-04-20 | 深圳市保千里电子有限公司 | A kind of method and its system of panorama video conference |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110557554A (en) | 2019-12-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105306868B (en) | Video conferencing system and method | |
| CN105611167B (en) | focusing plane adjusting method and electronic equipment | |
| US9591229B2 (en) | Image tracking control method, control device, and control equipment | |
| JP2010250452A (en) | Arbitrary viewpoint image synthesizing device | |
| US11006072B2 (en) | Window system based on video communication | |
| JP2017511615A (en) | Video interaction between physical locations | |
| US11601731B1 (en) | Computer program product and method for auto-focusing a camera on an in-person attendee who is speaking into a microphone at a hybrid meeting that is being streamed via a videoconferencing system to remote attendees | |
| US10623698B2 (en) | Video communication device and method for video communication | |
| JP6410769B2 (en) | Information processing system, control method therefor, and computer program | |
| JP2009089324A (en) | Video conference system and program, and recoding medium | |
| US10788888B2 (en) | Capturing and rendering information involving a virtual environment | |
| KR20100121086A (en) | Ptz camera application system for photographing chase using sound source recognition and method therefor | |
| CN110557554B (en) | Image switching device and system | |
| CN104270616A (en) | Monitor platform for unitizing video information and geographic information | |
| JP2018019295A (en) | Information processing system, control method therefor, and computer program | |
| CN111325790A (en) | Target tracking method, device and system | |
| JP2004007283A (en) | Video distributing system, its program, and recording medium | |
| Su et al. | Rgb-d camera network calibration and streaming for 3d telepresence in large environment | |
| US10701313B2 (en) | Video communication device and method for video communication | |
| JP6664456B2 (en) | Information processing system, control method therefor, and computer program | |
| WO2019174442A1 (en) | Adapterization equipment, voice output method, device, storage medium and electronic device | |
| TW201318424A (en) | Video communication device for communication system, image processor and processing method thereof | |
| CN117640874A (en) | Image processing method and device | |
| JP2019153943A (en) | Audio processing device, video and audio processing device, video and audio distribution server, and program thereof | |
| CN213213667U (en) | Interactive conference device based on visual and sound fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |