CA2656268A1 - Video surveillance system and method with combined video and audio recognition - Google Patents
Video surveillance system and method with combined video and audio recognition Download PDFInfo
- Publication number
- CA2656268A1 CA2656268A1 CA002656268A CA2656268A CA2656268A1 CA 2656268 A1 CA2656268 A1 CA 2656268A1 CA 002656268 A CA002656268 A CA 002656268A CA 2656268 A CA2656268 A CA 2656268A CA 2656268 A1 CA2656268 A1 CA 2656268A1
- Authority
- CA
- Canada
- Prior art keywords
- video
- audio
- recognition
- signals
- event
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims description 26
- 230000005236 sound signal Effects 0.000 claims description 29
- 238000012545 processing Methods 0.000 claims description 20
- 230000004044 response Effects 0.000 claims description 10
- 238000013500 data storage Methods 0.000 claims 2
- 238000001514 detection method Methods 0.000 abstract description 10
- 238000004880 explosion Methods 0.000 abstract description 7
- 230000006835 compression Effects 0.000 abstract description 5
- 238000007906 compression Methods 0.000 abstract description 5
- 206010039740 Screaming Diseases 0.000 abstract description 3
- 238000012544 monitoring process Methods 0.000 description 16
- 108010025925 alarin Proteins 0.000 description 5
- 230000000007 visual effect Effects 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 229930192851 perforin Natural products 0.000 description 2
- KHGNFPUMBJSZSM-UHFFFAOYSA-N Perforine Natural products COC1=C2CCC(O)C(CCC(C)(C)O)(OC)C2=NC2=C1C=CO2 KHGNFPUMBJSZSM-UHFFFAOYSA-N 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000003750 conditioning effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 235000012222 talc Nutrition 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B13/00—Burglar, theft or intruder alarms
- G08B13/16—Actuation by interference with mechanical vibrations in air or other fluid
- G08B13/1654—Actuation by interference with mechanical vibrations in air or other fluid using passive vibration detection systems
- G08B13/1672—Actuation by interference with mechanical vibrations in air or other fluid using passive vibration detection systems using sonic detecting means, e.g. a microphone operating in the audio frequency range
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B13/00—Burglar, theft or intruder alarms
- G08B13/18—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength
- G08B13/189—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems
- G08B13/194—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
- G08B13/196—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems using television cameras
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B19/00—Alarms responsive to two or more different undesired or abnormal conditions, e.g. burglary and fire, abnormal temperature and abnormal rate of flow
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B31/00—Predictive alarm systems characterised by extrapolation or other computation using updated historic data
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Business, Economics & Management (AREA)
- Computing Systems (AREA)
- Emergency Management (AREA)
- Signal Processing (AREA)
- Closed-Circuit Television Systems (AREA)
- Burglar Alarm Systems (AREA)
- Alarm Systems (AREA)
Abstract
A novel video surveillance system is made up of video and audio compression engine, a storage device and, a video and audio recognition engine. The video recognition engine detects such events as face recognition, motion detection etc, whereas audio recognition engine detects voice and other sound signatures indicating a potential alarm situation, e.g., panic voices such as screaming and yelling, or sounds such as gun shots, explosions. Combined recognition of audio and video signals provides for higher true alarm generation and lower false alarms level of the surveillance system. Additionally, the audio recognition engine provides information for directing video cameras in the direction of interest allowing better capture of an interesting scene.
Description
VIDEO SURVEILLANCE SYSTEM AND METHOD
WITH COMBINED VIDEO AND AUDIO RECOGNITION
BACKGROUND OF THE INVENTION
Field of the Invention [0001] The present invention generally relates to surveillance systems and methods for providing security, and, inore particularly to a novel on-line (real-time) video and audio recognition system and process for surveillance systems.
Description of the Prior Art [0002] Conventional video surveillance systems typically do not include any functionality or provision for monitoring audio; i.e., surveillance systeins do not include audio inputs at all. At best, typical video surveillance systems such as described in U.S. Patent Nos.
6,724,421 and 6,175,382 provide simultaneous recording of visual and audio information. In both types of video surveillance systems described in these references, video data is being analyzed by smart surveillance engines and are compressed for digital storage. These engines implement various recognition algorithms such as face recognition, motion detection, panic detection, stabbing motion detection etc. One alarming situation, for example, when monitoring an entrance to a high-rise building, involves a sudden fast motion of one person towards another one, iinplying a potential robbery, battery, or similar activity. A smart surveillance engine in this case will recognize (with some level of success which is less than 100%) fast sudden inotion and generate an alarm at the monitoring station. Police forces can be dispatched to the monitored location as a consequence of such an alarin. Obviously, fast sudden motion could have been generated by a child running towards his/her parent/friend and in this case the generated alarin becomes a false alarm which will cause an expensive dispatch of the police force. Another outcome of sinart surveillance engine inisdetection is an absence of alarni generation in case of a real emergency.
This case may arise, for example, wheri there is more than one person at the scene. Not sending a police force when the true emergency situation is talcing place is yet another drawback of current surveillance systems.
WITH COMBINED VIDEO AND AUDIO RECOGNITION
BACKGROUND OF THE INVENTION
Field of the Invention [0001] The present invention generally relates to surveillance systems and methods for providing security, and, inore particularly to a novel on-line (real-time) video and audio recognition system and process for surveillance systems.
Description of the Prior Art [0002] Conventional video surveillance systems typically do not include any functionality or provision for monitoring audio; i.e., surveillance systeins do not include audio inputs at all. At best, typical video surveillance systems such as described in U.S. Patent Nos.
6,724,421 and 6,175,382 provide simultaneous recording of visual and audio information. In both types of video surveillance systems described in these references, video data is being analyzed by smart surveillance engines and are compressed for digital storage. These engines implement various recognition algorithms such as face recognition, motion detection, panic detection, stabbing motion detection etc. One alarming situation, for example, when monitoring an entrance to a high-rise building, involves a sudden fast motion of one person towards another one, iinplying a potential robbery, battery, or similar activity. A smart surveillance engine in this case will recognize (with some level of success which is less than 100%) fast sudden inotion and generate an alarm at the monitoring station. Police forces can be dispatched to the monitored location as a consequence of such an alarin. Obviously, fast sudden motion could have been generated by a child running towards his/her parent/friend and in this case the generated alarin becomes a false alarm which will cause an expensive dispatch of the police force. Another outcome of sinart surveillance engine inisdetection is an absence of alarni generation in case of a real emergency.
This case may arise, for example, wheri there is more than one person at the scene. Not sending a police force when the true emergency situation is talcing place is yet another drawback of current surveillance systems.
[0003] Prior art video-only surveillance system is depicted in Figure 1. A
camera array 10 feeds video infonnation into a video coinpression engine 12 through video linlc 11.
The video information is compressed and sent through linlc 16 to a storage device 14 for a long-terin storing. Video infonnation is additionally fed to video recognition engine 13 through the same video link 11. Video recognition engine 13 perfonns video recognition tasks, such as face recognition, motion detection and others, and generates events and alarms that are sent through liiilc 17 to an events data base 15 and monitoring station 18. Monitoring station 18 may comprise a manned monitoring station whereby an operator perfonns real-time visual monitoring of a particular amount of cameras. When an emergency situation talces place, as interpreted by the operator, it is his/her decision whether or not to dispatch a police force or other emergency response team to the monitored area. It is clear from the above description that there is no use of audio information although such information is very often available at the monitored area.
camera array 10 feeds video infonnation into a video coinpression engine 12 through video linlc 11.
The video information is compressed and sent through linlc 16 to a storage device 14 for a long-terin storing. Video infonnation is additionally fed to video recognition engine 13 through the same video link 11. Video recognition engine 13 perfonns video recognition tasks, such as face recognition, motion detection and others, and generates events and alarms that are sent through liiilc 17 to an events data base 15 and monitoring station 18. Monitoring station 18 may comprise a manned monitoring station whereby an operator perfonns real-time visual monitoring of a particular amount of cameras. When an emergency situation talces place, as interpreted by the operator, it is his/her decision whether or not to dispatch a police force or other emergency response team to the monitored area. It is clear from the above description that there is no use of audio information although such information is very often available at the monitored area.
[0004] Prior Art video surveillance system with audio recording is shown in Figure 2. Camera array 20 feeds video information into video and audio compression engine 22 through video link 21. Siinultaneously, audio information is fed from microphone array 29 through audio link 30 to the video and audio coinpression engine 22. The video and audio information is compressed and sent through linlc 26 to a storage device 24 for a long-term storing. Video inforination is similarly fed to the video recognition engine 23 through the saine video linlc 21. Video recognition engine 23 perfonns video recognition tasks, such as face recognition, motion detection and others, and generates events and alanns that are sent through link 27 to a database 25 and monitoring station 28. Monitoring station 28 is a manned monitoring station whereby an operator perforins visual monitoring of a particular amount of cameras. When an emergency sittiation takes place, as interpreted by the operator, it is his/her decisioii wliether or not to dispatch a police force or other emergency response team to the monitored area. It is clear from the above description that there is no extraction of usefiil infornlation from the audio inputs although such infonnation is very often available in the audio signals obtained from at the monitored area.
[0005] As described above, a second type of surveillance system siinultaneously records video and audio infon-nation as well as iinplements smart surveillance engines for various video recognition tasks. Today, in these systems, audio information is coinpressed and recorded without being analyzed.
[0006] Today's surveillance systeins simply do not utilize rather precious audio infon-nation when analyzing video input. Obviously, this audio information is available and in many surveillance scenarios can be used very extensively.
[0007] Thus, it would be highly desirable to incorporate the use of audio infon-nation in video surveillance systems with the expectation that use of audio information will decrease the number of false alarins generated by surveillance system as well as increase the percentage of true alarms detected, while at the saine time, providing more information to the person evaluating an alarm.
Additionally, some events may be detected using audio and video information as opposed to such events being undetected using video information only.
SUMMARY OF THE INVENTION
Additionally, some events may be detected using audio and video information as opposed to such events being undetected using video information only.
SUMMARY OF THE INVENTION
[0008] It is thus an object of the present invention to provide a video surveillance system and method that incorporates the use of video inforination coupled with audio infon-nation obtained from the area under surveillance.
[0009] The surveillance system of the invention includes both video and audio signal inputs.
Video inputs are sourced from digital or analog cameras and audio inputs are received from microphones installed at a monitored area. Video and audio inforination is compressed and sent to a digital storage device. Gompression of the audio and video inforination is preferred in order to save amount of digital storage required for all cameras and microphones implemented.
Simultaneously with the recording, video and audio inputs are fed into a smart recognition engine that performs video recognition, audio recognition and performs instantaneous correlation of the results from video-audio recognition for detecting/recognizing a particular set of events, indicative of a panic situation, e.g., high-pitch screaming voices, explosion, gun shots, etc.
Alarins generated by the smart recognition engine may be sent to a monitoring station where a human operator decides whether to dispatch a police or emergency personnel to a monitored area.
Video inputs are sourced from digital or analog cameras and audio inputs are received from microphones installed at a monitored area. Video and audio inforination is compressed and sent to a digital storage device. Gompression of the audio and video inforination is preferred in order to save amount of digital storage required for all cameras and microphones implemented.
Simultaneously with the recording, video and audio inputs are fed into a smart recognition engine that performs video recognition, audio recognition and performs instantaneous correlation of the results from video-audio recognition for detecting/recognizing a particular set of events, indicative of a panic situation, e.g., high-pitch screaming voices, explosion, gun shots, etc.
Alarins generated by the smart recognition engine may be sent to a monitoring station where a human operator decides whether to dispatch a police or emergency personnel to a monitored area.
[0010] According to one aspect of the invention, the smart recognition engine executes available video recognition algoritluns, such as face recognition, motion detection, etc., as well as audio/speech recognition algorithins for speech recognition of a particular vocabulary ("Help", "Robbery", etc.). The audio recognition engine may be trained to recognize special audio signals such as gun shots, explosions, etc. as well as high-pitch and other voice signatures indicative of an alarm or emergency situation.
[0011] Using arrays of microphones placed in particular orientations, directions of sounds can be deterinined. Directional audio infonnation may then be delivered to a camera control unit for directing a cainera/caineras in the direction of interest. Further video/audio recognition may then be performed with better efficiency. Thus, for exainple, an explosion sound may be detected by audio recognition engine using an array of microphones in a monitored area. As a consequence, cameras will be directed into explosion direction and follow-on actions will talce place in the video recognition engine - from alarming the monitoring station up to scene recognition/understanding. The instantaneous use of results from video and audio recognition to direct the fiirther evaluation of recorded audio and video, and to direct iinproved recording of new video and audio inputs, advantageously iinproves the accuracy of the detection, reduces the time it takes to determine the nature of an alarin, and provides more inforination to a human operator evaluating the sittlation.
[0012] Outputs from the video recognition engine and the audio recognition engine are analyzed by mutual recognition engine and as a consequence final alarms are generated and forwarded to the monitoring station.
[0013] In keeping with these and other objects, according to a preferred aspect of the invention, there is provided a surveillance system and method, and computer program product, wherein the system comprises:
a means for generating real-time video signals coinprising video infonnation taken over an area under surveillance;
a means for obtaining real-time audio signals comprising audio information from the area under surveillance;
a means for simultaneously receiving the video signals and audio signals, determining relevant video and audio recognition information therefrom, and mutually correlating the real-time audio and video inforination to deterinine likelihood of occurrence of a particular event;
and, a means for generating an alarm condition based on occurrence of the particular event.
BRIEF DESCRIPTION OF THE DRAWINGS
a means for generating real-time video signals coinprising video infonnation taken over an area under surveillance;
a means for obtaining real-time audio signals comprising audio information from the area under surveillance;
a means for simultaneously receiving the video signals and audio signals, determining relevant video and audio recognition information therefrom, and mutually correlating the real-time audio and video inforination to deterinine likelihood of occurrence of a particular event;
and, a means for generating an alarm condition based on occurrence of the particular event.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Further features, aspects and advantages of the structures and methods of the present invention will becoine better understood with regard to the following description, appended claims, and accompanying drawings where:
[0015] Figure 1 illustrates a video only surveillance system according to the prior art;
[0016] Figure 2 illustrates a Video Stuveillance System with Audio Recording capability according to the prior art;
[0017] Figure 3 illustrates a Video Surveillance System witll Video and Audio Recognition according to the invention; and, [0018] Figure 4 illustrates details of the Smart Recognition Engine according to the invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] Figure 3 illustrates a Video Surveillance System with video and audio recognition according to the invention. As shown in Figure 3 a camera array 40 coinprising one or more still or video electronic cameras, e.g., CCD or CMOS cameras, either color or monochrome or having an equivalent combination of coinponents that capture an area under surveillance feeds video signals iinto a digital video and audio compression engine 42 through a video conununications link 41. Motion and operation of each camera device of the camera array 40 may be controlled by received control signals, e.g., under computer and/or software control.
Moreover, operational parameters for each camera in cainera array 40 including pan/tilt mirror, lens system, focus motor, pan motor, and tilt motor control are controlled by received control signals, as will be explained in greater detail herein. Prior to outputting the digital video signals, many signal processing techniques may be applied for reducing noise or providing filtering/image enhancing techniques, for exainple.
Moreover, operational parameters for each camera in cainera array 40 including pan/tilt mirror, lens system, focus motor, pan motor, and tilt motor control are controlled by received control signals, as will be explained in greater detail herein. Prior to outputting the digital video signals, many signal processing techniques may be applied for reducing noise or providing filtering/image enhancing techniques, for exainple.
[0020] Siinultaneously, a microphone array 49 comprising microphone sensor devices (omni-directional and/or highly directional microphones) that can convert acoustic pressure into electrical signals are provided to feed audio infonnation into the digital video and audio compression engine 42 through audio communications link 50. As known to slcilled artisans, a directivity level of the microphone array varies with respect to sound frequencies so that the number of microphones and the distance between the microphones may be determined in consideration of a required frequency range capable in order to provide any given degree of directivity. The microphones iinplemented in the array may be controlled under software control, for exainple, to accomplish these ends and, include transducers configured to have a pick-up pattern that may be distinctly biased towards various frequency receptions, e.g., in the range of human speech, explosions, gun shots, etc. In this maiuler the microphone array is ensured to be receptive to respond to an acoustic event's soundfield with a high degree of accuracy. Further audio signal conditioning techniques may be applied for digitizing the analog audio signals obtained using an A/D converter, for exainple, and for providing gaul control, reducing/filtering noise, for example. The digitized video and audio infonnation is digitally compressed and sent through link 46 to a memory storage device 44 for a long-term storage, e.g., a database, a hard disk drive, magnetic or optical media including but not limited to: a CD-ROM, DVD, tape, platter, disk array, or the like. The output of each camera of the camera array 40 is stored in the storage medium in a coinpressed format, such as MPEG 1, MPEG2, and the like.
Furthennore, the output of each camera of the array may be stored in a particular location on the storage medium associated with that camera or, is stored with an indication to which camera each stored output corresponds.
Furthennore, the output of each camera of the array may be stored in a particular location on the storage medium associated with that camera or, is stored with an indication to which camera each stored output corresponds.
[0021] As further shown in Figure 3, the same video inforlnation and audio infonnation is additionally siinultaneously fed to a smart recognition engine 43 through respective video link 41 and audio link 50. It is understood that the cominunication linlcs 41 and 50 between the respective camera array and audio microphone array and the video and audio compression engine 42 and smart recognition engine 43 may be hardwired, or wireless links may be einployed. Moreover, it is within the scope of the present invention for these coinmunication linlcs to take the forin of cable, satellite, RF and microwave transmission, fiber optics, and the lilce.
[0022] As will be described in greater detail herein, as further depicted in Figure 4, the sinart recognition engine 43 coinprises a video recognition engine 62, audio recognition engine 63, a inutual recognition engine and an alarin generation module 64. The smart recognition engine 43 iinplements software for controlling a computer device to perforin methods and processes for executing video recognition algorithins and face recognition algorithins.
These may be executed with and in conjunction with motion detection algorithins (for example, the well-known patch correlation or traclcing algoritluns that tracks the individual points) to estimate the motion of features in the image streain), etc. The smart recognition engine 43 additionally implements software for controlling a coinputer device to perform metliods and processes for executing audio recognition and speech recognition algoritluns. Speech recognition algoritluns implemented as computer readable instructions, data structures, program modules, etc. may be used for recognizing particular spoken words that may be potentially indicative of an emergency or alann-worthy situation ("Help", "Robbery", etc.).
These may be executed with and in conjunction with motion detection algorithins (for example, the well-known patch correlation or traclcing algoritluns that tracks the individual points) to estimate the motion of features in the image streain), etc. The smart recognition engine 43 additionally implements software for controlling a coinputer device to perform metliods and processes for executing audio recognition and speech recognition algoritluns. Speech recognition algoritluns implemented as computer readable instructions, data structures, program modules, etc. may be used for recognizing particular spoken words that may be potentially indicative of an emergency or alann-worthy situation ("Help", "Robbery", etc.).
[0023] An audio recognition engine 63, comprising coinputer readable instructions, data structures, prograin modules or other data, may be trained to recognize special audio signals such as gun shots, explosions, etc., as well as high-pitch sounds, e.g., screains, shrieks, and other sound and voice signatures associated with known potential alann provoking events. It is understood however, that the various recognition algorithms may be einployed according to the invention, that do not require prior training.
[0024] The computing device(s) implemented includes a general purpose coinputer device such as a PC, device, laptop, mobile device, and the like, having components including, but not limited to a processing unit, a system memory, and a system bus that couples various system components including the system memory to the processing unit. The computer device implements these components for executing the smart recogiution engine and audio recognition engine that are stored on a well-known computer-readable mediuin comprising any available media that can be accessed by the computer device including both removable, non-removable media, volatile, and nonvolatile media. The computer-readable recording may be centralized at one location or decentralized over computer systems connected via networlc, for exainple, and coinputer-readable recognition algorithms can be stored in the computer-readable recording medium and be executed in a decentralized manner.
[0025] Returning to Figure 3, using the array of microphones 49 in particular orientations, directions of sounds are determinable. Directional information concerning a sensed audio event is delivered to camera microphone control inodule 52 tluough a wired or wireless coinmunications link 53. The cainera/microphone control module 52 includes all of the software necessary to iinplement motor position control for directing camera/cameras of array 40 and controlling the positions of the microphone array 49 in the direction of interest by means of control signals 54. For instance, the control signals may be input to camera array 40 to adjust or control camera pan/tilt mirrors, lens system(s), focus motor, pan inotor, and tilt motor components and sub-systems. These control signals are additionally used to automatically direct the field of view seen by the cameras in order to obtain a better centered image or, inore zoomed, focused or more resolved image with more information regarding the actual alarm or alarm event. In one non-limiting example, in response to audio recognition of a gun shot audio signal by the sinart recognition engine, control signals may be generated that direct one or more caineras of the camera array to the scene to "look" in the direction of the gun-shot. If video cainera array is directed at the location of a crime from audio recognition of the gun-shot, then the "criine event" recognition will be better off because inore information about the gun-shot is available. Alternately, or in addition, these control signals inay be generated are used to automatically adjust the orientation of the microphones and the distance between the microphones to better receive the accompanying audio information. The microphones orientation may be additionally adjusted in consideration of detecting audio signals of a required frequency range, or for providing any given degree of directivity. Thus, for example, one or more microphones may be redirected to "listen" from a particular direction in response to a video recognition event.
[0026] More specifically, as shown in Figure 4, outputs from video recognition 62 engine and audio recognition engine 63 are analyzed by the mutual recognition engine 64 for processing the simultaneously received video and audio recognition information and ultimately deterinining whether an alarin condition exists. In this manner, alarms may be generated that are forwarded to the manned monitoring station 48 through coininunications linlc 47. That is, the recognition processes einployed as coinputer readable instructions, data structures, prograin modules, etc.
used in the inutual recognition engine 64 are generally based upon a pattern matching and/or hypotlieses evaluation. During an evaluation phase, there is determined an estimate of the probabilities of various events. This may be accoinplished by determining from the real-time video recognition inforination and audio signals to what extent a correlation exists between the respective recognized video scenes and accompanying recognized voice or audio signatures. Li an exainple recognition event, for recognizing a stabbing motion, the video information is used for the purpose of trying to evaluate probabilities of various video scenes.
If it is laiown that such scenes would be accoinpanied by high pitch voice (screaming etc) then detecting a high-pitch from the audio input will increase the probability of it being a result of a stabbing motion as captured in the video signals. An operator perfonns visual monitoring of a particular area surveyed by the camera aiTay 40 and when an alarm indication is provided by the alarm generating unit takes place, it the operator's decision to dispatch or not to dispatch a police or emergency personnel to the monitored area. It is clear from the above description that there is an extraction of usefiil inforination from the audio inputs which is, being combined with video recognition events, improves the total operation of the surveillance system.
used in the inutual recognition engine 64 are generally based upon a pattern matching and/or hypotlieses evaluation. During an evaluation phase, there is determined an estimate of the probabilities of various events. This may be accoinplished by determining from the real-time video recognition inforination and audio signals to what extent a correlation exists between the respective recognized video scenes and accompanying recognized voice or audio signatures. Li an exainple recognition event, for recognizing a stabbing motion, the video information is used for the purpose of trying to evaluate probabilities of various video scenes.
If it is laiown that such scenes would be accoinpanied by high pitch voice (screaming etc) then detecting a high-pitch from the audio input will increase the probability of it being a result of a stabbing motion as captured in the video signals. An operator perfonns visual monitoring of a particular area surveyed by the camera aiTay 40 and when an alarm indication is provided by the alarm generating unit takes place, it the operator's decision to dispatch or not to dispatch a police or emergency personnel to the monitored area. It is clear from the above description that there is an extraction of usefiil inforination from the audio inputs which is, being combined with video recognition events, improves the total operation of the surveillance system.
[0027] As further shown in Figure 4, communications linlc 60 between video recognition engine 62 and inutual recognition engine 64 is bidirectional, as are the communications link 61 between audio recognition engine 63 and mutual recognition engine 64. Bi-directionality of links 60 and 61 allows mutual influence of video and audio recognition algorithtns in the manner as described, which, as a consequence, gives better recognition level for video and audio as well as possibility to iinplement detection of particular events that were heretofore impossible to detect.
[0028] While the invention has been particularly shown and described with respect to ilhtstrative and prefonned embodiinents tliereof, it will be understood by those slcilled in the art that the foregoing and other changes in form and details may be made therein without departing from the spirit and scope of the invention which should be limited only by the scope of the appended claims.
Claims (27)
Having thus described our invention, what we claim as new, and desire to secure by Letters Patent is:
1. A surveillance system utilizing video and audio recognition comprising:
a means for generating real-time video signals comprising video information taken over an area under surveillance;
a means for obtaining real-time audio signals comprising audio information from said area under surveillance;
a means for simultaneously receiving said video signals and audio signals, determining relevant video and audio recognition information therefrom, and mutually correlating the real-time audio and video information to determine likelihood of occurrence of a particular event;
and, a means for generating an alarm condition based on occurrence of said particular event.
a means for generating real-time video signals comprising video information taken over an area under surveillance;
a means for obtaining real-time audio signals comprising audio information from said area under surveillance;
a means for simultaneously receiving said video signals and audio signals, determining relevant video and audio recognition information therefrom, and mutually correlating the real-time audio and video information to determine likelihood of occurrence of a particular event;
and, a means for generating an alarm condition based on occurrence of said particular event.
2. The system as claimed in Claim 1, wherein said processing means comprises a first recognition engine for processing said video signals for determining said video recognition information.
3. The system as claimed in Claim 2, wherein said processing means comprises a second recognition engine for processing said audio signals for determining said audio recognition information.
4. The system as claimed in Claim 1, wherein said processing means comprises a mutual recognition means for correlating the audio and video recognition information and increase ability of detecting occurrence of a particular event.
5. The system as claimed in Claim 4, wherein said means for generating real time video signals comprises one or more video camera devices, said mutual recognition means further comprising means for generating control signals for directing one or more cameras of the camera devices to capture video signals in the direction of the particular event in response to recognizing occurrence of that event based on said audio recognition of the event
6. The system as claimed in Claim 5, wherein each of said video camera devices comprise one or more of pan/tilt mirrors, lens system, focus motor, pan motor, and tilt motor components responsive to said control signals for adjusting one or more of pan, tilt, zoom, rotation, dolly, translate control parameters of the video camera devices.
7. The system as claimed in Claim 4, wherein said means for generating real time audio signals comprises one or more microphone devices, said mutual recognition means further comprising means for generating control signals to direct one or more microphones of the microphone devices to enable the capture of audio recognition information in the direction of the particular event in response to recognizing occurrence of a potential event based on said video recognition of the event.
8. The system as claimed in Claim 7, wherein each of said microphone devices are responsive to said control signals to automatically adjust the orientation of the microphones in consideration of detecting audio signals of a required frequency range.
9. The system as claimed in Claim 7, wherein each of said microphone devices are responsive to said control signals to automatically adjust the orientation of the microphones in consideration of receiving audio signals at any given degree of directivity.
10. The system as claimed in Claim 1, further comprising means for storing said audio and video data.
11. The system as claimed in Claim 10, further comprising means for compressing said audio and video data prior to storing it in said storage means.
12. A surveillance method utilizing video and audio recognition comprising the steps of:
simultaneously receiving at a processing means real-time video signals comprising video information taken over an area under surveillance and real-time audio signals comprising audio information from said area under surveillance, determining relevant video recognition and audio recognition information from said received video and audio signals;
mutually correlating the real-time audio and video recognition information to determine likelihood of occurrence of a particular event; and, generating an alarm condition based on occurrence of said particular event.
simultaneously receiving at a processing means real-time video signals comprising video information taken over an area under surveillance and real-time audio signals comprising audio information from said area under surveillance, determining relevant video recognition and audio recognition information from said received video and audio signals;
mutually correlating the real-time audio and video recognition information to determine likelihood of occurrence of a particular event; and, generating an alarm condition based on occurrence of said particular event.
13. The surveillance method as claimed in Claim 12, wherein said processing means comprises a first recognition engine implementing processing steps for determining said video recognition information from said video signals.
14. The surveillance method as claimed in Claim 13, wherein said processing means comprises a second recognition engine implementing processing steps for determining said audio recognition information from said audio signals.
15. The surveillance method as claimed in Claim 12, wherein said processing means comprises a mutual recognition means for correlating the audio and video recognition information and increasing ability of detecting occurrence of a particular event.
16. The surveillance method as claimed in Claim 15, wherein concurrent with said receiving step, a step of obtaining said real-time video signals by one or more video camera devices, said mutual recognition means further comprising means for generating control signals adapted for directing one or more cameras of the camera devices to capture video signals in the direction of the particular event in response to recognizing potential occurrence of that event based on said audio recognition of the event.
17. The surveillance method as claimed in Claim 16, wherein each of said one or more video camera devices comprise one or more of pan/tilt mirrors, lens system, focus motor, pan motor, and tilt motor components that are responsive to said control signals for adjusting one or more of pan, tilt, zoom, rotation, dolly, translate control parameters of the video camera devices.
18. The surveillance method as claimed in Claim 15, wherein concurrent with said receiving step, a step of obtaining said real-time audio signals by one or more microphone devices, said mutual recognition means further comprising means for generating control signals adapted for directing one or more microphones of the microphone devices to capture audio signals in the direction of the particular event in response to recognizing potential occurrence of that event based on video recognition of the event.
19. The surveillance method as claimed in Claim 18, wherein each of said microphone devices are responsive to said control signals to automatically adjust the orientation of the microphones in consideration of detecting audio signals of a required frequency range.
20. The surveillance method as claimed in Claim 18, wherein each of said microphone devices are responsive to said control signals to automatically adjust the orientation of the microphones in consideration of receiving audio signals at any given degree of directivity.
21. The surveillance method as claimed in Claim 12, further comprising the step of storing said audio and video data in a data storage device.
22. The surveillance method as claimed in Claim 21, further comprising the step of:
compressing audio and video data prior to said storing in said data storage device.
compressing audio and video data prior to said storing in said data storage device.
23. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to implement method steps for performing surveillance of an area using video and audio recognition, said method steps including the steps of:
simultaneously receiving at a processing means real-time video signals comprising video information taken over an area under surveillance and real-time audio signals comprising audio information from said area under surveillance, determining relevant video recognition and audio recognition information from said received video and audio signals;
mutually correlating the real-time audio and video recognition information to determine likelihood of occurrence of a particular event; and, generating an alarm condition based on occurrence of said particular event.
simultaneously receiving at a processing means real-time video signals comprising video information taken over an area under surveillance and real-time audio signals comprising audio information from said area under surveillance, determining relevant video recognition and audio recognition information from said received video and audio signals;
mutually correlating the real-time audio and video recognition information to determine likelihood of occurrence of a particular event; and, generating an alarm condition based on occurrence of said particular event.
24. The program storage device readable by a machine as claimed in Claim 23, wherein said processing means comprises: a first recognition engine implementing processing steps for determining said video recognition information from said video signals, and a second recognition engine implementing processing steps for determining said audio recognition information from said audio signals.
25. The program storage device readable by a machine as claimed in Claim 24, wherein said processing means comprises a mutual recognition means for correlating the audio and video recognition information and increasing ability of detecting occurrence of a particular event.
26. The program storage device readable by a machine as claimed in Claim 25, wherein concurrent with said receiving step, a step of obtaining said real-time video signals by one or more video camera devices, said mutual recognition means further comprising means for generating control signals adapted for directing one or more cameras of the camera devices to capture video signals in the direction of the particular event in response to recognizing potential occurrence of that event based on said audio recognition of the event.
27. The program storage device readable by a machine as claimed in Claim 25, wherein concurrent with said receiving step, a step of obtaining said real-time audio signals by one or more microphone devices, said mutual recognition means further comprising means for generating control signals adapted for directing one or more microphones of the microphone devices to capture audio signals in the direction of the particular event in response to recognizing potential occurrence of that event based on video recognition of the event.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/US2006/030560 WO2008016360A1 (en) | 2006-08-03 | 2006-08-03 | Video surveillance system and method with combined video and audio recognition |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA2656268A1 true CA2656268A1 (en) | 2008-02-07 |
Family
ID=38997456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002656268A Abandoned CA2656268A1 (en) | 2006-08-03 | 2006-08-03 | Video surveillance system and method with combined video and audio recognition |
Country Status (6)
| Country | Link |
|---|---|
| JP (1) | JP5043940B2 (en) |
| CN (1) | CN101501564B (en) |
| BR (1) | BRPI0621897B1 (en) |
| CA (1) | CA2656268A1 (en) |
| MX (1) | MX2009001254A (en) |
| WO (1) | WO2008016360A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022017702A1 (en) * | 2020-07-20 | 2022-01-27 | Robert Bosch Gmbh | Method for determining a noteworthy sub-sequence of a monitoring image sequence |
Families Citing this family (38)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2466242B (en) * | 2008-12-15 | 2013-01-02 | Audio Analytic Ltd | Sound identification systems |
| US9286911B2 (en) | 2008-12-15 | 2016-03-15 | Audio Analytic Ltd | Sound identification systems |
| CN102082948B (en) * | 2009-11-30 | 2012-07-25 | ***通信集团北京有限公司 | System, method and equipment for acquiring video information |
| CN103067655A (en) * | 2011-10-24 | 2013-04-24 | 鸿富锦精密工业（深圳）有限公司 | System and method of controlling video camera device |
| CN103136899B (en) * | 2013-01-23 | 2016-01-20 | 宁凯 | Based on the intelligent alarm method for supervising of Kinect somatosensory device |
| JP5958833B2 (en) | 2013-06-24 | 2016-08-02 | パナソニックＩｐマネジメント株式会社 | Directional control system |
| CN103747217A (en) * | 2014-01-26 | 2014-04-23 | 国家电网公司 | Video monitoring method and device |
| EP2927885A1 (en) * | 2014-03-31 | 2015-10-07 | Panasonic Corporation | Sound processing apparatus, sound processing system and sound processing method |
| US10182280B2 (en) | 2014-04-23 | 2019-01-15 | Panasonic Intellectual Property Management Co., Ltd. | Sound processing apparatus, sound processing system and sound processing method |
| EP2938097B1 (en) * | 2014-04-24 | 2017-12-27 | Panasonic Corporation | Sound processing apparatus, sound processing system and sound processing method |
| CN105338294A (en) * | 2014-08-07 | 2016-02-17 | 富士通株式会社 | Monitoring device and method |
| CN104269016A (en) * | 2014-09-22 | 2015-01-07 | 北京奇艺世纪科技有限公司 | Alarm method and device |
| CN104333686B (en) * | 2014-11-27 | 2018-03-27 | 天地伟业技术有限公司 | Intelligent monitoring camera and its control method based on face and Application on Voiceprint Recognition |
| US9813484B2 (en) | 2014-12-31 | 2017-11-07 | Motorola Solutions, Inc. | Method and apparatus analysis of event-related media |
| US20160241818A1 (en) * | 2015-02-18 | 2016-08-18 | Honeywell International Inc. | Automatic alerts for video surveillance systems |
| JP6682222B2 (en) | 2015-09-24 | 2020-04-15 | キヤノン株式会社 | Detecting device, control method thereof, and computer program |
| US9598076B1 (en) * | 2015-10-22 | 2017-03-21 | Ford Global Technologies, Llc | Detection of lane-splitting motorcycles |
| CN105491336B (en) * | 2015-12-08 | 2018-07-06 | 成都芯软科技发展有限公司 | A kind of low power image identification module |
| CN106028217B (en) * | 2016-06-20 | 2020-01-21 | 咻羞科技(深圳)有限公司 | Intelligent equipment interaction system and method based on audio recognition technology |
| CN106023515A (en) * | 2016-07-06 | 2016-10-12 | 中警科技（江苏）开发有限公司 | Remote automatic alarm police kiosk |
| WO2018075068A1 (en) | 2016-10-21 | 2018-04-26 | Empire Technology Development Llc | Selecting media from mass social monitoring devices |
| CN106600876A (en) * | 2017-01-24 | 2017-04-26 | 璧典寒 | Automatic machine room duty alarming system and alarming method |
| US10810854B1 (en) | 2017-12-13 | 2020-10-20 | Alarm.Com Incorporated | Enhanced audiovisual analytics |
| CN109033997A (en) * | 2018-07-02 | 2018-12-18 | 厦门快商通信息技术有限公司 | A kind of lumbering event detecting method and system |
| WO2020022055A1 (en) * | 2018-07-24 | 2020-01-30 | ソニー株式会社 | Information processing device and method, and program |
| CN109089087B (en) * | 2018-10-18 | 2020-09-29 | 广州市盛光微电子有限公司 | Multi-channel audio-video linkage device |
| CN109543538A (en) * | 2018-10-23 | 2019-03-29 | 深圳壹账通智能科技有限公司 | Obtain method, apparatus, computer equipment and the storage medium of the track of alert object |
| TWI687753B (en) * | 2018-12-06 | 2020-03-11 | 宏碁股份有限公司 | Panoramic camera and panoramic photography system |
| CN110336976A (en) * | 2019-06-13 | 2019-10-15 | 长江大学 | A kind of intelligent monitoring probe and system |
| CN111091073A (en) * | 2019-11-29 | 2020-05-01 | 清华大学 | Abnormal event monitoring equipment and method combining video and audio |
| EP3839909A1 (en) * | 2019-12-18 | 2021-06-23 | Koninklijke Philips N.V. | Detecting the presence of an object in a monitored environment |
| CN111460907B (en) * | 2020-03-05 | 2023-06-20 | 浙江大华技术股份有限公司 | Malicious behavior identification method, system and storage medium |
| CN111818237A (en) * | 2020-07-21 | 2020-10-23 | 南京智金科技创新服务中心 | Video monitoring analysis system and method |
| CN112396801A (en) * | 2020-11-16 | 2021-02-23 | 苏州思必驰信息科技有限公司 | Monitoring alarm method, monitoring alarm device and storage medium |
| GB202019713D0 (en) * | 2020-12-14 | 2021-01-27 | Vaion Ltd | Security system |
| CN112929372A (en) * | 2021-02-06 | 2021-06-08 | 北京第七九七音响股份有限公司 | Network intelligent audio terminal, monitoring method and monitoring system |
| CN113920660B (en) * | 2021-09-30 | 2023-04-18 | 中国工商银行股份有限公司 | Safety monitoring method and system suitable for safety storage equipment |
| GB2620594A (en) * | 2022-07-12 | 2024-01-17 | Ava Video Security Ltd | Computer-implemented method, security system, video-surveillance camera, and server |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3381343B2 (en) * | 1993-12-03 | 2003-02-24 | 株式会社日立製作所 | Monitoring device |
| JPH0983856A (en) * | 1995-09-07 | 1997-03-28 | Nippon Telegr & Teleph Corp <Ntt> | Intelligent camera equipment |
| US6175382B1 (en) * | 1997-11-24 | 2001-01-16 | Shell Oil Company | Unmanned fueling facility |
| US6611206B2 (en) * | 2001-03-15 | 2003-08-26 | Koninklijke Philips Electronics N.V. | Automatic system for monitoring independent person requiring occasional assistance |
| CN1186923C (en) * | 2003-04-03 | 2005-01-26 | 上海交通大学 | Abnormal object automatic finding and tracking video camera system |
| JP4175180B2 (en) * | 2003-05-29 | 2008-11-05 | 松下電工株式会社 | Monitoring and reporting system |
| CN1716329A (en) * | 2004-06-29 | 2006-01-04 | 乐金电子（沈阳）有限公司 | Baby monitoring system and its method using baby's crying frequency |
| CN200966113Y (en) * | 2006-11-08 | 2007-10-24 | 天津三星电子有限公司 | A monitor with the audio locking functions |
-
2006
- 2006-08-03 MX MX2009001254A patent/MX2009001254A/en active IP Right Grant
- 2006-08-03 WO PCT/US2006/030560 patent/WO2008016360A1/en active Application Filing
- 2006-08-03 CA CA002656268A patent/CA2656268A1/en not_active Abandoned
- 2006-08-03 BR BRPI0621897-0A patent/BRPI0621897B1/en active IP Right Grant
- 2006-08-03 CN CN2006800555140A patent/CN101501564B/en active Active
- 2006-08-03 JP JP2009522745A patent/JP5043940B2/en not_active Expired - Fee Related
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022017702A1 (en) * | 2020-07-20 | 2022-01-27 | Robert Bosch Gmbh | Method for determining a noteworthy sub-sequence of a monitoring image sequence |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101501564B (en) | 2012-02-08 |
| BRPI0621897A2 (en) | 2011-03-29 |
| WO2008016360A1 (en) | 2008-02-07 |
| MX2009001254A (en) | 2009-02-11 |
| CN101501564A (en) | 2009-08-05 |
| BRPI0621897B1 (en) | 2018-03-20 |
| JP2009545911A (en) | 2009-12-24 |
| JP5043940B2 (en) | 2012-10-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2656268A1 (en) | Video surveillance system and method with combined video and audio recognition | |
| US20060227237A1 (en) | Video surveillance system and method with combined video and audio recognition | |
| CN109300471B (en) | Intelligent video monitoring method, device and system for field area integrating sound collection and identification | |
| JP4861723B2 (en) | Monitoring system | |
| CN102737480B (en) | Abnormal voice monitoring system and method based on intelligent video | |
| CN101123722B (en) | Panorama video intelligent monitoring method and system | |
| USRE43147E1 (en) | Abnormality detection and surveillance system | |
| WO1997008896A1 (en) | Open area security system | |
| Valenzise et al. | Scream and gunshot detection and localization for audio-surveillance systems | |
| USRE44225E1 (en) | Abnormality detection and surveillance system | |
| CN111601074A (en) | Security monitoring method and device, robot and storage medium | |
| WO2014174760A1 (en) | Action analysis device, action analysis method, and action analysis program | |
| CN109551500A (en) | Supervisory control of robot alarm system | |
| Andersson et al. | Fusion of acoustic and optical sensor data for automatic fight detection in urban environments | |
| KR101899436B1 (en) | Safety Sensor Based on Scream Detection | |
| JP2012048689A (en) | Abnormality detection apparatus | |
| KR101384781B1 (en) | Apparatus and method for detecting unusual sound | |
| SG178563A1 (en) | Method and system for event detection | |
| CN110634506A (en) | Voice data processing method and device | |
| Park et al. | Sound learning–based event detection for acoustic surveillance sensors | |
| KR20120103286A (en) | Method and apparatus of detecting event of interest related to vehicles using audio signals | |
| KR101882309B1 (en) | safety light and safety system using voice recognition | |
| JP4175180B2 (en) | Monitoring and reporting system | |
| KR102319687B1 (en) | Surveillance system adopting wireless acoustic sensors | |
| CN114511978A (en) | Intrusion early warning method and device, vehicle and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| FZDE | Discontinued |
Effective date: 20170929 |
|
| FZDE | Discontinued |
Effective date: 20170929 |