CN108921012A - A method of utilizing artificial intelligence chip processing image/video frame - Google Patents
A method of utilizing artificial intelligence chip processing image/video frame Download PDFInfo
- Publication number
- CN108921012A CN108921012A CN201810470989.0A CN201810470989A CN108921012A CN 108921012 A CN108921012 A CN 108921012A CN 201810470989 A CN201810470989 A CN 201810470989A CN 108921012 A CN108921012 A CN 108921012A
- Authority
- CN
- China
- Prior art keywords
- image
- emergency event
- video frame
- neural network
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/44—Event detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
Present disclose provides a kind of methods using artificial intelligence chip processing image/video frame, including：The monitoring image of image processing apparatus acquisition monitoring system real time shooting；Image processing apparatus receives the video frame in the monitoring image, carries out artificial neural network operation to the video frame, output corresponds to the type of emergency event data of monitoring image after operation.This method can be judged by way of machine learning by the emergency event type in computer program real-time perfoming monitor video, save a large amount of human resources.
Description
Technical field
This disclosure relates to technical field of information processing, and in particular to a kind of emergency event automatic monitoring method.
Background technique
The emergency event in video is analyzed in the prior art, mainly by way of manually monitoring and judging, it is clear that should
The main problem of mode is：Artificial monitor and detection needs huge human resources, and manually browsing is difficult to implementation not
It is interrupted monitor and detection judgement；Further, manual retrieval's inefficiency, time complexity is high, and video resource substantial amounts,
It is affected by screen quality, artificial browsing can not necessarily accomplish accurate judgement；There are also be exactly that personal monitoring is set by hardware
Standby limitation can not realize the portability of system.
Summary of the invention
(1) technical problems to be solved
In view of this, the disclosure is designed to provide a kind of emergency event automatic monitoring method, to solve the above
At least partly technical problem.
(2) technical solution
To achieve the above object, the disclosure provides a kind of emergency event automatic monitoring method, including：
The monitoring image of image processing apparatus acquisition monitoring system real time shooting；
Image processing apparatus receives the video frame in the monitoring image, carries out artificial neural network fortune to the video frame
It calculates, output corresponds to the type of emergency event data of monitoring image after operation.
It further include to nerve net before the monitoring image for obtaining monitoring system real time shooting in further embodiment
Network model carries out adaptive training.
In further embodiment, the adaptivity training includes：Input includes at least emergency event video image and regards
The corresponding emergency event type code tag of the image of frequency frame；Video frame is input to current neural network structure to work as
In, and the update gradient direction of the network parameter of the affiliated type of current image is calculated by loss function and updates amplitude, lead to
Cross associated losses function calculate the affiliated type of the video clip whole neural network parameter update gradient direction and update
Amplitude；According to above-mentioned update gradient direction and update amplitude update neural network parameter.
In further embodiment, before receiving the video frame in the monitoring image to by preprocessing module to institute
Monitoring image is stated to be pre-processed.
In further embodiment, the pretreatment includes：To the cutting of monitoring image data, gaussian filtering, binaryzation, just
Then change and/or normalizes.
In further embodiment, the categorical data of the emergency event includes n bit, for indicating different type
Emergency event, n is integer greater than 1.
In further embodiment, carrying out artificial neural network operation to the video frame includes：Memory module receives prison
Image is controlled, which includes video frame；By direct memory access DMA by the instruction in storage unit, video requency frame data
It is passed to instruction cache module respectively with weight, inputs in neuron cache module and weight cache module；Control circuit is from instruction
Instruction is read in cache module, and computing circuit is passed to after being decoded；According to instruction, computing circuit executes corresponding neural network
Operation, and operation result is passed to output neuron cache module；The result that operation is finished is as current video frame image
Judging result is by the corresponding judging result storage address of direct memory access DMA.
In further embodiment, when described image is multiple image, each image successively executes artificial neural network fortune
It calculates, the resulting result judging result formation of operation judges that queue is re-used as the input of computing circuit, is weighted addition, determines whole
Emergency event type judging result of a monitor video at current time.
In further embodiment, the adaptive training process is off-line training, the input data of adaptivity training
It can be from external continuous time image collecting device.
In further embodiment, the computing circuit executes corresponding neural network computing, including：Pass through mlultiplying circuit
Input neuron is multiplied with weight data；By add tree the mutually multiply-add mistake is added step by step by add tree, is added
Quan He, and according to weighted sum biasing being set or is not added biasing；By activation primitive computing circuit, biasing is set or is not added to biasing
Weighted sum as input carry out activation primitive operation, obtain output neuron.
(3) beneficial effect
(1) the emergency event automatic monitoring method of the disclosure can by way of machine learning by computer program in real time into
Emergency event type judgement in row monitor video, saves a large amount of human resources；
(2) complex environment and video background may be implemented by machine recognition in the emergency event automatic monitoring method of the disclosure
Under type of emergency event monitor judgement, make up personal monitoring and judge that suffered monitor video picture quality and environmental disturbances are brought
Judging nicety rate reduction；
(3) include the image procossing for being able to carry out neural network computing in disclosed method, filled by the image procossing
Setting can make the hardware result size for entirely judging that early warning system needs be substantially reduced, and not need huge display system, can
Realize that mobile phone, tablet computer even specialized signal occur receiver and can be realized, it is easy to realize the portable design of system；
(4) disclosed method can greatly push monitor video emergency event to monitor universal, mention for social safety
For ensureing personal monitoring.
Detailed description of the invention
Fig. 1 is the emergency event automatic monitored control system block diagram of the embodiment of the present disclosure.
Fig. 2 is a kind of block diagram of image processing apparatus of automatic monitored control system in Fig. 1.
Fig. 3 is the block diagram of another image processing apparatus of automatic monitored control system in Fig. 1.
Fig. 4 is a kind of method flow diagram of the embodiment of the present disclosure handled monitoring image.
Fig. 5 is another method flow diagram of the embodiment of the present disclosure handled monitoring image.
Specific embodiment
Below with reference to the attached drawing in the embodiment of the present disclosure, the technical solution in the embodiment of the present disclosure is carried out clear, complete
Ground description, it is clear that described embodiment is only disclosure a part of the embodiment, instead of all the embodiments.Based on this
Disclosed embodiment, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, belongs to the protection scope of the disclosure.
In the disclosure, " video frame ", which refers to, carries out the time for exposure point diagram that of short duration exposure shooting obtains when video capture
Picture, these images, which continuously play, could constitute video；Video frame can be the current video frame of pending neural network computing, also
It can be and have already passed through neural network computing, and have the history video frame of corresponding true emergency event type code tag.This
In open, " emergency event " refers to be occurred suddenly, causes or may cause natural event, the accident calamity of serious social danger
Difficult, public accident or social event, including but not limited to flood, terrorist incident, social conflict, fire or power failure.
Existing video monitoring is often limited by picture quality, monitoring personnel's individual by personal monitoring and judgement
The image of the factors such as factor and environment, judging nicety rate and efficiency are lower.On the one hand the embodiment of the present disclosure provides a kind of prominent
Hair event automatic monitored control system and emergency event automatic monitoring method realize complex environment and video by machine automatic identification
Type of emergency event under background monitors judgement, makes up personal monitoring and judges suffered monitor video picture quality and environmental disturbances
The reduction of bring judging nicety rate.
Fig. 1 is the emergency event automatic monitored control system block diagram of the embodiment of the present disclosure.According to the embodiment of the present disclosure
On the one hand, a kind of emergency event automatic monitored control system 100, including monitoring device 110 and image processing apparatus 120 are provided.Wherein,
Monitoring device 110 is used to absorb the monitoring image of monitoring area；Image processing apparatus 120 is for receiving in the monitoring image
Video frame carries out artificial neural network operation to the video frame, and output corresponds to the emergency event class of monitoring image after operation
Type data.By the way that the image is exported type of emergency event data after neural network computing, emergency event type may be implemented
Judge automatically.The monitoring device 110 can be the various equipment that can shoot with video-corder image in the prior art, include but is not limited to take the photograph
Camera, camera or mobile phone are then converted to electronic format image (electronic format figure after image recording or picture frame
As may be pretreated).The image processing apparatus 120 of the embodiment of the present disclosure receives above-mentioned electronic format image, after to pass through
Hardware circuit carries out neural network computing to the electronic format image, obtains type of emergency event and (such as judges emergency event class
Type is event of fire) data.In neural network computing, used network model can be the existing various moulds of the prior art
Type, including but not limited to DNN (deep neural network), CNN (convolutional neural networks) or RNN (Recognition with Recurrent Neural Network) (such as
LSTM shot and long term memory network), and it is corresponding prominent comprising image or video frame in the neuron of the output layer of neural network
Send out event type data；The neural network operation is accelerated by the hardware device of the embodiment of the present disclosure, can be improved integral operation
Effect improves the efficiency of emergency event judgement.
Fig. 2 is a kind of block diagram of image processing apparatus of automatic monitored control system in Fig. 1.In some embodiments,
As shown in Fig. 2, image processing apparatus 120 includes memory module 121 and computing circuit 123；Wherein, memory module 121 is for depositing
Storage instruction, neural network parameter and operational data, operational data here include video frame (including current video frame and history view
Frequency frame) and the corresponding type of emergency event data of history video frame, computing circuit 123 is for executing phase to the operational data
The neural network computing answered.Wherein, memory module 121 can also store the output neuron number obtained after computing circuit operation
According to.Here neural network parameter includes but is not limited to weight, biasing and activation primitive.Preferably, initial in parameter
Changing weight is the weight that updates after historical data training, which can be realized by offline mode, can directly into
Pedestrian's artificial neural networks operation, saves the process being trained to neural network.
In some embodiments, it is transported in computing circuit 123 for executing corresponding neural network to the operational data
It calculates, including：Mlultiplying circuit is multiplied for that will input neuron with weight data；Add tree, for the mutually multiply-add mistake by adding
Method tree is added step by step, obtains weighted sum, and according to weighted sum biasing being set or is not added biasing；And activation primitive operation electricity
Road, the weighted sum for biasing to be set or be not added to biasing carry out activation primitive operation as input, obtain output neuron.As
Preferably, activation primitive can be sigmoid function, tanh function, ReLU function or softmax function.
In some embodiments, image processing apparatus 120 further includes control circuit 122, the control circuit 122 and storage mould
Block 121 and computing circuit 123 are electrically connected and (are directly or indirectly electrically connected), for translating the instruction stored in mould 121
Code is at operational order and is input to computing circuit 123, is also used to control the reading data of memory module 121 and computing circuit 123
Or calculating process.
In some embodiments, as shown in Fig. 2, image processing apparatus 120 can also include direct memory access DMA124
(Direct Memory Access), the input data for being stored in memory module 121, neural network parameter and instruction, with
It is called for control circuit 122 and computing circuit 123；Further it is also used to after computing circuit 123 calculates output neuron,
The output neuron is written to memory module 121.
In some embodiments, as shown in Fig. 2, image processing apparatus 120 further includes instruction cache module 125, for from
The direct memory access DMA124 cache instruction is called for control circuit 122.The instruction cache module 125 can be on piece
Caching, is integrated on processor by preparation process, can be improved processing speed when instruction is transferred, be saved integral operation
Time.
In some embodiments, image processing apparatus 120 further includes input neuron cache module 126, input nerve
First cache module 126 is used to cache input neuron from direct memory access DMA124, calls for computing circuit；Image procossing dress
Setting 120 can also include weight cache module 127, be used to cache weight from the direct memory access DMA124, for operation
Circuit 123 calls；Image processing apparatus 120 can also include output neuron cache module 128, be used to store from the fortune
It calculates circuit 123 and obtains the output neuron after operation, with output to direct memory access DMA124.Above-mentioned input neuron caching
Module, weight cache module and output neuron cache module may be on piece caching, are integrated in by semiconductor technology
On image processing apparatus 120, processing speed can be improved when reading and writing for computing circuit 123, saves the integral operation time.
Fig. 3 is the block diagram of another image processing apparatus 120 of automatic monitored control system in Fig. 1.As shown in figure 3,
Image processing apparatus 120 in the embodiment may include preprocessing module 129, be used for the prison absorbed to monitoring device 110
Control image is pre-processed, and the data for meeting neural network input format are converted into.Preferably, the pretreatment includes that will monitor
The image of device intake and/or video data cutting, gaussian filtering, binaryzation, regularization and/or normalization, to be met
The data of neural network input format.The Effect of Pretreatment is to improve the accuracy of subsequent neural network computing, to obtain standard
True number judgement.
It should be noted that the preprocessing module 129 of the embodiment of the present disclosure can be set in image processing apparatus 120,
It is integrally formed with image processing apparatus 120 by semiconductor technology, naturally it is also possible to be set to outside the image processing apparatus 120
Portion includes but is not limited to be set in monitoring device 110.
In some embodiments, the parameter (such as weight, biasing) in neural network, Ke Yitong can adaptively be trained
It crosses one pair of input or several includes the label (e.g. corresponding coding) in images and corresponding emergency event of video frame, it is defeated
Enter to the graphic processing facility 120 containing neural network structure, the corresponding network of present image is calculated and judged by loss function
The update gradient direction of parameter and update amplitude, so it is adaptive loss function is reduced by continuous iteration so that singly
The emergency event type of width video frame images and integral monitoring video judgement ground error rate constantly reduces, finally can be preferable
It returns to correct emergency event type and differentiates result.Preferably, above-mentioned adaptive training process is handled in real time.
In some embodiments, emergency event automatic monitored control system 100 can also include：Result treatment and displaying device,
For receiving the calculated type of emergency event data of image processing apparatus, format can be recognized by being converted into user, described to recognize
Format is picture, table, text, video and/or voice.Wherein, which can be according to image processing apparatus
The 120 type of emergency event data (an e.g. string encoding) calculated are converted to the cognizable format of user, such as are counted
Mould conversion, such as be converted to the analog signals such as sound；Such as format, picture format is converted to, then passes through exhibition again
Showing device (such as touch screen, display) shows user, selects for user；Such as control signal is converted to, control corresponding dress
It sets etc. and to react (such as control extinguishing device carries out fire-extinguishing operations to monitoring area) to the emergency event.
According to the another aspect of the embodiment of the present disclosure, a kind of emergency event automatic monitored control system is also provided, including at image
Device is managed, is used to receive the video frame in monitoring image, artificial neural network operation is carried out to the video frame, it is defeated after operation
Correspond to the type of emergency event data of monitoring image out.Wherein the set-up mode of the image processing apparatus can be with above-described embodiment
Middle image processing apparatus 120, it will not be described here.
Embodiment of the present disclosure still further aspect also provides a kind of emergency event automatic monitoring method.Fig. 4 is that the disclosure is implemented
A kind of method flow diagram that monitoring image is handled of example.A kind of emergency event automatic monitoring method as shown in Figure 4, packet
It includes：
S401：The monitoring image of image processing apparatus acquisition monitoring device real time shooting；
S402：Image processing apparatus receives the video frame in the monitoring image, carries out artificial neuron to the video frame
Network operations, output corresponds to the type of emergency event data of monitoring image after operation.
In step S401, calculation process is carried out to the image that monitoring device is shot with video-corder by way of obtaining in real time.It is this
Mode can judge in time whether emergency event occurs, in order to which related personnel handles emergency event scene.
In step S402, acquisition can be one section of video (comprising multiple images), be also possible to a single image (view
Frequency frame), by successively carrying out after neural network computing and being weighted to multiple image, finally provide a judgement knot
Fruit, or by the judging result for directly giving type of emergency event after single image progress neural network computing.
It in some embodiments, further include that adaptive training is carried out to neural network model before step S401.It is described
Adaptivity training may include steps of：The image that input includes at least emergency event video image video frame is corresponding
Emergency event type code tag；Video frame is input in current neural network structure, and passes through loss function meter
It calculates the update gradient direction of the network parameter of the affiliated type of current image and updates amplitude, being calculated by associated losses function should
The update gradient direction and update amplitude of the whole neural network parameter of the affiliated type of video clip；According to above-mentioned update gradient
Direction and update amplitude update neural network parameter.The adaptive training process is off-line training, and adaptivity is trained defeated
Entering data can be from external continuous time image collecting device.
In some embodiments, before receiving the video frame in the monitoring image to by preprocessing module to described
Monitoring image is pre-processed.The pretreatment includes：To the cutting of monitoring image data, gaussian filtering, binaryzation, regularization
And/or normalization.Corresponding preprocessing function can be realized by setting preprocessing module, for setting for corresponding preprocessing module
It sets and can refer to preprocessing module 129 in above-mentioned emergency event automatic monitored control system, it will not be described here.
In some embodiments, the categorical data of the emergency event includes n bit, for indicating different types of prominent
Hair event, n are the integer greater than 1.Certainly, for the image not comprising emergency event, also there is corresponding data type, such as adopt
It is indicated with coding n ' b0, but the data type is needed to be formed with the above-mentioned image containing emergency event and be distinguished.
In some embodiments, carrying out artificial neural network operation to the video frame includes：Memory module receives monitoring
Image, the monitoring image include video frame；By direct memory access DMA by storage unit instruction, video requency frame data and
Weight is passed to instruction cache module respectively, inputs in neuron cache module and weight cache module；Control circuit is slow from instruction
Instruction is read in storing module, and computing circuit is passed to after being decoded；According to instruction, computing circuit executes corresponding neural network fortune
It calculates, and operation result is passed to output neuron cache module；And the result for finishing operation is as current video frame image
Judging result by the corresponding judging result storage address of direct memory access DMA.
Further, when described image is multiple image, each image successively executes artificial neural network operation, operation institute
The result judging result formation obtained judges that queue is re-used as the input of computing circuit, is weighted addition, determines that entire monitoring regards
Emergency event type judging result of the frequency at current time.
In some embodiments, the computing circuit executes corresponding neural network computing, including：It will by mlultiplying circuit
Input neuron is multiplied with weight data；By add tree the mutually multiply-add mistake is added step by step by add tree, is weighted
With, and according to weighted sum biasing being set or is not added biasing；By activation primitive computing circuit, biasing is set or is not added to biasing
Weighted sum carries out activation primitive operation as input, obtains output neuron.
In addition, similar but there are difference with the method for above-described embodiment, the embodiment of the present disclosure is also providing a kind of burst
Event automatic monitoring method.Fig. 5 is another method flow diagram of the embodiment of the present disclosure handled monitoring image.Such as figure
A kind of emergency event automatic monitoring method shown in 5, including：
S501：Image processing apparatus obtains the multiple groups history image of external incoming type of emergency event to be judged；
S502：Image processing apparatus screens the video frame in multiple groups history image, successively carries out to the video frame artificial
Neural network computing, there are the correspondence type of emergency event data of emergency event in output multistage image after operation.
For step S501, by obtaining external multiple groups history image, the later period passes through calculating again, and screening is set out wherein
The image of raw emergency event, the and judge type of emergency event, the later period, to the non-emergent emergency event, (such as traffic was separated again
Chapter) it is handled.By carrying out operation and screening to a large amount of image automatically, a large amount of manual labors can be saved.
In step S52, by successively carrying out after neural network computing and being weighted to multiple image, finally give
A judging result out can carry out comprehensive descision to one section of image, further increase entire screening efficiency.
Specific neural network computational details can with and training method and preprocess method, be referred to above-described embodiment
Method in corresponding step carry out, it will not be described here.
It is exemplified below specific example above-mentioned emergency event automatic monitoring method is specifically described, wherein embodiment 1
It is handled in real time corresponding to the image to monitoring device, calculates the corresponding type of emergency event of image in time；Embodiment 2 is right
The segment that emergency event occurs should be searched in multiple (such as a large amount of) video clips.Specific device that following embodiment is mentioned,
The function and connection type of module, circuit and unit please refer to and are situated between in above-mentioned emergency event automatic monitored control system embodiment
The content to continue.
Embodiment 1：
The present embodiment provides a kind of method that can be handled in real time monitoring image and detect emergency event type, this method energy
It is enough to judge whether emergency event occurs in time, in order to which related personnel handles emergency event scene.
In the present embodiment 1, the memory module real time monitoring apparatus interaction monitoring image of image processing apparatus, by monitoring image
Video frame, which is stored in memory module, is used as input data, and input data includes but are not limited to the view of one or more groups of monitor videos
Frequency frame；Device combines the history video frame of a period of time and image/video frame tagging to be instructed according to input monitoring video frame
Practice, predict and provides the type of emergency event coding for this input.The video frame images of the monitor video wherein inputted both may be used
To be to be originally inputted, it is also possible to be originally inputted the result after pretreatment.
Image processing apparatus can carry out adaptive training, such as：The device inputs one group and (belongs to an emergency event
Video) or a width include monitoring video frame the corresponding emergency event type label of image (form of expression is coding, if
It is not that emergency event also has corresponding label coding).Device is input an image into current neural network structure, and is led to
It crosses loss function (cost function for measuring the corresponding emergency event type misjudgment of this image) and calculates and judge current image institute
Belong to the update gradient direction of the network parameter (such as weight, biasing) of type and update amplitude, passes through associated losses function (weighing apparatus
The cost function of all video frames misjudgment in the amount short time) calculate the whole neural network of the affiliated type of monitoring segment
The update gradient direction and update amplitude of parameter (such as weight, biasing), and then the adaptive continuous iteration that passes through reduces damage
Function is lost, so that the emergency event type of single width video frame images and integral monitoring video judgement ground error rate constantly subtracts
It is small, it finally can preferably return to correct emergency event type and differentiate result.
It in the emergency event type coding of input, needs at least n bit to indicate, then occurs without emergency event with volume
Code n ' b0 indicates that other emergency events are successively with n bit binary number come coded representation.These codings are as training screen simultaneously
The video frame tagging of monitoring inputs network as the training label of neural network and the output result of video to be judged.
Above-mentioned adaptive training process off-line；The type judgement of above-mentioned monitor video to be judged is handled in real time, this
In image processing apparatus be artificial neural network chip.
Above-mentioned apparatus work overall process be：
Step 1, the preprocessed module of input data is passed to memory module or directly incoming memory module；
Step 2, it is passed to instruction buffer, input by direct memory access DMA (Direct Memory Access) in batches
Neuron caches, in weight caching；
Step 3, control circuit reads instruction from instruction buffer, and computing circuit is passed to after being decoded；
Step 4, according to instruction, computing circuit executes corresponding operation,：In each layer of neural network, operation is main
It is divided into three steps：Step 4.1, corresponding input neuron is multiplied with weight；Step 4.2, execute add tree operation, i.e., it will step
Rapid 4.1 result is added step by step by add tree, obtains weighted sum, weighted sum biasing is set or is not processed as needed；Step
Rapid 4.3, activation primitive operation is executed to the result that step 4.2 obtains, obtains output neuron, and be passed to output neuron
In caching.
Step 5, from Step 2 to Step 4 is repeated, until all data operations finish.The result that operation is finished is as current
The judging result of video frame images is stored in corresponding judging result storage address by DMA.
Step 6, the resulting result of step 5 is judged into input of the queue as computing circuit, is weighted addition, obtains
It as a result is exactly emergency event type judging result of the entire monitor video at this moment.
According to affiliated functional requirement：If it is desired to obtain the judging result of video image emergency event, then above-mentioned neural network
Final weighted sum correspond to the judging result that emergency event coding result is the final video.
Embodiment 2：
Multiple history images can be screened the present embodiment provides a kind of, judge whether there is emergency event generation in image
And the judging result of type of emergency event is provided, operation and screening are carried out to a large amount of image by automation process, can be saved
The method that a large amount of manual labors can handle in real time monitoring image and detect emergency event type, this method can be judged in time
Whether emergency event is occurred, in order to which related personnel handles emergency event scene.
In the present embodiment 2, the storage circuit of image processing apparatus receives multiple video images, and video image video frame is deposited
Enter as input data in storage circuit, input data includes but are not limited to one group or a video image video frame；Device
It is trained, predicts and is provided for the prominent of this input according to inputted video image video frame and video image video frame tagging
Send out event type coding.The video image video frame images that wherein input are also possible to be originally inputted either be originally inputted
By the result after pretreatment.
In some embodiments, image processing apparatus is able to carry out adaptive training, such as：The device inputs one group (together
Belong to an emergency event video) or the corresponding emergency event of a secondary image comprising emergency event video image video frame
Type code tag (also has corresponding coding if not emergency event).The image of input is input to current mind by device
In network structure, and pass through loss function (cost function for measuring the corresponding emergency event type misjudgment of this image)
It calculates the update gradient direction for judging the network parameter (such as weight, biasing) of the affiliated type of current image and updates amplitude,
The video clip institute is calculated by associated losses function (cost function for measuring all video frames misjudgment in the short time)
Belong to the update gradient direction of the whole neural network parameter (such as weight, biasing) of type and update amplitude, and then is adaptive
Loss function is reduced by continuous iteration so that the emergency event type of single width video frame images and whole video is sentenced
Disconnected ground error rate constantly reduces, and finally can preferably return to correct emergency event type and differentiate result.
In some embodiments, in the emergency event type coding of input, at least n bit are needed to indicate, then without prominent
Hair event occurs to indicate that other emergency events are successively with n bit binary number come coded representation with coding n ' b0.These volumes simultaneously
Code inputs training label and to be judged video of the network as neural network as the video frame tagging for inputting training screen monitoring
Output result.
In some embodiments, above-mentioned adaptive training process off-line (does not need to take by being connected to the network to cloud
It is engaged on device, can be handled by local computer).Preferably, the type judgement of above-mentioned monitor video to be judged is to locate in real time
Reason.Preferably, image processing apparatus is artificial neural network chip.
Above-mentioned apparatus work overall process be：
Step 1, the preprocessed module of input data is passed to memory module or directly incoming memory module；
Step 2, it is passed to instruction buffer, input by DMA (Direct Memory Access, direct memory access) in batches
Neuron caches, in weight caching；
Step 3, control circuit reads instruction from instruction buffer, and computing circuit is passed to after being decoded；
Step 4, according to instruction, computing circuit executes corresponding operation：In each layer of neural network, operation mainly divides
For three steps：Step 4.1, corresponding input neuron is multiplied with weight；Step 4.2, add tree operation is executed, i.e., by step
4.1 result is added step by step by add tree, obtains weighted sum, weighted sum biasing is set or is not processed as needed；Step
4.3, activation primitive operation is executed to the result that step 4.2 obtains, obtains output neuron, and be passed to output neuron and delayed
In depositing.
Step 5, from Step 2 to Step 4 is repeated, it is known that all data operations finish.The result that operation is finished is as current
The judging result of video frame images is stored in corresponding judging result storage address by DMA.
Step 6, the resulting result of step 5 is judged into input of the queue as computing circuit, is weighted addition, obtains
As a result be exactly entire video emergency event type judging result.
According to affiliated functional requirement：If it is desired to obtain the judging result of video image emergency event, then above-mentioned neural network
Final weighted sum correspond to the judging result that emergency event coding result is the final video.
In embodiment provided by the disclosure, it should be noted that, disclosed relevant apparatus and method can pass through others
Mode is realized.For example, the apparatus embodiments described above are merely exemplary, such as the division of the part or module,
Only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple portions or module can be with
In conjunction with being perhaps desirably integrated into a system or some features can be ignored or does not execute.
In the disclosure, term "and/or" may be had been used.As used herein, term "and/or" means one
Or other or both (for example, A and/or B mean A or B or both A and B).
In the above description, for purpose of explanation, elaborate numerous details in order to provide each reality to the disclosure
Apply the comprehensive understanding of example.However, the skilled person will be apparent that, without certain in these details
Implementable one or more other embodiments.Described specific embodiment be not limited to the disclosure but in order to illustrate.
The scope of the present disclosure is not determined by specific example provide above, is only determined by following claim.At other
In the case of, in form of a block diagram, rather than it is illustrated in detail known circuit, structure, equipment, and operation is so as not to as making to retouching
The understanding stated thickens.In place of thinking to be suitable for, the ending of appended drawing reference or appended drawing reference is weighed in all attached drawings
It is multiple to indicate optionally correspondence or similar element with similar characteristics or same characteristic features, unless otherwise specifying or
Obviously.
Various operations and methods have been described.Certain methods are carried out in a manner of comparative basis in way of flowchart
Description, but these operations are optionally added to these methods and/or remove from these methods.In addition, although process
The particular order of the operation according to each example embodiment is illustrated, it is to be understood that, which is exemplary.Replacement is real
These operations can optionally be executed in different ways, combine certain operations, staggeredly certain operations etc. by applying example.Equipment is herein
Described component, feature and specific optional details can also may be optionally applied to method described herein, in each reality
It applies in example, these methods can be executed by such equipment and/or be executed in such equipment.
Each functional unit/subelement/module/submodule can be hardware in the disclosure, for example the hardware can be electricity
Road, including digital circuit, analog circuit etc..The physics realization of hardware configuration includes but is not limited to physical device, physics device
Part includes but is not limited to transistor, memristor etc..The memory module can be any magnetic storage medium appropriate or
Magnetic-optical storage medium, such as RRAM, DRAM, SRAM, EDRAM, HBM, HMC etc..
It is apparent to those skilled in the art that for convenience and simplicity of description, only with above-mentioned each function
The division progress of module can according to need and for example, in practical application by above-mentioned function distribution by different function moulds
Block is completed, i.e., the internal structure of device is divided into different functional modules, to complete all or part of function described above
Energy.
Particular embodiments described above has carried out further in detail the purpose of the disclosure, technical scheme and beneficial effects
Describe in detail bright, it should be understood that the foregoing is merely the specific embodiment of the disclosure, be not limited to the disclosure, it is all
Within the spirit and principle of the disclosure, any modification, equivalent substitution, improvement and etc. done should be included in the protection of the disclosure
Within the scope of.
Claims (10)
1. a kind of emergency event automatic monitoring method, which is characterized in that including：
The monitoring image of image processing apparatus acquisition monitoring system real time shooting；
Image processing apparatus receives the video frame in the monitoring image, carries out artificial neural network operation to the video frame,
Output corresponds to the type of emergency event data of monitoring image after operation.
2. the method according to claim 1, wherein obtain monitoring system real time shooting monitoring image it
Before, it further include that adaptive training is carried out to neural network model.
3. according to the method described in claim 2, it is characterized in that, adaptivity training includes：
The input emergency event type code tag corresponding including at least the image of emergency event video image video frame；
Video frame is input in current neural network structure, and the affiliated type of current image is calculated by loss function
The update gradient direction and update amplitude of network parameter, calculate the whole of the affiliated type of the video clip by associated losses function
The update gradient direction and update amplitude of somatic nerves network parameter；
According to above-mentioned update gradient direction and update amplitude update neural network parameter.
4. the method according to claim 1, wherein to logical before receiving the video frame in the monitoring image
Preprocessing module is crossed to pre-process the monitoring image.
5. according to the method described in claim 4, it is characterized in that, the pretreatment includes：To the cutting of monitoring image data, height
This filtering, binaryzation, regularization and/or normalization.
6. the method according to claim 1, wherein the categorical data of the emergency event include n bit,
For indicating different types of emergency event, n is the integer greater than 1.
7. the method according to claim 1, wherein carrying out artificial neural network operation packet to the video frame
It includes：
Memory module receives monitoring image, which includes video frame；
Instruction, video requency frame data and the weight in storage unit are passed to instruction buffer mould respectively by direct memory access DMA
Block inputs in neuron cache module and weight cache module；
Control circuit reads instruction from instruction cache module, and computing circuit is passed to after being decoded；
According to instruction, computing circuit executes corresponding neural network computing, and operation result is passed to output neuron caching mould
Block；
The result that operation finishes is judged to tie accordingly as the judging result of current video frame image by direct memory access DMA
Fruit storage address.
8. the method according to the description of claim 7 is characterized in that each image is successively held when described image is multiple image
Pedestrian's artificial neural networks operation, the resulting result judging result formation of operation judge that queue is re-used as the input of computing circuit, into
Row weighting summation determines entire monitor video in the emergency event type judging result at current time.
9. according to the method described in claim 3, it is characterized in that, the adaptive training process is off-line training, adaptively
Property training input data can be from external continuous time image collecting device.
10. the method according to the description of claim 7 is characterized in that the computing circuit executes corresponding neural network computing,
Including：
Neuron will be inputted by mlultiplying circuit to be multiplied with weight data；
By add tree the mutually multiply-add mistake is added step by step by add tree, obtains weighted sum, and add according to weighted sum
Bias or be not added biasing；
By activation primitive computing circuit, the weighted sum that biasing is set or be not added to biasing carries out activation primitive operation as input,
Obtain output neuron.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810470989.0A CN108921012B (en) | 2018-05-16 | 2018-05-16 | Method for processing image video frame by using artificial intelligence chip |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810470989.0A CN108921012B (en) | 2018-05-16 | 2018-05-16 | Method for processing image video frame by using artificial intelligence chip |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108921012A true CN108921012A (en) | 2018-11-30 |
| CN108921012B CN108921012B (en) | 2022-05-03 |
Family
ID=64402510
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810470989.0A Active CN108921012B (en) | 2018-05-16 | 2018-05-16 | Method for processing image video frame by using artificial intelligence chip |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108921012B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110503596A (en) * | 2019-08-12 | 2019-11-26 | 北京中科寒武纪科技有限公司 | Method for processing video frequency, device, electronic equipment and computer readable storage medium |
| CN110889497A (en) * | 2018-12-29 | 2020-03-17 | 中科寒武纪科技股份有限公司 | Learning task compiling method of artificial intelligence processor and related product |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8885929B2 (en) * | 2010-06-08 | 2014-11-11 | Gorilla Technology Inc. | Abnormal behavior detection system and method using automatic classification of multiple features |
| CN104636751A (en) * | 2014-12-11 | 2015-05-20 | 广东工业大学 | Crowd abnormity detection and positioning system and method based on time recurrent neural network |
| CN105160313A (en) * | 2014-09-15 | 2015-12-16 | 中国科学院重庆绿色智能技术研究院 | Method and apparatus for crowd behavior analysis in video monitoring |
| CN105512723A (en) * | 2016-01-20 | 2016-04-20 | 南京艾溪信息科技有限公司 | Artificial neural network calculating device and method for sparse connection |
| CN106022311A (en) * | 2016-06-13 | 2016-10-12 | 上海仪电（集团）有限公司中央研究院 | City monitoring video identification-based emergency event discovery method and system |
| CN106991477A (en) * | 2016-01-20 | 2017-07-28 | 南京艾溪信息科技有限公司 | A kind of artificial neural network compression-encoding device and method |
| CN107688795A (en) * | 2017-09-06 | 2018-02-13 | 全球能源互联网研究院有限公司 | A kind of monitoring method and terminal of power system video image |
| CN107818337A (en) * | 2017-10-09 | 2018-03-20 | 中国电子科技集团公司第二十八研究所 | Accident classification method and device based on depth convolutional neural networks |
-
2018
- 2018-05-16 CN CN201810470989.0A patent/CN108921012B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8885929B2 (en) * | 2010-06-08 | 2014-11-11 | Gorilla Technology Inc. | Abnormal behavior detection system and method using automatic classification of multiple features |
| CN105160313A (en) * | 2014-09-15 | 2015-12-16 | 中国科学院重庆绿色智能技术研究院 | Method and apparatus for crowd behavior analysis in video monitoring |
| CN104636751A (en) * | 2014-12-11 | 2015-05-20 | 广东工业大学 | Crowd abnormity detection and positioning system and method based on time recurrent neural network |
| CN105512723A (en) * | 2016-01-20 | 2016-04-20 | 南京艾溪信息科技有限公司 | Artificial neural network calculating device and method for sparse connection |
| CN106991477A (en) * | 2016-01-20 | 2017-07-28 | 南京艾溪信息科技有限公司 | A kind of artificial neural network compression-encoding device and method |
| CN106022311A (en) * | 2016-06-13 | 2016-10-12 | 上海仪电（集团）有限公司中央研究院 | City monitoring video identification-based emergency event discovery method and system |
| CN107688795A (en) * | 2017-09-06 | 2018-02-13 | 全球能源互联网研究院有限公司 | A kind of monitoring method and terminal of power system video image |
| CN107818337A (en) * | 2017-10-09 | 2018-03-20 | 中国电子科技集团公司第二十八研究所 | Accident classification method and device based on depth convolutional neural networks |
Non-Patent Citations (4)
| Title |
|---|
| BIT_TENG: "【深度学习】视频行为检测&分类方案整理", 《简书》 * |
| KEVINLIALI: "基于Deep Learning 的视频识别技术", 《阿里云开发者社区》 * |
| WEIXIN_33853794: "基于 Deep Learning 的视频识别方法概览", 《CSDN》 * |
| 阿里聚安全: "基于 Deep Learning 的视频识别方法概览", 《SEGMENTFAULT》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110889497A (en) * | 2018-12-29 | 2020-03-17 | 中科寒武纪科技股份有限公司 | Learning task compiling method of artificial intelligence processor and related product |
| CN110503596A (en) * | 2019-08-12 | 2019-11-26 | 北京中科寒武纪科技有限公司 | Method for processing video frequency, device, electronic equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108921012B (en) | 2022-05-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113936339B (en) | Fighting identification method and device based on double-channel cross attention mechanism | |
| CN106982359B (en) | Binocular video monitoring method and system and computer readable storage medium | |
| CN110135319A (en) | A kind of anomaly detection method and its system | |
| CN108566537A (en) | Image processing apparatus for carrying out neural network computing to video frame | |
| CN112733749A (en) | Real-time pedestrian detection method integrating attention mechanism | |
| CN108090458A (en) | Tumble detection method for human body and device | |
| CN110991362A (en) | Pedestrian detection model based on attention mechanism | |
| CN110363091A (en) | Face identification method, device, equipment and storage medium in the case of side face | |
| CN107169598A (en) | A kind of weather prediction method and system based on deep learning | |
| CN113065645B (en) | Twin attention network, image processing method and device | |
| CN110135476A (en) | A kind of detection method of personal safety equipment, device, equipment and system | |
| CN110659391A (en) | Video detection method and device | |
| CN110532959B (en) | Real-time violent behavior detection system based on two-channel three-dimensional convolutional neural network | |
| CN111783713A (en) | Weak supervision time sequence behavior positioning method and device based on relation prototype network | |
| CN109271912A (en) | Video classification methods, device, electronic equipment and storage medium | |
| CN114882530B (en) | Method for constructing lightweight convolutional neural network model for pedestrian detection | |
| CN108229432A (en) | Face calibration method and device | |
| Zeng et al. | A novel tensor decomposition-based efficient detector for low-altitude aerial objects with knowledge distillation scheme | |
| CN108921012A (en) | A method of utilizing artificial intelligence chip processing image/video frame | |
| CN113989931A (en) | Portable abnormal behavior intelligent analysis system | |
| CN117237547A (en) | Image reconstruction method, reconstruction model processing method and device | |
| CN108647660A (en) | A method of handling image using neural network chip | |
| CN107292284A (en) | Target re-detection method, device and unmanned plane | |
| CN115346169B (en) | Method and system for detecting sleep post behaviors | |
| Jiang et al. | Point-voxel absorbing graph representation learning for event stream based recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |