CN111915687A - Background extraction method with depth information and color information - Google Patents
Background extraction method with depth information and color information Download PDFInfo
- Publication number
- CN111915687A CN111915687A CN202010669614.4A CN202010669614A CN111915687A CN 111915687 A CN111915687 A CN 111915687A CN 202010669614 A CN202010669614 A CN 202010669614A CN 111915687 A CN111915687 A CN 111915687A
- Authority
- CN
- China
- Prior art keywords
- depth
- matrix
- background
- image
- vibe
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000605 extraction Methods 0.000 title claims abstract description 10
- 239000011159 matrix material Substances 0.000 claims abstract description 56
- 238000000034 method Methods 0.000 claims abstract description 15
- 238000001514 detection method Methods 0.000 claims abstract description 8
- 230000008569 process Effects 0.000 claims abstract description 3
- 238000012545 processing Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 abstract description 2
- 238000005516 engineering process Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Mathematical Physics (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Computational Mathematics (AREA)
- Pure & Applied Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
Abstract
A background extraction method with depth information and color information belongs to the technical field of automatic detection of video sequences. It includes step 1, initialization, decomposing the image matrix with depth information in the first frame into image BGR matrix v1Sum-depth image floating point type single-channel matrix d1And use of v1Initialize the ViBe algorithm, and d1Attaching the matrix element by element to a depth model matrix D; step 2: sequentially reading in the t frame depth imageDecomposition of t-frame depth image into image BGR matrix vtSum-depth image floating point single-channel type matrix dtAnd processes both matrices. According to the method, whether the foreground depth distance is larger than the background depth distance or not can be judged through the background floating point type matrix with the depth information and the current floating point type matrix with the depth information, and then the ghost problem of the ViBe is detected; moreover, a high-quality background BGR matrix with color information can be obtained by utilizing a ViBe sample set; the method has extremely high calculation speed and is suitable for industrial application.
Description
Technical Field
The invention belongs to the technical field of automatic detection of video sequences, and particularly relates to a method for extracting a background under the condition of an image sequence with depth information.
Background
With the development of society, the number of mobile terminals with camera functions gradually increases, the related technologies of the terminal camera functions gradually increase, and with the development of machine vision, the subversive technologies of automatic driving, virtual reality and the like gradually develop, a 3D camera is adopted to perform object recognition, behavior recognition and more related applications of scene modeling.
Background modeling is fundamental work of sequence image analysis and is a hot problem for researchers at home and abroad at present. The modeling result has important influence on the subsequent processing of video image such as motion detection, moving target classification, tracking and behavior understanding. However, due to the difference of the actual application environments and the diversity of the background, it is difficult to establish a good background sample, so that in the actual application, different algorithms are needed to optimize the design scheme, and a good result can be obtained.
Since 2011 a paper of the ViBe algorithm is published, the ViBe algorithm is applied to moving target detection of a visual screen sequence, but the technique has a ghost problem (a detected moving object does not correspond to any real moving object), and the ghost problem can cause misjudgment of a downstream technique.
Disclosure of Invention
In view of the above problems in the prior art, an object of the present invention is to provide a background extraction method capable of extracting a background with color information and a background with depth information from an image sequence with depth information, and capable of well suppressing a ghost problem of a ViBe and efficiently updating a color information model.
The invention provides the following technical scheme:
a background extraction method with depth information and color information is characterized by comprising the following steps:
step 1, initialization, specifically comprising the following steps:
step 1.1, decomposing the image matrix with depth information in the first frame into an image BGR matrix v1Sum-depth image floating point type single-channel matrix d1；
Step 1.2, use v1Initialize the ViBe algorithm, and d1Attaching the matrix element by element to a model single-channel floating-point type matrix D with the same size;
step 2: sequentially reading in the depth image of the t-th frame, and decomposing the depth image of the t-th frame into an image BGR matrix v according to the operation in the step 1.1tSum-depth image floating point single-channel type matrix dtAnd processing the two matrixes according to the following steps:
step 2.1: using vtUpdating the ViBe algorithm;
step 2.2: updating the matrix D;
step 2.3: judging whether the depth distance of the current pixel is closer than the depth distance of the background;
step 2.4: updating a ViBe sample space at the false detection position;
step 2.5: and (3) operating to obtain a color background BGR floating-point matrix B, and after the operation is finished, executing the step 2 again, wherein B is the background with color information, and D is the background with depth information.
The method for extracting the background with the depth information and the color information is characterized in that in the step 1.1, an image matrix v1And depth image d1The values of (A) and (B) are width × height.
The method for extracting the background with the depth information and the color information is characterized in that in the step 2.2, the matrix D is updated according to the following formula:
wherein,
representing depth floating point values of a depth moment floating point mono-channel matrix D at (x, y) at the t-th frame of a depth image sequence, Dx,yRepresenting the depth floating point value of the depth background matrix at (x, y), wherein alpha is a threshold value and is more than or equal to 0 and less than or equal to 1; wherein, α represents the weight of the current depth frame, and (1- α) is the weight of the background depth model.
The method for extracting the background with the depth information and the color information is characterized in that in the step 2.3, whether the current pixel depth distance is closer to the background depth distance is judged according to the following formula:
RD＝(max-min)/T
wherein max and min are the maximum distance and the minimum distance measurable by the depth camera, respectively, T is the threshold value, when C isx,yIf 0, the depth distance at (x, y) is farther than the background depth distance or close to the background depth, and if the place is judged as the foreground by the ViBe, the ViBe at the place will be judged as the foregroundThe background is misjudged as foreground, i.e. ghost.
The background extraction method with the depth information and the color information is characterized in that in the step 2.4, the ViBe sample space at the false detection position is updated according to the following formula:
wherein r is1,r2,r3,r4Are all random numbers, r1,r3∈[0,N)，r2,r4E [0, γ)), and omega (x, y) represents the coordinates of a random point in eight neighborhoods of coordinates (x, y), samples is the set of sample space matrices of ViBe,
represents the r-th sample value on (x, y), which is the color BGR value, N is the sample space size of each pixel of the ViBe algorithm, and γ is the sample update threshold of ViBe.
The background extraction method with depth information and color information is characterized in that in step 2.5, a color background BGR floating point matrix B is obtained according to the following formula:
wherein, Bx，yAnd (3) the BGR value of the color background BGR floating-point matrix B at the position (x, y) is a three-dimensional vector, the summation mode at the position is the addition of corresponding components, and after the operation is finished, the step 2 is executed again, wherein in the process, B is the background with color information, and D is the background with depth information.
By adopting the technology, compared with the prior art, the invention has the following beneficial effects:
the method can obtain the background floating point type matrix with the depth information in the image sequence with the depth information through calculation, judge whether the foreground depth distance is larger than the background depth distance through the background floating point type matrix with the depth information and the current floating point type matrix with the depth information, further detect the ghost problem of the ViBe, and then update the ViBe sample set to enable the color information in the ViBe sample to be closer to the background, so that the ViBe sample set is utilized to obtain the high-quality background BGR matrix with the color information.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
On the contrary, the invention is intended to cover alternatives, modifications, equivalents and alternatives which may be included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of the present invention, certain specific details are set forth in order to provide a better understanding of the present invention. It will be apparent to one skilled in the art that the present invention may be practiced without these specific details.
Step 1: initializing, specifically comprising the following steps:
step 1.1: we use RealSenseDepthParametric D435 depth camera to extract depth video, and decompose the image matrix with depth information in the first frame into image BGR matrix v through RealSensesSDK toolkit1Sum-depth image floating point type single-channel matrix d1The sizes of the two are width multiplied height;
step 1.2: using v1Initialize the ViBe algorithm, and d1Attaching the matrix element by element to a depth model single-channel floating-point matrix D with the same size;
step 2: according to step 1.1, reading in the t frame depth image in sequence and adding depthDecomposition of degree image into image BGR matrix vtSum-depth image floating point single-channel type matrix dtAnd processing the two matrixes according to the following steps:
step 2.1: using vtUpdating the ViBe algorithm;
step 2.2: updating the matrix D according to equation (1):
wherein,
representing depth floating point values of a depth moment floating point mono-channel matrix D at (x, y) at the t-th frame of a depth image sequence, Dx,yRepresenting a depth floating point value of the depth background matrix at (x, y), where α is a threshold value, and α is greater than or equal to 0 and less than or equal to 1, and a value of α needs to be defined according to an actual application environment, and is usually defined as α being 0.15;
step 2.3: according to the formula (2), judging whether the depth distance of the current pixel is closer than the depth distance of the background:
RD＝(max-min)/T (4)
wherein max and min are the longest distance and the shortest distance that can be measured by the depth camera, respectively, under the RealSenseDepthCameraD435 depth camera, max is 10, min is 0.1, T is the threshold, and T is 100, if Cx,yIf the distance of the depth at the position (x, y) is more distant than the depth of the background or close to the depth of the background, and if the position is judged as the foreground by the ViBe, the ViBe judges the background as the foreground by mistake, namely ghost image;
step 2.4: updating the ViBe sample space according to the formulas (5) and (6):
wherein r is1,r2,r3,r4Are all random numbers, r1,r3∈[0,N)，r2,r4E [0, γ)), and omega (x, y) represents the coordinates of a random point in eight neighborhoods of coordinates (x, y), samples is the set of sample space matrices of ViBe,
representing the r-th sample value on (x, y), which is a color BGR value, N is a sample space size of each pixel of the ViBe algorithm, N is 20 according to the ViBe algorithm, γ is a sample update threshold of the ViBe, and γ is 16 according to the ViBe algorithm;
step 2.5: the color background BGR floating-point matrix B is obtained according to the following formula:
wherein, Bx，yAnd (3) adding corresponding components for the BGR value of the color background BGR floating-point matrix B at (x, y), and executing the step 2 again after the operation is finished, wherein B is the background with color information, and D is the background with depth information.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (6)
1. A background extraction method with depth information and color information is characterized by comprising the following steps:
step 1, initialization, specifically comprising the following steps:
step 1.1, decomposing the image matrix with depth information in the first frame into an image BGR matrix v1Sum-depth image floating point type single-channel matrix d1；
Step 1.2, use v1Initialize the ViBe algorithm, and d1Attaching the matrix element by element to a model single-channel floating-point type matrix D with the same size;
step 2: sequentially reading in the depth image of the t-th frame, and decomposing the depth image of the t-th frame into an image BGR matrix v according to the operation in the step 1.1tSum-depth image floating point single-channel type matrix dtAnd processing the two matrixes according to the following steps:
step 2.1: using vtUpdating the ViBe algorithm;
step 2.2: updating the matrix D;
step 2.3: judging whether the depth distance of the current pixel is closer than the depth distance of the background;
step 2.4: updating a ViBe sample space at the false detection position;
step 2.5: and (3) operating to obtain a color background BGR floating-point matrix B, and after the operation is finished, executing the step 2 again, wherein B is the background with color information, and D is the background with depth information.
2. A background extraction method with depth information and color information as claimed in claim 1, wherein in step 1.1, the image matrix v is1And depth image d1The values of (A) and (B) are width × height.
3. A background extraction method with depth information and color information as claimed in claim 1, characterized in that in step 2.2, the matrix D is updated according to the following formula:
wherein,
representing depth floating point values of a depth moment floating point mono-channel matrix D at (x, y) at the t-th frame of a depth image sequence, Dx,yRepresenting the depth floating point value of the depth background matrix at (x, y), wherein alpha is a threshold value and is more than or equal to 0 and less than or equal to 1; wherein, α represents the weight of the current depth frame, and (1- α) is the weight of the background depth model.
4. The method as claimed in claim 1, wherein in step 2.3, it is determined whether the current pixel depth distance is closer than the background depth distance according to the following formula:
RD＝(max-min)/T
wherein max and min are the maximum distance and the minimum distance measurable by the depth camera, respectively, T is the threshold value, when C isx,yIf the distance of the depth at (x, y) is greater than the background depth distance or closer to the background depth, and if the place is judged as the foreground by the ViBe, the background is judged as the foreground by the ViBe, which is a ghost image.
5. The method as claimed in claim 1, wherein in step 2.4, the ViBe sample space at the false detection point is updated according to the following formula:
wherein r is1,r2,r3,r4Are all random numbers, r1,r3∈[0,N)，r2,r4E [0, γ)), and omega (x, y) represents the coordinates of a random point in eight neighborhoods of coordinates (x, y), samples is the set of sample space matrices of ViBe,
represents the r-th sample value on (x, y), which is the color BGR value, N is the sample space size of each pixel of the ViBe algorithm, and γ is the sample update threshold of ViBe.
6. The method of claim 1, wherein in the step 2.5, the color background BGR floating point matrix B is obtained according to the following formula:
wherein, Bx，yAnd (3) the BGR value of the color background BGR floating-point matrix B at the position (x, y) is a three-dimensional vector, the summation mode at the position is the addition of corresponding components, and after the operation is finished, the step 2 is executed again, wherein in the process, B is the background with color information, and D is the background with depth information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010669614.4A CN111915687B (en) | 2020-07-13 | 2020-07-13 | Background extraction method with depth information and color information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010669614.4A CN111915687B (en) | 2020-07-13 | 2020-07-13 | Background extraction method with depth information and color information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111915687A true CN111915687A (en) | 2020-11-10 |
| CN111915687B CN111915687B (en) | 2024-07-05 |
Family
ID=73226294
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010669614.4A Active CN111915687B (en) | 2020-07-13 | 2020-07-13 | Background extraction method with depth information and color information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111915687B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104408747A (en) * | 2014-12-01 | 2015-03-11 | 杭州电子科技大学 | Human motion detection method suitable for depth image |
| CN106251348A (en) * | 2016-07-27 | 2016-12-21 | 广东外语外贸大学 | A kind of self adaptation multi thread towards depth camera merges background subtraction method |
| CN110503659A (en) * | 2019-07-09 | 2019-11-26 | 浙江浩腾电子科技股份有限公司 | A kind of Extracting of Moving Object towards video sequence |
| CN110599523A (en) * | 2019-09-10 | 2019-12-20 | 江南大学 | ViBe ghost suppression method fused with interframe difference method |
| CN111405140A (en) * | 2020-03-18 | 2020-07-10 | 浙江工业大学 | Video anti-shake method based on target tracking |
-
2020
- 2020-07-13 CN CN202010669614.4A patent/CN111915687B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104408747A (en) * | 2014-12-01 | 2015-03-11 | 杭州电子科技大学 | Human motion detection method suitable for depth image |
| CN106251348A (en) * | 2016-07-27 | 2016-12-21 | 广东外语外贸大学 | A kind of self adaptation multi thread towards depth camera merges background subtraction method |
| CN110503659A (en) * | 2019-07-09 | 2019-11-26 | 浙江浩腾电子科技股份有限公司 | A kind of Extracting of Moving Object towards video sequence |
| CN110599523A (en) * | 2019-09-10 | 2019-12-20 | 江南大学 | ViBe ghost suppression method fused with interframe difference method |
| CN111405140A (en) * | 2020-03-18 | 2020-07-10 | 浙江工业大学 | Video anti-shake method based on target tracking |
Non-Patent Citations (4)
| Title |
|---|
| OLIVIER BARNICH 等: "ViBe: A Universal Background Subtraction Algorithm for Video Sequences", 《IEEE TRANSACTIONS ON IMAGE PROCESSING》, vol. 20, no. 6, 23 December 2010 (2010-12-23), pages 1709 - 1724, XP011323125, DOI: 10.1109/TIP.2010.2101613 * |
| 乔继梅: "改进ViBe算法和SSD在运动车辆目标检测与跟踪中的应用", 《中国优秀硕士学位论文全文数据库 工程科技Ⅱ辑》, no. 8, 15 August 2019 (2019-08-15) * |
| 杨硕 等: "基于立体匹配的改进ViBe算法", 《计算机技术与发展》, vol. 28, no. 3, 15 November 2017 (2017-11-15), pages 122 - 126 * |
| 胡涛 等: "融合颜色和深度信息的运动目标提取方法", 《武汉大学学报(信息科学版)》, vol. 44, no. 2, 25 January 2019 (2019-01-25), pages 276 - 282 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111915687B (en) | 2024-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106504288B (en) | A kind of domestic environment Xiamen localization method based on binocular vision target detection | |
| CN109815863B (en) | Smoke and fire detection method and system based on deep learning and image recognition | |
| US10614736B2 (en) | Foreground and background detection method | |
| CN110889464B (en) | Neural network training method for detecting target object, and target object detection method and device | |
| US20190197345A1 (en) | Object detection device and object detection method | |
| CN112967339B (en) | Vehicle pose determining method, vehicle control method and device and vehicle | |
| CN112184759A (en) | Moving target detection and tracking method and system based on video | |
| JP7093427B2 (en) | Object tracking methods and equipment, electronic equipment and storage media | |
| EP3352138A1 (en) | Method and apparatus for processing a 3d scene | |
| CN112528974B (en) | Distance measuring method and device, electronic equipment and readable storage medium | |
| CN108447060B (en) | Foreground and background separation method based on RGB-D image and foreground and background separation device thereof | |
| Zhu et al. | Salient object detection with complex scene based on cognitive neuroscience | |
| Meiju et al. | Application of improved Otsu threshold segmentation algorithm in mobile phone screen defect detection | |
| CN111915687A (en) | Background extraction method with depth information and color information | |
| US20190251695A1 (en) | Foreground and background detection method | |
| CN115205793A (en) | Electric power machine room smoke detection method and device based on deep learning secondary confirmation | |
| Zhou et al. | On contrast combinations for visual saliency detection | |
| CN103778629A (en) | Background model real-time updating method for non-coherent radar image | |
| CN110288630B (en) | Moving target ghost suppression method for background modeling | |
| CN111476087A (en) | Target detection method and related model training method, device and apparatus | |
| CN111523478B (en) | Pedestrian image detection method acting on target detection system | |
| CN111476821B (en) | Target tracking method based on online learning | |
| CN110992642B (en) | Fire detection method based on deep learning | |
| Čejka et al. | Tackling problems of marker-based augmented reality under water | |
| CN113506272B (en) | False video detection method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |