CN112597892A - System and method for detecting remnants in automobile cabin - Google Patents

System and method for detecting remnants in automobile cabin Download PDF

Info

Publication number
CN112597892A
CN112597892A CN202011539422.8A CN202011539422A CN112597892A CN 112597892 A CN112597892 A CN 112597892A CN 202011539422 A CN202011539422 A CN 202011539422A CN 112597892 A CN112597892 A CN 112597892A
Authority
CN
China
Prior art keywords
image
objects
row
passengers
suspicious
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011539422.8A
Other languages
Chinese (zh)
Other versions
CN112597892B (en
Inventor
米建勋
高翔
钱基业
陈涛
向菲
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing University of Post and Telecommunications
Original Assignee
Chongqing University of Post and Telecommunications
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chongqing University of Post and Telecommunications filed Critical Chongqing University of Post and Telecommunications
Priority to CN202011539422.8A priority Critical patent/CN112597892B/en
Publication of CN112597892A publication Critical patent/CN112597892A/en
Application granted granted Critical
Publication of CN112597892B publication Critical patent/CN112597892B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/254—Analysis of motion involving subtraction of images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/758—Involving statistics of pixels or of feature values, e.g. histogram matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30268—Vehicle interior

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention relates to the technical field of intelligent detection, in particular to a detection system and a detection method for a remnant in an automobile cabin, wherein the method comprises the steps of setting a frame number threshold value according to the characteristic that a picture of a monitoring video in the automobile cabin is stable before passengers get on the automobile, and establishing a seat background picture; the pressure sensor under the seat records the number of passengers and the sitting positions and continuously monitors the change of pressure; when the pressure quantity is changed, carrying out the carry-over detection on the front row seats, and carrying out the carry-over detection on the vacant position areas of the rear row seats; after detecting and judging the abandoned object, identifying the type of the abandoned object according to an image classification algorithm proposed by us; after the identification is successful, the vehicle-mounted system sends a short message to inform a mobile phone of a passenger or inform a driver of reminding the passenger who just gets off to pick up the lost article; the invention utilizes the target detection method to track the remnant, replaces the idea of generally using the target tracking method, realizes a detection, namely tracking method, and overcomes the difficulty of large calculation amount of the target tracking algorithm.

Description

System and method for detecting remnants in automobile cabin
Technical Field
The invention relates to the technical field of intelligent detection, in particular to a system and a method for detecting a left object in an automobile cabin.
Background
The existing algorithms for detecting the remnants are generally divided into three categories, one is a background modeling method, namely, a picture without the remnants is established as a background, and a subsequent image is subtracted from the background, so that new objects left in the picture are detected, and the method is relatively dependent on the quality of the background picture; secondly, a target tracking method needs to consume a large amount of resources, which brings huge calculation burden, and the requirement is difficult to meet under the commonly applied scene, so the scheme is usually not adopted at a mobile terminal, and the tracking method is generally difficult to deal with the challenge brought by object deformation and shielding; the last type is a target detection method, which has the advantages of high speed, intuition, easy operation and the like, can clearly show the position and the type of the left-over object, has relatively mature technology, but is easily influenced by factors such as shielding and illumination change which cause the unclear picture, so that the recognition rate is reduced, and the false positive rate is increased. In daily research, people generally adopt a mode of combining any two or three types of methods to invent a more excellent carry-over detection algorithm.
In the field of video surveillance, researchers have conducted many intensive studies on carry-over detection in recent years in order to detect events of general public interest, such as identification of dangerous explosives and illegally parked vehicles, to reduce the possibility of accidents, and to ensure the life safety and property safety of the people. Unlike the current hot application, a wide public space, detection of carry-over in the scene in the vehicle has recently begun to receive attention from practitioners as a problem that has been ignored by the public in the past but has been left alone.
The patent with the patent application number of CN201911244307.5 uses a visual method as well as the invention, and proposes a method of using multiple cameras and multiple visual angles to track and locate the left objects in the car, the technology relates to the cooperative work of the multiple cameras, but the burden of calculation is increased; meanwhile, the patent does not relate to the serious influence of illumination on the effect, and the practical application scene is limited.
Disclosure of Invention
Aiming at the problems, the invention provides a detection system and a detection method for the left objects in an automobile cabin, wherein the system comprises front and rear seat pressure sensors, front and rear image acquisition equipment, a center console display, a memory, an embedded processor and a vehicle control unit; wherein:
the front-row and rear-row pressure sensors are arranged below each seat of the front-row and rear-row passenger seats and used for monitoring the pressure from each seat and judging the positions of passengers and the actions of getting on and off the train according to the change of the pressure;
the system comprises front and rear rows of image acquisition equipment, a front row of image acquisition equipment and a camera, wherein the front row of image acquisition equipment is arranged above an automobile center console and is aimed at a front passenger seat through a lens, and the camera which is arranged at the top of a cabin and is obliquely positioned above a rear seat and is aimed at a rear seat through the lens uses a near infrared image format; the front row image acquisition equipment acquires an image of a passenger seat; the rear-row image acquisition equipment acquires an image of a rear-row seat;
the memory is used for storing data generated by the front and rear rows of image acquisition equipment, the front and rear rows of pressure sensors and the embedded processor;
the embedded processor is a vehicle gauge-level deep learning processor, can read, write and calculate data in the memory, and temporarily stores a calculation result in the memory or sends the calculation result to the vehicle control unit;
the vehicle control unit is used for transferring and processing the data of the whole vehicle and controlling the operation of each part of the vehicle, and in the invention, the embedded processor is communicated with the center console in a display way, so that the center console is communicated with the embedded processor in a data way;
and the console display is used for outputting a carry-over detection result from the embedded processor and informing a driver of reminding the passengers getting off to take back the carry-over.
The invention also provides a method for detecting the left objects in the automobile cabin, which comprises the following steps:
s1: reading a video, setting a frame number threshold according to the characteristic that a picture of a monitoring video in an automobile cabin is stable before passengers get on the automobile, and establishing a seat background picture;
s2: after passengers get on the vehicle, the pressure sensors under the seats record the number and the sitting positions of the passengers and continuously monitor the pressure change;
s3: when the pressure reaches a preset threshold value, carrying out remnant detection on the front row seats, and carrying out remnant detection on the vacant position areas of the rear row seats;
s4: after detecting and judging the abandoned object, identifying the type of the abandoned object according to an image classification algorithm proposed by us;
s5: after the identification is successful, the vehicle-mounted system sends a short message to inform a mobile phone of a passenger or inform a driver of the short message to remind the passenger who just gets off to pick up the lost article.
Further, if the automobile is in a flameout or non-driving state, a clear background picture of the front row and the rear row without passengers is obtained by adopting median modeling, otherwise, a clear background picture of the front row and the rear row without passengers is obtained by adopting Gaussian modeling.
Further, during background modeling, a background picture of a rear row needs to be respectively established according to positions of three seats of the rear row of the vehicle.
Further, performing carryover detection comprises:
subtracting the current frame from the background to obtain a difference image, performing morphological operation to obtain a plurality of connected regions, and removing passengers according to the areas of the connected regions;
storing the remaining areas into a suspicious object queue, and continuously tracking the objects in the queue in the subsequent frame;
counting the number of frames of each object in the suspicious object queue and the number of lost frames by calculating the distance between the centroid coordinates, the matching degree between the gray level histograms and the matching degree between the Hu moments;
when the number of frames of a certain object exceeds the set number of frames, the object is cut off as a remnant;
when the number of the lost frames of a certain object exceeds the set number of frames, the lost frames are removed from the suspicious object queue;
if a new object is detected, it is added to the suspect queue and tracking thereof is started.
Further, the centroid coordinate distance of different objects is the sum of the corresponding abscissa difference and ordinate difference of the centroids of different objects, and the abscissa and ordinate of the centroid of the jth object can be expressed as:
Figure BDA0002854127240000031
the matching degree between the gray level histograms is expressed as:
Figure BDA0002854127240000032
the degree of match between the Hu moments is expressed as:
Figure BDA0002854127240000041
wherein N is the number of objects, cjIs the abscissa or ordinate of the centroid of the jth object, ciIs the abscissa or ordinate of the ith pixel of the jth object, and n is the number of pixels of the jth object; p and q are the two objects being compared, L (p, q) is the Hu moment match of the p and q objects,
Figure BDA0002854127240000042
the kth component of the Hu moment of the p object,
Figure BDA0002854127240000043
the kth component of the Hu moment of q objects, D (p, q) the gray level histogram matching degrees of p and q objects,
Figure BDA0002854127240000044
the number of pixels of s gray in the gray histogram of the p object,
Figure BDA0002854127240000045
the number of s-gray pixels in the gray histogram of the q object is shown, and m is the range of the gray histogram.
Further, the acquiring the binarized image includes:
Ji(x,y)＝|Fi(x,y)-B(x,y)|,i＝s,s+1……n；
Figure BDA0002854127240000046
wherein, B (x, y) is the gray value of the pixel point corresponding to the background frame, Fi(x, y) is the gray value of the pixel point corresponding to the current frame; j. the design is a squarei(x, y) is a differential image of the background frame and the current frame; t is a threshold value for performing binarization operation, Oi(x, y) is a binarized image, s is the number of frames used to create a background picture, and n is the total number of frames.
Further, continuously tracking the objects in the queue comprises:
reading a first frame image, obtaining an original suspicious object queue Q1, creating and allocating a structural body for each suspicious object in Q1, wherein the structural body members comprise: the method comprises the following steps of (1) bounding box coordinates, total visible frame number, continuous invisible frame number, gray histogram array and Hu moment array;
reading the next frame of image, obtaining a suspicious object queue Q2, and similarly creating and distributing a structural body for each suspicious object in Q2;
calculating centroid coordinates, a gray histogram and a Hu moment of the suspicious objects in Q1 and Q2, when the centroid distance, the gray histogram matching degree and the Hu moment matching degree of the suspicious object A1 in Q1 and the suspicious object B1 in Q2 are all smaller than threshold values, matching is successful, the structural body of A1 is updated to the structural body of B1, and the total visible frame number is added with 1;
when the centroid distances, the gray histogram matching degrees and the Hu moment matching degrees of all the suspicious objects A2 in the Q1 and all the suspicious objects in the Q2 are all larger than the threshold values, adding 1 to the continuous invisible frame number of the A2, and when the value reaches the maximum invisible frame number, the A2 is successfully matched, and the continuous invisible frame number returns to zero;
the doubtful objects which are not matched in the Q2 are all added into the Q1, and the Q2 is emptied;
when the total visible frame number of a suspicious object is more than 10, the suspicious object is judged to be left, and when the continuous invisible frame number of the suspicious object is more than 20, the suspicious object is discarded.
Further, identifying the carryover category includes:
collecting historical data with labels, inputting the historical data into a neural network, and outputting the data after dimension reduction and feature extraction processing;
in the neural network, an objective function is solved for each kind of label according to historical data to obtain a coding coefficient x for training, and the objective function is expressed as:
Figure BDA0002854127240000051
where A is training set data, b is test set data, λ is a trade-off parameter that balances the fidelity term and the regularization term, αiIs a trade-off parameter, xiIs the coding coefficient for each class of samples; x is the coding coefficient of all classes of samples; II-1To calculate the L1 norm, |)2In order to calculate the L2 norm,
Figure BDA0002854127240000053
to calculate the square of the L2 norm.
And classifying according to the image feature vector of the remnant and the reconstructed residual error and the dispersion degree between classes, and outputting the label of the remnant.
Further, the label of the legacy is represented as:
Figure BDA0002854127240000052
wherein label (y) represents the category information of the feature vector y of the image of the remnant; x is the coding coefficient of all classes of samples.
The invention has the following advantages and beneficial effects:
(1) the invention uses the near-infrared camera as the image acquisition equipment, compared with the visible light camera, the invention can be used in the scene without light at night and can hardly be influenced by illumination change in the scene in the daytime;
(2) according to the invention, when the background image is established for the rear seats and the objects left by the rear seats are judged, the partition thought is adopted, compared with the non-partition thought, the invention fully considers various situations of the taking of the rear passengers, and the judgment for the objects left by the rear seats is more precise;
(3) the invention utilizes the target detection method to track the remnant, replaces the idea of generally using the target tracking method, realizes a detection, namely tracking method, and overcomes the difficulty of large calculation amount of the target tracking algorithm.
Drawings
FIG. 1 is a schematic illustration of an operating environment in an embodiment of the present invention;
FIG. 2 is an overall method flow diagram of an embodiment of the present invention;
FIG. 3 is a flow chart of carryover detection and determination in an embodiment of the present invention;
FIG. 4 is a flow chart of a carryover identification in an embodiment of the present invention;
FIG. 5 is a diagram of a legacy detection hardware system architecture according to an embodiment of the present invention;
FIG. 6 is a schematic diagram illustrating the identification result of a mobile phone left behind on a rear seat using a near-infrared image in a dark environment according to an embodiment of the present invention;
FIG. 7 is a schematic diagram illustrating the identification result of a mobile phone left on a rear seat by using a visible light image in a dark environment according to an embodiment of the present invention;
510, a high-speed bus; 520. a vehicle control unit (ECU); 530. an embedded processor; 540. a memory; 550. front and rear rows of pressure sensors; 560. front row image acquisition equipment; 570. a back row image acquisition device; 580. and displaying on the console.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
The embodiment provides a system for detecting a remnant in an automobile cabin, which comprises front and rear seat pressure sensors, front and rear image acquisition equipment, a center console display, a memory, an embedded processor and a vehicle control unit, as shown in fig. 1; wherein:
the front-row and rear-row pressure sensors are arranged below each seat of the front-row and rear-row passenger seats and used for monitoring the pressure from each seat and judging the positions of passengers and the actions of getting on and off the train according to the change of the pressure;
the system comprises front and rear rows of image acquisition equipment, a front row of image acquisition equipment and a camera, wherein the front row of image acquisition equipment is arranged above an automobile center console and is aimed at a front passenger seat through a lens, and the camera which is arranged at the top of a cabin and is obliquely positioned above a rear seat and is aimed at a rear seat through the lens uses a near infrared image format; the front row image acquisition equipment acquires an image of a passenger seat; the rear-row image acquisition equipment acquires an image of a rear-row seat;
the memory is used for storing data generated by the front and rear rows of image acquisition equipment, the front and rear rows of pressure sensors and the embedded processor;
the embedded processor is a vehicle gauge-level deep learning processor, can read, write and calculate data in the memory, and temporarily stores a calculation result in the memory or sends the calculation result to the vehicle control unit;
the vehicle control unit is used for transferring and processing the data of the whole vehicle and controlling the operation of each part of the vehicle, and in the invention, the embedded processor is electrically communicated with the display of the central console, so that the central console is communicated with the embedded processor in a data way;
and the console display is used for outputting a carry-over detection result from the embedded processor and informing a driver of reminding the passengers getting off to take back the carry-over.
In a specific implementation, as shown in fig. 5, the system includes a high-speed bus 510, a vehicle control unit (ECU)520, an embedded processor 530, a memory 540, a front-back pressure sensor 550, a front-back image capturing device 560, a back-back image capturing device 570, and a console display 580, where:
the high-speed bus 510 is used for data transmission of the whole system, including transmission of acquired image data, data of the pressure sensor and data output to the console for display;
the front and rear row pressure sensors are used for monitoring the pressure change of the seats and transmitting the numerical value generated at the moment to the memory 540 through the high-speed bus 510, the embedded processor 530 reads the pressure data at each moment in the memory 540 and judges whether a passenger gets off the vehicle or not, and when the passenger gets off the vehicle, the embedded processor 530 enables the memory 540 to stop receiving the data of the front and rear row pressure sensors 550;
the front row image acquisition equipment 560 and the rear row image acquisition equipment 570 are used for acquiring near-infrared images of front and rear seats and transmitting the near-infrared images to the memory 540 through the high-speed bus 510, the embedded processor 530 reads video frames in the memory 540, the legacy detection algorithm is operated, the calculated result is transmitted to the whole vehicle controller (ECU)520, the whole vehicle controller (ECU)520 outputs the video frames to the center console through the high-speed bus 510 to be displayed 580, after receiving information, a driver can select a closing message to remind, a closing signal is reversely transmitted to the embedded processor 530, and the algorithm operation is forcibly stopped; or when the passenger takes the article away, the algorithm can not detect the left article, and the algorithm automatically stops the operation of the algorithm;
the high-speed bus 510 includes: any one or combination of a plurality of automobile buses meeting high transmission rate, such as a CAN bus, a FlexRay bus or an MOST/1394 bus;
the memory 540 includes: any one or more of various media that can be stored, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
Example 2
In this embodiment, the present invention provides a method for detecting a remnant in an automobile cabin, as shown in fig. 2, specifically including the following steps:
s1: reading a video, setting a frame number threshold according to the characteristic that a picture of a monitoring video in an automobile cabin is stable before passengers get on the automobile, and establishing a seat background picture;
s2: after passengers get on the vehicle, the pressure sensors under the seats record the number and the sitting positions of the passengers and continuously monitor the pressure change;
s3: when the pressure reaches a preset threshold value, carrying out remnant detection on the front row seats, and carrying out remnant detection on the vacant position areas of the rear row seats;
s4: after detecting and judging the abandoned object, identifying the type of the abandoned object according to an image classification algorithm proposed by us;
s5: after the identification is successful, the vehicle-mounted system sends a short message to inform a mobile phone of a passenger or inform a driver of the short message to remind the passenger who just gets off to pick up the lost article.
A plurality of image acquisition devices, pressure sensors and an on-board computer for processing data and running a carry-over detection algorithm are arranged in the automobile cabin. The pressure sensor is arranged below each seat and used for monitoring the number and the riding position of passengers getting on the vehicle, the plurality of image acquisition devices comprise a front row of image acquisition devices and a rear row of image acquisition devices and are used for acquiring images of the front row of seats and the rear row of seats, the vehicle-mounted computer supports the operation of the whole system, stores and processes data of the pressure sensor and the image acquisition devices, operates a carry-over detection algorithm and outputs a result to the whole vehicle controller. Specifically, considering that the rear seats have different riding scenes with different numbers of people and different getting-off conditions, the algorithm is used for carrying out the leave-behind algorithm detection on vacant seat areas, and the invention summarizes the following conditions:
(1) two or three passengers take a bus, one person near the bus gets off the bus, and the positions of the rest passengers are not changed;
(2) two or three passengers take a bus, one person near the bus gets off the bus, and the rest passengers change positions;
(3) two or three people take a bus, and more than one person gets off the bus;
(4) when three passengers get on the bus, the passengers at the sides and the passengers at the middle get off the bus, and the passengers at the sides get on the bus again.
In view of this, the rear-row image capturing device needs to capture the first 50 frames of images in advance and perform background modeling, and divide the obtained background images into three regions according to the seat positions to be stored respectively. Meanwhile, a pressure sensor is arranged under each seat, the sensor sends a signal to an on-board computer after monitoring the pressure change, and the on-board computer judges the number change and the position change before and after getting off the vehicle and determines a detection area of the left-over object; and then, the detection algorithm of the left object in the vehicle-mounted computer starts to operate, the background image of the area and the current image of the area are used as input, the algorithm determines the position and the type information of the left object and outputs the position and the type information to the vehicle control unit, and the vehicle control unit sends the information to the central console for displaying, so that a driver is told to remind the passengers getting off the vehicle to pick up the left object.
When the pressure quantity changes, carry out the thing detection of leaving behind to the front seat beginning, carry out the thing detection of leaving behind to the vacant position region of back seat, as fig. 3, specifically include the following step:
step S210: subtracting the current frame from the background to obtain a difference image, performing morphological operation to obtain a plurality of connected regions, and removing passengers according to the areas of the connected regions;
the specific implementation of this step is as follows: when the vehicle-mounted computer determines the vacancy, cutting out an image of the current frame in a vacancy area, performing frame difference with a background image of the vacancy area to obtain a difference image, and binarizing the difference image, wherein the difference image is represented as:
Ji(x,y)＝|Fi(x,y)-B(x,y)|,i＝s,s+1……n；
Figure BDA0002854127240000101
wherein, B (x, y) is the gray value of the pixel point corresponding to the background frame, Fi(x, y) is the gray value of the pixel point corresponding to the current frame; j. the design is a squarei(x, y) is a differential image of the background frame and the current frame; t is a threshold value for performing binarization operation, Oi(x, y) is a binarized image, s is the number of frames used to create a background picture, and n is the total number of frames.
And then performing morphological operation and connected domain analysis on the difference image to obtain a plurality of connected regions, limiting the area of each connected region to be more than 450 and less than 4000, removing regions caused by noise and passengers, and obtaining a final remnant image.
Step S220: storing the remaining areas into a suspicious object queue, and continuously tracking the objects in the queue in the subsequent frame;
step S230: counting the number of frames of each object in the suspicious object queue and the number of lost frames by calculating the centroid coordinate distance, the gray level histogram matching degree and the Hu moment matching degree of different objects; the distance between the centroid coordinates of different objects is the sum of the corresponding horizontal coordinate difference value and vertical coordinate difference value of the centroids of different objects, and the horizontal coordinate and vertical coordinate of the centroid of the jth object can be expressed as:
Figure BDA0002854127240000102
the matching degree between the gray level histograms is expressed as:
Figure BDA0002854127240000103
the degree of match between the Hu moments is expressed as:
Figure BDA0002854127240000104
wherein N is the number of objects, cjIs the abscissa or ordinate of the centroid of the jth object, ciIs the abscissa or ordinate of the ith pixel of the jth object, and n is the number of pixels of the jth object; p and q are the two objects being compared, L (p, q) is the Hu moment match of the p and q objects,
Figure BDA0002854127240000105
the kth component of the Hu moment of the p object,
Figure BDA0002854127240000106
the kth component of the Hu moment of q objects, D (p, q) the gray level histogram matching degrees of p and q objects,
Figure BDA0002854127240000107
the number of pixels of s gray in the gray histogram of the p object,
Figure BDA0002854127240000108
the number of s-gray pixels in the gray histogram of the q object is shown, and m is the range of the gray histogram.
Step S240: when the number of the existing frames of a certain object exceeds a certain number, the object is judged to be a remnant. And when the number of the lost frames of a certain object exceeds a certain number, removing the lost frames from the suspicious object queue. If a new object is detected, it is added to the suspect queue and tracking thereof is started.
As an alternative implementation, the specific implementation steps of suspicious object tracking are as follows:
(1) reading a first frame image, obtaining an original suspicious object queue Q1, creating and allocating a structural body for each suspicious object in Q1, wherein the structural body members comprise: the method comprises the following steps of (1) bounding box coordinates, total visible frame number, continuous invisible frame number, gray histogram array and Hu moment array;
(2) reading the next frame of image, obtaining a suspicious object queue Q2, and similarly creating and distributing a structural body for each suspicious object in Q2;
(3) calculating centroid coordinates, a gray histogram and a Hu moment of the suspicious objects in Q1 and Q2, when the centroid distance, the gray histogram matching degree and the Hu moment matching degree of the suspicious object A1 in Q1 and the suspicious object B1 in Q2 are all smaller than threshold values, matching is successful, the structural body of A1 is updated to the structural body of B1, and the total visible frame number is added with 1;
(4) when the centroid distances, the gray histogram matching degrees and the Hu moment matching degrees of all the suspicious objects A2 in the Q1 and all the suspicious objects in the Q2 are all larger than the threshold values, adding 1 to the continuous invisible frame number of the A2, and when the value reaches the maximum invisible frame number, the A2 is successfully matched, and the continuous invisible frame number returns to zero;
(5) the doubtful objects which are not matched in the Q2 are all added into the Q1, and the Q2 is emptied;
(6) when the total number of visible frames of a suspicious object is more than 10, the suspicious object is judged to be left. When the number of continuous invisible frames of a suspicious object is more than 20, discarding the suspicious object;
(7) if the remnants are not taken away and the driver does not stop the detection of the remnants, returning to the step (2); otherwise, ending the detection.
The above-mentioned method is a process of counting the number of frames existing in each object in the suspicious queue and the number of frames disappeared by calculating the distance between centroid coordinates, the matching degree between gray level histograms and the matching degree between Hu moments.
In this example, the preset neural network may be any class classification network trained by using ImageNet data set, taking a pre-trained ResNet-50 neural network as an example, graying the obtained image, scaling the grayed image to 224 because the input image size needs 224 × 3, copying two same gray maps, generating an image format of 224 × 3, processing the image format by using ResNet-50, taking a1 × 1000 dimensional vector output by the last layer of the neural network as a result of dimension reduction and feature extraction, and finally saving a plurality of 1 × 1000 dimensional vectors obtained after processing each remnant image into an matable file.
The output of the neural network is used as the input of the image classification method proposed by the invention which is trained in advance, in this example, the classes of the belongings of the passengers are predefined to be five classes, namely, wallet, mobile phone, bag and umbrella, it should be clear that the classes of the belongings are not limited to the five classes, and can be arbitrarily added or deleted according to the actual use condition, as shown in fig. 4, and the method specifically includes the following steps:
(1) acquiring the images of the types of the objects left over on the Internet, wherein the quantity requirement of each type of image is the same, and the same operation is adopted for all the images, and the slightly different is that when an after-mat file is saved, a column of label information is added to the last column of a vector matrix;
(2) dividing the data in the step (1) into a training set and a test set, and training the image classification method provided by the invention by solving an objective function for each class:
Figure BDA0002854127240000121
obtaining a coding coefficient x, where xiIs the coding coefficient of each sample class, A is the training set data, b is the test set data, lambda is the trade-off parameter of the balance fidelity term and the regularization term, alphaiAre trade-off parameters that balance the constraints imposed on each class. The example splits the original objective function, and obtains the optimal solution of the original problem by alternately updating each parameter until the convergence condition is satisfied. Specifically, the method comprises the following steps:
the augmented lagrange form of its scaled version is:
Figure BDA0002854127240000122
st.z-x＝0
where z is an independent variable introduced in order to partition the original non-convex problem (objective function) into x and xiThe two convex problems of (2) are convenient to solve. n is the number of classes contained in the data set, p>0 is a penalty parameter, which is,
Figure BDA0002854127240000123
in order to obtain a scaled versionThe augmented Lagrangian function, y is called the Lagrangian multiplier, | - |1Is L1 norm, |)2Is a norm of L2 and,
Figure BDA0002854127240000124
is the square of the norm of L2.
The optimized iterative process is as follows:
Figure BDA0002854127240000131
Figure BDA0002854127240000132
uk+1:＝uk+xk+1-zk+1；
further:
xk+1:＝(A′A+ρI)-1[A′b+ρ(zk-uk)]；
order to
Figure BDA0002854127240000133
Then:
Figure BDA0002854127240000134
wherein A' is a transposed matrix of the training sample A, and I is a unit matrix with main diagonals all being 0; the stopping conditions for the above objective function optimization are two: 1. the maximum iteration times are reached; 2. the original problem residual and the dual residual are sufficiently small, and their formulas are:
‖pk‖2≤∈priand‖dk‖2≤∈dual；
wherein p iskX-z is the original problem residual, dk＝-ρ(zk+1-zk) For dual problem residuals, epri>0 and edual>0 is the tolerance of the original problem and the dual problem.
(3) After training is finished, classifying the legacy images by using the obtained coefficient x, dividing the legacy image feature vectors into classes with minimum reconstruction residuals, wherein the inter-class dispersion degree is also a reference factor for decision making, and the formula is as follows:
Figure BDA0002854127240000135
wherein y is the feature vector of the legacy image, label (y) is the class information thereof, and x is the coding coefficient of all class samples.
And classifying according to the image feature vector of the remnant and the reconstructed residual error and the dispersion degree between classes, and outputting the label of the remnant.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (10)

1. A detection system for the remnant in the automobile cabin is characterized by comprising front and back seat pressure sensors, front and back image acquisition equipment, a center console display, a memory, an embedded processor and a vehicle control unit; wherein:
the front-row and rear-row pressure sensors are arranged below each seat of the front-row and rear-row passenger seats and used for monitoring the pressure from each seat and judging the positions of passengers and the actions of getting on and off the train according to the change of the pressure;
the system comprises front and rear rows of image acquisition equipment, a front row of image acquisition equipment and a camera, wherein the front row of image acquisition equipment is arranged above an automobile center console and is aimed at a front passenger seat through a lens, and the camera which is arranged at the top of a cabin and is obliquely positioned above a rear seat and is aimed at a rear seat through the lens uses a near infrared image format; the front row image acquisition equipment acquires an image of a passenger seat; the rear-row image acquisition equipment acquires an image of a rear-row seat;
the memory is used for storing data generated by the front and rear rows of image acquisition equipment, the front and rear rows of pressure sensors and the embedded processor;
the embedded processor is a vehicle gauge-level deep learning processor, can read, write and calculate data in the memory, and temporarily stores a calculation result in the memory or sends the calculation result to the vehicle control unit;
the vehicle control unit is used for transferring and processing the data of the whole vehicle and controlling the operation of each part of the vehicle, and in the invention, the embedded processor is communicated with the center console in a display way, so that the center console is communicated with the embedded processor in a data way;
and the console display is used for outputting a carry-over detection result from the embedded processor and informing a driver of reminding the passengers getting off to take back the carry-over.
2. A method for detecting the left object in the automobile cabin is characterized by comprising a system for detecting the left object in the automobile cabin, and specifically comprising the following steps:
s1: reading a video, setting a frame number threshold according to the characteristic that a picture of a monitoring video in an automobile cabin is stable before passengers get on the automobile, and establishing a seat background picture;
s2: after passengers get on the vehicle, the pressure sensors under the seats record the number and the sitting positions of the passengers and continuously monitor the pressure change;
s3: when the pressure reaches a preset threshold value, carrying out remnant detection on the front row seats, and carrying out remnant detection on the vacant position areas of the rear row seats;
s4: after detecting and judging the abandoned object, identifying the type of the abandoned object according to an image classification algorithm proposed by us;
s5: after the identification is successful, the vehicle-mounted system sends a short message to inform a mobile phone of a passenger or inform a driver of the short message to remind the passenger who just gets off to pick up the lost article.
3. The method according to claim 2, wherein during the background modeling, if the vehicle is in a flameout or non-driving state, the median modeling is used to obtain a clean background picture of front and back rows without passengers, otherwise, the Gaussian modeling is used to obtain a clean background picture of front and back rows without passengers.
4. The method according to claim 2 or 3, wherein in the background modeling, the background pictures are respectively established according to the positions of three seats in the back row of the vehicle.
5. The method of claim 2, wherein performing the carryover detection comprises:
subtracting the current frame from the background to obtain a difference image, performing morphological operation to obtain a plurality of connected regions, and removing passengers according to the areas of the connected regions to obtain a binary image of the remnant;
storing the remaining areas into a suspicious object queue, and continuously tracking objects in the queue in a subsequent frame;
counting the number of frames of each object in the suspicious object queue and the number of lost frames by calculating the distance between the centroid coordinates, the matching degree between the gray level histograms and the matching degree between the Hu moments;
when the number of frames of a certain object exceeds the set number of frames, judging the object as a remnant;
when the number of continuous disappearing frames of a certain object exceeds the set number of frames, removing the object from the suspicious object queue;
if a new object is detected, it is added to the suspect queue and tracking thereof is started.
6. The method of claim 5, wherein the centroid coordinate distance of different objects is the sum of the corresponding abscissa and ordinate differences of the centroids of different objects, and the abscissa and ordinate of the centroid of the jth object can be expressed as:
Figure FDA0002854127230000031
the matching degree between the gray level histograms is expressed as:
Figure FDA0002854127230000032
the degree of match between the Hu moments is expressed as:
Figure FDA0002854127230000033
wherein N is the number of objects, cjIs the abscissa or ordinate of the centroid of the jth object, ciIs the abscissa or ordinate of the ith pixel of the jth object, and n is the number of pixels of the jth object; p and q are the two objects being compared, L (p, q) is the Hu moment match of the p and q objects,
Figure FDA0002854127230000034
the kth component of the Hu moment of the p object,
Figure FDA0002854127230000035
the kth component of the Hu moment of q objects, D (p, q) the gray level histogram matching degrees of p and q objects,
Figure FDA0002854127230000036
the number of pixels of s gray in the gray histogram of the p object,
Figure FDA0002854127230000037
the number of s-gray pixels in the gray histogram of the q object is shown, and m is the range of the gray histogram.
7. The method according to claim 5, wherein the step of obtaining the binarized image of the carry-over comprises:
Ji(x，y)＝|Fi(x，y)-B(x，y)|，i＝s，s+1......n；
Figure FDA0002854127230000038
wherein, B (x, y) is the gray value of the pixel point corresponding to the background frame, Fi(x, y) is the gray value of the pixel point corresponding to the current frame; j. the design is a squarei(x, y) is a differential image of the background frame and the current frame; t is a threshold value for performing binarization operation, Oi(x, y) is a binarized image, s is the number of frames used to create a background picture, and n is the total number of frames.
8. The method of claim 5, wherein the continuously tracking objects in the queue comprises:
reading a first frame image, obtaining an original suspicious object queue Q1, creating and allocating a structural body for each suspicious object in Q1, wherein the structural body members comprise: the method comprises the following steps of (1) bounding box coordinates, total visible frame number, continuous invisible frame number, gray histogram array and Hu moment array;
reading the next frame of image, obtaining a suspicious object queue Q2, and similarly creating and distributing a structural body for each suspicious object in Q2;
calculating centroid coordinates, a gray histogram and a Hu moment of the suspicious objects in Q1 and Q2, when the centroid distance, the gray histogram matching degree and the Hu moment matching degree of the suspicious object A1 in Q1 and the suspicious object B1 in Q2 are all smaller than threshold values, matching is successful, the structural body of A1 is updated to the structural body of B1, and the total visible frame number is added with 1;
when the centroid distances, the gray histogram matching degrees and the Hu moment matching degrees of all the suspicious objects A2 in the Q1 and all the suspicious objects in the Q2 are all larger than the threshold values, adding 1 to the continuous invisible frame number of the A2, and when the value reaches the maximum invisible frame number, the A2 is successfully matched, and the continuous invisible frame number returns to zero;
the doubtful objects which are not matched in the Q2 are all added into the Q1, and the Q2 is emptied;
when the total visible frame number of a suspicious object is more than 10, the suspicious object is judged to be left, and when the continuous invisible frame number of the suspicious object is more than 20, the suspicious object is discarded.
9. The method of claim 2, wherein identifying the carry-over category comprises:
collecting historical data with labels, inputting the historical data into a neural network, and outputting the data after dimension reduction and feature extraction processing;
solving an objective function for each kind of label according to historical data to obtain a coding coefficient x for training, wherein the objective function is expressed as:
Figure FDA0002854127230000041
where A is training set data, b is test set data, λ is a trade-off parameter that balances the fidelity term and the regularization term, αiIs a trade-off parameter; x is the number ofiIs the coding coefficient for each class of samples; x is the coding coefficient of all classes of samples; i | · | purple wind1To calculate the L1 norm, | ·| non-woven2In order to calculate the L2 norm,
Figure FDA0002854127230000042
to calculate the square of the L2 norm;
and classifying according to the image feature vector of the remnant and the reconstructed residual error and the dispersion degree between classes, and outputting the label of the remnant.
10. The method of claim 9, wherein the tag of the carry-over is expressed as:
Figure FDA0002854127230000043
wherein label (y) represents the category information of the feature vector y of the image of the remnant; x is the coding coefficient of all classes of samples.
CN202011539422.8A 2020-12-23 2020-12-23 System and method for detecting remnants in automobile cabin Active CN112597892B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011539422.8A CN112597892B (en) 2020-12-23 2020-12-23 System and method for detecting remnants in automobile cabin

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011539422.8A CN112597892B (en) 2020-12-23 2020-12-23 System and method for detecting remnants in automobile cabin

Publications (2)

Publication Number Publication Date
CN112597892A true CN112597892A (en) 2021-04-02
CN112597892B CN112597892B (en) 2022-09-23

Family

ID=75200443

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011539422.8A Active CN112597892B (en) 2020-12-23 2020-12-23 System and method for detecting remnants in automobile cabin

Country Status (1)

Country Link
CN (1) CN112597892B (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113456039A (en) * 2021-07-30 2021-10-01 芯海科技（深圳）股份有限公司 Vehicle-mounted vital sign early warning method and device, vehicle-mounted electronic equipment and medium
CN113642454A (en) * 2021-08-11 2021-11-12 汇纳科技股份有限公司 Seat use condition identification method, system, equipment and computer storage medium
CN113705543A (en) * 2021-10-27 2021-11-26 珠海市魅族科技有限公司 In-vehicle article anti-forgetting monitoring method, device, equipment and medium
CN113997898A (en) * 2021-11-30 2022-02-01 浙江极氪智能科技有限公司 Living body detection method, apparatus, device and storage medium
CN116596958A (en) * 2023-07-18 2023-08-15 四川迪晟新达类脑智能技术有限公司 Target tracking method and device based on online sample augmentation
CN117036482A (en) * 2023-08-22 2023-11-10 北京智芯微电子科技有限公司 Target object positioning method, device, shooting equipment, chip, equipment and medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102314695A (en) * 2011-08-23 2012-01-11 北京黄金视讯科技有限公司 Abandoned object detection method based on computer vision
EP3196863A1 (en) * 2014-08-01 2017-07-26 Shenzhen Cimc-tianda Airport Support Ltd. System and method for aircraft docking guidance and aircraft type identification
CN107314816A (en) * 2017-06-08 2017-11-03 贵阳学院 The multi-level information monitoring method for early warning of crops early disease
CN107316059A (en) * 2017-06-16 2017-11-03 陕西师范大学 Learner's gesture recognition method
CN107585096A (en) * 2016-07-08 2018-01-16 奥迪股份公司 Anti- forgetting system for prompting, method and vehicle

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102314695A (en) * 2011-08-23 2012-01-11 北京黄金视讯科技有限公司 Abandoned object detection method based on computer vision
EP3196863A1 (en) * 2014-08-01 2017-07-26 Shenzhen Cimc-tianda Airport Support Ltd. System and method for aircraft docking guidance and aircraft type identification
CN107585096A (en) * 2016-07-08 2018-01-16 奥迪股份公司 Anti- forgetting system for prompting, method and vehicle
CN107314816A (en) * 2017-06-08 2017-11-03 贵阳学院 The multi-level information monitoring method for early warning of crops early disease
CN107316059A (en) * 2017-06-16 2017-11-03 陕西师范大学 Learner's gesture recognition method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
JIANXUN MI等: "Image compression using principal component neural network", 《ICARCV 2004 8TH CONTROL, AUTOMATION, ROBOTICS AND VISION CONFERENCE, 2004》 *
ZOU HUI等: "Regularization and variable selection via the elastic net", 《J OURNAL OF THE ROYAL STATISTICAL SOCIETY》 *
隆武林等: "基于图像分割与特征匹配的目标检测及跟踪", 《信息技术》 *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113456039A (en) * 2021-07-30 2021-10-01 芯海科技（深圳）股份有限公司 Vehicle-mounted vital sign early warning method and device, vehicle-mounted electronic equipment and medium
CN113642454A (en) * 2021-08-11 2021-11-12 汇纳科技股份有限公司 Seat use condition identification method, system, equipment and computer storage medium
CN113642454B (en) * 2021-08-11 2024-03-01 汇纳科技股份有限公司 Seat use condition identification method, system, equipment and computer storage medium
CN113705543A (en) * 2021-10-27 2021-11-26 珠海市魅族科技有限公司 In-vehicle article anti-forgetting monitoring method, device, equipment and medium
CN113997898A (en) * 2021-11-30 2022-02-01 浙江极氪智能科技有限公司 Living body detection method, apparatus, device and storage medium
CN116596958A (en) * 2023-07-18 2023-08-15 四川迪晟新达类脑智能技术有限公司 Target tracking method and device based on online sample augmentation
CN116596958B (en) * 2023-07-18 2023-10-10 四川迪晟新达类脑智能技术有限公司 Target tracking method and device based on online sample augmentation
CN117036482A (en) * 2023-08-22 2023-11-10 北京智芯微电子科技有限公司 Target object positioning method, device, shooting equipment, chip, equipment and medium

Also Published As

Publication number Publication date
CN112597892B (en) 2022-09-23

Similar Documents

Publication Publication Date Title
CN112597892B (en) System and method for detecting remnants in automobile cabin
Silva et al. Automatic detection of motorcyclists without helmet
US20030169906A1 (en) Method and apparatus for recognizing objects
AU2007291884B2 (en) Method and system of identifying one or more features represented in a plurality of sensor acquired data sets
US20140376769A1 (en) Method for detecting large size and passenger vehicles from fixed cameras
US7650034B2 (en) Method of locating a human eye in a video image
CN111368612B (en) Overguard detection system, personnel detection method and electronic equipment
JP2019106193A (en) Information processing device, information processing program and information processing method
CN106022458B (en) Towards the number faster statistical approach of safety of school bus
CN112287861A (en) Road information enhancement and driving early warning method based on night environment perception
CN111950499A (en) Method for detecting vehicle-mounted personnel statistical information
CN115346197A (en) Driver distraction behavior identification method based on bidirectional video stream
JP6472504B1 (en) Information processing apparatus, information processing program, and information processing method
CN113408364B (en) Temporary license plate recognition method, system, device and storage medium
Mathew et al. Detecting new stable objects in surveillance video
CN116012822B (en) Fatigue driving identification method and device and electronic equipment
Xia et al. An automobile detection algorithm development for automated emergency braking system
CN113053127B (en) Intelligent real-time state detection system and method
CN114120250A (en) Video-based method for detecting illegal people carried by motor vehicle
CN111881843A (en) Taxi passenger carrying number counting method based on face detection
Bao et al. Water region and multiple ship detection for port surveillance
CN116152790B (en) Safety belt detection method and device
EP3937066A1 (en) Peer vehicle center line determination system and method
Mohamed Asjath et al. Automated vehicle parking slot recognition and monitoring using Optimised image processing techniques
Asjath et al. Automated Vehicle Parking Slot Recognition and Monitoring using Optimised Image Processing Techniques

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant