Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 shows an exemplary system architecture 100 to which an embodiment of a method for outputting information or an apparatus for outputting information of the present application may be applied.
As shown in fig. 1, the system architecture 100 may include cameras 101, 102, 103, a network 104, and a server 105. The network 104 is a medium used to provide a communication link between the cameras 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 105 through the network 104 using the cameras 101, 102, 103 to receive or send messages or the like.
Cameras 101, 102, 103 are broadly referred to as license plate number identifiable cameras for vehicle monitoring. The system can be an electronic police which can take a snapshot of illegal (such as crossing a lane to press a solid line, driving in a reverse direction, occupying a non-motor vehicle lane, driving without a guide mark, running a red light, and the like) vehicles at an intersection. The camera can also be a bayonet camera which is arranged at some key road sections of expressways, provincial roads and national roads and is used for capturing overspeed driving illegal behaviors. The cameras 101, 102, 103 may also be illicit capture cameras, traffic monitoring cameras, world wide web monitoring cameras, flow capture cameras, etc.
The server 105 may be a server providing various services, such as a background analysis server providing analysis of vehicle data collected on the cameras 101, 102, 103. The background analysis server can analyze and other processes on the received vehicle data, and output the processing results (such as early warning information of vehicle scratch and theft) to the terminal or auxiliary parking equipment of the vehicle owner.
The server may be hardware or software. When the server is hardware, the server may be implemented as a distributed server cluster formed by a plurality of servers, or may be implemented as a single server. When the server is software, it may be implemented as a plurality of software or software modules (e.g., to provide distributed services), or as a single software or software module. The present invention is not particularly limited herein. The server may also be a server of a distributed system or a server that incorporates a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology.
It should be noted that, the method for outputting information provided by the embodiment of the present application is generally performed by the server 105, and accordingly, the device for outputting information is generally disposed in the server 105.
It should be understood that the number of cameras, networks and servers in fig. 1 is merely illustrative. There may be any number of cameras, networks, and servers, as desired for implementation.
With continued reference to FIG. 2, a flow 200 of one embodiment of a method for outputting information in accordance with the present application is shown. The method for outputting information comprises the following steps:
In step 201, a real-time video stream is acquired.
In this embodiment, an execution subject of the method for outputting information (e.g., a server shown in fig. 1) may acquire a real-time video stream from a camera through a wired connection or a wireless connection. The camera can aim at the parking stall, and the potential safety hazard in the key monitoring parking process. Alternatively, the camera may be aimed at the road. As shown in fig. 3a and 3b, the camera may collect a video stream and then send the video stream to the server in real time.
And 202, detecting key points of the target vehicle in running and the parking space vehicle parked in the parking space from the real-time video stream.
In this embodiment, targets such as pedestrians, vehicles, and the like can be detected from the video frame by a target detection model commonly used in the related art. The keypoints are then extracted from the targets by various methods known in the art. The keypoints are extracted from the target, for example, by a keypoint detection model. The keypoint detection model can be a variety of existing neural network structures (e.g., denseBox, VGGNet, resNet, segNet, etc.). As shown in fig. 3c, the vehicle key points may be the wheel axle, license plate, door spindle, door handle, etc. 22 key points are marked from a large number of vehicle images and used as training samples to train a vehicle key point detection model. And detecting the key points of the vehicle in the image through the trained vehicle key point detection model. As shown in fig. 3d, the key points of the human body may be a head, a neck, a left and right shoulder, a left and right elbow, a left and right wrist, a left and right hip, a left and right knee, a left and right ankle, etc. 13 key points can be marked from a large number of pedestrian images and used as training samples to train out a human body key point detection model. And detecting key points of the human body in the image through the trained human body key point detection model. The target vehicle here refers to a vehicle in running, and the parking space vehicle refers to a vehicle parked in a parking space. The parking space area can be marked in advance according to the image acquired by the camera, the position of the parking space in the video frame acquired subsequently is unchanged, and the vehicle detected on the parking space is the parking space vehicle.
As shown in fig. 3c, 22 key points are extracted from the vehicle image. The number of key points detected by one object is not fixed under the influence of the view angle. Since it is necessary to detect the case of collision of the targets, it is necessary to detect at least two targets, and it is not necessary to perform the subsequent steps if no target or only one target is detected.
And 203, tracking the target according to the key points of the target vehicle to obtain a tracking track.
In the present embodiment, it is not necessary to perform target detection and key point extraction for each frame image. The problem of time consumption is mainly considered, and because the time consumption of extracting key points is long, the key point extraction cannot be performed on each frame. The method and the device realize tracking and prediction of the target by adopting a combination mode of detecting key points and tracking key points. Based on the extracted key points, the key points are tracked, and a traditional characteristic point target tracking method is adopted. The location of the target may be determined based on the location of the key points of the target. The tracking trajectory may include time and a location of the target.
And 204, if the tracking track intersects with the area where the parking space vehicle is located, performing coordinate transformation according to the key points of the target vehicle and the key points of the parking space vehicle to obtain a first area and a second area of the top view under the physical coordinates.
In the present embodiment, the tracking trajectory at this time is a trajectory in image coordinates. If the tracking track of the target vehicle and the area where the parking space vehicle is located have no intersection on a two-dimensional plane, the situation that the vehicle has no intersection in a 3D space can be basically judged, and scratch cannot exist. If the tracking track of the target vehicle and the area where the parking space vehicle is located are intersected on a two-dimensional plane, the tracking track of the target vehicle and the area where the parking space vehicle is located are converted into a 3D space for further judgment. There may be virtually no contact from a top view because of the overlap caused by the object being occluded in the sideways taken photograph. The 2D image coordinates of the key points can be converted into physical coordinates of 3D by a related art coordinate conversion method. And then, projecting the image to the ground to obtain a top view under physical coordinates. The plan view of the target vehicle under the physical coordinates is named as a first area, and the plan view of the parking space vehicle under the physical coordinates is named as a second area.
In step 205, if the first area and the second area overlap, a prompt message is output.
In this embodiment, the first region and the second region are obtained in the top plan, and the overlapping area and the distance between the targets are obtained. It is convenient to calculate in the top plan whether there is an overlap of two objects and the distance between the object boundary lines. When the targets are not overlapped, but the distance between the targets does not reach the alarm threshold value, the system does not process; when the targets are not overlapped, but the distance between the targets reaches an alarm threshold, the system needs to start video recording and push relevant alarm information to a user and on-site guiding equipment; when the targets overlap, indicating that the targets have scratch, the system needs to push scratch information and video information to the parking user to remind the parking user to pay attention to the vehicle, and if the hit vehicle is registered in the system, the system also feeds back relevant information to the hit driver.
According to the method provided by the embodiment of the disclosure, through the 3D modeling technology, the position relation between the target and the target is judged in the physical coordinate system, the problem that the current parking system cannot perform collision early warning can be solved, meanwhile, the parking space is monitored and guided in the whole process, on one hand, the parking lot order can be timely and manually interfered, the traffic incident occurrence is reduced, on the other hand, related data can be pushed to a traffic management department, the generation of traffic disputes is reduced, and powerful data support is provided for intelligent traffic. The stability of parking system has been strengthened in the use of this scheme, has promoted the accuracy of system data.
In some optional implementations of the present embodiment, the method further includes: if the number of the parking space vehicles parked in the parking space is greater than 1, calculating the distance between the target vehicle and each parking space vehicle; determining a parking space vehicle closest to a target vehicle as the target parking space vehicle; if the distance between the target vehicle and the target parking space vehicle is smaller than a preset threshold value, starting video recording and outputting alarm information. And calculating the distance between the target vehicle and each parking space vehicle, and searching the minimum distance between the target vehicle and the parking space vehicle. When the position relation between the subsequent target vehicle and the parking space vehicle is judged, only the relation between the two targets with the minimum distance is judged, and the target vehicle and each parking space vehicle are not required to be compared one by one, so that the calculation complexity is reduced.
In some optional implementations of this embodiment, outputting the hint information includes: and starting video recording and pushing video recording information to the car owners of the parking space vehicles. The prompt information can guide the target vehicle to stop, and can also leave video evidence for the car owners of the parking space vehicles, so that the car owners of the parking space vehicles can conveniently follow the responsibilities. The accident scene can be quickly restored through video recording without checking complete monitoring video, so that the accident handling time is saved.
In some optional implementations of the present embodiment, the method further includes: identifying a license plate number of the target vehicle; inquiring the contact way of the culprit driver according to the license plate number; and pushing the related information according to the contact mode. The license plate number may be identified by OCR (Optical Character Recognition ) methods. The contact way of the driver can be obtained through the license plate number, and the prompt information is pushed to the driver. The video can also be pushed to the driver. The driver can be timely reminded, so that accidents are avoided.
With continued reference to fig. 3e, fig. 3e is a schematic diagram of an application scenario of the method for outputting information according to the present embodiment. In the application scenario of fig. 3e, the server detects, from the real-time video acquired by the camera, a vehicle in a predetermined range from the parking space during traveling and having a vehicle speed lower than a predetermined value as a target vehicle. And then determining a plurality of parking space vehicles according to the marked parking spaces. And determining the closest parking space vehicle as the target parking space vehicle according to the distance between the target vehicle and each parking space vehicle. Because the video shot by the monitoring point is not overlook usually, the detected key points of the vehicle need to be used for converting the vehicle from the image coordinate system to the physical coordinate system, so that the overhead view of the target vehicle and the overhead view of the target parking space vehicle are obtained. Whether scratch occurs is judged by whether top views of two vehicles overlap. In fig. 3e, scratch occurs, video is started, and scratch information and video information are sent to a parking user to remind the parking user to pay attention to his own vehicle. The relevant information may also be sent to the culprit driver.
With further reference to fig. 4, a flow 400 of yet another embodiment of a method for outputting information is shown. The flow 400 of the method for outputting information comprises the steps of:
in step 401, a real-time video stream is acquired.
Step 401 is substantially the same as step 201 and will not be described again.
Step 402, detecting a pedestrian from a real-time video stream.
In this embodiment, the key points of the pedestrian can be detected from the video frame by a pre-trained human key point detection model. As shown in fig. 3d, 13 key points are extracted from the pedestrian image.
Step 403, determining the distance between the pedestrian and the parking space vehicle.
In this embodiment, the distance between the foot key point and the parking space vehicle may be used as the distance between the pedestrian and the parking space vehicle.
Optionally, knowing the image coordinates of the key points (1-13) and the height information of the key points in the world coordinate system (the height information of the person is obtained through statistical information), the specific coordinates P (x i,yi,zi) of the key points in the world coordinate system can be obtained through the relation between the image coordinate system and the real coordinate system, the coordinates P (x i,yi) can be obtained by projecting the coordinates on a road plane, and the minimum circumcircle of P (x i,yi) is calculated, wherein the circular area is the projection area of the pedestrian on the road plane. And calculating the distance from the projection area to the parking space vehicle as the distance between the pedestrian and the parking space vehicle. Thereby improving the accuracy of the distance estimation.
Step 404, if the distance is smaller than the predetermined threshold, the video recording is started, and the related information is recorded.
In this embodiment, if the distance is far, no processing is performed; if the distance meets the set threshold, the stay time (obtained by a tracking mode) does not reach the set threshold, the pedestrian stays in the parking space area for a short time, and the system needs to automatically start video recording and record related information at the moment so that the vehicle owner can check the vehicle state.
And 405, pushing scratch and theft early warning information to the car owners of the parking space vehicles if the distance is smaller than a preset threshold value and the residence time of the pedestrians exceeds a preset time length.
In this embodiment, if the distance meets the set threshold, the residence time also reaches the set threshold, and during this time, scratch and theft early warning needs to be triggered, so as to push relevant information to the user, and remind the user of paying attention to the safety of the vehicle.
As can be seen from fig. 4, compared with the embodiment corresponding to fig. 2, the flow 400 of the method for outputting information in this embodiment represents the step of early warning of injury actions such as theft and driving of a pedestrian. Therefore, the scheme described in the embodiment not only can prevent the vehicle from being scratched, but also can detect the injury behavior of pedestrians to the vehicle, so that the suspected person can be rapidly and accurately positioned, and the loss can be recovered for the vehicle owner.
With further reference to fig. 5, as an implementation of the method shown in the foregoing figures, the present disclosure provides an embodiment of an apparatus for outputting information, which corresponds to the method embodiment shown in fig. 2, and which is particularly applicable to various electronic devices.
As shown in fig. 5, the apparatus 500 for outputting information of the present embodiment includes: an acquisition unit 501, a detection unit 502, a tracking unit 503, a conversion unit 504, and an output unit 505. Wherein the acquisition unit 501 is configured to acquire a real-time video stream. The detection unit 502 is configured to detect key points of a target vehicle in running and a parking space vehicle parked in a parking space from a real-time video stream. The tracking unit 503 is configured to perform target tracking according to the key points of the target vehicle to obtain a tracking track. And the conversion unit 504 is configured to perform coordinate conversion according to the key points of the target vehicle and the key points of the parking space vehicle if the tracking track intersects with the area where the parking space vehicle is located, so as to obtain a first area and a second area of the top view under the physical coordinates. The output unit 505 is configured to output the prompt information if there is overlap between the first area and the second area.
In the present embodiment, specific processes of the acquisition unit 501, the detection unit 502, the tracking unit 503, the conversion unit 504, and the output unit 505 of the apparatus 500 for outputting information can refer to steps 201, 202, 203, 204, and 205 in the corresponding embodiment of fig. 2.
In some optional implementations of the present embodiment, the apparatus 500 further includes an alarm unit (not shown in the drawings) configured to: if the number of the parking space vehicles parked in the parking space is greater than 1, calculating the distance between the target vehicle and each parking space vehicle. And determining the parking space vehicle closest to the target vehicle as the target parking space vehicle. If the distance between the target vehicle and the target parking space vehicle is smaller than a preset threshold value, starting video recording and outputting alarm information.
In some optional implementations of the present embodiment, the output unit 505 is further configured to: and starting video recording and pushing video recording information to the car owners of the parking space vehicles.
In some optional implementations of the present embodiment, the apparatus 500 further includes an identification unit (not shown in the drawings) configured to: the license plate number of the target vehicle is identified. And inquiring the contact way of the culprit driver according to the license plate number. And pushing the related information according to the contact mode.
In some optional implementations of the present embodiment, the apparatus 500 further includes a video recording unit (not shown in the drawings) configured to: the pedestrian is detected from the real-time video stream. And determining the distance between the pedestrian and the vehicle in the parking space. If the distance is smaller than the preset threshold value, starting video recording and recording related information.
In some optional implementations of this embodiment, the video recording unit is further configured to: if the distance is smaller than the preset threshold value and the residence time of the pedestrian exceeds the preset time, pushing scratch and theft early warning information to the car owners of the parking space vehicles.
In some optional implementations of this embodiment, the video recording unit is further configured to: and detecting body key points of pedestrians. And determining the projection area of the pedestrian on the road surface according to the body key points. And calculating the distance between the projection area and the parking space vehicle.
According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of flow 200 or 400.
A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of flow 200 or 400.
A computer program product comprising a computer program that when executed by a processor implements the method of flow 200 or 400.
Fig. 6 illustrates a schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 6, the apparatus 600 includes a computing unit 601 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the device 600 may also be stored. The computing unit 601, ROM 602, and RAM603 are connected to each other by a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Various components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, mouse, etc.; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the respective methods and processes described above, for example, a method for outputting information. For example, in some embodiments, the method for outputting information may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for outputting information described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the method for outputting information by any other suitable means (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.