CN114022846A - Anti-collision monitoring method, device, equipment and medium for working vehicle - Google Patents
Anti-collision monitoring method, device, equipment and medium for working vehicle Download PDFInfo
- Publication number
- CN114022846A CN114022846A CN202111325161.4A CN202111325161A CN114022846A CN 114022846 A CN114022846 A CN 114022846A CN 202111325161 A CN202111325161 A CN 202111325161A CN 114022846 A CN114022846 A CN 114022846A
- Authority
- CN
- China
- Prior art keywords
- obstacle
- distance
- target obstacle
- vehicle
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an anti-collision monitoring method of an operation vehicle, which is applied to an anti-collision monitoring system, wherein the anti-collision monitoring system comprises a vehicle-mounted auxiliary terminal and data acquisition terminals which are arranged in each preset direction at the top end of an operation device of the operation vehicle, and the method comprises the following steps: the method comprises the steps of identifying a target obstacle in an acquired depth image, and constructing a three-dimensional scene model comprising a working vehicle and the target obstacle according to the depth image so as to realize safety protection monitoring suitable for different scenes. And then in the three-dimensional scene model, determining the future movement track of the operation device and the barrier distance between the operation device and the target barrier, generating an early warning signal according to the future movement track and the barrier distance, and sending out early warning prompt, so that the damage caused by the approach or collision of an operation vehicle to the barrier during construction can be prevented, and the safety operation of operation personnel can be effectively prompted. In addition, an anti-collision monitoring system, equipment and a storage medium are also provided.
Description
Technical Field
The invention relates to the technical field of operation safety, in particular to an anti-collision monitoring method, device, equipment and medium for an operation vehicle.
Background
The working vehicle is equipped with special equipment or appliances, such as an engineering emergency car, a sprinkler, a sewage suction truck, a cement mixer truck, a crane truck, a medical truck and the like, and can assist in completing various construction operations. In general, work vehicles avoid approaching obstacles, but due to the varying skill level of the operator, various dangerous situations often arise in which an obstacle is accidentally approached. In addition, due to construction requirements, construction has to be performed in a scene close to an obstacle, for example, hoisting work close to high-voltage power equipment, offline piling and the like.
The conditions of line collision, line tripping, power failure and casualties caused by the lack of safe distance can result in tall machines and nearby obstacles, so the conditions need to be avoided to the utmost extent.
Disclosure of Invention
In view of the above, it is necessary to provide an anti-collision monitoring method, apparatus, device, and medium that avoid a collision of a work vehicle with a nearby obstacle, in view of the above-described problems.
An anti-collision monitoring method of an operation vehicle is applied to an anti-collision monitoring system, the anti-collision monitoring system comprises a vehicle-mounted auxiliary terminal and data acquisition terminals which are arranged in each preset direction at the top end of an operation device of the operation vehicle, and the method comprises the following steps:
acquiring a depth image acquired by a data acquisition terminal in each preset direction at the top end of the operation device;
identifying a target obstacle in the depth image in each preset direction, and constructing a three-dimensional scene model comprising a working vehicle and the target obstacle according to the depth image in each preset direction;
determining a future movement track of the operation device and an obstacle distance between the operation device and a target obstacle in the three-dimensional scene model, and generating an early warning signal according to the future movement track and the obstacle distance;
and sending the early warning signal to the vehicle-mounted auxiliary terminal so that the vehicle-mounted auxiliary terminal responds to the early warning signal to send out early warning prompt.
In one embodiment, the method for identifying the target obstacle in the depth image in each preset direction comprises the following steps:
performing color threshold segmentation on the depth image in each preset direction to obtain an interested color area image, and performing binarization processing on the color area image to obtain a binarized image;
positioning a suspected obstacle in the binary image, describing key features of the suspected obstacle by using a feature descriptor, classifying the suspected obstacle according to the key features by using a support vector machine classifier, and determining a target obstacle in the binary image;
and acquiring a target binary image containing a target obstacle, inputting the target binary image into the trained convolutional neural network, and acquiring the obstacle characteristic of the target obstacle.
In one embodiment, constructing a three-dimensional scene model of the work vehicle and the target obstacle from the depth image in each preset direction includes:
determining three-dimensional coordinate information of a working vehicle and a target obstacle in a working scene according to the depth information of the depth image in each preset direction;
geometrically calibrating the three-dimensional coordinate information to obtain shape parameters for representing the working vehicle and the target obstacle in the working scene;
collecting RGB images of a job scene, extracting texture information of the job scene from the RGB images, and generating a scene texture map of the job scene;
and generating a three-dimensional scene model of the working vehicle and the target obstacle according to the shape parameters and the scene texture map.
In one embodiment, determining a future movement track of the working device and an obstacle distance between the working device and a target obstacle, and generating an early warning signal according to the future movement track and the obstacle distance comprises:
acquiring a plurality of pieces of three-dimensional coordinate information of an operation device of an operation vehicle in an operation process, and simulating a future motion track of the operation device according to the plurality of pieces of three-dimensional coordinate information;
acquiring imaging parameters of a data acquisition terminal and target obstacle parameters of a target obstacle, and determining the obstacle distance between an operation device and the target obstacle according to the imaging parameters and the target obstacle parameters;
and when the target barrier is recognized to appear on the future motion track and the barrier distance meets the collision early warning condition, generating an early warning signal.
In one embodiment, when a target obstacle appears on a future motion track and the obstacle distance satisfies a collision warning condition, generating a warning signal includes:
when a target obstacle appears on a future movement track and the obstacle distance between the operation device and the target obstacle is smaller than a first preset safety distance, generating an early warning signal; or the like, or, alternatively,
when a target obstacle appears on a future movement track, the obstacle distance between the operation device and the target obstacle is smaller than a second preset safety distance, and the movement speed of the operation device exceeds a preset safety speed, generating an early warning signal; the second preset safety distance is greater than the first preset safety distance.
In one embodiment, before acquiring the depth image in each preset direction of the work vehicle, the method further includes:
when a monitoring distribution request input by a user side is received, searching available edge computing resources;
determining the distribution share of the usable edge computing resources according to the access number of the monitoring nodes and the request content of the monitoring distribution request in a resource distribution period;
determining the priority of the monitoring allocation request according to the time sequence precedence relationship of the monitoring allocation request, and allocating the edge computing resources according to allocation shares and the priority;
and detecting the availability of the channel transmission mode of each channel in the edge computing resource allocation, and correcting the unavailable channel transmission mode.
In one embodiment, the anti-collision monitoring method further includes:
setting an electronic fence in the depth image;
when the fact that the moving obstacle touches the electronic fence is detected, the moving obstacle is identified, obstacle features of the moving obstacle are obtained, and an early warning signal is generated according to the obstacle features of the moving obstacle; and/or the presence of a gas in the gas,
and acquiring a preset operation dangerous distance, and generating an early warning signal when the linear distance between the electronic fence and the operation device is smaller than the dangerous distance.
An anti-collision monitoring system, comprising:
the data acquisition terminal is arranged in each preset direction at the top end of the operation device of the operation vehicle and is used for acquiring a depth image in each preset direction of the operation vehicle;
the main control unit is used for identifying a target obstacle in the depth image in each preset direction and constructing a three-dimensional scene model comprising a working vehicle and the target obstacle according to the depth image in each preset direction; determining a future movement track of the operation device and an obstacle distance between the operation device and a target obstacle in the three-dimensional scene model, and generating an early warning signal according to the future movement track and the obstacle distance;
and the vehicle-mounted auxiliary terminal is used for responding the early warning signal and sending out early warning prompt.
A computer-readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of:
acquiring a depth image acquired by a data acquisition terminal in each preset direction at the top end of the operation device;
identifying a target obstacle in the depth image in each preset direction, and constructing a three-dimensional scene model comprising a working vehicle and the target obstacle according to the depth image in each preset direction;
determining a future movement track of the operation device and an obstacle distance between the operation device and a target obstacle in the three-dimensional scene model, and generating an early warning signal according to the future movement track and the obstacle distance;
and sending the early warning signal to the vehicle-mounted auxiliary terminal so that the vehicle-mounted auxiliary terminal responds to the early warning signal to send out early warning prompt.
An anti-collision monitoring device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of:
acquiring a depth image acquired by a data acquisition terminal in each preset direction at the top end of the operation device;
identifying a target obstacle in the depth image in each preset direction, and constructing a three-dimensional scene model comprising a working vehicle and the target obstacle according to the depth image in each preset direction;
determining a future movement track of the operation device and an obstacle distance between the operation device and a target obstacle in the three-dimensional scene model, and generating an early warning signal according to the future movement track and the obstacle distance;
and sending the early warning signal to the vehicle-mounted auxiliary terminal so that the vehicle-mounted auxiliary terminal responds to the early warning signal to send out early warning prompt.
The invention provides an anti-collision monitoring method, an anti-collision monitoring device, an anti-collision monitoring equipment and an anti-collision monitoring medium for a working vehicle. And then in the three-dimensional scene model, determining the future movement track of the operation device and the barrier distance between the operation device and the target barrier, generating an early warning signal according to the future movement track and the barrier distance, and sending out early warning prompt, so that the damage caused by the approach or collision of an operation vehicle to the barrier during construction can be prevented, and the safety operation of operation personnel can be effectively prompted.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Wherein:
fig. 1 is a flow diagram of a method of collision avoidance monitoring of a work vehicle in one embodiment;
fig. 2 is a schematic diagram of the structure of the collision avoidance monitoring system in one embodiment;
fig. 3 is a block diagram of a collision avoidance monitoring device in one embodiment.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, fig. 1 is a flowchart illustrating a method for collision avoidance monitoring of a work vehicle according to an embodiment. The anti-collision monitoring method for the working vehicle is applied to an anti-collision monitoring system, and the anti-collision monitoring system is composed of a data acquisition terminal, a main control unit and a vehicle-mounted auxiliary terminal. The data acquisition terminal is installed on each preset direction of the top end of the operation device of the operation vehicle and used for acquiring depth images of the operation vehicle and surrounding obstacles. The main control unit constructs a three-dimensional scene model comprising the operation vehicle and surrounding obstacles based on a visual technology and a deep learning technology, and sends out an early warning signal to possible external force damage based on an anti-collision learning algorithm. The vehicle-mounted auxiliary terminal displays the space conditions of the working vehicle and surrounding obstacles in real time, and responds to the early warning signal to send out warning prompt and control the working vehicle to stop working before external force damage occurs.
The anti-collision monitoring method for the work vehicle in the embodiment provides the following steps:
and 102, acquiring a depth image acquired by a data acquisition terminal in each preset direction at the top end of the operation device.
Because the depth information is the basic data that needs to be used in the anti-collision monitoring method, and the monocular camera cannot acquire the depth information of the object, the data acquisition terminal in this embodiment uses a camera with a binocular (or trinocular) lens to shoot the depth image to acquire the depth information. The binocular camera is fixed in each preset direction at the top end of the working device in an adsorption mode, and meanwhile, in order to obtain a better shooting effect, the distance between the two lenses is generally kept to be about 20 cm.
In addition, when the binocular camera collects video information, the problem that images acquired by different lenses are inconsistent may occur, and then coordinate deviation occurs in a subsequently constructed three-dimensional scene model is considered, so that the data collection terminal is further provided with a radar sensor and a nine-axis sensor in the embodiment, and the radar sensor is used for calibrating the shooting position of the images so as to keep the consistency of the spatial positions among the images. The nine-axis sensor is used for reflecting motion conditions such as left-right inclination, front-back inclination, left-right swinging and the like, and realizing high-precision motion detection.
Furthermore, algorithms based on the vision technology and the deep learning technology require a large amount of computing resources in operation, and if only a centralized computing mode in the central server is adopted, the central server needs a large amount of computing resources, and the computing resources on the application side are in an idle state. Therefore, in order to make full use of the computing power of the main control unit, more computing tasks are performed on the application side by the edge computing technology, so that the system has faster service response, and the real-time performance of the system is improved.
The anti-collision monitoring system in the embodiment can realize real-time monitoring of multiple monitoring nodes, namely, synchronous monitoring of a plurality of operation devices of an operation vehicle. When a monitoring allocation request for part of monitoring nodes input by an operator at the vehicle-mounted auxiliary terminal is received, the main control unit firstly searches available edge computing resources, the edge computing resources are idle computing resources which are not currently subjected to task processing by the main control unit, and the load of the central server can be effectively relieved by using the edge computing resources. Determining the distribution share of the usable edge computing resources according to the access number of the monitoring nodes and the request content of the monitoring distribution request in a resource distribution period, wherein the share of the edge computing resources distributed to each monitoring node is kept equal to the greatest extent during initial distribution, and increasing or reducing the distribution share of each monitoring in a proper amount according to different importance degrees of the request content so as to meet the actual monitoring requirement. And meanwhile, determining the priority of the monitoring distribution request according to the time sequence precedence relationship of the monitoring distribution request, wherein the priority of the distribution request is correspondingly higher the earlier the time sequence of the monitoring distribution request is. And finally, distributing the edge computing resources according to the determined distribution share and priority, so as to realize reasonable distribution of the edge computing resources of different monitoring nodes. In addition, the availability of the channel transmission mode of each channel in the edge computing resource allocation is detected, and the unavailable channel transmission mode is corrected, so that the efficiency of the resource allocation is improved.
And 104, identifying a target obstacle in the depth image in each preset direction, and constructing a three-dimensional scene model comprising the working vehicle and the target obstacle according to the depth image in each preset direction.
The target obstacle is a specific obstacle needing to be avoided by the operation device in an operation scene and comprises a cable, a building, a pedestrian and the like. And constructing a three-dimensional scene model comprising the working vehicle and the target obstacle to provide a scene model basis for the subsequent anti-collision monitoring based on the vision technology.
In one embodiment, the process of identifying the target obstacle is: firstly, converting the depth image in each preset direction from an RGB space to an HSV color space, then segmenting a color area image of blue, yellow and red three-color components from a channel of the HSV color space according to the threshold distribution condition of each color in the HSV color space, and then carrying out bitwise OR and AND operation on the color area image and the original depth image to obtain a binary image after color threshold segmentation. And then, detecting the suspected obstacles in the binary image by using an object detection algorithm, and positioning the detected suspected obstacles.
And determining the positions of key feature points in the suspected obstacle by using a FAST feature point detection algorithm, for example, within the range of the located suspected obstacle area. Taking any key feature point as a center, taking a neighborhood window with a preset size, randomly selecting a pair of points in the window, comparing the sizes of the two pixels, and carrying out binary assignment. And then, randomly selecting N pairs of random points in the window, repeating pixel size comparison and binary assignment to form a binary code, wherein the binary code is the feature descriptor. The key features of all key feature points are described using the feature descriptor. And inputting the key features into a support vector machine for classification, and determining a target obstacle in the suspected obstacles according to a classification result. And then acquiring the target binary image containing the target obstacle, inputting the target binary image into a trained convolutional neural network, such as a VGG-16 convolutional neural network or a Resnet-18 convolutional neural network, and acquiring the obstacle characteristics of the output target obstacle, wherein the obstacle characteristics can be used as obstacle information to remind operators before external force damage occurs.
In one embodiment, the process of constructing the three-dimensional scene model comprises: the method comprises the steps of obtaining internal parameters of a depth camera of a binocular camera and depth information corresponding to important pixel points in a depth image, wherein the important pixel points comprise pixel points covered by an operation vehicle and a target obstacle in an operation scene. And determining three-dimensional coordinate information of the pixel points in a camera coordinate system corresponding to the depth camera. And aligning the determined three-dimensional coordinate information with the three-dimensional coordinate information in a preset three-dimensional scene model, realizing geometric calibration of the determined three-dimensional coordinate information, and acquiring shape parameters for expressing the working vehicle and the target obstacle in the working scene. Collecting RGB images of the operation scene, extracting texture information of the operation scene from the RGB images, and generating a scene texture map of the operation scene. And projecting the vertex of the shape parameter to the scene texture map to obtain a mapping relation between the shape parameter and the scene texture map, and filling the shape parameter by using the scene texture map according to the mapping relation to obtain a three-dimensional scene model of the working vehicle and the target obstacle, wherein the three-dimensional scene model comprises the scene texture.
And 106, determining a future motion track of the operation device and the barrier distance between the operation device and the target barrier in the three-dimensional scene model, and generating an early warning signal according to the future motion track and the barrier distance.
And in the three-dimensional scene model, determining the obstacle distance from the target obstacle and the operation position of the top end part of the operation device in the operation vehicle according to the distance sensing function of the binocular camera. And then, a future movement track is given based on the existing operation position, and finally, the false collision prevention early warning and risk assessment of the whole operation vehicle are given according to the future movement track and the barrier distance.
In one embodiment, in order to ensure the accuracy of the determined future motion track, a plurality of three-dimensional coordinate information of the working device of the working vehicle in the working process is acquired, and n continuous three-dimensional coordinate information is smoothly connected into a section of known motion track. Determining the historical movement distance and the historical movement angle of each track in the N sections of known movement tracks, performing linear fitting on the known movement tracks according to the historical movement distance and the historical movement angle of each section of known movement track in a plane coordinate to obtain a historical fitting curve, and generating a grid with the nodes as the centers based on the positions of the nodes which are likely to actually pass through. In order to generate candidate tracks with different bending degrees, M control points are marked in the grid of each node, and a plurality of possible future motion tracks can be obtained by fitting a historical fitting curve with the nodes and the control points.
And then acquiring imaging parameters of the data acquisition terminal and target obstacle parameters of the target obstacle, wherein the imaging comprises calibration results of an internal reference matrix, a distortion matrix, an external reference rotation matrix and an external reference translation matrix, and the imaging parameters can be acquired from a calibration document of the binocular camera. The target obstacle parameters include width, height, coordinates, and category information of the target obstacle in the field-of-view imaging. Because the distance between the binocular camera and the fixed structure of the working vehicle is unchanged, in the three-dimensional scene model, only the first relative distance between the binocular camera and the fixed structure needs to be acquired, then the second relative distance between the working device and the target obstacle is acquired based on the future movement track, and the first relative distance and the second relative distance are compared to determine the actual obstacle distance between the working device and the target obstacle. And finally, generating an early warning signal when the target barrier appears on the future motion track and the distance between the barriers meets the collision early warning condition.
Illustratively, the collision warning condition includes: when a target obstacle appears on the future movement track and the obstacle distance between the operation device and the target obstacle is smaller than a first preset safety distance. Or when the target obstacle appears on the future movement track, the obstacle distance between the working device and the target obstacle is smaller than a second preset safety distance, and the movement speed of the working device exceeds a preset safety speed. The second preset safety distance is greater than the first preset safety distance.
And step 108, sending the early warning signal to the vehicle-mounted auxiliary terminal so that the vehicle-mounted auxiliary terminal responds to the early warning signal to send out early warning prompt.
The vehicle-mounted auxiliary terminal is installed in a control room of an operation vehicle and mainly comprises a video display device, an early warning device, an emergency cut-off device and a communication device. And the main control unit sends the real-time video signal for monitoring the whole operation process and the generated early warning signal to the vehicle-mounted auxiliary terminal. The operator can know the operation condition in real time from the vehicle-mounted auxiliary terminal and analyze the operation condition. The video display device can display video information and historical running tracks of six directions, namely, upper, lower, left, right, front and back directions, and can also realize running track prediction and demonstration of the operation device. The early warning device sends out early warning reminding containing barrier characteristics when receiving the early warning signal, and the emergency cut-off device automatically interrupts the operation of the working vehicle. The operator can communicate with the field operation issuer and the related supervising personnel in real time through the communication device to report the monitored operation condition.
Furthermore, the embodiment can also use a plurality of binocular cameras to mark a range in the depth image, so as to construct an electronic fence. When the moving obstacle is detected to touch the electronic fence, the moving obstacle is identified, obstacle features of the moving obstacle are obtained, and an early warning signal is generated according to the obstacle features of the moving obstacle, so that moving obstacles such as external pedestrians or vehicles are prevented from entering an area in the electronic fence. And a preset operation dangerous distance can be acquired, and when the linear distance between the electronic fence and the operation device is smaller than the dangerous distance, an early warning signal is generated to ensure that the operation device works within a specified range.
According to the anti-collision monitoring method for the working vehicle, the target obstacle in the acquired depth image is identified, and the three-dimensional scene model comprising the working vehicle and the target obstacle is constructed according to the depth image, so that safety protection monitoring suitable for different scenes is realized. And then in the three-dimensional scene model, determining the future movement track of the operation device and the barrier distance between the operation device and the target barrier, generating an early warning signal according to the future movement track and the barrier distance, and sending out early warning prompt, so that the damage caused by the approach or collision of an operation vehicle to the barrier during construction can be prevented, and the safety operation of operation personnel can be effectively prompted.
In one embodiment, as shown in fig. 2, there is provided an anti-collision monitoring system, the apparatus comprising:
the data acquisition terminal 202 is installed in each preset direction at the top end of the operation device of the operation vehicle and is used for acquiring a depth image in each preset direction of the operation vehicle;
the main control unit 204 is configured to identify a target obstacle in the depth image in each preset direction, and construct a three-dimensional scene model including a work vehicle and the target obstacle according to the depth image in each preset direction; determining a future movement track of the operation device and an obstacle distance between the operation device and a target obstacle in the three-dimensional scene model, and generating an early warning signal according to the future movement track and the obstacle distance;
and the vehicle-mounted auxiliary terminal 206 is used for responding to the early warning signal and sending out early warning prompt.
According to the anti-collision monitoring system, the target obstacles in the acquired depth image are identified, and the three-dimensional scene model comprising the operation vehicle and the target obstacles is constructed according to the depth image, so that the safety protection monitoring under different scenes is realized. And then in the three-dimensional scene model, determining the future movement track of the operation device and the barrier distance between the operation device and the target barrier, generating an early warning signal according to the future movement track and the barrier distance, and sending out early warning prompt, so that the damage caused by the approach or collision of an operation vehicle to the barrier during construction can be prevented, and the safety operation of operation personnel can be effectively prompted.
In an embodiment, the main control unit 204 is further specifically configured to: performing color threshold segmentation on the depth image in each preset direction to obtain an interested color area image, and performing binarization processing on the color area image to obtain a binarized image; positioning a suspected obstacle in the binary image, describing key features of the suspected obstacle by using a feature descriptor, classifying the suspected obstacle according to the key features by using a support vector machine classifier, and determining a target obstacle in the binary image; and acquiring a target binary image containing a target obstacle, inputting the target binary image into the trained convolutional neural network, and acquiring the obstacle characteristic of the target obstacle.
In an embodiment, the main control unit 204 is further specifically configured to: determining three-dimensional coordinate information of a working vehicle and a target obstacle in a working scene according to the depth information of the depth image in each preset direction; geometrically calibrating the three-dimensional coordinate information to obtain shape parameters for representing the working vehicle and the target obstacle in the working scene; collecting RGB images of a job scene, extracting texture information of the job scene from the RGB images, and generating a scene texture map of the job scene; and generating a three-dimensional scene model of the working vehicle and the target obstacle according to the shape parameters and the scene texture map.
In an embodiment, the main control unit 204 is further specifically configured to: acquiring a plurality of pieces of three-dimensional coordinate information of an operation device of an operation vehicle in an operation process, and simulating a future motion track of the operation device according to the plurality of pieces of three-dimensional coordinate information; acquiring imaging parameters of a data acquisition terminal and target obstacle parameters of a target obstacle, and determining the obstacle distance between an operation device and the target obstacle according to the imaging parameters and the target obstacle parameters; and when the target barrier is recognized to appear on the future motion track and the barrier distance meets the collision early warning condition, generating an early warning signal.
In an embodiment, the main control unit 204 is further specifically configured to: when a target obstacle appears on a future movement track and the obstacle distance between the operation device and the target obstacle is smaller than a first preset safety distance, generating an early warning signal; or when a target obstacle appears on the future movement track, the obstacle distance between the operation device and the target obstacle is smaller than a second preset safety distance, and the movement speed of the operation device exceeds a preset safety speed, generating an early warning signal; the second preset safety distance is greater than the first preset safety distance.
In an embodiment, the main control unit 204 is further specifically configured to: before acquiring the depth image of the working vehicle in each preset direction acquired by the data acquisition terminal, the method further comprises the following steps: when a monitoring distribution request input by a user side is received, searching available edge computing resources; determining the distribution share of the usable edge computing resources according to the access number of the monitoring nodes and the request content of the monitoring distribution request in a resource distribution period; determining the priority of the monitoring allocation request according to the time sequence precedence relationship of the monitoring allocation request, and allocating the edge computing resources according to allocation shares and the priority; and detecting the availability of the channel transmission mode of each channel in the edge computing resource allocation, and correcting the unavailable channel transmission mode.
In an embodiment, the main control unit 204 is further specifically configured to: setting an electronic fence in the depth image; when the fact that the moving obstacle touches the electronic fence is detected, the moving obstacle is identified, obstacle features of the moving obstacle are obtained, and an early warning signal is generated according to the obstacle features of the moving obstacle; and/or acquiring a preset operation dangerous distance, and generating an early warning signal when the linear distance between the electronic fence and the operation device is smaller than the dangerous distance.
Fig. 3 shows an internal structure diagram of the collision avoidance monitoring device in one embodiment. As shown in fig. 3, the collision avoidance monitoring device includes a processor, a memory, and a network interface connected by a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the anti-collision monitoring device stores an operating system and may also store a computer program, which, when executed by the processor, may cause the processor to implement the anti-collision monitoring method for the work vehicle. The internal memory may also have stored therein a computer program that, when executed by the processor, causes the processor to perform a method of collision avoidance monitoring of a work vehicle. It will be understood by those skilled in the art that the configuration shown in fig. 3 is a block diagram of only a portion of the configuration relevant to the present application, and does not constitute a limitation on the collision avoidance monitoring apparatus to which the present application is applied, and a particular collision avoidance monitoring apparatus may include more or fewer components than those shown in the drawings, or may combine certain components, or have a different arrangement of components.
An anti-collision monitoring device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the following steps when executing the computer program: acquiring a depth image acquired by a data acquisition terminal in each preset direction at the top end of the operation device; identifying a target obstacle in the depth image in each preset direction, and constructing a three-dimensional scene model comprising a working vehicle and the target obstacle according to the depth image in each preset direction; determining a future movement track of the operation device and an obstacle distance between the operation device and a target obstacle in the three-dimensional scene model, and generating an early warning signal according to the future movement track and the obstacle distance; and sending the early warning signal to the vehicle-mounted auxiliary terminal so that the vehicle-mounted auxiliary terminal responds to the early warning signal to send out early warning prompt.
In one embodiment, identifying a target obstacle in the depth image in each preset direction comprises: performing color threshold segmentation on the depth image in each preset direction to obtain an interested color area image, and performing binarization processing on the color area image to obtain a binarized image; positioning a suspected obstacle in the binary image, describing key features of the suspected obstacle by using a feature descriptor, classifying the suspected obstacle according to the key features by using a support vector machine classifier, and determining a target obstacle in the binary image; and acquiring a target binary image containing a target obstacle, inputting the target binary image into the trained convolutional neural network, and acquiring the obstacle characteristic of the target obstacle.
In one embodiment, constructing a three-dimensional scene model of the work vehicle and the target obstacle from the depth image in each preset direction includes: determining three-dimensional coordinate information of a working vehicle and a target obstacle in a working scene according to the depth information of the depth image in each preset direction; geometrically calibrating the three-dimensional coordinate information to obtain shape parameters for representing the working vehicle and the target obstacle in the working scene; collecting RGB images of a job scene, extracting texture information of the job scene from the RGB images, and generating a scene texture map of the job scene; and generating a three-dimensional scene model of the working vehicle and the target obstacle according to the shape parameters and the scene texture map.
In one embodiment, determining a future movement trajectory of the working device and an obstacle distance between the working device and a target obstacle, and generating an early warning signal according to the future movement trajectory and the obstacle distance comprises: acquiring a plurality of pieces of three-dimensional coordinate information of an operation device of an operation vehicle in an operation process, and simulating a future motion track of the operation device according to the plurality of pieces of three-dimensional coordinate information; acquiring imaging parameters of a data acquisition terminal and target obstacle parameters of a target obstacle, and determining the obstacle distance between an operation device and the target obstacle according to the imaging parameters and the target obstacle parameters; and when the target barrier is recognized to appear on the future motion track and the barrier distance meets the collision early warning condition, generating an early warning signal.
In one embodiment, when a target obstacle appears on a future motion trajectory and the obstacle distance satisfies a collision warning condition, generating a warning signal includes: when a target obstacle appears on a future movement track and the obstacle distance between the operation device and the target obstacle is smaller than a first preset safety distance, generating an early warning signal; or when a target obstacle appears on the future movement track, the obstacle distance between the operation device and the target obstacle is smaller than a second preset safety distance, and the movement speed of the operation device exceeds a preset safety speed, generating an early warning signal; the second preset safety distance is greater than the first preset safety distance.
In one embodiment, before acquiring the depth image in each preset direction of the work vehicle, acquired by the data acquisition terminal, the method further comprises: when a monitoring distribution request input by a user side is received, searching available edge computing resources; determining the distribution share of the usable edge computing resources according to the access number of the monitoring nodes and the request content of the monitoring distribution request in a resource distribution period; determining the priority of the monitoring allocation request according to the time sequence precedence relationship of the monitoring allocation request, and allocating the edge computing resources according to allocation shares and the priority; and detecting the availability of the channel transmission mode of each channel in the edge computing resource allocation, and correcting the unavailable channel transmission mode.
In one embodiment, further performing: setting an electronic fence in the depth image; when the fact that the moving obstacle touches the electronic fence is detected, the moving obstacle is identified, obstacle features of the moving obstacle are obtained, and an early warning signal is generated according to the obstacle features of the moving obstacle; and/or acquiring a preset operation dangerous distance, and generating an early warning signal when the linear distance between the electronic fence and the operation device is smaller than the dangerous distance.
A computer-readable storage medium storing a computer program which, when executed by a processor, performs the steps of: acquiring a depth image acquired by a data acquisition terminal in each preset direction at the top end of the operation device; identifying a target obstacle in the depth image in each preset direction, and constructing a three-dimensional scene model comprising a working vehicle and the target obstacle according to the depth image in each preset direction; determining a future movement track of the operation device and an obstacle distance between the operation device and a target obstacle in the three-dimensional scene model, and generating an early warning signal according to the future movement track and the obstacle distance; and sending the early warning signal to the vehicle-mounted auxiliary terminal so that the vehicle-mounted auxiliary terminal responds to the early warning signal to send out early warning prompt.
In one embodiment, identifying a target obstacle in the depth image in each preset direction comprises: performing color threshold segmentation on the depth image in each preset direction to obtain an interested color area image, and performing binarization processing on the color area image to obtain a binarized image; positioning a suspected obstacle in the binary image, describing key features of the suspected obstacle by using a feature descriptor, classifying the suspected obstacle according to the key features by using a support vector machine classifier, and determining a target obstacle in the binary image; and acquiring a target binary image containing a target obstacle, inputting the target binary image into the trained convolutional neural network, and acquiring the obstacle characteristic of the target obstacle.
In one embodiment, constructing a three-dimensional scene model of the work vehicle and the target obstacle from the depth image in each preset direction includes: determining three-dimensional coordinate information of a working vehicle and a target obstacle in a working scene according to the depth information of the depth image in each preset direction; geometrically calibrating the three-dimensional coordinate information to obtain shape parameters for representing the working vehicle and the target obstacle in the working scene; collecting RGB images of a job scene, extracting texture information of the job scene from the RGB images, and generating a scene texture map of the job scene; and generating a three-dimensional scene model of the working vehicle and the target obstacle according to the shape parameters and the scene texture map.
In one embodiment, determining a future movement trajectory of the working device and an obstacle distance between the working device and a target obstacle, and generating an early warning signal according to the future movement trajectory and the obstacle distance comprises: acquiring a plurality of pieces of three-dimensional coordinate information of an operation device of an operation vehicle in an operation process, and simulating a future motion track of the operation device according to the plurality of pieces of three-dimensional coordinate information; acquiring imaging parameters of a data acquisition terminal and target obstacle parameters of a target obstacle, and determining the obstacle distance between an operation device and the target obstacle according to the imaging parameters and the target obstacle parameters; and when the target barrier is recognized to appear on the future motion track and the barrier distance meets the collision early warning condition, generating an early warning signal.
In one embodiment, when a target obstacle appears on a future motion trajectory and the obstacle distance satisfies a collision warning condition, generating a warning signal includes: when a target obstacle appears on a future movement track and the obstacle distance between the operation device and the target obstacle is smaller than a first preset safety distance, generating an early warning signal; or when a target obstacle appears on the future movement track, the obstacle distance between the operation device and the target obstacle is smaller than a second preset safety distance, and the movement speed of the operation device exceeds a preset safety speed, generating an early warning signal; the second preset safety distance is greater than the first preset safety distance.
In one embodiment, before acquiring the depth image in each preset direction of the work vehicle, acquired by the data acquisition terminal, the method further comprises: when a monitoring distribution request input by a user side is received, searching available edge computing resources; determining the distribution share of the usable edge computing resources according to the access number of the monitoring nodes and the request content of the monitoring distribution request in a resource distribution period; determining the priority of the monitoring allocation request according to the time sequence precedence relationship of the monitoring allocation request, and allocating the edge computing resources according to allocation shares and the priority; and detecting the availability of the channel transmission mode of each channel in the edge computing resource allocation, and correcting the unavailable channel transmission mode.
In one embodiment, further performing: setting an electronic fence in the depth image; when the fact that the moving obstacle touches the electronic fence is detected, the moving obstacle is identified, obstacle features of the moving obstacle are obtained, and an early warning signal is generated according to the obstacle features of the moving obstacle; and/or acquiring a preset operation dangerous distance, and generating an early warning signal when the linear distance between the electronic fence and the operation device is smaller than the dangerous distance.
It should be noted that the above-mentioned method, apparatus, device and computer-readable storage medium for monitoring collision avoidance of a work vehicle belong to a general inventive concept, and the contents in the embodiments of the method, apparatus, device and computer-readable storage medium for monitoring collision avoidance of a work vehicle are applicable to each other.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. An anti-collision monitoring method for a working vehicle is applied to an anti-collision monitoring system, the anti-collision monitoring system comprises a vehicle-mounted auxiliary terminal and data acquisition terminals which are arranged in each preset direction on the top end of a working device of the working vehicle, and the method comprises the following steps:
acquiring a depth image acquired by the data acquisition terminal in each preset direction at the top end of the operation device;
identifying a target obstacle in the depth image in each preset direction, and constructing a three-dimensional scene model comprising the working vehicle and the target obstacle according to the depth image in each preset direction;
determining a future movement track of the operation device and an obstacle distance between the operation device and the target obstacle in the three-dimensional scene model, and generating an early warning signal according to the future movement track and the obstacle distance;
and sending the early warning signal to the vehicle-mounted auxiliary terminal so that the vehicle-mounted auxiliary terminal responds to the early warning signal to send out early warning prompt.
2. The collision avoidance monitoring method of claim 1, wherein said identifying a target obstacle in the depth image in each of the preset directions comprises:
performing color threshold segmentation on the depth image in each preset direction to obtain an interested color area image, and performing binarization processing on the color area image to obtain a binarized image;
positioning a suspected obstacle in the binarized image, describing key features of the suspected obstacle by using a feature descriptor, classifying the suspected obstacle according to the key features by using a support vector machine classifier, and determining a target obstacle in the binarized image;
and acquiring a target binary image containing the target obstacle, inputting the target binary image into the trained convolutional neural network, and acquiring the obstacle characteristic of the target obstacle.
3. The anti-collision monitoring method according to claim 1, wherein the constructing a three-dimensional scene model of the work vehicle and the target obstacle from the depth image in each preset direction includes:
determining three-dimensional coordinate information of the working vehicle and the target obstacle in a working scene according to the depth information of the depth image in each preset direction;
geometrically calibrating the three-dimensional coordinate information to obtain shape parameters for representing the working vehicle and the target obstacle in the working scene;
collecting an RGB image of the operation scene, extracting texture information of the operation scene from the RGB image, and generating a scene texture map of the operation scene;
and generating a three-dimensional scene model of the working vehicle and the target obstacle according to the shape parameters and the scene texture map.
4. The method of claim 1, wherein determining a future movement trajectory of the work implement and an obstacle distance between the work implement and the target obstacle, and generating an alert signal based on the future movement trajectory and the obstacle distance comprises:
acquiring a plurality of pieces of three-dimensional coordinate information of an operation device of the operation vehicle in an operation process, and simulating a future motion track of the operation device according to the plurality of pieces of three-dimensional coordinate information;
acquiring imaging parameters of the data acquisition terminal and target obstacle parameters of the target obstacle, and determining the obstacle distance between the operation device and the target obstacle according to the imaging parameters and the target obstacle parameters;
generating the early warning signal when the target obstacle is identified to appear on the future motion track and the obstacle distance meets a collision early warning condition.
5. The collision avoidance monitoring method according to claim 4, wherein the generating the warning signal when the target obstacle appears on the future motion trajectory and the obstacle distance satisfies a collision warning condition comprises:
when a target obstacle appears on the future movement track and the obstacle distance between the operation device and the target obstacle is smaller than a first preset safety distance, generating the early warning signal; or the like, or, alternatively,
when a target obstacle appears on the future movement track, the obstacle distance between the operation device and the target obstacle is smaller than a second preset safety distance, and the movement speed of the operation device exceeds a preset safety speed, generating an early warning signal; the second preset safety distance is greater than the first preset safety distance.
6. The method according to claim 1, further comprising, before the obtaining the depth image in each preset direction of the work vehicle collected by the data collection terminal:
when a monitoring distribution request input by a user side is received, searching available edge computing resources;
determining the distribution share of the usable edge computing resources according to the access number of the monitoring nodes and the request content of the monitoring distribution request in a resource distribution period;
determining the priority of the monitoring allocation request according to the time sequence precedence relationship of the monitoring allocation request, and allocating the edge computing resources according to the allocation share and the priority;
and detecting the availability of the channel transmission mode of each channel in the edge computing resource allocation, and correcting the unavailable channel transmission mode.
7. The method of claim 1, wherein the collision avoidance monitoring method further comprises:
setting an electronic fence in the depth image;
when a moving obstacle is detected to touch the electronic fence, identifying the moving obstacle, acquiring obstacle features of the moving obstacle, and generating an early warning signal according to the obstacle features of the moving obstacle; and/or the presence of a gas in the gas,
and acquiring a preset operation dangerous distance, and generating an early warning signal when the linear distance between the electronic fence and the operation device is smaller than the dangerous distance.
8. An anti-collision monitoring system, comprising:
the data acquisition terminal is arranged in each preset direction at the top end of an operation device of the operation vehicle and is used for acquiring a depth image in each preset direction of the operation vehicle;
the main control unit is used for identifying a target obstacle in the depth image in each preset direction and constructing a three-dimensional scene model comprising the working vehicle and the target obstacle according to the depth image in each preset direction; determining a future movement track of the operation device and an obstacle distance between the operation device and the target obstacle in the three-dimensional scene model, and generating an early warning signal according to the future movement track and the obstacle distance;
and the vehicle-mounted auxiliary terminal is used for responding the early warning signal and sending out early warning prompt.
9. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 7.
10. A collision avoidance monitoring device comprising a memory and a processor, wherein the memory stores a computer program which, when executed by the processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111325161.4A CN114022846A (en) | 2021-11-10 | 2021-11-10 | Anti-collision monitoring method, device, equipment and medium for working vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111325161.4A CN114022846A (en) | 2021-11-10 | 2021-11-10 | Anti-collision monitoring method, device, equipment and medium for working vehicle |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114022846A true CN114022846A (en) | 2022-02-08 |
Family
ID=80063095
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111325161.4A Pending CN114022846A (en) | 2021-11-10 | 2021-11-10 | Anti-collision monitoring method, device, equipment and medium for working vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114022846A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114694423A (en) * | 2022-03-26 | 2022-07-01 | 太仓武港码头有限公司 | Ship loader safety early warning method, system, terminal and storage medium |
| WO2023216555A1 (en) * | 2022-05-10 | 2023-11-16 | 丰疆智能(深圳)有限公司 | Obstacle avoidance method and apparatus based on binocular vision, and robot and medium |
| CN117372427A (en) * | 2023-12-06 | 2024-01-09 | 南昌中展数智科技有限公司 | Engineering construction supervision method and system based on video analysis |
-
2021
- 2021-11-10 CN CN202111325161.4A patent/CN114022846A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114694423A (en) * | 2022-03-26 | 2022-07-01 | 太仓武港码头有限公司 | Ship loader safety early warning method, system, terminal and storage medium |
| WO2023216555A1 (en) * | 2022-05-10 | 2023-11-16 | 丰疆智能(深圳)有限公司 | Obstacle avoidance method and apparatus based on binocular vision, and robot and medium |
| CN117372427A (en) * | 2023-12-06 | 2024-01-09 | 南昌中展数智科技有限公司 | Engineering construction supervision method and system based on video analysis |
| CN117372427B (en) * | 2023-12-06 | 2024-03-22 | 南昌中展数智科技有限公司 | Engineering construction supervision method and system based on video analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114022846A (en) | Anti-collision monitoring method, device, equipment and medium for working vehicle | |
| CN113345019B (en) | Method, equipment and medium for measuring potential hazards of transmission line channel target | |
| CN110430401B (en) | Vehicle blind area early warning method, early warning device, MEC platform and storage medium | |
| CN113424121A (en) | Vehicle speed control method and device based on automatic driving and computer equipment | |
| CN111160302A (en) | Obstacle information identification method and device based on automatic driving environment | |
| US20220373353A1 (en) | Map Updating Method and Apparatus, and Device | |
| CN113470374A (en) | Vehicle overspeed monitoring method and device, computer equipment and storage medium | |
| CN111988524A (en) | Unmanned aerial vehicle and camera collaborative obstacle avoidance method, server and storage medium | |
| CN113012315B (en) | Safety monitoring system and method for working machine and working machine | |
| CN116110012B (en) | Dangerous violation identification method and system for intelligent construction site | |
| CN115917357A (en) | Method and device for detecting undefined class obstacle and computer equipment | |
| CN113189989A (en) | Vehicle intention prediction method, device, equipment and storage medium | |
| CN113965733A (en) | Binocular video monitoring method, system, computer equipment and storage medium | |
| JP7360520B1 (en) | Object tracking integration method and integration device | |
| CN112001336A (en) | Pedestrian boundary crossing alarm method, device, equipment and system | |
| Katsamenis et al. | Real time road defect monitoring from UAV visual data sources | |
| CN116597371A (en) | Dangerous object early warning method, system and computer equipment based on image monitoring | |
| CN112529953B (en) | Elevator space state judgment method and device and storage medium | |
| CN114170498A (en) | Detection method and device for spilled objects, computer equipment and storage medium | |
| CN115100632A (en) | Expansion point cloud identification method and device, computer equipment and storage medium | |
| CN114167443A (en) | Information completion method and device, computer equipment and storage medium | |
| CN113393423A (en) | Cliff detection method and device based on point cloud and mobile robot | |
| CN114078326B (en) | Collision detection method, device, visual sensor and storage medium | |
| CN114648689A (en) | Elevator space state judgment method, device, equipment and storage medium | |
| CN113807125A (en) | Emergency lane occupation detection method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |