CN114115246B - Remote driving auxiliary system and method thereof - Google Patents
Remote driving auxiliary system and method thereof Download PDFInfo
- Publication number
- CN114115246B CN114115246B CN202111322167.6A CN202111322167A CN114115246B CN 114115246 B CN114115246 B CN 114115246B CN 202111322167 A CN202111322167 A CN 202111322167A CN 114115246 B CN114115246 B CN 114115246B
- Authority
- CN
- China
- Prior art keywords
- driving
- remote
- unit
- video data
- control instruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000033001 locomotion Effects 0.000 claims abstract description 48
- 230000004424 eye movement Effects 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 12
- 238000005259 measurement Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 9
- 230000004886 head movement Effects 0.000 claims description 8
- 230000006399 behavior Effects 0.000 claims description 6
- 210000001508 eye Anatomy 0.000 description 16
- 238000004364 calculation method Methods 0.000 description 11
- 210000003128 head Anatomy 0.000 description 11
- 210000005252 bulbus oculi Anatomy 0.000 description 8
- 230000007246 mechanism Effects 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 7
- 230000008901 benefit Effects 0.000 description 4
- 238000004088 simulation Methods 0.000 description 4
- 230000001133 acceleration Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000009434 installation Methods 0.000 description 3
- 241000282414 Homo sapiens Species 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000011835 investigation Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Electromagnetism (AREA)
- Aviation & Aerospace Engineering (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- General Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Traffic Control Systems (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
The invention provides a remote driving auxiliary system, wherein a first auxiliary unit is used for acquiring motion information of a remote driver in remote driving, generating a gesture control instruction according to the motion information, and sending the gesture control instruction to a second auxiliary unit, wherein the motion information is used for representing physical motion information of the remote driver on a driving concern point in the remote driving, and the driving concern point is a key point of the remote driver for performing driving reaction on display unit display video data according to driving behavior; and the second auxiliary unit is used for adjusting the acquisition angle of the camera unit according to the gesture control instruction to acquire new video data when receiving the gesture control instruction, and transmitting the new video data to the display unit so that a remote driver can make driving reaction based on the new video data displayed by the display unit. The invention further provides a remote driving assistance method. The invention can truly realize real-time control of the remote vehicle by observing the video under the current network bandwidth condition.
Description
Technical Field
The invention belongs to the field of automatic driving, and particularly relates to a remote driving and remote taking over system.
Background
With the popularization of 5G communication technology and the gradual maturity of automatic driving technology, the remote driving and remote taking over of vehicles have been increasingly researched and paid attention.
Currently, the remote driving system/remote take-over system is basically realized based on multiple paths of high-definition videos of a remote vehicle, namely, a plurality of cameras are installed on the remote vehicle, camera data are transmitted back to a remote console in real time through a 5G network, and a remote driver controls the remote vehicle according to video pictures. Although the scheme is simple and visual, and the driving assistance indication line and other technologies are matched, a driver can remotely control the remote vehicle through proper training, for example, the scheme can be applied to a remote driving scene with single driving scene such as a mine, a port, a wharf and the like under the current technical conditions. However, the disadvantages of this approach are also apparent. This is because current 5G macro networks typically upload at 100Mbit/s and are only capable of transmitting 1 high definition video. If real-time transmission of multiple paths of high-definition video is to be realized, a 5G private network must be used, and meanwhile, the uploading speed of 5G needs to be optimized. Thus, not only the construction cost of the system is greatly increased, but also the use occasion is limited. Meanwhile, in order to adapt to the observation habit of an operator, multiple paths of videos are usually spliced into one picture at a local or remote end, so that the real-time performance of the system is further reduced, and the vehicle cannot be driven at a high speed (such as a vehicle speed higher than 30 km/h).
In addition, the driving scene is compared with a single remote driving system/remote take-over system, and the technical scheme is realized by combining a high-precision simulation scene with a dynamic object, as disclosed in a patent document (the publication number is CN 113022540A), namely, a high-precision map of a target area is acquired in advance, an object list identified by a local automatic driving system of a remote vehicle is returned to a remote driving end, and a simulation server of the remote driving end carries out simulation restoration. Although the scheme can reduce the requirement on network bandwidth, can be applied to a remote driving system/remote take-over system with a single driving scene and a relatively stable target, the scheme lacks real-time video, and has the safety to be improved when being applied to traffic scenes with a large number of dynamic targets (pedestrians and other traffic vehicles). Meanwhile, the scheme has very high requirements on the computing power of the simulation server.
Based on the above, the present application provides a technical solution to solve the above technical problems.
Disclosure of Invention
A first object of the present invention is to obtain a remote driving assistance system with greatly improved safety, which can truly realize real-time control of a remote vehicle by observing video under the current network bandwidth condition.
A second object of the present invention is to obtain a driving assistance method for remote driving that greatly improves safety, and that can truly realize real-time control of a remote vehicle by observing video under the current network bandwidth condition.
A first aspect of the present invention provides an assistance system for remote driving, which is applied to assist a remote driver in remote driving a remote vehicle, including a first assistance unit and a display unit disposed at the remote driving end, and a second assistance unit disposed at the remote vehicle end;
the first auxiliary unit is used for acquiring motion information of the remote driver in the remote driving, generating a gesture control instruction according to the motion information, and sending the gesture control instruction to the second auxiliary unit, wherein the motion information is used for representing body motion information of the remote driver on a driving attention point in the remote driving, and the driving attention point is a key point of the remote driver for performing driving reaction on the video data displayed by the display unit according to driving behaviors;
the second auxiliary unit is used for adjusting the acquisition angle of the camera unit according to the gesture control instruction to acquire new video data when receiving the gesture control instruction from the first auxiliary unit, and transmitting the new video data to the display unit so that the remote driver can make driving reaction based on the new video data displayed by the display unit.
In one embodiment of the present invention, the remote driving assistance system is a remote driving takeover system.
Specifically, the first auxiliary unit comprises a remote driver motion information unit, a camera gesture calculation unit and a display unit.
Specifically, the second auxiliary unit comprises a servo control system and an image pickup unit. Specifically, the camera unit is a front camera unit. For example, the front camera unit is mounted on the single-axis servo control system with adjustable position. Preferably, the front-end camera unit is mounted on a single-axis servo control system and integrally mounted at a proper position in front of the vehicle. In this position, images of the front, the left front, and the right front of the vehicle can be acquired by rotation of the camera.
In a preferred embodiment of the present invention, the number of cameras protected by the front camera unit is not more than 2, preferably one.
In a preferred embodiment of the invention, the information collected by the remote driver movement information unit comprises eye movement information and/or head movement information.
In a preferred embodiment of the present invention, the information collected by the remote driver movement information unit is derived from remote driver eye movement direction information collected by tracking by an eye movement tracking system. More preferably, the inertial measurement unit from the eye tracking system tracks the acquired remote driver's head movement information. In particular, the eye tracking system is a head-mounted device that is capable of tracking the direction of eye movement of a wearer (remote driver). Also, it includes an Inertial Measurement Unit (IMU) containing an acceleration sensor and a gyroscope, the inertial measurement unit being used to track the movement of the wearer's head.
In a specific embodiment, the camera gesture calculation unit calculates the rotation direction of the camera in the horizontal plane (i.e. rotation around the Z axis) in real time according to the output of the eye tracking system and in combination with the information such as the installation position of the camera on the vehicle, which is created in advance, and finally converts the rotation direction into a camera gesture control instruction, and sends the camera gesture control instruction to the single-axis servo control system at the vehicle-mounted end through a 4G or 5G network.
In one embodiment of the present invention, the single axis servo control system is a high precision, high dynamic micro servo mechanism comprising a micro motor, an embedded controller and corresponding mechanical structure which controls the rotation and positioning of the camera attached thereto about a vertical axis in accordance with instructions.
In a preferred embodiment of the invention, the gesture control instruction is sent to the second auxiliary unit via a 4G/5G network.
In a preferred embodiment of the invention, the new video data is transmitted to the display unit via a 4G/5G network.
In a second aspect of the present invention, there is provided a driving support method for remote driving, which is applied to the remote driving support system of the present invention, the method comprising:
acquiring motion information of the remote driver in remote driving through the first auxiliary unit, generating a gesture control instruction according to the motion information, and sending the gesture control instruction to the second auxiliary unit, wherein the motion information is used for representing body motion information of the remote driver on a driving attention point in remote driving, and the driving attention point is a key point of a driving response of the remote driver on video data displayed by the display unit according to driving behaviors;
and when the second auxiliary unit receives the gesture control instruction, adjusting the acquisition angle of the camera unit according to the gesture control instruction to acquire new video data, and transmitting the new video data to the display unit so that the remote driver can make driving reaction based on the new video data displayed by the display unit.
In a preferred embodiment of the invention, the gesture control instruction is sent to the second auxiliary unit via a 4G/5G network.
In a preferred embodiment of the invention, the new video data is transmitted to the display unit via a 4G/5G network.
In a preferred embodiment of the invention, the course of the driving assistance method is completed at a frequency of not less than 50 hz.
In a preferred embodiment of the present invention, the command tracking response delay of the single axis servo controller is no more than 5ms.
The invention can bring at least one of the following beneficial effects:
(1) The invention provides a remote driving auxiliary method which can truly realize real-time control of a remote vehicle by observing videos under the condition of the current network bandwidth, thereby greatly improving the safety.
(2) In one embodiment of the invention, the system provided by the invention can effectively solve the defects of the existing video-based remote driving/taking over system. According to the method, the follow-up control of the camera and the eye tracking of the driver are used, and only a single-path high-definition video is transmitted between the vehicle and a remote driving system, so that the requirement on computing capacity is greatly reduced; but can simultaneously meet the requirement of a remote driver on viewing the surrounding conditions of the vehicle in real time, and can truly realize the real-time control of the remote vehicle by observing videos under the condition of the current 5G network bandwidth, thereby greatly improving the safety.
Drawings
The above features, technical features, advantages and implementation thereof will be further described in the following detailed description of preferred embodiments with reference to the accompanying drawings in a clearly understandable manner.
FIG. 1 is an exemplary system architecture diagram of the present invention;
fig. 2 is a schematic diagram of an exemplary system operation of the present invention.
Detailed Description
In the invention, the inventor discovers an operator auxiliary facility and a device for an automatic driving vehicle remote take-over system through extensive and deep experiments, so that video of the surrounding environment of the remote vehicle is transmitted according to the attention point of an operator, the effectiveness of video transmission under the condition of low network bandwidth at present can be realized, and the driving habit of human beings is more met, thereby realizing the aim of improving the safety.
More specifically, the inventive concept of the present invention combines a remote control technique, a high-speed servo technique, an eye-tracking technique, and a high-precision indoor positioning technique to obtain a preferred embodiment.
The term "or" as used herein includes the relationship of "and" unless specifically stated and defined otherwise. The sum corresponds to the boolean logic operator AND, the OR corresponds to the boolean logic operator OR, AND the AND is a subset of OR.
The following details various aspects of the invention:
auxiliary system
The invention provides an auxiliary system for remote driving, which is applied to assisting a remote driver in remote driving of a remote vehicle, and comprises a first auxiliary unit and a display unit which are arranged at the remote driving end, and a second auxiliary unit which is arranged at the remote vehicle end;
the first auxiliary unit is used for acquiring motion information of the remote driver in remote driving, generating a gesture control instruction according to the motion information, and sending the gesture control instruction to the second auxiliary unit, wherein the motion information is used for representing body motion information of the remote driver on a driving attention point in remote driving, and the driving attention point is a key point of a driving response of the remote driver on the video data displayed by the display unit according to driving behaviors;
and the second auxiliary unit is used for adjusting the acquisition angle of the camera unit according to the gesture control instruction to acquire new video data when receiving the gesture control instruction, and transmitting the new video data to the display unit so that the remote driver can make driving reaction based on the new video data displayed by the display unit.
The inventor discovers that the video of the surrounding environment of the remote vehicle is transmitted according to the attention point of a driver, the video transmission effectiveness under the condition of the current lower network bandwidth can be realized, the driving habit of human beings is more met, the real-time control of the remote vehicle by observing the video can be truly realized under the condition of the current network bandwidth, and the safety is greatly improved.
In one specific embodiment of the invention, the gesture control instruction of the first auxiliary unit is sent to the second auxiliary unit through a 4G/5G network.
In one embodiment of the invention, the new video data is transmitted to the display unit via a 4G/5G network.
The "4G/5G network" of the present invention may be 4G, 5G, or a combination thereof.
It should be understood that although the present invention achieves the object of the present invention by adopting a new technical concept, that is, real-time control of a remote vehicle by observing video can be truly realized under the current network bandwidth condition; however, the present invention may employ various other networks for instruction transmission or data transmission in the art, provided that the object of the present invention is not limited thereto.
It should be understood that the gesture control command transmission and the transmission of the video data may be the same network or may be different networks.
In one embodiment of the invention, the data of the camera unit is turned back to the video receiving and displaying device of the remote driving end in real time through the 4G/5G network.
In a preferred embodiment of the present invention, the number of cameras protected by the front camera unit is not higher than 2, preferably one. It will be appreciated that one of the advantages of the solution of the present inventors is to allow remote driving of the vehicle in the case of the current 5G bandwidth. Even in the case where the bandwidth condition is satisfied, the present invention has another advantage that the demand for the number of cameras is reduced, and thus the setting of the number of cameras can be reduced, thereby obtaining technical advantages.
In one embodiment of the present invention, the remote driving assistance system is a remote driving takeover system.
First auxiliary unit
Specifically, the first auxiliary unit comprises a remote driver motion information unit and a camera gesture calculation unit. The remote driver movement information unit is arranged to obtain movement information of the remote driver in remote driving. The camera gesture calculation unit is configured to receive the motion information acquired by the remote driver motion information unit, generate a gesture control instruction according to the motion information, and send the generated gesture control instruction to the second auxiliary unit.
In a preferred embodiment of the invention, the information collected by the remote driver movement information unit comprises eye movement information and/or head movement information.
In a preferred embodiment of the present invention, the information collected by the remote driver movement information unit is derived from remote driver eye movement direction information collected by tracking by an eye movement tracking system.
More preferably, the head movement information from the remote driver tracked and collected by the inertial measurement unit.
In one embodiment, the eye tracking system is a head mounted device that is capable of tracking the direction of eye movement of a wearer (remote driver). Also, it includes an Inertial Measurement Unit (IMU) containing an acceleration sensor and a gyroscope, the inertial measurement unit being used to track the movement of the wearer's head.
In a specific embodiment, the camera gesture calculation unit calculates the rotation direction of the camera in the horizontal plane (i.e. rotation around the Z axis) in real time according to the output of the eye tracking system and in combination with the information such as the installation position of the camera on the vehicle, which is created in advance, and finally converts the rotation direction into a camera gesture control instruction, and sends the camera gesture control instruction to the single-axis servo control system at the vehicle-mounted end through a 4G or 5G network.
In one embodiment of the invention, the system provided by the invention can effectively solve the defects of the existing video-based remote driving/taking over system. According to the method, the follow-up control of the camera and the eye tracking of the driver are used, and only a single-path high-definition video is transmitted between the vehicle and a remote driving system, so that the requirement on computing capacity is greatly reduced; but can simultaneously meet the requirement of a remote driver on viewing the surrounding conditions of the vehicle in real time, and can truly realize the real-time control of the remote vehicle by observing videos under the condition of the current 5G network bandwidth, thereby greatly improving the safety.
The inventor finds that an eye tracker is used to acquire a focus of an operator (remote driver) on a vehicle-mounted video in a driving process, and a camera is adjusted to a desired position in real time; more preferably, the eye tracker not only tracks the eye movement of the operator, but also tracks the head movement of the operator through a built-in IMU, so that the precise acquisition of the region of interest and the position of the operator with respect to the vehicle-mounted video can be ensured.
The inventor has found through extensive and intensive investigation that the technical scheme of the invention only needs a servo control mechanism to adjust the gesture of the camera in real time according to the requirement of an operator (remote driver), so that the requirement threshold of the operator for the vehicle-mounted video in the driving process can be met by only needing one camera and one path of high-definition video, and the remote driving of the vehicle is realized under the condition of the current 5G bandwidth. The application range of 5G remote driving is enlarged, so that the method can be applied to a real traffic environment. It should be noted that although the invention can save resources to the greatest extent, only one camera and one path of high-definition video are needed to meet the requirement threshold of the driver, thereby allowing the application under the current 5G bandwidth limit; however, in order to achieve a better adjustment effect or improve the 5G bandwidth condition, the present invention is not limited to a single servo control mechanism, a single camera, and a single-path high-definition video.
Second auxiliary unit
Specifically, the second auxiliary unit includes a servo control system and an image capturing unit.
Specifically, the servo control system is configured to adjust the acquisition angle of the image capturing unit according to the attitude control instruction to acquire new video data when receiving the attitude control instruction.
Preferably, the servo control system is a single-axis servo control system.
The "camera unit" is arranged to acquire new video data and to transmit the new video data to the display unit so that the remote driver reacts to the driving based on the new video data displayed by the display unit. The book is provided with
It will be appreciated by those skilled in the art that the "camera unit" is used to collect driving information related to the surrounding environment, and is usually installed at a suitable collecting location of the vehicle, so long as the object of the present invention is not limited.
Specifically, the camera unit is a front camera unit. For example, the front camera unit is mounted on the single-axis servo control system with adjustable position.
In one embodiment of the invention, the front-end camera unit is mounted on a single-axis servo control system and integrally mounted in a suitable position in front of the vehicle. In this position, images of the front, the left front, and the right front of the vehicle can be acquired by rotation of the camera.
In one embodiment of the present invention, the single axis servo control system is a high precision, high dynamic micro servo mechanism comprising a micro motor, an embedded controller and corresponding mechanical structure which controls the rotation and positioning of the camera attached thereto about a vertical axis in accordance with instructions.
The inventor has found through extensive and intensive investigation that the technical scheme of the invention only needs a servo control mechanism to adjust the gesture of the camera in real time according to the requirement of an operator (remote driver), so that the requirement threshold of the operator for the vehicle-mounted video in the driving process can be met by only needing one camera and one path of high-definition video, and the remote driving of the vehicle is realized under the condition of the current 5G bandwidth. The application range of 5G remote driving is enlarged, so that the method can be applied to a real traffic environment. It should be noted that although the invention can save resources to the greatest extent, only one camera and one path of high-definition video are needed to meet the requirement threshold of the driver, thereby allowing the application under the current 5G bandwidth limit; however, in order to achieve a better adjustment effect or improve the 5G bandwidth condition, the present invention is not limited to a single servo control mechanism, a single camera, and a single-path high-definition video.
Display unit
The "display unit" is arranged to receive the video data stream of the second auxiliary unit. Specifically, herein, the "display unit" is configured to receive real-time video data from a camera of the vehicle-mounted terminal. It may be any device having a function of receiving video from a network and displaying it, as long as the object of the present invention is not limited. Specifically, the display unit is a video receiving display device.
Driving method
The invention provides a driving assisting method for remote driving, which is applied to the driving assisting system for remote driving, and comprises the following steps:
acquiring motion information of the remote driver in remote driving through the first auxiliary unit, generating a gesture control instruction according to the motion information, and sending the gesture control instruction to the second auxiliary unit, wherein the motion information is used for representing body motion information of the remote driver on a driving attention point in remote driving, and the driving attention point is a key point of a driving response of the remote driver on video data displayed by the display unit according to driving behaviors;
and when the second auxiliary unit receives the gesture control instruction, adjusting the acquisition angle of the camera unit according to the gesture control instruction to acquire new video data, and transmitting the new video data to the display unit so that the remote driver can make driving reaction based on the new video data displayed by the display unit.
In a preferred embodiment of the invention, the gesture control instruction is sent to the second auxiliary unit via a 4G/5G network.
In a preferred embodiment of the invention, the new video data is transmitted to the display unit via a 4G/5G network.
In a preferred embodiment of the invention, the course of the driving assistance method is completed at a frequency of not less than 50 hz.
In a preferred embodiment of the present invention, the command tracking response delay of the single axis servo controller is no more than 5ms.
In one embodiment, when the remote driver intends to turn the vehicle to the left, the operator (i.e., the remote driver) will naturally turn the head to the left or turn the eyeball in an effort to obtain video data of the left side of the vehicle. The eye tracking instrument acquires the eyeball horizontal movement distance of the operator or the rotation angle data of the head, and sends the eyeball horizontal movement distance or the rotation angle data to the camera pose calculation unit. The camera gesture calculation unit converts the eye movement data and the head rotation data into gesture data of the camera and sends the gesture data to a single-axis servo controller at the vehicle-mounted end through a wireless network (4G/5G). The single-shaft servo controller controls the motor to rotate according to expected camera posture data, drives the camera to rotate leftwards around the vertical axis to an expected position, and in the rotating process, the camera always acquires video data and sends the video data back to video receiving and displaying equipment of a remote driving end through a network. The whole process is completed at the frequency of not lower than 50hz, and the instruction tracking response delay of the single-axis servo controller is not more than 5ms, so that smooth pictures and no clamping and stopping are ensured in the process of rotating the head or the eyeball of an operator.
Based on the present application, one skilled in the art will appreciate that one aspect described herein may be implemented independently of any other aspect, and that two or more of these aspects may be combined in various ways. For example, apparatus may be implemented and/or methods practiced using any number and aspects set forth herein. In addition, such apparatus may be implemented and/or such methods practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.
The various technical means described in the present invention are commercially available or will be apparent to those skilled in the art unless specifically explained.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description will explain the specific embodiments of the present invention with reference to the accompanying drawings. It is evident that the drawings in the following description are only examples of the invention, from which other drawings and other embodiments can be obtained by a person skilled in the art without inventive effort.
It should also be noted that the illustrations provided in the following embodiments merely illustrate the basic concepts of the application by way of illustration, and only the components related to the application are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.
In addition, in the following description, specific details are provided in order to provide a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects may be practiced without these specific details. The terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining a description of "a first," "a second," etc. may explicitly or implicitly include one or more such feature. In the description of the present invention, unless otherwise indicated, the meaning of "a plurality" is two or more.
As shown in fig. 1, the whole system is composed of two parts, one part is a remote driving end 100, and the remote driving end is installed on a remote driving platform; the other part is the vehicle-mounted terminal 200, which is mounted on a vehicle requiring remote driving.
The remote driving end 100 includes an eye tracking system 101, a camera pose calculation unit 102, and a video receiving and displaying device 103.
The eye tracking system 101 is a head-mounted device that is capable of tracking the direction of eye movement of the wearer. Also, it includes an Inertial Measurement Unit (IMU) containing an acceleration sensor and a gyroscope, the inertial measurement unit being used to track the movement of the wearer's head.
The camera pose calculation unit 102 calculates the rotation direction of the camera 202 in the horizontal plane (i.e. the rotation around the Z axis) in real time according to the output of the eye tracking system and in combination with the information such as the installation position of the camera on the vehicle created in advance, and finally converts the rotation direction into a camera pose control instruction, and sends the camera pose control instruction to the single-axis servo control system 201 of the vehicle-mounted terminal 200 through the 4G or 5G network.
The video receiving and displaying device 103 is used to receive real-time video data from the camera 202 at the vehicle-mounted end, and may be any device that has a function of receiving video from the network and displaying the video.
The vehicle-mounted terminal 200 includes a single-axis servo control system 201 and a camera 202 mounted in a suitable position in the front of the vehicle.
The single axis servo control system 201 is a high precision, high dynamic micro servo mechanism comprising a micro motor, an embedded controller and corresponding mechanical structure which can control the rotation and positioning of the camera attached thereto about a vertical axis as commanded.
The camera 202 is mounted on the single-axis servo control system 201 and integrally mounted at a proper position in the front of the vehicle. In this position, images of the front, the left front, and the right front of the vehicle can be acquired by rotation of the camera.
The data of the camera 202 is turned back to the video receiving and displaying device 103 of the remote driving terminal 100 in real time through the 4G/5G network.
In summary, the embodiment of the present invention as shown in fig. 2 achieves the following effects:
using the eye tracker 101 to acquire the attention point of an operator to the vehicle-mounted video in the driving process, and adjusting the camera to a desired position in real time;
the eye tracker 101 not only tracks the eyeball movement of the operator, but also tracks the head movement of the operator through a built-in IMU, so that the attention area and the position of the operator to the vehicle-mounted video can be accurately acquired;
the gesture of the camera 202 is adjusted in real time according to the requirement of an operator by using the servo control mechanism 201, so that the requirement of the operator on the vehicle-mounted video in the driving process can be met only by using one camera 202 and one path of high-definition video, and the remote driving of the vehicle is realized under the condition of the current 5G bandwidth. The application range of 5G remote driving is enlarged, so that the method can be applied to a real traffic environment.
When the remote driver intends to turn the vehicle to the left, naturally, the operator will turn the head to the left or turn the eyeballs in the hope of obtaining video data of the left side of the vehicle.
The eye tracking apparatus 101 acquires the eyeball horizontal movement distance of the operator or the rotation angle data of the head, and sends it to the camera pose calculation unit 102.
The camera pose calculation unit 102 converts the eye movement data and the head rotation data into pose data of the camera 202, and transmits them to the single-axis servo controller 201 at the vehicle-mounted end via a wireless network (4G/5G).
The single-axis servo controller 201 controls the motor to rotate according to the expected camera posture data, drives the camera 202 to rotate leftwards around the vertical axis to an expected position, and in the rotation process, the camera 202 always acquires video data and sends the video data back to the video receiving and displaying device 103 of the remote driving end through the network.
The whole process is completed at a frequency not lower than 50hz, and the instruction tracking response delay of the single-axis servo controller 201 is not greater than 5ms, so that smooth pictures and no clamping in the process of rotating the head or the eyeball of an operator are ensured.
It should be noted that the above embodiments can be freely combined as needed. The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.
Claims (10)
1. An auxiliary system for remote driving is characterized by being applied to the remote driving of a remote vehicle by a remote driver and comprising a first auxiliary unit and a display unit which are arranged at a remote driving end and a second auxiliary unit which is arranged at the remote vehicle end;
the first auxiliary unit is used for acquiring motion information of the remote driver in the remote driving, generating a gesture control instruction according to the motion information, and sending the gesture control instruction to the second auxiliary unit, wherein the motion information is used for representing body motion information of the remote driver on a driving attention point in the remote driving, and the driving attention point is a key point of the remote driver for performing driving reaction on the video data displayed by the display unit according to driving behaviors;
the second auxiliary unit is used for adjusting the acquisition angle of the camera unit according to the gesture control instruction to acquire new video data when receiving the gesture control instruction from the first auxiliary unit, and transmitting the new video data to the display unit so that the remote driver can make driving reaction based on the new video data displayed by the display unit.
2. The assistance system according to claim 1, wherein the movement information acquired by the first assistance unit comprises eye movement information and/or head movement information of the remote driver.
3. The assistance system according to claim 1 or 2, wherein the movement information acquired by the first assistance unit is derived from eye movement information of the remote driver tracked and acquired by an eye automatic tracking system;
or the head movement information of the remote driver is tracked and acquired by an inertial measurement unit arranged on the eye movement tracking system.
4. The auxiliary system according to claim 1, wherein,
the gesture control instruction generated by the first auxiliary unit is sent to the second auxiliary unit through a 4G/5G network; and/or
And the new video data acquired by the second auxiliary unit are transmitted to the display unit through a 4G/5G network.
5. The assist system according to claim 4, wherein the second assist unit is provided with 1 camera of the camera unit.
6. A remote-driving assisting driving method applied to the remote-driving assisting system according to any one of claims 1 to 5, the method comprising:
acquiring motion information of the remote driver in remote driving through the first auxiliary unit, generating a gesture control instruction according to the motion information, and sending the gesture control instruction to the second auxiliary unit, wherein the motion information is used for representing body motion information of the remote driver on a driving attention point in remote driving, and the driving attention point is a key point of a driving response of the remote driver on video data displayed by the display unit according to driving behaviors;
and when the second auxiliary unit receives the gesture control instruction, adjusting the acquisition angle of the camera unit according to the gesture control instruction to acquire new video data, and transmitting the new video data to the display unit so that the remote driver can make driving reaction based on the new video data displayed by the display unit.
7. The driving assist method according to claim 6, wherein the attitude control instruction generated by the first assist unit is transmitted to the second assist unit through a 4G/5G network.
8. The driving assist method according to any one of claims 6 or 7, characterized in that the new video data obtained by the second assist unit is transmitted to the display unit via a 4G/5G network.
9. The driving assist method according to claim 6, wherein the process of the driving assist method is completed at a frequency of not less than 50 hz.
10. The driving assist method as set forth in claim 6, wherein the instruction tracking response delay of the uniaxial servo controller is not more than 5ms.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111322167.6A CN114115246B (en) | 2021-11-09 | 2021-11-09 | Remote driving auxiliary system and method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111322167.6A CN114115246B (en) | 2021-11-09 | 2021-11-09 | Remote driving auxiliary system and method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114115246A CN114115246A (en) | 2022-03-01 |
| CN114115246B true CN114115246B (en) | 2024-01-23 |
Family
ID=80377668
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111322167.6A Active CN114115246B (en) | 2021-11-09 | 2021-11-09 | Remote driving auxiliary system and method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114115246B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20130058991A (en) * | 2011-11-28 | 2013-06-05 | 현대자동차주식회사 | Driving assistant system having a function of gesture recognition |
| EP3530521A1 (en) * | 2018-02-22 | 2019-08-28 | Continental Automotive GmbH | Driver assistance method and apparatus |
| CN110228484A (en) * | 2019-06-17 | 2019-09-13 | 福州视驰科技有限公司 | A kind of low time delay intelligent remote control loop driving function with auxiliary |
| CN110850711A (en) * | 2019-12-06 | 2020-02-28 | 中国科学院自动化研究所 | Auxiliary driving control system and method based on cloud |
| KR20200141428A (en) * | 2020-12-09 | 2020-12-18 | 현대모비스 주식회사 | Method for driver attention-based video output and vehicle control determine |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6714036B2 (en) * | 2018-04-25 | 2020-06-24 | 株式会社日立物流 | Management support system |
-
2021
- 2021-11-09 CN CN202111322167.6A patent/CN114115246B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20130058991A (en) * | 2011-11-28 | 2013-06-05 | 현대자동차주식회사 | Driving assistant system having a function of gesture recognition |
| EP3530521A1 (en) * | 2018-02-22 | 2019-08-28 | Continental Automotive GmbH | Driver assistance method and apparatus |
| CN110228484A (en) * | 2019-06-17 | 2019-09-13 | 福州视驰科技有限公司 | A kind of low time delay intelligent remote control loop driving function with auxiliary |
| CN110850711A (en) * | 2019-12-06 | 2020-02-28 | 中国科学院自动化研究所 | Auxiliary driving control system and method based on cloud |
| KR20200141428A (en) * | 2020-12-09 | 2020-12-18 | 현대모비스 주식회사 | Method for driver attention-based video output and vehicle control determine |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114115246A (en) | 2022-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109891875B (en) | Image transmission apparatus, method, system, and storage medium | |
| CN108139799B (en) | System and method for processing image data based on a region of interest (ROI) of a user | |
| CN106485736B (en) | Panoramic visual tracking method for unmanned aerial vehicle, unmanned aerial vehicle and control terminal | |
| CA2737451C (en) | Method and apparatus for displaying stereographic images of a region | |
| KR101544524B1 (en) | Display system for augmented reality in vehicle, and method for the same | |
| US9609290B2 (en) | Telepresence method and system for supporting out of range motion by aligning remote camera with user's head | |
| JPWO2019012817A1 (en) | Image processing apparatus, image processing method of image processing apparatus, and program | |
| CN108495048B (en) | Double-camera image acquisition equipment based on holder control | |
| BR112015010280B1 (en) | Image display system and method | |
| WO2020100664A1 (en) | Image processing device, image processing method, and program | |
| EP3152900B1 (en) | System and method for remote monitoring at least one observation area | |
| CN108696724B (en) | Live broadcast system capable of instantly obtaining high-definition photos | |
| EP3174290B1 (en) | Method for generating an enhanced video signal and providing the enhanced video signal to the driver of a vehicle that is moving at a certain speed, system for generating an enhanced video signal, program and computer program product | |
| CN114240769A (en) | Image processing method and device | |
| CN114877872A (en) | Unmanned aerial vehicle, operating system thereof, method, medium and equipment for generating map | |
| CN102200445A (en) | Real-time augmented reality device and method thereof | |
| JP2019062348A (en) | Display control apparatus, display control method, and video output apparatus | |
| KR101957896B1 (en) | integrated image and situation display system for remote control and method of displaying the same | |
| CN102591014B (en) | Panoramic vision observing system and work method thereof | |
| CN114115246B (en) | Remote driving auxiliary system and method thereof | |
| CN205356558U (en) | System is watched in remote control and make a video recording and hold and watch end thereof | |
| CN216927439U (en) | Crawler-type unmanned target vehicle remote control system | |
| US20220244726A1 (en) | Information processing apparatus, information processing method, and program | |
| CN115689920A (en) | Auxiliary correction method, device and correction system for HUD imaging | |
| EP2175661A1 (en) | Method and apparatus for producing a visual representation of a region |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |