Classifications

• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
  • B60W60/005—Handover processes
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
  • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
  • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
  • G05D1/0038—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
  • B60K35/10—Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
  • B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
  • B60K35/21—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor using visual output, e.g. blinking lights or matrix displays
  • B60K35/22—Display screens
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
  • B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
  • B60K35/28—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
  • B60K35/80—Arrangements for controlling instruments
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
  • B60W30/06—Automatic manoeuvring for parking
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
  • B60W50/08—Interaction between the driver and the control system
  • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B62—LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
  • B62D—MOTOR VEHICLES; TRAILERS
  • B62D15/00—Steering not otherwise provided for
  • B62D15/02—Steering position indicators ; Steering position determination; Steering aids
  • B62D15/027—Parking aids, e.g. instruction means
  • B62D15/0285—Parking performed automatically
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
  • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
  • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
  • G05D1/0044—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
  • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
  • G05D1/20—Control system inputs
  • G05D1/22—Command input arrangements
  • G05D1/221—Remote-control arrangements
  • G05D1/222—Remote-control arrangements operated by humans
  • G05D1/224—Output arrangements on the remote controller, e.g. displays, haptics or speakers
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
  • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
  • G05D1/20—Control system inputs
  • G05D1/22—Command input arrangements
  • G05D1/221—Remote-control arrangements
  • G05D1/225—Remote-control arrangements operated by off-board computers
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/1523—Matrix displays
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/16—Type of output information
  • B60K2360/171—Vehicle or relevant part thereof displayed
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/16—Type of output information
  • B60K2360/176—Camera images
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/55—Remote control arrangements
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/55—Remote control arrangements
  • B60K2360/56—Remote control arrangements using mobile devices
  • B60K2360/566—Mobile devices displaying vehicle information
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/55—Remote control arrangements
  • B60K2360/56—Remote control arrangements using mobile devices
  • B60K2360/573—Mobile devices controlling vehicle functions
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
  • B60W50/08—Interaction between the driver and the control system
  • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
  • B60W2050/143—Alarm means
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
  • B60W50/08—Interaction between the driver and the control system
  • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
  • B60W2050/146—Display means
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
  • B60W2420/40—Photo, light or radio wave sensitive means, e.g. infrared sensors
  • B60W2420/403—Image sensing, e.g. optical camera
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W2554/00—Input parameters relating to objects
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B62—LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
  • B62D—MOTOR VEHICLES; TRAILERS
  • B62D1/00—Steering controls, i.e. means for initiating a change of direction of the vehicle

Definitions

• the invention relates to a method for assisting a user in remotely controlling a motor vehicle by means of a remote control device, wherein, depending on sensor data provided by means of at least one environment sensor of the motor vehicle, an environmental image of at least part of the environment of the motor vehicle is provided from a certain perspective and the first environmental image is provided by means of the Remote control device is displayed.
• the invention also includes a
• Computer program product a remote control device and a driver assistance system for a motor vehicle.
• US 2016/0096549 A1 describes a method for remote control of a motor vehicle with a trailer.
• a user can drive the motor vehicle trailer arrangement remotely by means of a remote control.
• appropriate control elements in particular also as a virtual display of such control elements, can be provided by the remote control, for example to remotely operate the accelerator pedal, the brake pedal, a gear selector lever and so on.
• a video stream of a 360-degree all-round vision system of the motor vehicle can also be displayed to a user.
• DE 10 2009 041 587 A1 describes a method for assisting a driver in monitoring an autonomous parking process of a motor vehicle, the monitoring user also changing during the autonomous parking process can be outside the motor vehicle.
• image data about a surrounding area of the motor vehicle can be obtained by means of at least one camera of the motor vehicle and this or image data calculated therefrom can be transmitted to a remote control of the user for display. If several cameras are provided, the user can select one of the several cameras via the remote control and then the image data of the selected camera is transmitted to the user by means of the
• the views of the various cameras can also be automatically displayed to the user via the remote control in a predetermined time change.
• the driver is shown a bird's eye view of the area surrounding the motor vehicle by means of the remote control.
• the environmental video recorded by a motor vehicle camera can be displayed as a live stream to a remote user who remotely controls the motor vehicle.
• Remote control of a motor vehicle by means of a remote control device is provided as a function of at least one environment sensor of the motor vehicle
• Sensor data a first environment image of at least part of the environment of the motor vehicle from a certain perspective and the first
• the particular perspective is selected from a plurality of perspectives that can be selected as a perspective that is independent of a position of the at least one environment sensor and is dependent on a perspective that the user detects by means of the remote control device
• the perspectives that can be selected by the user by means of the remote control device, from which the user can be shown the corresponding surrounding image of the surroundings of the motor vehicle are not limited to the perspectives provided by the positions of the one or more surroundings sensors of the motor vehicle, but can be, for example in principle be freely selectable in three-dimensional space.
• the view or representation of the environment from the selected perspective can then be calculated by known methods from the sensor data provided by the at least one environment sensor. This advantageously provides the user with significantly more flexibility with regard to the choice of the perspective on the motor vehicle environment, which in particular
• both an activation of a function to be carried out automatically by the motor vehicle can be understood from a distance, and also manual remote-controlled driving of the motor vehicle
• the motor vehicle to be remotely controlled is preferably an automatically driving motor vehicle, in particular of the autonomy level two, three or four. Remote control of the motor vehicle preferably takes place through the
• the user when the motor vehicle is controlled remotely, the user does not have to be in a predetermined proximity to the motor vehicle, but can be at any distance from the motor vehicle. That also has to be
• Motor vehicle is not in the user's field of vision.
• the communication between the motor vehicle and the remote control device can in principle take place by any communication standard, preferably by one
• the 4G or LTE (Long Term Evolution) mobile communications standard in particular LTE advanced, is currently particularly suitable for this.
• LTE advanced Long Term Evolution mobile communications standard
• the highest data transfer speeds can currently be achieved, thereby minimizing latency.
• other, further improved mobile radio standards such as the planned 5G mobile radio standard or others for data transmission between the motor vehicle and the
• the remote control device can take any form. This can be designed, for example, as a mobile device, such as a Wi-Fi connection, or a Wi-Fi connection, or a Wi-Fi connection.
• a mobile device such as a Wi-Fi connection
• Smartphone or another mobile communication device or as a non-mobile communication device, such as a desktop computer
• the remote control device has at least one for displaying the at least one first environment image
• Display device or a display.
• the at least one environment sensor is preferably designed as at least one camera.
• other types of sensors for example LIDAR (Light Detection And Ranging) sensors and / or radar sensors and / or ultrasound sensors and any combination thereof, can also be used as an alternative or in addition for environment detection and for providing the sensor data.
• LIDAR Light Detection And Ranging
• a 3D all-round view of the motor vehicle is based on the acquired sensor data
• a 360-degree camera and / or a 360-degree LIDAR sensor can be installed on the roof of the motor vehicle in order to detect the surroundings around the motor vehicle at full angle.
• several environment sensors in particular several cameras, are used to provide the sensor data, which are arranged on the outside around the motor vehicle, for example a front camera on the vehicle front, a rear camera on the rear of the vehicle and, for example, a left mirror camera in the Area of the left outside mirror of the motor vehicle and a right mirror camera in the area of the right outside mirror of the motor vehicle.
• Each of these cameras is preferably designed as a wide-angle camera, in particular as a fisheye camera.
• a respective camera can be designed to provide a field of view with an opening angle of 180 degrees in the horizontal. Such an arrangement also allows the surroundings of the motor vehicle to be at a full angle around the motor vehicle capture and based on this a 3D all-round view of the motor vehicle
• a perspective can be assigned a viewpoint, in particular a virtual viewpoint from which the surroundings are to be viewed, as well as a direction of view. Furthermore, a perspective that is independent of a position of the at least one environment sensor is to be understood as a perspective, the associated viewpoint of which is also from a position of the at least one
• Environment sensor can be different.
• the user can also choose perspectives that do not match the perspectives from which the respective motor vehicle cameras record the surroundings.
• the recorded user input can be used to select the perspective
• a user can select or change a perspective without having to change the position and / or orientation of the motor vehicle itself by remote control.
• a certain perspective within the motor vehicle coordinate system can thus be selected or changed by the user input.
• a view window is determined on a defined projective surface and the first surrounding image as part of one from the sensor data
• the available 3D all-round view of the motor vehicle is calculated as a function of the particular view window on the defined projective area, so that the first surrounding image represents the part of the surroundings lying within the view window from the selected perspective.
• the sensor data provided can be projected onto the defined projective area from the point of view of the selected perspective, and the surrounding image can thus advantageously be provided as part of the total 3D all-round view that can be provided around the motor vehicle within the specific view window.
• the projective surface can represent, for example, a plane that extends perpendicular to the vertical axis of the vehicle and with respect to the
• Vehicle vertical axis is below the motor vehicle.
• Such a projective area is particularly suitable for displaying or representing the surroundings from a bird's eye view or in a top view of the motor vehicle.
• the projective surface is a curved surface.
• the projective surface can be hemispherical or bowl-shaped, for example.
• the projective surface can also be composed of a
• a virtual viewpoint associated with the particular perspective can be freely selectable on one or more predetermined lines or in three-dimensional space.
• the user can thus freely select the perspective and change it as desired, in particular also dynamically and continuously, through the user input recorded by the remote control device.
• innumerable, in particular infinitely many, different perspectives can be suitably chosen to suit the respective situation.
• a virtual viewpoint assigned to the specific perspective can be selected within the motor vehicle and also outside the motor vehicle.
• a perspective can be selected that a driver sitting in the motor vehicle would have of the surroundings, as well as any number of perspectives outside the motor vehicle that a viewer would have next to, behind, in front of or above the motor vehicle. The user is thus advantageously provided with the option of having the environmental information recorded by the motor vehicle displayed comprehensively.
• a motor vehicle representation of the motor vehicle Surrounding image at least part of a motor vehicle representation of the motor vehicle is shown from the selected perspective.
• This motor vehicle representation can, for example, be provided in the form of a 2D bitmap image or else as a three-dimensional computer-generated motor vehicle model. This significantly increases the degree of reality for the user. Above all, however, the user can directly relate the environment shown in the first environment image and objects located therein to the position of the motor vehicle and, for example, distances, Size relationships, the current direction of travel or the like can be recorded much better and this can further increase safety.
• this embodiment has the great advantage that the data to be transmitted from the motor vehicle to the remote control device can be reduced to a minimum, since only the image data of the surrounding image is transmitted to the remote control device and, for example, no sensor data or ambient data that are outside the specific one Viewports as described above. This enables a significantly faster data transmission from the motor vehicle to the remote control device. This allows latency times to be reduced to a minimum. In this case it is advantageous if the
• Remote control device recorded user input for selecting the perspective as, for example, corresponding selection information is transmitted from the remote control device to the motor vehicle. This can then correspondingly calculate the environmental image on the basis of the acquired sensor data in accordance with the selected perspective and transmit it to the remote control device. This process is repeated in particular continuously, so that the user is shown repeatedly updated first surroundings images from the selected perspective in the form of a live video on the remote control device.
• the sensor data are transmitted from the motor vehicle to the remote control device and the first environmental image is calculated from the sensor data by the remote control device.
• this makes it possible, for example, to simultaneously compute and display environmental images from different perspectives from the sensor data transmitted to the remote control device.
• these different environmental images that is to say from the different perspectives, calculated in the motor vehicle on the basis of the sensor data and transmitted to the remote control device, but the computing capacity in motor vehicles is typically very limited, in particular due to that in the motor vehicle available space is limited. Such restrictions apply to the
• Remote control device not, for example, as a powerful home computer can be trained. So you can, for example, next to the first
• Remote control device and / or a bandwidth and / or one
• Data transmission speed of the data transmission between the motor vehicle and the remote control device determines whether the first environment image is calculated from the sensor data by the motor vehicle or by the remote control device.
• the two variants described above can be implemented depending on the situation, depending on the current connection quality of the communication link between the motor vehicle and the remote control device. Since at least the motor vehicle is moving, the communication link can be better or worse from time to time. If, for example, currently high data transmission speeds are available, all of the raw sensor data recorded by the motor vehicle can be transmitted to the remote control device, and the first of these sensor data
• Environment image and preferably other environment images are calculated from further perspectives and displayed to the user. If, on the other hand, the communication link between the motor vehicle and the remote control device is currently poor or only very low data transmission speeds are available, the motor vehicle can instead use the first environment image based on the acquired one
• the first environment image is calculated from the sensor data by the motor vehicle and transmitted to the remote control device and if the bandwidth of the Communication link and / or the data transmission speed of the communication link between the motor vehicle and remote control device does not fall below the predetermined limit, the sensor data to the
• Remote control device are transmitted and the first environment image from the
• Sensor data is calculated by the remote control device.
• At least one second environment image is provided from a second perspective and at the same time as the first by means of the remote control device
• the perspective assigned to the second environment image can also be selected or changed by the user or can also be predefined. For example, in addition to the first environment image with the perspective selected by the user, a top view image from a bird's eye view of the motor vehicle and the motor vehicle environment can be displayed as the second environment image, and / or a further second environment image from a driver's perspective or the like. There are no limits to the design options.
• the user can also set the perspectives associated with the environment images displayed in each case beforehand by a user using a configuration menu. The user can then, for example, change the set perspective at any time for each individual surrounding image.
• remote control of the motor vehicle is independent of a distance between the remote control device and the
• Motor vehicle is carried out or is feasible.
• the user does not have to be in a predetermined proximity to the motor vehicle, but can be in a user position that is independent of the position of the motor vehicle for remote control of the motor vehicle. Especially in connection with such a remote control, it is particularly important to provide the user with comprehensive
• Motor vehicle is, so that the motor vehicle is always within sight of the User.
• Such a comprehensive provision of information regarding the surroundings of the motor vehicle can advantageously be provided by the invention, which increases security, particularly with regard to such remote control systems
• the invention also relates to a computer program product comprising commands that are executed by a processor of an electronic
• Control device cause the latter to carry out a method according to the invention or one of its embodiments.
• the invention also relates to a remote control device for assisting a user in remotely controlling a motor vehicle, the remote control device being designed to display an environmental image of at least part of the environment of the vehicle
• the remote control device is designed to detect a user input made by the user and, depending on the recorded user input, to select the particular perspective from a plurality of selectable perspectives as a perspective independent of a position of the at least one environment sensor.
• the invention also includes a driver assistance system for a motor vehicle to assist a user in remotely controlling the motor vehicle by means of a
• the driver assistance system is designed to receive from the remote control device a selection information that is dependent on a user input acquired by means of the remote control device and that is made by the user and, depending on the received selection information, the specific perspective from several selectable perspectives as one from a position of the at least one Environment sensor independent perspective to select.
• a motor vehicle with a driver assistance system according to the invention or one of its configurations should also be regarded as belonging to the invention.
• the invention also includes a system comprising one
• Remote control device and a motor vehicle the system being designed to carry out the method according to the invention or one of its embodiments.
• Fig. 1 is a schematic representation of a remote control system with a
• FIG. 2 shows a schematic illustration of the provision of a 3D all-round view of the motor vehicle from the sensor data provided by the motor vehicle and the calculation of an environmental image as a function of a selected perspective according to an exemplary embodiment of the invention
• Fig. 3 is a schematic representation of the remote control device according to one
• Fig. 1 shows a schematic representation of a remote control system 1 with a
• the motor vehicle 2 has a driver assistance system 4 which comprises at least one environment sensor, in this example four cameras 5, 6, 7, 8, namely for example a front camera 5, a rear camera 6, a left mirror camera 7 and a right mirror camera 8. Each of these cameras 5, 6, 7, 8 can be used as
• an environment can thus be 9 of the motor vehicle 2 completely, that is, at a full angle around the motor vehicle 2.
• an environmental image of at least part of the environment 9 of the motor vehicle 2 can be calculated from a certain perspective and displayed on the remote control device 3, as will be described in more detail with reference to FIG. 3.
• each of these perspectives is now advantageously freely selectable by a user.
• This freely selectable perspective is therefore advantageously independent of the positions of the respective cameras 5, 6, 7, 8.
• a perspective can be selected that does not match a perspective of the respective cameras 5, 6, 7, 8.
• the remote control device 3 can have corresponding input means. The remote control device 3 can use this to record the perspective selection made by the user.
• both the motor vehicle 2 and the remote control device 3 are designed to calculate the environmental image on the basis of the acquired sensor data.
• the driver assistance system 4 has a control device 10, which is designed to calculate the image of the surroundings based on the predefined perspective on the basis of the sensor data.
• the remote control device 3 also has a corresponding control device 11, which is also designed to operate on the basis of a given perspective
• the user input acquired by the remote control device 3 for defining a desired perspective can be transmitted as a corresponding selection information item 12 to the motor vehicle 2, in particular to the control device 10 of the motor vehicle 2.
• the control device 10 calculates the environmental image 13 from the image based on the image data provided by the cameras 5, 6, 7, 8 in a respective time step
• control device 10 can repeatedly calculate the environment image 13 based on the sensor data from the same perspective chosen by the user until a new one Selection information 12 is received from the remote control device 3 specifying a different perspective. Then the control device 10 of the
• Motor vehicle 2 calculate the environment image 13 depending on the newly specified perspective and transmit it to the remote control device 3.
• this calculation of the environmental image 13 can also be carried out by the remote control device 3 itself.
• the remote control device 3 itself.
• Selection information 12 is not transmitted from the remote control device 3 to the motor vehicle 2. Instead, the control device 10 of the motor vehicle 2 transmits the image data captured by the cameras 5, 6, 7, 8, in particular as raw sensor data 14, to the remote control device 3 in a respective time step. In addition, other environmental and / or motor vehicle data, such as a Area map, the current motor vehicle position, the current driving speed, or other data are transmitted from the motor vehicle to the remote control device. This then calculates and displays the environmental image 13 as a function of the perspective chosen by the user.
• the determination of whether the surroundings image 13 is to be calculated by the motor vehicle 2 or by the remote control device 3 is made depending on how good the communication link between the motor vehicle 2 and the remote control device 3 is currently. For example, if a very high bandwidth is currently available for the data transmission and / or if currently high data transmission speeds are possible, it is preferred that the motor vehicle 2, in particular the control device 10, transmits the raw camera data 14 to the remote control device 3 and the calculation of the environmental image 13 is carried out by the remote control device 3.
• the remote control device 3 is not limited in its computing power due to the installation space, as is the case for the control device 10 of the motor vehicle 2, for example. This advantageously makes it possible, on the basis of the data transmitted to the remote control device 3
• the motor vehicle 2 itself calculates the environmental image 13 is calculated as a function of the perspective specified by the user and the calculated environment image 13 is transmitted to the remote control device 3, which then displays it.
• This has the advantage that significantly smaller amounts of data are generated for the transmission of the environmental image 13 from the motor vehicle 2 to the remote control device 3 than when the entire raw camera data 14 is transmitted. This enables data transmission to be carried out quickly even at low data transmission speeds.
• remote control commands 15 entered by the user via the remote control device 3 can also be carried out by the remote control device 3
• Remote control device 3 to the motor vehicle 2, in particular to the
• Control device 10 are transmitted, which implements these commands.
• Remote control commands are, in particular, drive commands to motor vehicle 2, in particular for controlling steering of motor vehicle 2 and / or for controlling acceleration and / or braking of motor vehicle 2.
• a user thus receives remote control of motor vehicle 2 by means of
• Remote control device 3 additionally provides comprehensive environmental information about the environment 9 of the motor vehicle 2, which increases security in the
• FIG. 2 shows a schematic illustration of the calculation of the environmental image based on the sensor data provided as part of a 3D all-round view of the motor vehicle 2.
• FIG. 2 shows a projective surface, which in this example is designed as a spherical or shell-shaped projective surface 15 , together with a three-dimensional model of the motor vehicle 2 as an example of a motor vehicle representation 17.
• the images of the surroundings 9 captured by the motor vehicle cameras 5, 6, 7, 8 can be combined and referenced to the
• View window 18 defines which depends on the position of the selected perspective 19, which is represented here by the virtual view point assigned to perspective 19.
• the perspective 19 is selected such that a rear view of the motor vehicle 2, which is represented here by the motor vehicle model or in general a motor vehicle representation 17, is provided.
• an environment image can be provided, which represents the first part of the environment 20 lying within the view window from the selected perspective 19.
• FIG. 3 shows a schematic illustration of the remote control device 3 in detail according to an exemplary embodiment of the invention.
• the remote control device 3 can have operating elements 21, of which only one steering wheel is shown here by way of example and by means of which a user can drive the motor vehicle 2 by remote control.
• Such a steering wheel or the operating elements 21 in general do not have to be provided objectively, but can also be represented as a virtual representation on a display device and / or a touchscreen.
• the remote control device 3 also provides input options which are also not explicitly shown here and by means of which one
• the user can choose the perspective from which the surroundings 9 of the motor vehicle 2 are to be represented by means of the remote control device 3.
• the remote control device 3 has at least one display device.
• a respective display device 22 shows a respective environmental image 13a, 13b, 13c.
• These environmental images 13a, 13b, 13c can alternatively also be on a common one
• Display device 22 for example side by side, one below the other or the like. At least one of the displayed surrounding images 13a, 13b, 13c is displayed from a perspective 19 specified by the user.
• the other two environment images can either also be displayed from a perspective that the user has actively selected and / or from one
• a first environment image 13a shows a part 20 of the environment 9 of the motor vehicle 2, which is in turn represented here as a corresponding motor vehicle representation 17, from a perspective 19, the associated viewpoint of which is outside the motor vehicle 2 or
• a second environment image 13b shows the part 20 of the environment 9 from a perspective, the associated viewpoint of which lies within the motor vehicle 2 or the motor vehicle model 17.
• the part 20 lying in the view window 18 is the
• the focus of perspective 19 is directly above motor vehicle 2 or motor vehicle display 17.
• the two surroundings images 13b, 13c can be displayed to the user as predetermined perspectives. This is particularly advantageous because the bird's eye view according to the third environment image 13c to a user
• a perspective according to the second environment image 13b is particularly advantageous since this simulates the perspective of a driver seated in the motor vehicle 2, which is a particularly familiar type of representation according to the remote-control user.

Landscapes

• Engineering & Computer Science (AREA)
• Mechanical Engineering (AREA)
• Transportation (AREA)
• Automation & Control Theory (AREA)
• Remote Sensing (AREA)
• Radar, Positioning & Navigation (AREA)
• Combustion & Propulsion (AREA)
• Chemical & Material Sciences (AREA)
• Aviation & Aerospace Engineering (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• Human Computer Interaction (AREA)
• Selective Calling Equipment (AREA)
• Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
• Traffic Control Systems (AREA)
• Steering Controls (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=69055946

Family Applications (1)

Country Status (6)

Families Citing this family (5)

Family Cites Families (25)

• 2018
  • 2018-12-06 DE DE102018131104.4A patent/DE102018131104A1/de active Pending
• 2019
  • 2019-11-21 CN CN201980090358.9A patent/CN113348125B/zh active Active
  • 2019-11-21 EP EP19828567.8A patent/EP3891038A1/de active Pending
  • 2019-11-21 WO PCT/EP2019/082007 patent/WO2020114785A1/de unknown
  • 2019-11-21 JP JP2021532038A patent/JP7317967B2/ja active Active
  • 2019-11-21 US US17/311,185 patent/US12007765B2/en active Active

Also Published As

Similar Documents

Legal Events