CN114202988A - Forklift driving simulation method, device, system and system control device - Google Patents

Abstract

The embodiment of the application provides a forklift driving simulation method, device and system and a system control device, and relates to the field of engineering equipment simulation training equipment. According to the method, firstly, a virtual scene is generated according to operation data collected by each sensor, gyroscope parameters of the virtual reality glasses and pictures collected by a camera of the virtual reality glasses, and the virtual scene is sent to the virtual reality glasses for display; and generating a seat control instruction according to the virtual scene, sending the seat control instruction to the seat control device, and controlling the seat to move by the seat control device based on the seat control instruction. The seat control device can be controlled to give corresponding feedback according to operation results in different virtual scenes, so that the operation experience is more real.

Description

Forklift driving simulation method, device, system and system control device

Technical Field

The application relates to the field of engineering equipment simulation training equipment, in particular to a forklift driving simulation method, device, system and system control device.

Background

Forklifts are vehicles used for the handling and transportation of goods and are widely used in the logistics industry. However, the forklift is a special vehicle for characteristic equipment with certain dangerousness, and in order to avoid safety accidents caused by the fact that students with insufficient experience perform real vehicle operation in the training process, it is necessary to develop a simulation training driving system capable of simulating the training experience of the forklift.

In the conventional forklift simulation system, a display displays a simulated picture to give feedback to the operation of a user. However, the display has limited contents, and thus cannot display the surrounding environment in all directions, so that the user's view is limited. In addition, the real feedback of the driving seat under a specific operating environment cannot be simulated, so that the simulation effect is limited.

Disclosure of Invention

The application aims to provide a forklift driving simulation method, a forklift driving simulation device, a forklift driving simulation system and a forklift driving simulation system control device, which can generate a virtual scene of panoramic simulation of an environment where a forklift driver is located, and can control a seat control device to give corresponding feedback according to operation results in different virtual scenes, so that the operation experience is more real.

In order to achieve the above purpose, the technical solutions adopted in the embodiments of the present application are as follows:

in a first aspect, an embodiment of the present application provides a method for simulating driving of a forklift, which is applied to a system control device in a system for simulating driving of a forklift, where the system for simulating driving of a forklift includes: the system comprises a system control device, a seat control device, virtual reality glasses, an operating platform, a three-degree-of-freedom seat and an operating assembly;

the method comprises the following steps:

acquiring a picture acquired by a camera of the virtual reality glasses and a gyroscope parameter of the virtual reality glasses, wherein the gyroscope parameter is used for indicating a moving angle of the virtual reality glasses;

acquiring operation data acquired by various sensors arranged on the operation table, the seat and the operation assembly;

generating a virtual scene according to the operation data acquired by the sensors, the gyroscope parameters of the virtual reality glasses and the pictures acquired by the cameras of the virtual reality glasses, and sending the virtual scene to the virtual reality glasses for display;

and generating a seat control instruction according to the virtual scene, sending the seat control instruction to the seat control device, and controlling the seat to move by the seat control device based on the seat control instruction.

In an optional implementation manner, before generating a virtual scene according to the operation data acquired by each of the sensors, the gyroscope parameters of the virtual reality glasses, and the pictures acquired by the cameras of the virtual reality glasses, the method further includes:

receiving a virtual reality glasses reset request;

adjusting the display picture of the virtual reality glasses to a preset initial visual angle according to the reset request to obtain a reset picture;

and sending the reset picture to the virtual reality glasses for display.

In an optional implementation manner, the generating a virtual scene according to the operation data collected by each of the sensors, the gyroscope parameters of the virtual reality glasses, and the pictures collected by the camera of the virtual reality glasses includes:

determining the current visual angle of the virtual reality glasses according to the gyroscope parameters of the virtual reality glasses;

determining a virtual picture corresponding to the current visual angle;

determining hand information in the picture according to the picture acquired by the camera of the virtual reality glasses;

and superposing the operation data acquired by the sensor and the hand information in the picture on the virtual picture corresponding to the current visual angle to obtain the virtual scene.

In an alternative embodiment, the operational data collected by the sensor includes:

steering direction and angle information of the steering wheel are acquired by a steering wheel steering shaft sensor on the operating platform;

the switch sensor on the operating platform acquires the on-off information of the switch;

the position change information of the first operating rod is acquired by a first operating rod sensor on the operating platform;

the position change information of the second operating rod is acquired by a second operating rod sensor on the operating platform;

the position change information of the third operating rod is acquired by a third operating rod sensor on the operating platform;

the position change information of the first deflector rod is acquired by a first deflector rod sensor on the operating platform;

the position change information of the second deflector rod is acquired by a second deflector rod sensor on the operating platform;

the opening information of the accelerator is acquired by an accelerator sensor in the operating assembly;

the on-off information of the first brake pad is acquired by a first brake pad sensor in the operating assembly;

the opening information of the second brake pad is acquired by a second brake pad sensor in the operating assembly;

and the safety belt sensor on the seat acquires the opening and closing information of the safety belt.

In an optional implementation manner, the determining, according to the picture collected by the camera of the virtual reality glasses, hand information in the picture includes:

performing edge segmentation on a picture acquired by a camera of the virtual reality glasses, and distinguishing a hand from a picture background to obtain a hand sample;

extracting key points of the hand sample to obtain finger quantity information, finger length information and hand direction information of the hand;

and determining the hand information in the picture according to the finger number information, the finger length information and the hand direction information.

In an optional embodiment, before acquiring the data collected by each sensor, the method further includes:

receiving a forklift starting signal sent by a switch sensor on the operating platform;

determining whether the forklift is in a safe state, and if the forklift is not in the safe state currently, sending a warning picture and a warning audio to the virtual reality glasses;

and if the forklift is in a safe state at present, generating a starting picture and a starting audio, and sending the starting picture and the starting audio to the virtual reality glasses.

In an optional embodiment, the determining whether the forklift is in a safe state includes:

and if a safety belt buckling signal sent by a safety belt sensor on the seat is received, determining that the forklift is in a safe state.

In a second aspect, the present application provides a forklift truck simulated driving device, where the device includes:

the virtual reality glasses comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring pictures acquired by a camera of the virtual reality glasses and gyroscope parameters of the virtual reality glasses, and the gyroscope parameters are used for indicating the movement angle of the virtual reality glasses;

the acquisition module is also used for acquiring operation data acquired by various sensors arranged on the operation table, the seat and the operation assembly;

the processing module is used for generating a virtual scene according to the operation data acquired by the sensors, the gyroscope parameters of the virtual reality glasses and the pictures acquired by the cameras of the virtual reality glasses, and sending the virtual scene to the virtual reality glasses for display;

the processing module is further specifically configured to generate a seat control instruction according to the virtual scene, send the seat control instruction to a seat control device, and control the seat to move by the seat control device based on the seat control instruction.

In a third aspect, an embodiment of the present application provides a system control apparatus, including:

a processor, a memory and a bus, wherein the memory stores machine readable instructions executable by the processor, when the system control device runs, the processor and the memory are communicated through the bus, and the processor executes the machine readable instructions to execute the steps of the forklift simulated driving method according to any one of the preceding embodiments.

In a fourth aspect, an embodiment of the present application provides a forklift simulated driving system, where the system includes: the system control device, the seat control device, the virtual reality glasses, the virtual reality calibration equipment, the operation table, the three-degree-of-freedom seat, the display and the operation assembly are described in the previous embodiment;

the system also comprises sensors arranged on the operating platform, the three-degree-of-freedom seat and the operating assembly;

the system control device is used for providing a virtual picture for the virtual reality glasses according to operation data, provided by the seat control device, executed on the operation table by a user;

the virtual reality glasses are used for displaying the virtual picture;

the seat control device is used for controlling the seat to move according to a seat control command sent by the system control device.

The beneficial effects of the embodiment of the application include:

by adopting the forklift driving simulation method, device, system and system control device provided by the application, firstly, the virtual scene can be generated according to the operation data collected by each sensor, the gyroscope parameters of the virtual reality glasses and the pictures collected by the virtual reality camera, and the pictures at the current visual angle in the virtual scene are displayed in the virtual reality glasses. Compared with a single display, the driver obtains a wider visual angle from the virtual reality glasses and can observe the surrounding environment in an all-around manner. When the operation such as backing a car is carried out, the situation behind the forklift in the virtual scene is convenient to observe back, and the safety of simulated driving is ensured. Secondly, still to the environmental aspect of virtual scene in this application, if the fork truck that the driver drove takes place to turn on one's side, collide etc. in the virtual scene, system control device all can send seat control command, and seat control device makes real feedback of jolting based on this instruction control seat, makes the driver feel the state change of fork truck with one's mind, has built lifelike driving environment for the driver.

In addition, after wearing the virtual reality glasses, the driver sees the picture as the virtual scene of driving the forklift, and in order to enable the driver to operate the forklift smoothly, firstly, the picture display visual angle of the virtual reality glasses is adjusted to be consistent with the actual forklift through the virtual reality glasses calibrating device in the application, and the position one-to-one correspondence between the operating parts in the virtual picture and the operating parts on the actual forklift is ensured. Secondly, the pictures shot by the virtual glasses camera are subjected to hand tracking processing, corresponding hand positions are superposed in the virtual pictures according to the processed hand information, and virtual pictures with hands are generated, so that a driver can quickly find corresponding operation parts.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.

Fig. 1 is a schematic structural diagram of a forklift simulated driving system provided in an embodiment of the present application;

fig. 2 is a schematic structural diagram of a forklift simulated driving system provided in the embodiment of the present application;

FIG. 3 is a flow chart illustrating steps of a method for simulating driving of a forklift according to an embodiment of the present disclosure;

FIG. 4 is a flowchart illustrating another step of a method for simulating driving of a forklift according to an embodiment of the present disclosure;

FIG. 5 is a flowchart illustrating another step of a method for simulating driving of a forklift according to an embodiment of the present disclosure;

FIG. 6 is a flowchart illustrating another step of a method for simulating driving of a forklift according to an embodiment of the present disclosure;

FIG. 7 is a flowchart illustrating another step of a method for simulating driving of a forklift according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of a forklift simulated driving device provided in the embodiment of the present application;

fig. 9 is a schematic structural diagram of a system control device according to an embodiment of the present application.

Icon: 101-a system control device; 102-a seat control device; 103-virtual reality glasses; 104-an operation table; 105-three degree of freedom seat; 106-a display; 107-switch; 108-a first toggle lever; 109-a second deflector rod; 110-a steering wheel; 111-a first lever; 112-a second lever; 113-a third lever; 114-a seat belt; 115-throttle; 116-a first brake pad; 117-second brake pad; 118-virtual reality glasses calibration equipment; 1181-virtual reality glasses groove; 1182-calibration key; 119-a throttle sensor; 120-a first brake pad sensor; 121-a second brake pad sensor; 122-seat belt sensor; 123-switch sensor; 124-a first lever sensor; 125-a second lever sensor; 126-third lever sensor; 127-a first stick sensor; 128-a second stick sensor; 129-steering wheel steering shaft sensor; 10-a forklift simulated driving device; 1001-acquisition module; 1002-a processing module; 2001-a processor; 2002-memory.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.

Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

In the description of the present application, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. are used for indicating the orientation or positional relationship based on the orientation or positional relationship shown in the drawings or the orientation or positional relationship which the present invention product is usually put into use, it is only for convenience of describing the present application and simplifying the description, but it is not intended to indicate or imply that the referred device or element must have a specific orientation, be constructed in a specific orientation and be operated, and thus, should not be construed as limiting the present application.

Furthermore, the appearances of the terms "first," "second," and the like, if any, are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.

It should be noted that the features of the embodiments of the present application may be combined with each other without conflict.

Fork truck is the vehicle that is used for the transport of transport and the transportation of goods in the commodity circulation trade of wide application. However, the forklift has certain dangerousness, and as drivers mostly perform zero-base operation during training, safety accidents can be caused, and great potential safety hazards are caused. Therefore, it is necessary to develop a forklift simulation training driving system.

Currently, forklift steering systems display simulated pictures through a display to provide feedback on user's operations. However, the display has limited contents, and thus cannot display the surrounding environment in all directions, so that the user's view is limited. In addition, the real feedback of the driving seat under a specific operating environment cannot be simulated, so that the simulation effect is limited.

Based on the above, the applicant has proposed a method, an apparatus, a system and a system control apparatus for simulating driving of a forklift truck, which can display a virtual scene through virtual reality glasses worn by a driver, so that the driver can obtain a wider viewing angle from the virtual reality glasses than a single display, and can observe surrounding environment in an all-around manner, thereby ensuring safety of the simulated driving. The real bumping feedback can be further carried out on the environment condition of the virtual scene by controlling the seat, so that the driver can feel the state change of the forklift truck by cutting the body, and a vivid driving environment is created for the driver.

The following explains a forklift simulated driving method, device, system and system control device provided in the embodiments of the present application with reference to a plurality of specific application examples.

Fig. 1 is a schematic structural diagram of a forklift simulated driving system applied to a forklift simulated driving method according to an embodiment of the present application, and includes a system control device 101, a seat control device 102, virtual reality glasses 103, an operation console 104, a three-degree-of-freedom seat 105, and an operation component. As shown in fig. 2, the system further includes: a virtual reality calibration device 118, and a display 106, and further comprising: and sensors disposed on the console 104, the three-degree-of-freedom seat 105, and the operation unit.

When the forklift simulation driving system is in standby, a driver wears the virtual reality glasses 103 and confirms whether the internal picture of the forklift in the virtual picture displayed by the virtual reality glasses 103 after wearing and the internal picture of the forklift before wearing the virtual reality glasses 103 are at the same visual angle. If so, the operation can be continued, and if not, the virtual reality glasses 103 need to be removed, placed in the virtual reality glasses groove 1181 of the virtual reality glasses calibration device 118, and the nearby calibration key 1182 is pressed. Upon pressing, the virtual reality glasses calibration device 118 will issue a reset request to the system control apparatus 101. When the system control device 101 receives the reset request sent by the virtual reality glasses calibration device 118, the reset picture corresponding to the preset initial viewing angle is sent to the virtual reality glasses 103, and when the driver takes up the virtual reality glasses 103 again to wear, the system control device 101 recalculates the movement angle of the virtual reality glasses 103 and sends the picture of the movement angle to the virtual reality glasses 103 to display.

Next, the driver turns on the switch 107 and fastens the seat belt 114, the switch sensor and the seat belt sensor transmit the acquired switch-on signal and the acquired seat belt fastening signal to the seat control device 102, and the seat control device 102 aggregates the signals and transmits the aggregated signals to the system control device 101 in real time through a preset communication protocol. The system control device 101 activates the display 106 and the virtual reality glasses 103 connected thereto, and transmits the activation screen to the virtual reality glasses 103 to be displayed. If the driver does not fasten the seat belt 114 before turning on the switch or looses the seat belt 114 during driving, the system control device 101 sends the calling warning screen and the warning audio to the virtual reality glasses 103 to be displayed, and stops the operation of other components in the system.

Then, the driver operates the forklift, the corresponding operation data to be collected by the sensors provided on the console 104, the three-degree-of-freedom seat 105, and the operation components is transmitted to the system control device 101 in real time via the seat control device 102, and the system control device 101 generates a dynamic image in which the corresponding operation member moves in the virtual image based on the operation data after receiving the operation data. Meanwhile, the system control device 101 obtains hand information of the driver by analyzing the picture acquired by the camera on the virtual reality glasses 103, and generates a virtual hand. In addition, a static virtual picture corresponding to the current viewing angle of the virtual reality glasses 103 is obtained through a gyroscope in the virtual reality glasses 103. And superposing the dynamic picture moved by the operating component and the virtual hand on the static virtual picture to obtain a virtual picture of the current visual angle, and sending the virtual picture to the virtual reality glasses 103 for display.

When a driver operates the forklift to drive in a virtual scene, the state of the forklift may change, for example, the forklift driven by the driver has accidents such as rollover and forward tilting in the virtual scene, or shakes due to bad road conditions. At this time, the system control device 101 calculates, according to the situations in the virtual scene, a physical expression of the current three-degree-of-freedom seat 105 of the forklift through the physical engine, sends a corresponding seat control instruction to the seat control device 102, and after receiving the corresponding control instruction, the seat control device 102 controls the electric cylinder below the three-degree-of-freedom seat 105 to perform telescopic motion, simulates a degree of bumping of the physical expression, and simultaneously sends corresponding accident audio play to the virtual reality glasses 103.

The system control device that this application embodiment provided is applied to the system control device among the fork truck simulation driving system, and this system structure schematic diagram is shown as figure 1, includes that fork truck simulation driving system includes: a system control device 101, a seat control device 102, virtual reality glasses 103, an operation table 104, a three-degree-of-freedom seat 105, and operation components.

As shown in fig. 2, the above-mentioned operation components include: the brake system comprises a switch 107, a first shift lever 108, a second shift lever 109, a steering wheel 110, a first operating lever 111, a second operating lever 112, a third operating lever 113, a safety belt 114, an accelerator 115, a first brake pad 116 and a second brake pad 117, wherein sensors for collecting operation information of a driver can be arranged on all the operating components. Optionally, the system for simulating driving of a forklift further includes: a display 106 and a virtual reality glasses calibration device 118, the virtual reality glasses calibration device 118 comprising: a virtual reality glasses recess 1181 and a calibration key 1182.

Alternatively, with continued reference to fig. 2, the sensors disposed on the console, the three-degree-of-freedom seat 105, and the operating assembly may include: a throttle sensor 119, a first brake pad sensor 120, a second brake pad sensor 121, a seatbelt sensor 122, a switch sensor 123, a first lever sensor 124, a second lever sensor 125, a third lever sensor 126, a first stick sensor 127, a second stick sensor 128, and a steering wheel sensor 129.

As shown in fig. 3, the method for simulating the driving of the forklift provided by the embodiment of the application includes the following steps:

s101, obtaining pictures collected by a camera of the virtual reality glasses and gyroscope parameters of the virtual reality glasses, wherein the gyroscope parameters are used for indicating the moving angle of the virtual reality glasses.

Wherein, the gyroscope parameter of virtual reality glasses can be gathered by the built-in gyroscope in the virtual reality glasses and obtain, the gyroscope can be the device of compriseing top part and power part, the gyroscope body is in the level with the silk thread hoist messenger rotation axis in the gyroscope device, the motion state of record virtual reality glasses, when the motion of virtual reality glasses, the gyroscope body has produced the motion of advancing time in the device, at this moment, according to the roll angle of gyroscope, pitch angle and yaw angle, can calculate the current position who obtains the virtual reality glasses.

The current position of the virtual reality glasses can be interpreted as an angle of movement of the virtual reality glasses on the spatial coordinate axis with respect to a preset initial position.

And after receiving the gyroscope parameters sent by the virtual reality glasses, the system control device sends the virtual pictures corresponding to the visual angles to the virtual reality glasses according to the current positions of the virtual reality glasses.

In addition, the camera of the virtual reality glasses can be arranged in the center of the glasses, and the current picture of the real situation inside the forklift which can be seen by the visual angle of the virtual reality glasses is collected and transmitted to the system control device in real time.

And S102, acquiring operation data acquired by various sensors arranged on the operation table, the seat and the operation assembly.

The sensors arranged on the operating platform, the seat and the operating assembly convert the operations into electric signals according to the operations performed by the driver, transmit the electric signals to the seat control device in real time, and send the electric signals to the system control device through a preset communication protocol after being collected by the seat control device. For example, when a driver operates a first operating lever disposed on an operating console, the position of the operating lever may be shifted, and a sensor may collect and convert the shift into a corresponding electrical signal and transmit the electrical signal.

And S103, generating a virtual scene according to the operation data acquired by the sensors, the gyroscope parameters of the virtual reality glasses and the pictures acquired by the camera of the virtual reality glasses, and sending the virtual scene to the virtual reality glasses for displaying.

Optionally, the system control device prestores a plurality of three-dimensional virtual scenes containing different road conditions, such as ramps, curved roads, multi-obstacle roads and the like, three-dimensional images of various positions inside the forklift, and three-dimensional images of external forks and gantries of the forklift which can be seen from the perspective of a driver inside the forklift, and specific three-dimensional image contents can be set according to training needs.

After the system control device receives the operation data collected by each real-time sensor, the gyroscope parameters of the real-time virtual reality glasses and the pictures collected by the camera of the real-time virtual reality glasses, firstly, the road condition picture of the current virtual scene, namely the background picture of the forklift work is obtained according to the gyroscope parameters of the virtual reality glasses, then, the three-dimensional pictures inside and outside the forklift at the current visual angle corresponding to the moving angle of the virtual reality glasses are obtained, and the three-dimensional pictures inside and outside the forklift are combined with the background picture, so that the static three-dimensional picture which can be seen by the current visual angle of a driver is obtained.

Then, the information of the operation and the self movement of the driver is obtained from the operation data collected by each sensor and the picture collected by the camera of the virtual reality glasses, the information is generated into a corresponding three-dimensional dynamic image on the basis of the static three-dimensional picture to obtain a virtual scene, and the virtual scene is sent to the virtual reality glasses worn by the driver and displayed in front of the driver.

And S104, generating a seat control command according to the virtual scene, sending the seat control command to a seat control device, and controlling the seat to move by the seat control device based on the seat control command.

When the driver operates the console, the seat and the operating components to drive in the virtual scene, the vehicle state may change, for example, the driver may turn over or lean forward a forklift in the virtual scene, or shake due to bad road conditions. At the moment, the system control device can calculate the physical expression of the current forklift seat through the physical engine according to the situations in the virtual scene, send a corresponding seat control instruction to the seat control device, and after receiving the corresponding control instruction, the seat control instruction can control the electric cylinder below the seat to perform telescopic motion, so that the bumping degree of the physical expression is simulated. For example, when the forklift in the virtual scene has a forward tilting accident due to overload, the physical engine calculates the current tilt angle and the physical state change of the corresponding seat at the tilt speed in real time, and controls the seat to make the corresponding change in real time.

In this embodiment, first, according to the present application, a virtual scene can be generated according to operation data collected by each sensor, a gyroscope parameter of the virtual reality glasses, and a picture collected by the virtual reality camera, and the picture at the current viewing angle in the virtual scene is displayed in the virtual reality glasses. Compared with a single display, the driver obtains a wider visual angle from the virtual reality glasses and can observe the surrounding environment in an all-around manner. When the operation such as backing a car is carried out, the situation behind the forklift in the virtual scene is convenient to observe back, and the safety of simulated driving is ensured. Secondly, still to the environmental aspect of virtual scene in this application, if the fork truck that the driver drove takes place to turn on one's side, collide etc. in the virtual scene, system control device all can send seat control command, and seat control device makes real feedback of jolting based on this instruction control seat, makes the driver feel the state change of fork truck with one's mind, has built lifelike driving environment for the driver.

Optionally, as shown in fig. 4, before the step S103 generates a virtual scene according to the operation data collected by each sensor, the gyroscope parameter of the virtual reality glasses, and the picture collected by the camera of the virtual reality glasses, the method may further include the following steps:

s201, receiving a virtual reality glasses resetting request.

Optionally, after the virtual reality glasses are started, the driver wears the virtual reality glasses and confirms whether the internal picture of the forklift is at the same visual angle in the internal picture of the forklift seen before wearing the virtual reality glasses and the virtual picture displayed by wearing the back virtual reality glasses. If so, the operation can be continued, if not, the glasses need to be taken down, placed in the virtual reality glasses groove of the virtual reality glasses calibration equipment, and the calibration key beside the glasses is pressed. After being pressed, the virtual reality glasses calibration equipment sends a reset request to the system control device.

S202, according to the reset request, the display picture of the virtual reality glasses is adjusted to be a preset initial visual angle, and a reset picture is obtained.

It should be noted that the system control device also prestores a virtual frame of a fixed viewing angle corresponding to the virtual reality glasses calibration device groove, which is referred to as a reset frame corresponding to a preset initial viewing angle.

And S203, sending the reset picture to the virtual reality glasses for displaying.

And after the system control device receives a reset request sent by the virtual reality glasses calibration equipment, sending a reset picture corresponding to the preset initial visual angle to the virtual reality glasses, and simultaneously resetting the parameters of the gyroscope in the virtual reality glasses to zero. When the driver picks up the virtual reality glasses again and wears the virtual reality glasses, the system control device recalculates the moving angle of the virtual reality glasses and sends the picture of the moving angle to the virtual reality glasses for display.

In this embodiment, through virtual reality calibration equipment and system control device, can adjust the visual angle of virtual reality glasses to initial visual angle, can avoid the overlength of live time, the accumulative error of gyroscope makes the visual angle of virtual reality glasses and the current inconsistent condition of real fork truck visual angle, promotes the driver and uses experience.

Alternatively, as shown in fig. 5, the generation of the virtual scene in step S103 according to the operation data collected by each sensor, the gyroscope parameters of the virtual reality glasses, and the picture collected by the camera of the virtual reality glasses may be implemented by steps S301 to S304 described below.

S301, determining the current visual angle of the virtual reality glasses according to the gyroscope parameters of the virtual reality glasses.

As described in the above embodiments, the method according to the gyroscope parameters in the virtual reality glasses includes: and the roll angle, the pitch angle and the yaw angle can obtain the current position of the virtual reality glasses on a space coordinate axis. After the current position of the virtual reality glasses is obtained, the system control device can determine the visual angle of the virtual reality glasses at the current position according to the corresponding relation between the internal preset position and the visual angle.

S302, determining a virtual picture corresponding to the current visual angle.

It can be understood that, the three-dimensional pictures of the inside and the outside of the forklift in the virtual scene and the background pictures containing various road conditions are stored in the system control device in advance. Therefore, according to the determined view angle of the virtual reality glasses in the above steps, the static virtual picture corresponding to the current view angle can be determined.

And S303, determining hand information in the picture according to the picture collected by the camera of the virtual reality glasses.

It can be understood that after the virtual reality glasses are worn by the driver, the seen pictures are the pictures of the virtual reality glasses, the audio played by the virtual reality glasses can be heard and is isolated from the external environment, and at the moment, the driver can not accurately find and operate the real operation part on the real forklift according to the virtual pictures in front of the driver.

For this reason, first, the size of the forklift in the virtual scene and the size of the forklift seen by the driver before wearing the virtual reality glasses are the same under the same viewing angle. Secondly, through the virtual reality glasses calibration equipment in the embodiment, when the driver changes the visual angle, the fork truck seen corresponds to the real fork truck in the virtual scene. For example, if the driver lowers his head to look at the steering wheel in the virtual scene and the angle of view is not changed when the driver takes off the virtual glasses, the driver can also look at the steering wheel at the same angle of view as in the virtual scene.

Finally, the visual angle of the camera on the virtual reality glasses is the same as the current visual angle of the virtual reality glasses, that is, the pictures collected by the camera correspond to the virtual pictures seen by the driver in the virtual reality glasses and the positions of the operating parts in the forklift one by one. In this case, the hand position in the captured image can be adaptively matched to the same position in the virtual image.

Therefore, the system control device calculates and analyzes the pictures of each frame in the video stream collected by the virtual reality glasses in sequence to obtain the hand information.

S304, the operation data collected by the sensor and the hand information in the picture are superposed on the virtual picture corresponding to the current visual angle to obtain a virtual scene.

When the hand information is obtained, the system control device generates a virtual hand on the virtual screen based on the hand information, and it is understood that the virtual hand and the actual hand are synchronized in real time.

In addition, the system control device generates a dynamic screen in which the corresponding operation member moves in the virtual screen even after receiving the operation data collected by the sensors provided in the console, the seat, and the operation unit. For example, when the driver holds the steering wheel and rotates the steering wheel clockwise, the system control device may generate a dynamic virtual screen that is rotated clockwise by the virtual hand holding the steering wheel, superimpose the virtual screen on the static virtual screen to obtain a complete virtual scene screen, and transmit the complete virtual scene screen to the virtual reality glasses for display.

In this embodiment, according to the picture that the camera was gathered to and the driver's operational data that the sensor gathered, obtained the virtual picture of each operating element of driver operation, can make the driver wear can be quick after the virtual reality glasses find the part that wants the operation, operate fork truck smoothly, promote driver's operation experience.

Alternatively, as shown in fig. 6, in the step S303, the hand information in the picture is determined according to the picture captured by the camera of the virtual reality glasses, and the following steps S401 to S403 may be implemented:

s401, performing edge segmentation on a picture acquired by a camera of the virtual reality glasses, and distinguishing a hand from a picture background to obtain a hand sample.

Optionally, the image collected by the virtual reality glasses camera may be divided into multiple frames, and each frame is subjected to edge segmentation processing.

The edge segmentation can be understood as separating the hand in each frame from other background information irrelevant to the hand, and only keeping a hand sample. The specific edge segmentation algorithm may be edge segmentation based on skin color, or edge segmentation based on semantic segmentation, which is not limited herein.

S402, extracting key points of the hand sample to obtain finger number information, finger length information and hand direction information of the hand.

After a hand sample is obtained, feature extraction is carried out on the hand sample through a hand recognition algorithm to obtain key point information of the hand, and the key points mark information such as positions of fingertips, positions of palms, positions of finger joints and the like. The key point information is combined to obtain finger number information, finger length information and hand direction information.

Optionally, the camera of the virtual reality glasses may be a binocular camera, and depth information of the hand may be obtained for calibration by the system control device, so that the obtained key point information may further include three-dimensional information of the hand, such as a palm thickness, a fingertip altitude, and the like.

And S403, determining the hand information in the picture according to the finger number information, the finger length information and the hand direction information.

After calculating and obtaining the finger number information, the finger length information and the hand direction information, the system control device reconstructs a virtual hand model according to the information in a three-dimensional coordinate system. It can be understood that the motion and the position of the virtual hand correspond to the motion and the position of the driver's hand in each frame of the picture acquired by the virtual reality glasses camera one by one.

In this embodiment, the action picture of the virtual hand in the virtual picture is constructed according to the real hand action of the driver by analyzing the picture collected by the virtual reality glasses camera. The driver can accurately recognize the position of the hand in the visual field in the operation process, so that the forklift simulation driving system can be operated smoothly.

Optionally, the operational data collected by the sensor includes: the steering wheel comprises a steering wheel steering shaft sensor on an operating platform, a switch sensor on the operating platform, a first operating rod sensor on the operating platform, a second operating rod sensor on the operating platform, a third operating rod sensor on the operating platform, a first deflector rod sensor on the operating platform, and a second deflector rod sensor on the operating platform.

And opening information of the accelerator collected by an accelerator sensor in the operation assembly.

The opening degree information of the second brake pad is acquired by the second brake pad sensor in the operation assembly, and the opening information of the safety belt is acquired by the safety belt sensor on the seat.

The steering wheel comprises a steering shaft sensor, a switch sensor, a first operating rod sensor, a second operating rod sensor, a third operating rod sensor, a first driving lever sensor, a second driving lever sensor, an accelerator sensor, a first brake pad sensor, a second brake pad sensor and a safety belt sensor, wherein the steering shaft sensor, the switch sensor, the first operating rod sensor, the second operating rod sensor, the third operating rod sensor, the first driving lever sensor, the second driving lever sensor, the accelerator sensor, the first brake pad sensor, the second brake pad sensor and the safety belt sensor are respectively electrically connected with a seat control device, and collected actions of all parts operated by a driver are converted into corresponding electric signals which are sent to the seat control device. The seat control device sends the information to the system control device in real time according to a preset communication protocol.

After receiving the electric signals corresponding to the operation data acquired by the sensors, the system control device generates dynamic virtual pictures corresponding to the movement of the operation part according to different preset corresponding rules of the electric signals of the sensors and the operation data, superposes the dynamic virtual pictures with the static virtual pictures and the dynamic pictures of the hands in the embodiment, and sends the dynamic virtual pictures to the virtual reality glasses worn by the driver for display. That is, from the perspective of the driver, the operation of the operating component by the driver is reflected in the virtual reality glasses worn by the driver in real time, for example, the driver performs the throttle-down operation, the throttle component in the virtual scene is moved down, the forklift moves forward, and at the same time, the driver hears the sound effect of the engine starting.

In the embodiment, the electric signals transmitted by the sensors on the operating platform, the seat and the operating assembly are analyzed, so that various operations of a driver in a virtual scene of a driving forklift are accurately acquired and transmitted in real time, and the system control device feeds back the operations to the pictures of the virtual reality glasses, so that the pictures seen by the driver are synchronous with the operations made, and the driver is provided with more real somatosensory feedback.

Optionally, as shown in fig. 7, before acquiring the data collected by each sensor in step S103, the following steps may be further included:

and S501, receiving a forklift starting signal sent by a switch sensor on the operating platform.

The driver can insert the switch through the button, perhaps the mode starting switch that the button pressed, the switch starts the back, and the switch sensor of inside setting can be the signal of telecommunication that the switch started with driver starting switch's action conversion to with this signal of telecommunication, fork truck starting signal promptly, send to system control device.

S502, whether the forklift is in a safe state or not is determined, and if the forklift is not in the safe state at present, a warning picture and a warning audio are sent to the virtual reality glasses.

After receiving the forklift starting signal, the system control device comprehensively judges whether the current forklift is in a safe state capable of being started or not according to other sensor signals. And after comprehensive analysis is carried out by the system control device, the current state of the forklift is determined not to meet the standard of the safety state, and the warning picture and the warning audio are called from the picture and the audio which are stored in advance and are sent to the visual field area of the virtual reality glasses for display.

And S503, if the forklift is in a safe state at present, generating a starting picture and a starting audio, and sending the starting picture and the starting audio to the virtual reality glasses.

On the contrary, if the system control device determines that the current forklift is in a safe state capable of being started after receiving the forklift starting signal and comprehensively analyzing other sensor signals, the system control device calls the starting picture and the starting audio from the picture and the audio stored in advance, and sends the starting picture and the starting audio to the visual field area of the virtual reality glasses for displaying. In addition, the system control device can send a seat control instruction to the seat control device, the seat control device controls the seat to make a starting state of up-and-down bump based on the instruction, and then the system control device can call a standby sound effect and send the standby sound effect to the virtual reality glasses for playing.

In the embodiment, the safety of the driver in the driving process is ensured by judging the safety state of the forklift after the driver starts the forklift.

Optionally, in step S502, determining whether the forklift is in a safe state includes: and if a safety belt buckling signal sent by a safety belt sensor on the seat is received, determining that the forklift is in a safe state.

The condition for determining whether the forklift is in the safe state may be determining whether the seat belt of the driver is engaged, but is not limited thereto.

After the forklift simulation driving system is started, if a driver buckles the safety belt before the operation of the starting switch, the safety belt sensor arranged at the buckle position of the safety belt or in the buckle box sends a safety belt buckling signal to the system control device, and after the system control device receives the signal, the safety state mark position 1 of the forklift can be used for indicating that the forklift is in the safety state at present. It can be understood that if the driver opens the safety belt in the operation process, the safety belt sensor also sends a safety belt opening signal to the system control device, and the system control device receives the signal and then marks the safety state of the forklift at the position 0, and at this time, the system control device calls a warning picture and a warning audio from a picture and an audio which are stored in advance and sends the warning picture and the warning audio to the visual field area of the virtual reality glasses for display.

In this embodiment, detect whether the driver has buckled the safety belt through the safety belt sensor, guaranteed the security that the driver drove. In addition, the safety belt sensor can also detect the action of opening the safety belt of the driver, and avoid the situation that the driver forgets to buckle during the driving.

Referring to fig. 8, the present embodiment further provides a device 10 for simulating steering of a forklift, including:

an obtaining module 1001 configured to obtain a picture collected by a camera of the virtual reality glasses and a gyroscope parameter of the virtual reality glasses, where the gyroscope parameter is used to indicate a movement angle of the virtual reality glasses.

The obtaining module 1001 is further configured to obtain operation data collected by each sensor disposed on the console, the seat, and the operating assembly.

The processing module 1002 is configured to generate a virtual scene according to operation data acquired by each sensor, the gyroscope parameters of the virtual reality glasses, and a picture acquired by a camera of the virtual reality glasses, and send the virtual scene to the virtual reality glasses for display.

The processing module 1002 is further specifically configured to generate a seat control instruction according to the virtual scene, send the seat control instruction to the seat control device, and control the seat to move by the seat control device based on the seat control instruction.

Optionally, the processing module 1002 is further specifically configured to receive a virtual reality glasses reset request, adjust a display image of the virtual reality glasses to a preset initial viewing angle according to the reset request, obtain a reset image, and send the reset image to the virtual reality glasses for display.

Optionally, the processing module 1002 is further specifically configured to determine a current viewing angle of the virtual reality glasses according to a gyroscope parameter of the virtual reality glasses, determine a virtual picture corresponding to the current viewing angle, determine hand information in the picture according to the picture acquired by a camera of the virtual reality glasses, and superimpose the operation data acquired by the sensor and the hand information in the picture on the virtual picture corresponding to the current viewing angle to obtain the virtual scene.

Optionally, the obtaining module 1002 is further configured to obtain steering direction and angle information of the steering wheel collected by a steering shaft sensor of the steering wheel on the console, on/off information of the switch collected by a switch sensor on the console, position change information of the first operating lever collected by a first operating lever sensor on the console, position change information of the second operating lever collected by a second operating lever sensor on the console, position change information of the third operating lever collected by a third operating lever sensor on the console, position change information of the first shift lever collected by a first shift lever sensor on the console, position change information of the second shift lever collected by a second shift lever sensor on the console, and opening degree information of the throttle collected by a throttle sensor in the operating assembly, the safety belt control device comprises a seat belt, a first brake pad sensor, a second brake pad sensor, an operation assembly and a safety belt sensor, wherein the first brake pad sensor in the operation assembly acquires the on-off information of the first brake pad, the second brake pad sensor in the operation assembly acquires the opening degree information of the second brake pad, and the safety belt sensor on the seat belt acquires the on-off information of the safety belt.

Optionally, the processing module 1002 is further specifically configured to perform edge segmentation on a picture acquired by a camera of the virtual reality glasses, and distinguish a hand from a picture background to obtain a hand sample. And extracting key points of the hand sample to obtain finger quantity information, finger length information and hand direction information of the hand. And determining the hand information in the picture according to the finger number information, the finger length information and the hand direction information.

Optionally, the processing module 1002 is further specifically configured to receive a forklift start signal sent by a switch sensor on the console. And determining whether the forklift is in a safe state, and if the forklift is not in the safe state at present, sending a warning picture and a warning audio to the virtual reality glasses. And if the forklift is in a safe state at present, generating a starting picture and a starting audio, and sending the starting picture and the starting audio to the virtual reality glasses.

Optionally, the processing module 1002 is further configured to determine that the forklift is in a safe state if a seat belt fastening signal sent by a seat belt sensor on the seat is received.

Optionally, an embodiment of the present application further provides a system control apparatus, as shown in fig. 9, the system control apparatus includes: a processor 2001, a memory 2002 and a bus, wherein the memory 2002 stores machine-readable instructions executable by the processor 2001, when the system control device operates, the processor 2001 communicates with the memory 2002 through the bus, and the processor 2001 executes the machine-readable instructions to execute the steps of the forklift simulated driving method in the foregoing embodiment.

The memory 2002, processor 2001, and bus elements are electrically coupled to each other, directly or indirectly, to enable data transfer or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The simulated driving device of the forklift includes at least one software functional module which can be stored in the memory 2002 in the form of software or firmware (firmware) or solidified in an Operating System (OS) of the processing device. The processor 2001 is used to execute executable modules stored in the memory 2002, such as software functional modules and computer programs included in the forklift simulated driving apparatus.

The Memory 2002 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. The forklift simulated driving method is applied to a system control device in a forklift simulated driving system, and the forklift simulated driving system comprises the following steps: the system comprises a system control device, a seat control device, virtual reality glasses, an operating platform, a three-degree-of-freedom seat and an operating assembly;

the method comprises the following steps:

acquiring a picture acquired by a camera of the virtual reality glasses and a gyroscope parameter of the virtual reality glasses, wherein the gyroscope parameter is used for indicating a moving angle of the virtual reality glasses;

acquiring operation data acquired by various sensors arranged on the operation table, the seat and the operation assembly;

generating a virtual scene according to the operation data acquired by the sensors, the gyroscope parameters of the virtual reality glasses and the pictures acquired by the cameras of the virtual reality glasses, and sending the virtual scene to the virtual reality glasses for display;

and generating a seat control instruction according to the virtual scene, sending the seat control instruction to the seat control device, and controlling the seat to move by the seat control device based on the seat control instruction.

2. The method for simulating the driving of a forklift truck according to claim 1, wherein before generating a virtual scene according to the operation data collected by each sensor, the gyroscope parameters of the virtual reality glasses, and the pictures collected by the cameras of the virtual reality glasses, the method further comprises:

receiving a virtual reality glasses reset request;

adjusting the display picture of the virtual reality glasses to a preset initial visual angle according to the reset request to obtain a reset picture;

and sending the reset picture to the virtual reality glasses for display.

3. The method for simulating the driving of a forklift truck according to claim 1, wherein the generating a virtual scene according to the operation data collected by each sensor, the gyroscope parameters of the virtual reality glasses, and the pictures collected by the cameras of the virtual reality glasses comprises:

determining the current visual angle of the virtual reality glasses according to the gyroscope parameters of the virtual reality glasses;

determining a virtual picture corresponding to the current visual angle;

determining hand information in the picture according to the picture acquired by the camera of the virtual reality glasses;

and superposing the operation data acquired by the sensor and the hand information in the picture on the virtual picture corresponding to the current visual angle to obtain the virtual scene.

4. The method of claim 3, wherein the operational data collected by the sensors comprises:

steering direction and angle information of the steering wheel are acquired by a steering wheel steering shaft sensor on the operating platform;

the switch sensor on the operating platform acquires the on-off information of the switch;

the position change information of the first operating rod is acquired by a first operating rod sensor on the operating platform;

the position change information of the second operating rod is acquired by a second operating rod sensor on the operating platform;

the position change information of the third operating rod is acquired by a third operating rod sensor on the operating platform;

the position change information of the first deflector rod is acquired by a first deflector rod sensor on the operating platform;

the position change information of the second deflector rod is acquired by a second deflector rod sensor on the operating platform;

the opening information of the accelerator is acquired by an accelerator sensor in the operating assembly;

the on-off information of the first brake pad is acquired by a first brake pad sensor in the operating assembly;

the opening information of the second brake pad is acquired by a second brake pad sensor in the operating assembly;

and the safety belt sensor on the seat acquires the opening and closing information of the safety belt.

5. The method for simulating the driving of the forklift according to claim 3, wherein the determining hand information in the picture according to the picture collected by the camera of the virtual reality glasses comprises:

performing edge segmentation on a picture acquired by a camera of the virtual reality glasses, and distinguishing a hand from a picture background to obtain a hand sample;

extracting key points of the hand sample to obtain finger quantity information, finger length information and hand direction information of the hand;

and determining the hand information in the picture according to the finger number information, the finger length information and the hand direction information.

6. The method for simulating the driving of a forklift truck according to claim 1, wherein before acquiring the data collected by each sensor, the method further comprises:

receiving a forklift starting signal sent by a switch sensor on the operating platform;

determining whether the forklift is in a safe state, and if the forklift is not in the safe state currently, sending a warning picture and a warning audio to the virtual reality glasses;

and if the forklift is in a safe state at present, generating a starting picture and a starting audio, and sending the starting picture and the starting audio to the virtual reality glasses.

7. The method of claim 6, wherein the determining whether the truck is in a safe state comprises:

and if a safety belt buckling signal sent by a safety belt sensor on the seat is received, determining that the forklift is in a safe state.

8. A simulated steering apparatus for a forklift, the apparatus comprising:

the virtual reality glasses comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring pictures acquired by a camera of the virtual reality glasses and gyroscope parameters of the virtual reality glasses, and the gyroscope parameters are used for indicating the movement angle of the virtual reality glasses;

the acquisition module is also used for acquiring operation data acquired by various sensors arranged on the operation table, the seat and the operation assembly;

the processing module is used for generating a virtual scene according to the operation data acquired by the sensors, the gyroscope parameters of the virtual reality glasses and the pictures acquired by the cameras of the virtual reality glasses, and sending the virtual scene to the virtual reality glasses for display;

the processing module is further specifically configured to generate a seat control instruction according to the virtual scene, send the seat control instruction to a seat control device, and control the seat to move by the seat control device based on the seat control instruction.

9. A system control apparatus, characterized in that the system control apparatus comprises:

a processor, a memory and a bus, the memory storing machine readable instructions executable by the processor, the processor and the memory communicating via the bus when the system control device is operating, the processor executing the machine readable instructions to perform the steps of the method of simulating driving of a forklift truck according to any one of claims 1 to 7.

10. A forklift simulated driving system, the system comprising: the system control apparatus, seat control apparatus, virtual reality glasses, virtual reality calibration device, console, three-degree-of-freedom seat, display, and operational assembly of claim 9;

the system also comprises sensors arranged on the operating platform, the three-degree-of-freedom seat and the operating assembly;

the system control device is used for providing a virtual picture for the virtual reality glasses according to operation data, provided by the seat control device, executed on the operation table by a user;

the virtual reality glasses are used for displaying the virtual picture;

the seat control device is used for controlling the seat to move according to a seat control command sent by the system control device.

Images