WO2022195988A1 - 遠隔操作支援サーバおよび遠隔操作支援システム - Google Patents
遠隔操作支援サーバおよび遠隔操作支援システム Download PDFInfo
- Publication number
- WO2022195988A1 WO2022195988A1 PCT/JP2021/045780 JP2021045780W WO2022195988A1 WO 2022195988 A1 WO2022195988 A1 WO 2022195988A1 JP 2021045780 W JP2021045780 W JP 2021045780W WO 2022195988 A1 WO2022195988 A1 WO 2022195988A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- remote
- real space
- space area
- remote operation
- operation support
- Prior art date
Links
- 238000003384 imaging method Methods 0.000 claims abstract description 74
- 230000008859 change Effects 0.000 claims abstract description 38
- 238000006073 displacement reaction Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims description 65
- 230000008602 contraction Effects 0.000 claims description 9
- 230000009467 reduction Effects 0.000 claims description 8
- 238000012806 monitoring device Methods 0.000 claims description 6
- 230000002093 peripheral effect Effects 0.000 claims description 4
- 230000003340 mental effect Effects 0.000 abstract 1
- 230000006870 function Effects 0.000 description 59
- 230000007246 mechanism Effects 0.000 description 18
- 238000004891 communication Methods 0.000 description 16
- 230000003287 optical effect Effects 0.000 description 11
- 206010034719 Personality change Diseases 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 5
- 238000012790 confirmation Methods 0.000 description 5
- 238000004091 panning Methods 0.000 description 4
- 239000010720 hydraulic oil Substances 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000034 method Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 239000002689 soil Substances 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 208000019901 Anxiety disease Diseases 0.000 description 1
- 230000036506 anxiety Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000010079 rubber tapping Methods 0.000 description 1
Images
Classifications
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/20—Drives; Control devices
- E02F9/2025—Particular purposes of control systems not otherwise provided for
- E02F9/205—Remotely operated machines, e.g. unmanned vehicles
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/26—Indicating devices
- E02F9/261—Surveying the work-site to be treated
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F3/00—Dredgers; Soil-shifting machines
- E02F3/04—Dredgers; Soil-shifting machines mechanically-driven
- E02F3/28—Dredgers; Soil-shifting machines mechanically-driven with digging tools mounted on a dipper- or bucket-arm, i.e. there is either one arm or a pair of arms, e.g. dippers, buckets
- E02F3/36—Component parts
- E02F3/42—Drives for dippers, buckets, dipper-arms or bucket-arms
- E02F3/43—Control of dipper or bucket position; Control of sequence of drive operations
- E02F3/435—Control of dipper or bucket position; Control of sequence of drive operations for dipper-arms, backhoes or the like
Definitions
- the present invention relates to technology for informing workers of the state of a working machine.
- the blind spot area from the driver's seat of the work machine (hydraulic excavator) is calculated based on information on the posture and movement of the work machine. , which determines the visibility level for an obstacle based on the blind spot area and the positional relationship between the obstacle and the working machine, and corrects the warning level according to the visibility level (for example, Patent Document 2). reference).
- JP 2019-156533 A Japanese Patent No. 6581139
- the present invention provides a system and the like that can reduce the psychological burden on an operator who remotely operates a working machine when changing the image range acquired by the actual imaging device mounted on the working machine. intended to
- the remote operation support server of the present invention is A server for supporting remote control of a work machine using a remote control device, A captured image representing the environment of the working machine is obtained through an actual imaging device mounted on the working machine, and a working environment image corresponding to the captured image is displayed on a remote image output device constituting the remote control device.
- a first support processing element The first support processing element recognizes the future change of the real space area reflected in the work environment image displayed on the remote image output device, and a remote output interface constituting the remote control device, a second support processing element for outputting a notice representing a future change in the real space area; It has
- the advance notice representing the future change of the real space area reflected in the working environment image displayed on the remote image output device is sent to the remote output interface constituting the remote operation device. output.
- the space occupation mode of the real space area reflected in the work environment image can change. Then, it is possible for the operator, who has come into contact with the notice, to grasp in advance the future change mode of the real space area reflected in the work environment image. Therefore, immediately after that, according to the notice, when the space occupation mode of the real space area reflected in the work environment image displayed on the remote image output device changes, the operator's psychological burden is reduced. reduction.
- FIG. 2 is an explanatory diagram regarding functions of the remote operation support system according to the first embodiment; Explanatory drawing about a work environment image. Explanatory drawing about the 1st displacement mode of a virtual sound source. Explanatory drawing about the 2nd displacement mode of a virtual sound source.
- FIG. 7 is an explanatory diagram regarding functions of a remote operation support system according to the second embodiment;
- the remote operation support system shown in FIG. 1 includes a remote operation support server 10 , a remote control device 20 , and a working machine 40 remotely controllable through the remote control device 20 . Only one of remote operation device 20 and work machine 14 may be a component of the remote operation support system.
- Remote operation support server 10, remote operation device 20, and work machine 40 are configured to be able to communicate with each other through a network.
- the mutual communication network between remote operation support server 10 and remote operation device 20 and the mutual communication network between remote operation support server 10 and work machine 40 may be the same or different.
- the remote operation support server 10 includes a database 102 , a first support processing element 121 and a second support processing element 122 .
- the database 102 stores and holds captured image data and the like.
- the database 102 may be configured by a database server separate from the remote operation support server 10 .
- Each support processing element is composed of an arithmetic processing unit (single-core processor or multi-core processor or a processor core that constitutes it), reads necessary data and software from a storage device such as memory, and processes the data into the software. Accordingly, arithmetic processing, which will be described later, is executed.
- the remote control device 20 includes a remote control device 200 , a remote input interface 210 and a remote output interface 220 .
- the remote control device 200 is configured by an arithmetic processing device (single-core processor or multi-core processor or a processor core constituting this), reads necessary data and software from a storage device such as a memory, and uses the data as a target for the software. Execute the arithmetic processing accordingly.
- the remote input interface 210 includes a remote control mechanism 211.
- Remote output interface 220 includes remote image output device 221 , remote audio output device 222 , and remote wireless communication device 224 .
- the remote control mechanism 211 includes a traveling operating device, a turning operating device, a boom operating device, an arm operating device, and a bucket operating device.
- Each operating device has an operating lever that receives a rotating operation.
- An operating lever (running lever) of the operating device for running is operated to move the lower running body 410 of the work machine 40 .
- the travel lever may also serve as a travel pedal.
- a traction pedal may be provided that is fixed to the base or lower end of the traction lever.
- a control lever (swing lever) of the swing operation device is operated to move a hydraulic swing motor that constitutes the swing mechanism 430 of the work machine 40 .
- An operating lever (boom lever) of the boom operating device is operated to move the boom cylinder 442 of the work machine 40 .
- An operating lever (arm lever) of the arm operating device is operated to move the arm cylinder 444 of the working machine 40 .
- An operating lever (bucket lever) of the bucket operating device is operated to move the bucket cylinder 446 of
- each control lever that constitutes the remote control mechanism 211 is arranged around the seat St on which the operator sits.
- the seat St is in the form of a high-back chair with armrests, a low-back chair without a headrest, or a chair without a backrest. may be
- a pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St.
- One operating lever may serve as a plurality of operating levers.
- the left operation lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 functions as an arm lever when operated in the longitudinal direction, and when operated in the lateral direction. function as a pivot lever.
- the right operation lever 2112 provided in front of the right frame of the seat St shown in FIG. It may function as a bucket lever in some cases.
- the lever pattern may be arbitrarily changed by an operator's operation instruction.
- the remote image output device 221 includes a central remote image output device 2210 having substantially rectangular screens arranged in front, diagonally forward left, and diagonally forward right of the sheet St. It consists of a left remote image output device 2211 and a right remote image output device 2212 .
- the shapes and sizes of the respective screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211 and the right remote image output device 2212 may be the same or different.
- the left remote image is tilted such that the screen of the central remote image output device 2210 and the screen of the left remote image output device 2211 form an inclination angle ⁇ 1 (for example, 120° ⁇ 1 ⁇ 150°).
- the right edge of output device 2211 is adjacent to the left edge of central remote image output device 2210 .
- the right remote image is tilted such that the screen of the central remote image output device 2210 and the screen of the right remote image output device 2212 form an inclination angle ⁇ 2 (eg, 120° ⁇ 2 ⁇ 150°).
- the left edge of output device 2212 is adjacent to the right edge of central remote image output device 2210 .
- the tilt angles ⁇ 1 and ⁇ 2 may be the same or different.
- the respective screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or may be inclined with respect to the vertical direction.
- At least one remote image output device among the central remote image output device 2210, the left remote image output device 2211 and the right remote image output device 2212 may be composed of a plurality of divided remote image output devices.
- the central remote image output device 2210 may comprise a pair of vertically adjacent remote image output devices having substantially rectangular screens.
- the remote sound output device 222 is composed of one or more speakers, for example, as shown in FIG. , a left sound output device 2221 and a right sound output device 2222 .
- the specifications of the central sound output device 2220, the left sound output device 2221, and the right sound output device 2222 may be the same or different.
- the remote sound output device 222 implements a virtual sound source V ss which is a positionally displaceable sound around the seat St.
- the working machine 40 includes a real machine control device 400 , a real machine input interface 41 , a real machine output interface 42 , and a working mechanism 440 .
- the actual machine control device 400 includes an image processing device 30 .
- the image processing device 30 comprises a state detection element 31 , an image prediction element 32 and an image compression element 34 .
- Each of the components of actual machine control device 400 and image processing device 30 is configured by an arithmetic processing device (single-core processor or multi-core processor, or a processor core constituting this), and reads necessary data and software from a storage device such as a memory. It reads the data and executes arithmetic processing on the data according to the software.
- the work machine 40 is, for example, a crawler excavator (construction machine), and as shown in FIG. and an upper revolving body 420 .
- a cab 424 (driver's cab) is provided on the front left side of the upper swing body 420 .
- a work mechanism 440 is provided in the front central portion of the upper swing body 420 .
- the real machine input interface 41 includes a real machine operation mechanism 411 , a real machine imaging device 412 , and a real machine peripheral monitoring device 414 .
- the actual machine operating mechanism 411 includes a plurality of operating levers arranged around a seat arranged inside the cab 424 in the same manner as the remote operating mechanism 211 .
- the cab 424 is provided with a drive mechanism or a robot that receives a signal corresponding to the operation mode of the remote control lever and moves the actual machine control lever based on the received signal.
- the actual machine imaging device 412 is installed inside the cab 424, for example, and images the environment including at least part of the working mechanism 440 through the front window and the pair of left and right side windows. Some or all of the front window and side windows may be omitted.
- the actual machine surroundings monitoring device 414 monitors real space areas (for example, the right side area, the left side area and/or the rear area of the cab 424) that cannot be captured by the actual machine imaging device 412 in a basic attitude (for example, an attitude in which the optical axis is directed forward). and/or blind spots) for detecting the presence of target objects (eg, workers, other work machines and/or vehicles).
- target objects eg, workers, other work machines and/or vehicles.
- the real machine output interface 42 includes a real machine wireless communication device 422 .
- a working mechanism 440 as a working mechanism includes a boom 441 attached to the upper rotating body 420 so as to be able to rise and fall, and an arm 443 rotatably connected to the tip of the boom 441. and a bucket 445 rotatably connected to the tip of the arm 443 .
- the working mechanism 440 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446, which are configured by telescopic hydraulic cylinders.
- the boom cylinder 442 is interposed between the boom 441 and the upper slewing body 420 so as to expand and contract when supplied with hydraulic oil and rotate the boom 441 in the hoisting direction.
- the arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to expand and contract when supplied with hydraulic oil to rotate the arm 443 about the horizontal axis with respect to the boom 441 .
- the bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to expand and contract when supplied with hydraulic oil to rotate the bucket 445 with respect to the arm 443 about the horizontal axis.
- FIG. 4 is a flow chart for explaining the functions of the remote operation support system having the above configuration.
- the block "C ⁇ " is used for simplification of the description, means transmission and/or reception of data, and processing in the branch direction is executed on the condition of transmission and/or reception of the data. It means a conditional branch.
- an environment confirmation request is transmitted to the remote operation support server 10 through the remote wireless communication device 224 (FIG. 4/STEP 210). For example, it may be determined that the operator performs a designated operation through the remote input interface 210 as a requirement for starting transmission of the environment confirmation request.
- the “specifying operation” is, for example, an operation such as tapping on the remote input interface 210 for specifying the work machine 40 intended for remote operation by the operator.
- the first support processing element 121 transmits the environment confirmation request to the relevant work machine 40 (FIG. 4/C10).
- actual device control device 400 acquires a captured image through actual device imaging device 412, and through actual device wireless communication device 422. , the captured image data representing the captured image is transmitted to the remote operation support server 10 (FIG. 4/STEP 410).
- the working environment image data corresponding to the captured image is transmitted to the remote control device 20 by the second support processing element 122.
- the work environment image data is image data representing a simulated work environment image generated based on the captured image as well as the captured image data itself.
- the remote control device 200 When the remote operation device 20 receives the working environment image data through the remote wireless communication device 224 (FIG. 4/C21), the remote control device 200 outputs the working environment image corresponding to the working environment image data to the remote image output device 221. (FIG. 4/STEP 212).
- the actual machine control device 400 determines whether or not the working machine 40 is in the specified state (FIG. 4/STEP 411). At this time, it may be determined which one of a plurality of predetermined different designated states the designated state corresponds to.
- the “specified state” is set in advance so that the actual machine control device 400 causes the actual machine imaging device 412 to perform its pan function and/or tilt function and zoom function in response to the work machine 40 being in the designated state. It is programmed.
- the specified state may be defined as a state in which the presence of a target object is detected in the blind spot area of the actual imaging device 412 in a normal posture by the actual peripheral monitoring device 414 .
- the left area of the cab 424 is a blind spot area for the actual imaging device 412 in a normal posture. and its optical axis is directed downward (or upward) by the tilt function as required.
- the right side area of the cab 424 is a blind spot area for the actual imaging device 412 in a normal posture. direction, and its optical axis is directed downward (or upward) by a tilt function if necessary.
- the specified state may be defined as a state in which the presence of a specified type of target object is detected in the real space area to be imaged by the actual imaging device 412 .
- the zoom-in function of the actual imaging device 412 increases the display magnification of the image area containing the target object.
- the zoom-out function of the actual imaging device 412 reduces the display magnification of the image area containing the target object.
- pan and/or tilt functions may be controlled to adjust the position of the target object in the captured image.
- a state in which it is detected that the displacement amount and/or the displacement speed of the target object in the real space area to be imaged by the real imaging device 412 is equal to or greater than a predetermined value may be defined as the specified state.
- the pan function and/or tilt function of the actual imaging device 412 adjusts the optical axis of the actual imaging device 412 so as to follow the target object. Orientation is changed.
- the actual machine control device 400 controls the attitude change function (pan function and/or tilt function) of the actual machine imaging device 412 according to the specified state. ) and/or the control mode of the zoom function is recognized, and the recognition result is transmitted to the remote operation support server 10 through the actual wireless communication device 422 (FIG. 4/STEP 412).
- the correspondence relationship between the specified state (type) and the control mode of the attitude change function and/or the zoom function of the actual imaging device 412 may be stored in the storage device or database 102 that constitutes the actual device control device 400 .
- the subsequent processing is executed without recognizing the control mode such as the zoom function of the actual imaging device 412 .
- the second support processing element 122 When the remote operation support server 10 receives the control mode of the attitude change function and/or the zoom function of the actual imaging device 412 (FIG. 4/C12), the second support processing element 122 generates the captured image (and thus the work environment image). ) is recognized and transmitted to the work machine 40 (FIG. 4/STEP 112).
- the panning function of the actual imaging device 412 changes the direction of the optical axis to the left or right
- the second support processing element 122 recognizes that the real space area reflected in the captured image is similarly displaced to the left or right.
- the second support processing element 122 recognizes that the real space area reflected in the captured image is similarly displaced vertically.
- the second support processing element 122 recognizes that the real space area reflected in the captured image is enlarged.
- the zoom magnification is reduced by the zoom-out function of the actual imaging device 412, the second support processing element 122 recognizes that the real space area reflected in the captured image is reduced.
- remote control device 20 When remote control device 20 receives a change in the real space area through remote wireless communication device 224 (FIG. 4/C22), remote control device 200 outputs a notice indicating the change through remote output interface 220. (FIG. 4/STEP 214).
- the remote sound output device The virtual sound source V ss realized by 222 is controlled to be displaced leftward (the same direction as the displacement direction of the real space area).
- the virtual sound source Vss in addition to or instead of displacing the virtual sound source Vss , as shown in FIG. Icon A1 may be output.
- the designated icon may be output so as to move in the same direction as the virtual sound source V ss on the remote image output device 221 .
- control is performed so that the sound gradually becomes stronger and/or the frequency of the sound gradually becomes higher with the localized virtual sound source Vss .
- An arrow-shaped figure or icon A2 pointing to the outside of the image area S may be output.
- the designated icon may be output so as to be enlarged or reduced in accordance with the enlargement or reduction of the real space area or the image area in the remote image output device 221 .
- the virtual sound source V ss realized by the remote sound output device 222 is localized.
- the sound is controlled so that it becomes progressively weaker and/or the frequency of the sound becomes progressively lower.
- the remote image output device 221 displays an image area to be reduced by zooming out and/or an arrow-shaped figure pointing inward of the image area.
- icon A2 may be output.
- the remote control device 200 transmits a notice that the notice has been outputted through the remote output interface 220 to the remote operation support server 10 through the remote wireless communication device 224 (FIG. 4/STEP 216). .
- the second support processing element 122 transmits the advance notice output completion notice to the work machine 40 (FIG. 4/C14).
- the actual machine control device 400 changes the posture of the actual imaging device 412 and/or Alternatively, the zoom function is controlled (FIG. 4/STEP 414).
- the optical axis direction changes from the front of the work machine 40 to the left oblique front by controlling the panning function of the actual machine imaging device 412
- the real space area reflected in the work environment image is shifted from the front area of the work machine 40 to the left. Change to oblique front area.
- the optical axis direction changes from the forward horizontal direction to the forward obliquely downward direction by controlling the tilt function of the actual imaging device 412
- the real space area reflected in the work environment image changes to a lower area than before. .
- the zoom-in function of the actual imaging device 412 controls the zoom-in function of the actual imaging device 412 to reduce the zoom magnification, the real space area reflected in the working environment image is reduced (or the image area is enlarged). Also, by controlling the zoom-out function of the actual imaging device 412 to increase the zoom magnification, the real space area reflected in the work environment image is enlarged (or the image area is reduced).
- the remote control device 200 recognizes the operation mode of the remote operation mechanism 211, and transmits a remote operation command corresponding to the operation mode to the remote operation support server 10 through the remote wireless communication device 224. (FIG. 4/STEP 220).
- the remote operation command when the remote operation command is received by the second support processing element 122, the remote operation command is transmitted to the work machine 40 by the first support processing element 121 (FIG. 4/ C16).
- the actual machine control device 400 when the actual machine control device 400 receives an operation command through the actual machine wireless communication device 422 (FIG. 4/C46), the operation of the working mechanism 440 and the like is controlled (FIG. 4/STEP 420). For example, an operation of scooping soil in front of the work machine 40 with the bucket 445 and removing the soil from the bucket 445 after rotating the upper rotating body 420 is performed.
- the operator who is in contact with the virtual sound source Vss displaced according to the displacement mode of the real space area and/or the arrow-shaped icon image A1 representing the displacement mode of the real space area, sees the real object reflected in the work environment image. It is possible to grasp in advance that the spatial region will be displaced according to the displaced mode in the future (see FIG. 6).
- the space occupation mode of the real space area reflected in the working environment image can change according to changes in the optical axis direction of the real machine imaging device 412 (see FIG. 5). Therefore, it is possible to reduce the psychological burden on the operator when the real space area reflected in the working environment image displayed on the remote image output device 221 is displaced as previously announced.
- the operator who is in contact with the virtual sound source V ss that emits a sound whose intensity and/or frequency changes according to the expansion/contraction of the real space region and/or the arrow-shaped icon image A2 representing the expansion/contraction of the real space region it is possible to grasp in advance that the real space area reflected in the work environment image will be enlarged or reduced in accordance with the aspect of enlargement or reduction in the future (see FIG. 7).
- the space occupation mode of the real space area reflected in the work environment image can be enlarged or reduced according to changes in the imaging magnification of the real imaging device 412 or the like. Therefore, it is possible to reduce the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device 221 is enlarged or reduced as previously announced.
- first support processing element 121 and the second support processing element 122 are configured by the remote operation support server 10, but as another embodiment, the first A support processing element 121 and/or a second support processing element 122 may be configured.
- the actual machine control device 400 controls the attitude change function and/or the zoom function of the actual machine imaging device 412 according to the designated state. is recognized (see FIG. 4/STEP 411 . . . YES ⁇ STEP 412), but as another embodiment, the remote operation support server 10 or the second support processing element 122 can change the attitude of the actual imaging device 412 according to the designated state and /or Control aspects of the zoom function may be recognized.
- a series of processes may be executed according to the flowchart shown in FIG. 8 instead of the flowchart shown in FIG.
- the block "C ⁇ " is used for simplification of the description, means transmission and/or reception of data, and processing in the branch direction is executed on the condition of transmission and/or reception of the data. It means a conditional branch.
- the same reference numerals are used for the same processing as in FIG. 4, and the description thereof is omitted.
- the actual machine control device 400 transmits the designated state to the remote operation support server 10 through the actual machine wireless communication device 422 ( FIG. 8/STEP 413).
- the second support processing element 122 When the remote operation support server 10 receives the designated state (FIG. 8/C13), the second support processing element 122 performs the attitude change function (pan function and/or tilt function) of the actual imaging device 412 according to the designated state. function) and/or control aspects of the zoom function are recognized (FIG. 8/STEP 111). Furthermore, the second support processing element 122 recognizes the change mode of the real space region reflected in the captured image (and thus the work environment image) according to the control mode of the attitude change function and/or the zoom function of the actual imaging device 412. After that, it is transmitted to the working machine 40 (FIG. 8/STEP 112).
- the second support processing element 122 causes the actual imaging device 412 to A control command for the attitude change function and/or the zoom function is generated and transmitted to work machine 40 (FIG. 8/STEP 114).
- the actual machine control device 400 When the work machine 40 receives a control command through the actual wireless communication device 422 (FIG. 8/C44), the actual machine control device 400 performs the attitude change function and/or the zoom function of the actual imaging device 412 according to the control command. controlled (FIG. 8/STEP 414).
- the actual machine control device 400 controls the attitude change function and/or the zoom function of the actual machine imaging device 412 according to the designated state. is recognized (FIG. 4/STEP 411 . . . . . . . . , STEP 412). Control aspects of functions may be recognized.
- the pan function and/or tilt function of the actual imaging device 412 continuously displaces the real space region reflected in the captured image and the corresponding work environment image.
- a plurality of actual imaging devices having an imaging area are mounted on the work machine 40, and the actual imaging device that acquires the captured image serving as the basis of the working environment image is switched from one actual imaging device to another actual imaging device.
- the real space area reflected in the work environment image may be discontinuously displaced.
- one real-machine imaging device may be switched to another real-machine imaging device whose imaging region is the real-space region where the target object is detected by the real-machine peripheral monitoring device 414 .
- the advance notice output by the remote output interface 220 reduces the psychological burden on the operator even when the work environment image output to the remote image output device 221 is discontinuously displaced or changed. .
- the pan function and/or tilt function of the actual imaging device 412 continuously displaces the real space region reflected in the captured image and the corresponding work environment image.
- the real space area reflected in the work environment image may be displaced and/or scaled by image processing when the work environment image is generated by the 1 support processing element 121 .
- the psychological burden on the operator can be reduced. be done.
- the zoom-in and zoom-out of the actual imaging device 412 are announced by the intensity of the sound (sound pressure level) and/or the frequency level emitted from the virtual sound source Vss (see FIG. 7).
- the real imaging device 412 has only a panning function, and before and after the change in the panning angle is announced by the left and right displacement of the virtual sound source Vss (see FIG. 6), the real imaging device 412 may be heralded by up and down displacements of the virtual source Vss .
- the actual imaging device 412 has only a tilt function, and before and after the change in the tilt angle is announced by the vertical displacement of the virtual sound source Vss , the actual imaging device 412 zooms in. and zoom out may be heralded by left and right displacements of the virtual sound source Vss .
- the second support processing element recognizes a future change mode of the real space area reflected in the work environment image in accordance with the specified state detected by the actual machine surroundings monitoring device mounted on the work machine. is preferred.
- the real space area reflected in the work environment image will be displaced in the future according to the displaced mode, and the operator who has come into contact with the advance notice of the displaced output location will be able to perform the work. It is possible to let the operator know in advance that the machine is in the specified state. Therefore, it is possible to reduce the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device is displaced as previously announced.
- the second support processing element recognizes a future displacement mode of the real space area as a future change mode of the real space area, and is configured by the remote output interface according to the future displacement mode of the real space area. It is preferable to displace the output location of the previous notice.
- the operator who has received a notice of the displacement of the output part, can grasp in advance that the real space area reflected in the working environment image will be displaced in accordance with the displacement mode in the future. be able to. For example, depending on changes in the optical axis direction of the actual imaging device, switching of one actual imaging device corresponding to the working environment image among a plurality of actual imaging devices, image processing by the first support processing element, etc., the working environment image The space occupation mode of the reflected real space area can be displaced. Therefore, it is possible to reduce the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device is displaced as previously announced.
- the second support processing element further recognizes a future expansion/contraction mode of the real space area as a future change mode of the real space area, and is configured by the remote output interface according to the future displacement mode of the real space area.
- the remote output interface displaces the output point of the notice according to a future expansion/contraction mode of the real space area.
- the remote operation support server having this configuration, when the operator comes into contact with the advance notice in which the output location is displaced in two steps, the real space area reflected in the work environment image is displaced in the future according to the first displacement mode, and It is possible to grasp in advance that the expansion/contraction is performed according to the second displacement mode. Therefore, it is possible to reduce the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device is displaced and enlarged/reduced according to the notice.
- the second support processing element displaces a virtual sound source that emits sound as the announcement to a remote sound output device constituting the remote output interface.
- the operator who comes in contact with the advance notice of the displacement of the virtual sound source is notified that the real space area reflected in the work environment image will be displaced in the future in accordance with the displacement mode (and will be enlarged or reduced). ) can be grasped in advance.
- the virtual sound source is displaced to the right (left, up, down)
- the operator can understand that the real space area reflected in the work environment image is immediately displaced to the right (left, up, down). .
- the notice is conveyed to the operator not visually but aurally, it is possible to avoid distracting the operator's visual attention to the work environment image. Therefore, it is possible to reduce the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device is displaced as previously announced.
- the second support processing element determines a future image of the real space area in accordance with a mode of switching from one of the plurality of real imaging devices mounted on the work machine to another real imaging device. It is preferable to recognize the future displacement mode of the real space area as the change mode.
- the operator who has come into contact with the notice that the output mode will change, is informed that the expansion/contraction mode in the future of the real space area reflected in the work environment image is derived from the switching of the actual imaging device, and the output mode is changed. It is possible to grasp in advance that it will change according to the change of the mode. Therefore, it is possible to reduce the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device is enlarged or reduced as previously announced.
- the second support processing element recognizes a future expansion/reduction mode of the real space area as a future change mode of the real space area, and according to the future expansion/reduction mode of the real space area, the notification is sent to the remote output interface. It is preferable to change the output mode of .
- the operator who comes in contact with the notice that the output mode will change, is notified that the future expansion/contraction mode of the real space area reflected in the work environment image will change in accordance with the change in the output mode.
- the second support processing element changes at least one of sound pressure level and frequency level of the sound as the notice to the remote sound output device constituting the remote output interface.
- the remote operation support server having this configuration, an operator who has come into contact with an announcement as sound in which the sound pressure level and/or the frequency level changes is notified that the real space area reflected in the work environment image will change in the future. It is possible to grasp in advance that the image will be enlarged or reduced accordingly. For example, when the sound pressure gradually increases, the operator can be made aware in advance that the real space area will shrink. On the other hand, when the sound pressure gradually decreases, the operator can be made aware in advance that the real space area will expand. At this time, since the notice is conveyed to the operator not visually but aurally, it is possible to avoid distracting the operator's visual attention to the work environment image. Therefore, it is possible to reduce the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device is enlarged or reduced as previously announced.
Landscapes
- Engineering & Computer Science (AREA)
- Mining & Mineral Resources (AREA)
- Civil Engineering (AREA)
- General Engineering & Computer Science (AREA)
- Structural Engineering (AREA)
- Component Parts Of Construction Machinery (AREA)
- Manipulator (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
遠隔操作装置を用いた作業機械の遠隔操作を支援するためのサーバであって、
前記作業機械に搭載されている実機撮像装置を通じて当該作業機械の環境を表わす撮像画像を取得し、前記撮像画像に応じた作業環境画像を、前記遠隔操作装置を構成する遠隔画像出力装置に表示させる第1支援処理要素と、
前記第1支援処理要素により前記遠隔画像出力装置に表示されている前記作業環境画像に映り込んでいる実空間領域の未来の変化態様を認識し、前記遠隔操作装置を構成する遠隔出力インターフェースに、前記実空間領域の未来の変化態様を表わす予告を出力させる第2支援処理要素と、
を備えている。
図1に示されている遠隔操作支援システムは、遠隔操作支援サーバ10と、遠隔操作装置20と、遠隔操作装置20を通じて遠隔操作可能な作業機械40と、により構成されている。遠隔操作装置20および作業機械14のうち一方のみが、遠隔操作支援システムの構成要素であってもよい。遠隔操作支援サーバ10、遠隔操作装置20および作業機械40は相互にネットワーク通信可能に構成されている。遠隔操作支援サーバ10および遠隔操作装置20の相互通信ネットワークと、遠隔操作支援サーバ10および作業機械40の相互通信ネットワークと、は同一であってもよく相違していてもよい。
遠隔操作支援サーバ10は、データベース102と、第1支援処理要素121と、第2支援処理要素122と、を備えている。データベース102は、撮像画像データ等を記憶保持する。データベース102は、遠隔操作支援サーバ10とは別個のデータベースサーバにより構成されていてもよい。各支援処理要素は、演算処理装置(シングルコアプロセッサまたはマルチコアプロセッサもしくはこれを構成するプロセッサコア)により構成され、メモリなどの記憶装置から必要なデータおよびソフトウェアを読み取り、当該データを対象として当該ソフトウェアにしたがった後述の演算処理を実行する。
遠隔操作装置20は、遠隔制御装置200と、遠隔入力インターフェース210と、遠隔出力インターフェース220と、を備えている。遠隔制御装置200は、演算処理装置(シングルコアプロセッサまたはマルチコアプロセッサもしくはこれを構成するプロセッサコア)により構成され、メモリなどの記憶装置から必要なデータおよびソフトウェアを読み取り、当該データを対象として当該ソフトウェアにしたがった演算処理を実行する。
遠隔操作機構211には、走行用操作装置と、旋回用操作装置と、ブーム用操作装置と、アーム用操作装置と、バケット用操作装置と、が含まれている。各操作装置は、回動操作を受ける操作レバーを有している。走行用操作装置の操作レバー(走行レバー)は、作業機械40の下部走行体410を動かすために操作される。走行レバーは、走行ペダルを兼ねていてもよい。例えば、走行レバーの基部または下端部に固定されている走行ペダルが設けられていてもよい。旋回用操作装置の操作レバー(旋回レバー)は、作業機械40の旋回機構430を構成する油圧式の旋回モータを動かすために操作される。ブーム用操作装置の操作レバー(ブームレバー)は、作業機械40のブームシリンダ442を動かすために操作される。アーム用操作装置の操作レバー(アームレバー)は作業機械40のアームシリンダ444を動かすために操作される。バケット用操作装置の操作レバー(バケットレバー)は作業機械40のバケットシリンダ446を動かすために操作される。
作業機械40は、実機制御装置400と、実機入力インターフェース41と、実機出力インターフェース42と、作業機構440と、を備えている。実機制御装置400は、画像処理装置30を備えている。画像処理装置30は、状態検知要素31と、画像予測要素32と、画像圧縮要素34と、を備えている。実機制御装置400および画像処理装置30の構成要素のそれぞれは、演算処理装置(シングルコアプロセッサまたはマルチコアプロセッサもしくはこれを構成するプロセッサコア)により構成され、メモリなどの記憶装置から必要なデータおよびソフトウェアを読み取り、当該データを対象として当該ソフトウェアにしたがった演算処理を実行する。
図4は前記構成の遠隔操作支援システムの機能について説明するフローチャートである。当該フローチャートにおいて「C●」というブロックは、記載の簡略のために用いられ、データの送信および/または受信を意味し、当該データの送信および/または受信を条件として分岐方向の処理が実行される条件分岐を意味している。
当該構成の遠隔操作支援システムによれば、遠隔画像出力装置221に表示されている作業環境画像に映り込んでいる実空間領域の未来の変化態様を表わす予告が遠隔出力インターフェース220に出力される。
前記実施形態では、遠隔操作支援サーバ10により第1支援処理要素121および第2支援処理要素122が構成されていたが、他の実施形態として、作業機械40および/または遠隔操作装置20により第1支援処理要素121および/または第2支援処理要素122が構成されていてもよい。
作業機械40が指定状態にあると判定された場合(図8/STEP411‥YES)、実機制御装置400により、当該指定状態が実機無線通信機器422を通じて遠隔操作支援サーバ10に対して送信される(図8/STEP413)。
前記第2支援処理要素が、前記作業機械に搭載されている実機周辺監視装置により検知された指定状態に応じて、前記作業環境画像に映り込んでいる実空間領域の未来の変化態様を認識する
ことが好ましい。
前記第2支援処理要素が、前記実空間領域の未来の変化態様として当該実空間領域の未来の変位態様を認識し、前記実空間領域の未来の変位態様にしたがって、前記遠隔出力インターフェースにより構成される前記予告の出力箇所を変位させる
ことが好ましい。
前記第2支援処理要素が、前記実空間領域の未来の変化態様として当該実空間領域の未来の拡縮態様をさらに認識し、前記実空間領域の未来の変位態様にしたがって前記遠隔出力インターフェースにより構成される前記予告の出力箇所を変位させる前または後に、前記実空間領域の未来の拡縮態様にしたがって前記遠隔出力インターフェースに前記予告の出力箇所を変位させる
ことが好ましい。
前記第2支援処理要素が、前記遠隔出力インターフェースを構成する遠隔音響出力装置に前記予告としての音響を発する仮想音源を変位させる
ことが好ましい。
前記第2支援処理要素が、前記作業機械に搭載されている複数の前記実機撮像装置のうち一の実機撮像装置から他の実機撮像装置への切り替え態様に応じて、前記実空間領域の未来の変化態様として当該実空間領域の未来の変位態様を認識する
ことが好ましい。
前記第2支援処理要素が、前記実空間領域の未来の変化態様として当該実空間領域の未来の拡縮態様を認識し、前記実空間領域の未来の拡縮態様にしたがって、前記遠隔出力インターフェースに前記予告の出力態様を変化させる
ことが好ましい。
前記第2支援処理要素が、前記遠隔出力インターフェースを構成する遠隔音響出力装置に前記予告としての音響の音圧の高低および周波数の高低のうち少なくとも一方を変化させる
ことが好ましい。
Claims (9)
- 遠隔操作装置を用いた作業機械の遠隔操作を支援するためのサーバであって、
前記作業機械に搭載されている実機撮像装置を通じて当該作業機械の環境を表わす撮像画像を取得し、前記撮像画像に応じた作業環境画像を、前記遠隔操作装置を構成する遠隔画像出力装置に表示させる第1支援処理要素と、
前記第1支援処理要素により前記遠隔画像出力装置に表示されている前記作業環境画像に映り込んでいる実空間領域の未来の変化態様を認識し、前記遠隔操作装置を構成する遠隔出力インターフェースに、前記実空間領域の未来の変化態様を表わす予告を出力させる第2支援処理要素と、
を備えている遠隔操作支援サーバ。 - 請求項1に記載の遠隔操作支援サーバにおいて、
前記第2支援処理要素が、前記作業機械に搭載されている実機周辺監視装置により検知された指定状態に応じて、前記作業環境画像に映り込んでいる前記実空間領域の未来の変化態様を認識する
遠隔操作支援サーバ。 - 請求項1または2に記載の遠隔操作支援サーバにおいて、
前記第2支援処理要素が、前記実空間領域の未来の変化態様として当該実空間領域の未来の変位態様を認識し、前記実空間領域の未来の変位態様にしたがって前記遠隔出力インターフェースにより構成される前記予告の出力箇所を変位させる
遠隔操作支援サーバ。 - 請求項3に記載の遠隔操作支援サーバにおいて、
前記第2支援処理要素が、前記実空間領域の未来の変化態様として当該実空間領域の未来の拡縮態様をさらに認識し、前記実空間領域の未来の変位態様にしたがって前記遠隔出力インターフェースにより構成される前記予告の出力箇所を変位させる前または後に、前記実空間領域の未来の拡縮態様にしたがって前記遠隔出力インターフェースに前記予告の出力箇所を変位させる
遠隔操作支援サーバ。 - 請求項3または4に記載の遠隔操作支援サーバにおいて、
前記第2支援処理要素が、前記遠隔出力インターフェースを構成する遠隔音響出力装置に前記予告としての音響を発する仮想音源を変位させる
遠隔操作支援サーバ。 - 請求項3~5のうちいずれか1項に記載の遠隔操作支援サーバにおいて、
前記第2支援処理要素が、前記作業機械に搭載されている複数の前記実機撮像装置のうち一の実機撮像装置から他の実機撮像装置への切り替え態様に応じて、前記実空間領域の未来の変化態様として当該実空間領域の未来の変位態様を認識する
遠隔操作支援サーバ。 - 請求項1~6のうちいずれか1項に記載の遠隔操作支援サーバにおいて、
前記第2支援処理要素が、前記実空間領域の未来の変化態様として当該実空間領域の未来の拡縮態様を認識し、前記実空間領域の未来の拡縮態様にしたがって、前記遠隔出力インターフェースに前記予告の出力態様を変化させる
遠隔操作支援サーバ。 - 請求項1~7のうちいずれか1項に記載の遠隔操作支援サーバにおいて、
前記第2支援処理要素が、前記遠隔出力インターフェースを構成する遠隔音響出力装置に前記予告としての音響の音圧の高低および周波数の高低のうち少なくとも一方を変化させる
遠隔操作支援サーバ。 - 請求項1~8のうちいずれか1項に記載の遠隔操作支援サーバと、前記遠隔操作装置および前記作業機械のうち少なくとも一方と、により構成されている遠隔操作支援システム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP21931730.2A EP4279667A1 (en) | 2021-03-19 | 2021-12-13 | Remote operation assistance server and remote operation assistance system |
US18/278,209 US20240141616A1 (en) | 2021-03-19 | 2021-12-13 | Remote operation assistance server and remote operation asistance system |
CN202180095647.5A CN117043732A (zh) | 2021-03-19 | 2021-12-13 | 远程操作支援服务器及远程操作支援*** |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2021-045675 | 2021-03-19 | ||
JP2021045675A JP2022144601A (ja) | 2021-03-19 | 2021-03-19 | 遠隔操作支援サーバおよび遠隔操作支援システム |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022195988A1 true WO2022195988A1 (ja) | 2022-09-22 |
Family
ID=83320041
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2021/045780 WO2022195988A1 (ja) | 2021-03-19 | 2021-12-13 | 遠隔操作支援サーバおよび遠隔操作支援システム |
Country Status (5)
Country | Link |
---|---|
US (1) | US20240141616A1 (ja) |
EP (1) | EP4279667A1 (ja) |
JP (1) | JP2022144601A (ja) |
CN (1) | CN117043732A (ja) |
WO (1) | WO2022195988A1 (ja) |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2005286590A (ja) * | 2004-03-29 | 2005-10-13 | Fuji Photo Film Co Ltd | 画像表示方法および画像表示装置 |
JP2008294724A (ja) * | 2007-05-24 | 2008-12-04 | Panasonic Corp | 画像受信装置 |
JP2013109687A (ja) * | 2011-11-24 | 2013-06-06 | Kyocera Corp | 携帯端末装置、プログラムおよび表示制御方法 |
WO2016174977A1 (ja) * | 2015-04-28 | 2016-11-03 | 株式会社小松製作所 | 作業機械の周辺監視装置及び作業機械の周辺監視方法 |
JP2019156533A (ja) | 2018-03-09 | 2019-09-19 | 株式会社タダノ | クレーン |
JP6581139B2 (ja) | 2017-03-31 | 2019-09-25 | 日立建機株式会社 | 作業機械の周囲監視装置 |
JP2020077173A (ja) * | 2018-11-07 | 2020-05-21 | コベルコ建機株式会社 | 建設機械の遠隔操作装置 |
JP2020200659A (ja) * | 2019-06-10 | 2020-12-17 | コベルコ建機株式会社 | 遠隔操作システムおよび遠隔操作サーバ |
-
2021
- 2021-03-19 JP JP2021045675A patent/JP2022144601A/ja active Pending
- 2021-12-13 WO PCT/JP2021/045780 patent/WO2022195988A1/ja active Application Filing
- 2021-12-13 EP EP21931730.2A patent/EP4279667A1/en active Pending
- 2021-12-13 US US18/278,209 patent/US20240141616A1/en active Pending
- 2021-12-13 CN CN202180095647.5A patent/CN117043732A/zh active Pending
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2005286590A (ja) * | 2004-03-29 | 2005-10-13 | Fuji Photo Film Co Ltd | 画像表示方法および画像表示装置 |
JP2008294724A (ja) * | 2007-05-24 | 2008-12-04 | Panasonic Corp | 画像受信装置 |
JP2013109687A (ja) * | 2011-11-24 | 2013-06-06 | Kyocera Corp | 携帯端末装置、プログラムおよび表示制御方法 |
WO2016174977A1 (ja) * | 2015-04-28 | 2016-11-03 | 株式会社小松製作所 | 作業機械の周辺監視装置及び作業機械の周辺監視方法 |
JP6581139B2 (ja) | 2017-03-31 | 2019-09-25 | 日立建機株式会社 | 作業機械の周囲監視装置 |
JP2019156533A (ja) | 2018-03-09 | 2019-09-19 | 株式会社タダノ | クレーン |
JP2020077173A (ja) * | 2018-11-07 | 2020-05-21 | コベルコ建機株式会社 | 建設機械の遠隔操作装置 |
JP2020200659A (ja) * | 2019-06-10 | 2020-12-17 | コベルコ建機株式会社 | 遠隔操作システムおよび遠隔操作サーバ |
Also Published As
Publication number | Publication date |
---|---|
EP4279667A1 (en) | 2023-11-22 |
CN117043732A (zh) | 2023-11-10 |
US20240141616A1 (en) | 2024-05-02 |
JP2022144601A (ja) | 2022-10-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6427597B2 (ja) | 作業機械の周辺監視装置及び作業機械の周辺監視方法 | |
JP5779244B2 (ja) | 作業機械の周囲監視装置 | |
JP5546427B2 (ja) | 作業機械の周囲監視装置 | |
EP2717570A1 (en) | Device for monitoring area around working machine | |
WO2016158265A1 (ja) | 作業機械 | |
WO2019189430A1 (ja) | 建設機械 | |
JP6071786B2 (ja) | 作業機械の周囲監視装置 | |
JP7472578B2 (ja) | 遠隔操作支援サーバ、遠隔操作支援システムおよび遠隔操作支援方法 | |
EP3739131B1 (en) | Construction machine | |
JP6917167B2 (ja) | 建設機械用俯瞰画像表示装置 | |
JP7318258B2 (ja) | 遠隔操作システムおよび遠隔操作サーバ | |
WO2021230090A1 (ja) | 遠隔操作支援装置および遠隔操作支援方法 | |
WO2021176883A1 (ja) | 作業支援サーバ、作業支援方法 | |
WO2022195988A1 (ja) | 遠隔操作支援サーバおよび遠隔操作支援システム | |
JP2021099017A (ja) | 遠隔操作装置および遠隔操作システム | |
JP2018121221A (ja) | 作業機械用周辺監視システム | |
JP2023040971A (ja) | 遠隔操作支援装置、遠隔操作支援システムおよび遠隔操作支援方法 | |
US11939744B2 (en) | Display system, remote operation system, and display method | |
WO2021166476A1 (ja) | 遠隔操作支援サーバ、遠隔操作支援システムおよび遠隔操作支援方法 | |
CN112433502A (zh) | 基于5g通讯和vr双环视技术的平地机远程遥控***及方法 | |
WO2021166475A1 (ja) | 遠隔操作装置、遠隔操作支援サーバ、遠隔操作支援システムおよび遠隔操作支援方法 | |
US20230150358A1 (en) | Collision avoidance system and method for avoiding collision of work machine with obstacles | |
WO2023100533A1 (ja) | 画像表示システム、遠隔操作支援システムおよび画像表示方法 | |
WO2023136070A1 (ja) | 遠隔操作支援システムおよび遠隔操作支援方法 | |
US20230151583A1 (en) | Collision avoidance system and method for avoiding collision of work machine with obstacles |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21931730 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2021931730 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18278209 Country of ref document: US |
|
ENP | Entry into the national phase |
Ref document number: 2021931730 Country of ref document: EP Effective date: 20230818 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202180095647.5 Country of ref document: CN |
|
NENP | Non-entry into the national phase |
Ref country code: DE |