WO2018178506A1

WO2018178506A1 - A method, a system and a device for displaying real-time video images from around a vehicle

Info

Publication number: WO2018178506A1
Application number: PCT/FI2018/000008
Authority: WO
Inventors: Jyrki Portin; Esa Pollari
Original assignee: Scopesensor Oy
Priority date: 2017-03-30
Filing date: 2018-03-29
Publication date: 2018-10-04

Abstract

The invention implements a display system of live video camera feeds for vehicles. In a departure from traditional systems featuring fixed cameras and monitors, the invention utilises one or more pan-tilt or pan-tilt-zoom cameras (102, 202, 203, 204) that are controlled by a portable display device (301, 601) having a touchscreen (302, 602). The system has two camera control methods: a first camera control mode and a second camera control mode. In the first camera control mode the directional movements of the camera are slaved to the orientation of the display device. In the second camera control mode the directional movements of the camera are controlled by swiping the touchscreen. The two camera control modes are intuitively combined so that both methods can be used intermittently, without the user explicitly having to switch camera control modes.

Description

A method, a system and a device for displaying real-time video images from around a vehicle

Technical field

Various embodiments relate to systems and display devices for displaying video images from around a vehicle.

Background

The invention relates to using video cameras in navigation and manoeuvring. Video cameras are commonly used for monitoring the surroundings of a vehicle. The video feed captured by the video cameras is transferred to the bridge, cabin, cockpit, re- mote control station or similar. This is useful especially in maritime navigation because ships are quite large and have structures which could obstruct the view from the bridge. Also remote controlled vehicles, such as unmanned aerial, ground, surface and underwater vehicles, use video cameras. The video camera could be placed for example on the bow of a ship and the watchstanding officer on the bridge can view the video feed from the video camera. Usually the viewing direction of the video camera can be controlled.

Remote-control cameras are usually controlled by a joystick or similar device. However, when viewing this type of video feed, the viewing direction is not always intuitive. Also in maritime navigation the watchstanding officer is exposed to a myriad of information. This information is displayed on different screens and monitors. To read this information the officer has to turn his or her attention away from the view outside of the bridge or that on the video camera display. To improve this situation the display device could be a movable or a portable device. However, using this type of display device could easily lead to disorientation when the officer's viewing direction is different from the direction of the camera. To complicate matters even further, the previously described situation could be sometimes the desirable one.

The concept of using a remotely controlled camera on a boat has been proposed for example in patent publication US6057880. However, it is a problem in the known art that using them is not intuitive and problems may arise if several cameras are used.

Summary Various embodiments are achieved through a method, a system and a display device characterized in what is disclosed in the independent claims. Some embodiments of the invention are disclosed in the dependent claims.

The invention implements a display system of live video camera feeds for vehicles operating over water, on land and in the air. In a departure from traditional systems featuring fixed cameras and monitors, the invention utilises one or more pan-tilt or pan-tilt-zoom cameras that are slaved to a tablet computer or similar portable device. The user controls the cameras either by rotating the tablet computer, or by swiping its touchscreen. The two camera control methods are intuitively combined so that both methods can be used intermittently, without the user explicitly having to switch control modes.

It must be noted that here the camera direction and the direction of the camera are directions that camera is directed i.e. the direction the camera is viewing.

A method for displaying real-time video images from around a vehicle uses a video camera and a display device. According to an embodiment, the method comprises steps where the video image is captured by the video camera mounted directionally controllable on the vehicle, the video image is transferred to a portable display device and the display device has a touchscreen for displaying the video images. The method further comprise steps where the directional movements of the video cam- era are slaved to the orientation of the display device, when a user swipes the touchscreen the video camera is no longer slaved to the orientation of the display device, and the directional movements of the video camera are controlled by swiping the touchscreen, and the directional movements of the video camera are slaved again to the orientation of the display device, when the user performs a predeter- mined action with the display device.

With the method a system and a display device can be configured to produce a way to control video cameras for displaying real-time video images from around a vehicle that is intuitive and easy to use.

In an embodiment, the predetermined action is setting the display device orientation to the same direction as that of the video camera. By the same direction it is meant that the directional vectors of the video camera and the display device are parallel.

In a second embodiment, there is an area on the touchscreen that activates a range- finder that measures the range to an aimpoint. Preferably the aimpoint is on the said area. In a third embodiment, activating the rangefinder also activates tracking of the aimpoint area. Then the video camera tracks the aimpont area which could be another vehicle, some object or such.

In a fourth embodiment, the touchscreen displays an icon indicating if the directional movements of the video camera are slaved to the orientation of the display device or if the directional movements of the video camera are controlled by swiping the touchscreen. From that the user can see which video camera control mode is in use.

A system for displaying real-time video images from around a vehicle comprises a video camera that is arranged to be mounted directionally controllable on the vehicle and a display device that is arranged to receive a video feed from the video camera, and the display device comprises a touchscreen for displaying the video feed. The display device comprises further an arrangement for detecting the orientation of the display device and an arrangement for sending directional movement instructions to the video camera. The display device has a first camera control mode and a second camera control mode. In the first camera control mode the directional move- ments of the video camera movements are slaved to the orientation of the display device, and in the second camera control mode swiping the touchscreen is arranged to control the directional movements of the video camera. The display device is arranged to switch from the first control mode to the second control mode when a swipe is detected on the touchscreen, and switch from the second camera control mode to the first camera control mode when the display device is detecting movement or has detected movement and the display device is set to a position where the orientation of the display device and the direction of the video camera are matching. The directional movements include azimuth and elevation.

In an embodiment of the system is a rangefinder that is arranged to be activated by tapping a predetermined area on the touchscreen and the distance is calculated to a target that is on the video feed inside the predetermined area, and activating the rangefinder also activates tracking of the target.

In a second embodiment, the system comprises at least one additional video camera producing at least one additional video feed and when changing the view on the display device the video feed is changed accordingly. This means that when the viewing direction is changed in either camera control mode, the video camera that is in use is changed when the limits of the field of view of the video camera are approached sufficiently. In a third embodiment, the display device is arranged to display an icon indicating which camera control mode is in use. It must be noted that the icon here may be a text, a colour code, a pictogram or such.

The display device for displaying real-time video image from around a vehicle ac- cording to an embodiment is arranged for displaying the video feed of a video camera that is mounted directionally controllable on the vehicle. The display device is portable and comprises a touchscreen. The display device further comprises a first camera control mode and a second camera control mode, and in the first camera control mode the directional movements of the video camera are slaved to the ori- entation of the display device, and in the second camera control mode swiping the touchscreen is arranged to control the directional movements of the video camera, and the display device is arranged to switch from the first control mode to the second control mode when a swiping gesture is detected on the touchscreen, and switch from the second control mode to the first control mode when the display device is detecting movement or has detected movement and the display device is set to a direction where the orientation of the display device and the video camera direction are matching.

Various embodiments will become apparent from the detailed description given hereafter. However, it should be understood that the detailed description and spe- cific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the spirit and scope of the invention will become apparent to those skilled in the art from this detailed description.

With the method a user interface can be produced on a display device. These embodiments are especially suitable for maritime navigation but are applicable to other purposes as well.

Description of the figures

In the following, embodiments will be described in detail. In the description, reference is made to the enclosed drawings, in which: Figure 1 shows an example of a vehicle having a video camera for capturing video images from around a motor yacht; Figure 2 shows an example of a vehicle having several video cameras for capturing video images from around a cargo ship;

Figure 3 shows an example of an embodiment of a first camera control mode of a system; Figure 4 shows an example of an embodiment of a second camera control mode of a system;

Figure 5 shows an example of a system where the camera control mode is switched from the second camera control mode back to the first control mode; Figure 6 shows an example of a system where one to several objects are being actively tracked, and

Figure 7 shows a flow chart according to an embodiment.

Detailed description of Figures The embodiments in the following description are given as examples only, and a person skilled in the art may realise the basic idea of the invention also in some other way than what is described in the description. Though the description may refer to a certain embodiment or embodiments in several places, this does not mean that the reference would be directed towards only one described embodiment or that the described characteristic would be usable only in one described embodiment. The individual characteristics of two or more embodiments may be combined and new embodiments of the invention may thus be provided.

Figure 1 shows a vessel 101 having a video camera 102. The vessel is a motor yacht, and the video camera is fitted onto a pylon. The direction of the video camera is controllable. Figure 2 shows a cargo ship 201 having three video cameras: a first video camera 202, a second video camera 203 and a third video camera 204. The directions of the video cameras are controllable. Video cameras are located at the bow, stern and sides so that each camera has its own field of view.

It must be noted that even though the examples here use maritime vehicles the embodiments are naturally applicable to other types of vehicles. In an embodiment a video camera or video cameras mounted on a vehicle such as a vessel are controlled by a portable display device having a touchscreen. This means that the display device has a means for controlling the directional movements of the video camera. The display device has two camera control modes: a first cam- era control mode and a second camera control mode. The display device could be a tablet or a similar device.

The first control mode relies on the orientation of the display device. The direction of the video camera is slaved to the orientation of the display device. This means that the direction of the video camera is the same as the facing direction of the display device. For example, if the facing direction of the display device i.e. the orientation of the display device is west, the video camera is also pointing west. The orientation is derived from an orientation detector arrangement that is in the display device. This orientation detector arrangement could be implemented in many ways such as with an accelerometer, a gyroscope, a magnetometer, an inertia system or such. For example, the magnetometer detects changes in the magnetic field and thus determines the horizontal orientation of the display device and adapts the azimuth of the camera accordingly. When using the first control method, the user, for example, points the top edge of the tablet perpendicular to the direction that he or she wants to view, and the camera turns to that direction. The main benefit of the method is its intuitiveness; the user instinctively knows where the camera is pointing, because he or she is facing the same direction.

The second control mode has the user swiping his or her finger over the touchscreen of the display device to control the video camera. In doing so, the finger is placed on the touchscreen and is swiped over it, as if to drag a point in the displayed image to a different location on the screen. The swipes can be horizontal or vertical, controlling respectively the azimuth and the elevation of the video camera. Diagonal swipes are also possible. Swiping unlocks the video camera slaving so that the orientation of the display device and the direction of the video camera no longer match. The video camera can be reslaved either by rotating the display device, in which case the direction of the video camera does not change until it matches the orientation of the video camera once more, or by swiping, which changes the direction of the video camera as per usual. The former behaviour allows the user to turn quickly towards a point of interest, for example to switch to binoculars for a closer view.

In some embodiments, status information relating to the video camera is superim- posed on the image of the display device in graphic and alphanumeric format. For example, a compass rose could provide a compass reference and a large arrow indicates the heading of the vehicle. The length of the arrow can be indicative of the speed at which the vehicle travels. In-between the compass rose and the arrow, a video camera icon moves in a circular motion to indicate the current direction of the video camera. The video camera icon is also an indicator of the control mode of the video camera. For example, a bare green camera symbol represents a unslaved video camera, while the same symbol in reverse video represents a slaved video camera. A quick-lock function is provided to return the direction of the video camera to match the orientation of the display device. Optionally, a timer function slaves the video camera if the display device is left untouched for long enough. The two camera control modes complement each other by catering to different circumstances. On a roomy bridge of a ship it is usually not a problem to turn around with the display device in hand i.e. using the first camera control mode. While the bridge provides a good view outside, the display device allows the user to look through obstacles as it were, such as shipping containers stowed on deck, since the video camera, because of its placement, provides an unimpeded view as is shown in figure 2. Also, optional night vision and zoom capabilities reveal details that the naked eye cannot perceive. In confined spaces, however, the user turning around may prove inconvenient, so there swiping, i.e. the second camera control mode, is the better alternative. When the size, shape or nature of the vehicle necessitates more than one video camera to cover an unimpeded 360° view around the vehicle, one, several or all video cameras can be involved in producing the desired image as is depicted in figure 2. Their number and field of view permitting, a 360° panoramic compound image can be presented to the user. The selection of active video cameras is auto- matic and necessary handovers from one video camera to another occur seamlessly.

In some embodiments, in the very centre of the touchscreen, a circle or some other shape serves as a reticle. Double-tapping the reticle activates a rangefinder that measures the range to an aimpoint. The range, the azimuth and the elevation of the video camera are recorded for tracking purposes. Tracking involves recalculating the bearing of the point of interest and the applicable azimuth and elevation for the video camera as the vehicle moves. The video camera can be quickly retrained at the point of interest by double-tapping the related entry at the bottom left of the touchscreen. For the sake of simplicity, the tagged point can be treated as stationary object, meaning its speed and heading are disregarded, or if external tracking assistance is available - for example, in the form of AIS transceivers that report the position, course and speed of a vessel - the system can be enhanced to track moving objects as well. The tagged points of interests are also shown in relation to the compass rose, in a fashion similar to the camera icon, but with, for example, danger triangle signs as the icons. A dashed circle can also be placed over the calculated position of the point of interest to accentuate it in the image.

In figure 3 is an example of a first camera control mode. There is a display device 301 having a touchscreen 302. In this example the display device is a tablet computer, but naturally other kinds of display devices can be used. On the touchscreen a video feed from a video camera is displayed. In addition to the video feed other information can be displayed by compounding this information to the image to be displayed. This information could be for example the heading of the vessel and the direction of the video camera.

When the first camera control mode is in use the movements of the video camera are slaved to the orientation of the display device 301. In figure 3A a user holds the display device 301 and the orientation of the display device is such that the top edge of the display device is perpendicular to the heading of the vessel that is 350°. The direction of the video camera is the same 350°. The user turns and at the same time rotates the display device to the orientation that is presented in figure 3B. The display device has an orientation detector arrangement that detects the orientation of the display device. The orientation detector arrangement detects that the direction the display device is facing i.e. the direction what is perpendicular to the top edge of the display device, is now 39°. The display device sends instructions to the video camera to follow the orientation of the display device and the video camera turns to the direction of 39°. In the first camera control mode the direction of the video cam- era matches the orientation of the display device.

Of course, some embodiments of the first camera control mode could include elevation control. In that case, tilting the display device 301 controls the elevation of the video camera. Because there is a possibility that careless tilting could turn the video camera to an unwanted direction, such as to sky, there could be some prede- termined action to activate elevation control. One example from these could be two consecutive tilting motions in a predetermined time window.

In figure 4 is an example of a second camera control mode using the same display device 301 that was depicted in figure 3. When the second camera control mode is in use the movements of the video camera are controlled by swiping the touchscreen 302. In figure 4A a user holds the display device 301 and the orientation of the display device is such that the top edge of the display device is perpendicular to the heading of the vessel that is 350°. The direction of the video camera is the same 350°. The user starts to swipe the touchscreen. In this example the user wants to turn the video camera to the right towards another vessel seen on the video feed. The user then touches the touchscreen and swipes to the left. The display device recognises the swipe and sends instructions to the video camera to turn right. In figure 4B the direction of the video camera is 39° and another vessel is in centred on the touchscreen. The user then ends the swiping motion and the display device stops sending turn instructions to the video camera.

Of course, some embodiments of the second camera control mode could include elevation control of the video camera. In that case, diagonal or vertical swipes on the touchscreen also control the vertical movements of the video camera. Because there is a possibility that careless tilting could turn the video camera to an unwanted direction, such as to sky, there could be some predetermined action to activate vertical movement. One example from these could be two consecutive vertical swiping motions in a predetermined time window. In figures 3 and 4 is also presented an icon 303 indicating which camera control mode is in use. The icon represents a camera, and it is in reverse video when the first camera control mode is use and normal when the second camera control mode is in use. The icon indicates that the first camera control mode is in use in figures 3A, 3B and 4A and the second camera control mode is in use in figure 4B. In figure 4A the user starts to swipe the touchscreen and thus the display device switches from the first camera control mode to the second camera control mode. In this example the camera control mode indication is implemented by an icon but there are a multitude of other arrangements for this.

When the display device 301 is using the first camera control mode it switches to the second camera control mode when the display device detects a swipe on the touchscreen 302. To prevent accidental mode switching there could be some additional gestures to activate mode switching. One could be for example a touching gesture where a finger is resting on the touchscreen for a predetermined time and if after that a swipe is detected without lifting the finger, the mode is changed from the first camera control mode to the second camera control mode. In figure 5 is presented an example how the camera control mode is switched from the second camera control mode back to the first control mode. The same display device 301 is in use that was depicted in figure 3. In figure 5A the second camera control mode is in use. The azimuth of the video camera is 39° and the heading of the vehicle is 350°. It must be noted that the orientation of the display device 301 could be arbitrary. In this example it is same as the heading of the vehicle. To switch the camera control mode from the second camera control mode to the first camera control mode the user re-orients the display device in a way that its horizontal orientation is the same as the azimuth of the video camera. In this case the top edge of the display device is perpendicular to the directional vector of the video camera. When the display device detects this situation, it switches from the second camera control mode to the first camera control mode, where the directional movements of the video camera are slaved to the orientation of the display device. It must be noted that the orientation detector arrangement is active also in the second camera control mode.

To prevent the accidental mode switching some embodiments could include some additional action or actions to activate mode switching. One example could be a single tap on the touchscreen when the orientation of the display device is close enough to the direction of the video camera. This could be a sector of 5° that in- eludes the azimuth of the video camera. The camera azimuth could be in the middle of the sector or this sector is from the camera azimuth towards the orientation of the display device. When the orientation of the display device is in this sector, the user can execute the action and switch modes.

Figure 6 presents an example of an embodiment where one to several objects are being actively tracked. There is a display device 601 having a touchscreen 602. The display device is in this example identical to the display device described in figure 3. The display device includes some embodiments of previously described camera control modes.

In figure 6A a user operates the display device 601 in the second camera control mode. He or she moves the viewing direction so that a sailing boat is centred on the touchscreen 602. A rangefinder activation area 603 is superimposed onto the touchscreen image. The user adjusts the view on the display device to place the sailing boat inside the rangefinder activation area. Then the user activates a range- finder by tapping the rangefinder activation area. The rangefinder is co-located with the video camera or it could be installed somewhere else on the vessel. The area or the object that is inside the rangefinder activation area on the image displayed on the display device serves as an aimpoint. The rangefinder measures the distance to the aimpoint. Activating the rangefinder also activates tracking of the aimpoint area. In figure 6B the viewing direction is changed but the previously activated aim- point area i.e. the sailing boat, is tracked both visually with tracked target markings and with alphanumeric information. If a tracked target disappears from the view, the alphanumeric information still shows the bearing and the range of the tracked target. It is also possible to track more targets simultaneously. The user is ready in figure 6B to track a second target. The tracked targets can be removed with some predetermined action. This action could be for example tapping the tracked target. Figure 7 shows an example of an embodiment as a flow chart. Hereby the method is described step by step.

The method for using a system for displaying real-time video images from around a vehicle is started at step 7.0. The system comprises a display device and at least one video camera. The display device in this embodiment is a tablet computer. The tablet computer is portable and has a touchscreen. The tablet computer has a means for controlling the directional movements of the video camera.

At step 7.1 the system is initialized. The active camera is chosen in that step. The camera control mode is set to the first camera control mode.

At step 7.2 it is checked if the orientation of the tablet is changed. If no change is detected, it is checked at step 7.9 if a horizontal swipe is detected. If no swipes are detected the active camera is selected if there is more than one video camera. The active camera is the camera that was used last or was chosen in the initialization step. At this step the azimuth of the active camera is adjusted with control instructions sent by the tablet computer. If there are no new control instructions, the direc- tion of the camera is left unchanged. Also, the elevation of the camera could be adjusted in this step. At step 7.7. an image from the active camera is captured. At step 7.7 a compound image is prepared. At that step additional images, icons, information and such are added to the image. At step 7.8. the compound image prepared at step 7.7 is displayed on the tablet computer. Then it is checked at step 7.2 if the orientation of the tablet computer has changed.

If at step 7.1 it is detected that the orientation of the tablet computer has not changed but at step 7.9 a swipe is detected, then execution moves to step 7.10. At step 7.10 the camera view is unlocked i.e. camera slaving is removed, if it was active beforehand. This means that a switch is made from the first camera control mode to the second camera control mode. Then steps 7.4 to 7.8 are executed as previously described.

If at step 7.1 it is detected that the orientation of the tablet computer has changed, it is checked at step 7.3 if the camera view is locked i.e. the camera is slaved to the orientation of the tablet computer. If the camera view is not locked it is checked at step 7.11 if the orientation of the tablet computer and the direction of the video camera match. If they match, the camera view is locked, i.e. the directional movements of the camera are slaved to the orientation of the tablet, at step 7.12. This means that it is switched from the second camera control mode to the first camera control mode. If the orientation of the tablet and the direction of the camera do not match, then steps 7.6 - 7.8 are executed as described previously.

If at step 7.2 it is detected that the orientation of tablet computer has changed and at step 7.3 it is checked that the camera view is locked, then then steps 7.4 to 7.8 are executed as described previously. Above, some preferred embodiments according to the invention have been described. The invention is not limited to the solutions described above, but the inventive idea can be applied in numerous ways within the scope of the claims.

Claims

1. A method for displaying real-time video images from around a vehicle, the method comprising steps where:

- the video images are captured by a video camera mounted directionally controllable on the vehicle (7.6);

- the video images are transferred to a portable display device and the display device has a touchscreen for displaying the video images (7.8);

- the directional movements of the video camera are slaved to the orientation of the display device (7.12);

- when a user swipes the touchscreen the video camera is no longer slaved to the orientation of the display device, and the directional movements of the video camera are controlled by swiping the touchscreen (7.5); and

- the directional movements of the video camera are slaved to the orientation of the display device once more, when the user performs a predetermined action with the display device (7.11 ).

2. The method according to claim 1 , wherein the predetermined action is setting the display device orientation to the same direction as that of the video camera.

3. The method according to any of claims 1-2, wherein on the touchscreen is an area that activates a rangefinder that measures the range to an aimpoint.

4. The method according to claim 3, wherein activating the rangefinder also activates tracking of the aimpoint area.

5. The method according to any of claims 1 to 4, wherein on the touchscreen is displayed an icon indicating if the directional movements of the video camera are slaved to the orientation of the display device or if the directional movements of the video camera are controlled by swiping the touchscreen.

6. A system for displaying real-time video images from around a vehicle (101 ; 201 ) comprising:

- a video camera (102; 202, 203, 204) that is configured to be mounted direc- tionally controllable on the vehicle;

- a display device (301 ; 601 ) that is configured to receive a video feed from the video camera, and the display device comprises a touchscreen (302; 602) for displaying the video feed, an arrangement for detecting the orientation of the display device and an arrangement for sending directional movement instructions to the video camera, and the display device has a first camera control mode and a second camera control mode, and in the first camera control mode the directional movements of the video camera are slaved to the orientation of the display device, and in the second camera control mode swiping the touchscreen is configured to control the directional movements of the video camera, and the display device is configured to switch from the first control mode to the second control mode when a swipe is detected on the touchscreen, and switch from the second camera control mode to the first camera control mode when the display device is detecting movement or has detected movement and the display device is set to a position where the orientation of the display device and the direction of the video camera are matching.

7. The system according to claim 6 wherein in the system is a rangefinder that is configured to be activated by tapping a predetermined area (603) on the touchscreen (302; 602) and the distance is calculated to a target that is on the video feed in the predetermined area, and activating the rangefinder also activates tracking of the target.

8. The system according to claim 6 or 7 wherein the system comprises at least one additional video camera (203, 204) producing at least one an additional video feed and when changing the view on the display device (301 ; 601 ) the video feed is changed accordingly.

9. The system according to any of claims 6 to 8 wherein the display device (301 ; 601 ) is configured to display an icon (303) indicating which camera control mode is in use.

10. The display device (301 ; 601 ) for displaying real-time video images from around a vehicle (101 ; 201 ), which display device is configured to display a video feed of a video camera (102; 202, 203, 204) that is mounted directionally controlla- ble on the vehicle, and the display device is portable and comprises a touchscreen (302; 602), and the display device has a first camera control mode and a second camera control mode, and in the first camera control mode the directional movements of the video camera movements are slaved to the orientation of the display device, and in the second camera control mode swiping the touchscreen is config- ured to control the directional movements of the video camera, and the display device is arranged to switch from the first camera control mode to the second camera control mode when a swiping gesture is detected on the touchscreen, and switch from the second camera control mode to the first camera control mode when the display device is detecting movement or has detected movement and the display device is set to a direction where the orientation of the display device and the azi- muth of the video camera are matching.