KR20170089228A

KR20170089228A - Method for controlling a virtual reality head mounted display using smart phone

Info

Publication number: KR20170089228A
Application number: KR1020160009373A
Authority: KR
Inventors: 이재민
Original assignee: 이재민
Priority date: 2016-01-26
Filing date: 2016-01-26
Publication date: 2017-08-03

Abstract

The present invention relates to a method of controlling a virtual reality head mounted display using a smart phone, and when a user wears an HMD (Head Mounted Display) equipped with a smart phone and reproduces VR (Virtual Reality) contents, The control unit detects the user's line of sight, and the control unit determines whether the detected line of sight satisfies a first condition set in advance. If the detected line of sight satisfies the first condition set in advance, The method comprising the steps of: displaying a button at a predetermined position on the screen; detecting, by the control unit, a line of sight of the user while the interaction button is displayed; Judging whether the condition is satisfied, and judging whether the detected visual line satisfies a second condition If the control unit determines that the corresponding interaction button is selected, the control unit changes the shape or color of the button, and if the selected specific interaction button is not canceled, controlling the VR content with the function set on the selected button.

Description

TECHNICAL FIELD [0001] The present invention relates to a method for controlling a virtual reality head mounted display using a smart phone,

The present invention relates to a method of controlling a virtual reality head mounted display using a smart phone, and more particularly, to a method of controlling a virtual reality head mounted display using a smart phone, in which an interaction button is additionally displayed at the time of content reproduction, To a control method of a virtual reality head mounted display using a smart phone.

In general, virtual reality (VR) means a human-computer interface that makes a certain environment or situation computerized and makes the person who uses it behave as if they are interacting with the actual environment. do. The virtual reality (VR) is a concept including an artificial reality, a cyber space, a virtual world, a virtual environment, a synthetic environment, and an artificial environment.

The virtual reality (VR) system applying this concept is a system that allows people to show and manipulate the environment as if they are in the environment without experiencing the environment that is difficult for people to experience on a daily basis.

The virtual reality system senses a change in the user's viewpoint or motion and can give appropriate changes corresponding to the change to the virtual environment. In order to enhance the sense of presence of the user, a stereoscopic display device or a head mounted display (HMD) Are typically used.

In particular, the above-mentioned head-mounted display (HMD) has merely been attached to a monitor in front of the user, but recently it has developed into a higher level device through various sensor technologies and wireless technologies. For example, when the user turns his head to the left, the scene displayed on the screen also moves to the left. That is, the head movement of the user can be reflected in the content manipulation. That is, the HMD is an output device and simultaneously recognizes and reflects the user's motion (ie, head, finger, body motion, etc.).

In addition, HMDs that do not even require a display are beginning to emerge. That is, instead of mounting the display, the user's smartphone is used as a display. The smartphone is a device equipped with a high resolution display, a wireless communication device, various sensors (e.g., gyroscope, accelerometer, etc.), a storage device for storing contents, In fact, most of the parts and functions needed for the HMD are available.

However, in most cases, no separate interaction is applied to VR contents that can be watched and watched with HMD. Therefore, in order to change the image viewed by the user to another image or to stop the image of the currently viewed content, it is necessary to cancel the wearing of the HMD and to operate the smartphone after separating the HMD and the smart phone .

Accordingly, in order to solve the above-mentioned troublesome problems, there is a method of using the magnet in the case of the interaction provided by the HMD itself, but when it is not operated according to the user's terminal (e.g. smart phone) (For example, HMD or smartphone) application (APP), which can be normally operated in conjunction with a user terminal (for example, a smart phone) There is a problem that it can be used only when viewing the VR content operating in the Internet.

Therefore, even if a separate additional device (for example, a remote controller, a sensor for HMD, a communication device, etc.) that requires interlocking is not attached to the HMD, the camera of the smartphone is used as an input device, There is a need for an interaction method capable of simultaneously controlling the reproduction of the VR content while displaying it.

The background art of the present invention is disclosed in Korean Patent No. 10-1530788 (Registered, 3D HMD system using Smartphone).

According to an aspect of the present invention, there is provided an information processing apparatus, which is created in order to solve the above-described problems, and is characterized in that an interaction button is additionally displayed at the time of content reproduction, and a button is selected using the user's eyes, The present invention provides a control method of a virtual reality head mounted display using a smart phone.

A method of controlling a virtual reality head mounted display using a smartphone according to an aspect of the present invention includes: when a user wears a head mounted display (HMD) equipped with a smartphone and then plays VR (Virtual Reality) According to the algorithm, the control unit detects the user's gaze; Determining whether the detected line of sight satisfies a predetermined first condition; Displaying the interaction button at a predetermined position on the screen if the detected sight line satisfies a first condition set in advance; The control unit detecting a user's gaze while the interaction button is displayed; Determining whether the detected user's gaze satisfies a predetermined second condition when the interaction button is displayed; And if the detected line of sight satisfies a predetermined second condition, the control unit determines that the corresponding interaction button is selected and changes the shape or color of the button, and if the selected specific interaction button is not canceled, And controlling the VR content.

In the present invention, the interaction button may include at least one of a home button, a play button, a stop button, a cancel button, a forward button, and a backward button, and the interaction button may include a pull- And an interaction menu.

In the present invention, the first condition may be that the user's gaze gazes at a predetermined direction for a predetermined time or more, the user's gaze moves in a pattern in which a pre-designated pupil or eyelid moves, And at least one of the following cases.

In the present invention, the second condition may include at least one of a case where the user's gaze gazes at a predetermined direction for a predetermined time or more, or a case where the user's gaze moves in a pattern in which a predetermined eyeball or eyelid moves .

In the present invention, the interaction buttons are OSD (On Screen Display) type buttons.

In the present invention, if no predetermined line of sight is detected after the interaction button is displayed, the controller removes the display of the interaction button after a preset time.

The method may further include performing at least one set-up for matching the position of the interaction button displayed on the screen with the user's gaze, if the control information is stored in the internal memory, The controller does not perform the additional setup, and if the setup information is not stored, the controller performs the setup at least once.

According to an aspect of the present invention, an interactive button is additionally displayed at the time of reproducing a content, and a button is selected using a user's line of sight to control the reproduction of the content.

1 is a view illustrating a configuration of a virtual reality head-mounted display device using a smartphone according to an embodiment of the present invention;
BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a virtual reality head mounted display device using a smart phone, and more particularly,
3 is a flowchart illustrating a method of controlling a virtual reality head mounted display using a smartphone according to an embodiment of the present invention.

Hereinafter, a method of controlling a virtual reality head-mounted display using a smartphone according to the present invention will be described with reference to the accompanying drawings.

In this process, the thicknesses of the lines and the sizes of the components shown in the drawings may be exaggerated for clarity and convenience of explanation. In addition, the terms described below are defined in consideration of the functions of the present invention, which may vary depending on the intention or custom of the user, the operator. Therefore, definitions of these terms should be made based on the contents throughout this specification.

1 is a view illustrating a configuration of a virtual reality head-mounted display device using a smartphone according to an embodiment of the present invention.

As shown in FIG. 1, a virtual reality head mounted display device using a smart phone is equipped with a smart phone for displaying VR contents on the front side of the HMD.

For reference, in the present embodiment, a smart phone means a portable smart terminal of a user, and means a portable terminal capable of adding and deleting functions by downloading and installing an application by a wireless communication method.

Each of the smart phones is equipped with a camera 120 on the front side (e.g., a direction with a display part) and on the rear side thereof. The cameras 120 mounted on the front and rear surfaces may have different resolutions, and at least one may be mounted on the front surface and the rear surface, respectively. For example, a dual camera may be mounted on the front and rear, respectively.

In order to prevent the camera 120 mounted on the smartphone from being hidden when the smartphone is mounted on the HMD, a camera window or a hole (Hall) is formed on the HMD ) May be formed.

In the present embodiment, the camera 120 is used as a sensor (or an input device) for sensing a user's gaze (or eye and eyelid movement). For example, the camera 120 detects that the pupil of the wearer wearing the HMD is moving upward, downward, leftward, and rightward, or detects that the eyelid is wound or opened (i.e., eyes open or closed).

The internal control unit 110 of the smart phone may set the corresponding condition set through an algorithm (for example, an algorithm set through an app (APP)) according to the embodiment (e.g., a smartphone mounted on the HMD, (E.g., wearing of an HMD or the like), the user's eyes (i.e., the movement of eyes and eyelids) are detected through the camera 120, and information on the eyes of the user And controls the reproduction of the VR content on the basis of this.

For example, when the user wears a virtual reality head-mounted display device using a smart phone, the controller 110 detects the user's gaze (i.e., eye and eyelid movement) using the camera 120, The control unit 110 displays an interaction button (e.g., an OSD (On Screen Display) button) when the user gazes at a specified direction for a predetermined time or moves the eyes of the user in a predetermined pattern (e.g., a pattern for moving a pupil or an eyelid) Is displayed at a predetermined position (e.g., the lower end) on the screen (see Fig. 2).

FIG. 2 is a view illustrating a screen in which an interaction button is displayed on the lower side of a virtual reality head-mounted display device using a smartphone according to an embodiment of the present invention.

At this time, the interaction button (e.g., OSD (On Screen Display) button) may be immediately displayed and continuously displayed when a user wears a virtual reality head-mounted display device using a smartphone according to a predetermined algorithm.

In addition, the above-described interaction button (e.g., an OSD (On Screen Display) button) may be configured such that when the user's head movement is detected according to a predetermined algorithm, Button) may be displayed at a predetermined position (e.g., lower side) on the screen.

Here, the interaction buttons include at least one of a home button, a play button, a stop button, a forward button, a backward button, and a cancel button.

If no input (e.g., eye movement) is detected after the interaction button is displayed, the control unit 110 removes the display of the interaction button after a preset time, and displays it again when a preset condition is reached have.

Meanwhile, in a state in which the interaction button is displayed as described above, the controller 110 displays a button (for example, a home button, a playback button, or the like) that gazes at a user's gaze Button, a stop button, a cancel button, a forward button, a backward button, etc.).

When the interaction button that the user is looking at is detected as described above, the controller 110 changes the shape or color of the detected button so that the user can recognize that the interaction button being examined is correctly detected.

Accordingly, if the interaction button that the user is examining is not correctly detected, the user can cancel the selection of the button (i.e., the gazing of the button) and select again. For example, if the user is gazing at a specific interaction button and then changes his or her gaze to another position, direction, or button before a predetermined time elapses, or moves the head in another direction, the controller 110 selects the corresponding interaction button (Ie, deselect the corresponding interaction button).

Meanwhile, in the present embodiment, the interaction button is a concept including an interaction menu such as a pull-up method, a pull-down method, and a pop-up method.

In order to match the positions of the user's eyes (i.e., eyes and eyelids) with the positions of the interaction buttons (or interaction menus) displayed on the screen (i.e., to accurately detect the buttons to be looked at by the user) ) You can perform at least one setup setup process.

If the specific interaction button selected by the user's eyes is selected without being canceled, the controller 110 controls the VR contents with the function set on the selected button.

3 is a flowchart illustrating a method of controlling a virtual reality head-mounted display using a smartphone according to an embodiment of the present invention.

As shown in FIG. 3, the controller 110 may be configured to match the positions of the user's eyes (i.e., the eyes and the eyelids) with the positions of the interaction buttons (or interaction menus) displayed on the screen (For detecting the button correctly) (S101).

The set-up for matching the interaction button position and the user's gaze is performed at least once. If the setup information is stored, the controller 110 may not perform additional setup. However, if the setup information is not stored, the controller 110 may forcibly display a setup screen.

When the next user plays the VR content after wearing the HMD equipped with the smartphone, the control unit 110 detects the user's line of sight according to a predetermined algorithm (e.g., an algorithm set through the APP) S102).

That is, when the VR content is reproduced, the control unit 110 detects the user's gaze (i.e., eye and eyelid movement).

Next, the controller 110 determines whether the detected line satisfies a predetermined first condition (S103).

For example, the first condition may be that the user's gaze gazes at a predetermined direction for a predetermined time or more, the user's gaze moves in a predetermined pattern (for example, a pupil or an eyelid movement pattern) A case in which a real head-mounted display device is worn, or a head movement of a pre-designated user is detected.

If the detected line of sight satisfies the first condition (S103), the control unit 110 displays an interaction button (e.g., OSD (On Screen Display) button) at a predetermined position (S104).

Also, the controller 110 detects the user's gaze in a state where the interaction button is displayed (S105).

If no input (e.g., eye movement) is detected after the interaction button is displayed, the controller 110 may remove the display of the interaction button after a preset time.

In operation S106, it is determined whether the user's gaze (i.e., the movement of the eyes and the eyelids) detected in the state where the interaction button is displayed satisfies a preset second condition.

For example, the second condition includes at least one of a case where the user gazes at a specific button for a predetermined time (e.g., 3 seconds) or a case where the user's gaze moves in a predetermined pattern (e.g., a pattern for moving a pupil or an eyelid) can do.

If the detected line of sight satisfies the predetermined second condition (YES in S106), the controller 110 determines that the corresponding interaction button is selected and changes the shape or color of the button, and if the selected specific interaction button is not canceled The VR contents are controlled by the function set in the selected button (S107).

Here, the interaction button includes at least one of a home button, a play button, a stop button, a cancel button, a forward button, and a back button, and the interaction button may include an interaction menu such as a pull-up method, a pull- It is a concept to include.

As described above, the present embodiment can easily add or delete interaction buttons of various functions without requiring a magnet and a separate remote control device, and it is necessary to consider a separate additional device for interlocking with a smart phone or an HMD owned by the user There is an effect that any smartphone or HMD in which the VR content is being reproduced can be easily controlled using the user's gaze.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, I will understand the point. Accordingly, the technical scope of the present invention should be defined by the following claims.

110:
120: camera

Claims

When a user wears a head mounted display (HMD) equipped with a smartphone and then plays VR (Virtual Reality) content, the control unit detects the user's gaze according to a predetermined algorithm;
Determining whether the detected line of sight satisfies a predetermined first condition;
Displaying the interaction button at a predetermined position on the screen if the detected sight line satisfies a first condition set in advance;
The control unit detecting a user's gaze while the interaction button is displayed;
Determining whether the detected user's gaze satisfies a predetermined second condition when the interaction button is displayed; And
If the detected visual line satisfies the second predetermined condition, the controller determines that the corresponding interaction button is selected and changes the shape or color of the button. If the selected specific interaction button is not canceled, the VR And controlling the contents of the virtual reality head-mounted display.

The method according to claim 1,
A home button, a playback button, a stop button, a cancel button, a forward button, and a back button,
Wherein the interaction button is a concept including an interaction menu such as a pull-up method, a pull-down method, a pop-up method, and the like.

2. The method of claim 1,
A case where the user's gaze gazes at a predetermined direction for a predetermined time or more, or a case where the user's gaze moves in a pattern in which a predetermined eyeball or eyelid moves, or a head movement of a designated user is detected, A method for controlling a virtual reality head mount display using a smart phone.

2. The method of claim 1,
And a case where the user's gaze gazes at a predetermined direction for a predetermined time or more, or a case where the user's gaze moves in a pattern for moving a predetermined eyeball or eyelids. / RTI >

The method according to claim 1,
Further comprising performing at least one set-up for matching the position of the interaction button displayed on the screen with the user's line of sight,
Wherein if the setup information is stored in the internal memory, the control unit does not perform additional setup, and if the setup information is not stored, the control unit forces the setup at least once. Control method of virtual reality head mount display.