WO2020016969A1 - Information processing device, information processing method, and program - Google Patents

Information processing device, information processing method, and program Download PDF

Info

Publication number
WO2020016969A1
WO2020016969A1 PCT/JP2018/026940 JP2018026940W WO2020016969A1 WO 2020016969 A1 WO2020016969 A1 WO 2020016969A1 JP 2018026940 W JP2018026940 W JP 2018026940W WO 2020016969 A1 WO2020016969 A1 WO 2020016969A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
interest
information processing
processing apparatus
video
Prior art date
Application number
PCT/JP2018/026940
Other languages
French (fr)
Japanese (ja)
Inventor
英彦 小笠原
洋一 西牧
平田　真一
Original Assignee
株式会社ソニー・インタラクティブエンタテインメント
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社ソニー・インタラクティブエンタテインメント filed Critical 株式会社ソニー・インタラクティブエンタテインメント
Priority to PCT/JP2018/026940 priority Critical patent/WO2020016969A1/en
Publication of WO2020016969A1 publication Critical patent/WO2020016969A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer

Definitions

  • the present invention relates to an information processing device, an information processing method, and a program for performing control for displaying a video on a video display device.
  • the present invention has been made in view of the above circumstances, and one of its objects is to provide an information processing apparatus capable of specifying a position where a user different from a gazing point is potentially interested.
  • An information processing method and a program are provided.
  • the information processing device is a gaze direction identification unit that acquires information that identifies a gaze direction of a user who is browsing a video displayed on a display device, and a movement of a specific pattern included in a temporal change in the gaze direction.
  • An interest position estimating unit for estimating an interest position and / or an interest direction to which the user is potentially interested, which is different from the position where the user's gaze is directed.
  • the line-of-sight direction specifying unit obtains information for specifying the line-of-sight direction of the user who is browsing the video displayed by the display device, and the interest position estimating unit determines the time of the line-of-sight direction. Estimating, based on the movement of the specific pattern included in the change, an interest position and / or an interest direction in which the user is potentially interested, which is different from the position where the user is looking And characterized in that:
  • the program according to the present invention includes a step of acquiring information for identifying a gaze direction of a user who is browsing an image displayed by a display device, and the user based on a movement of a specific pattern included in a temporal change of the gaze direction. Estimating an interesting position and / or a direction of interest that the user is potentially interested in, which is different from the position where the user is looking.
  • This program may be stored in a computer-readable non-transitory information storage medium and provided.
  • FIG. 1 is a diagram illustrating an overall outline of a video display system including an information processing device according to an embodiment of the present invention. It is a block diagram of a structure of a video display system. It is a figure showing a situation on the back side of a display.
  • FIG. 3 is a diagram illustrating an internal configuration of a display device. It is a functional block diagram of an information processing device. It is a figure showing an example of a temporal change of a gaze direction.
  • FIG. 7 is a diagram illustrating an example of a flow of a process executed by the information processing device according to the embodiment of the present invention. It is a figure showing the state before change in the example which changes the display mode of the target according to the direction of interest. It is a figure showing a state after change in an example which changes a display mode of a target according to a direction of interest.
  • FIG. 11 is a diagram illustrating an example in which fovea rendering is performed according to an interesting direction.
  • FIG. 1 is an overall schematic diagram of a video display system 1 including an information processing device 10 according to an embodiment of the present invention.
  • FIG. 2 is a configuration block diagram showing the configuration of the video display system 1.
  • the video display system 1 includes an information processing device 10, a camera 20, an operation device 30, and a head-mounted display device 40.
  • the information processing device 10 is a device that supplies a video to be displayed by the display device 40, and may be, for example, a home game machine, a portable game machine, a personal computer, a smartphone, a tablet, or the like. As shown in FIG. 2, the information processing device 10 includes a control unit 11, a storage unit 12, and an interface unit 13.
  • the control unit 11 includes at least one processor such as a CPU, and executes various programs by executing programs stored in the storage unit 12. Note that a specific example of the processing executed by the control unit 11 in the present embodiment will be described later.
  • the storage unit 12 includes at least one memory device such as a RAM, and stores a program executed by the control unit 11 and data processed by the program.
  • the interface unit 13 is an interface for data communication between the camera 20, the operation device 30, and the display device 40.
  • the information processing device 10 is connected to each of the camera 20, the operation device 30, and the display device 40 via the interface unit 13 by wire or wirelessly.
  • the interface unit 13 includes a multimedia interface such as HDMI (registered trademark) (High-Definition Multimedia Interface) in order to transmit video and audio data supplied by the information processing device 10 to the display device 40.
  • a data communication interface such as a USB (Universal Serial Bus) for receiving video data captured by the camera 20 and a signal indicating the content of a user's operation input to the operation device 30.
  • USB Universal Serial Bus
  • the camera 20 is installed in front of a user who uses the video display system 1 and captures an image including the user and its surroundings. As will be described later, the information processing apparatus 10 specifies the position of the display device 40 in the real space by analyzing the image captured by the camera 20.
  • the operation device 30 is a device for receiving an operation input from a user, and has an operation member such as an operation button disposed on a surface thereof. The user performs an operation input on the operation member by holding the operation device 30 with a hand or mounting the operation device 30 on the hand. The operation device 30 transmits to the information processing apparatus 10 an operation signal indicating the content of the operation input performed by the user.
  • the display device 40 is a video display device worn by the user on the head and used, and displays a video corresponding to a video signal transmitted from the information processing device 10 and allows the user to browse.
  • a light emitting element is disposed on the front surface of the display device 40.
  • the camera 20 captures the light emitted by the light emitting element, and the information processing device 10 analyzes the captured image, so that the information processing device 10 can specify the position of the display device 40 in the real space.
  • a plurality of light emitting elements are provided on the front surface of the display device 40, and the information processing apparatus 10 specifies a change in the posture of the display device 40 by specifying the positional relationship between these light emitting elements. It shall be.
  • the information processing apparatus 10 can specify the change in the direction.
  • the display device 40 includes a mechanism for identifying a change in the user's line of sight in real time.
  • FIG. 3 is a diagram illustrating a state on the back side of the display device 40.
  • FIG. 4 is a diagram schematically illustrating an internal configuration of the display device 40, and illustrates a positional relationship of components arranged for specifying the orientation of one pupil of the user.
  • the display device 40 includes a support 41, a plurality of LEDs 42, a lens 43, a hot mirror 44, and an eyeball camera 45 in order to specify the direction of the user's line of sight.
  • an annular support 41 is disposed on the back of the display device 40 at a position facing the left and right eyes of the user.
  • a plurality of LEDs 42 are arranged on each of the left and right supports 41. Each of these LEDs 42 is fixed in a direction in which the user's pupil can be irradiated with infrared rays.
  • the hot mirror 44 is an optical element having a property of transmitting visible light and reflecting infrared light.
  • the image displayed on the display element of the display device 40 is presented to the user through the hot mirror 44 and the lens 43.
  • light emitted from the LED 42 is reflected by the user's pupil, passes through the lens 43, is further reflected by the hot mirror 44, and enters the eyeball camera 45. That is, the eyeball camera 45 captures an image of the user's eyeball in which the light spot due to the irradiation of the LED 42 is captured.
  • the position of the light spot on the surface changes according to the movement of the user's eyeball. Therefore, the information processing apparatus 10 can identify the direction of the user's pupil (the direction of the line of sight) by analyzing the distribution of light spots included in the image captured by the eyeball camera 45.
  • the image display system 1 may include one or more vital sensors for detecting biological information of the user.
  • a vital sensor is connected to the information processing device 10 and transmits the detected biological information of the user to the information processing device 10 in real time.
  • Examples of information that can be acquired by the vital sensor include brain waves, heartbeats, sweating, body temperature, saliva, subcutaneous blood flow, and the like.
  • the vital sensor in this case may be arranged on the display device 40. Since the display device 40 is mounted on the user's head, by arranging the vital sensor at a position on the surface of the display device 40 that contacts the user's head, biological information such as the user's brain waves can be detected. .
  • the eyeball camera 45 may be used as a vital sensor.
  • the eye camera 45 captures an image around the user's eye at a relatively high resolution and a high frame rate. Therefore, by analyzing the image captured by the eyeball camera 45, it is possible to measure the time change of the thickness of the blood vessel near the user's eyeball.
  • the information processing device 10 can estimate a change in the blood flow volume of the user.
  • the user's heart rate can be measured by specifying a periodic change in blood flow.
  • the information processing apparatus 10 functionally includes a gaze direction identification unit 51, an interest position estimation unit 52, and a video display control unit 53. These functions are realized by the control unit 11 executing a program stored in the storage unit 12. This program may be provided to the information processing apparatus 10 via a communication network such as the Internet, or may be provided by being stored in a computer-readable information storage medium such as an optical disk.
  • the line-of-sight direction specifying unit 51 specifies a change in the line-of-sight direction of the user while the user is viewing an image displayed on the display device 40 by the image display control unit 53 described later. Specifically, the line-of-sight direction identification unit 51 transmits a control signal for operating the above-described LED 42 to the display device 40, and the eyeball camera 45 captures an image of the user's pupil while the LED 42 is emitting infrared light. To obtain a captured image. Then, using this captured image, the user's line-of-sight direction (direction of the point of gaze) at the time when the image is captured is specified.
  • the line-of-sight direction identification processing itself can be realized by a known method.
  • the gaze direction identification unit 51 repeatedly executes such gaze direction identification processing at short time intervals, and stores the execution result in the storage unit 12. Thereby, the gaze direction identification unit 51 can record the trajectory of the user's gaze movement (time change in the gaze direction) while monitoring the gaze direction.
  • the interest position estimating unit 52 is based on the time change of the past user's line of sight specified by the line of sight specifying unit 51, and the potential of the user is different from the direction of the user's current gaze (gaze point). Estimating the position and / or direction of interest in. In the following, such a position where it is estimated that a user other than the point of interest is interested is referred to as an interest position.
  • the direction in which the user is presumed to be interested is called an interest direction.
  • the direction of interest is a direction from the point of gaze to the position of interest in the image plane.
  • the video display control unit 53 draws a video to be presented to the user, and outputs a video signal indicating the content to the display device 40.
  • the video display control unit 53 draws a space image indicating a state in the virtual space where the virtual objects are arranged, and causes the display device 40 to display the space image.
  • the video display control unit 53 analyzes the image captured by the camera 20 to specify the operation, and identifies the user's head direction. The direction of the viewpoint set in the virtual space is moved so as to interlock with the change of the viewpoint.
  • the user can browse the state in the virtual space while moving his / her head and changing the direction of the viewpoint.
  • the user can look over a range that can be displayed on the display device 40 at a time. Therefore, the user may turn his or her consciousness to the position where the object that was displayed in the past but is no longer displayed on the display device 40 according to the movement of the user's head is displayed. That is, the interest position estimated by the interest position estimation unit 52 does not necessarily need to be a position in the video currently displayed, and may be a position outside the screen.
  • the video display control unit 53 changes the content of the video supplied to the display device 40 based on the interest position specified by the interest position estimation unit 52.
  • the video display control unit 53 draws a video in which the resolution of an area specified based on the user's point of interest and the position of interest is higher than other resolutions.
  • a specific example of the content of the video drawn by the video display control unit 53 based on the position of interest will be described later.
  • the interest position estimation unit 52 estimates the interest position of the user based on the time change of the user's gaze direction recorded by the gaze direction identification unit 51 in the most recent past. Specifically, when the user's eyeball is pointed near a specific position (gaze point), a reciprocating motion of slightly moving from the position to another direction and returning immediately may be performed. Such eye movements are not performed intentionally by the user, but are usually performed unconsciously. As one kind of such eye movement, a so-called microsaccade is known. When such eye movements are repeatedly performed in a specific direction, there is a high possibility that the user is potentially paying attention (interesting) in that direction.
  • the interest position estimation unit 52 estimates the direction of interest based on the movement of the specific pattern. Further, it is estimated that there is an interesting position at the position indicated by the interesting direction.
  • FIG. 6 shows an example of a temporal change in the gaze direction of the user specified by the gaze direction specifying unit 51.
  • the horizontal axis in this figure indicates the angle ⁇ x in the horizontal direction and the vertical axis indicates the angle ⁇ y in the vertical direction
  • the solid line in the figure indicates the time change in the line of sight.
  • the user's point of gaze shifts from area A to area B to area C over time.
  • the user's line of sight repeats a fine reciprocating movement that moves rightward and back. It is estimated that the direction of this reciprocation is the direction of interest.
  • the user does not move the eyeball until the position of interest is located at the center of the visual field.
  • the interest position estimation unit 52 may estimate the interest position from the reciprocating motion of the latest past for some time.
  • the gazing point exists in the middle area A, it is estimated from the direction of the reciprocating movement that the direction of the broken line Da that is slightly downward toward the right is the direction of interest. Further, it is estimated that the directions of interest when the gazing point is present in each of the upper region B and the lower region C are also directions as shown by broken lines Db and Dc in the figure. If the position of interest does not change while the user's line of sight moves from the region A to the region B and then to the region C, the position X corresponding to the intersection of the three broken lines Da, Db and Dc is determined to be the position of interest. Presumed. Note that, even when a plurality of directions of interest specified at a plurality of gazing points do not intersect at a single point, it is possible to estimate that the vicinity of a position where such a plurality of directions of interest gather is an interesting position.
  • the interest position estimation unit 52 may specify the interest position with reference to the content of the video being displayed by the video display control unit 53. For example, in the above-described example of FIG. 6, in the state where the gazing point remains in the area A, it is not possible to specify where on the broken line Da the interested position is based only on the line-of-sight direction information. In this case, when an object (candidate object) that may attract the user's interest is displayed on the broken line Da, the interest position estimation unit 52 may estimate the display position of the object as the interest position. In order to perform such estimation, the interest position estimation unit 52 receives information indicating the position of the candidate object currently displayed on the screen from the video display control unit 53. Then, of the candidate objects being displayed, the display position of the candidate object located closest to the direction of interest is estimated as the interest position.
  • the image display control unit 53 may cause the display device 40 to display an image obtained by capturing an image of the real space, or an image prepared in advance, instead of drawing the state of the virtual space. Even in such a case, by analyzing the displayed video, an object that may appear in the image and may be of interest to the user may be specified as a candidate object.
  • the interest position estimation unit 52 may acquire information indicating the browsing status of the user, such as whether or not the user is concentrated on the video, and use the information to estimate the interest position. For example, by analyzing an image captured by the eyeball camera 45, the interest position estimation unit 52 can specify an event such as a timing at which the user blinks or a change in pupil diameter. In general, it is assumed that the user blinks before concentrating on the image, or that the pupil diameter increases as the degree of concentration increases. Therefore, by using such information obtained by observing the user's pupil, it is possible to estimate the degree of concentration (index value of how much the user is interested in the image) at that time.
  • the pupil diameter of a person is affected by changes in the luminance of the image that the person is watching. Furthermore, there are individual differences in how the luminance of the image affects the pupil diameter, and there are people who are susceptible to the luminance of the entire image and those who are susceptible to the luminance near the gazing point. , And the way in which it affects them depends on the type, such as the person in between. Therefore, the interest position estimating unit 52 obtains, for each user, information on the luminance of the image and information on the pupil diameter observed when the image is displayed, and uses the obtained information to determine the pupil of the user. Information indicating how the diameter changes depending on the luminance of the image may be specified.
  • the interest position estimating unit 52 causes the display device 40 to display images of various background colors and brightness prepared for calibration when each user starts using the display device 40. Then, the size of the pupil diameter of the user who is browsing these calibration images is measured, and the measurement result is used as teacher data to determine a reference value used when estimating the degree of concentration of the user. I do. In addition, even during actual use of the display device 40, the interest position estimation unit 52 periodically acquires the luminance of the displayed image and the measurement result of the pupil diameter as teacher data, performs machine learning, and performs user learning. An estimator for estimating the degree of concentration for each may be constructed.
  • the interest position estimation unit 52 may estimate the degree of concentration of the user using the measurement result of the vital sensor described above. In this case as well, machine learning is performed using the biological information measured by the vital sensor as teacher data, and how the biological information affects the degree of concentration is learned in advance for each user, so that the degree of concentration can be accurately estimated. It can be carried out.
  • the interest position estimation unit 52 may estimate the interest position using the information of the dominant eye. It is known that the left and right eyes of a person have a dominant eye as well as a dominant hand. By using this dominant eye information, for example, the two dominant eye estimation results obtained based on the left and right eye movements are weighted according to the dominant eye, and then both are estimated. By combining them, the user's interest position may be estimated. Alternatively, when the estimation results of the left and right interest positions are contradictory, the estimation result of the dominant eye may be preferentially adopted. Either of the user's dominant eyes may be received in advance from the user himself or may be estimated from the movement of the line of sight with respect to the user's video.
  • the interest position estimation unit 52 determines whether or not the user is gazing at a specific area based on a temporal change in the user's line of sight (S1). Specifically, if the moving range of the user's line of sight in the latest past predetermined period falls within a range of a predetermined size, it is estimated that the user is gazing at a specific area.
  • the interest position estimation unit 52 determines whether or not a specific pattern of eye movement has occurred based on the trajectory of the user's line of sight while gazing at the area. Is determined (S2).
  • the eye movement of the specific pattern may be a short distance reciprocating movement in the specific direction described above.
  • the interest position estimating unit 52 determines that the eye movement of the specific pattern has occurred when the reciprocating motion has been performed a predetermined number of times or more within the latest past predetermined period.
  • the interest position estimation unit 52 subsequently determines the concentration of the user at that time based on a given determination criterion (S3). Specifically, the interest position estimation unit 52 determines whether or not the user is concentrated on the image based on the blink frequency, the pupil diameter, and other various types of biological information as described above. When it is determined that the user is concentrated on the video, the process proceeds to S4, and a process of estimating the user's interest position is executed. On the other hand, if any of the determination conditions up to this point is not satisfied, the interest position estimation unit 52 returns to S1 and repeats the processing of S1 to S3 until each determination condition is satisfied.
  • S3 determination criterion
  • the reliability of the estimation result (an index value indicating how reliable the estimation result is) is also calculated. If the reliability is low (that is, it is not possible to accurately estimate how concentrated the user is), the process may proceed to S4.
  • the interest position estimation unit 52 determines whether the user is paying attention to a specific direction based on the eye movement data of the specific pattern detected in S2. It is determined whether or not it is (S4). If the direction of the reciprocating movement of the eyeball is not concentrated in a specific direction, it is assumed that the user is merely concentrated and is consciously focused on the current gazing point. Therefore, the interest position estimation unit 52 determines that the interest position does not exist (S5).
  • the interest position estimating unit 52 specifies the interest direction and the interest position based on the direction of the reciprocating motion and other conditions (such as position information of the candidate object being displayed on the screen) (S6). By performing such processing periodically, the video display control unit 53 can present a video in consideration of the position of interest to the user.
  • the video display control unit 53 may change the display mode between an object existing at an interest position or an interest direction and other objects.
  • an object existing near the position of interest or in a position close to the direction of interest may be highlighted, and its color, brightness, size, etc. may be changed.
  • the brightness may be reduced, the transparency may be increased, or the focus may be changed to a blurred display so that the object located at a position distant from the interest position becomes less conspicuous. According to such control, it is possible for the user to intentionally pay attention to an object that has been unconsciously attracted.
  • FIG. 8 shows an example of an image displayed by such control
  • FIG. 8A shows a state before the display is changed
  • FIG. 8B shows a state after the display is changed according to the direction of interest.
  • a plurality of targets T are displayed in the video, but the user's line of sight is directed to the target Tp, and the lower left direction from the target Tp is specified as the direction of interest D.
  • the lower left targets T4, T8, and T9 that are close to the direction of interest do not change between FIGS. 8A and 8B, but the other targets are the same in FIG. 8B. It has changed to be less noticeable.
  • the control is performed to make the object that matches the position of interest or the direction of interest stand out.
  • the video display control unit 53 may perform the reverse control. For example, in a game or the like, a target object that the user wants to pay attention to is displayed in the video. In this case, when it is estimated that the user is interested in a different position or direction from the target object, display control may be performed to gradually emphasize the target object.
  • the video display control unit 53 may realize foveated rendering in consideration of an interest direction or an interest position.
  • Foveal rendering is a method that draws a region of interest centered on the user's gaze point at a relatively high resolution, while drawing peripheral regions outside the region of interest at a relatively low resolution, thereby reducing the drawing load. It is a technique to suppress.
  • a range extending therefrom in the direction of interest is set as the attention area (that is, the center of the attention area is shifted from the point of interest to the direction of interest. By setting the position, it is possible to draw at high resolution a range that the user is truly interested in browsing or is likely to pay attention to.
  • the video display control unit 53 determines the attention area so as to include both the position of interest and the point of interest. May be.
  • the resolution of the peripheral region which is assumed to be of little interest to the user, is reduced.
  • the user may notice this reduction in resolution.
  • the video display control unit 53 interrupts fovea rendering or increases the resolution of the peripheral region. For example, control for improving image quality may be executed. According to such control, the drawing load can be suppressed while preventing the user from feeling uncomfortable.
  • the video display control unit 53 may change the update frequency of the image for each region. Also in this example, as in the second example, the drawing method of the displayed image is changed between the attention area assumed to be the user's attention and the surrounding area outside the attention area. However, instead of changing the resolution in a frame image displayed at a time for each region as in the second example, the video update frequency (frame rate) is changed for each region. For example, the video display control unit 53 controls the frequency of updating the frame image in the peripheral region to be half that of the region of interest, and controls the frequency of updating the video in the region of interest to be relatively higher than the frequency of updating the peripheral region.
  • the video display control unit 53 may preliminarily draw a region near the position of interest with high resolution. Also in this example, the video display control unit 53 performs the same control as the fovea rendering, but unlike the previous example, the attention area is determined based on the gazing point. On the other hand, when there is an interesting position that the user is interested in separately from the point of interest, it is expected that the user will move his / her line of sight to that position (that is, make the interested position a new point of interest). Therefore, the image display control unit 53 also prepares an image drawn at a high resolution in advance in the region near the interest position in the same manner as the region near the gazing point.
  • an image to be displayed on the display device 40 is generated using the image drawn in advance. According to such control, when the user moves the line of sight from the current point of interest to the position of interest, a video in which a new point of interest is drawn at a high resolution can be presented to the user immediately without waiting time. .
  • the video display control unit 53 may reduce the region of interest to be subjected to fovea rendering. If the user is presumed to be interested in a position that is far away from the current point of interest, the user changes the direction of his / her head and moves from the current point of interest to the point of interest at a relatively high speed. It is highly likely that you are trying to move the viewpoint from a distance. During such a large movement of the viewpoint, it is difficult to check the displayed image in detail over a wide range. Therefore, even if the area to be drawn at a high resolution is reduced, the user does not easily feel uncomfortable.
  • the image display control unit 53 determines that the viewpoint movement to such a distant interest position is started, the image display control unit 53 reduces the region of interest and, in parallel, the region near the destination interest position with high resolution. Drawing is performed in advance. This makes it possible to advance preparations for displaying an area of interest to the user at a high resolution while suppressing an increase in the amount of data transmission and an increase in the drawing load.
  • the video display control unit 53 may draw the position of interest with high resolution with priority.
  • processing similar to foveal rendering in which the resolution is changed for each region is executed.
  • the region near the position of interest is set as the region of interest to be drawn with high resolution regardless of the point of regard.
  • the video display control unit 53 may draw not only the vicinity of the interest position but also an area including the path from the current point of interest to the interest position at a high resolution.
  • the video display control unit 53 may further lower the resolution of the area other than the vicinity of the interest position or the route to the position.
  • the various processes described above may be applied in combination.
  • the frequency of updating the image may be higher than that of the other area.
  • the resolution is changed in two stages of the attention region and the peripheral region.
  • the present invention is not limited to this, and the image is divided into three or more regions.
  • the resolution and the update frequency may be changed for each area.
  • a position or a direction in which a user other than the point of interest is attracted is estimated based on a temporal change of the user's line of sight. can do. Furthermore, by presenting the user with an image that changes according to the position of interest and the direction of interest estimated in this way, the user can naturally guide the user's line of sight and draw drawing load while preventing the user from feeling uncomfortable. It is possible to realize a display in which the communication band is suppressed.
  • the change in the direction of the display device 40 is specified by the camera 20, but the change in the direction of the display device 40 may be specified by various other methods.
  • the method for specifying the change in the user's line of sight in the display device 40 is not limited to the method described above, and various methods can be used.
  • 1 image display system 10 information processing device, 11 control unit, 12 storage unit, 13 interface unit, 20 camera, 30 operation device, 40 display device, 41 support, 42 LED, 43 lens, 44 half mirror, 45 eyeball camera , 51 gaze direction identification unit, 52 interest position estimation unit, 53 video display control unit.

Abstract

An information processing device (10) which acquires information identifying the line of sight of a user who is viewing a video displayed by a display device (40), and which, on the basis of movement of a specific pattern included in the time changes of the line of sight, estimates an interesting position and/or an interesting direction in which the user is subconsciously interested, said interesting position being out of the line of sight of the user.

Description

情報処理装置、情報処理方法、及びプログラムInformation processing apparatus, information processing method, and program
　本発明は、映像表示装置に映像を表示させる制御を行う情報処理装置、情報処理方法、及びプログラムに関する。 The present invention relates to an information processing device, an information processing method, and a program for performing control for displaying a video on a video display device.
　ユーザーに映像を提示する場合において、ユーザーの瞳の向きを検出するなどの方法で、ユーザーの注視点（ユーザーの視線が向けられている位置）を特定する技術が知られている。このような技術によれば、映像内に登場する複数のオブジェクトのうちユーザーがどのオブジェクトに注目しているかなどを特定することができる。 (2) When presenting an image to a user, a technique is known that specifies the point of gaze of the user (the position where the user's line of sight is directed) by, for example, detecting the direction of the user's pupil. According to such a technique, it is possible to specify which object the user is paying attention to among a plurality of objects appearing in the video.
　上記従来例の技術では、ユーザーが明確に映像内の一点を注視している場合には、その位置を特定することができる。しかしながら、人の視野にはある程度の広さがあり、視線の向きも絶えず変動するので、ユーザーは視野の中心点以外の場所も視認しており、そのような場所に無意識に興味を向けている場合がある。例えば、ユーザーが今現在視線を向けている位置からは離れた位置に表示されているオブジェクトにも気をとられているなどのケースもあり得る。 (4) According to the above-described conventional technique, when the user is clearly gazing at a point in the video, the position can be specified. However, since a person's visual field has a certain size and the direction of the line of sight changes constantly, the user sees places other than the center point of the visual field, and is unconsciously interested in such places. There are cases. For example, there may be a case where the user is paying attention to an object displayed at a position distant from the position where the user is currently looking.
　本発明は上記実情を考慮してなされたものであって、その目的の一つは、注視点とは別のユーザーが潜在的に興味を向けている位置を特定することのできる情報処理装置、情報処理方法、及びプログラムを提供することにある。 The present invention has been made in view of the above circumstances, and one of its objects is to provide an information processing apparatus capable of specifying a position where a user different from a gazing point is potentially interested. An information processing method and a program are provided.
　本発明に係る情報処理装置は、表示装置が表示する映像を閲覧中のユーザーの視線方向を特定する情報を取得する視線方向特定部と、前記視線方向の時間変化に含まれる特定パターンの動きに基づいて、前記ユーザーの視線が向けられている位置とは別の、当該ユーザーが潜在的に興味を向けている興味位置、及び／又は興味方向を推定する興味位置推定部と、を含むことを特徴とする。 The information processing device according to the present invention is a gaze direction identification unit that acquires information that identifies a gaze direction of a user who is browsing a video displayed on a display device, and a movement of a specific pattern included in a temporal change in the gaze direction. An interest position estimating unit for estimating an interest position and / or an interest direction to which the user is potentially interested, which is different from the position where the user's gaze is directed. Features.
　本発明に係る情報処理方法は、視線方向特定部が、表示装置が表示する映像を閲覧中のユーザーの視線方向を特定する情報を取得するステップと、興味位置推定部が、前記視線方向の時間変化に含まれる特定パターンの動きに基づいて、前記ユーザーの視線が向けられている位置とは別の、当該ユーザーが潜在的に興味を向けている興味位置、及び／又は興味方向を推定するステップと、を含むことを特徴とする。 In the information processing method according to the present invention, the line-of-sight direction specifying unit obtains information for specifying the line-of-sight direction of the user who is browsing the video displayed by the display device, and the interest position estimating unit determines the time of the line-of-sight direction. Estimating, based on the movement of the specific pattern included in the change, an interest position and / or an interest direction in which the user is potentially interested, which is different from the position where the user is looking And characterized in that:
　本発明に係るプログラムは、表示装置が表示する映像を閲覧中のユーザーの視線方向を特定する情報を取得するステップと、前記視線方向の時間変化に含まれる特定パターンの動きに基づいて、前記ユーザーの視線が向けられている位置とは別の、当該ユーザーが潜在的に興味を向けている興味位置、及び／又は興味方向を推定するステップと、をコンピュータに実行させるためのプログラムである。このプログラムは、コンピュータ読み取り可能で非一時的な情報記憶媒体に格納されて提供されてよい。 The program according to the present invention includes a step of acquiring information for identifying a gaze direction of a user who is browsing an image displayed by a display device, and the user based on a movement of a specific pattern included in a temporal change of the gaze direction. Estimating an interesting position and / or a direction of interest that the user is potentially interested in, which is different from the position where the user is looking. This program may be stored in a computer-readable non-transitory information storage medium and provided.
本発明の実施の形態に係る情報処理装置を含む映像表示システムの全体概要を示す図である。1 is a diagram illustrating an overall outline of a video display system including an information processing device according to an embodiment of the present invention. 映像表示システムの構成ブロック図である。It is a block diagram of a structure of a video display system. 表示装置の背面側の様子を示す図である。It is a figure showing a situation on the back side of a display. 表示装置の内部構成を示す図である。FIG. 3 is a diagram illustrating an internal configuration of a display device. 情報処理装置の機能ブロック図である。It is a functional block diagram of an information processing device. 視線方向の時間変化の一例を示す図である。It is a figure showing an example of a temporal change of a gaze direction. 本発明の実施の形態に係る情報処理装置が実行する処理の流れの一例を示す図である。FIG. 7 is a diagram illustrating an example of a flow of a process executed by the information processing device according to the embodiment of the present invention. 興味方向に応じてターゲットの表示態様を変化させる例における変化前の状態を示す図である。It is a figure showing the state before change in the example which changes the display mode of the target according to the direction of interest. 興味方向に応じてターゲットの表示態様を変化させる例における変化後の状態を示す図である。It is a figure showing a state after change in an example which changes a display mode of a target according to a direction of interest. 興味方向に応じて中心窩レンダリングを行う例について説明する図である。FIG. 11 is a diagram illustrating an example in which fovea rendering is performed according to an interesting direction.
　以下、本発明の実施の形態について、図面に基づき詳細に説明する。 Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
　図１は、本発明の一実施形態に係る情報処理装置１０を含む映像表示システム１の全体概要図である。また、図２は映像表示システム１の構成を示す構成ブロック図である。これらの図に示されるように、映像表示システム１は、情報処理装置１０と、カメラ２０と、操作デバイス３０と、頭部装着型の表示装置４０と、を含んで構成されている。 FIG. 1 is an overall schematic diagram of a video display system 1 including an information processing device 10 according to an embodiment of the present invention. FIG. 2 is a configuration block diagram showing the configuration of the video display system 1. As shown in these figures, the video display system 1 includes an information processing device 10, a camera 20, an operation device 30, and a head-mounted display device 40.
　情報処理装置１０は、表示装置４０が表示すべき映像を供給する装置であって、例えば家庭用ゲーム機、携帯型ゲーム機、パーソナルコンピューター、スマートフォン、タブレット等であってよい。図２に示されるように、情報処理装置１０は、制御部１１と、記憶部１２と、インタフェース部１３と、を含んで構成される。 The information processing device 10 is a device that supplies a video to be displayed by the display device 40, and may be, for example, a home game machine, a portable game machine, a personal computer, a smartphone, a tablet, or the like. As shown in FIG. 2, the information processing device 10 includes a control unit 11, a storage unit 12, and an interface unit 13.
　制御部１１は、ＣＰＵ等のプロセッサを少なくとも一つ含み、記憶部１２に記憶されているプログラムを実行して各種の情報処理を実行する。なお、本実施形態において制御部１１が実行する処理の具体例については、後述する。記憶部１２は、ＲＡＭ等のメモリデバイスを少なくとも一つ含み、制御部１１が実行するプログラム、及び当該プログラムによって処理されるデータを格納する。 The control unit 11 includes at least one processor such as a CPU, and executes various programs by executing programs stored in the storage unit 12. Note that a specific example of the processing executed by the control unit 11 in the present embodiment will be described later. The storage unit 12 includes at least one memory device such as a RAM, and stores a program executed by the control unit 11 and data processed by the program.
　インタフェース部１３は、カメラ２０、操作デバイス３０、及び表示装置４０との間のデータ通信のためのインタフェースである。情報処理装置１０は、インタフェース部１３を介して有線又は無線のいずれかでカメラ２０、操作デバイス３０、及び表示装置４０のそれぞれと接続される。具体的にインタフェース部１３は、情報処理装置１０が供給する映像や音声のデータを表示装置４０に送信するために、ＨＤＭＩ（登録商標）（High-Definition Multimedia Interface）等のマルチメディアインタフェースを含むこととする。また、カメラ２０によって撮像された映像データや、操作デバイス３０に対するユーザーの操作入力の内容を示す信号を受信するために、ＵＳＢ（Universal Serial Bus）等のデータ通信インタフェースを含んでいる。 The interface unit 13 is an interface for data communication between the camera 20, the operation device 30, and the display device 40. The information processing device 10 is connected to each of the camera 20, the operation device 30, and the display device 40 via the interface unit 13 by wire or wirelessly. Specifically, the interface unit 13 includes a multimedia interface such as HDMI (registered trademark) (High-Definition Multimedia Interface) in order to transmit video and audio data supplied by the information processing device 10 to the display device 40. And Further, it includes a data communication interface such as a USB (Universal Serial Bus) for receiving video data captured by the camera 20 and a signal indicating the content of a user's operation input to the operation device 30.
　カメラ２０は、映像表示システム１を利用するユーザーの前方に設置され、ユーザー及びその周辺を含んだ映像を撮像する。後述するように、カメラ２０の撮像画像を解析することによって、情報処理装置１０は表示装置４０の現実空間内における位置を特定する。 The camera 20 is installed in front of a user who uses the video display system 1 and captures an image including the user and its surroundings. As will be described later, the information processing apparatus 10 specifies the position of the display device 40 in the real space by analyzing the image captured by the camera 20.
　操作デバイス３０は、ユーザーからの操作入力を受け付けるためのデバイスであって、その表面には操作ボタン等の操作部材が配置されている。ユーザーは、操作デバイス３０を手で把持したり、手に装着したりして、操作部材に対する操作入力を行う。操作デバイス３０は、ユーザーが行った操作入力の内容を示す操作信号を、情報処理装置１０に送信する。 The operation device 30 is a device for receiving an operation input from a user, and has an operation member such as an operation button disposed on a surface thereof. The user performs an operation input on the operation member by holding the operation device 30 with a hand or mounting the operation device 30 on the hand. The operation device 30 transmits to the information processing apparatus 10 an operation signal indicating the content of the operation input performed by the user.
　表示装置４０は、ユーザーが頭部に装着して使用する映像表示装置であって、情報処理装置１０から送信される映像信号に応じた映像を表示し、ユーザーに閲覧させる。また、表示装置４０の前面には、発光素子が配置されている。この発光素子が発する光をカメラ２０が撮像し、情報処理装置１０がその撮像画像を解析することによって、情報処理装置１０は表示装置４０の現実空間内における位置を特定できる。特に本実施形態では、表示装置４０の前面に複数の発光素子が設けられており、これらの発光素子の位置関係を特定することによって、情報処理装置１０は表示装置４０の姿勢の変化を特定することとする。これにより、ユーザーが頭を動かして顔の向きを変化させたりした際に、情報処理装置１０はその向きの変化を特定できる。 The display device 40 is a video display device worn by the user on the head and used, and displays a video corresponding to a video signal transmitted from the information processing device 10 and allows the user to browse. In addition, a light emitting element is disposed on the front surface of the display device 40. The camera 20 captures the light emitted by the light emitting element, and the information processing device 10 analyzes the captured image, so that the information processing device 10 can specify the position of the display device 40 in the real space. In particular, in the present embodiment, a plurality of light emitting elements are provided on the front surface of the display device 40, and the information processing apparatus 10 specifies a change in the posture of the display device 40 by specifying the positional relationship between these light emitting elements. It shall be. Thus, when the user changes the direction of the face by moving the head, the information processing apparatus 10 can specify the change in the direction.
　さらに本実施形態において、表示装置４０は、ユーザーの視線方向の変化をリアルタイムに特定するための機構を備えている。以下、この機構の一例について説明する。図３は、表示装置４０の背面側の様子を示す図である。また、図４は、表示装置４０の内部構成を模式的に示す図であって、ユーザーの片方の瞳の向きを特定するために配置される構成要素の位置関係を示している。これらの図に示されるように、表示装置４０は、ユーザーの視線方向を特定するために、支持体４１と、複数のＬＥＤ４２と、レンズ４３と、ホットミラー４４と、眼球カメラ４５と、を含んでいる。 {Furthermore, in the present embodiment, the display device 40 includes a mechanism for identifying a change in the user's line of sight in real time. Hereinafter, an example of this mechanism will be described. FIG. 3 is a diagram illustrating a state on the back side of the display device 40. FIG. 4 is a diagram schematically illustrating an internal configuration of the display device 40, and illustrates a positional relationship of components arranged for specifying the orientation of one pupil of the user. As shown in these drawings, the display device 40 includes a support 41, a plurality of LEDs 42, a lens 43, a hot mirror 44, and an eyeball camera 45 in order to specify the direction of the user's line of sight. In.
　図３に示されるように、表示装置４０背面の、ユーザーの左右それぞれの目に対向する位置には、環状の支持体４１が配置されている。そして、左右の支持体４１のそれぞれには、複数のＬＥＤ４２が配置されている。これらのＬＥＤ４２は、いずれもユーザーの瞳に赤外線を照射可能な向きで固定されている。 環状 As shown in FIG. 3, an annular support 41 is disposed on the back of the display device 40 at a position facing the left and right eyes of the user. A plurality of LEDs 42 are arranged on each of the left and right supports 41. Each of these LEDs 42 is fixed in a direction in which the user's pupil can be irradiated with infrared rays.
　ホットミラー４４は、可視光を透過し、赤外線を反射する特性を持つ光学素子である。表示装置４０の表示素子に表示される映像は、ホットミラー４４、及びレンズ４３を通過してユーザーに提示される。一方、ＬＥＤ４２から照射された光はユーザーの瞳に反射され、レンズ４３を通過し、さらにホットミラー４４に反射されて眼球カメラ４５に入射する。すなわち、眼球カメラ４５は、ＬＥＤ４２の照射による光点が写っているユーザーの眼球の様子を撮像することになる。ユーザーの眼球の動きに応じて、その表面に写っている光点の位置が変化する。そのため情報処理装置１０は、眼球カメラ４５の撮像画像内に含まれる光点の分布を解析することによって、ユーザーの瞳の向き（視線方向）を特定することができる。 The hot mirror 44 is an optical element having a property of transmitting visible light and reflecting infrared light. The image displayed on the display element of the display device 40 is presented to the user through the hot mirror 44 and the lens 43. On the other hand, light emitted from the LED 42 is reflected by the user's pupil, passes through the lens 43, is further reflected by the hot mirror 44, and enters the eyeball camera 45. That is, the eyeball camera 45 captures an image of the user's eyeball in which the light spot due to the irradiation of the LED 42 is captured. The position of the light spot on the surface changes according to the movement of the user's eyeball. Therefore, the information processing apparatus 10 can identify the direction of the user's pupil (the direction of the line of sight) by analyzing the distribution of light spots included in the image captured by the eyeball camera 45.
　また、ここでは図示されていないが、映像表示システム１は、ユーザーの生体情報を検出するための１又は複数のバイタルセンサーを含んでもよい。このようなバイタルセンサーは、情報処理装置１０と接続され、検出したユーザーの生体情報をリアルタイムで情報処理装置１０に送信する。バイタルセンサーによって取得可能な情報の一例としては、脳波、心拍、発汗量、体温、唾液量、皮下血流量などが挙げられる。 Although not shown here, the image display system 1 may include one or more vital sensors for detecting biological information of the user. Such a vital sensor is connected to the information processing device 10 and transmits the detected biological information of the user to the information processing device 10 in real time. Examples of information that can be acquired by the vital sensor include brain waves, heartbeats, sweating, body temperature, saliva, subcutaneous blood flow, and the like.
　この場合のバイタルセンサーは、表示装置４０に配置されてもよい。表示装置４０はユーザーの頭部に装着されるため、この表示装置４０表面のユーザーの頭部に接触する位置にバイタルセンサーを配置することによって、ユーザーの脳波等の生体情報を検出することができる。 The vital sensor in this case may be arranged on the display device 40. Since the display device 40 is mounted on the user's head, by arranging the vital sensor at a position on the surface of the display device 40 that contacts the user's head, biological information such as the user's brain waves can be detected. .
　また、眼球カメラ４５をバイタルセンサーとして用いてもよい。ユーザーの眼球運動を記録するために、眼球カメラ４５は、比較的高解像度、かつ高フレームレートでユーザーの眼球周辺の映像を撮像する。そのため、この眼球カメラ４５が撮像した映像を解析することによって、ユーザーの眼球近くの血管の太さの時間変化を計測できる。この血管の太さの情報を用いて、情報処理装置１０は、ユーザーの血流量の変化を推定することができる。また、血流量の周期的な変化を特定することによって、ユーザーの心拍を計測することができる。 眼 Alternatively, the eyeball camera 45 may be used as a vital sensor. In order to record the user's eye movement, the eye camera 45 captures an image around the user's eye at a relatively high resolution and a high frame rate. Therefore, by analyzing the image captured by the eyeball camera 45, it is possible to measure the time change of the thickness of the blood vessel near the user's eyeball. Using the information on the thickness of the blood vessel, the information processing device 10 can estimate a change in the blood flow volume of the user. In addition, the user's heart rate can be measured by specifying a periodic change in blood flow.
　次に、情報処理装置１０が実現する機能について図５を用いて説明する。図５に示すように、情報処理装置１０は、機能的に、視線方向特定部５１と、興味位置推定部５２と、映像表示制御部５３と、を含んでいる。これらの機能は、制御部１１が記憶部１２に記憶されたプログラムを実行することにより実現される。このプログラムは、インターネット等の通信ネットワークを介して情報処理装置１０に提供されてもよいし、光ディスク等のコンピュータ読み取り可能な情報記憶媒体に格納されて提供されてもよい。 Next, the functions realized by the information processing apparatus 10 will be described with reference to FIG. As shown in FIG. 5, the information processing apparatus 10 functionally includes a gaze direction identification unit 51, an interest position estimation unit 52, and a video display control unit 53. These functions are realized by the control unit 11 executing a program stored in the storage unit 12. This program may be provided to the information processing apparatus 10 via a communication network such as the Internet, or may be provided by being stored in a computer-readable information storage medium such as an optical disk.
　視線方向特定部５１は、後述する映像表示制御部５３によって表示装置４０に表示される映像をユーザーが閲覧している間の、ユーザーの視線方向の変化を特定する。具体的に視線方向特定部５１は、前述したＬＥＤ４２を動作させる制御信号を表示装置４０に対して送信し、ＬＥＤ４２が赤外線を放射している状態で眼球カメラ４５がユーザーの瞳を撮像して得られる撮像画像を取得する。そして、この撮像画像を用いて、その撮像が行われた時点におけるユーザーの視線方向（注視点の方向）を特定する。この視線方向の特定処理自体は、公知の手法によって実現可能である。視線方向特定部５１は、このような視線方向の特定処理を短い時間間隔で繰り返し実行し、実行結果を記憶部１２内に記憶する。これにより視線方向特定部５１は、視線方向のモニターを行っている間のユーザーの視線の動きの軌跡（視線方向の時間変化）を記録することができる。 The line-of-sight direction specifying unit 51 specifies a change in the line-of-sight direction of the user while the user is viewing an image displayed on the display device 40 by the image display control unit 53 described later. Specifically, the line-of-sight direction identification unit 51 transmits a control signal for operating the above-described LED 42 to the display device 40, and the eyeball camera 45 captures an image of the user's pupil while the LED 42 is emitting infrared light. To obtain a captured image. Then, using this captured image, the user's line-of-sight direction (direction of the point of gaze) at the time when the image is captured is specified. The line-of-sight direction identification processing itself can be realized by a known method. The gaze direction identification unit 51 repeatedly executes such gaze direction identification processing at short time intervals, and stores the execution result in the storage unit 12. Thereby, the gaze direction identification unit 51 can record the trajectory of the user's gaze movement (time change in the gaze direction) while monitoring the gaze direction.
　興味位置推定部５２は、視線方向特定部５１によって特定された過去のユーザーの視線方向の時間変化に基づいて、ユーザーが現在注視している方向（注視点）とは別の、ユーザーが潜在的に興味を向けている位置、及び／又は方向を推定する。以下では、このように注視点とは別のユーザーが興味を向けていると推定される位置を、興味位置という。また、ユーザーが興味を向けていると推定される方向を、興味方向という。興味方向は、映像面内において注視点から興味位置に向かう方向である。興味位置推定部５２が興味方向や興味位置を推定する方法の具体例については、後述する。 The interest position estimating unit 52 is based on the time change of the past user's line of sight specified by the line of sight specifying unit 51, and the potential of the user is different from the direction of the user's current gaze (gaze point). Estimating the position and / or direction of interest in. In the following, such a position where it is estimated that a user other than the point of interest is interested is referred to as an interest position. The direction in which the user is presumed to be interested is called an interest direction. The direction of interest is a direction from the point of gaze to the position of interest in the image plane. A specific example of a method in which the interest position estimation unit 52 estimates an interest direction and an interest position will be described later.
　映像表示制御部５３は、ユーザーに提示する映像を描画し、その内容を示す映像信号を表示装置４０に対して出力する。例えば映像表示制御部５３は、仮想オブジェクトが配置された仮想空間内の様子を示す空間画像を描画し、表示装置４０に表示させる。さらに映像表示制御部５３は、表示装置４０を頭部に装着したユーザーが自分の頭の向きを変化させた場合、カメラ２０の撮像画像を解析してその動作を特定し、ユーザーの頭の向きの変化に連動するように仮想空間内に設定された視点の向きを移動させる。これによりユーザーは、自分の頭を動かして視点の向きを変化させながら仮想空間内の様子を閲覧することができる。なお、このような制御を行う場合、ユーザーは表示装置４０が一度に表示可能な範囲を超える範囲を見渡すことができる。そのため、過去に表示されたが現在はユーザーの頭部の動きに合わせて表示装置４０に表示されなくなったオブジェクトが表示されていた位置の方にユーザーが意識を向けていることもあり得る。すなわち、興味位置推定部５２によって推定される興味位置は、必ずしもその時点で表示されている映像内の位置である必要はなく、画面外の位置であってもよい。 The video display control unit 53 draws a video to be presented to the user, and outputs a video signal indicating the content to the display device 40. For example, the video display control unit 53 draws a space image indicating a state in the virtual space where the virtual objects are arranged, and causes the display device 40 to display the space image. Furthermore, when the user wearing the display device 40 on his / her head changes his / her head direction, the video display control unit 53 analyzes the image captured by the camera 20 to specify the operation, and identifies the user's head direction. The direction of the viewpoint set in the virtual space is moved so as to interlock with the change of the viewpoint. Thus, the user can browse the state in the virtual space while moving his / her head and changing the direction of the viewpoint. When such control is performed, the user can look over a range that can be displayed on the display device 40 at a time. Therefore, the user may turn his or her consciousness to the position where the object that was displayed in the past but is no longer displayed on the display device 40 according to the movement of the user's head is displayed. That is, the interest position estimated by the interest position estimation unit 52 does not necessarily need to be a position in the video currently displayed, and may be a position outside the screen.
　さらに本実施形態において、映像表示制御部５３は、興味位置推定部５２によって特定された興味位置に基づいて、表示装置４０に供給する映像の内容を変化させることとする。一例として、映像表示制御部５３は、ユーザーの注視点、及び興味位置に基づいて特定される領域の解像度が、他の解像度よりも高い映像を描画する。映像表示制御部５３が興味位置に基づいて描画する映像の内容の具体例については、後述する。 Further, in the present embodiment, the video display control unit 53 changes the content of the video supplied to the display device 40 based on the interest position specified by the interest position estimation unit 52. As an example, the video display control unit 53 draws a video in which the resolution of an area specified based on the user's point of interest and the position of interest is higher than other resolutions. A specific example of the content of the video drawn by the video display control unit 53 based on the position of interest will be described later.
　以下、興味位置推定部５２が興味方向、及び興味位置を推定する方法の具体例について、説明する。興味位置推定部５２は、直近の過去に視線方向特定部５１が記録したユーザーの視線方向の時間変化に基づいて、ユーザーの興味位置を推定する。具体的に、ユーザーの眼球が特定の位置（注視点）近傍に向けられている場合に、その位置から別の方向にわずかに動きすぐに戻るという往復運動を行うことがある。このような眼球運動はユーザーが意図的に行うのではなく、無意識に行われているのが通常である。このような眼球運動の一種として、マイクロサッカードと呼ばれるものが知られている。このような眼球運動が特定の方向に向けて繰り返し行われる場合、その方向にユーザーが潜在的に注目している（興味を向けている）可能性が高い。そこで興味位置推定部５２は、視線方向特定部５１が特定した視線方向の時間変化に特定パターンの動きが含まれる場合に、その特定パターンの動きに基づいて興味方向を推定する。さらに、その興味方向が指し示す位置に興味位置があると推定する。 Hereinafter, a specific example of a method in which the interest position estimation unit 52 estimates the direction of interest and the interest position will be described. The interest position estimation unit 52 estimates the interest position of the user based on the time change of the user's gaze direction recorded by the gaze direction identification unit 51 in the most recent past. Specifically, when the user's eyeball is pointed near a specific position (gaze point), a reciprocating motion of slightly moving from the position to another direction and returning immediately may be performed. Such eye movements are not performed intentionally by the user, but are usually performed unconsciously. As one kind of such eye movement, a so-called microsaccade is known. When such eye movements are repeatedly performed in a specific direction, there is a high possibility that the user is potentially paying attention (interesting) in that direction. Thus, when the temporal change in the line-of-sight direction specified by the line-of-sight direction specifying unit 51 includes the movement of the specific pattern, the interest position estimation unit 52 estimates the direction of interest based on the movement of the specific pattern. Further, it is estimated that there is an interesting position at the position indicated by the interesting direction.
　図６は、視線方向特定部５１によって特定されるユーザーの視線方向の時間変化の一例を示している。この図の横軸は水平方向の角度θｘ、縦軸は垂直方向の角度θｙを示しており、図中の実線が視線方向の時間変化を示している。この例では、ユーザーの注視点は時間とともにおおよそ領域Ａから領域Ｂ、そして領域Ｃに移っている。しかしながら、どの領域にユーザーの視線が向けられている状態においても、ユーザーの視線方向は向かって右方向に移動して戻る細かな往復運動を繰り返している。この往復運動の方向が興味方向であると推定される。ただし、ユーザーは、興味位置が視野の中心に位置する状態まで眼球を動かすわけではない。したがって、わずかな回数の往復運動だけでは、興味方向が概ねどの方向かを推定することはできても、具体的に興味位置がどこなのか（すなわち、注視点から興味位置までどの程度の距離があるか）を推定することは難しい。そこで興味位置推定部５２は、直近の過去のある程度の時間における往復運動から、興味位置を推定してもよい。 FIG. 6 shows an example of a temporal change in the gaze direction of the user specified by the gaze direction specifying unit 51. The horizontal axis in this figure indicates the angle θx in the horizontal direction and the vertical axis indicates the angle θy in the vertical direction, and the solid line in the figure indicates the time change in the line of sight. In this example, the user's point of gaze shifts from area A to area B to area C over time. However, no matter what area the user's line of sight is directed, the user's line of sight repeats a fine reciprocating movement that moves rightward and back. It is estimated that the direction of this reciprocation is the direction of interest. However, the user does not move the eyeball until the position of interest is located at the center of the visual field. Therefore, even with a small number of reciprocating movements, it is possible to estimate the general direction of the interesting direction, but to determine the specific position of the interesting position (that is, how much distance from the point of gaze to the interesting position) Is difficult to estimate. Therefore, the interest position estimation unit 52 may estimate the interest position from the reciprocating motion of the latest past for some time.
　図６の例では、中段の領域Ａに注視点が存在する場合、その往復運動の向きから、向かって右やや下方向に向かう破線Ｄａの方向が興味方向であると推定される。さらに、上方の領域Ｂ、及び下方の領域Ｃのそれぞれに注視点が存在する場合の興味方向も、図中の破線Ｄｂ、及びＤｃに示すような向きであると推定される。ユーザーの視線が領域Ａから領域Ｂ、そして領域Ｃに移動している間に興味位置が変化していなければ、３つの破線Ｄａ，Ｄｂ，Ｄｃの交点に対応する位置Ｘが興味位置であると推定される。なお、複数の注視点において特定される複数の興味方向が一点で交差しない場合であっても、このような複数の興味方向が集まる位置の近傍が、興味位置であると推定できる。 In the example of FIG. 6, when the gazing point exists in the middle area A, it is estimated from the direction of the reciprocating movement that the direction of the broken line Da that is slightly downward toward the right is the direction of interest. Further, it is estimated that the directions of interest when the gazing point is present in each of the upper region B and the lower region C are also directions as shown by broken lines Db and Dc in the figure. If the position of interest does not change while the user's line of sight moves from the region A to the region B and then to the region C, the position X corresponding to the intersection of the three broken lines Da, Db and Dc is determined to be the position of interest. Presumed. Note that, even when a plurality of directions of interest specified at a plurality of gazing points do not intersect at a single point, it is possible to estimate that the vicinity of a position where such a plurality of directions of interest gather is an interesting position.
　また、興味位置推定部５２は、映像表示制御部５３が表示中の映像の内容を参照して、興味位置を特定してもよい。例えば前述した図６の例において注視点が領域Ａ内に留まっている状態では、視線方向の情報だけでは破線Ｄａ上のどこに興味位置があるのか特定できない。この場合において、ユーザーの興味を惹く可能性のあるオブジェクト（候補オブジェクト）が破線Ｄａ上に表示されている場合、興味位置推定部５２はそのオブジェクトの表示位置を興味位置として推定してもよい。このような推定を行うために、興味位置推定部５２は、映像表示制御部５３から現在画面内に表示中の候補オブジェクトの位置を示す情報を受け付ける。そして、表示中の候補オブジェクトのうち、興味方向に最も近い位置に存在する候補オブジェクトの表示位置を、興味位置として推定する。 興味 Furthermore, the interest position estimation unit 52 may specify the interest position with reference to the content of the video being displayed by the video display control unit 53. For example, in the above-described example of FIG. 6, in the state where the gazing point remains in the area A, it is not possible to specify where on the broken line Da the interested position is based only on the line-of-sight direction information. In this case, when an object (candidate object) that may attract the user's interest is displayed on the broken line Da, the interest position estimation unit 52 may estimate the display position of the object as the interest position. In order to perform such estimation, the interest position estimation unit 52 receives information indicating the position of the candidate object currently displayed on the screen from the video display control unit 53. Then, of the candidate objects being displayed, the display position of the candidate object located closest to the direction of interest is estimated as the interest position.
　なお、映像表示制御部５３は、仮想空間の様子を描画するのではなく、現実空間の様子を撮像して得られる映像や、予め用意された映像などを表示装置４０に表示させる場合もある。このような場合においても、表示される映像を解析することによって、その中に映っている、ユーザーの興味を惹く可能性のある物体を候補オブジェクトとして特定してもよい。 The image display control unit 53 may cause the display device 40 to display an image obtained by capturing an image of the real space, or an image prepared in advance, instead of drawing the state of the virtual space. Even in such a case, by analyzing the displayed video, an object that may appear in the image and may be of interest to the user may be specified as a candidate object.
　さらに、興味位置推定部５２は、各種の情報に基づいて、ユーザーが映像に集中しているか否かなど、ユーザーの閲覧状況を示す情報を取得し、興味位置の推定に利用してもよい。例えば、眼球カメラ４５の撮像映像を解析することによって、興味位置推定部５２は、ユーザーが瞬きをしたタイミングや、瞳孔径の変化といった事象を特定することができる。一般的に、ユーザーは映像に集中する前に瞬きをしたり、集中度が高くなると瞳孔径が大きくなったりすると想定される。そのため、このようなユーザーの瞳を観測して得られる情報を用いることにより、その時点でのユーザーの映像に対する集中度（ユーザーがどの程度映像に興味を持っているかの指標値）を推定できる。 Further, based on various types of information, the interest position estimation unit 52 may acquire information indicating the browsing status of the user, such as whether or not the user is concentrated on the video, and use the information to estimate the interest position. For example, by analyzing an image captured by the eyeball camera 45, the interest position estimation unit 52 can specify an event such as a timing at which the user blinks or a change in pupil diameter. In general, it is assumed that the user blinks before concentrating on the image, or that the pupil diameter increases as the degree of concentration increases. Therefore, by using such information obtained by observing the user's pupil, it is possible to estimate the degree of concentration (index value of how much the user is interested in the image) at that time.
　ただし、人の瞳孔径は、その人が見ている映像の輝度の変化による影響を受ける。さらに、映像の輝度が瞳孔径にどのように影響を与えるかについては個人差があり、映像全体の輝度の影響を受けやすいタイプの人や、注視点近傍の輝度の影響を受けやすいタイプの人、その中間の人など、タイプによっても影響の仕方が異なる。そこで興味位置推定部５２は、ユーザーごとに、映像の輝度の情報と、その映像が表示されているときに観測される瞳孔径の情報とを取得し、取得した情報を用いてそのユーザーの瞳孔径が映像の輝度によってどのように変化するかを示す情報を特定してもよい。 However, the pupil diameter of a person is affected by changes in the luminance of the image that the person is watching. Furthermore, there are individual differences in how the luminance of the image affects the pupil diameter, and there are people who are susceptible to the luminance of the entire image and those who are susceptible to the luminance near the gazing point. , And the way in which it affects them depends on the type, such as the person in between. Therefore, the interest position estimating unit 52 obtains, for each user, information on the luminance of the image and information on the pupil diameter observed when the image is displayed, and uses the obtained information to determine the pupil of the user. Information indicating how the diameter changes depending on the luminance of the image may be specified.
　具体的に興味位置推定部５２は、各ユーザーが表示装置４０の使用を開始する際に、キャリブレーションのために用意された各種の背景色や輝度の画像を表示装置４０に表示させる。そして、これらのキャリブレーション用画像を閲覧中のユーザーの瞳孔径の大きさを計測し、その計測結果を教師データとして利用して、そのユーザーの集中度を推定する際に使用する基準値を決定する。また、興味位置推定部５２は、実際の表示装置４０の使用中にも、表示される映像の輝度と瞳孔径の計測結果とを定期的に教師データとして取得して、機械学習を行い、ユーザーごとの集中度を推定するための推定器を構築してもよい。なお、このような学習を行う際には、表示中の映像全体の輝度、及びユーザーが現在注視している領域の輝度の双方を入力として利用することが望ましい。こうすれば、現在表示中の映像の輝度の影響や、使用中のユーザーの個人差によらずに、精度よく集中度を推定できる。 Specifically, the interest position estimating unit 52 causes the display device 40 to display images of various background colors and brightness prepared for calibration when each user starts using the display device 40. Then, the size of the pupil diameter of the user who is browsing these calibration images is measured, and the measurement result is used as teacher data to determine a reference value used when estimating the degree of concentration of the user. I do. In addition, even during actual use of the display device 40, the interest position estimation unit 52 periodically acquires the luminance of the displayed image and the measurement result of the pupil diameter as teacher data, performs machine learning, and performs user learning. An estimator for estimating the degree of concentration for each may be constructed. When such learning is performed, it is desirable to use both the luminance of the entire image being displayed and the luminance of the area currently being watched by the user as inputs. In this way, the degree of concentration can be accurately estimated without depending on the influence of the luminance of the currently displayed image or the individual difference of the user in use.
　また、興味位置推定部５２は、前述したバイタルセンサーの計測結果を用いてユーザーの集中度を推定してもよい。この場合も、バイタルセンサーによって計測される生体情報を教師データとして機械学習を行い、予めユーザーごとに生体情報が集中度にどのような影響を与えるか学習することによって、精度よく集中度の推定を行うことができる。 興味 Furthermore, the interest position estimation unit 52 may estimate the degree of concentration of the user using the measurement result of the vital sensor described above. In this case as well, machine learning is performed using the biological information measured by the vital sensor as teacher data, and how the biological information affects the degree of concentration is learned in advance for each user, so that the degree of concentration can be accurately estimated. It can be carried out.
　また、興味位置推定部５２は、利き目の情報を用いて興味位置の推定を行ってもよい。利き手などと同様、人の左右の目にも利き目があることが知られている。この利き目の情報を利用することで、例えば左右それぞれの眼球運動に基づいて得られる２個の興味位置の推定結果に対して、利き目がどちらかに応じた重み付けを行ったうえで両者を合成することによって、ユーザーの興味位置を推定してもよい。あるいは、左右の興味位置の推定結果が相反するものである場合、利き目の推定結果を優先して採用してもよい。なお、ユーザーの利き目がどちらかは、予めユーザー自身から入力を受け付けてもよいし、ユーザーの映像に対する視線の動きなどから推定してもよい。 興味 Furthermore, the interest position estimation unit 52 may estimate the interest position using the information of the dominant eye. It is known that the left and right eyes of a person have a dominant eye as well as a dominant hand. By using this dominant eye information, for example, the two dominant eye estimation results obtained based on the left and right eye movements are weighted according to the dominant eye, and then both are estimated. By combining them, the user's interest position may be estimated. Alternatively, when the estimation results of the left and right interest positions are contradictory, the estimation result of the dominant eye may be preferentially adopted. Either of the user's dominant eyes may be received in advance from the user himself or may be estimated from the movement of the line of sight with respect to the user's video.
　以下、興味位置推定部５２が実行する処理の流れの一例について、図７のフロー図を用いて説明する。まず興味位置推定部５２は、ユーザーの視線方向の時間変化に基づいて、ユーザーが特定の領域を注視している状態か否かを判定する（Ｓ１）。具体的に、直近の過去所定期間内において、ユーザーの視線方向の移動範囲が所定の大きさの範囲内に収まっていれば、ユーザーは特定の領域を注視していると推定される。 Hereinafter, an example of a flow of a process executed by the interest position estimation unit 52 will be described with reference to a flowchart of FIG. First, the interest position estimation unit 52 determines whether or not the user is gazing at a specific area based on a temporal change in the user's line of sight (S1). Specifically, if the moving range of the user's line of sight in the latest past predetermined period falls within a range of a predetermined size, it is estimated that the user is gazing at a specific area.
　ユーザーが特定の領域を注視している場合、さらに興味位置推定部５２は、その領域を注視している間のユーザーの視線方向の軌跡に基づいて、特定パターンの眼球運動が生じているか否かを判定する（Ｓ２）。この特定パターンの眼球運動とは、前述した特定方向への短距離の往復運動などであってよい。例えば興味位置推定部５２は、直近の過去所定期間内において所定回数以上の往復運動が行われている場合に、特定パターンの眼球運動が生じていると判定する。 When the user is gazing at a specific area, the interest position estimation unit 52 further determines whether or not a specific pattern of eye movement has occurred based on the trajectory of the user's line of sight while gazing at the area. Is determined (S2). The eye movement of the specific pattern may be a short distance reciprocating movement in the specific direction described above. For example, the interest position estimating unit 52 determines that the eye movement of the specific pattern has occurred when the reciprocating motion has been performed a predetermined number of times or more within the latest past predetermined period.
　Ｓ２で特定パターンの眼球運動が生じたと判定された場合、続いて興味位置推定部５２は、所与の判定基準に基づいてその時点でのユーザーの集中度を判定する（Ｓ３）。具体的に興味位置推定部５２は、前述したように瞬きの頻度や瞳孔径、またその他の各種生体情報に基づいて、ユーザーが映像に集中しているか否かを判定する。ユーザーが映像に集中していると判定される場合、Ｓ４に進んでユーザーの興味位置を推定する処理を実行する。一方、ここまでの判定条件のいずれかが満たされない場合、興味位置推定部５２は、Ｓ１に戻って各判定条件が満たされた状態になるまでＳ１～Ｓ３の処理を繰り返すこととする。なお、機械学習などの結果に基づいて集中度を推定する場合、推定結果の信頼度（どの程度その推定結果が信頼できるかを示す指標値）も合わせて算出される。この信頼度が低い（すなわち、ユーザーがどの程度集中しているか精度よく推定できていない）場合、Ｓ４の処理に進んでもよい。 If it is determined in S2 that the eye movement of the specific pattern has occurred, the interest position estimation unit 52 subsequently determines the concentration of the user at that time based on a given determination criterion (S3). Specifically, the interest position estimation unit 52 determines whether or not the user is concentrated on the image based on the blink frequency, the pupil diameter, and other various types of biological information as described above. When it is determined that the user is concentrated on the video, the process proceeds to S4, and a process of estimating the user's interest position is executed. On the other hand, if any of the determination conditions up to this point is not satisfied, the interest position estimation unit 52 returns to S1 and repeats the processing of S1 to S3 until each determination condition is satisfied. When estimating the degree of concentration based on the result of machine learning or the like, the reliability of the estimation result (an index value indicating how reliable the estimation result is) is also calculated. If the reliability is low (that is, it is not possible to accurately estimate how concentrated the user is), the process may proceed to S4.
　Ｓ３でユーザーが映像に集中していると判定された場合、興味位置推定部５２は、Ｓ２で検出された特定パターンの眼球運動のデータに基づいて、ユーザーが特定の方向に注意を向けているか否かを判定する（Ｓ４）。眼球の往復運動の向きが特定の方向に集中していない場合、ユーザーは単に集中して現在の注視点に意識を集中していると想定される。そのため、興味位置推定部５２は、興味位置が存在しないと判定する（Ｓ５）。 If it is determined in S3 that the user is concentrated on the video, the interest position estimation unit 52 determines whether the user is paying attention to a specific direction based on the eye movement data of the specific pattern detected in S2. It is determined whether or not it is (S4). If the direction of the reciprocating movement of the eyeball is not concentrated in a specific direction, it is assumed that the user is merely concentrated and is consciously focused on the current gazing point. Therefore, the interest position estimation unit 52 determines that the interest position does not exist (S5).
　一方、眼球の往復運動が特定の方向に向けられている場合、その方向が興味方向であると推定される。そこで興味位置推定部５２は、往復運動の向き、及びその他の条件（画面内に表示中の候補オブジェクトの位置情報など）に基づいて、興味方向及び興味位置を特定する（Ｓ６）。このような処理を定期的に実行することにより、映像表示制御部５３は、興味位置を考慮した映像をユーザーに提示することができる。 On the other hand, when the reciprocating movement of the eyeball is directed in a specific direction, it is estimated that the direction is the direction of interest. Therefore, the interest position estimating unit 52 specifies the interest direction and the interest position based on the direction of the reciprocating motion and other conditions (such as position information of the candidate object being displayed on the screen) (S6). By performing such processing periodically, the video display control unit 53 can present a video in consideration of the position of interest to the user.
　以下、映像表示制御部５３がユーザーに提示する映像の内容を興味位置や興味方向に応じて制御する方法の具体例について、説明する。 Hereinafter, a specific example of a method in which the video display control unit 53 controls the content of the video presented to the user according to the position of interest and the direction of interest will be described.
　第１の例として、映像表示制御部５３は、画面内に複数のオブジェクトを表示する場合に、興味位置又は興味方向に存在するオブジェクトと、それ以外のオブジェクトとで、表示態様を変化させてもよい。具体的には、興味位置近傍、又は興味方向に近い位置に存在するオブジェクトを強調表示の対象として、その色や明度、大きさなどを変化させてもよい。また、逆に興味位置から離れた位置に存在するオブジェクトが目立たなくなるように、その明度を下げたり、透明度を上げたり、ピントをぼかした表示に変化させたりしてもよい。このような制御によれば、ユーザーが無意識に興味を惹かれていた対象に意識的に注意を向けさせるようにすることができる。 As a first example, when displaying a plurality of objects on a screen, the video display control unit 53 may change the display mode between an object existing at an interest position or an interest direction and other objects. Good. Specifically, an object existing near the position of interest or in a position close to the direction of interest may be highlighted, and its color, brightness, size, etc. may be changed. Conversely, the brightness may be reduced, the transparency may be increased, or the focus may be changed to a blurred display so that the object located at a position distant from the interest position becomes less conspicuous. According to such control, it is possible for the user to intentionally pay attention to an object that has been unconsciously attracted.
　図８は、このような制御によって表示される映像の一例を示しており、図８Ａは表示を変化させる前、図８Ｂは興味方向に応じて表示を変化させた後の様子を示している。この図の例では、映像内に複数のターゲットＴが表示されているが、そのうちターゲットＴｐにユーザーの視線が向けられており、ターゲットＴｐから向かって左下方向が、興味方向Ｄとして特定されたこととしている。この場合において、映像中に含まれる複数のターゲットのうち、興味方向に近い左下のターゲットＴ４，Ｔ８，Ｔ９は図８Ａと図８Ｂとで変化がないが、その他のターゲットについては、図８Ｂにおいては目立たなくなるように変化している。 FIG. 8 shows an example of an image displayed by such control, FIG. 8A shows a state before the display is changed, and FIG. 8B shows a state after the display is changed according to the direction of interest. In the example of this figure, a plurality of targets T are displayed in the video, but the user's line of sight is directed to the target Tp, and the lower left direction from the target Tp is specified as the direction of interest D. And In this case, among the plurality of targets included in the video, the lower left targets T4, T8, and T9 that are close to the direction of interest do not change between FIGS. 8A and 8B, but the other targets are the same in FIG. 8B. It has changed to be less noticeable.
　なお、ここでは興味位置や興味方向に合致するオブジェクトを目立たせるような制御を行うこととしたが、映像表示制御部５３はこれとは逆の制御を行ってもよい。例えばゲームなどにおいて、ユーザーに注目してもらいたい目標オブジェクトを映像内に表示することとする。この場合において、ユーザーが目標オブジェクトとは別の位置や方向に興味を惹かれていると推定された場合、徐々に目標オブジェクトを強調するような表示制御を行ってもよい。 In this case, the control is performed to make the object that matches the position of interest or the direction of interest stand out. However, the video display control unit 53 may perform the reverse control. For example, in a game or the like, a target object that the user wants to pay attention to is displayed in the video. In this case, when it is estimated that the user is interested in a different position or direction from the target object, display control may be performed to gradually emphasize the target object.
　第２の例として、映像表示制御部５３は、興味方向又は興味位置を考慮した中心窩レンダリング（Foveated rendering）を実現してもよい。中心窩レンダリングとは、ユーザーの注視点などを中心とした注目領域を比較的高解像度で描画する一方で、注目領域より外側の周辺領域については比較的低解像度で描画することによって、描画負荷を抑える手法である。本実施形態では、注視点を注目領域の中心とするのではなく、そこから興味方向に対して広がった範囲を注目領域とする（すなわち、注目領域の中心を注視点よりも興味方向にずれた位置に設定する）ことで、ユーザーが真に興味を持って閲覧している、又はこれから注目する可能性の高い範囲も高解像度で描画することができる。図９は、このように注視点Ｐ及び興味方向Ｄに基づいて設定された注目領域Ａの一例を示している。また、映像表示制御部５３は、興味方向だけでなく、興味位置が画面内のどこなのかまで推定できている場合には、その興味位置と注視点の双方が含まれるように注目領域を決定してもよい。 As a second example, the video display control unit 53 may realize foveated rendering in consideration of an interest direction or an interest position. Foveal rendering is a method that draws a region of interest centered on the user's gaze point at a relatively high resolution, while drawing peripheral regions outside the region of interest at a relatively low resolution, thereby reducing the drawing load. It is a technique to suppress. In the present embodiment, instead of setting the point of interest at the center of the attention area, a range extending therefrom in the direction of interest is set as the attention area (that is, the center of the attention area is shifted from the point of interest to the direction of interest. By setting the position, it is possible to draw at high resolution a range that the user is truly interested in browsing or is likely to pay attention to. FIG. 9 shows an example of the attention area A set based on the gazing point P and the interest direction D as described above. If the video display control unit 53 can estimate not only the direction of interest but also the position of interest in the screen, the video display control unit 53 determines the attention area so as to include both the position of interest and the point of interest. May be.
　さらに、このような中心窩レンダリングを実行した場合、ユーザーがあまり注目していないと想定される周辺領域の解像度を下げることになるが、ユーザーはこの解像度の低下に気がつく場合もある。ユーザーが解像度の低下に気がついた場合、解像度の低下した領域に急に注視点や興味位置が向けられるなど、ユーザーの視線に不自然な動きが生じると想定される。そこで、興味位置推定部５２がこのような解像度の低下に反応するユーザーの視線の動きを検出した場合、映像表示制御部５３は、中心窩レンダリングを中断したり、周辺領域の解像度を上げたりするなど、画質を改善する制御を実行してもよい。このような制御によれば、ユーザーに違和感を生じさせないようにしながら描画負荷を抑えることができる。 Furthermore, when such foveal rendering is performed, the resolution of the peripheral region, which is assumed to be of little interest to the user, is reduced. However, the user may notice this reduction in resolution. When the user notices a decrease in the resolution, it is assumed that an unnatural movement occurs in the user's line of sight, such as a point of gaze or an interest position being suddenly turned to the region in which the resolution has decreased. Therefore, when the interest position estimating unit 52 detects the movement of the user's line of sight in response to such a decrease in resolution, the video display control unit 53 interrupts fovea rendering or increases the resolution of the peripheral region. For example, control for improving image quality may be executed. According to such control, the drawing load can be suppressed while preventing the user from feeling uncomfortable.
　第３の例として、映像表示制御部５３は、画像の更新頻度を領域ごとに変化させてもよい。この例でも、第２の例と同様に、ユーザーが注目していると想定される注目領域とその外側の周辺領域とで、表示される画像の描画手法を変化させる。ただし、第２の例のように一度に表示されるフレーム画像内における解像度を領域ごとに変化させるのではなく、映像の更新頻度（フレームレート）を領域ごとに変化させる。例えば映像表示制御部５３は、周辺領域についてはフレーム画像の更新頻度を注目領域の半分に低下させ、相対的に注目領域内の映像の更新頻度が周辺領域の更新頻度より高くなるよう制御する。このような手法によれば、表示装置４０が表示するフレーム画像の全体を毎フレーム情報処理装置１０から表示装置４０に対して送信する必要がなくなる。そのため、情報処理装置１０から表示装置４０に対して単位時間あたりに送信すべきデータ量を削減でき、比較的通信帯域が小さい場合であってもユーザーに違和感を生じさせにくい映像を提示できる。この第３の例においても、注目領域の位置、及び大きさを興味位置や興味方向に基づいて決定することで、ユーザーが注視点以外の場所に興味を惹かれている場合に、そのような場所も含めて高解像度の映像を提示することができる。 As a third example, the video display control unit 53 may change the update frequency of the image for each region. Also in this example, as in the second example, the drawing method of the displayed image is changed between the attention area assumed to be the user's attention and the surrounding area outside the attention area. However, instead of changing the resolution in a frame image displayed at a time for each region as in the second example, the video update frequency (frame rate) is changed for each region. For example, the video display control unit 53 controls the frequency of updating the frame image in the peripheral region to be half that of the region of interest, and controls the frequency of updating the video in the region of interest to be relatively higher than the frequency of updating the peripheral region. According to such a method, it is not necessary to transmit the entire frame image displayed by the display device 40 from the frame information processing device 10 to the display device 40. Therefore, the amount of data to be transmitted per unit time from the information processing device 10 to the display device 40 can be reduced, and even when the communication band is relatively small, it is possible to present an image that does not cause a user to feel strange. Also in the third example, by determining the position and size of the attention area based on the position of interest and the direction of interest, when the user is interested in a place other than the gazing point, such a case is obtained. A high-resolution image including a place can be presented.
　第４の例として、映像表示制御部５３は、興味位置の近傍領域を高解像度で先行描画してもよい。この例でも映像表示制御部５３は中心窩レンダリングと同様の制御を行うこととするが、以前の例と異なり注目領域は注視点に基づいて決定されることとする。一方で、注視点とは別にユーザーが興味を持っている興味位置が存在する場合、ユーザーがその位置に視線を移動させる（すなわち、興味位置を新たな注視点とする）ことが予想される。そこで映像表示制御部５３は、その興味位置近傍の領域についても、注視点近傍の領域と同様に予め高解像度で描画した画像を用意しておく。そして、ユーザーの注視点がこの興味位置近傍に移動した際には、予め描画された画像を用いて表示装置４０に表示させる映像を生成する。このような制御によれば、ユーザーが視線方向をそれまでの注視点から興味位置に動かした場合に、待ち時間なく直ちに新たな注視点を高解像度で描画した映像をユーザーに提示することができる。 As a fourth example, the video display control unit 53 may preliminarily draw a region near the position of interest with high resolution. Also in this example, the video display control unit 53 performs the same control as the fovea rendering, but unlike the previous example, the attention area is determined based on the gazing point. On the other hand, when there is an interesting position that the user is interested in separately from the point of interest, it is expected that the user will move his / her line of sight to that position (that is, make the interested position a new point of interest). Therefore, the image display control unit 53 also prepares an image drawn at a high resolution in advance in the region near the interest position in the same manner as the region near the gazing point. Then, when the user's gazing point moves to the vicinity of the interest position, an image to be displayed on the display device 40 is generated using the image drawn in advance. According to such control, when the user moves the line of sight from the current point of interest to the position of interest, a video in which a new point of interest is drawn at a high resolution can be presented to the user immediately without waiting time. .
　さらに、興味位置が現在の注視点から所定距離以上離れている場合、映像表示制御部５３は、中心窩レンダリングの対象となる注目領域を小さくしてもよい。現在の注視点から大きく離れた位置にユーザーが興味を惹かれていると推定された場合、ユーザーが頭部の向きを変化させるなどして、現在の注視点から興味位置まで比較的高速で大きな距離の視点移動を行おうとしている可能性が高い。このような大きな視点移動の間は、表示されている映像を広い範囲にわたって詳細に確認することは難しい。そのため、高解像度で描画する領域を小さくしてもユーザーが違和感を感じにくくなる。そこで映像表示制御部５３は、このような遠くの興味位置までの視点移動が開始されると判定した場合、注目領域を小さくしつつ、並行して移動先の興味位置近傍の領域を高解像度で予め描画することとする。これにより、データ送信量の増大や描画負荷の増大を抑えつつ、ユーザーの興味を惹く領域を高解像度で表示する準備を進めることができる。 Furthermore, when the position of interest is further than a predetermined distance from the current gaze point, the video display control unit 53 may reduce the region of interest to be subjected to fovea rendering. If the user is presumed to be interested in a position that is far away from the current point of interest, the user changes the direction of his / her head and moves from the current point of interest to the point of interest at a relatively high speed. It is highly likely that you are trying to move the viewpoint from a distance. During such a large movement of the viewpoint, it is difficult to check the displayed image in detail over a wide range. Therefore, even if the area to be drawn at a high resolution is reduced, the user does not easily feel uncomfortable. Therefore, when the image display control unit 53 determines that the viewpoint movement to such a distant interest position is started, the image display control unit 53 reduces the region of interest and, in parallel, the region near the destination interest position with high resolution. Drawing is performed in advance. This makes it possible to advance preparations for displaying an area of interest to the user at a high resolution while suppressing an increase in the amount of data transmission and an increase in the drawing load.
　第５の例として、映像表示制御部５３は、興味位置を優先的に高解像度で描画してもよい。この例では、第２の例などと同様に領域ごとに解像度を変化させる中心窩レンダリングと同様の処理を実行する。ただし、第２の例などと異なり、注視点とは無関係に興味位置近傍の領域を高解像度で描画する注目領域として設定する。これにより、ユーザーの視線を自然に興味位置に誘導することができる。また、映像表示制御部５３は、興味位置近傍だけでなく、現在の注視点から興味位置までの経路を含む領域についても、高解像度で描画する対象としてもよい。また、併せて映像表示制御部５３は、興味位置の近傍やその位置までの経路以外の領域を、さらに低解像度化してもよい。 と し て As a fifth example, the video display control unit 53 may draw the position of interest with high resolution with priority. In this example, similar to the second example, processing similar to foveal rendering in which the resolution is changed for each region is executed. However, unlike the second example, the region near the position of interest is set as the region of interest to be drawn with high resolution regardless of the point of regard. Thus, the user's line of sight can be naturally guided to the position of interest. Further, the video display control unit 53 may draw not only the vicinity of the interest position but also an area including the path from the current point of interest to the interest position at a high resolution. In addition, the video display control unit 53 may further lower the resolution of the area other than the vicinity of the interest position or the route to the position.
　なお、以上説明した各種の処理は、組み合わせて適用してもよい。例えば第３の例以外の各例において他の領域より高解像度で描画するとした領域については、併せて画像の更新頻度も他領域より高くしてもよい。また、以上の説明では、中心窩レンダリングに相当する処理を実行する場合、注目領域と周辺領域の２段階で解像度を変化させることとしたが、これに限らず３段階以上の領域に映像を分割し、領域ごとに解像度や更新頻度を変化させてもよい。 The various processes described above may be applied in combination. For example, in each of the examples other than the third example, with respect to an area that is drawn at a higher resolution than the other area, the frequency of updating the image may be higher than that of the other area. Further, in the above description, when the processing corresponding to the fovea rendering is performed, the resolution is changed in two stages of the attention region and the peripheral region. However, the present invention is not limited to this, and the image is divided into three or more regions. However, the resolution and the update frequency may be changed for each area.
　以上説明したように、本発明の実施の形態に係る情報処理装置１０によれば、ユーザーの視線の時間変化に基づいて、注視点以外のユーザーが興味を惹かれている位置、又は方向を推定することができる。さらに、このようにして推定された興味位置や興味方向に応じて変化する映像をユーザーに提示することで、ユーザーの視線を自然に誘導したり、ユーザーに違和感を生じさせないようにしながら描画負荷や通信帯域を抑えた表示を実現したりすることができる。 As described above, according to the information processing apparatus 10 according to the embodiment of the present invention, a position or a direction in which a user other than the point of interest is attracted is estimated based on a temporal change of the user's line of sight. can do. Furthermore, by presenting the user with an image that changes according to the position of interest and the direction of interest estimated in this way, the user can naturally guide the user's line of sight and draw drawing load while preventing the user from feeling uncomfortable. It is possible to realize a display in which the communication band is suppressed.
　なお、本発明の実施の形態は、以上説明したものに限られない。例えば以上の説明では、カメラ２０によって表示装置４０の向きの変化を特定することとしたが、これ以外にも各種の方法で表示装置４０の向きの変化を特定してもよい。また、表示装置４０内でユーザーの視線方向の変化を特定するための方法も、以上説明したものに限られず、各種の方法を利用することができる。 The embodiments of the present invention are not limited to those described above. For example, in the above description, the change in the direction of the display device 40 is specified by the camera 20, but the change in the direction of the display device 40 may be specified by various other methods. In addition, the method for specifying the change in the user's line of sight in the display device 40 is not limited to the method described above, and various methods can be used.
　１　映像表示システム、１０　情報処理装置、１１　制御部、１２　記憶部、１３　インタフェース部、２０　カメラ、３０　操作デバイス、４０　表示装置、４１　支持体、４２　ＬＥＤ、４３　レンズ、４４　ハーフミラー、４５　眼球カメラ、５１　視線方向特定部、５２　興味位置推定部、５３　映像表示制御部。 1 image display system, 10 information processing device, 11 control unit, 12 storage unit, 13 interface unit, 20 camera, 30 operation device, 40 display device, 41 support, 42 LED, 43 lens, 44 half mirror, 45 eyeball camera , 51 gaze direction identification unit, 52 interest position estimation unit, 53 video display control unit.

Claims (8)

  1. 　表示装置が表示する映像を閲覧中のユーザーの視線方向を特定する情報を取得する視線方向特定部と、
    　前記視線方向の時間変化に含まれる特定パターンの動きに基づいて、前記ユーザーの視線が向けられている位置とは別の、当該ユーザーが潜在的に興味を向けている興味位置、及び／又は興味方向を推定する興味位置推定部と、
    　を含むことを特徴とする情報処理装置。     A line-of-sight direction identification unit that acquires information that identifies the line-of-sight direction of a user who is viewing a video displayed by the display device,
    An interest location where the user is potentially interested, and / or an interest location that is different from the location where the user's gaze is directed, based on the movement of the specific pattern included in the temporal change in the gaze direction. An interest position estimation unit for estimating a direction,
    An information processing apparatus comprising:
  2. 　請求項１に記載の情報処理装置において、
    　前記特定パターンの動きは、複数回の往復運動であって、
    　前記興味位置推定部は、前記複数回の往復運動の方向に基づいて、前記興味方向を推定する
    　ことを特徴とする情報処理装置。     The information processing apparatus according to claim 1,
    The movement of the specific pattern is a plurality of reciprocating movements,
    The information processing apparatus, wherein the interest position estimation unit estimates the interest direction based on directions of the plurality of reciprocating motions.
  3. 　請求項１又は２に記載の情報処理装置において、
    　前記興味位置推定部は、前記映像内に含まれ、前記ユーザーが興味を向ける可能性がある候補オブジェクトの位置情報を取得し、当該候補オブジェクトの位置情報を用いて、前記興味位置を推定する
    　ことを特徴とする情報処理装置。     The information processing apparatus according to claim 1, wherein
    The interest position estimating unit obtains position information of a candidate object included in the video and to which the user may be interested, and estimates the interest position using the position information of the candidate object. An information processing apparatus characterized by the above-mentioned.
  4. 　請求項１から３のいずれか一項に記載の情報処理装置において、
    　前記興味位置推定部は、前記ユーザーの前記映像に対する集中度を示す情報を取得し、当該集中度に基づいて前記興味位置、及び／又は興味方向を推定する
    　ことを特徴とする情報処理装置。     The information processing apparatus according to any one of claims 1 to 3,
    The information processing apparatus, wherein the interest position estimation unit acquires information indicating a degree of concentration of the user on the video, and estimates the position of interest and / or the direction of interest based on the degree of concentration.
  5. 　請求項４に記載の情報処理装置において、
    　前記興味位置推定部は、前記集中度を示す情報として、前記ユーザーの瞳孔径の時間変化に関する情報を取得する
    　ことを特徴とする情報処理装置。     The information processing apparatus according to claim 4,
    The information processing apparatus according to claim 1, wherein the interest position estimating unit acquires, as the information indicating the degree of concentration, information on a temporal change in a pupil diameter of the user.
  6. 　請求項５に記載の情報処理装置において、
    　前記興味位置推定部は、表示中の前記映像の輝度と、前記ユーザーの瞳孔径の計測結果と、に基づいて、前記ユーザーの集中度を推定する
    　ことを特徴とする情報処理装置。     The information processing apparatus according to claim 5,
    The information processing apparatus according to claim 1, wherein the interest position estimating unit estimates the degree of concentration of the user based on luminance of the video being displayed and a measurement result of a pupil diameter of the user.
  7. 　視線方向特定部が、表示装置が表示する映像を閲覧中のユーザーの視線方向を特定する情報を取得するステップと、
    　興味位置推定部が、前記視線方向の時間変化に含まれる特定パターンの動きに基づいて、前記ユーザーの視線が向けられている位置とは別の、当該ユーザーが潜在的に興味を向けている興味位置、及び／又は興味方向を推定するステップと、
    　を含むことを特徴とする情報処理方法。     The gaze direction identification unit acquires information that identifies the gaze direction of the user who is browsing the video displayed by the display device,
    An interest position estimating unit, based on the movement of the specific pattern included in the temporal change of the line of sight direction, is different from the position to which the user's line of sight is directed, and the interest that the user is potentially interested in Estimating a position and / or a direction of interest;
    An information processing method comprising:
  8. 　表示装置が表示する映像を閲覧中のユーザーの視線方向を特定する情報を取得するステップと、
    　前記視線方向の時間変化に含まれる特定パターンの動きに基づいて、前記ユーザーの視線が向けられている位置とは別の、当該ユーザーが潜在的に興味を向けている興味位置、及び／又は興味方向を推定するステップと、
    　をコンピュータに実行させるためのプログラム。     Acquiring information identifying the gaze direction of the user who is viewing the video displayed by the display device,
    An interest location where the user is potentially interested, and / or an interest location that is different from the location where the user's gaze is directed, based on the movement of the specific pattern included in the temporal change in the gaze direction. Estimating the direction;
    A program for causing a computer to execute.
PCT/JP2018/026940 2018-07-18 2018-07-18 Information processing device, information processing method, and program WO2020016969A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2018/026940 WO2020016969A1 (en) 2018-07-18 2018-07-18 Information processing device, information processing method, and program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2018/026940 WO2020016969A1 (en) 2018-07-18 2018-07-18 Information processing device, information processing method, and program

Publications (1)

Publication Number Publication Date
WO2020016969A1 true WO2020016969A1 (en) 2020-01-23

Family

ID=69164798

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/026940 WO2020016969A1 (en) 2018-07-18 2018-07-18 Information processing device, information processing method, and program

Country Status (1)

Country Link
WO (1) WO2020016969A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022131178A1 (en) * 2020-12-15 2022-06-23 株式会社Ｊｖｃケンウッド Web conferencing system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011042989A1 (en) * 2009-10-09 2011-04-14 Kikuchi Kouichi Viewer's feeling determination device for visually-recognized scene
WO2015008654A1 (en) * 2013-07-19 2015-01-22 ソニー株式会社 Detection device and method
WO2017025487A1 (en) * 2015-08-07 2017-02-16 SensoMotoric Instruments Gesellschaft für innovative Sensorik mbH System and method for displaying a stream of images
US20170090562A1 (en) * 2015-09-24 2017-03-30 Tobii Ab Eye-tracking enabled wearable devices

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011042989A1 (en) * 2009-10-09 2011-04-14 Kikuchi Kouichi Viewer's feeling determination device for visually-recognized scene
WO2015008654A1 (en) * 2013-07-19 2015-01-22 ソニー株式会社 Detection device and method
WO2017025487A1 (en) * 2015-08-07 2017-02-16 SensoMotoric Instruments Gesellschaft für innovative Sensorik mbH System and method for displaying a stream of images
US20170090562A1 (en) * 2015-09-24 2017-03-30 Tobii Ab Eye-tracking enabled wearable devices

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022131178A1 (en) * 2020-12-15 2022-06-23 株式会社Ｊｖｃケンウッド Web conferencing system

Similar Documents

Publication Publication Date Title
US10915180B2 (en) Systems and methods for monitoring a user's eye
US10559065B2 (en) Information processing apparatus and information processing method
US20190235624A1 (en) Systems and methods for predictive visual rendering
US8466875B2 (en) Electroencephalogram interface system, electroencephalogram interface apparatus, method, and computer program
CN107272904B (en) Image display method and electronic equipment
US11487354B2 (en) Information processing apparatus, information processing method, and program
US11561392B2 (en) Method for generating and displaying a virtual object by an optical system
US20150317956A1 (en) Head mounted display utilizing compressed imagery in the visual periphery
WO2020016970A1 (en) Information processing device, information processing method, and program
US11507184B2 (en) Gaze tracking apparatus and systems
WO2020016969A1 (en) Information processing device, information processing method, and program
TWI811613B (en) Head mounted display and control method and calibration method thereof
GB2597917A (en) Gaze tracking method and apparatus
US20220004253A1 (en) Data processing apparatus and method
JP7258620B2 (en) Image processing system and image processing method
WO2020116348A1 (en) Image processing device, image processing system, image processing method, and program
US10345899B2 (en) Method for adapting the sensorial output mode of a sensorial output device to a user
KR102564202B1 (en) Electronic device providing interaction with virtual animals for user's stress relief and control method thereof
US11874961B2 (en) Managing display of an icon in an eye tracking augmented reality device
EP4068055A1 (en) System and method for determining reference gaze data
CN115834858A (en) Display method and device, head-mounted display equipment and storage medium
KR20170090877A (en) Apparatus and Method for Interactive User Calibration

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18926887

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18926887

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP