WO2022267992A1

WO2022267992A1 - Method and apparatus for acquiring target of fixation in head-mounted display device

Info

Publication number: WO2022267992A1
Application number: PCT/CN2022/099421
Authority: WO
Inventors: 李晨
Original assignee: 北京有竹居网络技术有限公司
Priority date: 2021-06-24
Filing date: 2022-06-17
Publication date: 2022-12-29
Also published as: CN115525139A

Abstract

A method and apparatus for acquiring a target of fixation in a head-mounted display device. The method comprises: determining information of the distances from at least two target objects, which are within a field of view of a user, to the user, respectively (S110); determining information of the distance from a fixation point of a line of sight of the user to the user (S120); and on the basis of the information of the distances from the at least two target objects to the user, respectively, and the information of the distance from the fixation point of the line of sight of the user to the user, determining a target of fixation from among the at least two target objects (S130). By means of the method, a target of fixation can be determined from among a plurality of target objects, which are within a field of view of a user, and the method is particularly applicable to the situation where at least two target objects within the field of view of the user are located in the same direction of the user.

Description

在头戴式显示设备中获取注视目标的方法及装置Method and device for obtaining gaze target in head-mounted display device

相关申请的交叉引用Cross References to Related Applications

本申请基于申请号为202110703043.6、申请日为2021年06月24日，名称为“在头戴式显示设备中获取注视目标的方法及装置”的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。This application is based on a Chinese patent application with the application number 202110703043.6 and the filing date is June 24, 2021, entitled "Method and device for obtaining fixation targets in head-mounted display devices", and requests the priority of this Chinese patent application Right, the entire content of this Chinese patent application is hereby incorporated into this application as a reference.

技术领域technical field

本公开涉及图像识别技术领域，尤其涉及一种在头戴式显示设备中获取注视目标的方法及装置。The present disclosure relates to the technical field of image recognition, and in particular to a method and device for acquiring a gaze target in a head-mounted display device.

背景技术Background technique

随着智能计算机技术的发展，智能产品不断涌现，继智能手机、平板电脑之后，增强现实(AR，Augmented Reality)已有潜力成为下一个重大通用计算平台。AR头戴式显示设备是一种实现AR技术且可佩戴在用户头部进行展示的穿戴式设备，它通过计算机技术可将虚拟的信息叠加到真实世界，使真实的环境和虚拟的物体能够实时地叠加到同一个画面中，实现两种信息的相互补充，并通过诸如头盔、眼镜等设备在用户眼前进行画面展示，增强用户的现实感。With the development of smart computer technology, smart products continue to emerge. After smart phones and tablet computers, augmented reality (AR, Augmented Reality) has the potential to become the next major general-purpose computing platform. The AR head-mounted display device is a wearable device that implements AR technology and can be worn on the user's head for display. It can superimpose virtual information on the real world through computer technology, so that the real environment and virtual objects can be displayed in real time. Superimposed on the same screen, the two kinds of information complement each other, and the screen is displayed in front of the user's eyes through equipment such as helmets and glasses to enhance the user's sense of reality.

例如，在博物馆内，AR头戴式显示设备自动识别用户视场内文物，并显示视场内所有文物的介绍信息。但是就目前技术而言，当用户视场内文物数量大于或等于2件时，所有文物的介绍信息被混叠在一起显示，导致用户无法区分各介绍信息，也就无法获得所关注的文物的介绍信息。For example, in a museum, an AR head-mounted display device automatically identifies cultural relics in the user's field of view and displays introduction information of all cultural relics in the field of view. However, as far as the current technology is concerned, when the number of cultural relics in the user's field of view is greater than or equal to 2, the introduction information of all cultural relics will be mixed and displayed together, causing the user to be unable to distinguish each introduction information, and will not be able to obtain the information of the cultural relics concerned. Introduce information.

发明内容Contents of the invention

为了解决上述技术问题或者至少部分地解决上述技术问题，本公开提供了一种注视目标识别方法、装置、电子设备以及存储介质。In order to solve the above technical problem or at least partly solve the above technical problem, the present disclosure provides a gaze target recognition method, device, electronic device and storage medium.

第一方面，本公开提供了一种在头戴式显示设备中获取注视目标的方法，包括：In a first aspect, the present disclosure provides a method for obtaining a gaze target in a head-mounted display device, including:

确定用户视场内至少两个目标物分别到所述用户的距离信息；Determining distance information from at least two targets in the user's field of view to the user;

确定所述用户视线注视点到所述用户的距离信息；Determine the distance information from the user's gaze point to the user;

基于所述至少两个目标物分别到所述用户的距离信息以及所述用户视线注视点到所述用户的距离信息，在所述至少两个目标物中确定注视目标。Based on the distance information from the at least two targets to the user and the distance information from the gaze point of the user's line of sight to the user, determine a gaze target among the at least two targets.

第二方面，本公开还提供了一种在头戴式显示设备中获取注视目标的装置，包括：In the second aspect, the present disclosure also provides an apparatus for acquiring a gaze target in a head-mounted display device, including:

第一距离确定模块，用于确定用户视场内至少两个目标物分别到所述用户的距离信息；A first distance determination module, configured to determine distance information from at least two targets within the user's field of view to the user;

第二距离确定模块，用于确定所述用户视线注视点到所述用户的距离信息；A second distance determining module, configured to determine distance information from the user's gaze point to the user;

识别模块，用于基于所述至少两个目标物分别到所述用户的距离信息以及所述用户视线注视点到所述用户的距离信息，在所述至少两个目标物中确定注视目标。The identification module is configured to determine the gaze target among the at least two targets based on the distance information from the at least two targets to the user and the distance information from the gaze point of the user to the user.

第三方面，本公开还提供了一种电子设备，所述电子设备包括：In a third aspect, the present disclosure also provides an electronic device, the electronic device comprising:

一个或多个处理器；one or more processors;

存储装置，用于存储一个或多个程序；storage means for storing one or more programs;

当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如上所述的在头戴式显示设备中获取注视目标的方法。When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the method for acquiring a gaze target in a head-mounted display device as described above.

第四方面，本公开还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如上所述的在头戴式显示设备中获取注视目标的方法。In a fourth aspect, the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned method for acquiring a gaze target in a head-mounted display device is implemented.

本公开实施例提供的技术方案与现有技术相比具有如下优点：Compared with the prior art, the technical solutions provided by the embodiments of the present disclosure have the following advantages:

本公开实施例提供的技术方案通过确定用户视场内至少两个目标物分别到用户的距离信息；确定用户视线注视点到用户的距离信息；基于至少两个目标物分别到用户的距离信息以及用户视线注视点到用户的距离信息，在至少两个目标物中确定注视目标，可以实现从处于用户视场内的多个目标物中确定出注视目标。The technical solution provided by the embodiments of the present disclosure determines the distance information from at least two targets in the user's field of view to the user; determines the distance information from the user's gaze point to the user; based on the distance information from the at least two targets to the user and The distance information from the gazing point of the user's line of sight to the user determines the gazing target among at least two targets, so that the gazing target can be determined from multiple targets within the user's field of view.

本公开实施例提供的技术方案尤其适用于用户视场内至少两个目标物位于用户同一方向的情况。这是因为，假设存在两个目标物，分别为目标物A和目标物B。这两个目标物同时位于用户正前方，且呈一前一后布置。此种情况下，如果用户注视目标为目标物A，仅凭借现有的视线追踪技术仅能够确定用户正在注视其正前方，无法进一步确定用户到底在注视目标物A还是目标物B。通过本公开提供的技术方案可以唯一地确定用户到底注视的是目标物A还是目标物B，其可以提高注视目标确定的准确性。The technical solutions provided by the embodiments of the present disclosure are especially applicable to a situation where at least two targets within the user's field of view are located in the same direction of the user. This is because it is assumed that there are two targets, the target A and the target B, respectively. These two targets are located directly in front of the user at the same time, and are arranged in tandem. In this case, if the user is gazing at target A, the existing gaze tracking technology can only determine that the user is gazing directly in front of it, and it is impossible to further determine whether the user is gazing at target A or target B. The technical solution provided by the present disclosure can uniquely determine whether the user is gazing at the target A or the target B, which can improve the accuracy of gazing target determination.

本公开实施例提供的技术方案，在确定注视目标时，即使注视目标唯一的位于用户的某一方向，相对于通过用户视线方向，确定注视目标的方案，本公开中提供的技术方案通过两个距离的匹配确定注视目标，其计算量较少，确定注视目标需要的耗时少，能耗低，对头戴式显示设备性能要求低，有利于减轻设备重量，提高识别速度。The technical solutions provided by the embodiments of the present disclosure, when determining the gaze target, even if the gaze target is uniquely located in a certain direction of the user, compared with the solution of determining the gaze target through the direction of the user's line of sight, the technical solution provided in the present disclosure adopts two The matching of the distance determines the gaze target, which requires less calculation, takes less time to determine the gaze target, has low energy consumption, and has low requirements on the performance of the head-mounted display device, which is conducive to reducing the weight of the device and improving the recognition speed.

本公开实施例提供的技术方案可以实现无论用户视场内有多少个目标物，都可以准确确定出用户最可能注视的目标物，并展示该最可能注视的目标物的相关信息，确保头戴式显示设备始终显示一个物体的相关信息，其可以避免因多个物体的介绍信息被混叠在一起显示，导致用户无法区分各个信息，也就无法获得所关注的物体的介绍信息的不良现象出现，可以提高用户满意度。The technical solution provided by the embodiments of the present disclosure can accurately determine the most likely target that the user is looking at no matter how many targets are in the user's field of view, and display the relevant information of the most likely target to ensure that the head-mounted The type display device always displays the relevant information of an object, which can avoid the undesirable phenomenon that the introduction information of multiple objects is mixed and displayed, causing the user to be unable to distinguish each information, and thus unable to obtain the introduction information of the object concerned. , can improve user satisfaction.

附图说明Description of drawings

此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

为了更清楚地说明本公开实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, In other words, other drawings can also be obtained from these drawings without paying creative labor.

图1为本公开实施例提供的一种在头戴式显示设备中获取注视目标的方法的流程图；FIG. 1 is a flow chart of a method for acquiring a gaze target in a head-mounted display device according to an embodiment of the present disclosure;

图2为本公开实施例提供的一种双目测距的原理图；FIG. 2 is a schematic diagram of binocular ranging provided by an embodiment of the present disclosure;

图3为本公开实施例提供的一种用于S120的方法的流程图；FIG. 3 is a flow chart of a method for S120 provided by an embodiment of the present disclosure;

图4为本公开实施例提供的一种确定用户视线注视点到用户的距离信息的原理图；FIG. 4 is a schematic diagram of determining the distance information from the user's gaze point to the user provided by an embodiment of the present disclosure;

图5为本公开实施例提供的一种用于S121的方法的流程图；FIG. 5 is a flow chart of a method for S121 provided by an embodiment of the present disclosure;

图6为本公开实施例提供的一种眼球的结构示意图；Fig. 6 is a schematic structural diagram of an eyeball provided by an embodiment of the present disclosure;

图7为本公开实施例提供的另一种用于S121的方法的流程图；FIG. 7 is a flow chart of another method for S121 provided by an embodiment of the present disclosure;

图8为本公开实施例提供的另一种在头戴式显示设备中获取注视目标的方法的流程图；FIG. 8 is a flow chart of another method for acquiring a gaze target in a head-mounted display device according to an embodiment of the present disclosure;

图9为本公开实施例提供的一种AR头戴式显示设备的结构框图；FIG. 9 is a structural block diagram of an AR head-mounted display device provided by an embodiment of the present disclosure;

图10为本公开实施例中的一种在头戴式显示设备中获取注视目标的装置的结构示意图；FIG. 10 is a schematic structural diagram of an apparatus for acquiring a gaze target in a head-mounted display device in an embodiment of the present disclosure;

图11为本公开实施例中的一种电子设备的结构示意图。FIG. 11 is a schematic structural diagram of an electronic device in an embodiment of the present disclosure.

具体实施方式detailed description

为了能够更清楚地理解本公开的上述目的、特征和优点，下面将对本公开的方案进行进一步描述。需要说明的是，在不冲突的情况下，本公开的实施例及实施例中的特征可以相互组合。In order to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, in the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

在下面的描述中阐述了很多具体细节以便于充分理解本公开，但本公开还可以采用其他不同于在此描述的方式来实施；显然，说明书中的实施例只是本公开的一部分实施例，而不是全部的实施例。In the following description, many specific details are set forth in order to fully understand the present disclosure, but the present disclosure can also be implemented in other ways than described here; obviously, the embodiments in the description are only some of the embodiments of the present disclosure, and Not all examples.

图1为本公开实施例提供的一种在头戴式显示设备中获取注视目标的方法的流程图，本实施例可适用于从用户视场内至少两个目标物中识别出注视目标的情况，该方法可以由在头戴式显示设备中获取注视目标的装置执行，该装置可以采用软件和/或硬件的方式实现，该装置可配置于头戴式显示设备中，例如AR头盔或AR眼镜等。FIG. 1 is a flow chart of a method for obtaining a gaze target in a head-mounted display device provided by an embodiment of the present disclosure. This embodiment is applicable to the situation where a gaze target is identified from at least two targets within the user's field of view. , the method can be executed by a device for obtaining a gaze target in a head-mounted display device, the device can be implemented in software and/or hardware, and the device can be configured in a head-mounted display device, such as an AR helmet or AR glasses Wait.

如图1所示，该方法具体可以包括：As shown in Figure 1, the method may specifically include:

S110、确定用户视场内至少两个目标物分别到用户的距离信息。S110. Determine distance information from at least two targets in the field of view of the user to the user.

在本申请中，用户指头戴式显示设备的佩戴者。In this application, the user refers to the wearer of the head-mounted display device.

本步骤的实现方法有多种，示例性地，头戴式显示设备利用双目测距方法或红外测距的方法确定用户视场内至少两个目标物分别到用户的距离信息。There are many ways to implement this step. Exemplarily, the head-mounted display device uses a binocular ranging method or an infrared ranging method to determine distance information from at least two objects in the user's field of view to the user.

进一步地，若头戴式显示设备利用双目测距方法确定用户视场内至少两个目标物分别到用户的距离信息，具体可以为：头戴式显示设备利用双鱼眼摄像头获取至少两个目标物的双目图像信息；双鱼眼摄像头的视场大于用户视场；头戴式显示设备基于至少两个目标物的双目图像信息，利用双目测距方法确定用户视场内至少两个目标物分别到用户的距离信息。Further, if the head-mounted display device uses the binocular distance measurement method to determine the distance information of at least two targets in the user's field of view to the user, the specific may be: the head-mounted display device uses dual fisheye cameras to obtain at least two target objects The binocular image information of the object; the field of view of the dual fisheye camera is larger than the user's field of view; based on the binocular image information of at least two objects, the head-mounted display device uses the binocular ranging method to determine at least two targets within the user's field of view distance information from objects to users.

图2为本公开实施例提供的一种双目测距的原理图。参见图2，在头戴式显示设备中设置两个具有深度信息识别能力的鱼眼摄像头。两个鱼眼摄像头连线与用户左眼和右眼连线位于同一直线上。此处，“两个鱼眼摄像头连线与用户左眼和右眼连线位于同一直线上”应当理解为在误差允许的范围内，两个鱼眼摄像头连线与用户左眼和右眼连线位于同一直线上。FIG. 2 is a schematic diagram of binocular ranging provided by an embodiment of the present disclosure. Referring to FIG. 2 , two fisheye cameras capable of recognizing depth information are set in the head-mounted display device. The connection between the two fisheye cameras is on the same straight line as the connection between the user's left and right eyes. Here, "the connection between the two fisheye cameras and the connection between the user's left and right eyes is on the same straight line" should be understood as the connection between the two fisheye cameras and the user's left and right eyes within the allowable range of error. lines lie on the same straight line.

此时，参见图2，点O _R和点O _T代表两个鱼眼摄像头位置，点P为目标物；点P在两个摄像头的成像点分别为点P1和点P2(摄像头的成像平面经过旋转后放在镜头前方)，两个摄像头焦距均为f，B为两个摄像头之间的距离，Z为目标物P到用户的距离。设点P1和点P2之间的距离为dis，则 At this time, referring to Fig. 2, points _OR and point O _T represent the positions of two fisheye cameras, and point P is the target object; the imaging points of point P at the two cameras are points P1 and point P2 respectively (the imaging plane of the camera passes through Rotate and place in front of the lens), the focal length of the two cameras is f, B is the distance between the two cameras, and Z is the distance from the target P to the user. Let the distance between point P1 and point P2 be dis, then

dis＝B-(X _R-X _T) (1) dis=B-(X _R -X _T ) (1)

根据相似三角形原理，有According to the principle of similar triangles, we have

进一步，可得，Further, available,

在一确定的头戴式显示设备中，若两个鱼眼摄像头型号确定，焦距f确定，为定值。两个鱼眼摄像头位置设置固定，两个摄像头之间的距离B确定，为定值。因此在执行本步骤时，可以通过确定X _R-X _T(即视差)，结合式(3)就可以得到目标物P到用户的距离Z。 In a certain head-mounted display device, if the models of the two fisheye cameras are determined, the focal length f is determined and is a constant value. The positions of the two fisheye cameras are fixed, and the distance B between the two cameras is determined as a constant value. Therefore, when this step is performed, the distance Z from the target object P to the user can be obtained by determining X _R -X _T (that is, the parallax) and combining formula (3).

S120、确定用户视线注视点到用户的距离信息。S120. Determine distance information from the gaze point of the user to the user.

由于用户视线注视点为注视目标，确定用户视线注视点到用户的距离信息实质是确定被注视目标到用户的距离信息。Since the gaze point of the user's gaze is the gaze target, determining the distance information from the gaze gaze point of the user to the user is essentially determining the distance information from the gazed object to the user.

S130、基于至少两个目标物分别到用户的距离信息以及用户视线注视点到用户的距离信息，在至少两个目标物中确定注视目标。S130. Based on the distance information of the at least two targets to the user and the distance information of the gaze point of the user's line of sight to the user, determine a fixation target among the at least two targets.

可选地，逐一将各目标物到用户的距离信息以及用户视线注视点到用户的距离信息进行比较，若某一目标物到用户的距离信息和用户视线注视点到用户的距离信息一致，则该目标物为注视目标，否则该目标物不是注视目标。Optionally, compare the distance information from each target object to the user and the distance information from the user's gaze point to the user one by one, if the distance information from a certain object to the user is consistent with the distance information from the user's gaze point to the user, then The target object is the gaze target, otherwise the target object is not the gaze target.

需要说明的是，在实际执行上述技术方案时，由于存在数据采集误差、测量误差以及计算误差等，在执行本步骤时，可以设置若某个目标物到用户的距离与用户视线注视点到用户的距离之差的绝对值小于或等于预设值，则该目标物为注视目标，该预设值根据数据采集误差、测量误差以及计算误差中的一个或多个确定。It should be noted that, due to the existence of data collection errors, measurement errors, and calculation errors, etc. during the actual implementation of the above technical solutions, when performing this step, it is possible to set the distance from a certain target to the user and the distance from the user's gaze point to the user. If the absolute value of the distance difference is less than or equal to a preset value, then the target is a gaze target, and the preset value is determined according to one or more of data collection error, measurement error and calculation error.

上述技术方案通过确定用户视场内至少两个目标物分别到用户的距离信息；确定用户视线注视点到用户的距离信息；基于至少两个目标物分别到用户的距离信息以及用户视线注视点到用户的距离信息，在至少两个目标物中确定注视目标，可以实现从处于用户视场内的多个目标物中确定出注视目标。The above technical solution determines the distance information from at least two targets in the user's field of view to the user; determines the distance information from the user's gaze point to the user; based on the distance information from at least two targets to the user and the user's gaze point to The user's distance information determines the fixation target among at least two targets, so that the fixation target can be determined from multiple targets within the user's field of view.

上述技术方案尤其适用于用户视场内至少两个目标物位于用户同一方向的情况。这是因为，假设存在两个目标物，分别为目标物A和目标物B。这两个目标物同时位于用户正前方，且呈一前一后布置。此种情况下，如果用户注视目标为目标物A，仅凭借现有的视线追踪技术仅能够确定用户正在注视其正前方，无法进一步确定用户到底在注视目标物A还是目标物B。通过本公开提供的技术方案可以唯一地确定用户的注视目标，其可以提高注视目标确定的准确性。The above technical solution is especially applicable to the situation where at least two targets in the user's field of view are located in the same direction of the user. This is because it is assumed that there are two targets, the target A and the target B, respectively. These two targets are located directly in front of the user at the same time, and are arranged in tandem. In this case, if the user is gazing at target A, the existing gaze tracking technology can only determine that the user is gazing directly in front of it, and it is impossible to further determine whether the user is gazing at target A or target B. The technical solution provided by the present disclosure can uniquely determine the gaze target of the user, which can improve the accuracy of gaze target determination.

另外，在确定注视目标时，即使注视目标唯一的位于用户的某一方向，相对于通过用户视线方向，确定注视目标的方案，本公开中提供的技术方案通过两个距离的匹配确定注视目标，其计算量较少，确定注视目标需要的耗时少，能耗低，对头戴式显示设备性能要求低，有利于减轻设备重量，提高识别速度。In addition, when determining the gazing target, even if the gazing target is only located in a certain direction of the user, compared to the solution of determining the gazing target through the user's line of sight direction, the technical solution provided in this disclosure determines the gazing target through the matching of two distances, The calculation amount is less, the time-consuming to determine the gaze target is less, the energy consumption is low, and the performance requirement of the head-mounted display device is low, which is conducive to reducing the weight of the device and improving the recognition speed.

上述技术方案中，S120的具体实现方法有多种，实施例地，图3为本公开实施例提供的一种用于S120的方法的流程图。参见图3，该方法包括：In the above technical solution, there are various specific implementation methods of S120. For an embodiment, FIG. 3 is a flow chart of a method for S120 provided by an embodiment of the present disclosure. Referring to Figure 3, the method includes:

S121、头戴式显示设备确定用户左眼的视线方向信息和右眼的视线方向信息。S121. The head-mounted display device determines the gaze direction information of the user's left eye and the gaze direction information of the right eye.

S122、头戴式显示设备基于用户左眼的视线方向信息和右眼的视线方向信息，确定用户视线注视点位置信息。S122. The head-mounted display device determines the location information of the gaze point of the user's gaze based on the gaze direction information of the user's left eye and the gaze direction information of the right eye.

S123、头戴式显示设备基于用户视线注视点位置信息，确定用户视线注视点到用户的距离信息。S123. The head-mounted display device determines distance information from the user's gaze point to the user based on the location information of the user's gaze point.

图4为本公开实施例提供的一种确定用户视线注视点到用户的距离信息的原理图。参见图4，假设用户左眼的视线方向为Q1，右眼的视线方向为Q2。基于用户左眼的视线方向Q1与右眼的视线方向Q2可以得到用户视线注视点N的位置信息，基于用户视线注视点N的位置信息，可以确定用户视线注视点N到用户的距离信息M。FIG. 4 is a schematic diagram of determining the distance information from a gaze point of a user to a user provided by an embodiment of the present disclosure. Referring to FIG. 4 , it is assumed that the viewing direction of the user's left eye is Q1 , and the viewing direction of the right eye is Q2 . Based on the gaze direction Q1 of the user's left eye and the gaze direction Q2 of the right eye, the position information of the user's gaze point N can be obtained, and based on the position information of the user's gaze point N, the distance information M from the user's gaze point N to the user can be determined.

上述用于实现S120的方法，计算方法简单，易于实现。The above method for implementing S120 has a simple calculation method and is easy to implement.

进一步地，在上述技术方案中，用于实现S121的方法有多种，示例性地，可以利用巩膜-虹膜边缘法、双普尔钦像法以及瞳孔角膜反射法中的一个或多个，确定用户左眼的视线方向信息和右眼的视线方向信息。Further, in the above technical solution, there are many methods for realizing S121. For example, one or more of the sclera-iris edge method, the double Purchin image method, and the pupil-cornea reflection method can be used to determine the user's The gaze direction information of the left eye and the gaze direction information of the right eye.

下面以瞳孔角膜反射法为例，对实现S121的方法进行详细说明。图5为本公开实施例提供的一种用于S121的方法的流程图。参见图5，该方法包括：The method for implementing S121 will be described in detail below by taking the pupil cornea reflection method as an example. FIG. 5 is a flow chart of a method for S121 provided by an embodiment of the present disclosure. Referring to Figure 5, the method includes:

S1211、头戴式显示设备控制左右光源分别照射用户左眼和右眼。S1211. The head-mounted display device controls the left and right light sources to irradiate the user's left eye and right eye respectively.

左右光源固定安装于头戴式显示设备中，当用户佩戴头戴式显示设备时，左光源相对于人左眼位置固定，右光源相对于人右眼位置固定。The left and right light sources are fixedly installed in the head-mounted display device. When the user wears the head-mounted display device, the position of the left light source is fixed relative to the left eye of the person, and the position of the right light source is fixed relative to the right eye of the person.

S1212、头戴式显示设备获取图像采集设备所采集的用户左眼的图像信息和右眼的图像信息。S1212. The head-mounted display device acquires the image information of the user's left eye and the image information of the right eye collected by the image acquisition device.

图像采集设备包括用于对用户左眼图像进行采集的第一图像采集设备和用于对用户右眼图像进行采集的第二图像采集设备。第一图像采集设备和第二图像采集设备均固定安装于头戴式显示设备中。当用户佩戴头戴式显示设备时，第一图像采集设备相对于人左眼位置固定，第二图像采集设备相对于人右眼位置固定。The image acquisition device includes a first image acquisition device for acquiring the image of the user's left eye and a second image acquisition device for acquiring the image of the user's right eye. Both the first image acquisition device and the second image acquisition device are fixedly installed in the head-mounted display device. When the user wears the head-mounted display device, the position of the first image capture device is fixed relative to the left eye of the person, and the position of the second image capture device is fixed relative to the right eye of the person.

可选地，S1211中左右光源可以为红外光源，S1212图像采集设备可以为红外摄像头。由于用户眼角膜对红外光有较大的反射率，这样设置可以确保后续视线方向信息的计算结果具有较高的准确性。Optionally, the left and right light sources in S1211 may be infrared light sources, and the image acquisition device in S1212 may be an infrared camera. Since the cornea of the user's eye has a relatively high reflectivity to infrared light, this setting can ensure that the calculation result of the subsequent line-of-sight direction information has high accuracy.

S1213、头戴式显示设备基于左眼的图像信息，确定用户左眼角膜曲率中心位置信息以及左眼瞳孔中心位置信息；基于右眼的图像信息，确定用户右眼角膜曲率中心位置信息以及右眼瞳孔中心位置信息。S1213. Based on the image information of the left eye, the head-mounted display device determines the position information of the center of curvature of the user's left eye cornea and the position information of the center of the pupil of the left eye; based on the image information of the right eye, determines the position information of the center of curvature of the user's right eye cornea and the position information of the right eye Pupil center position information.

本步骤的实现方法有多种，可选地，在执行本步骤时，头戴式显示设备基于左眼的图像信息，确定左眼普尔钦斑位置信息；基于左眼普尔钦斑位置信息以及左光源的位置信息，确定用户左眼角膜曲率中心位置信息；基于左眼的图像信息以及眼球模型结构数据，确定左眼瞳孔中心位置信息。头戴式显示设备基于右眼的图像信息，确定右眼普尔钦斑位置信息；基于右眼普尔钦斑位置信息以及右光源的位置信息，确定用户右眼角膜曲率中心位置信息；基于右眼的图像信息以及眼球模型结构数据，确定右眼瞳孔中心位置信息。There are many ways to realize this step. Optionally, when performing this step, the head-mounted display device determines the position information of Purchin's spots in the left eye based on the image information of the left eye; The position information of the light source determines the position information of the center of curvature of the cornea of the user's left eye; based on the image information of the left eye and the structural data of the eyeball model, the position information of the pupil center of the left eye is determined. Based on the image information of the right eye, the head-mounted display device determines the position information of the Purchin spot of the right eye; based on the position information of the Purchin spot of the right eye and the position information of the right light source, determines the position information of the center of curvature of the user's right eye cornea; The image information and the eyeball model structure data are used to determine the position information of the pupil center of the right eye.

当光源照射在角膜上会产生一个闪烁点，该闪烁点为普尔钦斑(Purkinje image)，该闪烁点由进入瞳孔的光线在角膜外表面上反射(cornealreflection，CR)而形成。由于眼球近似球体，照射在上面的闪烁点位置基本不会随着眼球的转动而改变。When the light source shines on the cornea, a flickering point will be produced, which is called Purkinje image. The flickering point is formed by the reflection of light entering the pupil on the outer surface of the cornea (corneal reflection, CR). Since the eyeball is similar to a sphere, the position of the flickering point irradiated on it basically does not change with the rotation of the eyeball.

S1214、头戴式显示设备基于用户左眼角膜曲率中心位置信息以及左眼瞳孔中心位置信息，确定用户左眼的视线方向信息。S1214. The head-mounted display device determines the gaze direction information of the user's left eye based on the position information of the center of curvature of the cornea of the user's left eye and the position information of the center of the pupil of the left eye.

研究表明，用户在注视物体时，用户的眼球的视线会随着所注视的物体的位置进行变化。但是在这个过程中，角膜的绝对位置并不随眼球的转动而变化，而角膜的位置相对于瞳孔和眼球的位置则是在不断变化的。例如，当用户眼目视前方时，角膜在瞳孔的下方；当用户眼看向右方时，角膜在瞳孔的左侧；当用户眼球看向左侧时，角膜点在瞳孔的右侧。因此，只要确定了瞳孔和角膜的相对位置，它们之间形成的向量的方向可以作为用户眼视线的方向。Studies have shown that when a user is gazing at an object, the line of sight of the user's eyeballs will change with the position of the gazing object. But in this process, the absolute position of the cornea does not change with the rotation of the eyeball, but the position of the cornea relative to the pupil and the position of the eyeball is constantly changing. For example, when the user looks forward, the cornea is below the pupil; when the user looks right, the cornea is on the left side of the pupil; when the user looks left, the cornea is on the right side of the pupil. Therefore, as long as the relative positions of the pupil and the cornea are determined, the direction of the vector formed between them can be used as the direction of the line of sight of the user's eyes.

因此，可选地，可以将由左眼角膜曲率中心位置指向左眼瞳孔中心位置的方向作为用户左眼的视线方向。示例性地，图6为本公开实施例提供的一种眼球的结构示意图。图6中，点O ₁为瞳孔中心，点O ₂为角膜曲率中心。由角膜曲率中心O ₂指向瞳孔中心O1的方向为该眼的视线方向。 Therefore, optionally, the direction from the center of curvature of the cornea of the left eye to the center of the pupil of the left eye may be taken as the line of sight direction of the user's left eye. Exemplarily, FIG. 6 is a schematic structural diagram of an eyeball provided by an embodiment of the present disclosure. In FIG. ₆ , point O1 is the pupil center, and point _O2 is the corneal curvature center. The direction from the corneal curvature center _O2 to the pupil center O1 is the line of sight direction of the eye.

S1215、头戴式显示设备基于用户右眼角膜曲率中心位置信息以及右眼瞳孔中心位置信息，确定用户右眼的视线方向信息。S1215. The head-mounted display device determines the sight direction information of the user's right eye based on the position information of the center of curvature of the cornea of the user's right eye and the position information of the center of the pupil of the right eye.

类似地，可以将由右眼角膜曲率中心位置指向右眼瞳孔中心位置的方向，作为用户右眼的视线方向。Similarly, the direction from the center of curvature of the cornea of the right eye to the center of the pupil of the right eye may be taken as the line of sight direction of the user's right eye.

上述技术方案的本质是根据人眼的生理特性及视觉成像原理，利用图像处理技术对采集到的用户眼部的图像信息进行处理，获得用于视线估计的用户眼睛特征参数，以得到的用户眼睛特征参数为基准点，采用相应的映射模型得到视线落点坐标，实现视线的跟踪。这种方法精度较高，对用户无干扰，允许用户头部自由转动。The essence of the above-mentioned technical solution is to use image processing technology to process the collected image information of the user's eye according to the physiological characteristics of the human eye and the principles of visual imaging, to obtain the characteristic parameters of the user's eye for line of sight estimation, so as to obtain the user's eye The characteristic parameter is the reference point, and the coordinates of the line of sight are obtained by using the corresponding mapping model, so as to realize the tracking of the line of sight. This method has high precision, is non-disturbing to the user, and allows the user to rotate the head freely.

继续参见图6，根据眼睛生理结构，由黄斑中心凹P2指向瞳孔中心O ₁的方向为视轴方向，即用户眼实际注视方向。而利用由角膜中心O ₂指向瞳孔中心O ₁的方向为光轴方向。显然，光轴方向并不是用户眼睛实际注视方向，只是接近于用户眼睛实际注视方向。因此，图6中技术方案最终所得的左眼的视线方向信息和右眼的视线方向信息与用户眼睛实际注视方向略有差别，该差别会导致误差出现。针对于该误差，可以采用在执行S130时，通过设置合适的误差范围，以实现最终在至少两个目标物中确定注视目标的目的。 Continuing to refer to FIG. 6 , according to the physiological structure of the eye, the direction from the fovea P2 to the pupil center O1 is the visual axis direction, that is, the actual gaze direction _of the user's eyes. And the direction from the cornea center _O2 to the pupil center _O1 is used as the optical axis direction. Obviously, the direction of the optical axis is not the actual gaze direction of the user's eyes, but close to the actual gaze direction of the user's eyes. Therefore, the sight direction information of the left eye and the sight direction information of the right eye finally obtained by the technical solution in FIG. 6 are slightly different from the actual gaze direction of the user's eyes, and this difference will lead to errors. For this error, it may be used to set an appropriate error range when performing S130, so as to finally determine the fixation target among at least two targets.

可选地，还可以对该光轴方向进行修正，以使得光轴方向无限接近用户眼睛实际注视方向。对光轴方向进行修正的方法有多种，示例性地，图7为本公开实施例提供的另一种用于S121的方法的流程图。参见图7，该方法包括：Optionally, the optical axis direction may also be corrected so that the optical axis direction is infinitely close to the actual gaze direction of the user's eyes. There are multiple methods for correcting the direction of the optical axis. Exemplarily, FIG. 7 is a flow chart of another method for S121 provided by an embodiment of the present disclosure. Referring to Figure 7, the method includes:

S1210、头戴式显示设备获取左眼误差补偿角信息和右眼误差补偿角信息。S1210. The head-mounted display device acquires error compensation angle information for the left eye and error compensation angle information for the right eye.

参见图6，误差补偿角即图6中的∠θ。Referring to Fig. 6, the error compensation angle is ∠θ in Fig. 6 .

本步骤的实现方法有多种，示例性地，本步骤的实现方法包括：There are many ways to implement this step. Exemplarily, the implementation methods of this step include:

首先，头戴式显示设备输出标定指令，以使用户双眼注视位置信息确定的标定物。First, the head-mounted display device outputs a calibration instruction, so that the user's eyes are fixed on the calibration object determined by the position information.

此处，“位置信息确定的标定物”是指，标定物与用于执行本注视目标识别方法的头戴式显示设备的相对位置关系固定，且标定物的位置信息是已知的。Here, "a calibration object determined by position information" means that the relative positional relationship between the calibration object and the head-mounted display device used to implement the gaze target recognition method is fixed, and the location information of the calibration object is known.

标定指令是指，提示用户双眼注视标定物的指令，例如，可以通过发出语音提醒信息的方式，以提醒用户双眼注视标定物。The calibration instruction refers to an instruction that prompts the user to gaze at the calibration object with both eyes. For example, a voice reminder message may be issued to remind the user to gaze at the calibration object with both eyes.

其次、头戴式显示设备控制左右光源分别照射用户左眼和右眼。Secondly, the head-mounted display device controls the left and right light sources to illuminate the user's left eye and right eye respectively.

再次、在用户双眼注视标定物的情况下，头戴式显示设备获取图像采集设备所采集的用户左眼的图像信息和右眼的图像信息。Thirdly, when the user's eyes are fixed on the calibration object, the head-mounted display device acquires the image information of the user's left eye and the image information of the right eye collected by the image acquisition device.

再次、头戴式显示设备基于左眼的图像信息，分别确定用户左眼角膜曲率中心位置信息和左眼瞳孔中心位置信息；基于右眼的图像信息，分别确定用户右眼角膜曲率中心位置信息和右眼瞳孔中心位置信息。Again, based on the image information of the left eye, the head-mounted display device determines the position information of the center of curvature of the user's left eye cornea and the position information of the center of the pupil of the left eye; based on the image information of the right eye, respectively determines the position information of the center of curvature of the user's right eye cornea and The position information of the pupil center of the right eye.

再次、头戴式显示设备基于左眼角膜曲率中心位置信息和左眼瞳孔中心位置信息，确定与标定物对应的用户左眼的光轴方向信息；基于右眼角膜曲率中心位置信息和右眼瞳孔中心位置信息，确定与标定物对应的用户右眼的光轴方向信息。Thirdly, the head-mounted display device determines the optical axis direction information of the user's left eye corresponding to the calibration object based on the position information of the center of curvature of the left eye cornea and the position information of the center of the pupil of the left eye; The center position information determines the optical axis direction information of the user's right eye corresponding to the calibration object.

再次、头戴式显示设备基于标定物的位置信息，确定与标定物对应的用户左眼的视轴方向信息和右眼的视轴方向信息。Thirdly, based on the position information of the calibration object, the head-mounted display device determines the visual axis direction information of the user's left eye and the visual axis direction information of the right eye corresponding to the calibration object.

最后、头戴式显示设备基于与标定物对应的用户左眼的光轴方向信息和左眼视轴方向信息，确定左眼误差补偿角信息；基于与标定物对应的用户右眼的光轴方向信息和右眼视轴方向信息，确定右眼误差补偿角信息。Finally, the head-mounted display device determines the error compensation angle information of the left eye based on the optical axis direction information of the user's left eye corresponding to the calibration object and the visual axis direction information of the left eye; based on the optical axis direction of the user's right eye corresponding to the calibration object information and the direction information of the visual axis of the right eye to determine the error compensation angle information of the right eye.

需要说明的是，上述实现S1210的过程可视为标定过程，在标定过程，用户注视的是位置信息确定的标定物。标定过程可以理解为用户在初次使用头戴式显示设备时的参数配置过程。而下面S1211-S1215不属于标定过程，是用户在实际使用头戴式显示设备时，对用户视野中注视目标进行识别的过程。It should be noted that the above process of implementing S1210 can be regarded as a calibration process, in which the user looks at the calibration object determined by the position information. The calibration process can be understood as a parameter configuration process when the user uses the head-mounted display device for the first time. The following S1211-S1215 do not belong to the calibration process, but are the process of identifying the gaze target in the user's field of vision when the user actually uses the head-mounted display device.

S1216、头戴式显示设备基于左眼误差补偿角信息，对用户左眼的视线方向信息进行修正。S1216. The head-mounted display device corrects the line-of-sight direction information of the user's left eye based on the left-eye error compensation angle information.

S1217、头戴式显示设备基于右眼误差补偿角信息，对用户右眼的视线方向信息进行修正。S1217. The head-mounted display device corrects the line-of-sight direction information of the user's right eye based on the right-eye error compensation angle information.

上述技术方案利用左眼误差补偿角信息对用户左眼的视线方向信息进行修正，利用右眼误差补偿角信息对用户右眼的视线方向信息进行修正，其可以消除用户眼视轴与光轴固有的生理偏差，得到真正视线的方向以及注视点的位置，可以提高注视目标识别的准确性。The above technical solution uses the left-eye error compensation angle information to correct the line-of-sight direction information of the user's left eye, and uses the right-eye error compensation angle information to correct the line-of-sight direction information of the user's right eye, which can eliminate the user's visual axis and optical axis. The physiological deviation, the direction of the real line of sight and the position of the fixation point can be obtained, which can improve the accuracy of fixation target recognition.

图8为本公开实施例提供的另一种在头戴式显示设备中获取注视目标的方法的流程图。图8为图1中的一个具体示例。图9为本公开实施例提供的一种AR头戴式显示设备的结构框图。该AR头戴式显示设备可以执行图8中提供的在头戴式显示设备中获取注视目标的方法。FIG. 8 is a flow chart of another method for acquiring a gaze target in a head-mounted display device according to an embodiment of the present disclosure. FIG. 8 is a specific example in FIG. 1 . FIG. 9 is a structural block diagram of an AR head-mounted display device provided by an embodiment of the present disclosure. The AR head-mounted display device may execute the method for acquiring a gaze target in the head-mounted display device provided in FIG. 8 .

参见图9，该头戴式显示设备包括互相连接的数字信号处理(DSP，Digital Signal Process)模块和中央处理器(CPU，central processing unit)。数字信号处理模块与多个传感器连接，用于对多个传感器的原始数据进行预处理。与数字信号处理模块连接的传感器包括但不限于惯性测量单元(IMU，Inertial Measurement Unit)、RGB摄像头、双鱼眼摄像头、红外摄像头。此外数字信号处理模块还与红外光源连接。中央处理器与麦克风、喇叭、光学显示模块以及电池连接，用于对麦克风，喇叭，光学显示模块等进行驱动，数据处理，算法比较等。Referring to FIG. 9, the head-mounted display device includes a digital signal processing (DSP, Digital Signal Process) module and a central processing unit (CPU, central processing unit) connected to each other. The digital signal processing module is connected with multiple sensors, and is used for preprocessing the raw data of multiple sensors. The sensors connected to the digital signal processing module include but are not limited to an inertial measurement unit (IMU, Inertial Measurement Unit), an RGB camera, a dual fisheye camera, and an infrared camera. In addition, the digital signal processing module is also connected with the infrared light source. The central processing unit is connected with the microphone, the speaker, the optical display module and the battery, and is used for driving the microphone, the speaker, the optical display module, etc., data processing, algorithm comparison, etc.

参见图8和图9，该头戴式显示设备佩戴于用户身上，该方法包括：Referring to FIG. 8 and FIG. 9, the head-mounted display device is worn on the user, and the method includes:

S210、头戴式显示设备获取用户视场图像信息，并识别处于用户视场内的目标物。S210. The head-mounted display device acquires image information of the user's field of view, and identifies a target within the user's field of view.

示例性地，启动头戴式显示设备后，当检测到用户动作变化时，自动进行目标物搜索，获取用户视场内所有目标物。Exemplarily, after the head-mounted display device is started, when a change in the user's motion is detected, an object search is automatically performed to obtain all objects within the user's field of view.

本步骤中，通过惯性测量单元和/或红外摄像头检测用户动作变化。用户动作变化包括但不限于移动、转头或者转动眼球。通过用户动作变化来触发并带动摄像头搜索目标物。In this step, changes in user actions are detected by an inertial measurement unit and/or an infrared camera. User motion changes include but are not limited to moving, turning head, or eyeballs. The camera is triggered and driven to search for the target by changing the user's actions.

通常，双鱼眼摄像头的视野范围会大于用户眼视野。但在实际中，图像识别只需要识别用户眼正常观察事物的视野范围即可，其通常小于用户眼的极限视场和双鱼眼视场。因此，设置用户视场为水平130°，垂直90°。超出后认为需要转头或移动去调整。Usually, the field of view of the dual fisheye camera will be larger than that of the user's eye. But in practice, image recognition only needs to recognize the field of view of the user's eyes to observe things normally, which is usually smaller than the limit field of view of the user's eyes and the double fisheye field of view. Therefore, set the user field of view to 130° horizontally and 90° vertically. After exceeding, it is considered that the head needs to be turned or moved to adjust.

具体地，RGB摄像头采集用户视场内所有目标物RGB信息，协助完成目标物识别。若该AR头戴式显示设备用于向用户介绍博物馆展示的文物。这里目标物识别应当理解为识别目标物具体为何种文物。Specifically, the RGB camera collects the RGB information of all objects in the user's field of view to assist in object recognition. If the AR head-mounted display device is used to introduce the cultural relics displayed in the museum to the user. Here, target recognition should be understood as identifying what kind of cultural relic the target is.

S220、头戴式显示设备判断用户视场内是否仅包括一个目标物；若是，执行S230；若否，执行S240。S220. The head-mounted display device judges whether there is only one target in the field of view of the user; if yes, execute S230; if not, execute S240.

S230、控制头戴式显示设备通过虚拟图像显示与该目标物关联的信息。S230. Control the head-mounted display device to display information associated with the target object through a virtual image.

示例性地，若该AR头戴式显示设备用于向用户介绍博物馆展示的文物。“与该目标物对应的信息”为目标物的介绍信息。Exemplarily, if the AR head-mounted display device is used to introduce cultural relics displayed in a museum to a user. "Information corresponding to the object" is introduction information of the object.

可选地，在本步骤之前，还包括：头戴式显示设备将目标物的图像与数据库中的图像进行匹配，得到与目标物关联的信息。Optionally, before this step, it also includes: the head-mounted display device matches the image of the target with the images in the database to obtain information associated with the target.

数据库中存储有多个物体的相关信息，如物体的图像以及物体的介绍信息。由于用户视场内仅有一个目标物，首先从用户视场图像信息中确定目标物，然后利用目标物图像与数据库中各物体的图像进行匹配，若目标物图像与数据库中某个物体的图像的相似度大于设定阈值，确定目标物图像为该物体，将该物体的介绍信息作为与目标物关联的信息，通过虚拟图像的方式予以展示。The database stores information related to multiple objects, such as images of the objects and introduction information of the objects. Since there is only one target in the user's field of view, first determine the target from the image information of the user's field of view, and then use the target image to match the images of each object in the database. If the target image matches the image of an object in the database If the similarity is greater than the set threshold, the image of the target object is determined to be the object, and the introduction information of the object is used as the information associated with the target object, and displayed in the form of a virtual image.

S240、确定用户视场内至少两个目标物分别到用户的距离信息。S240. Determine distance information from at least two targets within the user's field of view to the user.

具体地，双鱼眼摄像头完成环境深度信息采集，协助中央处理器得到各目标物到用户的距离信息。Specifically, the dual fisheye camera completes the collection of environmental depth information and assists the central processing unit in obtaining distance information from each target object to the user.

S250、确定用户视线注视点到用户的距离信息。S250. Determine distance information from the gaze point of the user to the user.

具体地，利用红外摄像头与红外光源完成角膜图像采集，协助中央处理器检测角膜中心与瞳孔中心，并计算出视线注视点，进而得到用户视线注视点到用户的距离信息。Specifically, the infrared camera and infrared light source are used to collect corneal images, assist the central processor to detect the center of the cornea and the center of the pupil, and calculate the gaze point, and then obtain the distance information from the user's gaze point to the user.

S260、基于至少两个目标物分别到用户的距离信息以及用户视线注视点到用户的距离信息，在至少两个目标物中确定注视目标。S260. Based on the distance information from the at least two targets to the user and the distance information from the gaze point of the user's line of sight to the user, determine a fixation target among the at least two targets.

S270、控制头戴式显示设备通过虚拟图像仅显示与注视目标关联的信息。S270. Control the head-mounted display device to display only information associated with the gaze target through the virtual image.

示例性地，若该AR头戴式显示设备用于向用户介绍博物馆展示的文物。“与该注视目标对应的信息”为注视目标的介绍信息。Exemplarily, if the AR head-mounted display device is used to introduce cultural relics displayed in a museum to a user. "Information corresponding to the fixation target" is introduction information of the fixation target.

可选地，在本步骤之前，还包括：头戴式显示设备将注视目标的图像与数据库中的图像进行匹配，得到与注视目标关联的信息。Optionally, before this step, it also includes: the head-mounted display device matches the image of the gaze target with images in the database to obtain information associated with the gaze target.

数据库中存储有多个物体的相关信息，如物体的图像以及物体的介绍信息。在确定出注视目标后，首先从用户视场图像信息中确定注视目标图像，然后利用注视目标图像与数据库中各物体的图像进行匹配，若注视目标图像与数据库中某个物体的图像的相似度大于设定阈值，确定注视目标图像为该物体，将该物体的介绍信息作为与注视目标关联的信息，通过虚拟图像的方式予以展示。The database stores information related to multiple objects, such as images of the objects and introduction information of the objects. After determining the gaze target, first determine the gaze target image from the image information of the user's field of view, and then use the gaze target image to match the images of each object in the database. If the similarity between the gaze target image and the image of an object in the database If it is greater than the set threshold, it is determined that the gaze target image is the object, and the introduction information of the object is used as information associated with the gaze target, and displayed in the form of a virtual image.

上述技术方案可以实现无论用户视场内有多少个目标物，都可以准确确定出用户最可能注视的目标物，并展示该最可能注视的目标物的相关信息，确保头戴式显示设备始终显示一个物体的相关信息，其可以避免因多个物体的介绍信息被混叠在一起显示，导致用户无法区分各个信息，也就无法获得所关注的物体的介绍信息的不良现象出现，可以提高用户满意度。The above technical solution can realize that no matter how many targets are in the user's field of view, it can accurately determine the target that the user is most likely to focus on, and display the relevant information of the target that is most likely to focus on, ensuring that the head-mounted display device always displays The related information of an object can avoid the undesirable phenomenon that the introduction information of multiple objects is mixed together, causing the user to be unable to distinguish each information, and thus unable to obtain the introduction information of the object concerned, which can improve user satisfaction Spend.

需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本发明并不受所描述的动作顺序的限制，因为依据本发明，某些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本发明所必须的。It should be noted that for the foregoing method embodiments, for the sake of simple description, they are expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence. Because of the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

图10为本公开实施例中的一种在头戴式显示设备中获取注视目标的装置的结构示意图。本公开实施例所提供的在头戴式显示设备中获取注视目标的装置可以配置于头戴式显示设备中。参见图10，该在头戴式显示设备中获取注视目标的装置具体包括：FIG. 10 is a schematic structural diagram of an apparatus for acquiring a gaze target in a head-mounted display device in an embodiment of the present disclosure. The apparatus for obtaining a fixation target in a head-mounted display device provided in an embodiment of the present disclosure may be configured in the head-mounted display device. Referring to FIG. 10, the device for obtaining a gaze target in a head-mounted display device specifically includes:

第一距离确定模块310，用于确定用户视场内至少两个目标物分别到用户的距离信息；The first distance determination module 310 is configured to determine the distance information of at least two targets within the user's field of view to the user;

第二距离确定模块320，用于确定用户视线注视点到用户的距离信息；The second distance determination module 320 is used to determine the distance information from the user's gaze point to the user;

识别模块330，用于基于所述至少两个目标物分别到用户的距离信息以及所述用户视线注视点到用户的距离信息，在所述至少两个目标物中确定注视目标。The identification module 330 is configured to determine a gaze target among the at least two targets based on the distance information from the at least two targets to the user and the distance information from the gaze point of the user's line of sight to the user.

进一步地，识别模块330，还用于确定用户视场内至少两个目标物分别到所述用户的距离信息之前，控制头戴式显示设备获取用户视场图像信息，并识别处于用户视场内的目标物。Further, the recognition module 330 is also configured to control the head-mounted display device to obtain image information of the user's field of view and identify objects within the user's field of view before determining the distance information of at least two objects in the user's field of view to the user. target.

进一步地，若用户视场内目标物的个数大于或等于2，第一距离确定模块310，执行确定用户视场内至少两个目标物分别到所述用户的距离信息的步骤；Further, if the number of targets in the user's field of view is greater than or equal to 2, the first distance determination module 310 executes the step of determining the distance information of at least two targets in the user's field of view to the user;

该装置还包括显示模块，用于基于所述至少两个目标物分别到所述用户的距离信息以及所述用户视线注视点到所述用户的距离信息，在所述至少两个目标物中确定注视目标之后，控制头戴式显示设备通过虚拟图像仅显示与所述注视目标关联的信息。The device also includes a display module, configured to determine among the at least two targets based on the distance information from the at least two targets to the user and the distance information from the gaze point of the user to the user. After gazing at the target, the head-mounted display device is controlled to display only the information associated with the gazing target through the virtual image.

进一步地，识别模块330，还用于控制头戴式显示设备通过虚拟图像仅显示与所述注视目标关联的信息之前，控制头戴式显示设备将所述注视目标的图像与数据库中的图像进行匹配，得到与所述注视目标关联的信息。Furthermore, the recognition module 330 is also used to control the head-mounted display device to compare the image of the gaze target with the images in the database before displaying only the information associated with the gaze target through the virtual image. Match to obtain the information associated with the gaze target.

进一步地，第一距离确定模块310，用于控制头戴式显示设备利用双目测距方法确定用户视场内至少两个目标物分别到用户的距离信息。Further, the first distance determination module 310 is configured to control the head-mounted display device to determine distance information from at least two targets within the user's field of view to the user by using a binocular ranging method.

进一步地，第一距离确定模块310，用于控制头戴式显示设备利用双鱼眼摄像头获取至少两个目标物的双目图像信息；所述双鱼眼摄像头的视场大于用户视场；控制头戴式显示设备基于至少两个目标物的双目图像信息，利用双目测距方法确定用户视场内至少两个目标物分别到用户的距离信息。Further, the first distance determination module 310 is used to control the head-mounted display device to acquire binocular image information of at least two targets by using the dual fisheye camera; the field of view of the dual fisheye camera is larger than the user's field of view; Based on the binocular image information of at least two targets, the display device uses a binocular ranging method to determine the distance information between at least two targets in the user's field of view and the user respectively.

进一步地，第二距离确定模块320，用于：Further, the second distance determining module 320 is used for:

控制头戴式显示设备确定用户左眼的视线方向信息和右眼的视线方向信息；Control the head-mounted display device to determine the gaze direction information of the user's left eye and the gaze direction information of the right eye;

控制头戴式显示设备基于用户左眼的视线方向信息和右眼的视线方向信息，确定用户视线注视点位置信息；Control the head-mounted display device to determine the position information of the gaze point of the user's gaze based on the gaze direction information of the user's left eye and the gaze direction information of the right eye;

控制头戴式显示设备基于用户视线注视点位置信息，确定所述用户视线注视点到所述用户的距离信息。Controlling the head-mounted display device to determine distance information from the user's gaze point to the user based on the location information of the user's gaze point.

进一步地，第二距离确定模块320，用于：控制头戴式显示设备利用瞳孔角膜反射法确定用户左眼的视线方向信息和右眼的视线方向信息。Further, the second distance determination module 320 is configured to: control the head-mounted display device to determine the gaze direction information of the user's left eye and the gaze direction information of the right eye by using the pupil cornea reflection method.

控制头戴式显示设备控制左右光源分别照射用户左眼和右眼；Control the head-mounted display device to control the left and right light sources to illuminate the user's left and right eyes respectively;

控制头戴式显示设备获取图像采集设备所采集的用户左眼的图像信息和右眼的图像信息；Control the head-mounted display device to acquire the image information of the user's left eye and the image information of the right eye collected by the image acquisition device;

控制头戴式显示设备基于所述左眼的图像信息，确定用户左眼角膜曲率中心位置信息以及左眼瞳孔中心位置信息；基于所述右眼的图像信息，确定用户右眼角膜曲率中心位置信息以及右眼瞳孔中心位置信息；Controlling the head-mounted display device to determine the position information of the center of curvature of the cornea of the user's left eye and the position of the center of the pupil of the left eye based on the image information of the left eye; determine the position information of the center of curvature of the cornea of the user's right eye based on the image information of the right eye And the position information of the pupil center of the right eye;

控制头戴式显示设备基于所述用户左眼角膜曲率中心位置信息以及所述左眼瞳孔中心位置信息，确定用户左眼的视线方向信息；Controlling the head-mounted display device to determine the sight direction information of the user's left eye based on the position information of the center of curvature of the user's left eye cornea and the position information of the center of the pupil of the left eye;

控制头戴式显示设备基于所述用户右眼角膜曲率中心位置信息以及所述右眼瞳孔中心位置信息，确定用户右眼的视线方向信息。Controlling the head-mounted display device to determine the line-of-sight direction information of the user's right eye based on the position information of the center of curvature of the cornea of the user's right eye and the position information of the center of the pupil of the right eye.

控制头戴式显示设备获取左眼误差补偿角信息和右眼误差补偿角信息；Control the head-mounted display device to obtain the error compensation angle information of the left eye and the error compensation angle information of the right eye;

控制头戴式显示设备基于所述左眼误差补偿角信息，对用户左眼的视线方向信息进行修正；Controlling the head-mounted display device to correct the line-of-sight direction information of the user's left eye based on the left-eye error compensation angle information;

控制头戴式显示设备基于所述右眼误差补偿角信息，对用户右眼的视线方向信息进行修正；Controlling the head-mounted display device to correct the line-of-sight direction information of the user's right eye based on the right-eye error compensation angle information;

控制头戴式显示设备基于修正后的左眼的视线方向信息和修正后的右眼的视线方向信息，确定用户视线注视点位置信息。The head-mounted display device is controlled to determine position information of the gaze point of the user based on the corrected gaze direction information of the left eye and the corrected gaze direction information of the right eye.

控制头戴式显示设备输出标定指令，以使用户双眼注视位置信息确定的标定物；Control the head-mounted display device to output a calibration command, so that the user's eyes are fixed on the calibration object determined by the position information;

在用户双眼注视所述标定物的情况下，控制头戴式显示设备获取所述图像采集设备所采集的用户左眼的图像信息和右眼的图像信息；When the user's eyes are watching the calibration object, control the head-mounted display device to acquire the image information of the user's left eye and the image information of the right eye collected by the image acquisition device;

控制头戴式显示设备基于所述左眼的图像信息，分别确定用户左眼角膜曲率中心位置信息和左眼瞳孔中心位置信息；基于所述右眼的图像信息，分别确定用户右眼角膜曲率中心位置信息和右眼瞳孔中心位置信息；Controlling the head-mounted display device to determine the position information of the center of curvature of the cornea of the user's left eye and the position of the center of the pupil of the left eye based on the image information of the left eye; respectively determine the center of curvature of the cornea of the user's right eye based on the image information of the right eye Position information and right eye pupil center position information;

控制头戴式显示设备基于左眼角膜曲率中心位置信息和左眼瞳孔中心位置信息，确定与所述标定物对应的用户左眼的光轴方向信息；基于右眼角膜曲率中心位置信息和右眼瞳孔中心位置信息，确定与所述标定物对应的用户右眼的光轴方向信息；Control the head-mounted display device to determine the optical axis direction information of the user's left eye corresponding to the calibration object based on the position information of the center of curvature of the left eye cornea and the position information of the center of the pupil of the left eye; Pupil center position information, determining the optical axis direction information of the user's right eye corresponding to the calibration object;

控制头戴式显示设备基于所述标定物的位置信息，确定与所述标定物对应的用户左眼的视轴方向信息和右眼的视轴方向信息；Controlling the head-mounted display device to determine the visual axis direction information of the user's left eye and the visual axis direction information of the right eye corresponding to the calibration object based on the position information of the calibration object;

控制头戴式显示设备基于与所述标定物对应的用户左眼的光轴方向信息和左眼视轴方向信息，确定左眼误差补偿角信息；基于与所述标定物对应的用户右眼的光轴方向信息和右眼视轴方向信息，确定右眼误差补偿角信息。controlling the head-mounted display device to determine the error compensation angle information of the left eye based on the optical axis direction information of the user's left eye corresponding to the calibration object and the visual axis direction information of the left eye; The optical axis direction information and the visual axis direction information of the right eye determine the error compensation angle information of the right eye.

本公开实施例提供的在头戴式显示设备中获取注视目标的装置，可执行本公开方法实施例所提供的在头戴式显示设备中获取注视目标的方法中头戴式显示设备所执行的步骤，具备执行步骤和有益效果，此处不再赘述。The apparatus for obtaining a gaze target in a head-mounted display device provided by an embodiment of the present disclosure may execute the method performed by the head-mounted display device in the method for obtaining a gaze target in a head-mounted display device provided in the method embodiment of the present disclosure. The steps have execution steps and beneficial effects, and will not be repeated here.

图11为本公开实施例中的一种电子设备的结构示意图。下面具体参考图11，其示出了适于用来实现本公开实施例中的电子设备1000的结构示意图。可选地，所述电子设备为AR头戴式显示设备。如AR眼镜或AR头盔等。图11示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。FIG. 11 is a schematic structural diagram of an electronic device in an embodiment of the present disclosure. Referring specifically to FIG. 11 , it shows a schematic structural diagram of an electronic device 1000 suitable for implementing an embodiment of the present disclosure. Optionally, the electronic device is an AR head-mounted display device. Such as AR glasses or AR helmets, etc. The electronic device shown in FIG. 11 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.

如图11所示，电子设备1000可以包括处理装置(例如中央处理器、图形处理器等)1001，其可以根据存储在只读存储器(ROM)1002中的程序或者从存储装置1008加载到随机访问存储器(RAM)1003中的程序而执行各种适当的动作和处理以实现如本公开所述的实施例的在头戴式显示设备中获取注视目标的方法。在RAM 1003中，还存储有电子设备1000操作所需的各种程序和信息。处理装置1001、ROM 1002以及RAM 1003通过总线1004彼此相连。输入/输出(I/O)接口1005也连接至总线1004。As shown in FIG. 11, an electronic device 1000 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 1001, which may be randomly accessed according to a program stored in a read-only memory (ROM) 1002 or loaded from a storage device 1008. The program in the memory (RAM) 1003 executes various appropriate actions and processes to implement the method for acquiring a gaze target in a head-mounted display device according to an embodiment of the present disclosure. In the RAM 1003, various programs and information necessary for the operation of the electronic device 1000 are also stored. The processing device 1001, ROM 1002, and RAM 1003 are connected to each other through a bus 1004. An input/output (I/O) interface 1005 is also connected to the bus 1004 .

通常，以下装置可以连接至I/O接口1005：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置1006；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置1007；包括例如磁带、硬盘等的存储装置1008；以及通信装置1009。通信装置1009可以允许电子设备1000与其他设备进行无线或有线通信以交换信息。虽然图11示出了具有各种装置的电子设备1000，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。Typically, the following devices can be connected to the I/O interface 1005: input devices 1006 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibration an output device 1007 such as a computer; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication means 1009 may allow the electronic device 1000 to perform wireless or wired communication with other devices to exchange information. While FIG. 11 shows electronic device 1000 having various means, it is to be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.

特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码，从而实现如上所述的在头戴式显示设备中获取注视目标的方法。在这样的实施例中，该计算机程序可以通过通信装置1009从网络上被下载和安装，或者从存储装置1008被安装，或者从ROM 1002 被安装。在该计算机程序被处理装置1001执行时，执行本公开实施例的方法中限定的上述功能。In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, and the computer program includes program code for executing the method shown in the flow chart, thereby realizing the above The method for obtaining a gaze target in a head-mounted display device. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 1009, or from storage means 1008, or from ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的***、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行***、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的信息信号，其中承载了计算机可读的程序代码。这种传播的信息信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行***、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(射频)等等，或者上述的任意合适的组合。It should be noted that the above-mentioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include an information signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated information signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

在一些实施方式中，客户端、服务器可以利用诸如HTTP(HyperText Transfer Protocol，超文本传输协议)之类的任何已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字信息通信(例如，通信网络)互连。通信网络的示例包括局域网(“LAN”)，广域网(“WAN”)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何已知或未来研发的网络。In some embodiments, the client and the server can communicate using any known or future-developed network protocols such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol), and can communicate with digital information in any form or medium (eg, communication network) interconnections. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any known or future developed network.

上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device:

可选的，当上述一个或者多个程序被该电子设备执行时，该电子设备还可以执行上述实施例所述的其他步骤。Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).

附图中的流程图和框图，图示了按照本公开各种实施例的***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的***来实现，或者可以用专用硬件与计算机指令的组合来实现。The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元的名称在某种情况下并不构成对该单元本身的限定。The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of a unit does not constitute a limitation of the unit itself under certain circumstances.

本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、片上***(SOC)、复杂可编程逻辑设备(CPLD)等等。The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chips (SOCs), Complex Programmable Logical device (CPLD) and so on.

在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行***、装置或设备使用或与指令执行***、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体***、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

需要说明的是，在本文中，诸如“第一”和“第二”等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。It should be noted that in this article, relative terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these No such actual relationship or order exists between entities or operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

以上所述仅是本公开的具体实施方式，使本领域技术人员能够理解或实现本公开。对这些实施例的多种修改对本领域的技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本公开的精神或范围的情况下，在其它实施例中实现。因此，本公开将不会被限制于本文所述的这些实施例，而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。The above descriptions are only specific implementation manners of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

一种在头戴式显示设备中获取注视目标的方法，其特征在于，包括：A method for obtaining a gaze target in a head-mounted display device, characterized in that it includes:

确定用户视场内至少两个目标物分别到所述用户的距离信息；Determining distance information from at least two targets in the user's field of view to the user;

确定所述用户视线注视点到所述用户的距离信息；Determine the distance information from the user's gaze point to the user;

基于所述至少两个目标物分别到所述用户的距离信息以及所述用户视线注视点到所述用户的距离信息，在所述至少两个目标物中确定注视目标。Based on the distance information from the at least two targets to the user and the distance information from the gaze point of the user's line of sight to the user, determine a gaze target among the at least two targets.
根据权利要求1所述的方法，其特征在于，确定用户视场内至少两个目标物分别到所述用户的距离信息之前，还包括：The method according to claim 1, wherein before determining the distance information of at least two targets within the user's field of view to the user, further comprising:

头戴式显示设备获取用户视场图像信息，并识别处于用户视场内的目标物。The head-mounted display device acquires image information of the user's field of view, and identifies a target within the user's field of view.
根据权利要求2所述的方法，其特征在于，若用户视场内目标物的个数大于或等于2，执行确定用户视场内至少两个目标物分别到所述用户的距离信息的步骤；The method according to claim 2, characterized in that, if the number of targets in the user's field of view is greater than or equal to 2, the step of determining the distance information of at least two targets in the user's field of view to the user is performed;

基于所述至少两个目标物分别到所述用户的距离信息以及所述用户视线注视点到所述用户的距离信息，在所述至少两个目标物中确定注视目标之后，还包括：Based on the distance information from the at least two targets to the user and the distance information from the user's gaze point to the user, after determining the gaze target among the at least two targets, the method further includes:

控制头戴式显示设备通过虚拟图像仅显示与所述注视目标关联的信息。The head-mounted display device is controlled to display only the information associated with the gaze target through the virtual image.
根据权利要求3所述的方法，其特征在于，所述控制头戴式显示设备通过虚拟图像仅显示与所述注视目标关联的信息之前，还包括：The method according to claim 3, wherein before controlling the head-mounted display device to display only the information associated with the gaze target through a virtual image, further comprising:

头戴式显示设备将所述注视目标的图像与数据库中的图像进行匹配，得到与所述注视目标关联的信息。The head-mounted display device matches the image of the gaze target with images in the database to obtain information associated with the gaze target.
根据权利要求1所述的方法，其特征在于，所述确定用户视场内至少两个目标物分别到所述用户的距离信息，包括：The method according to claim 1, wherein the determining the distance information of at least two targets in the user's field of view to the user respectively includes:

头戴式显示设备利用双目测距方法确定用户视场内至少两个目标物分别到用户的距离信息。The head-mounted display device uses a binocular ranging method to determine distance information from at least two targets within the user's field of view to the user.
根据权利要求5所述的方法，其特征在于，头戴式显示设备利用双目测距方法确定用户视场内至少两个目标物分别到用户的距离信息，包括：The method according to claim 5, wherein the head-mounted display device uses a binocular ranging method to determine distance information from at least two targets in the user's field of view to the user, including:

头戴式显示设备利用双鱼眼摄像头获取至少两个目标物的双目图像信息；所述双鱼眼摄像头的视场大于用户视场；The head-mounted display device uses a dual fisheye camera to obtain binocular image information of at least two targets; the field of view of the dual fisheye camera is larger than the user's field of view;

头戴式显示设备基于至少两个目标物的双目图像信息，利用双目测距方法确定用户视场内至少两个目标物分别到用户的距离信息。Based on the binocular image information of at least two targets, the head-mounted display device determines distance information from at least two targets within the user's field of view to the user by using a binocular ranging method.
根据权利要求1所述的方法，其特征在于，所述确定所述用户视线注视点到所述用户的距离信息，包括：The method according to claim 1, wherein the determining the distance information from the point of gaze of the user to the user comprises:

头戴式显示设备确定用户左眼的视线方向信息和右眼的视线方向信息；The head-mounted display device determines the gaze direction information of the user's left eye and the gaze direction information of the right eye;

头戴式显示设备基于用户左眼的视线方向信息和右眼的视线方向信息，确定用户视线注视点位置信息；The head-mounted display device determines the position information of the gaze point of the user's gaze based on the gaze direction information of the user's left eye and the gaze direction information of the right eye;

头戴式显示设备基于用户视线注视点位置信息，确定所述用户视线注视点到所述用户的距离信息。The head-mounted display device determines distance information from the user's gaze point to the user based on the location information of the user's gaze point.
根据权利要求7所述的方法，其特征在于，所述头戴式显示设备确定用户左眼的视线方向信息和右眼的视线方向信息，包括：The method according to claim 7, wherein the head-mounted display device determines the line-of-sight direction information of the user's left eye and the line-of-sight direction information of the right eye, comprising:

头戴式显示设备利用瞳孔角膜反射法确定用户左眼的视线方向信息和右眼的视线方向信息。The head-mounted display device determines the gaze direction information of the user's left eye and the gaze direction information of the right eye by using the pupil cornea reflection method.
根据权利要求8所述的方法，其特征在于，所述头戴式显示设备利用瞳孔角膜反射法确定用户左眼的视线方向信息和右眼的视线方向信息，包括：The method according to claim 8, wherein the head-mounted display device uses a pupil corneal reflection method to determine the sight direction information of the user's left eye and the sight direction information of the right eye, including:

头戴式显示设备控制左右光源分别照射用户左眼和右眼；The head-mounted display device controls the left and right light sources to illuminate the user's left and right eyes respectively;

头戴式显示设备获取图像采集设备所采集的用户左眼的图像信息和右眼的图像信息；The head-mounted display device acquires the image information of the user's left eye and the image information of the right eye collected by the image acquisition device;

头戴式显示设备基于所述左眼的图像信息，确定用户左眼角膜曲率中心位置信息以及左眼瞳孔中心位置信息；基于所述右眼的图像信息，确定用户右眼角膜曲率中心位置信息以及右眼瞳孔中心位置信息；Based on the image information of the left eye, the head-mounted display device determines the position information of the center of curvature of the user's left eye cornea and the position information of the center of the pupil of the left eye; based on the image information of the right eye, determines the position information of the center of curvature of the user's right eye cornea and The position information of the pupil center of the right eye;

头戴式显示设备基于所述用户左眼角膜曲率中心位置信息以及所述左眼瞳孔中心位置信息，确定用户左眼的视线方向信息；The head-mounted display device determines the line-of-sight direction information of the user's left eye based on the position information of the center of curvature of the cornea of the user's left eye and the position information of the center of the pupil of the left eye;

头戴式显示设备基于所述用户右眼角膜曲率中心位置信息以及所述右眼瞳孔中心位置信息，确定用户右眼的视线方向信息。The head-mounted display device determines the line-of-sight direction information of the user's right eye based on the position information of the center of curvature of the cornea of the user's right eye and the position information of the center of the pupil of the right eye.
根据权利要求9所述的方法，其特征在于，还包括：The method according to claim 9, further comprising:

头戴式显示设备获取左眼误差补偿角信息和右眼误差补偿角信息；The head-mounted display device obtains the error compensation angle information of the left eye and the error compensation angle information of the right eye;

头戴式显示设备基于所述左眼误差补偿角信息，对用户左眼的视线方向信息进行修正；The head-mounted display device corrects the line-of-sight direction information of the user's left eye based on the left-eye error compensation angle information;

头戴式显示设备基于所述右眼误差补偿角信息，对用户右眼的视线方向信息进行修正；The head-mounted display device corrects the line-of-sight direction information of the user's right eye based on the right-eye error compensation angle information;

所述头戴式显示设备基于用户左眼的视线方向信息和右眼的视线方向信息，确定用户视线注视点位置信息包括：The head-mounted display device determines the position information of the gaze point of the user's gaze based on the gaze direction information of the user's left eye and the gaze direction information of the right eye, including:

头戴式显示设备基于修正后的左眼的视线方向信息和修正后的右眼的视线方向信息，确定用户视线注视点位置信息。Based on the corrected gaze direction information of the left eye and the corrected gaze direction information of the right eye, the head-mounted display device determines position information of the gaze point of the user.
根据权利要求10所述的方法，其特征在于，所述头戴式显示设备获取左眼误差补偿角信息和右眼误差补偿角信息，包括：The method according to claim 10, wherein the head-mounted display device acquires the error compensation angle information of the left eye and the error compensation angle information of the right eye, comprising:

头戴式显示设备输出标定指令，以使用户双眼注视位置信息确定的标定物；The head-mounted display device outputs a calibration command, so that the user's eyes are fixed on the calibration object determined by the position information;

头戴式显示设备控制左右光源分别照射用户左眼和右眼；The head-mounted display device controls the left and right light sources to illuminate the user's left and right eyes respectively;

在用户双眼注视所述标定物的情况下，头戴式显示设备获取所述图像采集设备所采集的用户左眼的图像信息和右眼的图像信息；When the user's eyes are fixed on the calibration object, the head-mounted display device acquires the image information of the user's left eye and the image information of the right eye collected by the image acquisition device;

头戴式显示设备基于所述左眼的图像信息，分别确定用户左眼角膜曲率中心位置信息和左眼瞳孔中心位置信息；基于所述右眼的图像信息，分别确定用户右眼角膜曲率中心位置信息和右眼瞳孔中心位置信息；Based on the image information of the left eye, the head-mounted display device respectively determines the position information of the center of curvature of the user's left eye cornea and the position information of the center of the pupil of the left eye; based on the image information of the right eye, respectively determines the position of the center of curvature of the user's right eye cornea information and the position information of the pupil center of the right eye;

头戴式显示设备基于左眼角膜曲率中心位置信息和左眼瞳孔中心位置信息，确定与所述标定物对应的用户左眼的光轴方向信息；基于右眼角膜曲率中心位置信息和右眼瞳孔中心位置信息，确定与所述标定物对应的用户右眼的光轴方向信息；The head-mounted display device determines the optical axis direction information of the user's left eye corresponding to the calibration object based on the position information of the center of curvature of the left eye cornea and the position information of the center of the pupil of the left eye; Center position information, determining the optical axis direction information of the user's right eye corresponding to the calibration object;

头戴式显示设备基于所述标定物的位置信息，确定与所述标定物对应的用户左眼的视轴方向信息和右眼的视轴方向信息；The head-mounted display device determines the visual axis direction information of the user's left eye and the visual axis direction information of the right eye corresponding to the calibration object based on the position information of the calibration object;

头戴式显示设备基于与所述标定物对应的用户左眼的光轴方向信息和左眼视轴方向信息，确定左眼误差补偿角信息；基于与所述标定物对应的用户右眼的光轴方向信息和右眼视轴方向信息，确定右眼误差补偿角信息。The head-mounted display device determines the error compensation angle information of the left eye based on the optical axis direction information of the user's left eye corresponding to the calibration object and the visual axis direction information of the left eye; based on the optical axis direction information of the user's right eye corresponding to the calibration object Axis direction information and visual axis direction information of the right eye are used to determine error compensation angle information of the right eye.
一种在头戴式显示设备中获取注视目标的装置，其特征在于，包括：A device for obtaining a gaze target in a head-mounted display device, characterized in that it includes:

第一距离确定模块，用于确定用户视场内至少两个目标物分别到所述用户的距离信息；A first distance determination module, configured to determine distance information from at least two targets within the user's field of view to the user;

第二距离确定模块，用于确定所述用户视线注视点到所述用户的距离信息；A second distance determining module, configured to determine distance information from the user's gaze point to the user;

识别模块，用于基于所述至少两个目标物分别到所述用户的距离信息以及所述用户视线注视点到所述用户的距离信息，在所述至少两个目标物中确定注视目标。The identification module is configured to determine the gaze target among the at least two targets based on the distance information from the at least two targets to the user and the distance information from the gaze point of the user to the user.
一种电子设备，其特征在于，所述电子设备包括：An electronic device, characterized in that the electronic device comprises:

一个或多个处理器；one or more processors;

存储装置，用于存储一个或多个程序；storage means for storing one or more programs;

当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-11中任一项所述的方法。When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the method according to any one of claims 1-11.
根据权利要求13所述的电子设备，其特征在于，所述电子设备为AR头戴式显示设备。The electronic device according to claim 13, wherein the electronic device is an AR head-mounted display device.
一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该程序被处理器执行时实现如权利要求1-11中任一项所述的方法。A computer-readable storage medium, on which a computer program is stored, wherein, when the program is executed by a processor, the method according to any one of claims 1-11 is realized.