CN109542231B - Method and device for feeding back information, electronic equipment and storage medium - Google Patents
Method and device for feeding back information, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN109542231B CN109542231B CN201811442589.5A CN201811442589A CN109542231B CN 109542231 B CN109542231 B CN 109542231B CN 201811442589 A CN201811442589 A CN 201811442589A CN 109542231 B CN109542231 B CN 109542231B
- Authority
- CN
- China
- Prior art keywords
- information
- user
- determining
- image
- feeding back
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The embodiment of the invention relates to the field of computer vision, and discloses a method and a device for feeding back information, electronic equipment and a storage medium. In some embodiments of the present application, a method for feeding back information includes: acquiring operation information of a user; determining first indication information according to the operation information; wherein the first indication information indicates a region of interest of the user in the image of the current environment; determining a first object according to the boundary information and the first indication information of the object in the image of the current environment; the first object is an object in the region of interest of the user; and feeding back the characteristic information of the first object and/or the distance information of the first object.
Description
Technical Field
The embodiment of the invention relates to the field of computer vision, in particular to a method and a device for feeding back information, electronic equipment and a storage medium.
Background
Environmental awareness is crucial to people's daily life because it provides a reference for further decisions. Many systems and methods have been provided to assist visually impaired people in performing various tasks such as navigation, reading, etc.
However, the inventors have found in the course of studying the prior art that there is currently no system or method that can help the blind to understand the surroundings and the things they are interested in, which hinders them from knowing the world.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
An object of embodiments of the present invention is to provide a method, an apparatus, an electronic device, and a storage medium for feeding back information, so that information of an object in which a user is interested can be fed back to the user according to a selection of the user, thereby better helping the user to know a surrounding environment and the object in which the user is interested and helping the user to know the world.
In order to solve the above technical problem, an embodiment of the present invention provides a method for feeding back information, including the following steps: acquiring operation information of a user; determining first indication information according to the operation information; wherein the first indication information indicates a region of interest of the user in the image of the current environment; determining a first object according to the boundary information and the first indication information of the object in the image of the current environment; the first object is an object in the region of interest of the user; and feeding back the characteristic information and/or the distance information of the first object.
The embodiment of the invention also provides a device for feeding back information, which comprises: the device comprises a first acquisition module, a first determination module, a second determination module and a feedback module; the first acquisition module is used for acquiring operation information of a user; the first determining module is used for determining first indicating information according to the operation information; wherein the first indication information indicates a region of interest of the user in the image of the current environment; the second determining module is used for determining the first object according to the boundary information and the first indicating information of the object in the image of the current environment; the first object is an object in the region of interest of the user; the feedback module is used for feeding back the characteristic information of the first object and/or the distance information of the first object.
An embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; and wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of feeding back information as set forth in the above embodiments.
Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which when executed by a processor implements the method for feeding back information mentioned in the above embodiments.
Compared with the prior art, the electronic equipment determines the object interested by the user according to the operation of the user and the boundary information of the object in the image of the current environment, and feeds back the characteristic information and/or the distance information of the object interested by the user to the user, so that the user can know the information of the object in the surrounding environment according to the fed-back information, and further can better sense the surrounding environment.
In addition, the feature information of the object includes information of N layers of the object, where N is a positive integer greater than 1. In this implementation, the feature information of the object is divided into a plurality of layers, which contributes to management of the feature information of the object.
In addition, N is equal to 3, the first layer information of the object is rough classification information of the object, the second layer information of the object is fine classification information of the object, and the third layer information of the object is attribute information of the object.
In addition, feeding back the characteristic information of the first object and the distance information of the first object specifically includes: determining second indication information according to the operation information; the second indication information is used for indicating the Tth layer information of the feedback object, and T is a positive integer not greater than N; and feeding back the T-th layer information of the first object and the distance information of the first object. In the implementation, the information of a certain level of the feedback object is determined according to the operation of the user, so that the user can know the information of the interested object more hierarchically and comprehensively, and can filter the information of the uninterested object more quickly.
In addition, feeding back the characteristic information of the first object and the distance information of the first object specifically includes: determining voice volume according to the distance information of the first object; and playing the characteristic information of the first object according to the determined voice volume in a voice playing mode. In the realization, the distance information of the object is fed back through the voice volume, so that a user can sense the distance between the user and the object more intuitively.
In addition, before obtaining the operation information of the user, the method for feeding back the information further comprises: acquiring an image of a current environment; determining information of the object in the image of the current environment, wherein the information of the object in the image of the current environment comprises boundary information of the object, and characteristic information of the object and/or distance information of the object.
In addition, the determining information of the object in the image specifically includes: transmitting the image of the current environment to a cloud server; the cloud server receives the image of the current environment, and determines and returns the information of the object in the image according to the image of the current environment; and receiving the information of the object in the image returned by the cloud server. In the implementation, the cloud server has stronger data processing capacity, so that the cloud server analyzes the image of the current environment, the processing speed can be increased, the speed of feedback information of the electronic equipment is increased, and the waiting time of a user is reduced.
In addition, acquiring the image of the current environment specifically includes: and receiving the image of the current environment transmitted by the acquisition device.
In addition, the operation information is touch information of the touch screen, and the touch information comprises touch position information; according to the operation information, determining first indication information, which comprises the following steps: taking the touch position information as first indication information; determining a first object according to the boundary information and the first indication information of the object, specifically comprising: determining pixels corresponding to the touch position information in the image; and determining the first object according to the pixel corresponding to the touch position information and the boundary information of the object in the image.
In addition, the operation information is touch information of the touch screen, and the touch information comprises touch position information; determining second indication information according to the operation information, which specifically comprises: judging whether the distance between the touch position indicated by the touch position information of this time and the touch position indicated by the touch position information of the last time is larger than a first threshold value or not; if yes, determining that the second indication information in the current feedback process indicates that the first layer information of the object is fed back; if not, determining the second indication information in the last feedback process, and determining the second indication information in the current feedback process according to the second indication information in the last feedback process.
In addition, the operation information includes posture information of the head of the user; according to the operation information, determining first indication information, which comprises the following steps: determining sight line information of the user according to the posture information of the head of the user; using the sight line information of the user as first indication information; determining a first object according to the boundary information and the first indication information of the object, specifically comprising: determining pixels corresponding to the sight line information of the user in the image; and determining the first object according to the pixels corresponding to the sight line information of the user and the boundary information of the object in the image.
In addition, the operation information includes posture information of the head of the user; determining second indication information according to the operation information, which specifically comprises: determining the posture of the head of the user in the feedback process according to the posture information of the head of the user; judging whether the posture of the head of the user in the feedback process changes relative to the posture of the head of the user in the feedback process of the last time; if yes, determining that the second indication information in the current feedback process indicates that the first layer information of the object is fed back; otherwise, determining the second indication information in the last feedback process, and determining the second indication information in the current feedback process according to the second indication information in the last feedback process.
Drawings
One or more embodiments are illustrated by way of example in the accompanying drawings, which correspond to the figures in which like reference numerals refer to similar elements and which are not to scale unless otherwise specified.
Fig. 1 is a flowchart of a method of feeding back information according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a method of feeding back information according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of an apparatus for feeding back information according to a third embodiment of the present invention;
fig. 4 is a schematic configuration diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments.
It should be noted that the method for providing feedback information according to the embodiment of the present invention is mainly applied to providing information of a surrounding environment for a visually impaired person to help the visually impaired person to perceive the surrounding environment, but those skilled in the art can understand that in practical application, the embodiment of the present invention may also be applied to other scenarios, and the embodiment of the present invention is not limited to the application scenario.
The first embodiment of the present invention relates to a method for feeding back information, which is applied to an electronic device, for example, a device with a processing function, such as a touch terminal (e.g., a mobile phone, a tablet, etc.). As shown in fig. 1, the method for feeding back information includes:
step 101: and acquiring the operation information of the user.
The following exemplifies the types and the obtaining manners of the operation information of the user.
In the first case, the operation information is touch information of the touch screen, and the touch information includes touch position information. The electronic equipment can be a smart phone, the touch screen is the touch screen of the smart phone, and the smart phone determines the position touched by a user on the touch screen and generates touch information of the touch screen.
In the second case, the operation information is posture information of the head of the user. The electronic device determines posture information of the head of the user according to information provided by an Inertial Measurement Unit (IMU) by receiving the information provided by the IMU.
It should be noted that, as can be understood by those skilled in the art, in practical applications, the operation information of the user may be other types of information, and the electronic device may determine the operation information of the user in other manners, which are not listed here, and the embodiment does not limit the type and the obtaining manner of the operation information of the user.
Step 102: and determining first indication information according to the operation information.
Specifically, the first indication information indicates a region of interest to the user in the image of the current environment.
In specific implementation, when a user starts an environment information feedback function of the electronic device, the electronic device acquires an image of a current environment and determines information of an object in the image, wherein the information of the object in the image of the current environment includes boundary information of the object, and feature information of the object and/or distance information of the object. The feature information of the object may be any one or more of rough classification information of the object, fine classification information of the object, attribute information of the object, and other information. The distance information of the object is used for indicating the distance between the object and the user, and the electronic device can determine the distance between the object and the electronic device according to the depth information of the object in the image of the current environment, and determine the distance between the object and the user by combining a preset distance between the electronic device and the user, or by sensing the distance between the electronic device and the user in real time. Before the user turns off the environmental information feedback function of the electronic device or resends the shooting instruction, the electronic device executes steps 101 to 104 based on the image and the information of the object in the image.
In a specific implementation, the feature information of the object includes rough classification information of the object, fine classification information of the object, and attribute information of the object. The rough classification information of the object may be the category information to which the object belongs, and the fine classification information of the object is the name information of the object. For example, the object is a table, the rough classification information of the table is the category information of the table, i.e., furniture, the fine classification information of the table is the name of the table, i.e., the table, and the feature information of the table is the attribute information of the object, i.e., brown.
It should be noted that, as can be understood by those skilled in the art, in practical application, the feature information of the object may be divided into more layers or fewer layers according to needs, the coarse classification information of the object and the fine classification information of the object may be set as information of other layers of the object according to needs, and the embodiment does not limit the layering manner of the information of the object.
The electronic equipment can acquire the image of the current environment through a camera of the electronic equipment, and can also receive the image of the current environment transmitted by the acquisition device through establishing communication connection with the acquisition device.
In specific implementation, the acquisition device shoots an image of the current environment after acquiring the shooting instruction. The shooting instruction can be sent by the electronic device, or can be a shooting instruction generated after the acquisition device determines that the user presses the shooting button.
In specific implementation, the device with the image acquisition function and the wearable device of the user can be combined to form the acquisition device. For example, a Red Green Blue-Depth (RGB-D) sensor may be affixed to a sunglasses frame worn by the user, the RGB-D sensor transmitting the captured RGB image and Depth map to the electronic device.
In specific implementation, the electronic device can transmit the image of the current environment to the cloud server, and the cloud server receives the image of the current environment, and determines and returns the information of the object in the image according to the image of the current environment. The electronic equipment receives the information of the object in the image returned by the cloud server.
In specific implementation, after the electronic device completes the confirmation of the information of the object in the image of the current environment, the electronic device may prompt the user in a vibration mode or the like to inform the user that the confirmation is completed.
It is worth mentioning that, because the cloud server has stronger data processing capability, the cloud server analyzes the image of the current environment, so that the processing speed can be increased, the speed of the feedback information of the electronic device is increased, and the waiting time of the user is reduced.
It should be noted that, as can be understood by those skilled in the art, the electronic device may transmit the image of the current environment to the cloud server, and the cloud server analyzes the image of the current environment to obtain information of the object in the image, or may analyze the image of the current environment by the electronic device itself to obtain information of the object in the image.
In a specific implementation, the cloud server or the electronic device may determine information of an object in an image of a current environment through a vision-based scene understanding technology. Such as object detection techniques, image semantic segmentation techniques, and the like. For example, by a semantic segmentation technique based on a neural network, objects in an image of the current environment are identified and segmented, and boundary information of each object, feature information of each object, and distance information of each object are determined.
The following illustrates a method for determining the first indication information in conjunction with different operation information.
In specific implementation, the operation information is touch information of the touch screen, and the touch information includes touch position information. The electronic equipment takes the touch position information as first indication information. For example, when the user touches the upper right corner of the smartphone, the first indication information indicates that the region of interest of the user is the upper right corner region of the image.
In another specific implementation, the operation information is posture information of the head of the user. The electronic equipment determines sight line information of the user according to the posture information of the head of the user; and using the sight line information of the user as first indication information. For example, when a coordinate system is established with an intersection point of a central axis of an image and a lower frame of the image as an origin, the lower frame of the image as an abscissa, and the central axis of the image as an ordinate, the electronic device determines, according to the posture information of the head of the user, that the line-of-sight information of the user is an angle of 45 degrees between the line of sight of the user and the abscissa, and the first indication information indicates that the region of interest of the user is a region around a position where a ray obtained by rotating the abscissa by 45 degrees in the counterclockwise direction with the origin as a starting point.
It should be noted that, as will be understood by those skilled in the art, in practical applications, the area of interest to the user may also be determined in other manners, for example, a method based on voice interaction, and the present embodiment is not limited to the manner of determining the area of interest to the user.
Step 103: and determining the first object according to the boundary information and the first indication information of the object in the image of the current environment.
In particular, the first object is an object within an area of interest to the user.
In specific implementation, the operation information is touch information of the touch screen, and the touch information includes touch position information. The electronic equipment determines a pixel corresponding to the touch position information in the image; and determining the first object according to the pixel corresponding to the touch position information and the boundary information of the object in the image.
In another specific implementation, the operation information is posture information of the head of the user. The electronic equipment determines a pixel corresponding to the sight line information of the user in the image; and determining the first object according to the pixels corresponding to the sight line information of the user and the boundary information of the object in the image.
It should be noted that, as can be understood by those skilled in the art, the person skilled in the art can also determine the first object by other ways, and the present embodiment is not limited to the method for determining the first object.
It should be noted that, as will be understood by those skilled in the art, the objects in the present embodiment may include tables, computers, sky, clouds, roads, etc., and in practical applications, when objects are classified, the sky, clouds, roads, etc. may be classified as "background objects", and the present embodiment is not limited to the types of the objects.
Step 104: and feeding back the characteristic information of the first object and/or the distance information of the first object.
Specifically, the electronic device may feed back feature information of an object (i.e., the first object) of interest to the user and/or distance information of the first object based on the operation information of the user.
It should be noted that, in practical applications, the electronic device may feed back other information of the first object on the basis of feeding back the characteristic information of the first object and/or the distance information of the first object, so as to facilitate a user to better know about the first object.
It should be noted that, as can be understood by those skilled in the art, in practical applications, the electronic device may feed back the characteristic information of the first object, or feed back the distance information of the first object, or feed back the characteristic information of the first object and the distance information of the first object according to the setting of the user or the factory setting, and the like.
The following illustrates a method for feeding back the characteristic information of the first object and the distance information of the first object by the electronic device.
In specific implementation, the electronic equipment determines the voice volume according to the distance information of the first object; and playing the characteristic information of the first object according to the determined voice volume in a voice playing mode. For example, for an indoor scene, the distance interval is divided into three levels, i.e., 0m-2m, 2m-5m, and 5m or more. For outdoor scenes, the distance interval is also divided into three levels, i.e., 0m-5m, 5m-15m and more than 15m, different volumes are set for the price setting tables of different levels, e.g., high volume is set for 0m-2m, medium volume is set for 2m-5m, and low volume is set for more than 5m of the indoor scene, so that the user can know the relative distance of each object.
It should be noted that, as can be understood by those skilled in the art, in practical applications, when a functional relationship between the volume and the distance information of the object is established, the volume of the voice may be determined according to the functional relationship and the distance information of the object.
It is worth mentioning that the distance information of the object is fed back through the volume of the voice, so that the user can know the distance between the object and the user more visually, and the user can know the surrounding environment more conveniently.
In another specific implementation, the electronic device directly plays the distance information of the first object and the characteristic information of the first object, and feeds back the information of the first object.
It should be noted that, in practical applications, the information of the first object may also be fed back in other manners, for example, when the electronic device is a smartphone, the distance information of the first object may be fed back by a strong vibration, for example, the distance information of the first object indicates that the first object is close to the user due to the strong vibration.
It is worth mentioning that the electronic device feeds back information according to the operation information of the user and the boundary information of the object, so that the user can know information such as the size of the object in the image, the distance between the objects, the relative position relationship between the objects and the like according to the information fed back by the electronic device and by combining the ratio of the image to the reality.
For example, when the operation information is touch information, when the user touches the position a with a finger, the information fed back by the electronic device is information of the object a, when the user moves directly below the finger, and when the moving distance is about 1cm, the information fed back by the electronic device is information of the object B, the object B is an environment, the user continues to move the finger directly below the finger, and when the user moves 2cm, the information fed back by the electronic device is information of the object C, and the user determines that the distance between the object a and the object C is 20cm based on the information fed back by the electronic device, the moving distance of the finger, and the ratio of the image to the reality (here, 1: 10 is taken as an example).
The above description is only for illustrative purposes and does not limit the technical aspects of the present invention.
Compared with the prior art, in the method for feeding back information provided in this embodiment, the electronic device determines the object interested by the user according to the operation of the user and the boundary information of the object in the image of the current environment, and feeds back the feature information and/or the distance information of the object interested by the user to the user, so that the user can know the information of the object in the surrounding environment according to the fed-back information, and further can better perceive the surrounding environment. In addition, when the user continuously operates, the user can know the relative position relation between objects in the surrounding environment and the like by combining the information fed back by the electronic equipment, so that the user is more comprehensively helped to perceive the world.
The second embodiment of the present invention relates to a method for feeding back information, and the second embodiment is a further improvement of the first embodiment, and the specific improvements are as follows: in the present embodiment, the feature information of the object is divided into a plurality of layers, that is, the feature information of the object includes information of N layers of the object, where N is a positive integer greater than 1.
Specifically, as shown in fig. 2, the present embodiment includes steps 201 to 205, wherein steps 201 to 203 are substantially the same as steps 101 to 103 in the first embodiment, and are not repeated herein. The following mainly introduces the differences:
step 201 to step 203 are executed.
Step 204: and determining second indication information according to the operation information.
Specifically, the second indication information is used to indicate the T-th layer information of the feedback object, T being a positive integer not greater than N.
A method of determining the second indication information is exemplified below.
In specific implementation, the operation information is touch information of the touch screen, and the touch information includes touch position information. The electronic equipment judges whether the distance between the touch position indicated by the touch position information of this time and the touch position indicated by the touch position information of the last time is larger than a first threshold value or not; if yes, determining that the second indication information in the current feedback process indicates that the first layer information of the object is fed back; if not, determining the second indication information in the last feedback process, and determining the second indication information in the current feedback process according to the second indication information in the last feedback process.
In another particular implementation, the operational information includes pose information of a head of the user. The electronic equipment determines the posture of the head of the user in the feedback process according to the posture information of the head of the user; judging whether the posture of the head of the user in the feedback process changes relative to the posture of the head of the user in the feedback process of the last time; if yes, determining that the second indication information in the current feedback process indicates that the first layer information of the object is fed back; otherwise, determining the second indication information in the last feedback process, and determining the second indication information in the current feedback process according to the second indication information in the last feedback process.
In a specific implementation, the method for the electronic device to determine whether the posture of the head of the user in the current feedback process changes relative to the posture of the head of the user in the last feedback process may be: the electronic equipment determines the sight line information of the user according to the posture of the head of the user in the feedback process; acquiring the sight line information of the user in the last feedback process, and judging whether the sight line deviation of the user is greater than a second threshold value or not according to the sight line information of the user at this time and the sight line information of the user in the last feedback process; if so, determining that the posture of the head of the user in the feedback process is changed relative to the posture of the head of the user in the feedback process of the last time, otherwise, determining that the posture of the head of the user in the feedback process of the current time is not changed relative to the posture of the head of the user in the feedback process of the last time.
The process of the electronic device determining whether the posture of the head of the user changes is described below in conjunction with an actual scene. Assuming that the second threshold is 10 degrees, the electronic device determines that the included angle between the sight line of the user and the abscissa is 45 degrees according to the posture information of the head of the user in the last feedback process, and determines that the included angle between the sight line of the user and the abscissa is 90 degrees according to the posture information of the head of the user in the current feedback process. The electronic equipment compares the sight line information of the user in the last feedback process with the sight line information of the user in the current feedback process, and determines that the sight line deviation of the user is 45 degrees and is larger than a second threshold value, so that the electronic equipment determines that the posture of the head of the user changes.
The following describes that the electronic device determines the second indication information in the current feedback process according to the second indication information in the last feedback process. And if the second indication information in the last feedback process of the electronic equipment indicates the first layer information of the feedback object, determining that the second indication information in the current feedback process indicates the second layer information of the feedback object. If the second indication information in the last feedback process of the electronic device indicates the second layer information of the feedback object, it is determined that the second indication information in the current feedback process indicates the third layer information … … of the feedback object, and so on, until the second indication information in the last feedback process indicates the nth layer information of the feedback object, the second indication information in the current feedback process indicates the first layer information of the feedback object, or the electronic device prompts that the user does not have more detailed information.
It should be noted that, as can be understood by those skilled in the art, in practical applications, the second indication information may be determined in other manners, and the method of determining the second indication information is not limited in the present embodiment.
Step 205: and feeding back the T-th layer information of the first object and the distance information of the first object.
Specifically, the electronic device may determine a voice volume according to the distance information of the first object; and playing the T-th layer information of the first object according to the determined voice volume in a voice playing mode. T is a positive integer not greater than N.
It should be noted that, as will be understood by those skilled in the art, the information of each layer of the object may include one or more pieces of detailed information of the object. For example, the third layer information of the object is attribute information of the object, and the attribute information of the object may include any one or any combination of color, shape, material and other information of the object.
The method for feeding back information according to the present embodiment will be described below by taking N equal to 3 and an electronic device as an example.
It is assumed that the first layer information of the object is rough classification information of the object, the second layer information of the object is fine classification information of the object, the third layer information of the object is attribute information of the object, the rough classification information of the object is information of a class to which the object belongs, and the fine classification information of the object is name information of the object. The user obtains the image of the current environment through the mobile phone, the image of the current environment is transmitted to the cloud server, the cloud server receives the image of the current environment, and the information of the object in the image is determined and returned according to the image of the current environment. And the mobile phone receives the information of the object in the image returned by the cloud server and displays the image of the current environment. At this time, the user touches the touch screen of the mobile phone with a finger, the mobile phone determines the touch position information of the user, further determines the first indication information, namely the region of interest of the user, and determines the objects contained in the region of interest according to the boundary information of all the objects in the image. Assuming that the object in the region of interest is a table, the mobile phone acquires the distance information of the table and the characteristic information of the table, and determines the first voice volume according to the distance information of the table. Since the user touches the touch screen of the mobile phone for the first time, the first layer of information of the table, namely the furniture, which is the information of the category to which the table belongs is played at the first volume. If the user wants to know not the furniture at the moment but other kinds of objects, such as appliances, the finger can be moved to other positions. If the object that the user wants to know belongs to furniture, the user can touch the same position or an area around the same position (the distance between any position in the area and the position touched last time is not larger than a first threshold value). At this time, the mobile phone plays the second layer of information of the table, namely the name information of the table, namely the table. If the furniture the user wants to know at the moment is not a table, the fingers can be moved to other positions. If the user really knows that the object is a table, but the table at the position is not determined to be the table which the user wants to find, the user can touch the position again, and at the moment, the mobile phone plays the third layer of information of the table, namely the attribute, color, material and the like of the table.
It is worth mentioning that dividing the feature information of the object into a plurality of layers is helpful for managing the feature information of the object.
It is worth mentioning that the information of a certain level of the feedback object is determined according to the operation of the user, so that the user can know the information of the object interested by the user more hierarchically and comprehensively, and can filter the information of the object not interested by the user more quickly.
In this embodiment, for clarity of description, step 204 is set as a step subsequent to step 203. However, those skilled in the art can understand that in practical applications, step 204 only needs to be set before step 205 and after step 201, and step 204 can be set as any step before step 205 and after step 201 as needed, which is not limited in this embodiment.
The above description is only for illustrative purposes and does not limit the technical aspects of the present invention.
Compared with the prior art, in the method for feeding back information provided in this embodiment, the electronic device determines the object interested by the user according to the operation of the user and the boundary information of the object in the image of the current environment, and feeds back the feature information and/or the distance information of the object interested by the user to the user, so that the user can know the information of the object in the surrounding environment according to the fed-back information, and further can better perceive the surrounding environment. In addition, the characteristic information of the object is divided into a plurality of layers, which is helpful for managing the characteristic information of the object. The information of a certain level of the feedback object is determined according to the operation of the user, so that the user can know the information of the object interested by the user more hierarchically and comprehensively, and the information of the object not interested by the user can be filtered more quickly.
The steps of the above methods are divided for clarity, and the implementation may be combined into one step or split some steps, and the steps are divided into multiple steps, so long as the same logical relationship is included, which are all within the protection scope of the present patent; it is within the scope of the patent to add insignificant modifications to the algorithms or processes or to introduce insignificant design changes to the core design without changing the algorithms or processes.
A third embodiment of the present invention relates to an apparatus for feeding back information, as shown in fig. 3, including: a first obtaining module 301, a first determining module 302, a second determining module 303 and a feedback module 304; the first obtaining module 301 is configured to obtain operation information of a user; the first determining module 302 is configured to determine first indication information according to the operation information; wherein the first indication information indicates a region of interest of the user in the image of the current environment; the second determining module 303 is configured to determine the first object according to the boundary information of the object in the image of the current environment and the first indication information; the first object is an object in the region of interest of the user; the feedback module 304 is configured to feed back the characteristic information of the first object and/or the distance information of the first object.
It should be understood that this embodiment is an example of the apparatus corresponding to the first embodiment, and may be implemented in cooperation with the first embodiment. The related technical details mentioned in the first embodiment are still valid in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related-art details mentioned in the present embodiment can also be applied to the first embodiment.
It should be noted that each module referred to in this embodiment is a logical module, and in practical applications, one logical unit may be one physical unit, may be a part of one physical unit, and may be implemented by a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, elements that are not so closely related to solving the technical problems proposed by the present invention are not introduced in the present embodiment, but this does not indicate that other elements are not present in the present embodiment.
A fourth embodiment of the present invention relates to an electronic apparatus, as shown in fig. 4, including: at least one processor 401; and a memory 402 communicatively coupled to the at least one processor 401; and wherein the memory 402 stores instructions executable by the at least one processor 401, the instructions being executable by the at least one processor 401 to enable the at least one processor 401 to perform the method of feeding back information as mentioned in the above embodiments.
The electronic device includes: one or more processors 401 and a memory 402, one processor 401 being exemplified in fig. 4. The processor 401 and the memory 402 may be connected by a bus or other means, and fig. 4 illustrates the connection by a bus as an example. Memory 402, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The processor 401 executes various functional applications of the device and data processing, i.e. implements the above-mentioned method of feeding back information, by running non-volatile software programs, instructions and modules stored in the memory 402.
The memory 402 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store a list of options, etc. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 402 may optionally include memory located remotely from processor 401, which may be connected to an external device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
One or more modules are stored in the memory 402 and when executed by the one or more processors 401 perform the method of feeding back information in any of the method embodiments described above.
The product can execute the method provided by the embodiment of the application, has corresponding functional modules and beneficial effects of the execution method, and can refer to the method provided by the embodiment of the application without detailed technical details in the embodiment.
A fifth embodiment of the present invention relates to a computer-readable storage medium storing a computer program. The computer program realizes the above-described method embodiments when executed by a processor.
That is, as can be understood by those skilled in the art, all or part of the steps in the method for implementing the embodiments described above may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples for carrying out the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.
Claims (13)
1. A method for feeding back information, comprising:
acquiring operation information of a user;
determining first indication information according to the operation information; wherein the first indication information indicates a region of interest of a user in an image of a current environment;
determining a first object according to the boundary information of the object in the image of the current environment and the first indication information; wherein the first object is an object within the user's region of interest;
feeding back characteristic information of the first object and distance information of the first object;
the characteristic information of the first object comprises N levels of information of the first object, wherein N is a positive integer greater than 1;
feeding back the characteristic information of the first object and the distance information of the first object, specifically comprising:
determining second indication information according to the operation information; the second indication information is used for indicating feedback of Tth layer information of the object, and T is a positive integer not greater than N; and feeding back the T-th layer information of the first object and the distance information of the first object.
2. The method for feeding back information according to claim 1, wherein N is equal to 3, the first layer information of the object is coarse classification information of the object, the second layer information of the object is fine classification information of the object, and the third layer information of the object is attribute information of the object.
3. The method for feeding back information according to claim 1, wherein feeding back the feature information of the first object and the distance information of the first object specifically comprises:
determining voice volume according to the distance information of the first object;
and playing the characteristic information of the first object according to the determined voice volume in a voice playing mode.
4. The method for feeding back information according to any one of claims 1 to 3, wherein before the obtaining of the operation information of the user, the method for feeding back information further comprises:
acquiring an image of a current environment;
determining information of an object in the image of the current environment, wherein the information of the object in the image of the current environment includes boundary information of the object, and feature information of the object and/or distance information of the object.
5. The method for feeding back information according to claim 4, wherein the determining information of the object in the image specifically comprises:
transmitting the image of the current environment to a cloud server; the cloud server receives the image of the current environment, and determines and returns information of objects in the image according to the image of the current environment;
and receiving the information of the object in the image returned by the cloud server.
6. The method for feeding back information according to claim 5, wherein the acquiring the image of the current environment specifically includes:
and receiving the image of the current environment transmitted by the acquisition device.
7. The method for feeding back information according to any one of claims 1 to 3, wherein the operation information is touch information of a touch screen, and the touch information includes touch position information;
the determining first indication information according to the operation information includes:
taking the touch position information as the first indication information;
the determining the first object according to the boundary information of the object and the first indication information specifically includes:
determining a pixel corresponding to the touch position information in the image;
and determining the first object according to the pixels corresponding to the touch position information and the boundary information of the object in the image.
8. The method for feeding back information according to claim 1, wherein the operation information is touch information of a touch screen, and the touch information includes touch position information;
the determining second indication information according to the operation information specifically includes:
judging whether the distance between the touch position indicated by the touch position information of this time and the touch position indicated by the touch position information of the last time is larger than a first threshold value or not;
if yes, determining that the second indication information in the feedback process indicates that the first layer information of the object is fed back;
if not, determining the second indication information in the last feedback process, and determining the second indication information in the current feedback process according to the second indication information in the last feedback process.
9. A method of feeding back information according to any one of claims 1 to 3, wherein the operation information includes posture information of a head of the user;
the determining first indication information according to the operation information includes:
determining sight line information of the user according to the posture information of the head of the user;
using the sight line information of the user as the first indication information;
the determining the first object according to the boundary information of the object and the first indication information specifically includes:
determining pixels in the image corresponding to the user's gaze information;
and determining the first object according to the pixel corresponding to the sight line information of the user and the boundary information of the object in the image.
10. The method of feeding back information according to claim 1, wherein the operation information includes posture information of a head of the user;
the determining second indication information according to the operation information specifically includes:
determining the posture of the head of the user in the feedback process according to the posture information of the head of the user;
judging whether the posture of the head of the user in the feedback process changes relative to the posture of the head of the user in the feedback process of the last time;
if yes, determining that the second indication information in the feedback process indicates that the first layer information of the object is fed back;
otherwise, determining the second indication information in the last feedback process, and determining the second indication information in the current feedback process according to the second indication information in the last feedback process.
11. An apparatus for feeding back information, comprising: the device comprises a first acquisition module, a first determination module, a second determination module and a feedback module;
the first acquisition module is used for acquiring operation information of a user;
the first determining module is used for determining first indicating information according to the operation information; wherein the first indication information indicates a region of interest of a user in an image of a current environment;
the second determining module is used for determining a first object according to the boundary information of the object in the image of the current environment and the first indicating information; wherein the first object is an object within the user's region of interest;
the feedback module is used for feeding back the characteristic information of the first object and the distance information of the first object;
the characteristic information of the first object comprises N levels of information of the first object, wherein N is a positive integer greater than 1;
feeding back the characteristic information of the first object and the distance information of the first object, specifically comprising:
determining second indication information according to the operation information; the second indication information is used for indicating feedback of Tth layer information of the object, and T is a positive integer not greater than N; and feeding back the T-th layer information of the first object and the distance information of the first object.
12. An electronic device, comprising: at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; and wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of feedback information according to any one of claims 1 to 10.
13. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of feeding back information of any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811442589.5A CN109542231B (en) | 2018-11-29 | 2018-11-29 | Method and device for feeding back information, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811442589.5A CN109542231B (en) | 2018-11-29 | 2018-11-29 | Method and device for feeding back information, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109542231A CN109542231A (en) | 2019-03-29 |
| CN109542231B true CN109542231B (en) | 2022-04-08 |
Family
ID=65851241
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811442589.5A Active CN109542231B (en) | 2018-11-29 | 2018-11-29 | Method and device for feeding back information, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109542231B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN204744865U (en) * | 2015-06-08 | 2015-11-11 | 深圳市中科微光医疗器械技术有限公司 | Device for environmental information around reception and registration of visual disability personage based on sense of hearing |
| CN105892630A (en) * | 2015-11-02 | 2016-08-24 | 乐视致新电子科技（天津）有限公司 | List content display method and device |
| CN106250144A (en) * | 2016-07-28 | 2016-12-21 | 北京珠穆朗玛移动通信有限公司 | A kind of informing message display method and mobile terminal thereof |
| CN106249866A (en) * | 2016-05-12 | 2016-12-21 | 温州职业技术学院 | Vision auxiliary eyeglasses and image-recognizing method thereof |
-
2018
- 2018-11-29 CN CN201811442589.5A patent/CN109542231B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN204744865U (en) * | 2015-06-08 | 2015-11-11 | 深圳市中科微光医疗器械技术有限公司 | Device for environmental information around reception and registration of visual disability personage based on sense of hearing |
| CN105892630A (en) * | 2015-11-02 | 2016-08-24 | 乐视致新电子科技（天津）有限公司 | List content display method and device |
| CN106249866A (en) * | 2016-05-12 | 2016-12-21 | 温州职业技术学院 | Vision auxiliary eyeglasses and image-recognizing method thereof |
| CN106250144A (en) * | 2016-07-28 | 2016-12-21 | 北京珠穆朗玛移动通信有限公司 | A kind of informing message display method and mobile terminal thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109542231A (en) | 2019-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11189037B2 (en) | Repositioning method and apparatus in camera pose tracking process, device, and storage medium | |
| US9661214B2 (en) | Depth determination using camera focus | |
| CN113762252B (en) | Unmanned aerial vehicle intelligent following target determining method, unmanned aerial vehicle and remote controller | |
| US9865062B2 (en) | Systems and methods for determining a region in an image | |
| EP3742250B1 (en) | Method, apparatus and system for controlling unmanned aerial vehicle | |
| CN107613202B (en) | Shooting method and mobile terminal | |
| CN111935393A (en) | Shooting method, shooting device, electronic equipment and storage medium | |
| CN111722245B (en) | Positioning method, positioning device and electronic equipment | |
| CN112132113A (en) | Vehicle re-identification method and device, training method and electronic equipment | |
| CN104081317A (en) | Image processing device, and computer program product | |
| KR20160140700A (en) | Automated selective upload of images | |
| CN109413399B (en) | Apparatus for synthesizing object using depth map and method thereof | |
| US11509809B2 (en) | Following control method, control terminal, and unmanned aerial vehicle | |
| CN109495616B (en) | Photographing method and terminal equipment | |
| CN109992111B (en) | Augmented reality extension method and electronic device | |
| KR102337209B1 (en) | Method for notifying environmental context information, electronic apparatus and storage medium | |
| CN112487979A (en) | Target detection method, model training method, device, electronic device and medium | |
| EP2888716B1 (en) | Target object angle determination using multiple cameras | |
| CN108881846B (en) | Information fusion method and device and computer readable storage medium | |
| CN109218620B (en) | Photographing method and device based on ambient brightness, storage medium and mobile terminal | |
| CN109542231B (en) | Method and device for feeding back information, electronic equipment and storage medium | |
| US20230043122A1 (en) | Electronic device comprising multi-camera, and photographing method | |
| CN113378605A (en) | Multi-source information fusion method and device, electronic equipment and storage medium | |
| US11756302B1 (en) | Managing presentation of subject-based segmented video feed on a receiving device | |
| CN114659450B (en) | Robot following method, device, robot and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |