WO2023273056A1 - Robot navigation method, robot and computer-readable storage medium - Google Patents

Robot navigation method, robot and computer-readable storage medium Download PDF

Info

Publication number
WO2023273056A1
WO2023273056A1 PCT/CN2021/126711 CN2021126711W WO2023273056A1 WO 2023273056 A1 WO2023273056 A1 WO 2023273056A1 CN 2021126711 W CN2021126711 W CN 2021126711W WO 2023273056 A1 WO2023273056 A1 WO 2023273056A1
Authority
WO
WIPO (PCT)
Prior art keywords
key point
captured image
robot
image
target object
Prior art date
Application number
PCT/CN2021/126711
Other languages
French (fr)
Chinese (zh)
Inventor
郭渺辰
程骏
汤志超
胡淑萍
林灿然
张惊涛
蔡洁心
郭德骏
谭欢
Original Assignee
深圳市优必选科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202110736327.5A external-priority patent/CN113515143B/en
Application filed by 深圳市优必选科技股份有限公司 filed Critical 深圳市优必选科技股份有限公司
Publication of WO2023273056A1 publication Critical patent/WO2023273056A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/12—Target-seeking control
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Definitions

  • the present application belongs to the technical field of robots, and in particular relates to a robot navigation method, a robot and a computer-readable storage medium.
  • Distribution robot is a common service robot, which is mainly used for the distribution of goods.
  • food delivery robots are used to deliver meals to designated locations.
  • markers such as barcodes, logo images, etc.
  • the robot can achieve precise positioning by recognizing markers during the movement.
  • the distribution robot can only move along a fixed route, which has poor flexibility; in addition, markers need to be arranged in the application scene, which is costly.
  • Embodiments of the present application provide a robot navigation method, a robot, and a computer-readable storage medium, which can avoid dependence on markers during the movement of the robot, and effectively improve the flexibility of the movement of the robot.
  • the embodiment of the present application provides a robot navigation method, which is applied to a robot, and a shooting device is installed on the robot, and the method includes:
  • the detection result indicates that the target behavior exists in the captured image, detecting a target object corresponding to the target behavior in the captured image;
  • the target behavior in the captured image is determined by performing image detection on the captured image, and then the target object corresponding to the target behavior is determined, and then the depth information of the target object is obtained from the captured image; Taking depth images reduces the device cost of the robot; calculates the actual distance between the robot and the target object according to the depth information of the target object, and then plans the robot's movement route according to the actual distance.
  • the robot does not need to rely on markers during the movement process, and does not need to move along a fixed route, but can flexibly plan a movement route according to the identified target object, effectively improving the flexibility of the robot's movement.
  • the detecting the target object in the captured image includes:
  • the key point information includes respective position information of a plurality of key points in the captured image
  • the detecting the target behavior in the captured image according to the key point information to obtain the detection result includes:
  • the detection result indicates that the target behavior exists in the captured image.
  • detecting the target object corresponding to the target behavior in the captured image includes:
  • the detection result indicates that the target behavior exists in the captured image, obtaining a second key point associated with the first key point from the plurality of key points;
  • the second key point is determined as a target object in the captured image.
  • the target object includes a second key point
  • the acquiring the depth information of the target object from the captured image includes:
  • the converting the captured image into a grayscale image includes:
  • the depth estimation model includes an encoding layer and a decoding layer; the encoding layer is used to perform feature extraction on the captured image to obtain a feature map; the decoding layer is used to perform upsampling processing on the feature map to obtain The grayscale image.
  • the obtaining the gray value corresponding to the second key point in the gray image includes:
  • mapping the second key point to a preset coordinate system to obtain a mapping point
  • the calculating the depth information of the target object according to the gray value corresponding to the second key point includes:
  • the average value is determined as the depth information of the target object.
  • the embodiment of the present application provides a robot navigation device, including:
  • an image acquisition unit configured to acquire a photographed image through the photographing device
  • a target detection unit configured to detect a target object in the captured image
  • a depth acquisition unit configured to acquire depth information of the target object from the captured image
  • a distance calculation unit configured to determine the actual distance between the target object and the robot according to the depth information
  • a navigation unit configured to plan the movement route of the robot according to the actual distance.
  • an embodiment of the present application provides a robot, including a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor executes the The computer program realizes the robot navigation method described in any one of the above first aspects.
  • the embodiment of the present application provides a computer-readable storage medium
  • the embodiment of the present application provides a computer-readable storage medium
  • the computer-readable storage medium stores a computer program, and it is characterized in that the When the computer program is executed by the processor, the robot navigation method according to any one of the above-mentioned first aspects is realized.
  • an embodiment of the present application provides a computer program product, which, when the computer program product is run on a terminal device, causes the terminal device to execute the robot navigation method described in any one of the above-mentioned first aspects.
  • Fig. 1 is the schematic diagram of the robot that the embodiment of the present application provides;
  • Fig. 2 is a schematic flow chart of the robot navigation method provided by the embodiment of the present application.
  • FIG. 3 is a schematic diagram of an image of human body key point detection provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of an image of human body key point detection provided by an embodiment of the present application.
  • Fig. 5 is a schematic diagram of a depth estimation model provided by an embodiment of the present application.
  • Fig. 6 is a schematic diagram of the implementation flow of the robot navigation method provided by the embodiment of the present application.
  • Fig. 7 is a structural block diagram of a robot navigation device provided by an embodiment of the present application.
  • Fig. 8 is a schematic structural diagram of a robot provided by an embodiment of the present application.
  • the term “if” may be construed, depending on the context, as “when” or “once” or “in response to determining” or “in response to detecting ".
  • references to "one embodiment” or “some embodiments” or the like in the specification of the present application means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application.
  • appearances of the phrases “in one embodiment,” “in some embodiments,” “in other embodiments,” “in other embodiments,” etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean “one or more but not all embodiments” unless specifically stated otherwise.
  • FIG. 1 it is a schematic diagram of a robot provided in an embodiment of the present application.
  • the robot includes a robot body and a processor.
  • a wheeled structure is installed at the bottom of the robot body; there is a reversible display screen above the robot body, and an RGB camera is mounted on the top of the display screen; a tray for placing items is arranged behind the robot body.
  • the robot navigation method provided in the embodiment of the present application may be applied to the robot shown in FIG. 1 .
  • the processor of the robot acquires the captured image through the RGB camera, and then according to the robot navigation method provided in the embodiment of the application, determines the actual distance between the user and the robot, and according to the actual distance Plan the route; finally, the processor controls the movement of the robot body to the user's location according to the planned route.
  • the processor controls the display screen to flip to face the user through a proportional-integral-differential (PID) control algorithm.
  • PID proportional-integral-differential
  • FIG. 2 it is a schematic flowchart of a robot navigation method provided by an embodiment of the present application.
  • the method may include the following steps:
  • the photographed image is acquired by the photographing device.
  • the photographing device may be an RGB camera as shown in FIG. 1 .
  • the captured image is an RGB image.
  • the target behavior can be detected by the method of key point detection. Specifically, the captured image is input into the trained key point detection model, and the key point information is output; the target behavior in the captured image is detected according to the key point information.
  • the output of the key point detection model may only include key point information, such as the label of the key point and the position information of each key point.
  • the output of the key point detection model can also be a captured image carrying key point information, that is, the position of each key point is marked on the captured image. As shown in (a) of Figure 3, multiple key points are marked on the captured image.
  • the output of the key point detection model can also be a key point detection image carrying key point information, as shown in (b) in FIG. 3 .
  • an implementation manner of detecting the target behavior in the captured image according to the key point information may be:
  • the detection result indicates that there is a target behavior in the captured image.
  • the preset position relationship can also be set according to different key point detection methods.
  • the preset position relationship is that the ordinate of key point 4 is greater than the ordinate of 3, and the connection line between key point 4 and 3 forms a preset clip with the horizontal plane.
  • angle, or the ordinate of the key point 7 is greater than the ordinate of 6, and the line connecting the key points 7 and 6 forms a preset included angle with the horizontal plane.
  • the target object may be a person image in a captured image.
  • the person image included in the detection frame 31 as shown in (a) of FIG. 3 may be used as the target object.
  • the target object can also be used to represent the key point information of the person. Key points 0-17 as shown in (b) of FIG. 3 can be used as target objects.
  • an implementation manner of S202 may be:
  • the detected key point information includes two sets of key points, each set of key points includes 18 key points, and each set of key points corresponds to a character image; then according to each set of key points corresponding to each character image Point information identifies the behavior of each character. Assume that the behavior of the first character is recognized as raising hands, the behavior of the second character is standing, and the target behavior is raising hands. Then the person image of the first person is the target object.
  • another implementation of S202 may be:
  • the detected key point information includes multiple sets of key points. In this case, it is necessary to determine which character performed the target behavior.
  • the second key point associated with the first key point refers to a key point that belongs to the same shooting object (person or object) as the first key point. Therefore, the first key point may be included in the second key point.
  • FIG. 4 is a schematic diagram of an image of human body key point detection provided by the embodiment of the present application.
  • Fig. 4 there are two groups of human body key points 0-17 and 18-35 (that is, two persons are detected). If key points 3 and 4 are detected as the first key points, then the second key points associated with key points 3 and 4 are 0-17.
  • S203 may include the following steps:
  • a depth image corresponding to the captured image is acquired through a depth camera; depth information of a target object is acquired according to the depth image.
  • one implementation of acquiring the depth information of the target object according to the depth image is: respectively mapping the second key point and the depth image to a preset coordinate system; finding the depth corresponding to the second key point in the preset coordinate system value.
  • S203 may include the following steps:
  • an implementation of converting the captured image into a grayscale image is as follows:
  • the depth estimation model includes an encoding layer and a decoding layer; the encoding layer is used to extract features from the captured image to obtain a feature map; the decoding layer is used to upsample the feature map to obtain a grayscale image.
  • the conversion of the captured image into a grayscale image is realized based on a deep learning convolutional neural network.
  • Other methods can also be used, such as methods based on recurrent neural networks and methods based on generative adversarial networks in monocular depth estimation methods, which are not specifically limited here.
  • an implementation manner of obtaining the depth information of the target object according to the grayscale image is as follows:
  • an implementation manner of obtaining the depth information of the target object according to the grayscale image is as follows:
  • an implementation method of obtaining the gray value corresponding to the second key point in the gray image is as follows:
  • mapping the second key point to the preset coordinate system to obtain a mapping point mapping the grayscale image to the preset coordinate system to obtain a mapping graph; finding the gray value corresponding to the mapping point from the mapping graph.
  • mapping the second key point and the grayscale image into the same coordinate system is equivalent to establishing the corresponding relationship between the second key point and the gray value, so as to accurately find the gray value corresponding to the second key point .
  • an implementation manner of calculating the depth information of the target object according to the gray value corresponding to the second key point is as follows:
  • Another implementation manner of calculating the depth information of the target object according to the gray value corresponding to the second key point is:
  • the target key point is determined from the second key point; the average value of the gray value corresponding to the target key point is calculated; and the average value is determined as the depth information of the target object.
  • this method Compared with calculating the average value of the gray values corresponding to all the second key points, this method requires less calculation, which is beneficial to improve the efficiency of robot navigation.
  • Some of the key points of the human body can be preset as target key points. For example, 1 and 14-15 among the 18 human body key points shown in Fig. 3 are taken as target key points.
  • Depth information can be mapped into a geographic coordinate system to determine the actual distance between the target object and the robot.
  • an implementation of S205 may be as follows: determine the actual position of the target object according to the actual distance and the current position of the robot; use the current position of the robot as the starting point and the actual position of the target object as the end point, and plan the distance from the starting point to the target object. The navigation route between the end points; control the movement of the robot according to the navigation route.
  • FIG. 6 it is a schematic diagram of the implementation flow of the robot navigation method provided by the embodiment of the present application.
  • the robot acquires the photographed images of multiple people through the photographing device; then the recognition of the hand-raising behavior (i.e. the recognition of the target behavior) is performed on the photographed image; if the hand-raising behavior is recognized, the depth estimation is performed on the photographed image , to obtain a grayscale image; sample the position of the person raising the hand in the image (ie, the target object) to obtain the target key point; then align the target key point and the grayscale image offline (ie, map to the same coordinate system), To obtain the depth value corresponding to the target key point.
  • the recognition of the hand-raising behavior i.e. the recognition of the target behavior
  • the depth estimation is performed on the photographed image , to obtain a grayscale image
  • sample the position of the person raising the hand in the image (ie, the target object) to obtain the target key point
  • align the target key point and the grayscale image offline ie, map to the
  • the display screen of the robot can be controlled to turn to the person corresponding to the target object through the PID control method.
  • the head key points of the target object can be used to locate the position of the head of the target object in the captured image, and then the display screen of the robot is controlled to flip according to the position, so that the head of the target object is at the center of the captured image.
  • FIG. 7 is a structural block diagram of the robot navigation device provided by the embodiment of the present application. For the convenience of description, only the parts related to the embodiment of the present application are shown.
  • the device includes:
  • the behavior detection unit 71 is configured to detect the target behavior in the photographed image, and obtain the detection result, and the photographed image is acquired by the photographing device.
  • the target detection unit 72 is configured to detect a target object corresponding to the target behavior in the captured image if the detection result indicates that the target behavior exists in the captured image.
  • a depth acquisition unit 73 configured to acquire depth information of the target object from the captured image.
  • a distance calculation unit 74 configured to determine the actual distance between the target object and the robot according to the depth information.
  • the navigation unit 75 is configured to perform navigation control on the robot according to the actual distance.
  • the behavior detection unit 71 is also used for:
  • Key point detection is performed on the captured image to obtain key point information; target behavior in the captured image is detected according to the key point information to obtain a detection result.
  • the key point information includes respective position information of multiple key points in the captured image.
  • the behavior detection unit 71 is also used for:
  • the detection result indicates that the target behavior exists in the captured image.
  • the target detection unit 72 is also used for:
  • the detection result indicates that the target behavior exists in the captured image, then obtain a second key point associated with the first key point from the plurality of key points; determine the second key point is the target object in the captured image.
  • the depth acquisition unit 73 is also used for:
  • the depth acquisition unit 73 is also used for:
  • the captured image is input into a trained depth estimation model, and the grayscale image is output;
  • the depth estimation model includes a coding layer and a decoding layer;
  • the coding layer is used for feature extraction of the captured image , to obtain a feature map;
  • the decoding layer is used to perform upsampling processing on the feature map to obtain the grayscale image.
  • the depth acquisition unit 73 is also used for:
  • mapping the second key point to a preset coordinate system to obtain a mapping point
  • mapping the grayscale image to the preset coordinate system to obtain a mapping map
  • searching the mapping point corresponding to the mapping point from the mapping map grayscale value.
  • the depth acquisition unit 73 is also used for:
  • the robot navigation device shown in FIG. 7 can be a software unit, a hardware unit, or a combination of software and hardware built into existing terminal equipment, or it can be integrated into the terminal equipment as an independent pendant, or it can be Exists as an independent terminal device.
  • Fig. 8 is a schematic structural diagram of a robot provided by an embodiment of the present application.
  • the robot 8 of this embodiment comprises: at least one processor 80 (only one is shown in Figure 8) processor, memory 81 and be stored in described memory 81 and can be in described at least one processor A computer program 82 running on 80, when the processor 80 executes the computer program 82, it realizes the steps in any of the above embodiments of the robot navigation method.
  • the robot may include, but is not limited to, a processor, memory.
  • Fig. 8 is only an example of the robot 8, and does not constitute a limitation to the robot 8, and may include more or less components than those shown in the illustration, or combine some components, or different components, for example It may also include input and output devices, network access devices, etc.
  • the so-called processor 80 can be a central processing unit (Central Processing Unit, CPU), and the processor 80 can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit) , ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the memory 81 may be an internal storage unit of the robot 8 in some embodiments, such as a hard disk or memory of the robot 8 .
  • the memory 81 can also be an external storage device of the robot 8 in other embodiments, such as a plug-in hard disk equipped on the robot 8, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital) Digital, SD) card, flash memory card (Flash Card), etc.
  • the memory 81 may also include both an internal storage unit of the robot 8 and an external storage device.
  • the memory 81 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program.
  • the memory 81 can also be used to temporarily store data that has been output or will be output.
  • the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in each of the foregoing method embodiments can be realized.
  • An embodiment of the present application provides a computer program product.
  • the terminal device can implement the steps in the foregoing method embodiments when executed.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the procedures in the methods of the above embodiments in the present application can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a computer-readable storage medium.
  • the computer program When executed by a processor, the steps in the above-mentioned various method embodiments can be realized.
  • the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form.
  • the computer-readable medium may at least include: any entity or device capable of carrying computer program codes to a device/terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media.
  • ROM read-only memory
  • RAM Random Access Memory
  • electrical carrier signals telecommunication signals
  • software distribution media Such as U disk, mobile hard disk, magnetic disk or optical disk, etc.
  • computer readable media may not be electrical carrier signals and telecommunication signals under legislation and patent practice.
  • the disclosed devices/robots and methods may be implemented in other ways.
  • the device/robot embodiments described above are only illustrative.
  • the division of the modules or units is only a logical function division.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Automation & Control Theory (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)

Abstract

A robot navigation method, a robot and a computer-readable storage medium. The robot navigation method comprises: S201, detecting a target behavior in a captured image, so as to obtain a detection result; S202, if the detection result shows that there is a target behavior in the captured image, then detecting a target object corresponding to the target behavior in the captured image; S203, acquiring depth information of the target object from the captured image; S204, determining the actual distance between the target object and a robot according to the depth information; and S205, performing navigation control on the robot according to the actual distance. By means of the robot navigation method, the dependence of a robot on a marker during movement of the robot can be avoided, thereby effectively improving the flexibility of the movement of the robot.

Description

机器人导航方法、机器人及计算机可读存储介质Robot navigation method, robot and computer readable storage medium
本申请要求于2021年06月30日在中国专利局提交的、申请号为202110736327.5的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims priority to a Chinese patent application with application number 202110736327.5 filed at the China Patent Office on June 30, 2021, the entire contents of which are incorporated herein by reference.
技术领域technical field
本申请属于机器人技术领域，尤其涉及一种机器人导航方法、机器人及计算机可读存储介质。The present application belongs to the technical field of robots, and in particular relates to a robot navigation method, a robot and a computer-readable storage medium.
背景技术Background technique
随着智能控制技术的发展，机器人的应用越来越广泛。配送机器人是一种常见的服务型机器人，该类机器人主要用于物品的配送。例如，送餐机器人用于将餐品送达指定地点。With the development of intelligent control technology, the application of robots is becoming more and more extensive. Distribution robot is a common service robot, which is mainly used for the distribution of goods. For example, food delivery robots are used to deliver meals to designated locations.
现有技术中，配送机器人的导航通常依赖于标志物。例如，在场景中布置标志物(如条码、标识图像等)，机器人在移动过程中，通过识别标志物实现精准定位。通过现有方法，配送机器人只能按照固定的路线移动，灵活性较差；另外，需要在应用场景中布置标志物，成本较高。In the prior art, the navigation of a delivery robot usually relies on markers. For example, markers (such as barcodes, logo images, etc.) are arranged in the scene, and the robot can achieve precise positioning by recognizing markers during the movement. With the existing method, the distribution robot can only move along a fixed route, which has poor flexibility; in addition, markers need to be arranged in the application scene, which is costly.
技术问题technical problem
本申请实施例提供了一种机器人导航方法、机器人及计算机可读存储介质，可以避免机器人移动过程中对标志物的依赖，有效提高机器人移动的灵活性。Embodiments of the present application provide a robot navigation method, a robot, and a computer-readable storage medium, which can avoid dependence on markers during the movement of the robot, and effectively improve the flexibility of the movement of the robot.
技术解决方案technical solution
第一方面，本申请实施例提供了一种机器人导航方法，应用于机器人，所述机器人上安装有拍摄装置，所述方法包括：In the first aspect, the embodiment of the present application provides a robot navigation method, which is applied to a robot, and a shooting device is installed on the robot, and the method includes:
检测拍摄图像中的目标行为，得到检测结果，所述拍摄图像由所述拍摄装置获取；Detecting the target behavior in the photographed image to obtain a detection result, the photographed image being acquired by the photographing device;
如果所述检测结果表示所述拍摄图像中存在所述目标行为，则检测所述拍摄图像中所述目标行为对应的目标对象；If the detection result indicates that the target behavior exists in the captured image, detecting a target object corresponding to the target behavior in the captured image;
从所述拍摄图像中获取所述目标对象的深度信息；acquiring depth information of the target object from the captured image;
根据所述深度信息确定所述目标对象与所述机器人之间的实际距离；determining an actual distance between the target object and the robot according to the depth information;
根据所述实际距离对所述机器人进行导航控制。Perform navigation control on the robot according to the actual distance.
本申请实施例中，通过对拍摄图像进行图像检测，确定拍摄图像中的目标行为，进而确定目标行为对应的目标对象，然后从拍摄图像中获取目标对象的深度信息；通过这种方式，无需重新拍摄深度图像，减少了机器人的装置成本；根据目标对象的深度信息计算机器人与目标对象之间的实际距离，然后根据实际距离规划机器人的运动路线。通过上述方法，机器人在移动过程中无需依赖标志物，且无需按照固定的路线移动，而是可以根据识别出的目标对象灵活的规划运动路线，有效提高机器人移动的灵活性。In the embodiment of the present application, the target behavior in the captured image is determined by performing image detection on the captured image, and then the target object corresponding to the target behavior is determined, and then the depth information of the target object is obtained from the captured image; Taking depth images reduces the device cost of the robot; calculates the actual distance between the robot and the target object according to the depth information of the target object, and then plans the robot's movement route according to the actual distance. Through the above method, the robot does not need to rely on markers during the movement process, and does not need to move along a fixed route, but can flexibly plan a movement route according to the identified target object, effectively improving the flexibility of the robot's movement.
在第一方面的一种可能的实现方式中，所述检测所述拍摄图像中的目标对象，包括：In a possible implementation manner of the first aspect, the detecting the target object in the captured image includes:
对所述拍摄图像进行关键点检测，得到关键点信息；Carrying out key point detection on the captured image to obtain key point information;
根据所述关键点信息检测所述拍摄图像中的目标行为，得到所述检测结果。Detecting the target behavior in the captured image according to the key point information to obtain the detection result.
在第一方面的一种可能的实现方式中，所述关键点信息包括多个关键点在所述拍摄图像中各自的位置信息；In a possible implementation manner of the first aspect, the key point information includes respective position information of a plurality of key points in the captured image;
所述根据所述关键点信息检测所述拍摄图像中的目标行为，得到检测结果，包括：The detecting the target behavior in the captured image according to the key point information to obtain the detection result includes:
根据所述多个关键点在所述拍摄图像中各自的位置信息，判断所述多个关键点中是否存在满足预设位置关系的第一关键点；According to the respective position information of the plurality of key points in the captured image, determine whether there is a first key point satisfying a preset position relationship among the plurality of key points;
如果所述多个关键点中存在满足预设位置关系的第一关键点，则所述检测结果表示所述拍摄图像中存在所述目标行为。If there is a first key point satisfying a preset position relationship among the plurality of key points, the detection result indicates that the target behavior exists in the captured image.
在第一方面的一种可能的实现方式中，所述如果所述检测结果表示所述拍摄图像中存在所述目标行为，则检测所述拍摄图像中的目标行为对应的目标对象，包括：In a possible implementation manner of the first aspect, if the detection result indicates that the target behavior exists in the captured image, detecting the target object corresponding to the target behavior in the captured image includes:
如果所述检测结果表示所述拍摄图像中存在所述目标行为，则从所述多个关键点中获取与所述第一关键点相关联的第二关键点；If the detection result indicates that the target behavior exists in the captured image, obtaining a second key point associated with the first key point from the plurality of key points;
将所述第二关键点确定为所述拍摄图像中的目标对象。The second key point is determined as a target object in the captured image.
在第一方面的一种可能的实现方式中，所述目标对象包括第二关键点；In a possible implementation manner of the first aspect, the target object includes a second key point;
所述从所述拍摄图像中获取所述目标对象的深度信息，包括：The acquiring the depth information of the target object from the captured image includes:
将所述拍摄图像转换为灰度图；converting the captured image into a grayscale image;
获取所述灰度图中所述第二关键点对应的灰度值；Acquiring the gray value corresponding to the second key point in the gray image;
根据所述第二关键点对应的灰度值计算所述目标对象的深度信息。calculating the depth information of the target object according to the gray value corresponding to the second key point.
在第一方面的一种可能的实现方式中，所述将所述拍摄图像转换为灰度图，包括：In a possible implementation manner of the first aspect, the converting the captured image into a grayscale image includes:
将所述拍摄图像输入到训练后的深度估计模型中，输出所述灰度图；Inputting the captured image into the trained depth estimation model, and outputting the grayscale image;
其中，所述深度估计模型包括编码层和解码层；所述编码层用于对所述拍摄图像进行特征提取，得到特征图；所述解码层用于对所述特征图进行上采样处理，得到所述灰度图。Wherein, the depth estimation model includes an encoding layer and a decoding layer; the encoding layer is used to perform feature extraction on the captured image to obtain a feature map; the decoding layer is used to perform upsampling processing on the feature map to obtain The grayscale image.
在第一方面的一种可能的实现方式中，所述获取所述灰度图中所述第二关键点对应的灰度值，包括：In a possible implementation manner of the first aspect, the obtaining the gray value corresponding to the second key point in the gray image includes:
将所述第二关键点映射到预设坐标系，得到映射点；Mapping the second key point to a preset coordinate system to obtain a mapping point;
将所述灰度图映射到所述预设坐标系，得到映射图；Mapping the grayscale image to the preset coordinate system to obtain a mapping image;
从所述映射图中查找所述映射点对应的灰度值。Find the gray value corresponding to the mapping point from the mapping map.
在第一方面的一种可能的实现方式中，所述根据所述第二关键点对应的灰度值计算所述目标对象的深度信息，包括：In a possible implementation manner of the first aspect, the calculating the depth information of the target object according to the gray value corresponding to the second key point includes:
从所述第二关键点中确定出目标关键点；determining a target key point from the second key point;
计算所述目标关键点对应的灰度值的平均值；Calculate the average value of the gray value corresponding to the target key point;
将所述平均值确定为所述目标对象的深度信息。The average value is determined as the depth information of the target object.
第二方面，本申请实施例提供了一种机器人导航装置，包括：In the second aspect, the embodiment of the present application provides a robot navigation device, including:
图像获取单元，用于通过所述拍摄装置获取拍摄图像；an image acquisition unit, configured to acquire a photographed image through the photographing device;
目标检测单元，用于检测所述拍摄图像中的目标对象；a target detection unit, configured to detect a target object in the captured image;
深度获取单元，用于从所述拍摄图像中获取所述目标对象的深度信息；a depth acquisition unit, configured to acquire depth information of the target object from the captured image;
距离计算单元，用于根据所述深度信息确定所述目标对象与所述机器人之间的实际距离；a distance calculation unit, configured to determine the actual distance between the target object and the robot according to the depth information;
导航单元，用于根据所述实际距离规划所述机器人的运动路线。a navigation unit, configured to plan the movement route of the robot according to the actual distance.
第三方面，本申请实施例提供了一种机器人，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现如上述第一方面中任一项所述的机器人导航方法。In a third aspect, an embodiment of the present application provides a robot, including a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor executes the The computer program realizes the robot navigation method described in any one of the above first aspects.
第四方面，本申请实施例提供了一种计算机可读存储介质，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如上述第一方面中任一项所述的机器人导航方法。In a fourth aspect, the embodiment of the present application provides a computer-readable storage medium, the embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and it is characterized in that the When the computer program is executed by the processor, the robot navigation method according to any one of the above-mentioned first aspects is realized.
第五方面，本申请实施例提供了一种计算机程序产品，当计算机程序产品在终端设备上运行时，使得终端设备执行上述第一方面中任一项所述的机器人导航方法。In a fifth aspect, an embodiment of the present application provides a computer program product, which, when the computer program product is run on a terminal device, causes the terminal device to execute the robot navigation method described in any one of the above-mentioned first aspects.
可以理解的是，上述第二方面至第五方面的有益效果可以参见上述第一方面中的相关描述，在此不再赘述。It can be understood that, for the beneficial effects of the above-mentioned second aspect to the fifth aspect, reference can be made to the relevant description in the above-mentioned first aspect, and details will not be repeated here.
附图说明Description of drawings
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application, the accompanying drawings that need to be used in the descriptions of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are only for the present application For some embodiments, those of ordinary skill in the art can also obtain other drawings based on these drawings without paying creative efforts.
图1是本申请实施例提供的机器人的示意图；Fig. 1 is the schematic diagram of the robot that the embodiment of the present application provides;
图2是本申请实施例提供的机器人导航方法的流程示意图；Fig. 2 is a schematic flow chart of the robot navigation method provided by the embodiment of the present application;
图3是本申请实施例提供的人体关键点检测的图像示意图；FIG. 3 is a schematic diagram of an image of human body key point detection provided by an embodiment of the present application;
图4是本申请实施例提供的人体关键点检测的图像示意图；FIG. 4 is a schematic diagram of an image of human body key point detection provided by an embodiment of the present application;
图5是本申请实施例提供的深度估计模型的示意图；Fig. 5 is a schematic diagram of a depth estimation model provided by an embodiment of the present application;
图6是本申请实施例提供的机器人导航方法的实施流程示意图；Fig. 6 is a schematic diagram of the implementation flow of the robot navigation method provided by the embodiment of the present application;
图7是本申请实施例提供的机器人导航装置的结构框图；Fig. 7 is a structural block diagram of a robot navigation device provided by an embodiment of the present application;
图8是本申请实施例提供的机器人的结构示意图。Fig. 8 is a schematic structural diagram of a robot provided by an embodiment of the present application.
本发明的实施方式Embodiments of the present invention
以下描述中，为了说明而不是为了限定，提出了诸如特定***结构、技术之类的具体细节，以便透彻理解本申请实施例。然而，本领域的技术人员应当清楚，在没有这些具体细节的其它实施例中也可以实现本申请。在其它情况中，省略对众所周知的***、装置、电路以及方法的详细说明，以免不必要的细节妨碍本申请的描述。In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
应当理解，当在本申请说明书和所附权利要求书中使用时，术语“包括”指示所描述特征、整体、步骤、操作、元素和/或组件的存在，但并不排除一个或多个其它特征、整体、步骤、操作、元素、组件和/或其集合的存在或添加。It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and/or components, but does not exclude one or more other Presence or addition of features, wholes, steps, operations, elements, components and/or collections thereof.
如在本申请说明书和所附权利要求书中所使用的那样，术语“如果”可以依据上下文被解释为“当...时”或“一旦”或“响应于确定”或“响应于检测到”。As used in this specification and the appended claims, the term "if" may be construed, depending on the context, as "when" or "once" or "in response to determining" or "in response to detecting ".
另外，在本申请说明书和所附权利要求书的描述中，术语“第一”、“第二”、“第三”等仅用于区分描述，而不能理解为指示或暗示相对重要性。In addition, in the description of the specification and appended claims of the present application, the terms "first", "second", "third" and so on are only used to distinguish descriptions, and should not be understood as indicating or implying relative importance.
在本申请说明书中描述的参考“一个实施例”或“一些实施例”等意味着在本申请的一个或多个实施例中包括结合该实施例描述的特定特征、结构或特点。由此，在本说明书中的不同之处出现的语句“在一个实施例中”、“在一些实施例中”、“在其他一些实施例中”、“在另外一些实施例中”等不是必然都参考相同的实施例，而是意味着“一个或多个但不是所有的实施例”，除非是以其他方式另外特别强调。Reference to "one embodiment" or "some embodiments" or the like in the specification of the present application means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically stated otherwise.
参见图1，是本申请实施例提供的机器人的示意图。机器人包括机器人本体和处理器。如图1所示，机器人本体底部安装有轮式结构；机器人本体上方有一个可翻转的显示屏，显示屏上方搭载一个RGB相机；机器人本体后方设置有放置物品的托盘。Referring to FIG. 1 , it is a schematic diagram of a robot provided in an embodiment of the present application. The robot includes a robot body and a processor. As shown in Figure 1, a wheeled structure is installed at the bottom of the robot body; there is a reversible display screen above the robot body, and an RGB camera is mounted on the top of the display screen; a tray for placing items is arranged behind the robot body.
本申请实施例提供的机器人导航方法可以应用于图1所示的机器人。在一个应用场景中，当用户举手示意时，机器人的处理器通过RGB相机获取拍摄图像，然后根据本申请实施例提供的机器人导航方法，确定用户与机器人之间的实际距离，并根据实际距离规划路 线；最后处理器根据规划好的路线控制机器人本体运动向用户所在位置方向运动。当机器人靠近用户时，处理器通过比例-积分-微分(Proportional-Integral-Differential，PID)控制算法控制显示屏翻转，使之面向用户。The robot navigation method provided in the embodiment of the present application may be applied to the robot shown in FIG. 1 . In one application scenario, when the user raises his hand to signal, the processor of the robot acquires the captured image through the RGB camera, and then according to the robot navigation method provided in the embodiment of the application, determines the actual distance between the user and the robot, and according to the actual distance Plan the route; finally, the processor controls the movement of the robot body to the user's location according to the planned route. When the robot is close to the user, the processor controls the display screen to flip to face the user through a proportional-integral-differential (PID) control algorithm.
下面介绍本申请实施例提供的机器人导航方法。参见图2，是本申请实施例提供的机器人导航方法的流程示意图，作为示例而非限定，所述方法可以包括以下步骤：The following introduces the robot navigation method provided by the embodiment of the present application. Referring to FIG. 2 , it is a schematic flowchart of a robot navigation method provided by an embodiment of the present application. As an example but not a limitation, the method may include the following steps:
S201，检测拍摄图像中的目标行为，得到检测结果。S201. Detect a target behavior in a captured image, and obtain a detection result.
拍摄图像由拍摄装置获取。拍摄装置可以是如图1中所示的RGB相机。拍摄图像为RGB图像。The photographed image is acquired by the photographing device. The photographing device may be an RGB camera as shown in FIG. 1 . The captured image is an RGB image.
可以通过关键点检测的方法检测目标行为。具体的，将拍摄图像输入到训练后的关键点检测模型中，输出关键点信息；根据关键点信息检测拍摄图像中的目标行为。The target behavior can be detected by the method of key point detection. Specifically, the captured image is input into the trained key point detection model, and the key point information is output; the target behavior in the captured image is detected according to the key point information.
关键点信息包括多个关键点在所述拍摄图像中各自的位置信息。示例性的，参见图3，是本申请实施例提供的人体关键点检测的图像示意图。如图3所示，可以采用18个人体关键点的检测方式，18个人体关键点中包括5个头部关键点(0和14-17)和13个身体关键点(1-13)。The key point information includes respective position information of a plurality of key points in the captured image. For example, refer to FIG. 3 , which is a schematic diagram of an image of human body key point detection provided by the embodiment of the present application. As shown in FIG. 3 , 18 human body key points can be detected, and the 18 human body key points include 5 head key points (0 and 14-17) and 13 body key points (1-13).
关键点检测模型的输出可以仅包括关键点信息，如关键点的标号和每个关键点的位置信息这些数据。关键点检测模型的输出还可以为携带有关键点信息的拍摄图像，即在拍摄图像上标注有各个关键点的位置。如图3中的(a)所示，拍摄图像上标注有多个关键点。关键点检测模型的输出还可以为携带有关键点信息的关键点检测图像，如图3中的(b)所示。The output of the key point detection model may only include key point information, such as the label of the key point and the position information of each key point. The output of the key point detection model can also be a captured image carrying key point information, that is, the position of each key point is marked on the captured image. As shown in (a) of Figure 3, multiple key points are marked on the captured image. The output of the key point detection model can also be a key point detection image carrying key point information, as shown in (b) in FIG. 3 .
可选的，根据关键点信息检测拍摄图像中的目标行为的一种实现方式可以为：Optionally, an implementation manner of detecting the target behavior in the captured image according to the key point information may be:
根据多个关键点在拍摄图像中各自的位置信息，判断多个关键点中是否存在满足预设位置关系的第一关键点；如果多个关键点中存在满足预设位置关系的第一关键点，则检测结果表示拍摄图像中存在目标行为。According to the respective position information of multiple key points in the captured image, determine whether there is a first key point satisfying the preset position relationship among the multiple key points; if there is a first key point satisfying the preset position relationship among the multiple key points , the detection result indicates that there is a target behavior in the captured image.
示例性的，假设预设位置关系为，手腕关键点高于手肘关键点，且小臂与水平面成预设夹角。先确定关键点表示部位与关键点编号的对应关系。当采用如图3中所示的18个关键点时，手肘关键点对应关键点3和6，手腕关键点对应关键点4和6。判断过程为：判断关键点4是否高于3、且3和4的连线是否与水平面成预设夹角，或者判断关键点7是否高于6、且6和7的联系是否水平面成预设夹角。Exemplarily, it is assumed that the preset position relationship is that the key point of the wrist is higher than the key point of the elbow, and the forearm forms a preset angle with the horizontal plane. First determine the corresponding relationship between the key point representation part and the key point number. When using 18 key points as shown in Figure 3, the elbow key point corresponds to key points 3 and 6, and the wrist key point corresponds to key points 4 and 6. The judging process is: judging whether the key point 4 is higher than 3, and whether the line connecting 3 and 4 forms a preset angle with the horizontal plane, or judging whether the key point 7 is higher than 6, and whether the connection between 6 and 7 is a preset angle angle.
当然，也可以根据不同的关键点检测方式设置预设位置关系。例如，当采用如图3中所示的18个关键点时，预设位置关系为，关键点4的纵坐标大于3的纵坐标、且关键点4与3的连线与水平面成预设夹角，或者关键点7的纵坐标大于6的纵坐标、且关键点7与6的连线与水平面成预设夹角。Of course, the preset position relationship can also be set according to different key point detection methods. For example, when using 18 key points as shown in Figure 3, the preset position relationship is that the ordinate of key point 4 is greater than the ordinate of 3, and the connection line between key point 4 and 3 forms a preset clip with the horizontal plane. angle, or the ordinate of the key point 7 is greater than the ordinate of 6, and the line connecting the key points 7 and 6 forms a preset included angle with the horizontal plane.
S202，如果检测结果表示拍摄图像中存在目标行为，则检测拍摄图像述目标行为对应的目标对象。本申请实施例中，目标对象可以是拍摄图像中的人物图像。如图3中的(a)所示的检测框31所包括的人物图像可以作为目标对象。目标对象还可以是用于表示人物的 关键点信息。如图3中的(b)所示的关键点0-17可以作为目标对象。S202. If the detection result indicates that there is a target behavior in the captured image, detect a target object corresponding to the target behavior in the captured image. In this embodiment of the present application, the target object may be a person image in a captured image. The person image included in the detection frame 31 as shown in (a) of FIG. 3 may be used as the target object. The target object can also be used to represent the key point information of the person. Key points 0-17 as shown in (b) of FIG. 3 can be used as target objects.
在一个实施例中，S202的一种实现方式可以为：In an embodiment, an implementation manner of S202 may be:
对拍摄图像进行关键点检测，得到关键点信息；根据关键点信息确定拍摄图像中的人物图像；根据关键点信息识别每个人物图像对应的行为；将目标行为对应的人物图像确定为目标对象。Carry out key point detection on the captured image to obtain key point information; determine the person image in the captured image according to the key point information; identify the behavior corresponding to each person image according to the key point information; determine the person image corresponding to the target behavior as the target object.
示例性的，假设检测出的关键点信息中包括两组关键点，每组关键点中包括18个关键点，每组关键点对应一个人物图像；然后根据每个人物图像各自对应的一组关键点信息识别每个人物的行为。假设识别出第一个人物的行为是举手，第二个人物的行为是站立，目标行为是举手。那么第一个人物的人物图像即为目标对象。Exemplarily, it is assumed that the detected key point information includes two sets of key points, each set of key points includes 18 key points, and each set of key points corresponds to a character image; then according to each set of key points corresponding to each character image Point information identifies the behavior of each character. Assume that the behavior of the first character is recognized as raising hands, the behavior of the second character is standing, and the target behavior is raising hands. Then the person image of the first person is the target object.
这种方式需要检测拍摄图像中每个人物图像，数据处理量较大。This method needs to detect each person image in the captured image, and the amount of data processing is relatively large.
为了减少数据处理量，提高检测效率，在一个实施例中，S202的另一种实现方式可以为：In order to reduce the amount of data processing and improve detection efficiency, in one embodiment, another implementation of S202 may be:
从多个关键点中获取与第一关键点相关联的第二关键点；将第二关键点确定为拍摄图像中的目标对象。A second key point associated with the first key point is acquired from the plurality of key points; and the second key point is determined as a target object in the captured image.
实际应用中，拍摄图像中可能包括多个人物，相应的，检测出的关键点信息中包括多组关键点。这种情况下，需要确定是哪个人物发生了目标行为。本申请实施例中，与第一关键点相关联的第二关键点是指，与第一关键点属于同一拍摄对象(人或物)的关键点。所以，第二关键点中可以包括第一关键点。In practical applications, multiple people may be included in the captured image, and correspondingly, the detected key point information includes multiple sets of key points. In this case, it is necessary to determine which character performed the target behavior. In the embodiment of the present application, the second key point associated with the first key point refers to a key point that belongs to the same shooting object (person or object) as the first key point. Therefore, the first key point may be included in the second key point.
示例性的，参见图4，是本申请实施例提供的人体关键点检测的图像示意图。图4中有两组人体关键点0-17和18-35(即检测出两个人物)。检测出关键点3和4为第一关键点，则与关键点3和4相关联的第二关键点为0-17。For example, refer to FIG. 4 , which is a schematic diagram of an image of human body key point detection provided by the embodiment of the present application. In Fig. 4, there are two groups of human body key points 0-17 and 18-35 (that is, two persons are detected). If key points 3 and 4 are detected as the first key points, then the second key points associated with key points 3 and 4 are 0-17.
S203，从拍摄图像中获取目标对象的深度信息。S203. Obtain depth information of the target object from the captured image.
在一个实施例中，S203可以包括以下步骤：In one embodiment, S203 may include the following steps:
通过深度相机获取与拍摄图像对应的深度图像；根据深度图像获取目标对象的深度信息。A depth image corresponding to the captured image is acquired through a depth camera; depth information of a target object is acquired according to the depth image.
具体的，根据深度图像获取目标对象的深度信息的一种实现方式为：将第二关键点和深度图像分别映射到预设坐标系；在预设坐标系中查找与第二关键点对应的深度值。Specifically, one implementation of acquiring the depth information of the target object according to the depth image is: respectively mapping the second key point and the depth image to a preset coordinate system; finding the depth corresponding to the second key point in the preset coordinate system value.
这种方式下，需要在机器人上安装深度相机，增加了机器人的硬件成本。In this way, a depth camera needs to be installed on the robot, which increases the hardware cost of the robot.
在另一个实施例中，S203可以包括以下步骤：In another embodiment, S203 may include the following steps:
将拍摄图像转换为灰度图；根据灰度图获取目标对象的深度信息。Convert the captured image to a grayscale image; obtain the depth information of the target object according to the grayscale image.
这种方式下，无需在机器人上安装深度相机，利用拍摄图像即可获得深度信息，节约了机器人的硬件成本。In this way, there is no need to install a depth camera on the robot, and the depth information can be obtained by taking images, which saves the hardware cost of the robot.
可选的，将拍摄图像转换为灰度图的一种实现方式为：Optionally, an implementation of converting the captured image into a grayscale image is as follows:
将拍摄图像输入到训练后的深度估计模型中，输出灰度图。Input the captured image into the trained depth estimation model and output the grayscale image.
参见图5，是本申请实施例提供的深度估计模型的示意图。如图5所示，深度估计模型包括编码层和解码层；编码层用于对拍摄图像进行特征提取，得到特征图；解码层用于对特征图进行上采样处理，得到灰度图。Referring to FIG. 5 , it is a schematic diagram of a depth estimation model provided by an embodiment of the present application. As shown in Figure 5, the depth estimation model includes an encoding layer and a decoding layer; the encoding layer is used to extract features from the captured image to obtain a feature map; the decoding layer is used to upsample the feature map to obtain a grayscale image.
本申请实施例中，将拍摄图像转换为灰度图，是基于深度学习的卷积神经网络实现的。还可以采用其他方式，如单目深度估计方法中的基于递归神经网络的方法、基于生成对抗网络的方法等，在此不做具体限定。In the embodiment of the present application, the conversion of the captured image into a grayscale image is realized based on a deep learning convolutional neural network. Other methods can also be used, such as methods based on recurrent neural networks and methods based on generative adversarial networks in monocular depth estimation methods, which are not specifically limited here.
可选的，当目标对象为拍摄图像中的人物图像时，根据灰度图获取目标对象的深度信息的一种实现方式为：Optionally, when the target object is a person image in the captured image, an implementation manner of obtaining the depth information of the target object according to the grayscale image is as follows:
在目标对象上任意获取至少一个像素点；获取该像素点在灰度图中对应的灰度值；根据灰度值计算目标对象的深度信息。Arbitrarily obtain at least one pixel point on the target object; obtain the gray value corresponding to the pixel point in the gray scale image; calculate the depth information of the target object according to the gray value.
可选的，当目标对象为关键点信息时，即目标对象包括第二关键点，根据灰度图获取目标对象的深度信息的一种实现方式为：Optionally, when the target object is key point information, that is, the target object includes the second key point, an implementation manner of obtaining the depth information of the target object according to the grayscale image is as follows:
获取灰度图中第二关键点对应的灰度值；根据第二关键点对应的灰度值计算目标对象的深度信息。Obtain the gray value corresponding to the second key point in the gray image; calculate the depth information of the target object according to the gray value corresponding to the second key point.
由于关键点所属图像的图像尺寸和灰度图的图像尺寸可能不同，无法从灰度图中直接找到第二关键点的对应信息。为了解决上述问题，进一步的，获取灰度图中第二关键点对应的灰度值的一种实现方式为：Since the image size of the image to which the key point belongs and the image size of the grayscale image may be different, it is impossible to directly find the corresponding information of the second key point from the grayscale image. In order to solve the above problem, further, an implementation method of obtaining the gray value corresponding to the second key point in the gray image is as follows:
将第二关键点映射到预设坐标系，得到映射点；将灰度图映射到预设坐标系，得到映射图；从映射图中查找映射点对应的灰度值。Mapping the second key point to the preset coordinate system to obtain a mapping point; mapping the grayscale image to the preset coordinate system to obtain a mapping graph; finding the gray value corresponding to the mapping point from the mapping graph.
通过上述方法，将第二关键点和灰度图映射到同一个坐标系中，相当于建立了第二关键点与灰度值的对应关系，以便于准确找到第二关键点对应的灰度值。Through the above method, mapping the second key point and the grayscale image into the same coordinate system is equivalent to establishing the corresponding relationship between the second key point and the gray value, so as to accurately find the gray value corresponding to the second key point .
可选的，根据第二关键点对应的灰度值计算目标对象的深度信息的一种实现方式为：Optionally, an implementation manner of calculating the depth information of the target object according to the gray value corresponding to the second key point is as follows:
计算所有第二关键点对应的灰度值的平均值；将该平均值确定为目标对象的深度信息。Calculate the average value of the gray values corresponding to all the second key points; determine the average value as the depth information of the target object.
可选的，根据第二关键点对应的灰度值计算目标对象的深度信息的另一种实现方式为：Optionally, another implementation manner of calculating the depth information of the target object according to the gray value corresponding to the second key point is:
从第二关键点中确定出目标关键点；计算目标关键点对应的灰度值的平均值；将平均值确定为目标对象的深度信息。The target key point is determined from the second key point; the average value of the gray value corresponding to the target key point is calculated; and the average value is determined as the depth information of the target object.
与计算所有第二关键点对应的灰度值的平均值相比，这种方式计算量较少，有利于提高机器人导航的效率。Compared with calculating the average value of the gray values corresponding to all the second key points, this method requires less calculation, which is beneficial to improve the efficiency of robot navigation.
可以预先设定人体关键点中的某几个关键点作为目标关键点。例如，将图3所示的18个人体关键点中的1和14-15作为目标关键点。Some of the key points of the human body can be preset as target key points. For example, 1 and 14-15 among the 18 human body key points shown in Fig. 3 are taken as target key points.
S204，根据深度信息确定目标对象与机器人之间的实际距离。S204. Determine the actual distance between the target object and the robot according to the depth information.
可以将深度信息(深度值)映射到地理坐标系中，以确定目标对象与机器人之间的实际距离。Depth information (depth values) can be mapped into a geographic coordinate system to determine the actual distance between the target object and the robot.
S205，根据实际距离对机器人进行导航控制。S205, performing navigation control on the robot according to the actual distance.
可选的，S205的一种实现方式可以为：根据实际距离和机器人当前所在位置，确定目标对象的实际位置；将机器人当前所在位置作为起点、将目标对象的实际位置作为终点，规划从起点到终点之间的导航路线；根据导航路线控制机器人运动。Optionally, an implementation of S205 may be as follows: determine the actual position of the target object according to the actual distance and the current position of the robot; use the current position of the robot as the starting point and the actual position of the target object as the end point, and plan the distance from the starting point to the target object. The navigation route between the end points; control the movement of the robot according to the navigation route.
参见图6，是本申请实施例提供的机器人导航方法的实施流程示意图。如图6所示，机器人通过拍摄装置获取多个人物的拍摄图像；然后对拍摄图像进行举手行为的识别(即目标行为的识别)；若识别出举手行为，则对拍摄图像进行深度估计，获得灰度图；对举手者在图像中的位置(即目标对象)进行采样，获得目标关键点；然后将目标关键点和灰度图离线对其(即映射到相同的坐标系)，以获取目标关键点对应的深度值。另外，当机器人靠近目标对象的实际位置时，可以通过PID控制方法控制机器人的显示屏转向目标对象对应的人物。例如，可以利用目标对象的头部关键点定位目标对象的头部在拍摄图像中的位置，然后根据该位置控制机器人的显示屏翻转，以使目标对象的头部处于拍摄图像中的中心位置。Referring to FIG. 6 , it is a schematic diagram of the implementation flow of the robot navigation method provided by the embodiment of the present application. As shown in Figure 6, the robot acquires the photographed images of multiple people through the photographing device; then the recognition of the hand-raising behavior (i.e. the recognition of the target behavior) is performed on the photographed image; if the hand-raising behavior is recognized, the depth estimation is performed on the photographed image , to obtain a grayscale image; sample the position of the person raising the hand in the image (ie, the target object) to obtain the target key point; then align the target key point and the grayscale image offline (ie, map to the same coordinate system), To obtain the depth value corresponding to the target key point. In addition, when the robot is close to the actual position of the target object, the display screen of the robot can be controlled to turn to the person corresponding to the target object through the PID control method. For example, the head key points of the target object can be used to locate the position of the head of the target object in the captured image, and then the display screen of the robot is controlled to flip according to the position, so that the head of the target object is at the center of the captured image.
应理解，上述实施例中各步骤的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.
对应于上文实施例所述的机器人导航方法，图7是本申请实施例提供的机器人导航装置的结构框图，为了便于说明，仅示出了与本申请实施例相关的部分。Corresponding to the robot navigation method described in the above embodiments, FIG. 7 is a structural block diagram of the robot navigation device provided by the embodiment of the present application. For the convenience of description, only the parts related to the embodiment of the present application are shown.
参照图7，该装置包括：Referring to Figure 7, the device includes:
行为检测单元71，用于检测拍摄图像中的目标行为，得到检测结果，所述拍摄图像由所述拍摄装置获取。The behavior detection unit 71 is configured to detect the target behavior in the photographed image, and obtain the detection result, and the photographed image is acquired by the photographing device.
目标检测单元72，用于如果所述检测结果表示所述拍摄图像中存在所述目标行为，则检测所述拍摄图像中所述目标行为对应的目标对象。The target detection unit 72 is configured to detect a target object corresponding to the target behavior in the captured image if the detection result indicates that the target behavior exists in the captured image.
深度获取单元73，用于从所述拍摄图像中获取所述目标对象的深度信息。A depth acquisition unit 73, configured to acquire depth information of the target object from the captured image.
距离计算单元74，用于根据所述深度信息确定所述目标对象与所述机器人之间的实际距离。A distance calculation unit 74, configured to determine the actual distance between the target object and the robot according to the depth information.
导航单元75，用于根据所述实际距离对所述机器人进行导航控制。The navigation unit 75 is configured to perform navigation control on the robot according to the actual distance.
可选的，行为检测单元71还用于：Optionally, the behavior detection unit 71 is also used for:
对所述拍摄图像进行关键点检测，得到关键点信息；根据所述关键点信息检测所述拍摄图像中的目标行为，得到检测结果。Key point detection is performed on the captured image to obtain key point information; target behavior in the captured image is detected according to the key point information to obtain a detection result.
可选的，所述关键点信息包括多个关键点在所述拍摄图像中各自的位置信息。Optionally, the key point information includes respective position information of multiple key points in the captured image.
可选的，行为检测单元71还用于：Optionally, the behavior detection unit 71 is also used for:
根据所述多个关键点在所述拍摄图像中各自的位置信息，判断所述多个关键点中是否存在满足预设位置关系的第一关键点；如果所述多个关键点中存在满足预设位置关系的第一关键点，则所述检测结果表示所述拍摄图像中存在所述目标行为。According to the respective position information of the plurality of key points in the captured image, it is judged whether there is a first key point satisfying the preset position relationship among the plurality of key points; If the first key point of the position relationship is assumed, the detection result indicates that the target behavior exists in the captured image.
可选的，目标检测单元72还用于：Optionally, the target detection unit 72 is also used for:
如果所述检测结果表示所述拍摄图像中存在所述目标行为，则从所述多个关键点中获取与所述第一关键点相关联的第二关键点；将所述第二关键点确定为所述拍摄图像中的目标对象。If the detection result indicates that the target behavior exists in the captured image, then obtain a second key point associated with the first key point from the plurality of key points; determine the second key point is the target object in the captured image.
可选的，深度获取单元73还用于：Optionally, the depth acquisition unit 73 is also used for:
将所述拍摄图像转换为灰度图；获取所述灰度图中所述第二关键点对应的灰度值；根据所述第二关键点对应的灰度值计算所述目标对象的深度信息。Converting the captured image into a grayscale image; acquiring the grayscale value corresponding to the second key point in the grayscale image; calculating the depth information of the target object according to the grayscale value corresponding to the second key point .
可选的，深度获取单元73还用于：Optionally, the depth acquisition unit 73 is also used for:
将所述拍摄图像输入到训练后的深度估计模型中，输出所述灰度图；其中，所述深度估计模型包括编码层和解码层；所述编码层用于对所述拍摄图像进行特征提取，得到特征图；所述解码层用于对所述特征图进行上采样处理，得到所述灰度图。The captured image is input into a trained depth estimation model, and the grayscale image is output; wherein, the depth estimation model includes a coding layer and a decoding layer; the coding layer is used for feature extraction of the captured image , to obtain a feature map; the decoding layer is used to perform upsampling processing on the feature map to obtain the grayscale image.
可选的，深度获取单元73还用于：Optionally, the depth acquisition unit 73 is also used for:
将所述第二关键点映射到预设坐标系，得到映射点；将所述灰度图映射到所述预设坐标系，得到映射图；从所述映射图中查找所述映射点对应的灰度值。Mapping the second key point to a preset coordinate system to obtain a mapping point; mapping the grayscale image to the preset coordinate system to obtain a mapping map; searching the mapping point corresponding to the mapping point from the mapping map grayscale value.
可选的，深度获取单元73还用于：Optionally, the depth acquisition unit 73 is also used for:
从所述第二关键点中确定出目标关键点；计算所述目标关键点对应的灰度值的平均值；将所述平均值确定为所述目标对象的深度信息。Determine the target key point from the second key point; calculate the average value of the gray value corresponding to the target key point; determine the average value as the depth information of the target object.
需要说明的是，上述装置/单元之间的信息交互、执行过程等内容，由于与本申请方法实施例基于同一构思，其具体功能及带来的技术效果，具体可参见方法实施例部分，此处不再赘述。It should be noted that the information interaction and execution process between the above-mentioned devices/units are based on the same concept as the method embodiment of the present application, and its specific functions and technical effects can be found in the method embodiment section. I won't repeat them here.
另外，图7所示的机器人导航装置可以是内置于现有的终端设备内的软件单元、硬件单元、或软硬结合的单元，也可以作为独立的挂件集成到所述终端设备中，还可以作为独立的终端设备存在。In addition, the robot navigation device shown in FIG. 7 can be a software unit, a hardware unit, or a combination of software and hardware built into existing terminal equipment, or it can be integrated into the terminal equipment as an independent pendant, or it can be Exists as an independent terminal device.
所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，仅以上述各功能单元、模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能单元、模块完成，即将所述装置的内部结构划分成不同的功能单元或模块，以完成以上描述的全部或者部分功能。实施例中的各功能单元、模块可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中，上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。另外，各功能单元、模块的具体名称也只是为了便于相互区分，并不用于限制本申请的保护范围。上述***中单元、模块的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used for illustration. In practical applications, the above-mentioned functions can be assigned to different functional units, Completion of modules means that the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated into one processing unit, or each unit may exist separately physically, or two or more units may be integrated into one unit, and the above-mentioned integrated units may adopt hardware It can also be implemented in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the above system, reference may be made to the corresponding process in the foregoing method embodiments, and details will not be repeated here.
图8是本申请实施例提供的机器人的结构示意图。如图8所示，该实施例的机器人8包括：至少一个处理器80(图8中仅示出一个)处理器、存储器81以及存储在所述存储器81中并可在所述至少一个处理器80上运行的计算机程序82，所述处理器80执行所述计算机程序82时实现上述任意各个机器人导航方法实施例中的步骤。Fig. 8 is a schematic structural diagram of a robot provided by an embodiment of the present application. As shown in Figure 8, the robot 8 of this embodiment comprises: at least one processor 80 (only one is shown in Figure 8) processor, memory 81 and be stored in described memory 81 and can be in described at least one processor A computer program 82 running on 80, when the processor 80 executes the computer program 82, it realizes the steps in any of the above embodiments of the robot navigation method.
所述机器人可包括，但不仅限于，处理器、存储器。本领域技术人员可以理解，图8仅仅是机器人8的举例，并不构成对机器人8的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件，例如还可以包括输入输出设备、网络接入设备等。The robot may include, but is not limited to, a processor, memory. Those skilled in the art can understand that Fig. 8 is only an example of the robot 8, and does not constitute a limitation to the robot 8, and may include more or less components than those shown in the illustration, or combine some components, or different components, for example It may also include input and output devices, network access devices, etc.
所称处理器80可以是中央处理单元(Central Processing Unit，CPU)，该处理器80还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。The so-called processor 80 can be a central processing unit (Central Processing Unit, CPU), and the processor 80 can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit) , ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
所述存储器81在一些实施例中可以是所述机器人8的内部存储单元，例如机器人8的硬盘或内存。所述存储器81在另一些实施例中也可以是所述机器人8的外部存储设备，例如所述机器人8上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD)卡，闪存卡(Flash Card)等。进一步地，所述存储器81还可以既包括所述机器人8的内部存储单元也包括外部存储设备。所述存储器81用于存储操作***、应用程序、引导装载程序(Boot Loader)、数据以及其他程序等，例如所述计算机程序的程序代码等。所述存储器81还可以用于暂时地存储已经输出或者将要输出的数据。The memory 81 may be an internal storage unit of the robot 8 in some embodiments, such as a hard disk or memory of the robot 8 . The memory 81 can also be an external storage device of the robot 8 in other embodiments, such as a plug-in hard disk equipped on the robot 8, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital) Digital, SD) card, flash memory card (Flash Card), etc. Further, the memory 81 may also include both an internal storage unit of the robot 8 and an external storage device. The memory 81 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program. The memory 81 can also be used to temporarily store data that has been output or will be output.
本申请实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时实现可实现上述各个方法实施例中的步骤。The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in each of the foregoing method embodiments can be realized.
本申请实施例提供了一种计算机程序产品，当计算机程序产品在终端设备上运行时，使得终端设备执行时实现可实现上述各个方法实施例中的步骤。An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the foregoing method embodiments when executed.
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实现上述实施例方法中 的全部或部分流程，可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一计算机可读存储介质中，该计算机程序在被处理器执行时，可实现上述各个方法实施例的步骤。其中，所述计算机程序包括计算机程序代码，所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读介质至少可以包括：能够将计算机程序代码携带到装置/终端设备的任何实体或装置、记录介质、计算机存储器、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、电载波信号、电信信号以及软件分发介质。例如U盘、移动硬盘、磁碟或者光盘等。在某些司法管辖区，根据立法和专利实践，计算机可读介质不可以是电载波信号和电信信号。If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the procedures in the methods of the above embodiments in the present application can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a computer-readable storage medium. The computer program When executed by a processor, the steps in the above-mentioned various method embodiments can be realized. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program codes to a device/terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory ( RAM, Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media. Such as U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunication signals under legislation and patent practice.
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述或记载的部分，可以参见其它实施例的相关描述。In the above-mentioned embodiments, the descriptions of each embodiment have their own emphases, and for parts that are not detailed or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.
在本申请所提供的实施例中，应该理解到，所揭露的装置/机器人和方法，可以通过其它的方式实现。例如，以上所描述的装置/机器人实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个***，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通讯连接可以是通过一些接口，装置或单元的间接耦合或通讯连接，可以是电性，机械或其它的形式。In the embodiments provided in this application, it should be understood that the disclosed devices/robots and methods may be implemented in other ways. For example, the device/robot embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or Components may be combined or integrated into another system, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
以上所述实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围，均应包含在本申请的保护范围之内。The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still implement the foregoing Modifications to the technical solutions described in the examples, or equivalent replacement of some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the application, and should be included in the Within the protection scope of this application.

Claims (10)

  1. 一种机器人导航方法，其特征在于，应用于机器人，所述机器人上安装有拍摄装置，所述方法包括：A robot navigation method is characterized in that it is applied to a robot, and a camera is installed on the robot, and the method includes:
    检测拍摄图像中的目标行为，得到检测结果，所述拍摄图像由所述拍摄装置获取；Detecting the target behavior in the photographed image to obtain a detection result, the photographed image being acquired by the photographing device;
    如果所述检测结果表示所述拍摄图像中存在所述目标行为，则检测所述拍摄图像中所述目标行为对应的目标对象；If the detection result indicates that the target behavior exists in the captured image, detecting a target object corresponding to the target behavior in the captured image;
    从所述拍摄图像中获取所述目标对象的深度信息；acquiring depth information of the target object from the captured image;
    根据所述深度信息确定所述目标对象与所述机器人之间的实际距离；determining an actual distance between the target object and the robot according to the depth information;
    根据所述实际距离对所述机器人进行导航控制。Perform navigation control on the robot according to the actual distance.
  2. 如权利要求1所述的机器人导航方法，其特征在于，所述检测所述拍摄图像中的目标行为，得到检测结果，包括：The robot navigation method according to claim 1, wherein the detecting the target behavior in the captured image to obtain the detection result comprises:
    对所述拍摄图像进行关键点检测，得到关键点信息；Carrying out key point detection on the captured image to obtain key point information;
    根据所述关键点信息检测所述拍摄图像中的目标行为，得到所述检测结果。Detecting the target behavior in the captured image according to the key point information to obtain the detection result.
  3. 如权利要求2所述的机器人导航方法，其特征在于，所述关键点信息包括多个关键点在所述拍摄图像中各自的位置信息；The robot navigation method according to claim 2, wherein the key point information includes respective position information of a plurality of key points in the captured image;
    所述根据所述关键点信息检测所述拍摄图像中的目标行为，得到检测结果，包括：The detecting the target behavior in the captured image according to the key point information to obtain the detection result includes:
    根据所述多个关键点在所述拍摄图像中各自的位置信息，判断所述多个关键点中是否存在满足预设位置关系的第一关键点；According to the respective position information of the plurality of key points in the captured image, determine whether there is a first key point satisfying a preset position relationship among the plurality of key points;
    如果所述多个关键点中存在满足预设位置关系的第一关键点，则所述检测结果表示所述拍摄图像中存在所述目标行为。If there is a first key point satisfying a preset position relationship among the plurality of key points, the detection result indicates that the target behavior exists in the captured image.
  4. 如权利要求3所述的机器人导航方法，其特征在于，所述如果所述检测结果表示所述拍摄图像中存在所述目标行为，则检测所述拍摄图像中的目标行为对应的目标对象，包括：The robot navigation method according to claim 3, wherein if the detection result indicates that the target behavior exists in the captured image, detecting the target object corresponding to the target behavior in the captured image includes :
    如果所述检测结果表示所述拍摄图像中存在所述目标行为，则从所述多个关键点中获取与所述第一关键点相关联的第二关键点；If the detection result indicates that the target behavior exists in the captured image, obtaining a second key point associated with the first key point from the plurality of key points;
    将所述第二关键点确定为所述拍摄图像中的目标对象。The second key point is determined as a target object in the captured image.
  5. 如权利要求1或4所述的机器人导航方法，其特征在于，所述目标对象包括第二关键点；The robot navigation method according to claim 1 or 4, wherein the target object comprises a second key point;
    所述从所述拍摄图像中获取所述目标对象的深度信息，包括：The acquiring the depth information of the target object from the captured image includes:
    将所述拍摄图像转换为灰度图；converting the captured image into a grayscale image;
    获取所述灰度图中所述第二关键点对应的灰度值；Acquiring the gray value corresponding to the second key point in the gray image;
    根据所述第二关键点对应的灰度值计算所述目标对象的深度信息。calculating the depth information of the target object according to the gray value corresponding to the second key point.
  6. 如权利要求5所述的机器人导航方法，其特征在于，所述将所述拍摄图像转换为灰度图，包括：The robot navigation method according to claim 5, wherein said converting the captured image into a grayscale image comprises:
    将所述拍摄图像输入到训练后的深度估计模型中，输出所述灰度图；Inputting the captured image into the trained depth estimation model, and outputting the grayscale image;
    其中，所述深度估计模型包括编码层和解码层；所述编码层用于对所述拍摄图像进行特征提取，得到特征图；所述解码层用于对所述特征图进行上采样处理，得到所述灰度图。Wherein, the depth estimation model includes an encoding layer and a decoding layer; the encoding layer is used to perform feature extraction on the captured image to obtain a feature map; the decoding layer is used to perform upsampling processing on the feature map to obtain The grayscale image.
  7. 如权利要求5所述的机器人导航方法，其特征在于，所述获取所述灰度图中所述第二关键点对应的灰度值，包括：The robot navigation method according to claim 5, wherein said obtaining the gray value corresponding to the second key point in the gray image comprises:
    将所述第二关键点映射到预设坐标系，得到映射点；Mapping the second key point to a preset coordinate system to obtain a mapping point;
    将所述灰度图映射到所述预设坐标系，得到映射图；Mapping the grayscale image to the preset coordinate system to obtain a mapping image;
    从所述映射图中查找所述映射点对应的灰度值。Find the gray value corresponding to the mapping point from the mapping map.
  8. 如权利要求5所述的机器人导航方法，其特征在于，所述根据所述第二关键点对应的灰度值计算所述目标对象的深度信息，包括：The robot navigation method according to claim 5, wherein the calculating the depth information of the target object according to the gray value corresponding to the second key point comprises:
    从所述第二关键点中确定出目标关键点；determining a target key point from the second key point;
    计算所述目标关键点对应的灰度值的平均值；Calculate the average value of the gray value corresponding to the target key point;
    将所述平均值确定为所述目标对象的深度信息。The average value is determined as the depth information of the target object.
  9. 一种机器人，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现如权利要求1至8任一项所述的方法。A robot, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, characterized in that, when the processor executes the computer program, the computer program according to claims 1 to 8 is realized. any one of the methods described.
  10. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至8任一项所述的方法。A computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.
PCT/CN2021/126711 2021-06-30 2021-10-27 Robot navigation method, robot and computer-readable storage medium WO2023273056A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110736327.5A CN113515143B (en) 2021-06-30 Robot navigation method, robot and computer readable storage medium
CN202110736327.5 2021-06-30

Publications (1)

Publication Number Publication Date
WO2023273056A1 true WO2023273056A1 (en) 2023-01-05

Family

ID=78066123

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/126711 WO2023273056A1 (en) 2021-06-30 2021-10-27 Robot navigation method, robot and computer-readable storage medium

Country Status (1)

Country Link
WO (1) WO2023273056A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116412830A (en) * 2023-06-06 2023-07-11 深圳市磅旗科技智能发展有限公司 IHDR-based logistics robot self-adaptive navigation method and system

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150302239A1 (en) * 2012-11-27 2015-10-22 Sony Computer Entrtainment Inc. Information processor and information processing method
CN109341689A (en) * 2018-09-12 2019-02-15 北京工业大学 Vision navigation method of mobile robot based on deep learning
CN110210999A (en) * 2018-02-28 2019-09-06 阿里巴巴集团控股有限公司 Catering information processing method, apparatus and system
CN110710852A (en) * 2019-10-30 2020-01-21 广州铁路职业技术学院（广州铁路机械学校） Meal delivery method, system, medium and intelligent device based on meal delivery robot
CN111259839A (en) * 2020-01-20 2020-06-09 芯梯众和科技服务有限公司 Target object behavior monitoring method, device, equipment, system and storage medium
CN112753009A (en) * 2021-01-04 2021-05-04 华为技术有限公司 Man-machine interaction method and man-machine interaction device
CN113515143A (en) * 2021-06-30 2021-10-19 深圳市优必选科技股份有限公司 Robot navigation method, robot and computer readable storage medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150302239A1 (en) * 2012-11-27 2015-10-22 Sony Computer Entrtainment Inc. Information processor and information processing method
CN110210999A (en) * 2018-02-28 2019-09-06 阿里巴巴集团控股有限公司 Catering information processing method, apparatus and system
CN109341689A (en) * 2018-09-12 2019-02-15 北京工业大学 Vision navigation method of mobile robot based on deep learning
CN110710852A (en) * 2019-10-30 2020-01-21 广州铁路职业技术学院（广州铁路机械学校） Meal delivery method, system, medium and intelligent device based on meal delivery robot
CN111259839A (en) * 2020-01-20 2020-06-09 芯梯众和科技服务有限公司 Target object behavior monitoring method, device, equipment, system and storage medium
CN112753009A (en) * 2021-01-04 2021-05-04 华为技术有限公司 Man-machine interaction method and man-machine interaction device
CN113515143A (en) * 2021-06-30 2021-10-19 深圳市优必选科技股份有限公司 Robot navigation method, robot and computer readable storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116412830A (en) * 2023-06-06 2023-07-11 深圳市磅旗科技智能发展有限公司 IHDR-based logistics robot self-adaptive navigation method and system
CN116412830B (en) * 2023-06-06 2023-08-11 深圳市磅旗科技智能发展有限公司 IHDR-based logistics robot self-adaptive navigation method and system

Also Published As

Publication number Publication date
CN113515143A (en) 2021-10-19

Similar Documents

Publication Publication Date Title
WO2020259481A1 (en) Positioning method and apparatus, electronic device, and readable storage medium
US7733404B2 (en) Fast imaging system calibration
WO2020207190A1 (en) Three-dimensional information determination method, three-dimensional information determination device, and terminal apparatus
CN103189827B (en) Object display apparatus and object displaying method
CN112446919B (en) Object pose estimation method and device, electronic equipment and computer storage medium
US11940774B2 (en) Action imitation method and robot and computer readable storage medium using the same
KR101086068B1 (en) 3 dimensional marker detection method and device and method for providing augmented reality and mixed reality using the same
CN112506340B (en) Equipment control method, device, electronic equipment and storage medium
KR101165415B1 (en) Method for recognizing human face and recognizing apparatus
WO2021136386A1 (en) Data processing method, terminal, and server
CN112419388A (en) Depth detection method and device, electronic equipment and computer readable storage medium
WO2022099889A1 (en) Relocation method, robot, and computer-readable storage medium
CN111191582A (en) Three-dimensional target detection method, detection device, terminal device and computer-readable storage medium
CN116582653B (en) Intelligent video monitoring method and system based on multi-camera data fusion
WO2023273056A1 (en) Robot navigation method, robot and computer-readable storage medium
JP2015225353A (en) Image processor, image determination method, and program
JP5536124B2 (en) Image processing system and image processing method
US20230123879A1 (en) Method and apparatus for positioning express parcel
US20230048952A1 (en) Image registration method and electronic device
WO2022205841A1 (en) Robot navigation method and apparatus, and terminal device and computer-readable storage medium
CN113515143B (en) Robot navigation method, robot and computer readable storage medium
CN116311650A (en) Man-machine interaction method based on intelligent entrance guard's sight tracking and gesture recognition
KR102299902B1 (en) Apparatus for providing augmented reality and method therefor
WO2023272495A1 (en) Badging method and apparatus, badge detection model update method and system, and storage medium
JP2016045837A (en) Information processing apparatus, image determination method, and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21947983

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE