US20210227144A1 - Target tracking method and device, movable platform, and storage medium - Google Patents
Target tracking method and device, movable platform, and storage medium Download PDFInfo
- Publication number
- US20210227144A1 US20210227144A1 US17/222,627 US202117222627A US2021227144A1 US 20210227144 A1 US20210227144 A1 US 20210227144A1 US 202117222627 A US202117222627 A US 202117222627A US 2021227144 A1 US2021227144 A1 US 2021227144A1
- Authority
- US
- United States
- Prior art keywords
- image
- shooting
- target object
- shooting assembly
- position area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000003384 imaging method Methods 0.000 claims description 29
- 230000003287 optical effect Effects 0.000 claims description 24
- 238000003331 infrared imaging Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 6
- 230000000712 assembly Effects 0.000 abstract description 9
- 238000000429 assembly Methods 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 6
- 238000001931 thermography Methods 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 5
- 230000005855 radiation Effects 0.000 description 5
- 230000011218 segmentation Effects 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 241001465754 Metazoa Species 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- H04N5/23299—
-
- G06K9/0063—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/13—Satellite images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/17—Terrestrial scenes taken from planes or by drones
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
- H04N23/11—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/61—Control of cameras or camera modules based on recognised objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/28—Mobile studios
-
- G06K2009/00644—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30232—Surveillance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30241—Trajectory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/194—Terrestrial scenes using hyperspectral data, i.e. more or other wavelengths than RGB
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04B—TRANSMISSION
- H04B7/00—Radio transmission systems, i.e. using radiation field
- H04B7/14—Relay systems
- H04B7/15—Active relay systems
- H04B7/185—Space-based or airborne stations; Stations for satellite systems
- H04B7/18502—Airborne stations
- H04B7/18506—Communications with or from aircraft, i.e. aeronautical mobile service
Definitions
- the present disclosure generally relates to the field of image processing technology and, more particularly, to a target tracking method and device, a movable platform, and a storage medium.
- target tracking generally includes performing target recognition on an image captured by a shooting device to determine a target object in the image, and adjusting a shooting attitude of the shooting device to keep the target object in a shooting frame of the shooting device, to realize tracking of the target object in a monitoring environment.
- the tracking method used in current technologies is to perform the image recognition on the image shot by a visible light shooting device to determine the target object in the image, and adjust the shooting attitude of the visible light shooting device to keep the target object in the shooting frame of the visible light shooting device, to realize tracking of the target object by the visible light shooting device. Since the image captured by the visible light shooting device has rich feature information to facilitate an identification of the target object, the visible light shooting device can accurately track the target object. However, at present, the target object cannot or cannot accurately be recognized in the image output by certain types of shooting devices, and hence these types of shooting devices cannot or cannot accurately track the target object.
- a target tracking method applicable to a shooting device including a first shooting assembly and a second shooting assembly.
- the method includes calling the second shooting assembly to shoot an environment to obtain an image, performing target object recognition on the image to obtain a tracking position area of a target object to be tracked in the image, and adjusting a shooting attitude of the shooting device according to the tracking position area of the target object in the image to locate the target object in a shooting frame of the first shooting assembly.
- a movable platform including a shooting device including a first shooting assembly and a second shooting assembly, a memory storing a computer program including program instructions, and a processor. Imaging modes of the first shooting assembly and the second shooting assembly are different.
- the processor is configured to execute the program instructions to call the second shooting assembly to shoot the environment to obtain an image, perform target object recognition on the image to obtain a tracking position area of a target object to be tracked in the image, and adjust a shooting attitude of the shooting device according to the tracking position area of the target object in the image to locate the target object in a shooting frame of the first shooting assembly.
- FIG. 1 is an application scenario of a target tracking method consistent with embodiments of the disclosure.
- FIG. 2 is a schematic diagram of an imaging process of a shooting device consistent with embodiments of the disclosure.
- FIG. 3 is a flow chart of a target tracking method consistent with embodiments of the disclosure.
- FIG. 4 is a flow chart of another target tracking method consistent with embodiments of the disclosure.
- FIG. 5 is a schematic diagram of a target tracking device consistent with embodiments of the disclosure.
- FIG. 6 is a structural diagram of a movable platform consistent with embodiments of the disclosure.
- the present disclosure provides a target tracking method and device, a movable platform, and a storage medium, such that even if a target object cannot or cannot be accurately recognized in an image output by a shooting device, the shooting device can still track the target object.
- the present disclosure can realize target object recognition through a second image output by a second shooting assembly in the shooting device to obtain a tracking position area of a target object in the second image.
- a shooting attitude of the shooting device can be adjusted according to the tracking position area of the target object in the second image, such that the target object is located in a shooting frame of a first shooting assembly in the shooting device. As such, even if the target object cannot or cannot accurately be recognized in the image output by the first shooting assembly, the first shooting assembly can still track the target object.
- FIG. 1 is an application scenario of an example target tracking method consistent with the disclosure.
- the target tracking method can be applicable to a movable platform 01 .
- the movable platform 01 may be any apparatus that can be moved by an external force or through its own power system.
- the movable platform 01 may include an aircraft.
- the movable platform 01 includes the shooting device configured to shoot an environment, and the shooting device may be carried on a body of the movable platform 01 directly or through a movable component (e.g., a gimbal).
- the shooting device includes a first shooting assembly 0111 and a second shooting assembly 0112 having different imaging modes.
- the movable platform 01 can adjust an attitude of a body of the movable platform 01 or adjust an attitude of the movable component connected to the shooting device on the movable platform 01 (e.g., the gimbal), to adjust the shooting device to rotate up, down, left, and right and/or translate up, down, left, right, back, and forth, thereby adjusting a shooting attitude of the shooting device.
- the first shooting assembly 0111 and the second shooting assembly 0112 may be fixedly connected.
- the second shooting assembly 0112 of the shooting device can be called first to shoot the environment including the target object (e.g., people, animal, and the like) to obtain a second image.
- the second image may be any type of image that facilitates the target object recognition.
- the shooting attitude of the shooting device can be then controlled, such that the target object can be always in a shooting frame of the first shooting assembly 0111 . That is, the target object can be always within a shooting range of the first shooting assembly 0111 .
- the first shooting assembly 0111 can track the target object.
- the movable platform 01 can call the second shooting assembly 0112 to shoot the environment to obtain the second image.
- the target object recognition can be performed on the second image to obtain a tracking position area of the target object to be tracked in the second image.
- the shooting attitude of the shooting device can be adjusted according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly 0111 . Therefore, the present disclosure can use the second image to recognize the target object and the tracking position area of the target object in the second image, and adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly 0111 .
- the first shooting assembly can realize an indirect target tracking of the target object.
- the present disclose provides the target tracking method enabling the first shooting assembly 0111 to track the target object. Even if the target object cannot be accurately recognized in the image output by the first shooting assembly 0111 , the first shooting assembly 0111 can still track the target object.
- the shooting attitude of the shooting device can be adjusted directly based on the tracking position area of the target object in the second image.
- the target object can be in a preset position area of the shooting frame of the second shooting assembly 0112 .
- the target object can be in the target position area, which corresponds to the preset position area, in the shooting frame of the first shooting assembly 0111 .
- the target position area can be a central position area.
- a regional correspondence between the images captured by the first shooting assembly 0111 and the second shooting assembly 0112 needs to be determined.
- a position range of the target object in the image captured by the second shooting assembly 0112 can be used as the preset position area.
- a position range of the target object in the image captured by the second shooting assembly 0112 can be used as the preset position area.
- the tracking position area of the target object in the second image can be determined, and the shooting attitude of the shooting device can be adjusted directly based on the position of the target object in the second image.
- the target object can be in the preset position area of the image captured by the second shooting assembly 0112 . That is, the target object can be ensured to locate in the shooting frame of the first shooting assembly 0111 . It is further ensured that the target object can be located in the target position area, which corresponds to the preset position area, in the shooting frame of the first shooting assembly 0111 .
- the tracking position area of the target object in the shooting frame of the first shooting assembly 0111 can be determined.
- the shooting attitude of the shooting device can be adjusted according to the tracking position area of the target object in the shooting frame of the first shooting assembly 0111 , such that the target object is in the shooting frame of the first shooting assembly 0111 .
- the target object can be in the target position area in the shooting frame of the first shooting assembly 0111 .
- the tracking position area of the target object in the shooting frame of the first shooting assembly 0111 can be determined.
- the shooting attitude of the shooting device can be adjusted, such that the target object is located in the shooting frame of the first shooting assembly 0111 .
- the target object can be in the target position area in the shooting frame of the first shooting assembly 0111 .
- the tracking position area of the target object in the shooting frame of the first shooting assembly 0111 can be determined according to the tracking position area of the target object in the second image and the relative positional relationship between the first shooting assembly 0111 and the second shooting assembly 0112 .
- the shooting device can be then adjusted based on the tracking position area of the target object in the shooting frame of the first shooting assembly 0111 .
- the relative position between the first shooting assembly 0111 and the second shooting assembly 0112 may be fixed, or, the relative position may be not fixed but can be measured by a sensor in the movable platform 01 , and thus, the relative position between the first shooting assembly 0111 and the second shooting assembly 0112 can be known or knowable.
- the relative positional relationship between the first shooting assembly 0111 and the second shooting assembly 0112 i.e., a position conversion relationship between corresponding pixel points of a first image captured by the first shooting assembly 0111 and the second image captured by the second shooting assembly 0112 , can be determined according to relative positions between the first shooting assembly 0111 and the second shooting assembly 0112 .
- the present disclosure can quickly obtain the tracking position area of the target object in the shooting frame of the first shooting assembly 0111 according to the tracking position area of the target object in the second image.
- FIG. 2 is a schematic diagram of an imaging process of an example shooting device 011 consistent with the disclosure.
- the shooting device 011 will be described below with an example in which the relative position between the first shooting assembly 0111 and the second shooting assembly 0112 is fixed, the first shooting assembly 0111 can include a thermal infrared shooting device, a first image captured by the first shooting assembly 0111 can include a thermal infrared image and can be an image corresponding to the shooting frame of the first shooting assembly 0111 , the second shooting assembly 0112 can include a visible light shooting device, and a second image captured by the second shooting assembly 0112 can include an optical image.
- the shooting device 011 on the movable platform 01 includes a dual-light camera.
- the relative position between the first shooting assembly 0111 and the second shooting assembly 0112 in the dual-light camera can be fixed.
- the first shooting assembly 0111 can include the thermal infrared shooting device and the first image can include the thermal infrared image.
- the second shooting assembly 0112 can include the visible light shooting device and the second image can include the optical image.
- the first shooting assembly 0111 can shoot the environment in a thermal infrared imaging mode to obtain the thermal infrared image A1A2A3A4 as the first image
- the second shooting assembly 0112 can shoot the environment in a visible light imaging mode to obtain the optical image B1B2B3B4 as the second image.
- a first pixel on the first image A1A2A3A4 corresponds to a second pixel on the second image B1B2B3B4, i.e., the first pixel and the second pixel are imaged for a same target point on the target object.
- the relative position between the first shooting assembly 0111 and the second shooting assembly 0112 in the dual-light camera is fixed, external parameters of the two shooting assemblies in the dual-light camera can be known.
- the external parameters can be used to indicate the relative position of the two shooting assemblies, which can be determined according to a relative installation position and a relative installation angle of the two shooting assemblies.
- Internal parameters of the two shooting assemblies (determined according to a focal length of the shooting assembly and a position of an optical center) can be also known. Therefore, the second pixel in the second image can be easily projected to the first pixel in the first image. For example, assume that the coordinate of a target point on the target object in the camera coordinate system of the first shooting assembly 0111 are (X 1 , Y 1 , Z 1 ).
- X 1 , Y 1 , and Z 1 are a horizontal coordinate value, a vertical coordinate value, and a depth coordinate value.
- a relative positional offset between the second shooting assembly and the first shooting assembly can be ( ⁇ X, ⁇ Y, ⁇ Z), and thus, a coordinate of the target object in the camera coordinate system of the second shooting assembly 0112 can be (X 1 + ⁇ X, Y 1 + ⁇ Y, Z 1 + ⁇ Z).
- a coordinate of the first pixel of the target point of the target object in the first image captured by the first shooting assembly 0111 can be (u 1 , v 1 )
- a coordinate of the second pixel of the target point in the second image captured by the second shooting assembly 0112 can be (u 2 , v 2 ).
- the coordinate of the first pixel and the coordinate of the second pixel can have the following relative positional relationship:
- u 1 f 1 ⁇ X 1 Z 1
- v 1 f 1 ⁇ Y 1 Z 1
- u 2 f 2 ⁇ ( X 1 + ⁇ ⁇ ⁇ X ) Z 1 + ⁇ ⁇ ⁇ Z
- v 2 f 2 ⁇ ( Y 1 + ⁇ ⁇ ⁇ Y ) Z 1 + ⁇ ⁇ ⁇ Z ⁇ ⁇ u 1
- u 2 f 1 ⁇ X 1 f 2 ⁇ ( X 1 + ⁇ ⁇ ⁇ X ) * Z 1 + ⁇ ⁇ ⁇ Z Z 1 ⁇ f 1 f 2
- f 1 and f 2 are the focal lengths of the first shooting assembly 0111 and the second shooting assembly 0112 .
- An observation depth of the target object can be generally above 5 m, i.e., Z 1 >5 m, and the relative positional offset between the two shooting assemblies, ( ⁇ X, ⁇ Y, ⁇ Z), can be very small.
- the thermal infrared imaging mode can refer to that the thermal infrared shooting device can detect an infrared radiation emitted by the object itself, and convert a temperature distribution of the object into the thermal infrared image through photoelectric conversion, signal processing, and other means. Therefore, the first image A1A2A3A4 obtained by the thermal infrared mode can reflect the temperature distribution information of the object having thermal radiation.
- Objects having thermal radiation can include, for example, people, animals, electromagnetic equipment, and the like.
- the thermal imaging mode has the advantage of being able to shoot without light and with an occlusion. It can shoot objects well at night or in other special environments.
- the imaging principle of the visible light imaging mode can be to use a reflection of visible light from a surface of the object for imaging.
- the optical image B1B2B3B4 obtained by imaging can include detailed information such as the color and shape of the object, but the imaging result can be greatly affected by light and occlusion.
- both the first shooting assembly 0111 and the second shooting assembly 0112 can shoot the same target object in the environment. Because the imaging modes of the first shooting assembly 0111 and the second shooting assembly 0112 are different, sizes of the images obtained by the first shooting assembly 0111 and the second shooting assembly 0112 in the shooting environment may be different. For example, due to technical limitations, a range of infrared imaging can be smaller than that of visible light imaging. Generally, the image obtained by infrared imaging can be smaller than the image obtained by visible light imaging. The positions of the target object in the first image and the second image can be different. Since the relative positional relationship between the first shooting assembly 0111 and the second shooting assembly 0112 can be determined, once the position of the target object in one image is determined, the position of the target object in another image can be easily obtained through conversion.
- the thermal infrared image has insufficient texture information compared to the optical image. If the target recognition and tracking are directly performed on the thermal infrared image, the effect can be very poor. But thermal infrared images have the advantage that they can image objects with thermal radiation in the environment without being hindered by light and occlusion.
- the target tracking method based on the thermal infrared image can have very important practical significance.
- the present disclosure can use the optical image output by the visible light shooting device to realize the target object recognition to obtain the tracking position area of the target object in the optical image.
- the shooting attitude of the shooting device can be adjusted according to the tracking position area of the target object in the optical image, such that the target object is located in the shooting frame of the thermal infrared shooting device in the shooting device. As such, even if the thermal infrared image output by the thermal infrared camera cannot or cannot accurately recognize the target object, the thermal infrared camera can still track the target object.
- the user can specify the target object to be tracked based on the first image output by the first shooting assembly 0111 , so as to achieve the tracking of the target object.
- the movable platform 01 can send the first image to a control terminal 02 to cause the control terminal 02 to display the first image.
- the user can perform a selection operation on the first image displayed on the control terminal 02 , for example, frame an area including the target object to be tracked on the first image.
- the control terminal 02 can generate first area indication information according to the area including the target object to be tracked, and send the first area indication information to the movable platform 01 .
- the movable platform 01 After receiving the first area indication information sent by the control terminal 02 , the movable platform 01 can determine second area indication information of the second image according to the first area indication information and the relative positional relationship between the first shooting assembly 0111 and the second shooting assembly 0112 .
- the movable platform 01 can perform the target recognition on the area indicated by the second area indication information in the second image to determine the target object and obtain the tracking position area of the target object in the second image.
- the target recognition can be performed on the area indicated by the second area indication information in the second image, and the target object can be recognized in the area.
- the recognition may be recognized through a neural network.
- the determined target object may be recognized from the second image output by the second shooting assembly 0112 to obtain the tracking position area of the target object in the second image.
- the recognition of the determined target object from the second image output by the second shooting assembly 0112 may be recognized using the neural network, or by image tracking.
- the user can view target tracking result based on the first image at the control terminal 02 .
- the tracking position area of the target object in the second image can be determined according to the tracking position area of the target object in the second image and the relative positional relationship between the first shooting assembly 0111 and the second shooting assembly 0112 .
- the target object can be marked in the first image according to the tracking position area of the target object in the first image. All or some of image information in the second image information can be added to the marked first image to enrich contour features of the marked first image.
- the marked first image can be sent to the control terminal 02 , such that the control terminal 02 can display the marked first image.
- FIG. 3 is a flow chart of an example target tracking method consistent with the disclosure.
- the target tracking method may be executed by a movable platform, for example, the movable platform 01 in FIG. 1 .
- the movable platform calls the second shooting assembly to shoot the environment to obtain the second image, and calls the first shooting assembly to shoot the environment to obtain the first image at the same time.
- the first shooting assembly can include the thermal infrared imaging device and the second shooting assembly can include the visible light imaging device.
- the imaging modes of the first shooting assembly and the second shooting assembly can be different.
- the first shooting assembly can use the thermal infrared imaging mode to obtain the thermal infrared image as the first image
- the second shooting assembly can use the visible light imaging mode to obtain the optical image as the second image.
- the second image can include the optical image.
- the positions of the target object in the first image and the second image may be different. Since the relative positional relationship between the first shooting assembly and the second shooting assembly can be determined, once the position of the target object in one image is determined, the position of the target object in another image can be easily obtained through conversion.
- target object recognition is performed on the second image to recognize the target object in the second image, and the tracking position area of the target object to be tracked in the second image is obtained using segmentation.
- Target object recognition can determine the target object and the tracking position area of the target object in the second image through image processing methods of target detection and target segmentation.
- the target detection and target segmentation may include a conventional target detection method and a conventional segmentation method, or may include a target detection method and a target segmentation method based on deep learning (e.g., a neural network), which is not limited in here.
- the shooting attitude of the shooting device is adjusted according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly.
- the movable platform can adjust the shooting attitude of the shooting device by changing the attitude of the movable platform itself (e.g., the body), or control the shooting device to adjust the shooting attitude through the gimbal connected to the shooting device, i.e., adjust the shooting attitude of the shooting device by adjusting the attitude of the gimbal.
- adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the second image may also include determining the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image, and adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the shooting frame of the first shooting assembly.
- the thermal infrared image has insufficient texture information compared to the optical image. If the target recognition and tracking are directly performed on the thermal infrared image, the effect can be very poor. But thermal infrared images have the advantage that they can image objects having thermal radiation in the environment without being hindered by light and occlusion. Therefore, the target tracking method based on thermal infrared images can be very important.
- the present disclosure can solve the problem very well, because the present disclosure can use the optical image output by the visible light shooting device to recognize the target object to obtain the tracking position area of the target object in the optical image.
- the shooting attitude of the shooting device can be adjusted according to the tracking position area of the target object in the optical image, such that the target object is located in the shooting frame of the thermal infrared shooting device in the shooting device. In this way, even if the target object cannot be or cannot be accurately recognized in the thermal infrared image output by the thermal infrared shooting device, the thermal infrared shooting device can still track the target object.
- the first image can be first sent to the control terminal of the movable platform, such that the control terminal can display the first image.
- the user can perform the selection operation on the first image on the control terminal (e.g., frame the area where the target object is located), and then the terminal device can obtain the first area indication information for indicating the selected area of the user in the first image according to the selection operation, and send the first area indication information to the movable platform.
- the movable platform receives the first area indication information, the processes described above can be used to determine the position of the target object in the first image according to the tracking position area of the target object in the second image.
- the second area indication information can be determined according to the first area indication information and the relative positional relationship between the first shooting assembly and the second shooting assembly.
- the second area indication information can be used to indicate the area in the first image mapped from the area selected by the user in the second image, and finally, the target recognition can be performed on the area indicated by the second area indication information in the second image to determine the target object, and obtain the tracking position area of the target object in the second image.
- the user can specify the target object, and a target tracking efficiency of the target tracking can be improved.
- the tracking position area of the target object in the first image can be determined according to the tracking position area of the target object in the second image.
- the target object can be then marked in the first image (e.g., as shown in the first image in FIG. 2 , the location of the target object can be framed).
- the marked first image can be sent to the control terminal of the movable platform to cause the control terminal to display the marked first image. Determining the tracking position area of the target object in the first image according to the tracking position area of the target object in the second image has been described in connection with FIGS. 1 and 2 , and detailed description thereof is omitted herein.
- the present disclosure can determine the target object in the tracking position area of the second image by performing the target object recognition on the second image.
- the tracking position area of the target object in the first image can be determined according to the tracking position area of the target object in the second image, and the target object can be marked in the first image, and the marked target object can be showed to the user through the control terminal. Therefore, indirect target tracking based on the first image can be realized, especially when the first image is the thermal infrared image.
- the present disclosure can achieve the target tracking based on the thermal infrared image, which has very important practical value.
- all or some of the image information in the second image information can be first extracted, and all or some of the image information in the second image information can be added to the marked first image to enrich the contour features of the marked first image.
- the marked first image can be sent to the control terminal, such that the first image finally presented to the user on the control terminal not only can be marked with the target object, but also can have greatly enriched details.
- a defect that the details of the image, e.g., the thermal imaging image or the like, are not rich can be improved to a certain extent.
- the present disclosure can achieve not only the target tracking based on the thermal imaging image, but also an interaction with the user through the thermal imaging image, and use the details of the optical image to enrich the contour details of the thermal imaging image that is not rich in details.
- the practicality of thermal imaging images can be greatly improved.
- FIG. 4 is a flow chart of another example target tracking method consistent with the disclosure, and the target tracking method may be executed by a movable platform, e.g., the movable platform 01 in FIG. 1 .
- the first shooting assembly and the second shooting assembly of the shooting device of the movable platform are called to shoot the environment to obtain the first image and the second image.
- the first shooting assembly can include the thermal infrared imaging device and the second shooting assembly can include the visible light imaging device.
- the imaging modes of the first shooting assembly and the second shooting assembly can be different.
- the first shooting assembly can use the thermal infrared imaging mode to obtain the thermal infrared image as the first image
- the second shooting assembly can use the visible light imaging mode to obtain the optical image as the second image.
- the second image can include the optical image.
- the first image is sent to the control terminal of the movable platform to cause the control terminal to display the first image.
- the target object to be tracked in the second image is determined, and the tracking position area of the target object in the second image is obtained.
- the second area indication information can be determined according to the first area indication information and the relative positional relationship between the first shooting assembly and the second shooting assembly.
- the target recognition can be performed on the area indicated by the second area indication information in the second image to determine the target object, and obtain the tracking position area of the target object in the second image.
- the shooting attitude of the shooting device is adjusted according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly.
- the movable platform can adjust the shooting attitude of the shooting device by changing the attitude of the movable platform itself, or control the shooting device to adjust the shooting attitude through the gimbal connected to the shooting device, i.e., adjust the shooting attitude of the shooting device by adjusting the attitude of the gimbal.
- adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the second image may include adjusting the target object to the preset position area of the image captured by the second shooting assembly, such that the target object is in the shooting frame of the first shooting assembly, and furthermore, can be in the target position area of the shooting frame.
- adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the second image may include determining the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image, and the relative positional relationship between the first shooting assembly and the second shooting assembly, and adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the shooting frame of the first shooting assembly, such that the target object is in the shooting frame of the first shooting assembly, furthermore, can be in the target position area of the shooting frame.
- the tracking position area of the target object in the shooting frame of the first shooting assembly is determined. For example, according to the tracking position area of the target object in the second image and the relative positional relationship between the first shooting assembly and the second shooting assembly, the tracking position area of the target object in the first image can be determined.
- the target object is marked in the first image.
- the detailed information in the second image is extracted to enrich the contour features in the first image.
- the image information in the second image information can be first extracted, and all or some of the image information in the second image information can be added to the marked first image to enrich the contour features of the marked first image.
- the marked first image is sent to the control terminal of the movable platform to cause the control terminal to display the marked first image.
- the first image can be also referred to as a “displaying image,” and the second image can be also referred to as a “tracking image.”
- FIG. 5 is a schematic diagram of an example target tracking device consistent with the disclosure.
- the target tracking device can implement the target tracking method in FIGS. 3 and 4 .
- the target tracking device can be applied to a movable platform, for example, the movable platform 01 in FIG. 1 .
- the movable platform can include the shooting device, and the shooting device can include the first shooting assembly and the second shooting assembly.
- the target tracking device includes a calling circuit 510 , a recognition circuit 520 , and a tracking circuit 530 .
- the calling circuit 510 can be configured to call the second shooting assembly to shoot the environment to obtain the second image, and the imaging modes of the first shooting assembly and the second shooting assembly can be different.
- the recognition circuit 520 can be configured to perform the target object recognition on the second image to obtain the tracking position area of the target object to be tracked in the second image.
- the tracking circuit 530 can be configured to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly.
- the movable platform can include the gimbal carrying the shooting device
- the tracking unit 530 can be further configured to adjust the attitude of the movable platform and/or the attitude of the gimbal to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image.
- the first shooting assembly can include the thermal infrared imaging device
- the second shooting assembly can include the visible light imaging device
- the second image can include the optical image
- the tracking unit 530 can be specifically configured to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image. After the shooting attitude of the shooting device is adjusted, the target object can be located in the preset position area of the shooting frame of the second shooting assembly.
- the target tracking device can further include a determination circuit 540 configured to determine the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image.
- the tracking unit 530 can be further configured to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the shooting frame of the first shooting assembly.
- the calling unit 510 can be further configured to call the first shooting assembly to shoot the environment to obtain the first image.
- the target tracking device can further include a sending unit 550 configured to send the first image to the control terminal of the movable platform to cause the control terminal to display the first image.
- the target tracking device can further include an acquisition unit 560 configured to obtain the first area indication information sent by the control terminal.
- the first area indication information can be determined by the control terminal by detecting the user's selection operation of the target object on the first image displayed by the control terminal.
- the recognition unit 520 can be further configured to determine the target object to be tracked in the second image and obtain the tracking position area of the target object in the second image according to the first area indication information.
- the recognition unit 520 can be further configured to determine the second area indication information of the second image according to the first area indication information and the relative positional relationship between the first shooting assembly and the second shooting assembly, and perform the target recognition on the area indicated by the second area indication information in the second image to determine the target object and obtain the tracking position area of the target object in the second image.
- the calling unit 510 can be further configured to call the first shooting assembly to shoot the environment to obtain the first image.
- the determination unit 540 can be further configured to determine the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image.
- the target tracking device can further include a marking unit 570 configured to mark the target object in the first image according to the tracking position area of the target object in the shooting frame of the first shooting assembly.
- the sending unit 550 can be further configured to send the marked first image to the control terminal of the movable platform, such that the control terminal can display the marked first image.
- the target tracking device can further include an extraction unit 580 configured to extract all or some of the image information in the second image information.
- the target tracking device can further include an adding unit 590 configured to add all or some of the image information in the second image information to the marked first image to enrich the contour features of the marked first image.
- the determination unit 540 can be further configured to determine the tracking position area of the target object in the first image according to the tracking position area of the target object in the second image and the relative positional relationship between the first shooting assembly and the second shooting assembly.
- FIG. 6 is a structural diagram of an example movable platform consistent with the disclosure.
- the movable platform may at least include a processor 610 , a memory 620 , and a shooting device 630 .
- the shooting device includes a first shooting assembly 631 and a second shooting assembly 632 .
- the processor 610 , the memory 620 , and the shooting device 630 may be connected through a bus 640 or other means.
- FIG. 6 takes a connection through the bus as an example.
- the memory 620 may be configured to store the computer program.
- the computer program can include the program instructions.
- the processor 610 may be configured to execute the program instructions stored in the memory 620 .
- the processor 610 may be a central processing unit (CPU), and the processor may also be another general-purpose processor, i.e., a microprocessor or any conventional processor, such as a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, and the like.
- DSP digital signal processor
- ASIC Application Specific Integrated Circuit
- FPGA Field-Programmable Gate Array
- the memory 620 may include a read-only memory and a random access memory, and provide instructions and data to the processor 610 . Therefore, the processor 610 and the memory 620 are not limited herein.
- the processor 610 can load and execute one or more instructions stored in the computer storage medium to implement some or all processes of the methods in FIGS. 3 and 4 .
- at least one instruction in the computer storage medium can be loaded and executed by the processor 610 .
- the shooting device 630 can be configured to shoot the environment.
- the memory 620 can be configured to store the computer program and the computer program can include the program instructions.
- the processor 610 can be configured to call the program instructions and configured to call the second shooting assembly to shoot the environment to obtain the second image, the imaging modes of the first shooting assembly and the second shooting assembly being different, perform the target object recognition on the second image to obtain the tracking position area of the target object to be tracked in the second image, adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly.
- the processor 610 can be specifically configured to adjust the attitude of the movable platform and/or the attitude of the gimbal to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image.
- the first shooting assembly can include the thermal infrared imaging device
- the second shooting assembly can include the visible light imaging device
- the second image can include the optical image
- the processor 610 can be further configured to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image. After the shooting attitude of the shooting device is adjusted, the target object can be located in the preset position area of the shooting frame of the second shooting assembly.
- the processor can be further configured to determine the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image, and adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the shooting frame of the first shooting assembly.
- the processor can be further configured to call the first shooting assembly to shoot the environment to obtain the first image.
- the movable platform further includes a communication interface 650 for data interaction between the movable platform and other terminal devices, and configured to send the first image to the control terminal of the movable platform to cause the control terminal to display the first image.
- the processor 610 can be further configured to obtain the first area indication information sent by the control terminal, the first area indication information being determined by the control terminal by detecting the user's selection operation of the target object on the first image displayed by the control terminal, and determine the target object to be tracked in the second image and obtain the tracking position area of the target object in the second image according to the first area indication information.
- the processor 610 can be further configured to determine the second area indication information of the second image according to the first area indication information and the relative positional relationship between the first shooting assembly and the second shooting assembly, and perform the target recognition on the area indicated by the second area indication information in the second image to determine the target object and obtain the tracking position area of the target object in the second image.
- the processor 610 can be further configured to call the first shooting assembly to shoot the environment to obtain the first image.
- the processor 610 can be further configured to determine the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image.
- the processor 610 can be further configured to mark the target object in the first image according to the tracking position area of the target object in the shooting frame of the first shooting assembly.
- the communication interface 650 can be configured to send the marked first image to the control terminal of the movable platform, such that the control terminal can display the marked first image.
- the processor 610 can be further configured to extract all or some of the image information in the second image information, and add all or some of the image information in the second image information to the marked first image to enrich the contour features of the marked first image.
- the processor 610 can be further configured to determine the tracking position area of the target object in the first image according to the tracking position area of the target object in the second image and the relative positional relationship between the first shooting assembly and the second shooting assembly.
- Embodiments of the present disclosure also provide a control device.
- the control device can be in communication connection with the shooting device, and the shooting device can include the first shooting assembly and the second shooting assembly.
- the control device is characterized in that the control device includes a memory and a processor.
- the memory may be configured to store the computer program.
- the computer program can include the program instructions.
- the processor may be configured to call the program instructions and configured to call the second shooting assembly to shoot the environment to obtain the second image, the imaging modes of the first shooting assembly and the second shooting assembly being different, perform the target object recognition on the second image to obtain the tracking position area of the target object to be tracked in the second image, and adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly.
- the control device can be provided in the movable platform, and the movable platform can include the control device and the shooting device.
- the control device may be communicatively connected to the shooting device.
- the processor of the control device can execute the methods in FIGS. 3 and 4 . For the details, reference may be made to the previous sections, and will not be repeated herein.
- the integrated units can be stored in a computer readable storage medium when implemented in form of software functional units and sold or used as a standalone product. Based on such understanding, all or part of the technical solution of the disclosure can be embodied in the form of software product stored in a storage medium comprising a number of instructions for causing a computer processor to perform the entire or part of a method consistent with embodiments of the disclosure, such as one of the above-described exemplary methods.
- the storage medium can comprise a flask disk, a portable hard drive, a read only memory (ROM), a random access memory (RAM), a magnet disk, an optical disk, or other media capable of storing program code.
- the program may be stored in a computer-readable storage medium, and when the program is executed, it may include the processes of the foregoing method embodiments.
- the storage medium may include a magnetic disk, an optical disk, a ROM or RAM, or the like.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Remote Sensing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Astronomy & Astrophysics (AREA)
- Human Computer Interaction (AREA)
- Studio Devices (AREA)
Abstract
Description
- This application is a continuation of U.S. application Ser. No. 16/880,553, filed on May 21, 2020, which is a continuation of International Application No. PCT/CN2019/089248, filed May 30, 2019, the entire contents of both of which are incorporated herein by reference.
- The present disclosure generally relates to the field of image processing technology and, more particularly, to a target tracking method and device, a movable platform, and a storage medium.
- Currently, target tracking generally includes performing target recognition on an image captured by a shooting device to determine a target object in the image, and adjusting a shooting attitude of the shooting device to keep the target object in a shooting frame of the shooting device, to realize tracking of the target object in a monitoring environment.
- The tracking method used in current technologies is to perform the image recognition on the image shot by a visible light shooting device to determine the target object in the image, and adjust the shooting attitude of the visible light shooting device to keep the target object in the shooting frame of the visible light shooting device, to realize tracking of the target object by the visible light shooting device. Since the image captured by the visible light shooting device has rich feature information to facilitate an identification of the target object, the visible light shooting device can accurately track the target object. However, at present, the target object cannot or cannot accurately be recognized in the image output by certain types of shooting devices, and hence these types of shooting devices cannot or cannot accurately track the target object.
- Consistent with the disclosure, there is provided a target tracking method applicable to a shooting device including a first shooting assembly and a second shooting assembly. The method includes calling the second shooting assembly to shoot an environment to obtain an image, performing target object recognition on the image to obtain a tracking position area of a target object to be tracked in the image, and adjusting a shooting attitude of the shooting device according to the tracking position area of the target object in the image to locate the target object in a shooting frame of the first shooting assembly.
- Also consistent with the disclosure, there is provided a movable platform including a shooting device including a first shooting assembly and a second shooting assembly, a memory storing a computer program including program instructions, and a processor. Imaging modes of the first shooting assembly and the second shooting assembly are different. The processor is configured to execute the program instructions to call the second shooting assembly to shoot the environment to obtain an image, perform target object recognition on the image to obtain a tracking position area of a target object to be tracked in the image, and adjust a shooting attitude of the shooting device according to the tracking position area of the target object in the image to locate the target object in a shooting frame of the first shooting assembly.
- In order to provide a clearer illustration of technical solutions of disclosed embodiments, the drawings used in the description of the disclosed are briefly described below.
-
FIG. 1 is an application scenario of a target tracking method consistent with embodiments of the disclosure. -
FIG. 2 is a schematic diagram of an imaging process of a shooting device consistent with embodiments of the disclosure. -
FIG. 3 is a flow chart of a target tracking method consistent with embodiments of the disclosure. -
FIG. 4 is a flow chart of another target tracking method consistent with embodiments of the disclosure. -
FIG. 5 is a schematic diagram of a target tracking device consistent with embodiments of the disclosure. -
FIG. 6 is a structural diagram of a movable platform consistent with embodiments of the disclosure. - The present disclosure provides a target tracking method and device, a movable platform, and a storage medium, such that even if a target object cannot or cannot be accurately recognized in an image output by a shooting device, the shooting device can still track the target object.
- The present disclosure can realize target object recognition through a second image output by a second shooting assembly in the shooting device to obtain a tracking position area of a target object in the second image. A shooting attitude of the shooting device can be adjusted according to the tracking position area of the target object in the second image, such that the target object is located in a shooting frame of a first shooting assembly in the shooting device. As such, even if the target object cannot or cannot accurately be recognized in the image output by the first shooting assembly, the first shooting assembly can still track the target object.
-
FIG. 1 is an application scenario of an example target tracking method consistent with the disclosure. The target tracking method can be applicable to amovable platform 01. Themovable platform 01 may be any apparatus that can be moved by an external force or through its own power system. For example, themovable platform 01 may include an aircraft. As shown inFIG. 1 , themovable platform 01 includes the shooting device configured to shoot an environment, and the shooting device may be carried on a body of themovable platform 01 directly or through a movable component (e.g., a gimbal). The shooting device includes afirst shooting assembly 0111 and asecond shooting assembly 0112 having different imaging modes. Themovable platform 01 can adjust an attitude of a body of themovable platform 01 or adjust an attitude of the movable component connected to the shooting device on the movable platform 01 (e.g., the gimbal), to adjust the shooting device to rotate up, down, left, and right and/or translate up, down, left, right, back, and forth, thereby adjusting a shooting attitude of the shooting device. Thefirst shooting assembly 0111 and thesecond shooting assembly 0112 may be fixedly connected. When themovable platform 01 performs target tracking on a target object, thesecond shooting assembly 0112 of the shooting device can be called first to shoot the environment including the target object (e.g., people, animal, and the like) to obtain a second image. The second image may be any type of image that facilitates the target object recognition. According to a position area of the target object in the second image, the shooting attitude of the shooting device can be then controlled, such that the target object can be always in a shooting frame of thefirst shooting assembly 0111. That is, the target object can be always within a shooting range of thefirst shooting assembly 0111. Thus, thefirst shooting assembly 0111 can track the target object. For example, themovable platform 01 can call thesecond shooting assembly 0112 to shoot the environment to obtain the second image. The target object recognition can be performed on the second image to obtain a tracking position area of the target object to be tracked in the second image. The shooting attitude of the shooting device can be adjusted according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of thefirst shooting assembly 0111. Therefore, the present disclosure can use the second image to recognize the target object and the tracking position area of the target object in the second image, and adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of thefirst shooting assembly 0111. Thus, the first shooting assembly can realize an indirect target tracking of the target object. The present disclose provides the target tracking method enabling thefirst shooting assembly 0111 to track the target object. Even if the target object cannot be accurately recognized in the image output by thefirst shooting assembly 0111, thefirst shooting assembly 0111 can still track the target object. - For example, there are two adjustment methods for adjusting the shooting attitude of the shooting device. In some embodiments, the shooting attitude of the shooting device can be adjusted directly based on the tracking position area of the target object in the second image. After the shooting attitude of the shooting device is adjusted, the target object can be in a preset position area of the shooting frame of the
second shooting assembly 0112. When the target object is in the preset position area of the shooting frame of thesecond shooting assembly 0112, the target object can be in the target position area, which corresponds to the preset position area, in the shooting frame of thefirst shooting assembly 0111. The target position area can be a central position area. - Before adjusting the shooting attitude of the shooting device based on the tracking position area of the target object in the second image, a regional correspondence between the images captured by the
first shooting assembly 0111 and thesecond shooting assembly 0112 needs to be determined. For example, when the target object is maintained in the image captured by thefirst shooting assembly 0111, a position range of the target object in the image captured by thesecond shooting assembly 0112 can be used as the preset position area. In some embodiments, when the target object is maintained in the image captured by thefirst shooting assembly 0111, a position range of the target object in the image captured by thesecond shooting assembly 0112 can be used as the preset position area. Therefore, the tracking position area of the target object in the second image can be determined, and the shooting attitude of the shooting device can be adjusted directly based on the position of the target object in the second image. As such, the target object can be in the preset position area of the image captured by thesecond shooting assembly 0112. That is, the target object can be ensured to locate in the shooting frame of thefirst shooting assembly 0111. It is further ensured that the target object can be located in the target position area, which corresponds to the preset position area, in the shooting frame of thefirst shooting assembly 0111. - In some other embodiments, according to the tracking position area of the target object in the second image, the tracking position area of the target object in the shooting frame of the
first shooting assembly 0111 can be determined. The shooting attitude of the shooting device can be adjusted according to the tracking position area of the target object in the shooting frame of thefirst shooting assembly 0111, such that the target object is in the shooting frame of thefirst shooting assembly 0111. Furthermore, the target object can be in the target position area in the shooting frame of thefirst shooting assembly 0111. For example, according to the tracking position area of the target object in the second image and a relative positional relationship between thefirst shooting assembly 0111 and thesecond shooting assembly 0112, the tracking position area of the target object in the shooting frame of thefirst shooting assembly 0111 can be determined. According to the tracking position area of the target object in the shooting frame of thefirst shooting assembly 0111, the shooting attitude of the shooting device can be adjusted, such that the target object is located in the shooting frame of thefirst shooting assembly 0111. For example, the target object can be in the target position area in the shooting frame of thefirst shooting assembly 0111. - Therefore, the tracking position area of the target object in the shooting frame of the
first shooting assembly 0111 can be determined according to the tracking position area of the target object in the second image and the relative positional relationship between thefirst shooting assembly 0111 and thesecond shooting assembly 0112. The shooting device can be then adjusted based on the tracking position area of the target object in the shooting frame of thefirst shooting assembly 0111. For example, because the relative position between thefirst shooting assembly 0111 and thesecond shooting assembly 0112 may be fixed, or, the relative position may be not fixed but can be measured by a sensor in themovable platform 01, and thus, the relative position between thefirst shooting assembly 0111 and thesecond shooting assembly 0112 can be known or knowable. The relative positional relationship between thefirst shooting assembly 0111 and thesecond shooting assembly 0112, i.e., a position conversion relationship between corresponding pixel points of a first image captured by thefirst shooting assembly 0111 and the second image captured by thesecond shooting assembly 0112, can be determined according to relative positions between thefirst shooting assembly 0111 and thesecond shooting assembly 0112. As such, the present disclosure can quickly obtain the tracking position area of the target object in the shooting frame of thefirst shooting assembly 0111 according to the tracking position area of the target object in the second image. -
FIG. 2 is a schematic diagram of an imaging process of anexample shooting device 011 consistent with the disclosure. As shown inFIG. 2 , theshooting device 011 will be described below with an example in which the relative position between thefirst shooting assembly 0111 and thesecond shooting assembly 0112 is fixed, thefirst shooting assembly 0111 can include a thermal infrared shooting device, a first image captured by thefirst shooting assembly 0111 can include a thermal infrared image and can be an image corresponding to the shooting frame of thefirst shooting assembly 0111, thesecond shooting assembly 0112 can include a visible light shooting device, and a second image captured by thesecond shooting assembly 0112 can include an optical image. - In one embodiment, as shown in
FIG. 2 , theshooting device 011 on themovable platform 01 includes a dual-light camera. The relative position between thefirst shooting assembly 0111 and thesecond shooting assembly 0112 in the dual-light camera can be fixed. Thefirst shooting assembly 0111 can include the thermal infrared shooting device and the first image can include the thermal infrared image. Thesecond shooting assembly 0112 can include the visible light shooting device and the second image can include the optical image. - When the
movable platform 01 calls theshooting device 011 to shoot the environment including the target object (e.g., one of two people), thefirst shooting assembly 0111 can shoot the environment in a thermal infrared imaging mode to obtain the thermal infrared image A1A2A3A4 as the first image, and thesecond shooting assembly 0112 can shoot the environment in a visible light imaging mode to obtain the optical image B1B2B3B4 as the second image. Assume that a first pixel on the first image A1A2A3A4 corresponds to a second pixel on the second image B1B2B3B4, i.e., the first pixel and the second pixel are imaged for a same target point on the target object. Since the relative position between thefirst shooting assembly 0111 and thesecond shooting assembly 0112 in the dual-light camera is fixed, external parameters of the two shooting assemblies in the dual-light camera can be known. The external parameters can be used to indicate the relative position of the two shooting assemblies, which can be determined according to a relative installation position and a relative installation angle of the two shooting assemblies. Internal parameters of the two shooting assemblies (determined according to a focal length of the shooting assembly and a position of an optical center) can be also known. Therefore, the second pixel in the second image can be easily projected to the first pixel in the first image. For example, assume that the coordinate of a target point on the target object in the camera coordinate system of thefirst shooting assembly 0111 are (X1, Y1, Z1). X1, Y1, and Z1 are a horizontal coordinate value, a vertical coordinate value, and a depth coordinate value. A relative positional offset between the second shooting assembly and the first shooting assembly can be (ΔX, ΔY, ΔZ), and thus, a coordinate of the target object in the camera coordinate system of thesecond shooting assembly 0112 can be (X1+ΔX, Y1+ΔY, Z1+ΔZ). A coordinate of the first pixel of the target point of the target object in the first image captured by thefirst shooting assembly 0111 can be (u1, v1), and a coordinate of the second pixel of the target point in the second image captured by thesecond shooting assembly 0112 can be (u2, v2). The coordinate of the first pixel and the coordinate of the second pixel can have the following relative positional relationship: -
- where f1 and f2 are the focal lengths of the
first shooting assembly 0111 and thesecond shooting assembly 0112. - An observation depth of the target object can be generally above 5 m, i.e., Z1>5 m, and the relative positional offset between the two shooting assemblies, (ΔX, ΔY, ΔZ), can be very small. For example, a Euclidean distance d between the two shooting assemblies can be d=√{square root over (ΔX2+ΔY2+ΔZ2)}≈0.02 m. Therefore, Z1>>ΔX, ΔY, ΔZ and the relative position offset can be ignored, and the relative positional relationship between the first pixel in the first image and the second pixel in the second image can be obtained as u1/u2=f1/f2 and v1/v2=f1/f2. It can be seen that, according to the position of the second pixel in the second image and the relative positional relationship between the first shooting assembly and the second shooting assembly, the position of the first pixel in the first image can be easily obtained through conversion.
- The thermal infrared imaging mode can refer to that the thermal infrared shooting device can detect an infrared radiation emitted by the object itself, and convert a temperature distribution of the object into the thermal infrared image through photoelectric conversion, signal processing, and other means. Therefore, the first image A1A2A3A4 obtained by the thermal infrared mode can reflect the temperature distribution information of the object having thermal radiation. Objects having thermal radiation can include, for example, people, animals, electromagnetic equipment, and the like. The thermal imaging mode has the advantage of being able to shoot without light and with an occlusion. It can shoot objects well at night or in other special environments. The imaging principle of the visible light imaging mode, e.g., red-green-blue (RGB) imaging mode and the like, can be to use a reflection of visible light from a surface of the object for imaging. The optical image B1B2B3B4 obtained by imaging can include detailed information such as the color and shape of the object, but the imaging result can be greatly affected by light and occlusion.
- Although the relative position of the
first shooting assembly 0111 and thesecond shooting assembly 0112 may or may not be fixed, both thefirst shooting assembly 0111 and thesecond shooting assembly 0112 can shoot the same target object in the environment. Because the imaging modes of thefirst shooting assembly 0111 and thesecond shooting assembly 0112 are different, sizes of the images obtained by thefirst shooting assembly 0111 and thesecond shooting assembly 0112 in the shooting environment may be different. For example, due to technical limitations, a range of infrared imaging can be smaller than that of visible light imaging. Generally, the image obtained by infrared imaging can be smaller than the image obtained by visible light imaging. The positions of the target object in the first image and the second image can be different. Since the relative positional relationship between thefirst shooting assembly 0111 and thesecond shooting assembly 0112 can be determined, once the position of the target object in one image is determined, the position of the target object in another image can be easily obtained through conversion. - The thermal infrared image has insufficient texture information compared to the optical image. If the target recognition and tracking are directly performed on the thermal infrared image, the effect can be very poor. But thermal infrared images have the advantage that they can image objects with thermal radiation in the environment without being hindered by light and occlusion. The target tracking method based on the thermal infrared image can have very important practical significance. The present disclosure can use the optical image output by the visible light shooting device to realize the target object recognition to obtain the tracking position area of the target object in the optical image. The shooting attitude of the shooting device can be adjusted according to the tracking position area of the target object in the optical image, such that the target object is located in the shooting frame of the thermal infrared shooting device in the shooting device. As such, even if the thermal infrared image output by the thermal infrared camera cannot or cannot accurately recognize the target object, the thermal infrared camera can still track the target object.
- In one embodiment, the user can specify the target object to be tracked based on the first image output by the
first shooting assembly 0111, so as to achieve the tracking of the target object. For example, referring back toFIG. 1 , after thefirst shooting assembly 0111 and thesecond shooting assembly 0112 shoot the environment to obtain the first image and the second image, themovable platform 01 can send the first image to acontrol terminal 02 to cause thecontrol terminal 02 to display the first image. The user can perform a selection operation on the first image displayed on thecontrol terminal 02, for example, frame an area including the target object to be tracked on the first image. Thecontrol terminal 02 can generate first area indication information according to the area including the target object to be tracked, and send the first area indication information to themovable platform 01. After receiving the first area indication information sent by thecontrol terminal 02, themovable platform 01 can determine second area indication information of the second image according to the first area indication information and the relative positional relationship between thefirst shooting assembly 0111 and thesecond shooting assembly 0112. Themovable platform 01 can perform the target recognition on the area indicated by the second area indication information in the second image to determine the target object and obtain the tracking position area of the target object in the second image. For example, the target recognition can be performed on the area indicated by the second area indication information in the second image, and the target object can be recognized in the area. The recognition may be recognized through a neural network. After the target object is determined, the determined target object may be recognized from the second image output by thesecond shooting assembly 0112 to obtain the tracking position area of the target object in the second image. Furthermore, the recognition of the determined target object from the second image output by thesecond shooting assembly 0112 may be recognized using the neural network, or by image tracking. - In one embodiment, the user can view target tracking result based on the first image at the
control terminal 02. After the tracking position area of the target object in the second image is determined, the tracking position area of the target object in the first image can be determined according to the tracking position area of the target object in the second image and the relative positional relationship between thefirst shooting assembly 0111 and thesecond shooting assembly 0112. The target object can be marked in the first image according to the tracking position area of the target object in the first image. All or some of image information in the second image information can be added to the marked first image to enrich contour features of the marked first image. Finally, the marked first image can be sent to thecontrol terminal 02, such that thecontrol terminal 02 can display the marked first image. - It can be understood that the system architecture and application scenarios described in the embodiments of the present disclosure are intended to more clearly explain the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those of ordinary skill in the art may know that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure can be also applicable to similar technical problems.
-
FIG. 3 is a flow chart of an example target tracking method consistent with the disclosure. The target tracking method may be executed by a movable platform, for example, themovable platform 01 inFIG. 1 . - At S301, the movable platform calls the second shooting assembly to shoot the environment to obtain the second image, and calls the first shooting assembly to shoot the environment to obtain the first image at the same time. The first shooting assembly can include the thermal infrared imaging device and the second shooting assembly can include the visible light imaging device. The imaging modes of the first shooting assembly and the second shooting assembly can be different. The first shooting assembly can use the thermal infrared imaging mode to obtain the thermal infrared image as the first image, and the second shooting assembly can use the visible light imaging mode to obtain the optical image as the second image. The second image can include the optical image.
- Due to the different imaging modes of the first shooting assembly and the second shooting assembly, the positions of the target object in the first image and the second image may be different. Since the relative positional relationship between the first shooting assembly and the second shooting assembly can be determined, once the position of the target object in one image is determined, the position of the target object in another image can be easily obtained through conversion.
- At S302, target object recognition is performed on the second image to recognize the target object in the second image, and the tracking position area of the target object to be tracked in the second image is obtained using segmentation. Target object recognition can determine the target object and the tracking position area of the target object in the second image through image processing methods of target detection and target segmentation. The target detection and target segmentation may include a conventional target detection method and a conventional segmentation method, or may include a target detection method and a target segmentation method based on deep learning (e.g., a neural network), which is not limited in here.
- At 303, the shooting attitude of the shooting device is adjusted according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly. The movable platform can adjust the shooting attitude of the shooting device by changing the attitude of the movable platform itself (e.g., the body), or control the shooting device to adjust the shooting attitude through the gimbal connected to the shooting device, i.e., adjust the shooting attitude of the shooting device by adjusting the attitude of the gimbal.
- The relative position between the first shooting assembly and the second shooting assembly can be known, and the corresponding relationship between the first image and the second image can be also known. Therefore, when the target object is in the preset position area in the second image, the target object should also be in the shooting frame of the first shooting assembly. Furthermore, the target object can be in the target position area of the shooting frame. Adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the second image may include adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the second image, such that after the shooting attitude of the shooting device is adjusted, it can be ensured that the target object is in the preset position area of the image captured by the second shooting assembly. Therefore, the target object can be located in the shooting frame of the first shooting assembly, and further, it can be in the target position area of the shooting frame.
- In some embodiments, it can be ensured that after the shooting attitude of the shooting device is adjusted, the target object is in the shooting frame of the first shooting assembly, and further, the target object can be in the target position area of the shooting frame. For example, adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the second image may also include determining the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image, and adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the shooting frame of the first shooting assembly.
- The thermal infrared image has insufficient texture information compared to the optical image. If the target recognition and tracking are directly performed on the thermal infrared image, the effect can be very poor. But thermal infrared images have the advantage that they can image objects having thermal radiation in the environment without being hindered by light and occlusion. Therefore, the target tracking method based on thermal infrared images can be very important. The present disclosure can solve the problem very well, because the present disclosure can use the optical image output by the visible light shooting device to recognize the target object to obtain the tracking position area of the target object in the optical image. The shooting attitude of the shooting device can be adjusted according to the tracking position area of the target object in the optical image, such that the target object is located in the shooting frame of the thermal infrared shooting device in the shooting device. In this way, even if the target object cannot be or cannot be accurately recognized in the thermal infrared image output by the thermal infrared shooting device, the thermal infrared shooting device can still track the target object.
- In one embodiment, after the first image and the second image are obtained in the shooting environment of the two shooting assemblies of the shooting device, the first image can be first sent to the control terminal of the movable platform, such that the control terminal can display the first image. Thus, the user can perform the selection operation on the first image on the control terminal (e.g., frame the area where the target object is located), and then the terminal device can obtain the first area indication information for indicating the selected area of the user in the first image according to the selection operation, and send the first area indication information to the movable platform. After the movable platform receives the first area indication information, the processes described above can be used to determine the position of the target object in the first image according to the tracking position area of the target object in the second image. The second area indication information can be determined according to the first area indication information and the relative positional relationship between the first shooting assembly and the second shooting assembly. The second area indication information can be used to indicate the area in the first image mapped from the area selected by the user in the second image, and finally, the target recognition can be performed on the area indicated by the second area indication information in the second image to determine the target object, and obtain the tracking position area of the target object in the second image.
- Therefore, when multiple objects are included in the environment, the user can specify the target object, and a target tracking efficiency of the target tracking can be improved.
- In one embodiment, after the tracking position area of the target object in the second image is determined at S301, the tracking position area of the target object in the first image can be determined according to the tracking position area of the target object in the second image. According to the tracking position area of the target object in the first image, the target object can be then marked in the first image (e.g., as shown in the first image in
FIG. 2 , the location of the target object can be framed). The marked first image can be sent to the control terminal of the movable platform to cause the control terminal to display the marked first image. Determining the tracking position area of the target object in the first image according to the tracking position area of the target object in the second image has been described in connection withFIGS. 1 and 2 , and detailed description thereof is omitted herein. - The present disclosure can determine the target object in the tracking position area of the second image by performing the target object recognition on the second image. The tracking position area of the target object in the first image can be determined according to the tracking position area of the target object in the second image, and the target object can be marked in the first image, and the marked target object can be showed to the user through the control terminal. Therefore, indirect target tracking based on the first image can be realized, especially when the first image is the thermal infrared image. The present disclosure can achieve the target tracking based on the thermal infrared image, which has very important practical value.
- In one embodiment, after the target object is marked in the first image, all or some of the image information in the second image information can be first extracted, and all or some of the image information in the second image information can be added to the marked first image to enrich the contour features of the marked first image. The marked first image can be sent to the control terminal, such that the first image finally presented to the user on the control terminal not only can be marked with the target object, but also can have greatly enriched details. A defect that the details of the image, e.g., the thermal imaging image or the like, are not rich can be improved to a certain extent.
- Therefore, the present disclosure can achieve not only the target tracking based on the thermal imaging image, but also an interaction with the user through the thermal imaging image, and use the details of the optical image to enrich the contour details of the thermal imaging image that is not rich in details. The practicality of thermal imaging images can be greatly improved.
-
FIG. 4 is a flow chart of another example target tracking method consistent with the disclosure, and the target tracking method may be executed by a movable platform, e.g., themovable platform 01 inFIG. 1 . - As shown in
FIG. 4 , at S401, the first shooting assembly and the second shooting assembly of the shooting device of the movable platform are called to shoot the environment to obtain the first image and the second image. The first shooting assembly can include the thermal infrared imaging device and the second shooting assembly can include the visible light imaging device. The imaging modes of the first shooting assembly and the second shooting assembly can be different. The first shooting assembly can use the thermal infrared imaging mode to obtain the thermal infrared image as the first image, and the second shooting assembly can use the visible light imaging mode to obtain the optical image as the second image. The second image can include the optical image. - At S402, the first image is sent to the control terminal of the movable platform to cause the control terminal to display the first image.
- At S403, the first area indication information sent by the control terminal is obtained.
- At S404, according to the first area indication information, the target object to be tracked in the second image is determined, and the tracking position area of the target object in the second image is obtained. For example, the second area indication information can be determined according to the first area indication information and the relative positional relationship between the first shooting assembly and the second shooting assembly. The target recognition can be performed on the area indicated by the second area indication information in the second image to determine the target object, and obtain the tracking position area of the target object in the second image.
- At S405, the shooting attitude of the shooting device is adjusted according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly. The movable platform can adjust the shooting attitude of the shooting device by changing the attitude of the movable platform itself, or control the shooting device to adjust the shooting attitude through the gimbal connected to the shooting device, i.e., adjust the shooting attitude of the shooting device by adjusting the attitude of the gimbal.
- For example, adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the second image may include adjusting the target object to the preset position area of the image captured by the second shooting assembly, such that the target object is in the shooting frame of the first shooting assembly, and furthermore, can be in the target position area of the shooting frame.
- In some embodiments, adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the second image may include determining the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image, and the relative positional relationship between the first shooting assembly and the second shooting assembly, and adjusting the shooting attitude of the shooting device according to the tracking position area of the target object in the shooting frame of the first shooting assembly, such that the target object is in the shooting frame of the first shooting assembly, furthermore, can be in the target position area of the shooting frame.
- At S406, according to the tracking position area of the target object in the second image, the tracking position area of the target object in the shooting frame of the first shooting assembly is determined. For example, according to the tracking position area of the target object in the second image and the relative positional relationship between the first shooting assembly and the second shooting assembly, the tracking position area of the target object in the first image can be determined.
- At S407, according to the tracking position area of the target object in the first image, the target object is marked in the first image.
- In one embodiment, after the target object is marked in the first image, the detailed information in the second image is extracted to enrich the contour features in the first image. For example, all or some of the image information in the second image information can be first extracted, and all or some of the image information in the second image information can be added to the marked first image to enrich the contour features of the marked first image.
- At S408, the marked first image is sent to the control terminal of the movable platform to cause the control terminal to display the marked first image.
- It should be noted that the above description of the various embodiments tends to emphasize the differences between the various embodiments. The same aspects or similarities of the various embodiments can be referred to each other, and for the sake of brevity, they will not be repeated here.
- The first image can be also referred to as a “displaying image,” and the second image can be also referred to as a “tracking image.”
-
FIG. 5 is a schematic diagram of an example target tracking device consistent with the disclosure. The target tracking device can implement the target tracking method inFIGS. 3 and 4 . The target tracking device can be applied to a movable platform, for example, themovable platform 01 inFIG. 1 . The movable platform can include the shooting device, and the shooting device can include the first shooting assembly and the second shooting assembly. The target tracking device includes a callingcircuit 510, arecognition circuit 520, and atracking circuit 530. The callingcircuit 510 can be configured to call the second shooting assembly to shoot the environment to obtain the second image, and the imaging modes of the first shooting assembly and the second shooting assembly can be different. Therecognition circuit 520 can be configured to perform the target object recognition on the second image to obtain the tracking position area of the target object to be tracked in the second image. Thetracking circuit 530 can be configured to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly. - For example, the movable platform can include the gimbal carrying the shooting device, and the
tracking unit 530 can be further configured to adjust the attitude of the movable platform and/or the attitude of the gimbal to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image. - The first shooting assembly can include the thermal infrared imaging device, the second shooting assembly can include the visible light imaging device, and the second image can include the optical image.
- For example, the
tracking unit 530 can be specifically configured to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image. After the shooting attitude of the shooting device is adjusted, the target object can be located in the preset position area of the shooting frame of the second shooting assembly. - In some embodiment, the target tracking device can further include a
determination circuit 540 configured to determine the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image. Thetracking unit 530 can be further configured to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the shooting frame of the first shooting assembly. - In one embodiment, the calling
unit 510 can be further configured to call the first shooting assembly to shoot the environment to obtain the first image. The target tracking device can further include a sendingunit 550 configured to send the first image to the control terminal of the movable platform to cause the control terminal to display the first image. The target tracking device can further include anacquisition unit 560 configured to obtain the first area indication information sent by the control terminal. The first area indication information can be determined by the control terminal by detecting the user's selection operation of the target object on the first image displayed by the control terminal. Therecognition unit 520 can be further configured to determine the target object to be tracked in the second image and obtain the tracking position area of the target object in the second image according to the first area indication information. - In one embodiment, the
recognition unit 520 can be further configured to determine the second area indication information of the second image according to the first area indication information and the relative positional relationship between the first shooting assembly and the second shooting assembly, and perform the target recognition on the area indicated by the second area indication information in the second image to determine the target object and obtain the tracking position area of the target object in the second image. - In one embodiment, the calling
unit 510 can be further configured to call the first shooting assembly to shoot the environment to obtain the first image. Thedetermination unit 540 can be further configured to determine the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image. The target tracking device can further include a markingunit 570 configured to mark the target object in the first image according to the tracking position area of the target object in the shooting frame of the first shooting assembly. The sendingunit 550 can be further configured to send the marked first image to the control terminal of the movable platform, such that the control terminal can display the marked first image. - In one embodiment, the target tracking device can further include an
extraction unit 580 configured to extract all or some of the image information in the second image information. The target tracking device can further include an addingunit 590 configured to add all or some of the image information in the second image information to the marked first image to enrich the contour features of the marked first image. - In one embodiment, the
determination unit 540 can be further configured to determine the tracking position area of the target object in the first image according to the tracking position area of the target object in the second image and the relative positional relationship between the first shooting assembly and the second shooting assembly. -
FIG. 6 is a structural diagram of an example movable platform consistent with the disclosure. The movable platform may at least include aprocessor 610, amemory 620, and ashooting device 630. The shooting device includes afirst shooting assembly 631 and asecond shooting assembly 632. Theprocessor 610, thememory 620, and theshooting device 630 may be connected through abus 640 or other means.FIG. 6 takes a connection through the bus as an example. Thememory 620 may be configured to store the computer program. The computer program can include the program instructions. Theprocessor 610 may be configured to execute the program instructions stored in thememory 620. - In one embodiment, the
processor 610 may be a central processing unit (CPU), and the processor may also be another general-purpose processor, i.e., a microprocessor or any conventional processor, such as a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, and the like. - The
memory 620 may include a read-only memory and a random access memory, and provide instructions and data to theprocessor 610. Therefore, theprocessor 610 and thememory 620 are not limited herein. - In the embodiments of the present disclosure, the
processor 610 can load and execute one or more instructions stored in the computer storage medium to implement some or all processes of the methods inFIGS. 3 and 4 . In some embodiment, at least one instruction in the computer storage medium can be loaded and executed by theprocessor 610. - In some embodiments, the
shooting device 630 can be configured to shoot the environment. - The
memory 620 can be configured to store the computer program and the computer program can include the program instructions. - The
processor 610 can be configured to call the program instructions and configured to call the second shooting assembly to shoot the environment to obtain the second image, the imaging modes of the first shooting assembly and the second shooting assembly being different, perform the target object recognition on the second image to obtain the tracking position area of the target object to be tracked in the second image, adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly. - In some embodiments, the
processor 610 can be specifically configured to adjust the attitude of the movable platform and/or the attitude of the gimbal to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image. - The first shooting assembly can include the thermal infrared imaging device, the second shooting assembly can include the visible light imaging device, and the second image can include the optical image.
- In some embodiments, the
processor 610 can be further configured to adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image. After the shooting attitude of the shooting device is adjusted, the target object can be located in the preset position area of the shooting frame of the second shooting assembly. - In some embodiments, the processor can be further configured to determine the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image, and adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the shooting frame of the first shooting assembly.
- In one embodiment, the processor can be further configured to call the first shooting assembly to shoot the environment to obtain the first image. The movable platform further includes a communication interface 650 for data interaction between the movable platform and other terminal devices, and configured to send the first image to the control terminal of the movable platform to cause the control terminal to display the first image. The
processor 610 can be further configured to obtain the first area indication information sent by the control terminal, the first area indication information being determined by the control terminal by detecting the user's selection operation of the target object on the first image displayed by the control terminal, and determine the target object to be tracked in the second image and obtain the tracking position area of the target object in the second image according to the first area indication information. - In some embodiments, the
processor 610 can be further configured to determine the second area indication information of the second image according to the first area indication information and the relative positional relationship between the first shooting assembly and the second shooting assembly, and perform the target recognition on the area indicated by the second area indication information in the second image to determine the target object and obtain the tracking position area of the target object in the second image. - In one embodiment, the
processor 610 can be further configured to call the first shooting assembly to shoot the environment to obtain the first image. Theprocessor 610 can be further configured to determine the tracking position area of the target object in the shooting frame of the first shooting assembly according to the tracking position area of the target object in the second image. Theprocessor 610 can be further configured to mark the target object in the first image according to the tracking position area of the target object in the shooting frame of the first shooting assembly. - In one embodiment, the communication interface 650 can be configured to send the marked first image to the control terminal of the movable platform, such that the control terminal can display the marked first image.
- In one embodiment, the
processor 610 can be further configured to extract all or some of the image information in the second image information, and add all or some of the image information in the second image information to the marked first image to enrich the contour features of the marked first image. - In some embodiments, the
processor 610 can be further configured to determine the tracking position area of the target object in the first image according to the tracking position area of the target object in the second image and the relative positional relationship between the first shooting assembly and the second shooting assembly. - Embodiments of the present disclosure also provide a control device. The control device can be in communication connection with the shooting device, and the shooting device can include the first shooting assembly and the second shooting assembly. The control device is characterized in that the control device includes a memory and a processor.
- The memory may be configured to store the computer program. The computer program can include the program instructions.
- The processor may be configured to call the program instructions and configured to call the second shooting assembly to shoot the environment to obtain the second image, the imaging modes of the first shooting assembly and the second shooting assembly being different, perform the target object recognition on the second image to obtain the tracking position area of the target object to be tracked in the second image, and adjust the shooting attitude of the shooting device according to the tracking position area of the target object in the second image, such that the target object is located in the shooting frame of the first shooting assembly.
- The control device can be provided in the movable platform, and the movable platform can include the control device and the shooting device. The control device may be communicatively connected to the shooting device. The processor of the control device can execute the methods in
FIGS. 3 and 4 . For the details, reference may be made to the previous sections, and will not be repeated herein. - It should be noted that, for the specific working process of the movable platform described above, reference may be made to the related descriptions in the foregoing embodiments, and details will not be repeated herein.
- The integrated units can be stored in a computer readable storage medium when implemented in form of software functional units and sold or used as a standalone product. Based on such understanding, all or part of the technical solution of the disclosure can be embodied in the form of software product stored in a storage medium comprising a number of instructions for causing a computer processor to perform the entire or part of a method consistent with embodiments of the disclosure, such as one of the above-described exemplary methods. The storage medium can comprise a flask disk, a portable hard drive, a read only memory (ROM), a random access memory (RAM), a magnet disk, an optical disk, or other media capable of storing program code.
- Those of ordinary skill in the art can understand that all or some of the processes in the methods of the above embodiments can be completed by a computer program instructing relevant hardware. The program may be stored in a computer-readable storage medium, and when the program is executed, it may include the processes of the foregoing method embodiments. The storage medium may include a magnetic disk, an optical disk, a ROM or RAM, or the like.
- The above description merely illustrates some embodiments of the disclosure and is not intended to limit the scope of the disclosure. Those of ordinary skill in the art can understand all or some of the processes of implementing the above embodiments. Any equivalent changes in light of the claims of the present disclosure should all be covered by the scope of the present disclosure.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/222,627 US20210227144A1 (en) | 2019-05-30 | 2021-04-05 | Target tracking method and device, movable platform, and storage medium |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2019/089248 WO2020237565A1 (en) | 2019-05-30 | 2019-05-30 | Target tracking method and device, movable platform and storage medium |
US16/880,553 US10999519B2 (en) | 2019-05-30 | 2020-05-21 | Target tracking method and device, movable platform, and storage medium |
US17/222,627 US20210227144A1 (en) | 2019-05-30 | 2021-04-05 | Target tracking method and device, movable platform, and storage medium |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/880,553 Continuation US10999519B2 (en) | 2019-05-30 | 2020-05-21 | Target tracking method and device, movable platform, and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210227144A1 true US20210227144A1 (en) | 2021-07-22 |
Family
ID=71187716
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/880,553 Active US10999519B2 (en) | 2019-05-30 | 2020-05-21 | Target tracking method and device, movable platform, and storage medium |
US17/222,627 Abandoned US20210227144A1 (en) | 2019-05-30 | 2021-04-05 | Target tracking method and device, movable platform, and storage medium |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/880,553 Active US10999519B2 (en) | 2019-05-30 | 2020-05-21 | Target tracking method and device, movable platform, and storage medium |
Country Status (4)
Country | Link |
---|---|
US (2) | US10999519B2 (en) |
EP (1) | EP3771198B1 (en) |
CN (1) | CN111345029B (en) |
WO (1) | WO2020237565A1 (en) |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022000300A1 (en) * | 2020-06-30 | 2022-01-06 | 深圳市大疆创新科技有限公司 | Image processing method, image acquisition apparatus, unmanned aerial vehicle, unmanned aerial vehicle system, and storage medium |
CN113973171B (en) * | 2020-07-23 | 2023-10-10 | 宁波舜宇光电信息有限公司 | Multi-camera shooting module, camera shooting system, electronic equipment and imaging method |
CN114401371B (en) * | 2020-08-05 | 2024-03-26 | 深圳市浩瀚卓越科技有限公司 | tracking control method, device, object tracking unit, and storage medium |
WO2022094772A1 (en) * | 2020-11-03 | 2022-05-12 | 深圳市大疆创新科技有限公司 | Position estimation method, following control method, device and storage medium |
CN112601022B (en) * | 2020-12-14 | 2021-08-31 | 中标慧安信息技术股份有限公司 | On-site monitoring system and method based on network camera |
CN113327271B (en) * | 2021-05-28 | 2022-03-22 | 北京理工大学重庆创新中心 | Decision-level target tracking method and system based on double-optical twin network and storage medium |
CN113409358A (en) * | 2021-06-24 | 2021-09-17 | 浙江大华技术股份有限公司 | Image tracking method, image tracking device, storage medium and electronic equipment |
CN115190237B (en) * | 2022-06-20 | 2023-12-15 | 亮风台(上海)信息科技有限公司 | Method and device for determining rotation angle information of bearing device |
CN115623336B (en) * | 2022-11-07 | 2023-06-30 | 北京拙河科技有限公司 | Image tracking method and device for hundred million-level camera equipment |
Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120105634A1 (en) * | 2009-07-08 | 2012-05-03 | Elbit Systems Ltd. | Automatic video surveillance system and method |
US20120143808A1 (en) * | 2010-12-02 | 2012-06-07 | Pukoa Scientific, Llc | Apparatus, system, and method for object detection and identification |
US20140253737A1 (en) * | 2011-09-07 | 2014-09-11 | Yitzchak Kempinski | System and method of tracking an object in an image captured by a moving device |
US9164506B1 (en) * | 2014-07-30 | 2015-10-20 | SZ DJI Technology Co., Ltd | Systems and methods for target tracking |
US9274204B2 (en) * | 2010-08-16 | 2016-03-01 | Korea Research Institute Of Standards And Science | Camera tracing and surveillance system and method for security using thermal image coordinate |
KR101634966B1 (en) * | 2016-04-05 | 2016-06-30 | 삼성지투비 주식회사 | Image tracking system using object recognition information based on Virtual Reality, and image tracking method thereof |
US9769387B1 (en) * | 2013-11-05 | 2017-09-19 | Trace Live Network Inc. | Action camera system for unmanned aerial vehicle |
US9774797B2 (en) * | 2014-04-18 | 2017-09-26 | Flir Systems, Inc. | Multi-sensor monitoring systems and methods |
US9785147B1 (en) * | 2014-08-13 | 2017-10-10 | Trace Live Network Inc. | Pixel based image tracking system for unmanned aerial vehicle (UAV) action camera system |
US20190171201A1 (en) * | 2016-07-28 | 2019-06-06 | SZ DJI Technology Co., Ltd. | Target-based image exposure adjustment |
US10477157B1 (en) * | 2016-03-02 | 2019-11-12 | Meta View, Inc. | Apparatuses, methods and systems for a sensor array adapted for vision computing |
US20200053292A1 (en) * | 2018-08-10 | 2020-02-13 | Aurora Flight Sciences Corporation | Object-Tracking System |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050128291A1 (en) * | 2002-04-17 | 2005-06-16 | Yoshishige Murakami | Video surveillance system |
JP4107273B2 (en) * | 2004-08-04 | 2008-06-25 | 日産自動車株式会社 | Moving body detection device |
CN105915784A (en) * | 2016-04-01 | 2016-08-31 | 纳恩博(北京)科技有限公司 | Information processing method and information processing device |
CN106506941A (en) * | 2016-10-20 | 2017-03-15 | 深圳市道通智能航空技术有限公司 | The method and device of image procossing, aircraft |
EP3428884B1 (en) * | 2017-05-12 | 2020-01-08 | HTC Corporation | Tracking system and tracking method thereof |
WO2018214093A1 (en) * | 2017-05-25 | 2018-11-29 | 深圳市大疆创新科技有限公司 | Tracking method and apparatus |
JP6849272B2 (en) * | 2018-03-14 | 2021-03-24 | エスゼット ディージェイアイ テクノロジー カンパニー リミテッドSz Dji Technology Co.,Ltd | Methods for controlling unmanned aerial vehicles, unmanned aerial vehicles, and systems for controlling unmanned aerial vehicles |
-
2019
- 2019-05-30 CN CN201980005359.9A patent/CN111345029B/en active Active
- 2019-05-30 WO PCT/CN2019/089248 patent/WO2020237565A1/en unknown
- 2019-05-30 EP EP19877539.7A patent/EP3771198B1/en active Active
-
2020
- 2020-05-21 US US16/880,553 patent/US10999519B2/en active Active
-
2021
- 2021-04-05 US US17/222,627 patent/US20210227144A1/en not_active Abandoned
Patent Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120105634A1 (en) * | 2009-07-08 | 2012-05-03 | Elbit Systems Ltd. | Automatic video surveillance system and method |
US9274204B2 (en) * | 2010-08-16 | 2016-03-01 | Korea Research Institute Of Standards And Science | Camera tracing and surveillance system and method for security using thermal image coordinate |
US20120143808A1 (en) * | 2010-12-02 | 2012-06-07 | Pukoa Scientific, Llc | Apparatus, system, and method for object detection and identification |
US20140253737A1 (en) * | 2011-09-07 | 2014-09-11 | Yitzchak Kempinski | System and method of tracking an object in an image captured by a moving device |
US9769387B1 (en) * | 2013-11-05 | 2017-09-19 | Trace Live Network Inc. | Action camera system for unmanned aerial vehicle |
US9774797B2 (en) * | 2014-04-18 | 2017-09-26 | Flir Systems, Inc. | Multi-sensor monitoring systems and methods |
US9164506B1 (en) * | 2014-07-30 | 2015-10-20 | SZ DJI Technology Co., Ltd | Systems and methods for target tracking |
US20170322551A1 (en) * | 2014-07-30 | 2017-11-09 | SZ DJI Technology Co., Ltd | Systems and methods for target tracking |
US9785147B1 (en) * | 2014-08-13 | 2017-10-10 | Trace Live Network Inc. | Pixel based image tracking system for unmanned aerial vehicle (UAV) action camera system |
US10477157B1 (en) * | 2016-03-02 | 2019-11-12 | Meta View, Inc. | Apparatuses, methods and systems for a sensor array adapted for vision computing |
KR101634966B1 (en) * | 2016-04-05 | 2016-06-30 | 삼성지투비 주식회사 | Image tracking system using object recognition information based on Virtual Reality, and image tracking method thereof |
US20190171201A1 (en) * | 2016-07-28 | 2019-06-06 | SZ DJI Technology Co., Ltd. | Target-based image exposure adjustment |
US20200053292A1 (en) * | 2018-08-10 | 2020-02-13 | Aurora Flight Sciences Corporation | Object-Tracking System |
Also Published As
Publication number | Publication date |
---|---|
US20200288065A1 (en) | 2020-09-10 |
EP3771198A4 (en) | 2021-06-16 |
CN111345029B (en) | 2022-07-08 |
EP3771198A1 (en) | 2021-01-27 |
WO2020237565A1 (en) | 2020-12-03 |
EP3771198B1 (en) | 2022-08-24 |
CN111345029A (en) | 2020-06-26 |
US10999519B2 (en) | 2021-05-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10999519B2 (en) | Target tracking method and device, movable platform, and storage medium | |
US10288418B2 (en) | Information processing apparatus, information processing method, and storage medium | |
US10234873B2 (en) | Flight device, flight control system and method | |
CN108986164B (en) | Image-based position detection method, device, equipment and storage medium | |
CN111179358A (en) | Calibration method, device, equipment and storage medium | |
CN109993086B (en) | Face detection method, device and system and terminal equipment | |
CN108875730B (en) | Deep learning sample collection method, device, equipment and storage medium | |
US10659753B2 (en) | Photogrammetry system and method of operation | |
US20230366738A1 (en) | Systems and methods for temperature measurement | |
CN111970454B (en) | Shot picture display method, device, equipment and storage medium | |
CN110287907A (en) | A kind of method for checking object and device | |
CN109934873B (en) | Method, device and equipment for acquiring marked image | |
CN109447902B (en) | Image stitching method, device, storage medium and equipment | |
CN114612786A (en) | Obstacle detection method, mobile robot and machine-readable storage medium | |
JP2006090957A (en) | Surrounding object detecting device for moving body, and surrounding object detection method for moving body | |
WO2021114775A1 (en) | Object detection method, object detection device, terminal device, and medium | |
CN116160458B (en) | Multi-sensor fusion rapid positioning method, equipment and system for mobile robot | |
CN113125434A (en) | Image analysis system and method of controlling photographing of sample image | |
CN115861407A (en) | Safe distance detection method and system based on deep learning | |
CN116125489A (en) | Indoor object three-dimensional detection method, computer equipment and storage medium | |
US20220129660A1 (en) | System and method of acquiring coordinates of pupil center point | |
US20220270282A1 (en) | Information processing device, data generation method, and non-transitory computer-readable medium storing program | |
CN114155258A (en) | Detection method for highway construction enclosed area | |
CN117459688B (en) | Camera angle marking method, device and medium based on map system | |
US20230386055A1 (en) | Image feature matching method, computer device, and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SZ DJI TECHNOLOGY CO., LTD., CHINA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ZHANG, WEI;REEL/FRAME:055827/0395 Effective date: 20200515 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE |