WO2018142493A1 - Image processing device, image processing method, image processing program, image capturing method, and mobile object - Google Patents

Image processing device, image processing method, image processing program, image capturing method, and mobile object Download PDF

Info

Publication number
WO2018142493A1
WO2018142493A1 PCT/JP2017/003488 JP2017003488W WO2018142493A1 WO 2018142493 A1 WO2018142493 A1 WO 2018142493A1 JP 2017003488 W JP2017003488 W JP 2017003488W WO 2018142493 A1 WO2018142493 A1 WO 2018142493A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
subject
distance
cameras
feature point
Prior art date
Application number
PCT/JP2017/003488
Other languages
French (fr)
Japanese (ja)
Inventor
創輔 山尾
山 姜
成幸 小田嶋
Original Assignee
富士通株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 富士通株式会社 filed Critical 富士通株式会社
Priority to PCT/JP2017/003488 priority Critical patent/WO2018142493A1/en
Publication of WO2018142493A1 publication Critical patent/WO2018142493A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C15/00—Surveying instruments or accessories not provided for in groups G01C1/00 - G01C13/00
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00—Measuring distances in line of sight; Optical rangefinders
    • G01C3/02—Details
    • G01C3/06—Use of electric means to obtain final indication

Definitions

  • the present invention relates to an image processing device, an image processing method, an image processing program, an image photographing method, and a moving body.
  • Image processing technology that simultaneously performs self-position estimation and environmental map creation is known. For example, a technique for calibrating a plurality of cameras or restoring an observed image using a parameter block acquired from images captured by a plurality of cameras is known.
  • the image processing apparatus projects the acquired three-dimensional coordinates onto the three-dimensional map by using the same camera and the three-dimensional coordinates and the projection function acquired at different times.
  • processing for generating a three-dimensional map using the correspondence relationship of three-dimensional coordinates acquired at different times by the same camera may be referred to as “intra-camera processing”.
  • a three-dimensional map is generated by setting a baseline length that assumes changes in the moving distance and direction of the camera.
  • a different baseline length is used for each camera. May be set.
  • an object is to provide an image processing apparatus, an image processing method, an image processing program, an image capturing method, and a moving body that can integrate a three-dimensional map reflecting a real scale.
  • the image processing device associates each of the images captured by the first camera and the second camera with each of the images captured at the first timing by the first camera.
  • a distance from the first or second camera to the feature point of the subject is calculated based on the obtained feature point and information on the position of the first and second cameras stored in the storage unit, Information about the distance from the first camera to the subject generated using the same feature point associated with each of the images taken by the camera at the first and second timings is calculated.
  • the correction is made based on the distance from the second camera to the subject.
  • FIG. 1 is a diagram illustrating an example of a plurality of cameras mounted on a moving body.
  • FIG. 2 is a diagram illustrating an example of camera movement between different time points.
  • FIG. 3 is a diagram illustrating an example of subject shooting by the first camera in the background art.
  • FIG. 4 is a diagram illustrating an example of intra-camera processing using the first camera in the background art.
  • FIG. 5 is a diagram illustrating an example of an intra-camera process using the second camera in the background art.
  • FIG. 6 is a diagram illustrating an example of an image processing system according to the first embodiment.
  • FIG. 7 is a diagram illustrating an example of functional blocks according to the first embodiment.
  • FIG. 8 is a diagram illustrating an example of the feature point storage unit according to the first embodiment.
  • FIG. 1 is a diagram illustrating an example of a plurality of cameras mounted on a moving body.
  • FIG. 2 is a diagram illustrating an example of camera movement between different time points.
  • FIG. 3 is
  • FIG. 9 is a diagram illustrating an example of a three-dimensional map storage unit according to the first embodiment.
  • FIG. 10 is a diagram illustrating an example of the relative position and orientation storage unit according to the first embodiment.
  • FIG. 11 is a diagram illustrating an example of the scale storage unit according to the first embodiment.
  • FIG. 12 is a diagram illustrating an example of inter-camera processing according to the first embodiment.
  • FIG. 13 is a diagram illustrating an example of correction processing of coordinates acquired by the first camera in the first embodiment.
  • FIG. 14 is a diagram illustrating an example of a correction process of coordinates acquired by the second camera according to the first embodiment.
  • FIG. 15 is a diagram illustrating an example of the three-dimensional map integrated in the first embodiment.
  • FIG. 16 is a flowchart illustrating an example of a three-dimensional map generation process according to the first embodiment.
  • FIG. 17 is a diagram illustrating an example of a computer that executes an information processing program.
  • FIG. 1 is a diagram illustrating an example of a plurality of cameras mounted on a moving body.
  • a plurality of cameras 100a, 100b, and 100c are mounted on the moving body 10.
  • the cameras 100a, 100b, and 100c are installed separately from each other.
  • the cameras 100a, 100b, and 100c shown in FIG. 1 capture images in different directions, but may be configured such that each camera captures the same direction.
  • the camera 100a is an example of a first camera
  • the camera 100b is an example of a second camera.
  • the camera 100a acquires a plurality of images for performing intra-camera processing by capturing the same feature point at different timings while moving.
  • the feature point is, for example, a boundary point between a road and a white line, which is taken from a traveling automobile, and will be described in detail later.
  • FIG. 2 is a diagram illustrating an example of camera movement between different time points. As shown in FIG. 2, the moving body having the camera 100a moves by a distance S from the time point t0 to the time point t1 after the time point t0. In this case, the camera 100a captures a feature point reflected in the position and orientation 110 at time t0. In addition, the camera 100a captures the same feature point that appears in the position and orientation 111 at the time point t1.
  • FIG. 3 is a diagram illustrating an example of subject shooting by the first camera in the background art.
  • the camera 100a captures the feature points 3000a and 3000b of the subject 3000 at the time point t0.
  • the camera 100a captures the feature points 3000a and 3000b of the subject 3000 at the time point t1.
  • the camera 100a is moved by the distance S and rotated by the angle r from the time point t0 to the time point t1.
  • the position / orientation 110 of the camera 100a changes to the position / orientation 111.
  • the moving distance and the rotation angle are obtained from the coordinates of the feature points in the acquired image. Is estimated.
  • the estimation of the movement distance and the rotation angle may be referred to as “posture estimation”.
  • FIG. 4 is a diagram showing an example of intra-camera processing using the first camera in the background art.
  • the distance Se to the feature point 3100a acquired by the camera 100a is not equal to the distance Sr from the actual camera 100a to the feature point 3000a.
  • the image processing apparatus considers that the camera 100a has moved or rotated by a predetermined baseline “1” based on the distance Se.
  • the baseline “1” is not equal to the actually moved distance S and the rotated angle r
  • the position / orientation 111 ′ of the camera 100a at the time t1 obtained by the attitude estimation by the image processing apparatus is the actual camera. It is different from the position and orientation 111 at the time t1 of 100a.
  • the subject position 3100 recognized by the image processing apparatus is different from the actual subject 3000 position.
  • FIG. 5 is a diagram illustrating an example of an intra-camera process using the second camera in the background art. Similar to the example shown in FIG. 4, the distance to the feature points 3200a and 3200c acquired by the camera 100b is not equal to the actual distance to the feature points 3000a and 3000c. In this case, the image processing apparatus also regards the image acquired by the camera 100b as having moved or rotated by a predetermined baseline “1 ′” that is different from the distance S that the camera 100b has actually moved.
  • the baseline “1” estimated for the camera 100a does not always match the baseline “1 ′” estimated for the camera 100b.
  • the feature points acquired by the camera 100a and the feature points acquired by the camera 100b do not overlap correctly.
  • the map generated by the feature points acquired by the camera 100a may not overlap with the map generated by the feature points acquired by the camera 100b.
  • inter-camera process a process of generating a three-dimensional map using correspondence relationships of three-dimensional coordinates acquired at different times by a plurality of cameras.
  • FIG. 6 is a diagram illustrating an example of an image processing system according to the first embodiment.
  • the image processing system 1 in this embodiment includes a plurality of cameras 100 a and 100 b, an SLAM (Simultaneous Localization and Mapping) device 200, and a scale processing device 300.
  • Each device is connected to be communicable with each other via a network such as the Internet or a bus.
  • the scale processing device 300 is an example of an image processing device.
  • the camera 100a and the camera 100b may be expressed as “camera 100” when they are expressed without distinction.
  • a configuration including two cameras 100 is illustrated, but the configuration is not limited thereto.
  • the image processing system includes three or more cameras 100. May be.
  • the camera 100 is an imaging device that is mounted on a moving body such as an automobile and images the surrounding environment of the moving body.
  • the camera 100 captures an image using, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor as an image sensor.
  • the camera 100 photoelectrically converts light received by the image sensor and performs A / D (Analog / Digital) conversion to generate a captured image.
  • the camera 100 transmits the generated captured image to the SLAM device 200.
  • the SLAM device 200 extracts feature points using an image acquired from the camera 100 and generates a three-dimensional map.
  • FIG. 7 is a diagram illustrating an example of functional blocks according to the first embodiment. As illustrated in FIG. 7, the SLAM apparatus 200 includes a communication unit 210, a storage unit 220, and a control unit 230.
  • the communication unit 210 receives the data transmitted from the camera 100 and outputs it to the control unit 230.
  • the storage unit 220 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk.
  • the storage unit 220 includes a feature point storage unit 221 and a three-dimensional map storage unit 222.
  • the storage unit 220 stores, for example, a program executed by the control unit 230 and various types of information used for processing in the control unit 230.
  • the feature point storage unit 221 stores the coordinates of feature points extracted from the image acquired from the camera 100.
  • the feature point storage unit 221 is sequentially updated by the feature matching unit 233 described later when the camera 100 moves.
  • FIG. 8 is a diagram illustrating an example of a feature point storage unit according to the first embodiment.
  • the feature point storage unit 221 sets “acquisition time”, “three-dimensional coordinates (first camera)”, and “three-dimensional coordinates (second camera)” as “feature point IDs”. Store in association with each other.
  • feature point ID is an identifier for uniquely identifying a feature point extracted from an image.
  • acquisition time stores the time when the image from which the feature points are extracted is acquired.
  • the “three-dimensional coordinates (first camera)” stores the coordinates of feature points extracted from the image acquired at the acquisition time in the camera 100a.
  • “three-dimensional coordinates (second camera)” stores the coordinates of feature points extracted from the image acquired at the acquisition time in the camera 100b.
  • the three-dimensional coordinates are stored in different columns for each camera that has captured the image.
  • the present invention is not limited to this, and the feature point storage unit 221 further stores the identifier of the camera corresponding to the coordinates.
  • the configuration may be such that three-dimensional coordinates are stored in the same column.
  • the feature point storage unit 221 shown in FIG. 8 stores information about the same feature points photographed by different cameras at the same time, but in addition to this, the same camera has different times. Information on the same feature point photographed in is also stored.
  • the 3D map storage unit 222 stores a 3D map generated by a 3D map generation unit 235 described later.
  • the three-dimensional map storage unit 222 is sequentially updated by the three-dimensional map generation unit 235 when the camera 100 moves.
  • FIG. 9 is a diagram illustrating an example of a three-dimensional map storage unit according to the first embodiment.
  • the three-dimensional map storage unit 222 stores “acquisition time” and “three-dimensional coordinates (three-dimensional map after integration)” in association with “feature point IDs”.
  • three-dimensional coordinates (three-dimensional map after integration)” includes “three-dimensional coordinates (first camera)” and “three-dimensional coordinates (second camera)” integrated by the three-dimensional map generation unit 235.
  • the coordinates of the feature points after being stored are stored.
  • control unit 230 is realized, for example, by executing a program stored in an internal storage device using the RAM as a work area by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like.
  • the control unit 230 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
  • control unit 230 includes an image data acquisition unit 231, a feature extraction unit 232, a feature matching unit 233, a posture estimation unit 234, and a three-dimensional map generation unit 235, and performs information processing described below. Implement or execute a function or action.
  • the internal configuration of the control unit 230 is not limited to the configuration illustrated in FIG. 7, and may be another configuration as long as the information processing described later is performed.
  • the image data acquisition unit 231 When the captured image is input from the camera 100, the image data acquisition unit 231 starts acquiring the input captured image.
  • the image data acquisition unit 231 acquires image data captured by the camera 100 and outputs the acquired image data to the feature extraction unit 232.
  • the image data acquisition unit 231 acquires an accurate time when image data is input using, for example, an RTC (Real-Time Clock) (not shown).
  • RTC Real-Time Clock
  • the feature extraction unit 232 extracts a feature point included in the input image data candidate range for each of the image data input from a plurality of cameras. It is. For example, the feature extraction unit 232 identifies a corner point by performing Harris corner detection. The feature extraction unit 232 may identify corner points based on other corner point detection methods other than Harris corner detection. The feature extraction unit 232 stores the corner points detected by the corner detection method in the corresponding camera row of the feature point storage unit 221 as feature points. Note that the method of extracting the feature points by the feature extraction unit 232 is not limited to this, and the feature points may be extracted by edge detection, for example.
  • the feature matching unit 233 acquires data used for the intra-camera process with reference to the stored feature points. For example, the feature matching unit 233 acquires the coordinates of the same feature point ID extracted from images acquired by the same camera at different time points stored in the feature point storage unit 221, that is, the feature point correspondence. . Then, the feature matching unit 233 outputs the acquired set pointtra corresponding to the feature points at different times to the posture estimation unit 234 and the three-dimensional map generation unit 235.
  • the feature matching unit 233 acquires the coordinates of the same feature point ID, that is, the feature point correspondences extracted from images acquired by different cameras at the same time stored in the feature point storage unit 221. Then, the feature matching unit 233 outputs to the scale processing device 300 the set pinter corresponding to the feature points at the acquired simultaneous points and the acquired set pintra corresponding to the feature points at different time points.
  • the posture estimation unit 234 performs posture estimation processing of the camera 100 using the set pintra output from the feature matching unit 233. For example, the posture estimation unit 234 estimates the initial position and posture T from the set pintra using a known five-point algorithm.
  • the posture estimation unit 234 when the posture estimation unit 234 receives the corrected scale input from the scale processing device 300, the posture estimation unit 234 corrects the estimated initial position and posture T. For example, the posture estimation unit 234 calculates the corrected position and posture T ′ by multiplying the initial position and posture T by the input scale s.
  • the posture estimation unit 234 outputs the initial position / posture T and the corrected position / posture T ′ to the three-dimensional map generation unit 235.
  • the three-dimensional map generation unit 235 generates a three-dimensional map using the pintra output from the feature matching unit 233 and the initial position / posture T or the corrected position / posture T ′ output from the posture estimation unit 234.
  • the 3D map generation unit 235 stores the coordinates of the generated 3D map in the 3D map storage unit 222.
  • the scale processing device 300 in this embodiment includes a storage unit 320 and a control unit 330.
  • the scale processing apparatus 300 is also realized by a device such as a computer, similarly to the SLAM apparatus 200, and may have various functional units included in a known computer in addition to the functional units illustrated in FIG.
  • the storage unit 320 stores programs executed by the control unit 330, various data, and the like.
  • the storage unit 320 corresponds to a semiconductor memory element such as a RAM, a ROM, and a flash memory, and a storage device such as an HDD.
  • the storage unit 320 includes a relative position / posture storage unit 321 and a scale storage unit 322. Further, the storage unit 320 stores, for example, a program executed by the control unit 330 and various types of information used for processing in the control unit 330.
  • the relative position / posture storage unit 321 stores information on the relative positions and postures of the cameras 100a and 100b. Information stored in the relative position and orientation storage unit 321 is set in advance by, for example, an administrator of the image processing system 1 (not shown).
  • FIG. 10 is a diagram illustrating an example of the relative position and orientation storage unit according to the first embodiment.
  • the relative position and orientation storage unit 321 stores “coordinates” and “angles” in association with “camera IDs”.
  • “camera ID” is an identifier for uniquely identifying a camera included in the image processing system 1.
  • “Coordinates” and “angles” store the difference between the position and orientation of the camera and the position and orientation of the reference camera.
  • the camera with the camera ID “C001” is used as a reference camera.
  • the coordinates of the camera with the camera ID “C001” are the origin (0, 0, 0), and there are no angles (0 °, 0 °, 0 °).
  • the coordinates of the camera with the camera ID “C002” are moved “100 mm” in the x direction from the origin, and the angle is shifted by “5 °” in the x axis direction.
  • the scale storage unit 322 stores the scale corrected by the scale correction unit 332 described later.
  • FIG. 11 is a diagram illustrating an example of the scale storage unit according to the first embodiment. As illustrated in FIG. 11, the scale storage unit 322 stores “scale” in association with “camera ID”. In FIG. 11, “scale” is a numerical value for correcting coordinates acquired by the camera with the corresponding camera ID. For example, the corrected three-dimensional coordinates are calculated by multiplying the three-dimensional coordinates by a scale.
  • control unit 330 is realized by executing a program stored in an internal storage device using the RAM as a work area, for example, by a CPU, MPU, or the like.
  • the control unit 330 may be realized by an integrated circuit such as an ASIC or FPGA, for example.
  • the control unit 330 includes a scale estimation unit 331 and a scale correction unit 332, and realizes or executes information processing functions and operations described below.
  • the internal configuration of the control unit 330 is not limited to the configuration illustrated in FIG. 7, and may be another configuration as long as the information processing described below is performed.
  • the scale estimation unit 331 is identical between the pintra and the pinter.
  • a set p ′ corresponding to the feature point that refers to the three-dimensional point is extracted.
  • the scale estimation unit 331 refers to the relative position / posture storage unit 321 and calculates a three-dimensional point group set M ′ of the set p ′. For example, the scale estimation unit 331 corrects the three-dimensional coordinates acquired by the camera 100b in the pinter using the relative position and orientation of the camera 100b acquired from the relative position and orientation storage unit 321. Then, the scale estimation unit 331 calculates M ′ that is a set of corrected three-dimensional coordinates.
  • the scale estimator 331 generates all three-dimensional points between a set M of three-dimensional points p in the initial three-dimensional map generated from pintra and M ′, which is a set of calculated p ′. , The distance ratio p / p ′ is calculated. Then, the scale estimation unit 331 employs the median value of all the calculated distance ratios p / p ′ as the scale s between M and M ′. The scale estimation unit 331 stores the adopted scale s in the scale storage unit 322 in association with the camera ID. Further, the scale estimation unit 331 similarly calculates the scale s ′ for the three-dimensional coordinate pbintra generated by the intra-camera process from the image acquired from the camera 100b.
  • the scale correction unit 332 corrects the initial three-dimensional map M and the initial position / posture T generated from the pintra using the calculated scale s. Specifically, when receiving the initial three-dimensional map M and the initial position and orientation T from the SLAM device 200, the scale correction unit 332 multiplies the scale s stored in the scale storage unit 322. Then, the scale correcting unit 332 outputs the corrected three-dimensional map M ′ and the corrected position / posture T ′ to the SLAM device 200. The scale correction unit 332 performs the same process for each camera 100.
  • FIG. 12 is a diagram illustrating an example of inter-camera processing according to the first embodiment.
  • the cameras 100a and 100b respectively photograph the same subject 3000 at the same time point t0.
  • the feature extraction unit 232 of the SLAM device 200 specifies the three-dimensional coordinates of the feature point 3000a from the images captured by the cameras 100a and 100b.
  • the feature extraction unit 232 specifies the coordinates for each of the images photographed by the cameras 100 a and 100 b and stores them in the feature point storage unit 221.
  • the feature matching unit 233 reads the feature point correspondence at the specified simultaneous point from the feature point storage unit 221, and outputs the set pinter corresponding to the feature point to the scale processing device 300.
  • the scale estimation unit 331 of the scale processing apparatus 300 uses the coordinates of the three-dimensional point p included in the input pinter and the position and orientation of the camera 100b stored in the relative position and orientation storage unit 321 to correct the coordinates.
  • the set p ′ is calculated.
  • the scale estimation unit 331 calculates the scale s using the calculated ratio between p ′ and p, and stores it in the scale storage unit 322.
  • the scale correction unit 332 multiplies the three-dimensional coordinates of the feature points, which are input from the SLAM device 200 and calculated by the intra-camera process on the image acquired from the camera 100a, by the scale s, and performs the correction.
  • the three-dimensional coordinates of the feature points are calculated.
  • FIG. 13 is a diagram illustrating an example of correction processing of coordinates acquired by the first camera in the first embodiment. As shown in FIG. 13, by correcting the three-dimensional coordinate of the feature point 3100a calculated by the intra-camera process with the scale s, the corrected three-dimensional coordinate 3110a becomes the three-dimensional coordinate of the actual feature point 3000a of the subject. And overlap.
  • the scale correction unit 332 also applies the three-dimensional feature points after correction to the three-dimensional coordinates of the feature points calculated by the intra-camera process for the image input from the SLAM device 200 and acquired from the camera 100b. Calculate the coordinates.
  • FIG. 14 is a diagram illustrating an example of a correction process of coordinates acquired by the second camera according to the first embodiment. As shown in FIG. 14, the corrected three-dimensional coordinates 3210a overlap with the three-dimensional coordinates of the actual feature point 3000a of the subject, and the corrected three-dimensional coordinates 3210c overlap with the three-dimensional coordinates of the feature point 3000c of the subject. .
  • FIG. 15 is a diagram illustrating an example of the three-dimensional map integrated in the first embodiment.
  • a three-dimensional map reflecting the actual scale can be integrated.
  • the three-dimensional coordinates 3110b and 3210c of the feature points after correction also overlap with the feature points 3000b and 3000c of the actual subject, respectively.
  • FIG. 16 is a flowchart illustrating an example of a three-dimensional map generation process according to the first embodiment.
  • the image data acquisition unit 231 of the SLAM apparatus 200 stands by until image data is acquired from the first camera 100a (S100: No).
  • the image data acquisition unit 231 outputs the acquired image data to the feature extraction unit 232.
  • the feature extraction unit 232 extracts the feature point pa from the input image data and stores it in the feature point storage unit 221 (S101).
  • the feature matching unit 233 refers to the feature point storage unit 221, obtains a set pintra corresponding to feature points at different time points, and outputs them to the posture estimation unit 234 and the three-dimensional map generation unit 235 (S102).
  • the posture estimation unit 234 estimates the initial position and posture T of the camera 100a using the pintra output from the feature matching unit 233, and outputs the estimated initial position / posture T to the three-dimensional map generation unit 235 (S103).
  • the three-dimensional map generation unit 235 generates an initial three-dimensional map Mi using the pintra output from the feature matching unit 233 and the initial position / posture T output from the posture estimation unit 234. It memorize
  • the image data acquisition unit 231 waits until image data is acquired from the second camera 100b (S110: No).
  • the image data acquisition unit 231 outputs the acquired image data to the feature extraction unit 232.
  • the feature extraction unit 232 extracts the feature point pb from the input image data and stores it in the feature point storage unit 221 (S111).
  • the feature matching unit 233 refers to the feature point storage unit 221 and acquires a set pinter corresponding to feature points of pa and pb at the same point. Then, the feature matching unit 233 outputs the acquired pintra and pinter to the scale processing device 300 (S112).
  • the scale estimation unit 331 of the scale processing apparatus 300 receives the input of the pintra and the pinter from the SLAM apparatus 200, the scale estimation unit 331 extracts a set p ′ corresponding to the feature point that refers to the same three-dimensional point between the pintra and the pinter. (S113). Next, the scale estimation unit 331 calculates the three-dimensional point group set M ′ of the set p ′ using the relative position and orientation stored in the relative position and orientation storage unit 321 (S ⁇ b> 114).
  • the scale estimation unit 331 acquires a set M of three-dimensional points p of the initial three-dimensional map generated from pintra (S115). Then, the scale estimation unit 331 calculates the scale s using the calculated M ′ and M, and stores it in the scale storage unit 322 (S116).
  • the scale correcting unit 332 calculates a corrected three-dimensional map M′i obtained by multiplying Mi ′ by s using the scale s (S117). In addition, the scale correction unit 332 calculates a corrected position / posture T ′ obtained by multiplying the initial position / posture T by s using the scale s (S118). Then, the scale correcting unit 332 outputs the calculated corrected three-dimensional map M′i and the corrected position / posture T ′ to the SLAM device 200.
  • the three-dimensional map generation unit 235 that has received the output of the corrected three-dimensional map M′i and the corrected position / posture T ′ applies the processing of S122 described below to all the coordinates acquired from the n cameras 100. (S121).
  • the three-dimensional map generation unit 235 sequentially generates the three-dimensional map Mk using the corrected position / posture T ′ (S122). The three-dimensional map generation unit 235 returns to S121 and repeats the processing until the processing for the coordinates acquired from all the cameras 100 is completed (S123: No).
  • the 3D map generation unit 235 When the process of S122 is completed for the coordinates acquired from all the cameras 100 (S123: Yes), the 3D map generation unit 235 superimposes all the generated 3D maps in the 3D map storage unit 222. Store (S124).
  • the transmission apparatus divides the data to be transmitted into a plurality of pieces of divided data, and the divided data and the own apparatus hold each of the plurality of pieces of divided data. Generate primary verification data from the key. Then, after transmitting the transmission data for verification including the generated primary verification data to the receiving device, the transmitting device transmits a plurality of pieces of divided data to the receiving device. Also, in the data transmission / reception method according to the present embodiment, when the receiving device receives the verification transmission data and receives the divided data, the verification data is generated from the divided data and the common key held by the own device. Then, the receiving device determines the validity of the divided data according to the comparison result between the generated verification data and each primary verification data corresponding to the divided data included in the verification transmission data. Thereby, the receiving apparatus can integrate the three-dimensional map reflecting the actual scale.
  • the SLAM device 200 and the scale processing device 300 have been described in the above embodiment, the SLAM device 200 may be configured to have the functions of the scale processing device 300. Further, the camera 100a and 100b, the SLAM device 200, and the scale processing device 300 may be configured to be mounted on one moving body such as an automobile. Thereby, for example, a three-dimensional map without deviation can be provided in real time to a driver of a car.
  • the three-dimensional map generation process can be performed, for example, when acquisition of image data is started, but the timing for performing the process is not limited to this.
  • processing may be performed at predetermined intervals such as every second, and processing is performed when the number of acquired three-dimensional coordinates exceeds a predetermined threshold or when a key frame is displayed. May be.
  • the processing may be performed when information about the position of the camera 100 is changed, such as when the position and orientation of the camera 100a and the camera 100b change due to an external impact or the like.
  • the process may be performed when the moving speed or the rotation angle of the camera 100a and the camera 100b is changed, and the process may be repeated every time the process is finished. Thereby, the shift
  • each component of each part shown in the figure is functionally conceptual and does not necessarily need to be physically configured as shown.
  • the specific form of distribution / integration of each unit is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed / integrated in arbitrary units according to various loads or usage conditions. Can be configured.
  • various processing functions performed by each device may be executed entirely or arbitrarily on a CPU (or a microcomputer such as an MPU or MCU (Micro Controller Unit)).
  • various processing functions may be executed in whole or in any part on a program that is analyzed and executed by a CPU (or a microcomputer such as an MPU or MCU) or on hardware based on wired logic. Needless to say, it is good.
  • each component of each part illustrated does not necessarily need to be physically configured as illustrated.
  • the specific form of distribution / integration of each unit is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed / integrated in arbitrary units according to various loads or usage conditions. Can be configured.
  • the image data acquisition unit 231 and the feature extraction unit 232 may be integrated.
  • the illustrated processes are not limited to the above-described order, and may be performed at the same time as long as the processing contents do not contradict each other, or may be performed by changing the order.
  • each device may be executed entirely or arbitrarily on a CPU (or a microcomputer such as MPU or MCU (Micro Controller Unit)).
  • various processing functions may be executed in whole or in any part on a program that is analyzed and executed by a CPU (or a microcomputer such as an MPU or MCU) or on hardware based on wired logic. Needless to say, it is good.
  • FIG. 17 is a diagram illustrating an example of a computer that executes an information processing program.
  • a computer having a function equivalent to that of the scale processing device 300 will be described, but the SLAM device 200 and the moving body 10 can also be realized by the same configuration.
  • the computer 900 includes a CPU 901 that executes various arithmetic processes, an input device 902 that receives data input, and a monitor 903.
  • the computer 900 also includes a medium reading device 904 that reads a program and the like from a storage medium, an interface device 905 for connecting to various devices, and a communication device 906 for connecting to other information processing devices and the like by wire or wirelessly.
  • Have The computer 900 also includes a RAM 907 that temporarily stores various information and a hard disk device 908. Each device 901 to 908 is connected to a bus 909.
  • the hard disk device 908 stores an information processing program having the same functions as the processing units of the scale estimation unit 331 and the scale correction unit 332 shown in FIG. Also, the hard disk device 908 stores a relative position and orientation storage unit 321 and a scale storage unit 322, and various data for realizing an information processing program.
  • the input device 902 receives input of various information such as operation information from a user of the computer 900, for example.
  • the monitor 903 displays various screens such as a display screen for the user of the computer 900, for example.
  • a camera or the like is connected to the interface device 905.
  • the communication device 906 has the same function as the communication unit 210 illustrated in FIG. 7, for example, and exchanges various types of information with the camera 100.
  • the CPU 901 reads out each program stored in the hard disk device 908, develops it in the RAM 907, and executes it to perform various processes.
  • these programs can cause the computer 900 to function as the scale estimation unit 331 and the scale correction unit 332 illustrated in FIG.
  • the computer 900 may read and execute a program stored in a storage medium readable by the computer 900.
  • the storage medium readable by the computer 900 corresponds to, for example, a portable recording medium such as a CD-ROM, a DVD disk, a USB (Universal Serial Bus) memory, a semiconductor memory such as a flash memory, a hard disk drive, and the like.
  • the information processing program may be stored in a device connected to a public line, the Internet, a LAN, or the like, and the computer 900 may read out and execute the information processing program therefrom.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Electromagnetism (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Image Analysis (AREA)

Abstract

The present invention includes: in images of a subject image-captured by a first camera and a second camera, on the basis of feature points associated with one another respectively from the images captured by the first and second cameras at a first timing and information (321) on the positions of the first and second cameras, stored in a storage unit, calculating (331) a distance from the first or second camera to the feature point in the subject; and correcting (332), on the basis of the calculated distance from the first or second camera to the subject, information on the distance from the first camera to the subject, which is generated by using the same feature points associated with one another respectively from images captured by the first camera at the first and second timings. Thus, three-dimensional maps reflecting actual scale can be integrated.

Description

画像処理装置、画像処理方法、画像処理プログラム、画像撮影方法、及び移動体Image processing apparatus, image processing method, image processing program, image photographing method, and moving body
　本発明は、画像処理装置、画像処理方法、画像処理プログラム、画像撮影方法、及び移動体に関する。 The present invention relates to an image processing device, an image processing method, an image processing program, an image photographing method, and a moving body.
　自己位置推定と環境地図作成とを同時に行う画像処理技術が知られている。例えば、複数のカメラにより撮影された画像から取得されたパラメータブロックを用いて、複数のカメラを較正し、又は観測された画像を復元する技術が知られている。上記技術において、画像処理装置は、同一のカメラにより、異なる時点において取得された３次元座標と投射関数とを用いて、取得された３次元座標を３次元マップに投射する。なお、以下において、同一のカメラにより異なる時点において取得された３次元座標の対応関係を用いて３次元マップを生成する処理を、「intra-camera処理」と表記する場合がある。 Image processing technology that simultaneously performs self-position estimation and environmental map creation is known. For example, a technique for calibrating a plurality of cameras or restoring an observed image using a parameter block acquired from images captured by a plurality of cameras is known. In the above technique, the image processing apparatus projects the acquired three-dimensional coordinates onto the three-dimensional map by using the same camera and the three-dimensional coordinates and the projection function acquired at different times. In the following, processing for generating a three-dimensional map using the correspondence relationship of three-dimensional coordinates acquired at different times by the same camera may be referred to as “intra-camera processing”.
　しかし、異なる時点において座標を取得する際に、第一の時点から第二の時点までの間でのカメラの移動距離や方向の変化を正確に特定できない場合がある。この場合、カメラの移動距離や方向の変化を仮定するベースライン長を設定して３次元マップを生成することになるが、複数のカメラで画像を取得する場合、カメラごとに異なるベースライン長が設定される場合がある。このため、上記技術においては、生成された３次元マップを、別のカメラによるintra-camera処理において生成された別の３次元マップと統合する際に、マップが重ならない場合がある。 However, when acquiring coordinates at different points in time, it may not be possible to accurately identify changes in the moving distance and direction of the camera from the first point to the second point. In this case, a three-dimensional map is generated by setting a baseline length that assumes changes in the moving distance and direction of the camera. However, when acquiring images with a plurality of cameras, a different baseline length is used for each camera. May be set. For this reason, in the above technique, when the generated three-dimensional map is integrated with another three-dimensional map generated in the intra-camera processing by another camera, the maps may not overlap.
　一つの側面では、実スケールを反映した３次元マップを統合できる画像処理装置、画像処理方法、画像処理プログラム、画像撮影方法、及び移動体を提供することを目的とする。 In one aspect, an object is to provide an image processing apparatus, an image processing method, an image processing program, an image capturing method, and a moving body that can integrate a three-dimensional map reflecting a real scale.
　一つの態様において、画像処理装置は、第一のカメラおよび第二のカメラにより撮影された被写体の画像において、第一および第二のカメラが第一のタイミングにて撮影した画像のそれぞれから対応づけられた特徴点と、記憶部に記憶された第一および第二のカメラの位置に関する情報とに基づいて、第一又は第二のカメラから被写体の特徴点までの距離を算出し、第一のカメラが第一および第二のタイミングにて撮影した画像のそれぞれから対応付けられた同一の特徴点を用いて生成された、第一のカメラから被写体までの距離に関する情報を、算出された第一又は第二のカメラから被写体までの距離を基にして修正する。 In one embodiment, the image processing device associates each of the images captured by the first camera and the second camera with each of the images captured at the first timing by the first camera. A distance from the first or second camera to the feature point of the subject is calculated based on the obtained feature point and information on the position of the first and second cameras stored in the storage unit, Information about the distance from the first camera to the subject generated using the same feature point associated with each of the images taken by the camera at the first and second timings is calculated. Alternatively, the correction is made based on the distance from the second camera to the subject.
　一つの態様によれば、実スケールを反映した３次元マップを統合できる。 According to one aspect, it is possible to integrate a three-dimensional map reflecting the actual scale.
図１は、移動体に搭載された複数のカメラの一例を示す図である。FIG. 1 is a diagram illustrating an example of a plurality of cameras mounted on a moving body. 図２は、異時点間でのカメラの移動の一例を示す図である。FIG. 2 is a diagram illustrating an example of camera movement between different time points. 図３は、背景技術における第一のカメラによる被写体撮影の一例を示す図である。FIG. 3 is a diagram illustrating an example of subject shooting by the first camera in the background art. 図４は、背景技術における第一のカメラを用いたintra-camera処理の一例を示す図である。FIG. 4 is a diagram illustrating an example of intra-camera processing using the first camera in the background art. 図５は、背景技術における第二のカメラを用いたintra-camera処理の一例を示す図である。FIG. 5 is a diagram illustrating an example of an intra-camera process using the second camera in the background art. 図６は、実施例１における画像処理システムの一例を示す図である。FIG. 6 is a diagram illustrating an example of an image processing system according to the first embodiment. 図７は、実施例１における機能ブロックの一例を示す図である。FIG. 7 is a diagram illustrating an example of functional blocks according to the first embodiment. 図８は、実施例１における特徴点記憶部の一例を示す図である。FIG. 8 is a diagram illustrating an example of the feature point storage unit according to the first embodiment. 図９は、実施例１における３次元マップ記憶部の一例を示す図である。FIG. 9 is a diagram illustrating an example of a three-dimensional map storage unit according to the first embodiment. 図１０は、実施例１における相対位置姿勢記憶部の一例を示す図である。FIG. 10 is a diagram illustrating an example of the relative position and orientation storage unit according to the first embodiment. 図１１は、実施例１におけるスケール記憶部の一例を示す図である。FIG. 11 is a diagram illustrating an example of the scale storage unit according to the first embodiment. 図１２は、実施例１におけるinter-camera処理の一例を示す図である。FIG. 12 is a diagram illustrating an example of inter-camera processing according to the first embodiment. 図１３は、実施例１における第一のカメラにより取得された座標の補正処理の一例を示す図である。FIG. 13 is a diagram illustrating an example of correction processing of coordinates acquired by the first camera in the first embodiment. 図１４は、実施例１における第二のカメラにより取得された座標の補正処理の一例を示す図である。FIG. 14 is a diagram illustrating an example of a correction process of coordinates acquired by the second camera according to the first embodiment. 図１５は、実施例１において統合された三次元マップの一例を示す図である。FIG. 15 is a diagram illustrating an example of the three-dimensional map integrated in the first embodiment. 図１６は、実施例１における３次元マップ生成処理の一例を示すフローチャートである。FIG. 16 is a flowchart illustrating an example of a three-dimensional map generation process according to the first embodiment. 図１７は、情報処理プログラムを実行するコンピュータの一例を示す図である。FIG. 17 is a diagram illustrating an example of a computer that executes an information processing program.
　以下に、本願の開示する画像処理装置、画像処理方法、画像処理プログラム、画像撮影方法、及び移動体の実施例を図面に基づいて詳細に説明する。なお、本実施例により、開示技術が限定されるものではない。また、以下の実施例は、矛盾しない範囲で適宜組みあわせてもよい。 Hereinafter, embodiments of an image processing apparatus, an image processing method, an image processing program, an image capturing method, and a moving body disclosed in the present application will be described in detail with reference to the drawings. The disclosed technology is not limited by the present embodiment. Further, the following embodiments may be appropriately combined within a consistent range.
　本実施例における画像処理装置は、例えば、自動車などの移動体に搭載される複数のカメラにより撮影された画像を用いる。図１は、移動体に搭載された複数のカメラの一例を示す図である。図１に示すように、移動体１０には、複数のカメラ１００ａ、１００ｂ及び１００ｃが搭載される。図１に示すように、カメラ１００ａ、１００ｂ及び１００ｃは、それぞれ離間して設置される。なお、図１に示すカメラ１００ａ、１００ｂ及び１００ｃは、それぞれ異なる方向の画像を撮影するが、各カメラが同一の方向を撮影するような構成であってもよい。なお、カメラ１００ａは第一のカメラの一例であり、カメラ１００ｂは第二のカメラの一例である。 The image processing apparatus in the present embodiment uses, for example, images taken by a plurality of cameras mounted on a moving body such as an automobile. FIG. 1 is a diagram illustrating an example of a plurality of cameras mounted on a moving body. As shown in FIG. 1, a plurality of cameras 100a, 100b, and 100c are mounted on the moving body 10. As shown in FIG. 1, the cameras 100a, 100b, and 100c are installed separately from each other. The cameras 100a, 100b, and 100c shown in FIG. 1 capture images in different directions, but may be configured such that each camera captures the same direction. The camera 100a is an example of a first camera, and the camera 100b is an example of a second camera.
　カメラ１００ａは、移動しながら同一の特徴点を異なるタイミングで撮影することにより、intra-camera処理を行うための複数の画像を取得する。なお、特徴点は、例えば走行する自動車から撮影される、道路と白線との境界点などであり、詳細については後に説明する。図２は、異時点間でのカメラの移動の一例を示す図である。図２に示すように、カメラ１００ａを有する移動体は、時点ｔ０から、時点ｔ０より後の時点ｔ１までの間に、距離Ｓだけ移動する。この場合において、カメラ１００ａは、時点ｔ０において、位置姿勢１１０の状態で映る特徴点を撮影する。また、カメラ１００ａは、時点ｔ１において、位置姿勢１１１の状態で映る同一の特徴点を撮影する。 The camera 100a acquires a plurality of images for performing intra-camera processing by capturing the same feature point at different timings while moving. The feature point is, for example, a boundary point between a road and a white line, which is taken from a traveling automobile, and will be described in detail later. FIG. 2 is a diagram illustrating an example of camera movement between different time points. As shown in FIG. 2, the moving body having the camera 100a moves by a distance S from the time point t0 to the time point t1 after the time point t0. In this case, the camera 100a captures a feature point reflected in the position and orientation 110 at time t0. In addition, the camera 100a captures the same feature point that appears in the position and orientation 111 at the time point t1.
　ここで、背景技術における、画像処理装置が第一のカメラ及び第二のカメラから取得する画像を用いたintra-camera処理について、図３乃至図５を用いて説明する。図３は、背景技術における第一のカメラによる被写体撮影の一例を示す図である。図３に示すように、カメラ１００ａは、時点ｔ０において、被写体３０００の特徴点３０００ａ及び３０００ｂを撮影する。また、カメラ１００ａは、時点ｔ１においても同様に、被写体３０００の特徴点３０００ａ及び３０００ｂを撮影する。この際、カメラ１００ａは、時点ｔ０から時点ｔ１までの間に、距離Ｓだけ移動し、かつ角度ｒだけ回転している。この結果、カメラ１００ａの位置姿勢１１０は、位置姿勢１１１へと変化する。 Here, intra-camera processing using images acquired by the image processing apparatus from the first camera and the second camera in the background art will be described with reference to FIGS. 3 to 5. FIG. 3 is a diagram illustrating an example of subject shooting by the first camera in the background art. As shown in FIG. 3, the camera 100a captures the feature points 3000a and 3000b of the subject 3000 at the time point t0. Similarly, the camera 100a captures the feature points 3000a and 3000b of the subject 3000 at the time point t1. At this time, the camera 100a is moved by the distance S and rotated by the angle r from the time point t0 to the time point t1. As a result, the position / orientation 110 of the camera 100a changes to the position / orientation 111.
　この場合において、intra-camera処理を行う画像処理装置は、実際に移動した距離Ｓ及び回転した角度ｒを特定できない場合があるため、取得される画像における特徴点の座標から、移動距離及び回転角度を推定する。なお、以下において、当該移動距離及び回転角度の推定を、「姿勢推定」と表記する場合がある。 In this case, since the image processing apparatus that performs the intra-camera process may not be able to specify the actually moved distance S and the rotated angle r, the moving distance and the rotation angle are obtained from the coordinates of the feature points in the acquired image. Is estimated. In the following, the estimation of the movement distance and the rotation angle may be referred to as “posture estimation”.
　図４は、背景技術における第一のカメラを用いたintra-camera処理の一例を示す図である。図４に示すように、カメラ１００ａにより取得される特徴点３１００ａまでの距離Ｓｅは、実際のカメラ１００ａから特徴点３０００ａまでの距離Ｓｒとは等しくない。特徴点３１００ｂまでの距離についても同様である。この場合、画像処理装置は、かかる距離Ｓｅに基づき、カメラ１００ａが所定のベースライン「１」だけ移動し又は回転したとみなす。この場合、ベースライン「１」は実際に移動した距離Ｓ及び回転した角度ｒとは等しくないため、画像処理装置による姿勢推定により得られるカメラ１００ａの時点ｔ１における位置姿勢１１１’は、実際のカメラ１００ａの時点ｔ１における位置姿勢１１１とは異なる。その結果、画像処理装置が認識する被写体の位置３１００は、実際の被写体３０００の位置とは異なる。 FIG. 4 is a diagram showing an example of intra-camera processing using the first camera in the background art. As shown in FIG. 4, the distance Se to the feature point 3100a acquired by the camera 100a is not equal to the distance Sr from the actual camera 100a to the feature point 3000a. The same applies to the distance to the feature point 3100b. In this case, the image processing apparatus considers that the camera 100a has moved or rotated by a predetermined baseline “1” based on the distance Se. In this case, since the baseline “1” is not equal to the actually moved distance S and the rotated angle r, the position / orientation 111 ′ of the camera 100a at the time t1 obtained by the attitude estimation by the image processing apparatus is the actual camera. It is different from the position and orientation 111 at the time t1 of 100a. As a result, the subject position 3100 recognized by the image processing apparatus is different from the actual subject 3000 position.
　次に、画像処理装置は、第二のカメラであるカメラ１００ｂにより取得された画像についても、同様にintra-camera処理を行う。図５は、背景技術における第二のカメラを用いたintra-camera処理の一例を示す図である。図４に示す例と同様に、カメラ１００ｂにより取得される特徴点３２００ａ及び３２００ｃまでの距離も、実際の特徴点３０００ａ及び３０００ｃまでの距離とは等しくない。この場合において、画像処理装置は、カメラ１００ｂにより取得される画像についても、同様に実際にカメラ１００ｂが移動した距離Ｓとは異なる所定のベースライン「１’」だけ移動し又は回転したとみなす。 Next, the image processing apparatus similarly performs intra-camera processing on the image acquired by the camera 100b as the second camera. FIG. 5 is a diagram illustrating an example of an intra-camera process using the second camera in the background art. Similar to the example shown in FIG. 4, the distance to the feature points 3200a and 3200c acquired by the camera 100b is not equal to the actual distance to the feature points 3000a and 3000c. In this case, the image processing apparatus also regards the image acquired by the camera 100b as having moved or rotated by a predetermined baseline “1 ′” that is different from the distance S that the camera 100b has actually moved.
　この際、カメラ１００ａについて推定されるベースライン「１」と、カメラ１００ｂについて推定されるベースライン「１’」とが必ずしも一致するとは限らない。ベースラインが一致しない場合、カメラ１００ａにより取得された特徴点と、カメラ１００ｂにより取得された特徴点とが正しく重ならない。この場合、カメラ１００ａにより取得された特徴点により生成されたマップと、カメラ１００ｂにより取得された特徴点により生成されたマップとが重ならないことがある。 At this time, the baseline “1” estimated for the camera 100a does not always match the baseline “1 ′” estimated for the camera 100b. When the baselines do not match, the feature points acquired by the camera 100a and the feature points acquired by the camera 100b do not overlap correctly. In this case, the map generated by the feature points acquired by the camera 100a may not overlap with the map generated by the feature points acquired by the camera 100b.
　次に、本実施例における画像処理装置について説明する。本実施例においては、複数のカメラにより異なる時点において取得された３次元座標の対応関係を用いて３次元マップを生成する処理（以下において、「inter-camera処理」と表記する場合がある。）を行う。 Next, the image processing apparatus in this embodiment will be described. In the present embodiment, a process of generating a three-dimensional map using correspondence relationships of three-dimensional coordinates acquired at different times by a plurality of cameras (hereinafter, sometimes referred to as “inter-camera process”). I do.
［機能ブロック］
　まず、本実施例における画像処理システムの一例について、図６を用いて説明する。図６は、実施例１における画像処理システムの一例を示す図である。図６に示すように、本実施例における画像処理システム１は、複数のカメラ１００ａ及び１００ｂと、ＳＬＡＭ（Simultaneous　Localization　and　Mapping）装置２００と、スケール処理装置３００とを含む。各装置は、例えばインターネットなどのネットワーク、又はバス等によって相互に通信可能に接続されている。なお、スケール処理装置３００は画像処理装置の一例である。また、以下において、カメラ１００ａ及びカメラ１００ｂを区別せずに表現する場合に「カメラ１００」と表記する場合がある。また、本実施例においては、カメラ１００を２台含む構成を例示するが、これに限られず、例えば図１に示すように、画像処理システムが３台以上のカメラ１００を含むような構成であってもよい。 [Function block]
First, an example of the image processing system in the present embodiment will be described with reference to FIG. FIG. 6 is a diagram illustrating an example of an image processing system according to the first embodiment. As shown in FIG. 6, the image processing system 1 in this embodiment includes a plurality of cameras 100 a and 100 b, an SLAM (Simultaneous Localization and Mapping) device 200, and a scale processing device 300. Each device is connected to be communicable with each other via a network such as the Internet or a bus. The scale processing device 300 is an example of an image processing device. In the following description, the camera 100a and the camera 100b may be expressed as “camera 100” when they are expressed without distinction. In the present embodiment, a configuration including two cameras 100 is illustrated, but the configuration is not limited thereto. For example, as illustrated in FIG. 1, the image processing system includes three or more cameras 100. May be.
　カメラ１００は、例えば自動車などの移動体に搭載され、移動体の周辺環境を撮像する撮像装置である。カメラ１００は、例えば、撮像素子としてＣＭＯＳ（Complementary　Metal　Oxide　Semiconductor）イメージセンサまたはＣＣＤ（Charge　Coupled　Device）イメージセンサ等を用いて、画像を撮像する。カメラ１００は、撮像素子が受光した光を光電変換しＡ／Ｄ（Analog／Digital）変換を行って撮像画像を生成する。カメラ１００は、生成した撮像画像をＳＬＡＭ装置２００に送信する。 The camera 100 is an imaging device that is mounted on a moving body such as an automobile and images the surrounding environment of the moving body. The camera 100 captures an image using, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor as an image sensor. The camera 100 photoelectrically converts light received by the image sensor and performs A / D (Analog / Digital) conversion to generate a captured image. The camera 100 transmits the generated captured image to the SLAM device 200.
　ＳＬＡＭ装置２００は、カメラ１００から取得した画像を用いて特徴点を抽出し、３次元マップを生成する。図７は、実施例１における機能ブロックの一例を示す図である。図７に示すように、ＳＬＡＭ装置２００は、通信部２１０、記憶部２２０及び制御部２３０を有する。 The SLAM device 200 extracts feature points using an image acquired from the camera 100 and generates a three-dimensional map. FIG. 7 is a diagram illustrating an example of functional blocks according to the first embodiment. As illustrated in FIG. 7, the SLAM apparatus 200 includes a communication unit 210, a storage unit 220, and a control unit 230.
　通信部２１０は、カメラ１００から送信されたデータを受信し、制御部２３０に出力する。 The communication unit 210 receives the data transmitted from the camera 100 and outputs it to the control unit 230.
　記憶部２２０は、例えば、ＲＡＭ（Random　Access　Memory）、ＲＯＭ（Read　Only　Memory）、フラッシュメモリ（Flash　Memory）等の半導体メモリ素子、ハードディスクや光ディスク等の記憶装置によって実現される。記憶部２２０は、特徴点記憶部２２１と、３次元マップ記憶部２２２とを有する。また、記憶部２２０は、例えば制御部２３０が実行するプログラム及び、制御部２３０での処理に用いる各種の情報を記憶する。 The storage unit 220 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. The storage unit 220 includes a feature point storage unit 221 and a three-dimensional map storage unit 222. In addition, the storage unit 220 stores, for example, a program executed by the control unit 230 and various types of information used for processing in the control unit 230.
　特徴点記憶部２２１は、カメラ１００から取得される画像から抽出された特徴点の座標を記憶する。特徴点記憶部２２１は、カメラ１００が移動すると、後に説明する特徴マッチング部２３３により逐次更新される。 The feature point storage unit 221 stores the coordinates of feature points extracted from the image acquired from the camera 100. The feature point storage unit 221 is sequentially updated by the feature matching unit 233 described later when the camera 100 moves.
　図８は、実施例１における特徴点記憶部の一例を示す図である。図８に示すように、特徴点記憶部２２１は、「取得時刻」と、「３次元座標（第一カメラ）」と、「３次元座標（第二カメラ）」とを「特徴点ＩＤ」に対応付けて記憶する。 FIG. 8 is a diagram illustrating an example of a feature point storage unit according to the first embodiment. As illustrated in FIG. 8, the feature point storage unit 221 sets “acquisition time”, “three-dimensional coordinates (first camera)”, and “three-dimensional coordinates (second camera)” as “feature point IDs”. Store in association with each other.
　図８において、「特徴点ＩＤ」は、画像から抽出された特徴点を一意に識別する識別子（Identifier）である。「取得時刻」は、特徴点が抽出された画像が取得された時刻を記憶する。「３次元座標（第一カメラ）」は、カメラ１００ａにおいて取得時刻において取得された画像から抽出された特徴点の座標を記憶する。同様に、「３次元座標（第二カメラ）」は、カメラ１００ｂにおいて、取得時刻において取得された画像から抽出された特徴点の座標を記憶する。なお、図８においては、３次元座標は、画像を撮影したカメラごとに異なる列に記憶されるが、これに限られず、特徴点記憶部２２１が座標に対応するカメラの識別子をさらに記憶し、同一の列に３次元座標を記憶するような構成であってもよい。なお、図８に示す特徴点記憶部２２１には、同一の時点において、異なるカメラで撮影された同一の特徴点に関する情報が記憶されているが、これに加えて、同一のカメラで、異なる時点において撮影された同一の特徴点に関する情報も同様に記憶されている。 In FIG. 8, “feature point ID” is an identifier for uniquely identifying a feature point extracted from an image. The “acquisition time” stores the time when the image from which the feature points are extracted is acquired. The “three-dimensional coordinates (first camera)” stores the coordinates of feature points extracted from the image acquired at the acquisition time in the camera 100a. Similarly, “three-dimensional coordinates (second camera)” stores the coordinates of feature points extracted from the image acquired at the acquisition time in the camera 100b. In FIG. 8, the three-dimensional coordinates are stored in different columns for each camera that has captured the image. However, the present invention is not limited to this, and the feature point storage unit 221 further stores the identifier of the camera corresponding to the coordinates. The configuration may be such that three-dimensional coordinates are stored in the same column. Note that the feature point storage unit 221 shown in FIG. 8 stores information about the same feature points photographed by different cameras at the same time, but in addition to this, the same camera has different times. Information on the same feature point photographed in is also stored.
　３次元マップ記憶部２２２は、後に説明する３次元マップ生成部２３５が生成する３次元マップを記憶する。３次元マップ記憶部２２２は、カメラ１００が移動すると、３次元マップ生成部２３５により逐次更新される。 The 3D map storage unit 222 stores a 3D map generated by a 3D map generation unit 235 described later. The three-dimensional map storage unit 222 is sequentially updated by the three-dimensional map generation unit 235 when the camera 100 moves.
　図９は、実施例１における３次元マップ記憶部の一例を示す図である。図９に示すように、３次元マップ記憶部２２２は、「取得時刻」と、「３次元座標（統合後３次元マップ）」とを「特徴点ＩＤ」に対応付けて記憶する。 FIG. 9 is a diagram illustrating an example of a three-dimensional map storage unit according to the first embodiment. As illustrated in FIG. 9, the three-dimensional map storage unit 222 stores “acquisition time” and “three-dimensional coordinates (three-dimensional map after integration)” in association with “feature point IDs”.
　図９において、「３次元座標（統合後３次元マップ）」は、「３次元座標（第一カメラ）」と「３次元座標（第二カメラ）」とが、３次元マップ生成部２３５により統合された後の特徴点の座標を記憶する。 In FIG. 9, “three-dimensional coordinates (three-dimensional map after integration)” includes “three-dimensional coordinates (first camera)” and “three-dimensional coordinates (second camera)” integrated by the three-dimensional map generation unit 235. The coordinates of the feature points after being stored are stored.
　次に、制御部２３０は、例えば、ＣＰＵ（Central　Processing　Unit）やＭＰＵ（Micro　Processing　Unit）等によって、内部の記憶装置に記憶されているプログラムがＲＡＭを作業領域として実行されることにより実現される。また、制御部２３０は、例えば、ＡＳＩＣ（Application　Specific　Integrated　Circuit）やＦＰＧＡ（Field　Programmable　Gate　Array）等の集積回路により実現されるようにしてもよい。 Next, the control unit 230 is realized, for example, by executing a program stored in an internal storage device using the RAM as a work area by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. . The control unit 230 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
　図７に示すように、制御部２３０は、画像データ取得部２３１、特徴抽出部２３２、特徴マッチング部２３３、姿勢推定部２３４及び３次元マップ生成部２３５を有し、以下に説明する情報処理の機能や作用を実現または実行する。なお、制御部２３０の内部構成は、図７に示した構成に限られず、後述する情報処理を行う構成であれば他の構成であってもよい。 As shown in FIG. 7, the control unit 230 includes an image data acquisition unit 231, a feature extraction unit 232, a feature matching unit 233, a posture estimation unit 234, and a three-dimensional map generation unit 235, and performs information processing described below. Implement or execute a function or action. The internal configuration of the control unit 230 is not limited to the configuration illustrated in FIG. 7, and may be another configuration as long as the information processing described later is performed.
　画像データ取得部２３１は、カメラ１００から撮像画像が入力されると、入力された撮像画像の取得を開始する。画像データ取得部２３１は、カメラ１００で撮像された画像データを取得し、特徴抽出部２３２に出力する。なお、画像データ取得部２３１は、例えば図示しないＲＴＣ（Real-Time　Clock）を用いて、画像データが入力される正確な時刻を取得する。 When the captured image is input from the camera 100, the image data acquisition unit 231 starts acquiring the input captured image. The image data acquisition unit 231 acquires image data captured by the camera 100 and outputs the acquired image data to the feature extraction unit 232. Note that the image data acquisition unit 231 acquires an accurate time when image data is input using, for example, an RTC (Real-Time Clock) (not shown).
　特徴抽出部２３２は、画像データ取得部２３１から画像データが入力されると、複数のカメラから入力された画像データそれぞれについて、入力された画像データの候補範囲に含まれる特徴点を抽出する処理部である。例えば、特徴抽出部２３２は、Harrisのコーナ検出を行うことで、コーナ点を特定する。特徴抽出部２３２は、Harrisのコーナ検出以外のその他のコーナ点検出法に基づき、コーナ点を特定しても良い。特徴抽出部２３２は、コーナ検出法により検出したコーナ点を、特徴点として特徴点記憶部２２１の該当するカメラの列に記憶する。なお、特徴抽出部２３２が特徴点を抽出する方法はこれに限られず、例えばエッジ検出等により特徴点を抽出してもよい。 When image data is input from the image data acquisition unit 231, the feature extraction unit 232 extracts a feature point included in the input image data candidate range for each of the image data input from a plurality of cameras. It is. For example, the feature extraction unit 232 identifies a corner point by performing Harris corner detection. The feature extraction unit 232 may identify corner points based on other corner point detection methods other than Harris corner detection. The feature extraction unit 232 stores the corner points detected by the corner detection method in the corresponding camera row of the feature point storage unit 221 as feature points. Note that the method of extracting the feature points by the feature extraction unit 232 is not limited to this, and the feature points may be extracted by edge detection, for example.
　特徴マッチング部２３３は、記憶された特徴点を参照して、intra-camera処理に用いられるデータを取得する。例えば、特徴マッチング部２３３は、特徴点記憶部２２１に記憶された、異なる時点において、同一のカメラにより取得された画像から抽出された、同一の特徴点ＩＤの座標、すなわち特徴点対応を取得する。そして、特徴マッチング部２３３は、取得された異時点における特徴点対応の集合ｐintraを、姿勢推定部２３４及び３次元マップ生成部２３５に出力する。 The feature matching unit 233 acquires data used for the intra-camera process with reference to the stored feature points. For example, the feature matching unit 233 acquires the coordinates of the same feature point ID extracted from images acquired by the same camera at different time points stored in the feature point storage unit 221, that is, the feature point correspondence. . Then, the feature matching unit 233 outputs the acquired set pointtra corresponding to the feature points at different times to the posture estimation unit 234 and the three-dimensional map generation unit 235.
　また、特徴マッチング部２３３は、特徴点記憶部２２１に記憶された、同じ時点において、異なるカメラにより取得された画像から抽出された、同一の特徴点ＩＤの座標、すなわち特徴点対応を取得する。そして、特徴マッチング部２３３は、取得された同時点における特徴点対応の集合ｐinterと、取得された異時点における特徴点対応の集合ｐintraとを、スケール処理装置３００に出力する。 Also, the feature matching unit 233 acquires the coordinates of the same feature point ID, that is, the feature point correspondences extracted from images acquired by different cameras at the same time stored in the feature point storage unit 221. Then, the feature matching unit 233 outputs to the scale processing device 300 the set pinter corresponding to the feature points at the acquired simultaneous points and the acquired set pintra corresponding to the feature points at different time points.
　姿勢推定部２３４は、特徴マッチング部２３３から出力された集合ｐintraを用いて、カメラ１００の姿勢推定処理を行う。例えば、姿勢推定部２３４は、既知の５点アルゴリズムを用いて、集合ｐintraから初期位置姿勢Ｔを推定する。 The posture estimation unit 234 performs posture estimation processing of the camera 100 using the set pintra output from the feature matching unit 233. For example, the posture estimation unit 234 estimates the initial position and posture T from the set pintra using a known five-point algorithm.
　また、姿勢推定部２３４は、スケール処理装置３００から修正されたスケールの入力を受けた場合、推定した初期位置姿勢Ｔを修正する。例えば、姿勢推定部２３４は、初期位置姿勢Ｔに、入力されたスケールｓを乗じることにより、修正位置姿勢Ｔ’を算出する。 Also, when the posture estimation unit 234 receives the corrected scale input from the scale processing device 300, the posture estimation unit 234 corrects the estimated initial position and posture T. For example, the posture estimation unit 234 calculates the corrected position and posture T ′ by multiplying the initial position and posture T by the input scale s.
　また、姿勢推定部２３４は、初期位置姿勢Ｔ及び修正位置姿勢Ｔ’を、３次元マップ生成部２３５に出力する。 Also, the posture estimation unit 234 outputs the initial position / posture T and the corrected position / posture T ′ to the three-dimensional map generation unit 235.
　３次元マップ生成部２３５は、特徴マッチング部２３３から出力されたｐintraと、姿勢推定部２３４から出力された初期位置姿勢Ｔ又は修正位置姿勢Ｔ’とを用いて、３次元マップを生成する。３次元マップ生成部２３５は、生成した３次元マップの座標を、３次元マップ記憶部２２２に記憶する。 The three-dimensional map generation unit 235 generates a three-dimensional map using the pintra output from the feature matching unit 233 and the initial position / posture T or the corrected position / posture T ′ output from the posture estimation unit 234. The 3D map generation unit 235 stores the coordinates of the generated 3D map in the 3D map storage unit 222.
　次に、本実施例におけるスケール処理装置３００の機能構成について説明する。図７に示すように、本実施例におけるスケール処理装置３００は、記憶部３２０及び制御部３３０を有する。なお、スケール処理装置３００も、ＳＬＡＭ装置２００と同様に、コンピュータ等の機器により実現され、図７に示す機能部以外にも既知のコンピュータが有する各種の機能部を有することとしてもかまわない。 Next, the functional configuration of the scale processing apparatus 300 in this embodiment will be described. As shown in FIG. 7, the scale processing device 300 in this embodiment includes a storage unit 320 and a control unit 330. Note that the scale processing apparatus 300 is also realized by a device such as a computer, similarly to the SLAM apparatus 200, and may have various functional units included in a known computer in addition to the functional units illustrated in FIG.
　記憶部３２０は、制御部３３０が実行するプログラム、各種データなどを記憶する。記憶部３２０は、ＲＡＭ、ＲＯＭ、フラッシュメモリなどの半導体メモリ素子や、ＨＤＤなどの記憶装置に対応する。記憶部３２０は、相対位置姿勢記憶部３２１及びスケール記憶部３２２を有する。また、記憶部３２０は、例えば制御部３３０が実行するプログラム及び、制御部３３０での処理に用いる各種の情報を記憶する。 The storage unit 320 stores programs executed by the control unit 330, various data, and the like. The storage unit 320 corresponds to a semiconductor memory element such as a RAM, a ROM, and a flash memory, and a storage device such as an HDD. The storage unit 320 includes a relative position / posture storage unit 321 and a scale storage unit 322. Further, the storage unit 320 stores, for example, a program executed by the control unit 330 and various types of information used for processing in the control unit 330.
　相対位置姿勢記憶部３２１は、カメラ１００ａ及び１００ｂの相対的な位置及び姿勢に関する情報を記憶する。相対位置姿勢記憶部３２１に記憶される情報は、例えば図示しない画像処理システム１の管理者により、予め設定される。 The relative position / posture storage unit 321 stores information on the relative positions and postures of the cameras 100a and 100b. Information stored in the relative position and orientation storage unit 321 is set in advance by, for example, an administrator of the image processing system 1 (not shown).
　図１０は、実施例１における相対位置姿勢記憶部の一例を示す図である。図１０に示すように、相対位置姿勢記憶部３２１は、「座標」と、「角度」とを「カメラＩＤ」に対応付けて記憶する。図１０において、「カメラＩＤ」は、画像処理システム１が有するカメラを一意に識別する識別子である。「座標」及び「角度」は、カメラの位置及び姿勢と、基準となるカメラの位置及び姿勢との相違を記憶する。例えば、図１０においては、カメラＩＤ「Ｃ００１」のカメラを基準となるカメラとしている。このため、カメラＩＤ「Ｃ００１」のカメラの座標は原点（０，０，０）となり、角度も無し（０°，０°，０°）となる。また、カメラＩＤ「Ｃ００２」のカメラの座標は、原点からｘ方向に「１００ｍｍ」移動しており、角度はｘ軸方向に「５°」ずれていることを示す。 FIG. 10 is a diagram illustrating an example of the relative position and orientation storage unit according to the first embodiment. As illustrated in FIG. 10, the relative position and orientation storage unit 321 stores “coordinates” and “angles” in association with “camera IDs”. In FIG. 10, “camera ID” is an identifier for uniquely identifying a camera included in the image processing system 1. “Coordinates” and “angles” store the difference between the position and orientation of the camera and the position and orientation of the reference camera. For example, in FIG. 10, the camera with the camera ID “C001” is used as a reference camera. For this reason, the coordinates of the camera with the camera ID “C001” are the origin (0, 0, 0), and there are no angles (0 °, 0 °, 0 °). Further, the coordinates of the camera with the camera ID “C002” are moved “100 mm” in the x direction from the origin, and the angle is shifted by “5 °” in the x axis direction.
　スケール記憶部３２２は、後に説明するスケール修正部３３２により修正されたスケールを記憶する。図１１は、実施例１におけるスケール記憶部の一例を示す図である。図１１に示すように、スケール記憶部３２２は、「スケール」を「カメラＩＤ」に対応付けて記憶する。図１１において、「スケール」は、該当するカメラＩＤのカメラにより取得された座標を修正するための数値である。例えば、３次元座標にスケールを乗じることで、修正後の３次元座標が算出される。 The scale storage unit 322 stores the scale corrected by the scale correction unit 332 described later. FIG. 11 is a diagram illustrating an example of the scale storage unit according to the first embodiment. As illustrated in FIG. 11, the scale storage unit 322 stores “scale” in association with “camera ID”. In FIG. 11, “scale” is a numerical value for correcting coordinates acquired by the camera with the corresponding camera ID. For example, the corrected three-dimensional coordinates are calculated by multiplying the three-dimensional coordinates by a scale.
　次に、制御部３３０は、例えば、ＣＰＵやＭＰＵ等によって、内部の記憶装置に記憶されているプログラムがＲＡＭを作業領域として実行されることにより実現される。また、制御部３３０は、例えば、ＡＳＩＣやＦＰＧＡ等の集積回路により実現されるようにしてもよい。図７に示すように、制御部３３０は、スケール推定部３３１及びスケール修正部３３２を有し、以下に説明する情報処理の機能や作用を実現または実行する。なお、制御部３３０の内部構成は、図７に示した構成に限られず、後述する情報処理を行う構成であれば他の構成であってもよい。 Next, the control unit 330 is realized by executing a program stored in an internal storage device using the RAM as a work area, for example, by a CPU, MPU, or the like. The control unit 330 may be realized by an integrated circuit such as an ASIC or FPGA, for example. As illustrated in FIG. 7, the control unit 330 includes a scale estimation unit 331 and a scale correction unit 332, and realizes or executes information processing functions and operations described below. The internal configuration of the control unit 330 is not limited to the configuration illustrated in FIG. 7, and may be another configuration as long as the information processing described below is performed.
　スケール推定部３３１は、ＳＬＡＭ装置２００の特徴マッチング部２３３から異時点の特徴点対応の集合ｐintra及び同時点の特徴点対応の集合ｐinterが入力されると、ｐintraとｐinterとの間で、同一の３次元点を参照する特徴点対応の集合ｐ’を抽出する。 When a set pintra corresponding to feature points at different time points and a set pinter corresponding to feature points at the same time point are input from the feature matching unit 233 of the SLAM device 200, the scale estimation unit 331 is identical between the pintra and the pinter. A set p ′ corresponding to the feature point that refers to the three-dimensional point is extracted.
　次に、スケール推定部３３１は、相対位置姿勢記憶部３２１を参照し、集合ｐ’の３次元点群集合Ｍ’を算出する。例えば、スケール推定部３３１は、ｐinterのうち、カメラ１００ｂで取得された３次元座標を、相対位置姿勢記憶部３２１から取得されたカメラ１００ｂの相対的な位置姿勢を用いて補正する。そして、スケール推定部３３１は、補正された３次元座標の集合であるＭ’を算出する。 Next, the scale estimation unit 331 refers to the relative position / posture storage unit 321 and calculates a three-dimensional point group set M ′ of the set p ′. For example, the scale estimation unit 331 corrects the three-dimensional coordinates acquired by the camera 100b in the pinter using the relative position and orientation of the camera 100b acquired from the relative position and orientation storage unit 321. Then, the scale estimation unit 331 calculates M ′ that is a set of corrected three-dimensional coordinates.
　次に、スケール推定部３３１は、ｐintraから生成された初期の３次元マップの３次元点ｐの集合Ｍと、算出されたｐ’の集合であるＭ’との間で、全ての３次元点について、距離の比ｐ／ｐ’を算出する。そして、スケール推定部３３１は、算出された全ての距離の比ｐ／ｐ’の中央値を、ＭとＭ’とのスケールｓとして採用する。スケール推定部３３１は、採用されたスケールｓを、カメラＩＤと対応付けて、スケール記憶部３２２に記憶する。また、スケール推定部３３１は、カメラ１００ｂから取得された画像からintra-camera処理により生成された３次元座標ｐｂintraについても、同様にスケールｓ’を算出する。 Next, the scale estimator 331 generates all three-dimensional points between a set M of three-dimensional points p in the initial three-dimensional map generated from pintra and M ′, which is a set of calculated p ′. , The distance ratio p / p ′ is calculated. Then, the scale estimation unit 331 employs the median value of all the calculated distance ratios p / p ′ as the scale s between M and M ′. The scale estimation unit 331 stores the adopted scale s in the scale storage unit 322 in association with the camera ID. Further, the scale estimation unit 331 similarly calculates the scale s ′ for the three-dimensional coordinate pbintra generated by the intra-camera process from the image acquired from the camera 100b.
　スケール修正部３３２は、算出されたスケールｓを用いて、ｐintraから生成された初期の３次元マップＭ及び初期位置姿勢Ｔを修正する。具体的には、スケール修正部３３２は、ＳＬＡＭ装置２００から初期の３次元マップＭ及び初期位置姿勢Ｔの入力を受けると、スケール記憶部３２２に記憶されたスケールｓを乗じる。そして、スケール修正部３３２は、補正後の３次元マップＭ’及び修正位置姿勢Ｔ’を、ＳＬＡＭ装置２００に出力する。スケール修正部３３２は、それぞれのカメラ１００について、同様の処理を行う。 The scale correction unit 332 corrects the initial three-dimensional map M and the initial position / posture T generated from the pintra using the calculated scale s. Specifically, when receiving the initial three-dimensional map M and the initial position and orientation T from the SLAM device 200, the scale correction unit 332 multiplies the scale s stored in the scale storage unit 322. Then, the scale correcting unit 332 outputs the corrected three-dimensional map M ′ and the corrected position / posture T ′ to the SLAM device 200. The scale correction unit 332 performs the same process for each camera 100.
　次に、本実施例におけるinter-camera処理を用いた３次元マップを生成する処理について、図１２乃至図１５を用いて説明する。図１２は、実施例１におけるinter-camera処理の一例を示す図である。図１２に示すように、カメラ１００ａ及び１００ｂは、同一の時点ｔ０において、同一の被写体３０００をそれぞれ撮影する。本実施例におけるＳＬＡＭ装置２００の特徴抽出部２３２は、画像データが入力されると、カメラ１００ａ及び１００ｂがそれぞれ撮影した画像から、特徴点３０００ａの３次元座標を特定する。同様に、特徴抽出部２３２は、その他の特徴点についても、カメラ１００ａ及び１００ｂがそれぞれ撮影した画像のそれぞれについて座標を特定し、特徴点記憶部２２１に記憶する。 Next, processing for generating a three-dimensional map using inter-camera processing in the present embodiment will be described with reference to FIGS. FIG. 12 is a diagram illustrating an example of inter-camera processing according to the first embodiment. As shown in FIG. 12, the cameras 100a and 100b respectively photograph the same subject 3000 at the same time point t0. When the image data is input, the feature extraction unit 232 of the SLAM device 200 according to the present embodiment specifies the three-dimensional coordinates of the feature point 3000a from the images captured by the cameras 100a and 100b. Similarly, for the other feature points, the feature extraction unit 232 specifies the coordinates for each of the images photographed by the cameras 100 a and 100 b and stores them in the feature point storage unit 221.
　次に、特徴マッチング部２３３は、特定された同時点における特徴点対応を特徴点記憶部２２１から読み出し、特徴点対応の集合ｐinterをスケール処理装置３００に出力する。スケール処理装置３００のスケール推定部３３１は、入力されたｐinterに含まれる３次元点ｐの座標と、相対位置姿勢記憶部３２１に記憶されたカメラ１００ｂの位置姿勢とを用いて、補正後の座標の集合ｐ’を算出する。そして、スケール推定部３３１は、算出されたｐ’とｐとの比を用いて、スケールｓを算出し、スケール記憶部３２２に記憶する。 Next, the feature matching unit 233 reads the feature point correspondence at the specified simultaneous point from the feature point storage unit 221, and outputs the set pinter corresponding to the feature point to the scale processing device 300. The scale estimation unit 331 of the scale processing apparatus 300 uses the coordinates of the three-dimensional point p included in the input pinter and the position and orientation of the camera 100b stored in the relative position and orientation storage unit 321 to correct the coordinates. The set p ′ is calculated. Then, the scale estimation unit 331 calculates the scale s using the calculated ratio between p ′ and p, and stores it in the scale storage unit 322.
　次に、スケール修正部３３２は、ＳＬＡＭ装置２００から入力された、カメラ１００ａから取得された画像に対するintra-camera処理により算出された特徴点の３次元座標に、スケールｓを乗じて、補正後の特徴点の３次元座標を算出する。図１３は、実施例１における第一のカメラにより取得された座標の補正処理の一例を示す図である。図１３に示すように、intra-camera処理により算出された特徴点３１００ａの３次元座標をスケールｓで補正することにより、補正後の３次元座標３１１０ａは実際の被写体の特徴点３０００ａの３次元座標と重なる。 Next, the scale correction unit 332 multiplies the three-dimensional coordinates of the feature points, which are input from the SLAM device 200 and calculated by the intra-camera process on the image acquired from the camera 100a, by the scale s, and performs the correction. The three-dimensional coordinates of the feature points are calculated. FIG. 13 is a diagram illustrating an example of correction processing of coordinates acquired by the first camera in the first embodiment. As shown in FIG. 13, by correcting the three-dimensional coordinate of the feature point 3100a calculated by the intra-camera process with the scale s, the corrected three-dimensional coordinate 3110a becomes the three-dimensional coordinate of the actual feature point 3000a of the subject. And overlap.
　同様に、スケール修正部３３２は、ＳＬＡＭ装置２００から入力された、カメラ１００ｂから取得された画像に対するintra-camera処理により算出された特徴点の３次元座標についても、補正後の特徴点の３次元座標を算出する。図１４は、実施例１における第二のカメラにより取得された座標の補正処理の一例を示す図である。図１４に示すように、補正後の３次元座標３２１０ａは実際の被写体の特徴点３０００ａの３次元座標と重なり、補正後の３次元座標３２１０ｃは実際の被写体の特徴点３０００ｃの３次元座標と重なる。 Similarly, the scale correction unit 332 also applies the three-dimensional feature points after correction to the three-dimensional coordinates of the feature points calculated by the intra-camera process for the image input from the SLAM device 200 and acquired from the camera 100b. Calculate the coordinates. FIG. 14 is a diagram illustrating an example of a correction process of coordinates acquired by the second camera according to the first embodiment. As shown in FIG. 14, the corrected three-dimensional coordinates 3210a overlap with the three-dimensional coordinates of the actual feature point 3000a of the subject, and the corrected three-dimensional coordinates 3210c overlap with the three-dimensional coordinates of the feature point 3000c of the subject. .
　以上のように、全てのカメラ１００についてスケールを用いて３次元座標を補正することにより、図１５に示すような３次元マップが得られる。図１５は、実施例１において統合された三次元マップの一例を示す図である。図１５に示すように、補正後の特徴点の３次元座標３１１０ａ及び３２１０ａは、いずれも実際の被写体の特徴点３０００ａと重なるので、実スケールを反映した３次元マップを統合できる。同様に、補正後の特徴点の３次元座標３１１０ｂ及び３２１０ｃも、それぞれ実際の被写体の特徴点３０００ｂ及び３０００ｃと重なる As described above, the three-dimensional map as shown in FIG. 15 is obtained by correcting the three-dimensional coordinates using the scale for all the cameras 100. FIG. 15 is a diagram illustrating an example of the three-dimensional map integrated in the first embodiment. As shown in FIG. 15, since the corrected three- dimensional coordinates 3110a and 3210a of the feature points overlap with the actual feature points 3000a of the subject, a three-dimensional map reflecting the actual scale can be integrated. Similarly, the three- dimensional coordinates 3110b and 3210c of the feature points after correction also overlap with the feature points 3000b and 3000c of the actual subject, respectively.
［処理の流れ］
　次に、本実施例における処理の流れについて説明する。図１６は、実施例１における３次元マップ生成処理の一例を示すフローチャートである。まず、ＳＬＡＭ装置２００の画像データ取得部２３１は、第一のカメラ１００ａから画像データを取得するまで待機する（Ｓ１００：Ｎｏ）。画像データ取得部２３１は、画像データを取得すると（Ｓ１００：Ｙｅｓ）、取得した画像データを特徴抽出部２３２に出力する。特徴抽出部２３２は、入力された画像データから特徴点ｐａを抽出し、特徴点記憶部２２１に記憶する（Ｓ１０１）。 [Process flow]
Next, the flow of processing in the present embodiment will be described. FIG. 16 is a flowchart illustrating an example of a three-dimensional map generation process according to the first embodiment. First, the image data acquisition unit 231 of the SLAM apparatus 200 stands by until image data is acquired from the first camera 100a (S100: No). When acquiring the image data (S100: Yes), the image data acquisition unit 231 outputs the acquired image data to the feature extraction unit 232. The feature extraction unit 232 extracts the feature point pa from the input image data and stores it in the feature point storage unit 221 (S101).
　次に、特徴マッチング部２３３は、特徴点記憶部２２１を参照して、異時点における特徴点対応の集合ｐintraを取得し、姿勢推定部２３４及び３次元マップ生成部２３５に出力する（Ｓ１０２）。姿勢推定部２３４は、特徴マッチング部２３３から出力されたｐintraを用いて、カメラ１００ａの初期位置姿勢Ｔを推定して３次元マップ生成部２３５に出力する（Ｓ１０３）。また、３次元マップ生成部２３５は、特徴マッチング部２３３から出力されたｐintraと、姿勢推定部２３４から出力された初期位置姿勢Ｔとを用いて、初期３次元マップＭｉを生成し、３次元マップ記憶部２２２に記憶する（Ｓ１０４）。 Next, the feature matching unit 233 refers to the feature point storage unit 221, obtains a set pintra corresponding to feature points at different time points, and outputs them to the posture estimation unit 234 and the three-dimensional map generation unit 235 (S102). The posture estimation unit 234 estimates the initial position and posture T of the camera 100a using the pintra output from the feature matching unit 233, and outputs the estimated initial position / posture T to the three-dimensional map generation unit 235 (S103). Further, the three-dimensional map generation unit 235 generates an initial three-dimensional map Mi using the pintra output from the feature matching unit 233 and the initial position / posture T output from the posture estimation unit 234. It memorize | stores in the memory | storage part 222 (S104).
　次に、画像データ取得部２３１は、第二のカメラ１００ｂから画像データを取得するまで待機する（Ｓ１１０：Ｎｏ）。画像データ取得部２３１は、画像データを取得すると（Ｓ１１０：Ｙｅｓ）、取得した画像データを特徴抽出部２３２に出力する。特徴抽出部２３２は、入力された画像データから特徴点ｐｂを抽出し、特徴点記憶部２２１に記憶する（Ｓ１１１）。 Next, the image data acquisition unit 231 waits until image data is acquired from the second camera 100b (S110: No). When acquiring the image data (S110: Yes), the image data acquisition unit 231 outputs the acquired image data to the feature extraction unit 232. The feature extraction unit 232 extracts the feature point pb from the input image data and stores it in the feature point storage unit 221 (S111).
　次に、特徴マッチング部２３３は、特徴点記憶部２２１を参照して、同時点におけるｐａとｐｂとの特徴点対応の集合ｐinterを取得する。そして、特徴マッチング部２３３は、取得したｐintra及びｐinterを、スケール処理装置３００に出力する（Ｓ１１２）。 Next, the feature matching unit 233 refers to the feature point storage unit 221 and acquires a set pinter corresponding to feature points of pa and pb at the same point. Then, the feature matching unit 233 outputs the acquired pintra and pinter to the scale processing device 300 (S112).
　スケール処理装置３００のスケール推定部３３１は、ＳＬＡＭ装置２００からｐintra及びｐinterの入力を受けると、ｐintraとｐinterとの間で、同一の３次元点を参照する特徴点対応の集合ｐ’を抽出する（Ｓ１１３）。次に、スケール推定部３３１は、相対位置姿勢記憶部３２１に記憶された相対位置姿勢を用いて、集合ｐ’の３次元点群集合Ｍ’を算出する（Ｓ１１４）。 When the scale estimation unit 331 of the scale processing apparatus 300 receives the input of the pintra and the pinter from the SLAM apparatus 200, the scale estimation unit 331 extracts a set p ′ corresponding to the feature point that refers to the same three-dimensional point between the pintra and the pinter. (S113). Next, the scale estimation unit 331 calculates the three-dimensional point group set M ′ of the set p ′ using the relative position and orientation stored in the relative position and orientation storage unit 321 (S <b> 114).
　次に、スケール推定部３３１は、ｐintraから生成された初期の３次元マップの３次元点ｐの集合Ｍを取得する（Ｓ１１５）。そして、スケール推定部３３１は、算出されたＭ’とＭとを用いて、スケールｓを算出し、スケール記憶部３２２に記憶する（Ｓ１１６）。 Next, the scale estimation unit 331 acquires a set M of three-dimensional points p of the initial three-dimensional map generated from pintra (S115). Then, the scale estimation unit 331 calculates the scale s using the calculated M ′ and M, and stores it in the scale storage unit 322 (S116).
　そして、スケール修正部３３２は、スケールｓを用いて、Ｍｉ’をｓ倍した補正後の３次元マップＭ’ｉを算出する（Ｓ１１７）。また、スケール修正部３３２は、スケールｓを用いて、初期位置姿勢Ｔをｓ倍した修正位置姿勢Ｔ’を算出する（Ｓ１１８）。そして、スケール修正部３３２は、算出した補正後の３次元マップＭ’ｉ及び修正位置姿勢Ｔ’をＳＬＡＭ装置２００に出力する。 Then, the scale correcting unit 332 calculates a corrected three-dimensional map M′i obtained by multiplying Mi ′ by s using the scale s (S117). In addition, the scale correction unit 332 calculates a corrected position / posture T ′ obtained by multiplying the initial position / posture T by s using the scale s (S118). Then, the scale correcting unit 332 outputs the calculated corrected three-dimensional map M′i and the corrected position / posture T ′ to the SLAM device 200.
　補正後の３次元マップＭ’ｉ及び修正位置姿勢Ｔ’の出力を受けた３次元マップ生成部２３５は、以下に説明するＳ１２２の処理を、ｎ個のカメラ１００から取得された座標全てを対象に繰り返す（Ｓ１２１）。 The three-dimensional map generation unit 235 that has received the output of the corrected three-dimensional map M′i and the corrected position / posture T ′ applies the processing of S122 described below to all the coordinates acquired from the n cameras 100. (S121).
　３次元マップ生成部２３５は、修正位置姿勢Ｔ’を用いて、３次元マップＭｋを順次生成する（Ｓ１２２）。３次元マップ生成部２３５は、全てのカメラ１００から取得された座標について処理が完了するまで、Ｓ１２１に戻って処理を繰り返す（Ｓ１２３：Ｎｏ）。 The three-dimensional map generation unit 235 sequentially generates the three-dimensional map Mk using the corrected position / posture T ′ (S122). The three-dimensional map generation unit 235 returns to S121 and repeats the processing until the processing for the coordinates acquired from all the cameras 100 is completed (S123: No).
　３次元マップ生成部２３５は、全てのカメラ１００から取得された座標についてＳ１２２の処理を完了した場合（Ｓ１２３：Ｙｅｓ）、生成した全ての３次元マップを重ね合わせて、３次元マップ記憶部２２２に記憶する（Ｓ１２４）。 When the process of S122 is completed for the coordinates acquired from all the cameras 100 (S123: Yes), the 3D map generation unit 235 superimposes all the generated 3D maps in the 3D map storage unit 222. Store (S124).
［効果］
　以上説明したように、本実施例におけるデータ送受信方法は、送信装置が、送信対象となるデータを複数の分割データに分割し、複数の分割データのそれぞれにつき、分割データと自装置が保持する共通鍵とから一次検証データを生成する。そして、送信装置が、生成された各一次検証データを含む検証用送信データを、受信装置に送信した後に、複数の分割データを受信装置に送信する。また、本実施例におけるデータ送受信方法は、受信装置が、検証用送信データを受信し、分割データを受信すると、分割データと自装置が保持する共通鍵とから検証データを生成する。そして、受信装置が、生成した検証データと、検証用送信データに含まれる分割データに対応する各一次検証データと、の比較結果に応じて、分割データの正当性を判定する。これにより、受信装置は、実スケールを反映した３次元マップを統合できる。 [effect]
As described above, in the data transmission / reception method according to the present embodiment, the transmission apparatus divides the data to be transmitted into a plurality of pieces of divided data, and the divided data and the own apparatus hold each of the plurality of pieces of divided data. Generate primary verification data from the key. Then, after transmitting the transmission data for verification including the generated primary verification data to the receiving device, the transmitting device transmits a plurality of pieces of divided data to the receiving device. Also, in the data transmission / reception method according to the present embodiment, when the receiving device receives the verification transmission data and receives the divided data, the verification data is generated from the divided data and the common key held by the own device. Then, the receiving device determines the validity of the divided data according to the comparison result between the generated verification data and each primary verification data corresponding to the divided data included in the verification transmission data. Thereby, the receiving apparatus can integrate the three-dimensional map reflecting the actual scale.
　さて、これまで本発明の実施例について説明したが、本発明は上述した実施例以外にも、種々の異なる形態にて実施されてよいものである。 The embodiments of the present invention have been described so far, but the present invention may be implemented in various different forms other than the above-described embodiments.
［装置構成］
　例えば、上記実施例においては、ＳＬＡＭ装置２００とスケール処理装置３００とをそれぞれ説明したが、ＳＬＡＭ装置２００が、スケール処理装置３００の機能を併せ持つような構成であってもよい。また、カメラ１００ａ及び１００ｂと、ＳＬＡＭ装置２００と、スケール処理装置３００とが、自動車などの一つの移動体に実装されるような構成であってもよい。これにより、例えば自動車の運転者に対して、リアルタイムでずれのない３次元マップを提供することができる。 [Device configuration]
For example, although the SLAM device 200 and the scale processing device 300 have been described in the above embodiment, the SLAM device 200 may be configured to have the functions of the scale processing device 300. Further, the camera 100a and 100b, the SLAM device 200, and the scale processing device 300 may be configured to be mounted on one moving body such as an automobile. Thereby, for example, a three-dimensional map without deviation can be provided in real time to a driver of a car.
［座標の重み付け］
　また、上記実施例においては、全ての３次元点対応について距離の比ｐ／ｐ’を算出して、その中間値をスケールｓ’として採用する構成について説明したが、実施の形態はこれに限られない。例えば、距離の比ｐ／ｐ’を、算出されたカメラ１００からｐ又はｐ’までの距離により重み付けをした値の中間値や平均値をスケールとして採用するような構成であってもよい。これにより、例えばカメラ１００からの距離が近くintra-camera処理で得られる３次元座標のずれが小さい３次元点のスケールへの寄与度が高まるので、より正確なスケールを得ることができる。 [Weight of coordinates]
In the above-described embodiment, the configuration has been described in which the distance ratio p / p ′ is calculated for all three-dimensional points and the intermediate value is adopted as the scale s ′. However, the embodiment is not limited thereto. I can't. For example, a configuration may be adopted in which an intermediate value or an average value of values obtained by weighting the distance ratio p / p ′ by the calculated distance from the camera 100 to p or p ′ is used as a scale. As a result, for example, the degree of contribution to the scale of a three-dimensional point having a small distance from the camera 100 and a small three-dimensional coordinate shift obtained by the intra-camera process increases, so that a more accurate scale can be obtained.
［処理を行うタイミング］
　また、実施例１における３次元マップ生成処理は、例えば画像データの取得を開始した際に行うことができるが、処理を行うタイミングはこれに限られない。例えば１秒ごとのように所定の間隔にて処理を実施してもよく、また取得された３次元座標の数が所定の閾値以上となったときやキーフレームが表示されるタイミングに処理を実施してもよい。さらに、外部からの衝撃などでカメラ１００ａ及びカメラ１００ｂの位置姿勢が変わった場合など、カメラ１００の位置に関する情報が変更された場合に処理を実施してもよい。また、カメラ１００ａ及びカメラ１００ｂの移動速度又は回転角度が変更された場合に処理を実施してもよく、処理が終わる都度、繰り返し処理を実施してもよい。これにより、最適なタイミングで３次元マップのずれを補正できる。 [When to process]
In addition, the three-dimensional map generation process according to the first embodiment can be performed, for example, when acquisition of image data is started, but the timing for performing the process is not limited to this. For example, processing may be performed at predetermined intervals such as every second, and processing is performed when the number of acquired three-dimensional coordinates exceeds a predetermined threshold or when a key frame is displayed. May be. Furthermore, the processing may be performed when information about the position of the camera 100 is changed, such as when the position and orientation of the camera 100a and the camera 100b change due to an external impact or the like. Further, the process may be performed when the moving speed or the rotation angle of the camera 100a and the camera 100b is changed, and the process may be repeated every time the process is finished. Thereby, the shift | offset | difference of a three-dimensional map can be correct | amended with the optimal timing.
［スケールの大小］
　なお、上記実施例においては、intra-camera処理により得られる特徴点の座標が、実際の距離よりも遠くであると判定される例について説明したが、実際にはintra-camera処理により得られる特徴点の座標が実際の距離よりも近くであると判定される場合もある。この場合は、例えばスケールｓが１よりも大きくなるよう算出することにより、intra-camera処理により得られる特徴点の座標が、実際の距離に近づけることができる。 [Scale size]
In the above embodiment, the example in which the coordinates of the feature point obtained by the intra-camera process is determined to be farther than the actual distance has been described. However, the feature obtained by the intra-camera process is actually obtained. It may be determined that the coordinates of the point are closer than the actual distance. In this case, for example, by calculating so that the scale s is larger than 1, the coordinates of the feature points obtained by the intra-camera process can be brought closer to the actual distance.
［システム］
　また、図示した各部の各構成要素は機能概念的なものであり、必ずしも物理的に図示の如く構成されていることを要しない。すなわち、各部の分散・統合の具体的形態は図示のものに限られず、その全部または一部を、各種の負荷や使用状況等に応じて、任意の単位で機能的または物理的に分散・統合して構成することができる。さらに、各装置で行われる各種処理機能は、ＣＰＵ（またはＭＰＵ、ＭＣＵ（Micro　Controller　Unit）等のマイクロ・コンピュータ）上で、その全部または任意の一部を実行するようにしてもよい。また、各種処理機能は、ＣＰＵ（またはＭＰＵ、ＭＣＵ等のマイクロ・コンピュータ）で解析実行されるプログラム上、またはワイヤードロジックによるハードウェア上で、その全部または任意の一部を実行するようにしてもよいことは言うまでもない。 [system]
Each component of each part shown in the figure is functionally conceptual and does not necessarily need to be physically configured as shown. In other words, the specific form of distribution / integration of each unit is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed / integrated in arbitrary units according to various loads or usage conditions. Can be configured. Furthermore, various processing functions performed by each device may be executed entirely or arbitrarily on a CPU (or a microcomputer such as an MPU or MCU (Micro Controller Unit)). In addition, various processing functions may be executed in whole or in any part on a program that is analyzed and executed by a CPU (or a microcomputer such as an MPU or MCU) or on hardware based on wired logic. Needless to say, it is good.
　また、図示した各部の各構成要素は、必ずしも物理的に図示の如く構成されていることを要しない。すなわち、各部の分散・統合の具体的形態は図示のものに限られず、その全部または一部を、各種の負荷や使用状況等に応じて、任意の単位で機能的または物理的に分散・統合して構成することができる。例えば、画像データ取得部２３１と特徴抽出部２３２とを統合してもよい。また、図示した各処理は、上記の順番に限定されるものでなく、処理内容を矛盾させない範囲において、同時に実施してもよく、順序を入れ替えて実施してもよい。 In addition, each component of each part illustrated does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each unit is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed / integrated in arbitrary units according to various loads or usage conditions. Can be configured. For example, the image data acquisition unit 231 and the feature extraction unit 232 may be integrated. Further, the illustrated processes are not limited to the above-described order, and may be performed at the same time as long as the processing contents do not contradict each other, or may be performed by changing the order.
　さらに、各装置で行われる各種処理機能は、ＣＰＵ（またはＭＰＵ、ＭＣＵ（Micro　Controller　Unit）等のマイクロ・コンピュータ）上で、その全部または任意の一部を実行するようにしてもよい。また、各種処理機能は、ＣＰＵ（またはＭＰＵ、ＭＣＵ等のマイクロ・コンピュータ）で解析実行されるプログラム上、またはワイヤードロジックによるハードウェア上で、その全部または任意の一部を実行するようにしてもよいことは言うまでもない。 Furthermore, various processing functions performed by each device may be executed entirely or arbitrarily on a CPU (or a microcomputer such as MPU or MCU (Micro Controller Unit)). In addition, various processing functions may be executed in whole or in any part on a program that is analyzed and executed by a CPU (or a microcomputer such as an MPU or MCU) or on hardware based on wired logic. Needless to say, it is good.
　ところで、上記の実施例で説明した各種の処理は、予め用意されたプログラムをコンピュータで実行することで実現できる。そこで、以下では、上記の実施例と同様の機能を有するプログラムを実行するコンピュータの一例を説明する。図１７は、情報処理プログラムを実行するコンピュータの一例を示す図である。なお、以下においてはスケール処理装置３００と同等の機能を有するコンピュータの一例について説明するが、ＳＬＡＭ装置２００及び移動体１０についても、同様の構成により実現できる。 Incidentally, the various processes described in the above embodiments can be realized by executing a program prepared in advance on a computer. Therefore, in the following, an example of a computer that executes a program having the same function as in the above embodiment will be described. FIG. 17 is a diagram illustrating an example of a computer that executes an information processing program. In the following, an example of a computer having a function equivalent to that of the scale processing device 300 will be described, but the SLAM device 200 and the moving body 10 can also be realized by the same configuration.
　図１７に示すように、コンピュータ９００は、各種演算処理を実行するＣＰＵ９０１と、データ入力を受け付ける入力装置９０２と、モニタ９０３とを有する。また、コンピュータ９００は、記憶媒体からプログラム等を読み取る媒体読取装置９０４と、各種装置と接続するためのインタフェース装置９０５と、他の情報処理装置等と有線または無線により接続するための通信装置９０６とを有する。また、コンピュータ９００は、各種情報を一時記憶するＲＡＭ９０７と、ハードディスク装置９０８とを有する。また、各装置９０１～９０８は、バス９０９に接続される。 As illustrated in FIG. 17, the computer 900 includes a CPU 901 that executes various arithmetic processes, an input device 902 that receives data input, and a monitor 903. The computer 900 also includes a medium reading device 904 that reads a program and the like from a storage medium, an interface device 905 for connecting to various devices, and a communication device 906 for connecting to other information processing devices and the like by wire or wirelessly. Have The computer 900 also includes a RAM 907 that temporarily stores various information and a hard disk device 908. Each device 901 to 908 is connected to a bus 909.
　ハードディスク装置９０８には、図７に示したスケール推定部３３１及びスケール修正部３３２の各処理部と同様の機能を有する情報処理プログラムが記憶される。また、ハードディスク装置９０８には、相対位置姿勢記憶部３２１及びスケール記憶部３２２、並びに情報処理プログラムを実現するための各種データが記憶される。入力装置９０２は、例えば、コンピュータ９００のユーザから操作情報等の各種情報の入力を受け付ける。モニタ９０３は、例えば、コンピュータ９００のユーザに対して表示画面等の各種画面を表示する。インタフェース装置９０５は、例えばカメラ等が接続される。通信装置９０６は、例えば、図７に示した通信部２１０と同様の機能を有し、カメラ１００と各種情報をやりとりする。 The hard disk device 908 stores an information processing program having the same functions as the processing units of the scale estimation unit 331 and the scale correction unit 332 shown in FIG. Also, the hard disk device 908 stores a relative position and orientation storage unit 321 and a scale storage unit 322, and various data for realizing an information processing program. The input device 902 receives input of various information such as operation information from a user of the computer 900, for example. The monitor 903 displays various screens such as a display screen for the user of the computer 900, for example. For example, a camera or the like is connected to the interface device 905. The communication device 906 has the same function as the communication unit 210 illustrated in FIG. 7, for example, and exchanges various types of information with the camera 100.
　ＣＰＵ９０１は、ハードディスク装置９０８に記憶された各プログラムを読み出して、ＲＡＭ９０７に展開して実行することで、各種の処理を行う。また、これらのプログラムは、コンピュータ９００を図７に示したスケール推定部３３１及びスケール修正部３３２として機能させることができる。 The CPU 901 reads out each program stored in the hard disk device 908, develops it in the RAM 907, and executes it to perform various processes. In addition, these programs can cause the computer 900 to function as the scale estimation unit 331 and the scale correction unit 332 illustrated in FIG.
　なお、上記の情報処理プログラムは、必ずしもハードディスク装置９０８に記憶されている必要はない。例えば、コンピュータ９００が読み取り可能な記憶媒体に記憶されたプログラムを、コンピュータ９００が読み出して実行するようにしてもよい。コンピュータ９００が読み取り可能な記憶媒体は、例えば、ＣＤ－ＲＯＭやＤＶＤディスク、ＵＳＢ（Universal　Serial　Bus）メモリ等の可搬型記録媒体、フラッシュメモリ等の半導体メモリ、ハードディスクドライブ等が対応する。また、公衆回線、インターネット、ＬＡＮ等に接続された装置にこの情報処理プログラムを記憶させておき、コンピュータ９００がこれらから情報処理プログラムを読み出して実行するようにしてもよい。 Note that the above information processing program is not necessarily stored in the hard disk device 908. For example, the computer 900 may read and execute a program stored in a storage medium readable by the computer 900. The storage medium readable by the computer 900 corresponds to, for example, a portable recording medium such as a CD-ROM, a DVD disk, a USB (Universal Serial Bus) memory, a semiconductor memory such as a flash memory, a hard disk drive, and the like. Alternatively, the information processing program may be stored in a device connected to a public line, the Internet, a LAN, or the like, and the computer 900 may read out and execute the information processing program therefrom.
　１　画像処理システム
　１００ａ、１００ｂ、１００ｃ　カメラ
　２００　ＳＬＡＭ装置
　２１０　通信部
　２２０、３２０　記憶部
　２２１　特徴点記憶部
　２２２　３次元マップ記憶部
　３２１　相対位置姿勢記憶部
　３２２　スケール記憶部
　２３０、３３０　制御部
　２３１　画像データ取得部
　２３２　特徴抽出部
　２３３　特徴マッチング部
　２３４　姿勢推定部
　２３５　３次元マップ生成部
　３３１　スケール推定部
　３３２　スケール修正部 DESCRIPTION OF SYMBOLS 1 Image processing system 100a, 100b, 100c Camera 200 SLAM apparatus 210 Communication part 220, 320 Storage part 221 Feature point storage part 222 Three-dimensional map storage part 321 Relative position and orientation storage part 322 Scale storage part 230, 330 Control part 231 Image data Acquisition unit 232 Feature extraction unit 233 Feature matching unit 234 Posture estimation unit 235 3D map generation unit 331 Scale estimation unit 332 Scale correction unit

Claims (9)

  1. 　第一および第二のカメラにより撮影された被写体の画像において、前記第一および第二のカメラが第一のタイミングにて撮影した画像のそれぞれから対応づけられた特徴点と、記憶部に記憶された前記第一および第二のカメラの位置に関する情報とに基づいて、前記第一又は第二のカメラから被写体の特徴点までの距離を算出し、
    　前記第一のカメラが第一および第二のタイミングにて撮影した画像のそれぞれから対応付けられた同一の特徴点を用いて生成された、前記第一のカメラから前記被写体までの距離に関する情報を、算出された前記第一又は第二のカメラから前記被写体までの距離を基にして修正する画像処理装置。     In the image of the subject photographed by the first and second cameras, the feature points associated with each of the images photographed by the first and second cameras at the first timing are stored in the storage unit. And calculating the distance from the first or second camera to the feature point of the subject based on the information on the position of the first and second cameras,
    Information regarding the distance from the first camera to the subject generated using the same feature points associated with each of the images taken by the first camera at the first and second timings. An image processing apparatus for correcting based on the calculated distance from the first or second camera to the subject.
  2. 　修正された前記第一のカメラから前記被写体までの距離に関する情報を用いて、前記対応付けられた同一の特徴点の３次元座標を修正し、
    　修正された前記３次元座標を用いて３次元マップを生成する
    　ことを特徴とする請求項１に記載の画像処理装置。     Using the corrected information about the distance from the first camera to the subject, the three-dimensional coordinates of the same feature points that are associated are corrected,
    The image processing apparatus according to claim 1, wherein a three-dimensional map is generated using the corrected three-dimensional coordinates.
  3. 　前記第一のカメラにより前記第一のタイミングにて撮影された画像の特徴点までの距離と、前記第二のカメラにより前記第一のタイミングにて撮影された、前記特徴点に対応する画像の特徴点までの距離との比を算出し、
    　算出された複数の前記比の中央値を用いて、前記第一又は第二のカメラから前記被写体までの距離を算出することを特徴とする、請求項１に記載の画像処理装置。     The distance to the feature point of the image taken at the first timing by the first camera and the image corresponding to the feature point taken at the first timing by the second camera Calculate the ratio to the distance to the feature point,
    The image processing apparatus according to claim 1, wherein a distance from the first or second camera to the subject is calculated using a calculated median value of the plurality of ratios.
  4. 　前記比を、前記第一のカメラ又は前記第二のカメラにより前記第一のタイミングにて撮影された、前記画像の特徴点までの距離により重み付けした値を算出し、
    　算出された前記重み付けした値の中央値を、前記第一又は第二のカメラから前記被写体までの距離を算出することを特徴とする、請求項３に記載の画像処理装置。     Calculating the value weighted by the distance to the feature point of the image captured at the first timing by the first camera or the second camera;
    The image processing apparatus according to claim 3, wherein a distance from the first or second camera to the subject is calculated based on the median value of the calculated weighted values.
  5. 　前記修正する処理を、所定の時間ごと、前記特徴点の３次元座標の数が所定の閾値以上となったとき、キーフレーム表示のタイミング、前記第一及び第二のカメラの位置に関する情報が変更されたとき、前記第一及び第二のカメラの移動速度が変更されたとき及び前記修正する処理が終わったタイミングのうち少なくともいずれかにおいて実行することを特徴とする請求項１に記載の画像処理装置。 When the number of the three-dimensional coordinates of the feature points exceeds a predetermined threshold every predetermined time, the information regarding the timing of the key frame display and the positions of the first and second cameras is changed. 2. The image processing according to claim 1, wherein the image processing is executed at least one of when the moving speed of the first and second cameras is changed and when the correction process is finished. apparatus.
  6. 　第一および第二のカメラにより撮影された被写体の画像において、前記第一および第二のカメラが第一のタイミングにて撮影した画像のそれぞれから対応づけられた特徴点と、記憶部に記憶された前記第一および第二のカメラの位置に関する情報とに基づいて、前記第一又は第二のカメラから被写体の特徴点までの距離を算出し、
    　前記第一のカメラが第一および第二のタイミングにて撮影した画像のそれぞれから対応付けられた同一の特徴点を用いて生成された、前記第一のカメラから前記被写体までの距離に関する情報を、算出された前記第一又は第二のカメラから前記被写体までの距離を基にして修正する処理をコンピュータが実行する画像処理方法。     In the image of the subject photographed by the first and second cameras, the feature points associated with each of the images photographed by the first and second cameras at the first timing are stored in the storage unit. And calculating the distance from the first or second camera to the feature point of the subject based on the information on the position of the first and second cameras,
    Information regarding the distance from the first camera to the subject generated using the same feature points associated with each of the images taken by the first camera at the first and second timings. An image processing method in which a computer executes a process of correcting based on the calculated distance from the first or second camera to the subject.
  7. 　第一および第二のカメラにより撮影された被写体の画像において、前記第一および第二のカメラが第一のタイミングにて撮影した画像のそれぞれから対応づけられた特徴点と、記憶部に記憶された前記第一および第二のカメラの位置に関する情報とに基づいて、前記第一又は第二のカメラから被写体の特徴点までの距離を算出し、
    　前記第一のカメラが第一および第二のタイミングにて撮影した画像のそれぞれから対応付けられた同一の特徴点を用いて生成された、前記第一のカメラから前記被写体までの距離に関する情報を、算出された前記第一又は第二のカメラから前記被写体までの距離を基にして修正する処理をコンピュータに実行させる画像処理プログラム。     In the image of the subject photographed by the first and second cameras, the feature points associated with each of the images photographed by the first and second cameras at the first timing are stored in the storage unit. And calculating the distance from the first or second camera to the feature point of the subject based on the information on the position of the first and second cameras,
    Information regarding the distance from the first camera to the subject generated using the same feature points associated with each of the images taken by the first camera at the first and second timings. An image processing program for causing a computer to execute a correction process based on the calculated distance from the first or second camera to the subject.
  8. 　第一のカメラおよび第二のカメラを有する移動体が実行する画像撮影方法において、
    　前記第一および第二のカメラが被写体を撮影した画像を取得し、
    　前記第一および第二のカメラが第一のタイミングにて撮影した画像のそれぞれから同一の特徴点を対応づけ、
    　対応づけられた特徴点と、記憶部に記憶された前記第一および第二のカメラの位置に関する情報とに基づいて、前記第一又は第二のカメラから被写体の特徴点までの距離を算出し、
    　前記第一のカメラが第一および第二のタイミングにて撮影した画像のそれぞれから、同一の特徴点を対応付けると共に、対応付けられた前記特徴点を用いて前記第一のカメラから前記被写体までの距離に関する情報を生成し、
    　算出された前記第一又は第二のカメラから前記被写体までの距離を基にして、前記第一のカメラから前記被写体までの距離に関する情報を修正する画像撮影方法。     In an image capturing method executed by a moving body having a first camera and a second camera,
    The first and second cameras obtain images of the subject,
    Corresponding the same feature point from each of the images taken at the first timing by the first and second cameras,
    Based on the associated feature points and information on the positions of the first and second cameras stored in the storage unit, a distance from the first or second camera to the feature point of the subject is calculated. ,
    The same feature point is associated with each of the images taken by the first camera at the first and second timings, and the first camera to the subject is used using the associated feature point. Generate information about distance,
    An image capturing method for correcting information relating to a distance from the first camera to the subject based on the calculated distance from the first or second camera to the subject.
  9. 　第一のカメラおよび第二のカメラを有する移動体であって、
    　前記第一および第二のカメラが被写体を撮影した画像を取得し、
    　前記第一および第二のカメラが第一のタイミングにて撮影した画像のそれぞれから同一の特徴点を対応づけ、
    　対応づけられた特徴点と、記憶部に記憶された前記第一および第二のカメラの位置に関する情報とに基づいて、自装置から被写体の特徴点までの距離を算出し、
    　前記第一のカメラが第一および第二のタイミングにて撮影した画像のそれぞれから、同一の特徴点を対応付けると共に、対応付けられた前記特徴点を用いて前記第一のカメラから前記被写体までの距離に関する情報を生成し、
    　算出された前記自装置から前記被写体までの距離を基にして、前記第一のカメラから前記被写体までの距離に関する情報を修正する、移動体。     A moving body having a first camera and a second camera,
    The first and second cameras obtain images of the subject,
    Corresponding the same feature point from each of the images taken at the first timing by the first and second cameras,
    Based on the associated feature points and information on the positions of the first and second cameras stored in the storage unit, the distance from the device to the feature points of the subject is calculated,
    The same feature point is associated with each of the images taken by the first camera at the first and second timings, and the first camera to the subject is used using the associated feature point. Generate information about distance,
    A moving body that corrects information related to the distance from the first camera to the subject based on the calculated distance from the device to the subject.
PCT/JP2017/003488 2017-01-31 2017-01-31 Image processing device, image processing method, image processing program, image capturing method, and mobile object WO2018142493A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2017/003488 WO2018142493A1 (en) 2017-01-31 2017-01-31 Image processing device, image processing method, image processing program, image capturing method, and mobile object

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2017/003488 WO2018142493A1 (en) 2017-01-31 2017-01-31 Image processing device, image processing method, image processing program, image capturing method, and mobile object

Publications (1)

Publication Number Publication Date
WO2018142493A1 true WO2018142493A1 (en) 2018-08-09

Family

ID=63039396

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2017/003488 WO2018142493A1 (en) 2017-01-31 2017-01-31 Image processing device, image processing method, image processing program, image capturing method, and mobile object

Country Status (1)

Country Link
WO (1) WO2018142493A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110673947A (en) * 2019-08-12 2020-01-10 江苏博人文化科技有限公司 Method for reducing memory required by laser slam graph building

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005049999A (en) * 2003-07-30 2005-02-24 Ricoh Co Ltd Image input system, image input method, program described to execute method on information processing device, and storage medium with program stored
JP2007263657A (en) * 2006-03-28 2007-10-11 Denso It Laboratory Inc Three-dimensional coordinates acquisition system
WO2012172870A1 (en) * 2011-06-14 2012-12-20 日産自動車株式会社 Distance measurement device and environment map generation apparatus
JP2016516249A (en) * 2014-02-20 2016-06-02 エヌイーシー ラボラトリーズ アメリカ インクＮＥＣ Ｌａｂｏｒａｔｏｒｉｅｓ Ａｍｅｒｉｃａ， Ｉｎｃ． Positioning of moving objects in 3D using a single camera
US20160314593A1 (en) * 2015-04-21 2016-10-27 Hexagon Technology Center Gmbh Providing a point cloud using a surveying instrument and a camera device

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005049999A (en) * 2003-07-30 2005-02-24 Ricoh Co Ltd Image input system, image input method, program described to execute method on information processing device, and storage medium with program stored
JP2007263657A (en) * 2006-03-28 2007-10-11 Denso It Laboratory Inc Three-dimensional coordinates acquisition system
WO2012172870A1 (en) * 2011-06-14 2012-12-20 日産自動車株式会社 Distance measurement device and environment map generation apparatus
JP2016516249A (en) * 2014-02-20 2016-06-02 エヌイーシー ラボラトリーズ アメリカ インクＮＥＣ Ｌａｂｏｒａｔｏｒｉｅｓ Ａｍｅｒｉｃａ， Ｉｎｃ． Positioning of moving objects in 3D using a single camera
US20160314593A1 (en) * 2015-04-21 2016-10-27 Hexagon Technology Center Gmbh Providing a point cloud using a surveying instrument and a camera device

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
OZOG, PAUL ET AL.: "On the Importance of Modeling Camera Calibration Uncertainty in Visual SLAM", 2013 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION (ICRA, 6 May 2013 (2013-05-06), pages 3777 - 3784, XP032506698, ISSN: 1050-4729 *
vol. 2014-CVT, no. 2, pages 1 - 8 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110673947A (en) * 2019-08-12 2020-01-10 江苏博人文化科技有限公司 Method for reducing memory required by laser slam graph building
CN110673947B (en) * 2019-08-12 2022-04-05 江苏博人文化科技有限公司 Method for reducing memory required by laser slam graph building

Similar Documents

Publication Publication Date Title
EP3665506B1 (en) Apparatus and method for generating a representation of a scene
JP2017531976A (en) System and method for dynamically calibrating an array camera
KR101617078B1 (en) Apparatus and method for image matching unmanned aerial vehicle image with map image
EP3100234A1 (en) Data-processing system and method for calibration of a vehicle surround view system
JP6566768B2 (en) Information processing apparatus, information processing method, and program
JP2008506953A5 (en)
CN105335955A (en) Object detection method and object detection apparatus
CN108734738B (en) Camera calibration method and device
CN106908764B (en) Multi-target optical tracking method
JP4132068B2 (en) Image processing apparatus, three-dimensional measuring apparatus, and program for image processing apparatus
JP2014186004A (en) Measurement device, method and program
TWI820246B (en) Apparatus with disparity estimation, method and computer program product of estimating disparity from a wide angle image
JP5369873B2 (en) Judgment program and calibration device
JP2013246606A (en) Data derivation device and data derivation method
CN106683133B (en) Method for obtaining target depth image
TWI571099B (en) Device and method for depth estimation
WO2018142493A1 (en) Image processing device, image processing method, image processing program, image capturing method, and mobile object
CN107403448B (en) Cost function generation method and cost function generation device
US9538161B2 (en) System and method for stereoscopic photography
US11166005B2 (en) Three-dimensional information acquisition system using pitching practice, and method for calculating camera parameters
EP3343511A1 (en) Image processing apparatus and image processing method
CN115797466A (en) Rapid three-dimensional space calibration method
CN112422848B (en) Video stitching method based on depth map and color map
CN116704111A (en) Image processing method and apparatus
JP2017059998A (en) Image processing apparatus and method, and imaging device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17895390

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17895390

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP