JP7317157B2 - 緊急車両のオーディオ及びビジュアル検出のポスト融合 - Google Patents
緊急車両のオーディオ及びビジュアル検出のポスト融合 Download PDFInfo
- Publication number
- JP7317157B2 JP7317157B2 JP2022003280A JP2022003280A JP7317157B2 JP 7317157 B2 JP7317157 B2 JP 7317157B2 JP 2022003280 A JP2022003280 A JP 2022003280A JP 2022003280 A JP2022003280 A JP 2022003280A JP 7317157 B2 JP7317157 B2 JP 7317157B2
- Authority
- JP
- Japan
- Prior art keywords
- adv
- emergency vehicle
- neural network
- probability
- network model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000000007 visual effect Effects 0.000 title claims description 37
- 238000001514 detection method Methods 0.000 title description 60
- 238000000034 method Methods 0.000 claims description 26
- 238000003062 neural network model Methods 0.000 claims description 25
- 238000012545 processing Methods 0.000 claims description 21
- 230000005236 sound signal Effects 0.000 claims description 14
- 241000269400 Sirenidae Species 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 5
- 238000013527 convolutional neural network Methods 0.000 claims description 5
- 238000013519 translation Methods 0.000 claims description 4
- 230000008447 perception Effects 0.000 description 11
- 238000013528 artificial neural network Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 230000001133 acceleration Effects 0.000 description 5
- 230000004927 fusion Effects 0.000 description 5
- 238000007405 data analysis Methods 0.000 description 4
- 230000004807 localization Effects 0.000 description 4
- 230000002085 persistent effect Effects 0.000 description 4
- 238000002379 ultrasonic velocimetry Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000002411 adverse Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 3
- 230000001149 cognitive effect Effects 0.000 description 3
- 230000001276 controlling effect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 230000005534 acoustic noise Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001010 compromised effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000000875 corresponding effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009429 electrical wiring Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000014616 translation Effects 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W60/00—Drive control systems specially adapted for autonomous road vehicles
- B60W60/001—Planning or execution of driving tasks
- B60W60/0015—Planning or execution of driving tasks specially adapted for safety
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W60/00—Drive control systems specially adapted for autonomous road vehicles
- B60W60/001—Planning or execution of driving tasks
- B60W60/0027—Planning or execution of driving tasks using trajectory prediction for other traffic participants
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
- B60W40/02—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to ambient conditions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W10/00—Conjoint control of vehicle sub-units of different type or different function
- B60W10/18—Conjoint control of vehicle sub-units of different type or different function including control of braking systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W10/00—Conjoint control of vehicle sub-units of different type or different function
- B60W10/20—Conjoint control of vehicle sub-units of different type or different function including control of steering systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W30/18—Propelling the vehicle
- B60W30/18009—Propelling the vehicle related to particular drive situations
- B60W30/18163—Lane change; Overtaking manoeuvres
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W60/00—Drive control systems specially adapted for autonomous road vehicles
- B60W60/001—Planning or execution of driving tasks
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S5/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
- G01S5/18—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using ultrasonic, sonic, or infrasonic waves
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B13/00—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion
- G05B13/02—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
- G05B13/0265—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric the criterion being a learning criterion
- G05B13/027—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric the criterion being a learning criterion using neural networks only
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/047—Probabilistic or stochastic networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/809—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data
- G06V10/811—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data the classifiers operating on different input data, e.g. multi-modal recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/09—Arrangements for giving variable traffic instructions
- G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
- G08G1/0965—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages responding to signals from another vehicle, e.g. emergency vehicle
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
- G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/57—Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/08—Mouthpieces; Microphones; Attachments therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R1/00—Details of transducers, loudspeakers or microphones
- H04R1/20—Arrangements for obtaining desired frequency or directional characteristics
- H04R1/32—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
- H04R1/40—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers
- H04R1/406—Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only by combining a number of identical transducers microphones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R3/00—Circuits for transducers, loudspeakers or microphones
- H04R3/005—Circuits for transducers, loudspeakers or microphones for combining the signals of two or more microphones
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
- B60W2050/0001—Details of the control system
- B60W2050/0002—Automatic control, details of type of controller or control system architecture
- B60W2050/0004—In digital systems, e.g. discrete-time systems involving sampling
- B60W2050/0005—Processor details or data handling, e.g. memory registers or chip architecture
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
- B60W2420/40—Photo, light or radio wave sensitive means, e.g. infrared sensors
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
- B60W2420/54—Audio sensitive means, e.g. ultrasound
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2520/00—Input parameters relating to overall vehicle dynamics
- B60W2520/06—Direction of travel
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2554/00—Input parameters relating to objects
- B60W2554/40—Dynamic objects, e.g. animals, windblown objects
- B60W2554/402—Type
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2554/00—Input parameters relating to objects
- B60W2554/40—Dynamic objects, e.g. animals, windblown objects
- B60W2554/404—Characteristics
- B60W2554/4041—Position
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2554/00—Input parameters relating to objects
- B60W2554/40—Dynamic objects, e.g. animals, windblown objects
- B60W2554/404—Characteristics
- B60W2554/4044—Direction of movement, e.g. backwards
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S2205/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
- G01S2205/01—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations specially adapted for specific applications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S3/00—Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received
- G01S3/80—Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received using ultrasonic, sonic or infrasonic waves
- G01S3/802—Systems for determining direction or deviation from predetermined direction
- G01S3/803—Systems for determining direction or deviation from predetermined direction using amplitude comparison of signals derived from receiving transducers or transducer systems having differently-oriented directivity characteristics
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R2499/00—Aspects covered by H04R or H04S not otherwise provided for in their subgroups
- H04R2499/10—General applications
- H04R2499/13—Acoustic transducers and sound field adaptation in vehicles
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Mechanical Engineering (AREA)
- Transportation (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Automation & Control Theory (AREA)
- Signal Processing (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Acoustics & Sound (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Combustion & Propulsion (AREA)
- Chemical & Material Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Otolaryngology (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Computational Biology (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Evolutionary Biology (AREA)
Description
図1は、本開示の一実施形態による自律運転ネットワーク構成を示すブロック図である。図1を参照すると、ネットワーク構成100は、ネットワーク102を介して1つ以上のサーバ103~104に通信可能に結合され得る自律運転車両(ADV)101を含む。示されているADVは1つであるが、複数のADVがネットワーク102を介して互いに及び/又はサーバ103~104に結合され得る。ネットワーク102は、有線又は無線の、ローカルエリアネットワーク(LAN)、インターネットなどのワイドエリアネットワーク(WAN)、セルラーネットワーク、衛星ネットワーク又はそれらの組み合わせなどの任意のタイプのネットワークであり得る。サーバ(複数可)103~104は、ウェブサーバ又はクラウドサーバ、アプリケーションサーバ、バックエンドサーバ、又はそれらの組み合わせなどの、任意の種類のサーバ又は一群のサーバであり得る。サーバ103~104は、データ分析サーバ、コンテンツサーバ、交通情報サーバ、地図・特定地点(MPOI)サーバ、又は位置サーバなどであり得る。
図4は、一実施形態による緊急車両検出システム400の例を示す。
本願は、2021年1月14日に出願された「緊急車両のオーディオ及びビジュアル検出を融合するための機械学習モデル」と題された米国特許出願第17/149638号に関連する。本願は、その全体が参照により本明細書に組み込まれる。
Claims (18)
- 自律運転車両(ADV)を運転するためにコンピュータが実行する方法であって、
前記ADVの自律運転システム(ADS)で、1つ以上のオーディオ取込デバイスを使用して取り込まれたオーディオ信号のストリームと、前記ADVの周囲環境から前記ADVに取り付けられた1つ以上の画像取込デバイスを使用して取り込まれた一連の画像フレームとを受信し、
前記ADSによって第1のニューラルネットワークモデルを使用して、取り込まれたオーディオ信号のストリームの少なくとも一部がサイレン音からであるという第1の確率を決定し、
前記ADSによって第2のニューラルネットワークモデルを使用して、前記一連の画像フレームの少なくとも1つの画像フレームが緊急車両からであるという第2の確率を決定し、
前記ADSによって前記第1の確率及び前記第2の確率に基づいて、緊急車両が前記周囲環境に存在すると決定し、
前記第1のニューラルネットワークモデルを使用して、前記ADVと前記サイレン音の音源との間の角度、及び前記音源の移動方向を決定し、
前記第2のニューラルネットワークモデルを使用して、前記少なくとも1つの画像フレームを囲むバウンディングボックスのサイズ、及び、前記少なくとも1つの画像フレームを取り込むために使用される画像取込デバイスの1つ以上の外因性パラメータに基づいて、前記ADVと前記緊急車両との距離を決定し、前記外因性パラメータには、複数のカメラ配置におけるカメラ間の相対的な回転及び並進が含まれる、方法。 - 前記第1の確率が第1の閾値を超えている場合、
前記第2の確率が第2の閾値を超えている場合、又は、
前記第1の確率が前記第1の閾値を超え、前記第2の確率が前記第2の閾値を超えている場合、
緊急車両が前記周囲環境に存在すると決定する、請求項1に記載の方法。 - 緊急車両が前記周囲環境に存在するとの決定に応答して、前記サイレン音の音源と前記ADVとの間の前記角度、前記音源の前記移動方向、及び、前記緊急車両から前記ADVまでの前記距離を融合して、前記緊急車両の位置及び前記緊急車両の移動方向を決定する、請求項2に記載の方法。
- 前記緊急車両の前記位置及び前記移動方向に基づいて、前記ADVの位置の決定に応答して、前記ADVを制御して、前記ADVを操舵して現在の運転車線から出すこと、又は前記ADVを制動して減速することのうちの少なくとも1つを行う、請求項3に記載の方法。
- 前記第1のニューラルネットワークモデルを、複数の緊急車両から収集された緊急車両サイレンを表すオーディオデータで訓練し、前記第2のニューラルネットワークモデルを、前記オーディオデータの収集と同時的に収集されたビジュアルデータで訓練する、請求項1に記載の方法。
- 前記第1のニューラルネットワークモデル及び前記第2のニューラルネットワークモデルは夫々、畳み込みニューラルネットワークである、請求項1に記載の方法。
- 前記1つ以上のオーディオ取込デバイスは1つ以上のマイクロフォンを含み、前記1つ以上の画像取込デバイスは1つ以上のカメラを含む、請求項1に記載の方法。
- 命令が記憶されている非一時的な機械可読媒体であって、
前記命令は、プロセッサによって実行されると、前記プロセッサに自律運転車両(ADV)を運転する処理を行わせ、前記処理を行わせる際に、
前記ADVの自律運転システム(ADS)で、1つ以上のオーディオ取込デバイスを使用して取り込まれたオーディオ信号のストリームと、前記ADVの周囲環境から前記ADVに取り付けられた1つ以上の画像取込デバイスを使用して取り込まれた一連の画像フレームとを受信させ、
前記ADSによって第1のニューラルネットワークモデルを使用して、取り込まれたオーディオ信号のストリームの少なくとも一部がサイレン音からであるという第1の確率を決定させ、
前記ADSによって第2のニューラルネットワークモデルを使用して、前記一連の画像フレームの少なくとも1つの画像フレームが緊急車両からであるという第2の確率を決定させ、
前記ADSによって前記第1の確率及び前記第2の確率に基づいて、緊急車両が前記周囲環境に存在すると決定させ、
前記処理を行わせる際に、
前記第1のニューラルネットワークモデルを使用して、前記ADVと前記サイレン音の音源との間の角度、及び前記音源の移動方向を決定させ、
前記第2のニューラルネットワークモデルを使用して、前記少なくとも1つの画像フレームを囲むバウンディングボックスのサイズ、及び、前記少なくとも1つの画像フレームを取り込むために使用される画像取込デバイスの1つ以上の外因性パラメータに基づいて、前記ADVと前記緊急車両との距離を決定させ、前記外因性パラメータには、複数のカメラ配置におけるカメラ間の相対的な回転及び並進が含まれる、機械可読媒体。 - 前記処理を行わせる際に、
前記第1の確率が第1の閾値を超えている場合、
前記第2の確率が第2の閾値を超えている場合、又は、
前記第1の確率が前記第1の閾値を超え、前記第2の確率が前記第2の閾値を超えている場合、
緊急車両が前記周囲環境に存在すると決定させる、請求項8に記載の機械可読媒体。 - 前記処理を行わせる際に、
緊急車両が前記周囲環境に存在するとの決定に応答して、前記サイレン音の前記音源と前記ADVとの間の前記角度、前記音源の前記移動方向、及び、前記緊急車両から前記ADVまでの前記距離を融合して、前記緊急車両の位置及び前記緊急車両の移動方向を決定させる、請求項9に記載の機械可読媒体。 - 前記処理を行わせる際に、
前記緊急車両の前記位置及び前記移動方向に基づいて、前記ADVの位置の決定に応答して、前記ADVを制御させ、前記ADVを操舵して現在の運転車線から出すこと、又は前記ADVを制動して減速することのうちの少なくとも1つを行わせる、請求項10に記載の機械可読媒体。 - 前記第1のニューラルネットワークモデルは、複数の緊急車両から収集された緊急車両サイレンを表すオーディオデータで訓練されており、前記第2のニューラルネットワークモデルは、前記オーディオデータの収集と同時的に収集されたビジュアルデータで訓練されている、請求項8に記載の機械可読媒体。
- 前記第1のニューラルネットワークモデル及び前記第2のニューラルネットワークモデルは夫々、畳み込みニューラルネットワークである、請求項8に記載の機械可読媒体。
- 前記1つ以上のオーディオ取込デバイスは1つ以上のマイクロフォンを含み、前記1つ以上の画像取込デバイスは1つ以上のカメラを含む、請求項8に記載の機械可読媒体。
- プロセッサと、
前記プロセッサに結合されて命令を記憶するメモリと
備えており、
前記命令は、前記プロセッサによって実行されると、前記プロセッサに自律運転車両(ADV)を運転する処理を行わせ、前記処理を行わせる際に、
前記ADVの自律運転システム(ADS)で、1つ以上のオーディオ取込デバイスを使用して取り込まれたオーディオ信号のストリームと、前記ADVの周囲環境から前記ADVに取り付けられた1つ以上の画像取込デバイスを使用して取り込まれた一連の画像フレームとを受信させ、
前記ADSによって第1のニューラルネットワークモデルを使用して、オーディオ信号の前記ストリームの少なくとも一部がサイレン音からであるという第1の確率を決定させ、
前記ADSによって第2のニューラルネットワークモデルを使用して、前記一連の取り込まれた画像フレームの少なくとも1つの画像フレームが緊急車両からであるという第2の確率を決定させ、
前記ADSによって前記第1の確率及び前記第2の確率に基づいて、緊急車両が前記周囲環境に存在すると決定させ、
前記処理を行わせる際に、
前記第1のニューラルネットワークモデルを使用して、前記ADVと前記サイレン音の音源との間の角度、及び前記音源の移動方向を決定させ、
前記第2のニューラルネットワークモデルを使用して、前記少なくとも1つの画像フレームを囲むバウンディングボックスのサイズ、及び、前記少なくとも1つの画像フレームを取り込むために使用される画像取込デバイスの1つ以上の外因性パラメータに基づいて、前記ADVと前記緊急車両との距離を決定させ、前記外因性パラメータには、複数のカメラ配置におけるカメラ間の相対的な回転及び並進が含まれる、データ処理システム。 - 前記処理を行わせる際に、
前記第1の確率が第1の閾値を超えている場合、
前記第2の確率が第2の閾値を超えている場合、又は、
前記第1の確率が前記第1の閾値を超え、前記第2の確率が前記第2の閾値を超えている場合、
緊急車両が前記周囲環境に存在すると決定させる、請求項15に記載のデータ処理システム。 - 前記処理を行わせる際に、
緊急車両が前記周囲環境に存在するとの決定に応答して、前記サイレン音の前記音源と前記ADVとの間の前記角度、前記音源の前記移動方向、及び、前記緊急車両から前記ADVまでの前記距離を融合させて、前記緊急車両の位置及び前記緊急車両の移動方向を決定させる、請求項16に記載のデータ処理システム。 - コンピュータ上で動作しているときに、請求項1~7のいずれか1つに記載の方法を前記コンピュータに実行させるためのコンピュータプログラム。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/149,638 US20220219736A1 (en) | 2021-01-14 | 2021-01-14 | Emergency vehicle audio and visual detection post fusion |
US17/149,638 | 2021-01-14 |
Publications (2)
Publication Number | Publication Date |
---|---|
JP2022058594A JP2022058594A (ja) | 2022-04-12 |
JP7317157B2 true JP7317157B2 (ja) | 2023-07-28 |
Family
ID=80121748
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
JP2022003280A Active JP7317157B2 (ja) | 2021-01-14 | 2022-01-12 | 緊急車両のオーディオ及びビジュアル検出のポスト融合 |
Country Status (5)
Country | Link |
---|---|
US (1) | US20220219736A1 (ja) |
EP (1) | EP3971770A3 (ja) |
JP (1) | JP7317157B2 (ja) |
KR (1) | KR102607029B1 (ja) |
CN (1) | CN114379590B (ja) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11884298B2 (en) * | 2020-10-23 | 2024-01-30 | Tusimple, Inc. | Safe driving operations of autonomous vehicles |
US20230331253A1 (en) * | 2022-04-19 | 2023-10-19 | Gm Cruise Holdings Llc | Systems and methods for responding to detected emergency vehicles |
WO2024102431A1 (en) * | 2022-11-09 | 2024-05-16 | Aurora Operations, Inc. | Systems and methods for emergency vehicle detection |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190027032A1 (en) | 2017-07-24 | 2019-01-24 | Harman International Industries, Incorporated | Emergency vehicle alert system |
US20190049989A1 (en) | 2017-11-17 | 2019-02-14 | Intel Corporation | Identification of audio signals in surrounding sounds and guidance of an autonomous vehicle in response to the same |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107134160A (zh) * | 2016-02-29 | 2017-09-05 | 法拉第未来公司 | 紧急信号检测和响应 |
US20180137756A1 (en) * | 2016-11-17 | 2018-05-17 | Ford Global Technologies, Llc | Detecting and responding to emergency vehicles in a roadway |
US10127818B2 (en) * | 2017-02-11 | 2018-11-13 | Clear Commute Ventures Pty Ltd | Systems and methods for detecting and avoiding an emergency vehicle in the proximity of a substantially autonomous vehicle |
US10319228B2 (en) * | 2017-06-27 | 2019-06-11 | Waymo Llc | Detecting and responding to sirens |
US10852736B2 (en) * | 2018-04-03 | 2020-12-01 | Baidu Usa Llc | Method to track and to alert autonomous driving vehicles (ADVS) of emergency vehicles |
JP7048465B2 (ja) * | 2018-09-18 | 2022-04-05 | 株式会社東芝 | 移動体制御装置、方法及びプログラム |
US11532168B2 (en) * | 2019-11-15 | 2022-12-20 | Nvidia Corporation | Multi-view deep neural network for LiDAR perception |
US11814062B2 (en) * | 2020-11-18 | 2023-11-14 | Google Llc | Detecting and handling driving event sounds during a navigation session |
-
2021
- 2021-01-14 US US17/149,638 patent/US20220219736A1/en active Pending
- 2021-12-30 CN CN202111649429.XA patent/CN114379590B/zh active Active
-
2022
- 2022-01-12 JP JP2022003280A patent/JP7317157B2/ja active Active
- 2022-01-12 EP EP22151194.2A patent/EP3971770A3/en not_active Withdrawn
- 2022-01-13 KR KR1020220005369A patent/KR102607029B1/ko active IP Right Grant
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190027032A1 (en) | 2017-07-24 | 2019-01-24 | Harman International Industries, Incorporated | Emergency vehicle alert system |
US20190049989A1 (en) | 2017-11-17 | 2019-02-14 | Intel Corporation | Identification of audio signals in surrounding sounds and guidance of an autonomous vehicle in response to the same |
Non-Patent Citations (1)
Title |
---|
Aarajit Garg, Anchal Kumar Gupta, Divyansh Shrivastava, Yash Didwania, Prayash Jyoti Bora,Emergency Vehicle Detection by Autonomous Vehicle,International Journal of Engineering Research & Technology,Volume 08, Issue 05,IJERT,2019年05月11日,P. 190-194,https://www.ijert.org/research/emergency-vehicle-detection-by-autonomous-vehicle-IJERTV8IS050104.pdf |
Also Published As
Publication number | Publication date |
---|---|
EP3971770A3 (en) | 2022-07-20 |
CN114379590B (zh) | 2023-12-19 |
CN114379590A (zh) | 2022-04-22 |
KR102607029B1 (ko) | 2023-11-30 |
EP3971770A2 (en) | 2022-03-23 |
US20220219736A1 (en) | 2022-07-14 |
KR20220013580A (ko) | 2022-02-04 |
JP2022058594A (ja) | 2022-04-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112498365B (zh) | 基于置信度水平和距离、响应于障碍物的自动驾驶车辆的延迟决策 | |
CN111775945B (zh) | 用于自动驾驶的用于检测最接近的路径内对象的方法和装置 | |
JP7317157B2 (ja) | 緊急車両のオーディオ及びビジュアル検出のポスト融合 | |
JP7340046B2 (ja) | 緊急車両の音声および視覚の検出を融合した機械学習モデル | |
US11609576B2 (en) | Emergency vehicle audio detection | |
CN113129624A (zh) | 交通堵塞情况下的最快车道确定算法 | |
EP4024365B1 (en) | Audio logging for model training and onboard validation utilizing autonomous driving vehicle | |
CN112230646A (zh) | 设计用于单车运行的自动驾驶***下的车辆队列实施 | |
JP2022058593A (ja) | 自律運転車両を使用した自動オーディオデータラベル付け | |
US11613275B2 (en) | Grayscale-based camera perception | |
US11608056B2 (en) | Post collision damage reduction brake system incorporating front obstacle avoidance | |
CN113815640A (zh) | 用于具有不同速度限制的车道的车道变更*** | |
CN113247017A (zh) | 用于确保自动驾驶车辆的稳定绕行的双缓冲*** | |
KR102597917B1 (ko) | 자율 주행 차량을 위한 음원 검출 및 위치 측정 | |
US11325529B2 (en) | Early brake light warning system for autonomous driving vehicle | |
US20230202516A1 (en) | Planning under prediction with confidence region for an autonomous driving vehicle | |
JP7229278B2 (ja) | 経路計画のための二次計画法に基づき、片側に寄せる方法 | |
CN113799789B (zh) | 融合紧急车辆音频和视觉检测的机器学习模型 | |
US11662219B2 (en) | Routing based lane guidance system under traffic cone situation | |
US20240218911A1 (en) | Brake pad wear detection and warning for autonomous driving vehicles |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
A621 | Written request for application examination |
Free format text: JAPANESE INTERMEDIATE CODE: A621 Effective date: 20220112 |
|
A977 | Report on retrieval |
Free format text: JAPANESE INTERMEDIATE CODE: A971007 Effective date: 20230125 |
|
A131 | Notification of reasons for refusal |
Free format text: JAPANESE INTERMEDIATE CODE: A131 Effective date: 20230131 |
|
A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20230428 |
|
TRDD | Decision of grant or rejection written | ||
A01 | Written decision to grant a patent or to grant a registration (utility model) |
Free format text: JAPANESE INTERMEDIATE CODE: A01 Effective date: 20230704 |
|
A61 | First payment of annual fees (during grant procedure) |
Free format text: JAPANESE INTERMEDIATE CODE: A61 Effective date: 20230718 |
|
R150 | Certificate of patent or registration of utility model |
Ref document number: 7317157 Country of ref document: JP Free format text: JAPANESE INTERMEDIATE CODE: R150 |