WO2024102453A1 - Digitizing touch with artificial robotic fingertip - Google Patents
Digitizing touch with artificial robotic fingertip Download PDFInfo
- Publication number
- WO2024102453A1 WO2024102453A1 PCT/US2023/037113 US2023037113W WO2024102453A1 WO 2024102453 A1 WO2024102453 A1 WO 2024102453A1 US 2023037113 W US2023037113 W US 2023037113W WO 2024102453 A1 WO2024102453 A1 WO 2024102453A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- lens
- fingertip
- gel
- image
- hemispherical dome
- Prior art date
Links
- 230000003287 optical effect Effects 0.000 claims abstract description 36
- 229920001296 polysiloxane Polymers 0.000 claims abstract description 34
- 238000010801 machine learning Methods 0.000 claims abstract description 6
- 239000010410 layer Substances 0.000 claims description 51
- 239000000463 material Substances 0.000 claims description 49
- 238000000034 method Methods 0.000 claims description 35
- 238000005286 illumination Methods 0.000 claims description 30
- 238000012546 transfer Methods 0.000 claims description 27
- 238000013528 artificial neural network Methods 0.000 claims description 21
- 239000011248 coating agent Substances 0.000 claims description 20
- 238000000576 coating method Methods 0.000 claims description 20
- 238000005259 measurement Methods 0.000 claims description 19
- 230000008569 process Effects 0.000 claims description 16
- 239000007787 solid Substances 0.000 claims description 16
- 239000007789 gas Substances 0.000 claims description 13
- 238000004519 manufacturing process Methods 0.000 claims description 11
- 101710134784 Agnoprotein Proteins 0.000 claims description 9
- WQZGKKKJIJFFOK-GASJEMHNSA-N Glucose Natural products OC[C@H]1OC(O)[C@H](O)[C@@H](O)[C@@H]1O WQZGKKKJIJFFOK-GASJEMHNSA-N 0.000 claims description 9
- 239000008103 glucose Substances 0.000 claims description 9
- 230000007613 environmental effect Effects 0.000 claims description 7
- 229920002379 silicone rubber Polymers 0.000 claims description 7
- 239000011247 coating layer Substances 0.000 claims description 6
- 239000012636 effector Substances 0.000 claims description 6
- 238000007747 plating Methods 0.000 claims description 6
- 238000005266 casting Methods 0.000 claims description 5
- 238000007654 immersion Methods 0.000 claims description 5
- 238000002156 mixing Methods 0.000 claims description 5
- 150000001875 compounds Chemical class 0.000 claims description 4
- 238000003062 neural network model Methods 0.000 claims description 4
- 230000001681 protective effect Effects 0.000 claims description 4
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 claims description 3
- 239000004205 dimethyl polysiloxane Substances 0.000 claims description 3
- 235000013870 dimethyl polysiloxane Nutrition 0.000 claims description 3
- 239000001301 oxygen Substances 0.000 claims description 3
- 229910052760 oxygen Inorganic materials 0.000 claims description 3
- 238000005498 polishing Methods 0.000 claims description 3
- 229920000435 poly(dimethylsiloxane) Polymers 0.000 claims description 3
- 210000003813 thumb Anatomy 0.000 claims description 3
- -1 Poly dimethylsiloxane Polymers 0.000 claims description 2
- 230000003213 activating effect Effects 0.000 claims description 2
- 238000007605 air drying Methods 0.000 claims description 2
- XAGFODPZIPBFFR-UHFFFAOYSA-N aluminium Chemical compound [Al] XAGFODPZIPBFFR-UHFFFAOYSA-N 0.000 claims description 2
- 229910052782 aluminium Inorganic materials 0.000 claims description 2
- 238000004140 cleaning Methods 0.000 claims description 2
- 239000004945 silicone rubber Substances 0.000 claims description 2
- 239000000499 gel Substances 0.000 description 63
- 238000012545 processing Methods 0.000 description 53
- 230000035945 sensitivity Effects 0.000 description 26
- 210000003811 finger Anatomy 0.000 description 25
- 230000009471 action Effects 0.000 description 23
- 238000013461 design Methods 0.000 description 20
- 238000007373 indentation Methods 0.000 description 20
- 230000008901 benefit Effects 0.000 description 16
- 230000003993 interaction Effects 0.000 description 14
- 239000000523 sample Substances 0.000 description 13
- 230000004044 response Effects 0.000 description 12
- 230000002123 temporal effect Effects 0.000 description 12
- 241000282412 Homo Species 0.000 description 11
- 238000013459 approach Methods 0.000 description 11
- 238000005070 sampling Methods 0.000 description 11
- 230000000694 effects Effects 0.000 description 10
- 238000003384 imaging method Methods 0.000 description 10
- BQCADISMDOOEFD-UHFFFAOYSA-N Silver Chemical compound [Ag] BQCADISMDOOEFD-UHFFFAOYSA-N 0.000 description 9
- 238000011156 evaluation Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 9
- 229910052709 silver Inorganic materials 0.000 description 9
- 239000004332 silver Substances 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 239000007788 liquid Substances 0.000 description 8
- 230000000007 visual effect Effects 0.000 description 8
- 238000004458 analytical method Methods 0.000 description 7
- 229920005839 ecoflex® Polymers 0.000 description 7
- 229920001971 elastomer Polymers 0.000 description 7
- 239000010408 film Substances 0.000 description 7
- 230000033001 locomotion Effects 0.000 description 7
- 230000001953 sensory effect Effects 0.000 description 7
- 238000004088 simulation Methods 0.000 description 7
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 7
- 230000003068 static effect Effects 0.000 description 6
- 239000010409 thin film Substances 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 5
- 238000013480 data collection Methods 0.000 description 5
- 229920000642 polymer Polymers 0.000 description 5
- 238000010079 rubber tapping Methods 0.000 description 5
- 238000003860 storage Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 230000001133 acceleration Effects 0.000 description 4
- 238000007796 conventional method Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 4
- 239000000806 elastomer Substances 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000007781 pre-processing Methods 0.000 description 4
- 230000011514 reflex Effects 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 238000012512 characterization method Methods 0.000 description 3
- 238000013481 data capture Methods 0.000 description 3
- 230000000994 depressogenic effect Effects 0.000 description 3
- 238000013400 design of experiment Methods 0.000 description 3
- 230000004807 localization Effects 0.000 description 3
- 230000003278 mimic effect Effects 0.000 description 3
- 238000000465 moulding Methods 0.000 description 3
- 239000004033 plastic Substances 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 239000002344 surface layer Substances 0.000 description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 2
- 101150050048 SNCB gene Proteins 0.000 description 2
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 235000014121 butter Nutrition 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 230000001934 delay Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000009977 dual effect Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 235000013305 food Nutrition 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 210000001428 peripheral nervous system Anatomy 0.000 description 2
- 239000000049 pigment Substances 0.000 description 2
- 229910052710 silicon Inorganic materials 0.000 description 2
- 239000010703 silicon Substances 0.000 description 2
- 230000035943 smell Effects 0.000 description 2
- 239000000344 soap Substances 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 229910001094 6061 aluminium alloy Inorganic materials 0.000 description 1
- 241000238631 Hexapoda Species 0.000 description 1
- 241000237502 Ostreidae Species 0.000 description 1
- BLRPTPMANUNPDV-UHFFFAOYSA-N Silane Chemical compound [SiH4] BLRPTPMANUNPDV-UHFFFAOYSA-N 0.000 description 1
- 208000024313 Testicular Neoplasms Diseases 0.000 description 1
- NIXOWILDQLNWCW-UHFFFAOYSA-N acrylic acid group Chemical group C(C=C)(=O)O NIXOWILDQLNWCW-UHFFFAOYSA-N 0.000 description 1
- 230000002730 additional effect Effects 0.000 description 1
- 238000000540 analysis of variance Methods 0.000 description 1
- 239000006117 anti-reflective coating Substances 0.000 description 1
- 101150059062 apln gene Proteins 0.000 description 1
- 238000000149 argon plasma sintering Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 210000000481 breast Anatomy 0.000 description 1
- 210000003169 central nervous system Anatomy 0.000 description 1
- 235000013351 cheese Nutrition 0.000 description 1
- 238000005234 chemical deposition Methods 0.000 description 1
- 230000008133 cognitive development Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 239000002537 cosmetic Substances 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000003618 dip coating Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 238000004870 electrical engineering Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 238000009501 film coating Methods 0.000 description 1
- 238000003055 full factorial design Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000011065 in-situ storage Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000011068 loading method Methods 0.000 description 1
- 238000011089 mechanical engineering Methods 0.000 description 1
- 210000000412 mechanoreceptor Anatomy 0.000 description 1
- 108091008704 mechanoreceptors Proteins 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003647 oxidation Effects 0.000 description 1
- 238000007254 oxidation reaction Methods 0.000 description 1
- 235000020636 oyster Nutrition 0.000 description 1
- 238000010422 painting Methods 0.000 description 1
- 238000002559 palpation Methods 0.000 description 1
- 239000000843 powder Substances 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 108020003175 receptors Proteins 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
- 238000006748 scratching Methods 0.000 description 1
- 230000002393 scratching effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 229910000077 silane Inorganic materials 0.000 description 1
- 238000002444 silanisation Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000003756 stirring Methods 0.000 description 1
- 238000007920 subcutaneous administration Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
- 230000002459 sustained effect Effects 0.000 description 1
- 238000010408 sweeping Methods 0.000 description 1
- 238000002076 thermal analysis method Methods 0.000 description 1
- 210000003051 thermoreceptor Anatomy 0.000 description 1
- 108091008689 thermoreceptors Proteins 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 235000013311 vegetables Nutrition 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 239000003190 viscoelastic substance Substances 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
- B25J13/081—Touching devices, e.g. pressure-sensitive
- B25J13/084—Tactile sensors
Definitions
- This disclosure generally relates to robotics, and in particular relates to hardware and software for smart robotic sensing.
- Al Artificial intelligence
- Al technology is widely used throughout industry, government and science, such as advanced web search engines, recommendation systems, understanding human speech, self-driving cars, generative or creative tools, and competing at the highest level in strategic games.
- Robotics is an interdisciplinary branch of electronics and communication, computer science and engineering. Robotics involves the design, construction, operation, and use of robots. The goal of robotics is to design machines that can help and assist humans. Robotics integrates fields of mechanical engineering, electrical engineering, information engineering, mechatronics engineering, electronics, biomedical engineering, computer engineering, control systems engineering, software engineering, mathematics, etc. The field of robotics develops machines that can automate tasks and do various jobs that a human might not be able to do. Certain robots require user input to operate, while other robots function autonomously.
- Touch is a sensing modality that may provide rich information about object properties and interactions with the physical environment. Humans and robots both benefit from using touch to perceive and interact with the surrounding environment.
- no existing systems may provide rich, multimodal digital touch-sensing capabilities while retaining the form factor of a human finger.
- the embodiments disclosed herein may improve the digitization of touch with technological advances embodied in an artificial finger-shaped sensor with enhanced sensing capabilities.
- the artificial fingertip may comprise high-resolution sensors (e.g.. approximately 8.3 million taxels) that respond to omnidirectional touch, capture multimodal signals, and use on-device artificial intelligence to process the data in real time.
- evaluations show that the artificial fingertip can resolve spatial features as small as 7 um, sense normal and shear forces with a resolution between, e.g., 1 mN and 1.3 mN respectively, perceive vibrations up to, e.g., 9-11 kHz, sense odor, and even sense heat.
- the on-device Al neural- network accelerator may act as a peripheral nervous system on a robot and mimic the reflexarc found in humans.
- a system for touch digitization may comprise a silicone hemispherical dome comprising a surface comprising a reflective silver-film layer.
- the system may additionally comprise an omnidirectional optical system comprising a lens comprising a plurality of lens elements and an image sensor configured to generate image data from data captured by the lens.
- a first lens element of the plurality of lens elements may be in direct contact with the hemispherical dome without airgap.
- the lens may be configured to capture scattering of internal incident light generated by the reflective silver- film layer.
- the system may also comprise one or more non-image sensors disposed underneath the omnidirectional optical system.
- the system may further comprise one or more processors and a non-transitory memory coupled to the processors comprising instructions executable by the processors.
- the processors may be operable when executing the instructions to access the image data from the omnidirectional optical system and sensing data from the one or more non-image sensors and generate the touch digitization based on the accessed image and sensing data by one or more machine-learning models.
- an artificial fingertip for touch digitization may comprise a silicone hemispherical dome.
- the artificial fingertip may also comprise an omnidirectional optical system comprising a lens comprising a plurality of lens elements and an image sensor configured to generate image data from data captured by the lens.
- a first lens element of the plurality of lens elements may be in direct contact with the silicone hemispherical dome without airgap.
- the artificial fingertip may further comprise one or more non-image sensors disposed underneath the omnidirectional optical system.
- a technical advantage of the embodiments may include improved sensitivity to input stimuli by using dynamic lighting with variable wavelength and direction for reconstruction of touch surface topology signals (images) as the system disclosed herein moves beyond the traditional Lambertian scattering paradigm towards a controlled degree scattering surface along with a ground up approach on optimizing the material properties for the highest sensitivity to spatial features, which is achieved by developing a new process for chemically growing a thin-film layer of silver film onto the fingertip surface.
- Another technical advantage of the embodiments may include high spatial resolution in capturing minute details on the surface of the fingertip, which allows for bypassing traditional methods of capturing normal and shear forces through the use of markers.
- the embodiments disclosed herein utilize a 3- region solid immersion hyperfisheye lens design for touch digitization systems in a non-airgap configuration to a PDMS material. Due to the design and specific requirements for non-human imaging, the embodiments disclosed herein are able to maintain a superior performance of the modulation transfer function (MTF) through the entire field of the lens, thereby ensuring spatial performance along the tip, side and edges of the fingertip.
- MTF modulation transfer function
- the omnidirectional surface using a single camera and lens system may achieve full field view of over 200 degrees.
- Another technical advantage of the embodiments may include high temporal resolution based on interchangeable and modular electronic stack-up for touch digitization (vision capture, multi-modal capture, on-device Al processing), which combines all electronics and sensing elements into a familiar size and shape of a human thumb.
- Each individual system of the stack- up can be interchanged with a newer design to reduce the hardware development lifecycle, furthermore, components in the stack-up can be excluded to further reduce the size of the sensor based on individual requirements of the application.
- Another technical advantage of the embodiments may include digitizing touch signals for on-device Al processing for providing human-like reflex arc actions to robotic manipulators as a neural network processor is inside the fingertip for direct inference on input data and a processor can provide direct outputs to control a secondary device such as a robotic end effector.
- Certain embodiments disclosed herein may provide none, some, or all of the above technical advantages.
- One or more other technical advantages may be readily apparent to one skilled in the art in view of the figures, descriptions, and claims of the present disclosure.
- any subject matter resulting from a deliberate reference back to any previous claims can be claimed as well, so that any combination of claims and the features thereof are disclosed and can be claimed regardless of the dependencies chosen in the attached claims.
- the subject-matter which can be claimed comprises not only the combinations of features as set out in the attached claims but also any other combination of features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims.
- any of the embodiments and features described or depicted herein can be claimed in a separate claim and/or in any combination with any embodiment or feature described or depicted herein or with any of the features of the attached claims.
- FIG. 1 illustrates an example cutaway diagram of the artificial fingertip disclosed herein and its modular electronics.
- FIG. 2A illustrates example evaluations of normal and shear forces in three separate regions of the sensor, from the tip toward the side.
- FIG. 2B illustrates an example prediction of normal and shear forces from visuo- tactile image output.
- FIG. 2C illustrates example evaluations of spatial resolution by using a dual pronged microindenter depressed into the artificial fingertip with varying width.
- FIG. 2D illustrates example methods which create the artificial fingertip volume.
- FIG. 2E illustrates an example simulation of the effects of increasing the surface scattering along the internal reflective layer of the artificial fingertip.
- FIG. 3A illustrates the ability to determine the volume of water in an opaque container by tapping with one finger and recording the response on the fingers in contact with the container.
- FIG. 3B illustrates example spectrogram of the surface audio textures recorded for different objects.
- FIG. 3C illustrates example sensitivity’ of the artificial fingertip towards heat gradients.
- FIG. 3D illustrates example accuracy of object identification through local gas sensing.
- FIG. 3E illustrates example accuracy of object classification through scent.
- FIG. 3F illustrates an example localization of finger placement on an object during movement transients with empty' and full volumes of liquid.
- FIG. 5 illustrates an example visualization of the result of FEM analysis with surface map graphs.
- FIG. 6 illustrates example maximum normal and shear force applied to the artificial fingertip surface prior to detachment from the main body.
- FIG. 7 illustrates an example cross-section of the fingertip gel coating showing the 3 layers: outer, silver and base gel.
- FIGS. 8A-8H illustrate example hyperfisheye lens designed specifically for capturing omnidirectional tactile sensing images.
- FIG. 9 illustrates example inference latency measurements comparing on-device to host incorporating the tactile data pre-processing and transfer stages.
- FIG. 10 illustrates example evaluations of non-uniformity metrics.
- FIG. 11 illustrates an example data capture pipeline for the vision system of the disclosed artificial fingertip and touch information from external stimulus.
- FIG. 12 illustrates example data collection and prediction.
- FIG. 13 illustrates example simulated performance of the vision system of the artificial fingertip disclosed herein from on-axis to far-field contact for increasing line pairs per millimeters translated to spatial resolution for sagittal and tangential responses.
- FIG. 14 illustrates example average pipeline latency for a MobileNetV2 network from acquiring tactile data, transferring data, pre-processing, inference to providing actions for varying image and channel width sizes.
- FIG. 15 illustrates an example comparison of touch information from the artificial fingertip disclosed herein tapping a water bottle with varying volumes of water from empty, half filled, and full.
- FIG. 16 illustrates an example 6-DoF robot indenter for testing tactile sensor force resolution.
- FIGS. 17A-17B illustrate example image snapshots taken by the artificial fingertip disclosed herein from shear force data collection in two key moments.
- FIG. 18 illustrates an example normal force prediction error distribution by surface types and regions.
- FIG. 19 illustrates an example capture amongst vision-based touch sensors.
- FIG. 20A illustrates an example analog of the human reflex-arc by quickly processing sensory input within the fingertip, directly controlling the actuators of a robot hand to retract in response to touching an object.
- FIG. 20B illustrates example tactile processing and control paradigm transfer of sensory data to a remote computer for processing.
- FIG. 20C illustrates example local processing for mimicking the reflex-arc with the system fingertip.
- FIG. 20D illustrates example mean and standard deviation of the event-to-action latency.
- Touch is a sensing modality that may provide rich information about object properties and interactions with the physical environment. Humans and robots both benefit from using touch to perceive and interact with the surrounding environment. However, no existing systems may provide rich, multimodal digital touch-sensing capabilities while retaining the form factor of a human finger.
- the embodiments disclosed herein may improve the digitization of touch with technological advances embodied in an artificial finger-shaped sensor with enhanced sensing capabilities.
- the artificial fingertip may comprise high-resolution sensors (e.g., approximately 8.3 million taxels) that respond to omnidirectional touch, capture multimodal signals, and use on-device artificial intelligence to process the data in real time.
- evaluations show that the artificial fingertip can resolve spatial features as small as 7 um, sense normal and shear forces with a resolution between, e.g.. 1 mN and 1.3 mN respectively, perceive vibrations up to, e.g., 9-1 1 kHz, sense odor, and even sense heat.
- the on-device Al neural- network accelerator may act as a peripheral nervous system on a robot and mimic the reflexarc found in humans.
- a system for touch digitization may comprise a silicone hemispherical dome comprising a surface comprising a reflective silver-film layer.
- the system may additionally comprise an omnidirectional optical system comprising a lens comprising a plurality of lens elements and an image sensor configured to generate image data from data captured by the lens.
- a first lens element of the plurality of lens elements may be in direct contact with the silicone hemispherical dome without airgap.
- the lens may be configured to capture scattering of internal incident light generated by the reflective silver-film layer.
- the system may also comprise one or more non-image sensors disposed underneath the omnidirectional optical system.
- the system may further comprise one or more processors and a non-transitory memory coupled to the processors comprising instructions executable by the processors.
- the processors may be operable when executing the instructions to access the image data from the omnidirectional optical system and sensing data from the one or more non-image sensors and generate the touch digitization based on the accessed image and sensing data by one or more machine-learning models.
- an artificial fingertip for touch digitization may comprise a silicone hemispherical dome.
- the artificial fingertip may also comprise an omnidirectional optical system comprising a lens comprising a plurality of lens elements and an image sensor configured to generate image data from data captured by the lens.
- a first lens element of the plurality of lens elements may be in direct contact with the hemispherical dome without airgap.
- the artificial fingertip may further comprise one or more non-image sensors disposed underneath the omnidirectional optical system.
- touch may be the most critical in how humans interact with the world. Touch may enable humans to measure forces and recognize object properties, e.g., shape, weight, density, textures, friction, elasticity. Touch may also play an important role both in social relationships and in cognitive development. By comparison, the earliest efforts to impart that sense to robots came no closer than crude approximations. No solutions have emerged for digitizing touch with the same rich sensorial spectrum humans take for granted. Toward the advancement of robotic in-hand manipulation, the embodiments disclosed herein may mimic familiar features of the human hand: fingers. The touch digitization in the embodiments disclosed herein may enable intelligent systems to discern significantly higher levels of physical information during environmental interaction.
- Digitizing touch may depend on two features including one temporal in nature and one spatial. Temporal features may process information from basic signals of time variation, whereas spatial features may process a discrete multidimensional array of temporal signals.
- the embodiments disclosed herein combine these methods within a unified platform to improve the capabilities of touch digitization, i.e., a modular, finger-shaped, multimodal tactile sensor with on-device artificial intelligence (Al) capabilities and superhuman performance.
- the artificial fingertip may comprise a multimodal modular sensor.
- the multimodal modular sensor may comprise a main body mechanical housing.
- the main body mechanical housing may have a similar shape and size to the human thumb.
- the silicone hemispherical dome, the omnidirectional optical system, the one or more non-image sensors, the one or more processors, and the non-transitory memory may be disposed in the housing.
- the multimodal modular sensor may also comprise a soft silicone solid body fingertip.
- the soft silicone may be comprised of a polydimethysiloxane (PDMS) material.
- PDMS polydimethysiloxane
- the silicone hemispherical dome may be based on a Poly dimethylsiloxane (PDMS) material.
- the solid boy fingertip may comprise a chemically grown metallic silver reflective layer and an outer layer to protect the reflective layer.
- the multimodal modular sensor may additionally comprise multimodal electronics.
- the multi-modal electronics may extend past the traditional limits of the vision-based modality, which may be limited by the capture rate of the CMOS. Instead, the optical system disclosed herein may capture at a variable frames per second (e.g., 240 fps, depending on the capability of the CMOS) whereas the multimodal inclusion may allow' for capturing temporal data up to 10,000 Hz.
- the multimodal electronics may be enabled by a custom over molding and molding technology' by placing the electronics directly into the mold, injecting silicone, and placing the mold under vacuum to ensure the liquid silicone can entire all portholes and crevices around the off-the-shelf sensors.
- the one or more non-image sensors may comprise one or more of an inertial measurement unit (IMU) sensor, a microphone, an environmental sensor, a gas sensor, a pressure sensor, or a temperature sensor.
- the multimodal modular sensor may comprise inertial measurement units (IMU) in the main housing for processing electronics.
- the IMU may be rigidly coupled to the enclosure and fingertip.
- the IMU may be used for sensing vibrations, rotations, and position of the fingertip.
- the multimodal modular sensor may also comprise MEMS based microphones in the multimodal sensing printed circuit board (PCB), which may be over molded directly to the soft silicone solid body fingertip. In one example embodiment, there may be two microphones with a top porthole, and two with a bottom porthole.
- PCB printed circuit board
- Two microphones may be digital, and two microphones may be analog which differ in their sensing frequency ranges to cover a larger bandw idth. These microphones may provide surface audio textures similar to what human’s fingertip would perceive when scratching the surface of an object, sampling object-object or object-environment interactions.
- the multimodal modular sensor may additionally comprise an environmental sensor in the main housing. Internal air flow may be provided by a micro-sized inlet fan, which creates airflow through the device.
- the environmental sensor may be capable of sampling local air for air pressure, temperature, moisture and humidity.
- the multimodal modular sensor may also comprise a gas sensor in the main housing. Internal air flow may be provided by a micro-sized inlet fan which creates airflow through the device.
- the gas sensor may identify chemical compounds and compositions of local air samples to provide object state information and for classification of different objects.
- the multimodal modular sensor may further comprise pressure/temperature sensor in the multimodal sensing PCB, which may be over molded directly to the soft silicone solid body fingertip. The pressure/temperature sensor may measure absolute values of compression force being applied to the fingertip and capture heat gradients caused by a heat source on the fingertip and heat flow due to the metallic reflective layer.
- the multimodal modular sensor may comprise processing electronics.
- the system may further comprise a stack-up comprising a plurality printed circuit boards for the one or more processors, the omnidirectional optical system, and a data transfer system, wherein the plurality printed circuit boards share a common electrical interface and connector stack.
- the processing electronics may be in a stack-up that combines in a limited space five separate and unique PCB's which share a common electrical interface and connector stack.
- the processing electronics may comprise a microprocessor, a neural-network accelerator, an image capture system, and a data transfer system.
- the one or more processors may comprise one or more of a microprocessor or an accelerator.
- the microprocessor may be responsible for acquiring all the data from the sensors, lightweight processing, and configuration of processing and sampling parameters.
- the one or more machine-learning models may comprise one or more neural-network models.
- the one or more processors may comprise one or more neural-network accelerators which are configured for accelerating real-time inference on the accessed image and sensing data by the one or more neural -network models.
- the neural-network accelerator may have direct access to the microprocessor to obtain a selectable stream of data, whether this be image data or non-image data.
- the neural-network accelerator may provide neural-network acceleration which can be uploaded based on offline training to perform real-time inference on the input data streams.
- the neural-network accelerator may also have direct access to providing output control signals to a secondary device.
- the secondary device may comprise a robotic end effector.
- the secondary device may be an individual finger on a robotic hand, which may allow for rapid reflexes of the finger based on input data.
- the image capture system may be responsible for capturing image frames from the complementary metal-oxide semiconductor (CMOS) and configuring the sampling parameters towards different frames per second and resolutions.
- CMOS complementary metal-oxide semiconductor
- the data transfer system may provide a USB interface to a host computer which combines all multi-modal information and image/vision information over a single connector, this also powers the device.
- these applications may include medical applications such as prosthetics, palpation, sensing and localization (e.g., breast, testicular cancers, etc.), remote surgery, etc.
- these applications may include advertisements such as e-commerce, capturing the feeling of clothes, understanding the feeling of skin and the application of cosmetics, etc.
- these applications may include agriculture such as fruit/vegetable picking, food quality determination, food handling, etc.
- these applications may include haptics such as using artificial fingertips to capture the physical word accurately.
- FIG. 1 illustrates an example cutaway diagram 100 of the artificial fingertip disclosed herein and its modular electronics.
- Conventional efforts in the field may have involved subsets of sensing modalities, optimized for cost, for iterations in fingertip geometries, or for maximizing a particular design metric. But those choices may have limitations in performance.
- the main sensing modality of vision-based tactile sensors may capture the geometry of the object being touched. From that geometric data it may be possible to reconstruct normal and shear forces. However, that particular modality may fall short of the multimodal nature of human skin, which uses many different types of receptors (such as mechanoreceptors, thermoreceptors and nocioceptors).
- the disclosed artificial fingertip as shown in FIG.
- the disclosed artificial fingertip may include 5 printed circuit boards directly within the fingertip. This is a self-contained design. All of these printed circuit boards are custom designed. From top to bottom, the disclosed artificial fingertip comprise a multimodal sensor system, CMOS, an image capture system, processing and neural-network accelerator, and data transfer system.
- FIG. 2A illustrates example evaluations of normal and shear forces in three separate regions of the sensor, from the tip toward the side.
- the dots and error bar show the median and 95-percentile of the error, respectively.
- the median error from the deep-learning model for the three regions is 1.01 mN, 1.09 mN, 1.41 mN for normal forces, and 1.27 mN, 1.48 mN, 1.64 mN for shear forces.
- FIG. 2B illustrates an example prediction of normal and shear forces from visuo-tactile image output.
- Particular embodiments may train a deep-leaming model to predict normal and shear forces from visuo-tactile image output.
- normal and shear forces may be predicted with a median error of 1.01 mN and 1.27 mN respectively.
- predicting shear force may require the use of markers.
- FIG. 2C illustrates example evaluations of spatial resolution by using a dual pronged microindenter depressed into the artificial fingertip with vary ing width.
- FIG. 2D illustrates example methods which create the artificial fingertip volume.
- the top row corresponds to a method based on internal structure and the bottom row corresponds to a method based on solid gel with immersion lens.
- illumination artifacts are visible whereas with a solid volume, the resulting image is far higher quality with less illumination artifacts.
- FIG. 2E illustrates an example simulation of the effects of increasing the surface scattering along the internal reflective layer of the artificial fingertip. From left to right, machine polish of 1 ° to Lambertian scattering, the embodiments disclosed herein optimize for image contrast while constraining the background illumination uniformity'.
- the platform disclosed herein may have an elastomer that sen es as the touchsensing interface, with a subcutaneous camera that measures the deformation of the elastomer though structured light. But in addition to that ability, the platform disclosed herein may be capable of sensing multiple tactile modes (see FIG. 2), including contact intensity' and geometry, static and dynamic forces, surface audio textures and vibrations, heat change, fingertip velocity/acceleration/orientation. and even identification of some airborne chemical compounds. Further, to emulate the human reflex-arc, the embodiments disclosed herein introduce an on-device Al neural-network accelerator that provides next-generation touchprocessing capabilities, enables real-time local processing to minimize reaction delays, and reduces communication bandwidth.
- the artificial fingertip disclosed herein may be sensitive in both temporal and spatial domains. These domains may be obtained as modal signals encompassed within the artificial fingertip through visual, audio, vibration, pressure, heat and gas sensing. Commonly, conventional vision-based touch sensors using off-the-shelf imaging systems may be bound by slow visual capture rates which reduce the amount of sequential information resultant from frame encoding time and therefore limit the temporal nature of non-static touch interactions encountered during manipulation. Increasing the temporal frequency of the visual system may not be with benefit without an increase in spatial resolution for dynamic movements.
- the embodiments disclosed herein encompass the touch digitization system into the form of an artificial fingertip with similar geometry to a human finger.
- the surface of the fingertip may encode touch information from depressions in a reflective layer of which internal light reflections are captured by the camera.
- a visual tactile system may be utilized to resolve the minimum possible spatial features presented by an object interacting with the fingertip at high temporal rates.
- the artificial fingertip disclosed herein may advance through the spatial, temporal and multimodal performance through the embodiment of a modular platform for research into the digitization of touch. To achieve the high spatial and temporal performance demonstrated by the artificial fingertip disclosed herein, the embodiments disclosed herein leverage methodological breakthroughs in five different subsystems: elastomer interface, optical system, illumination system, multimodal sensing, and on-device Al processing.
- the surface layer material properties may directly relate to the spatial resolving capabilities.
- the embodiments disclosed herein developed a design-of-experiment technique to identify these 6 material parameters which affect sensor sensitivity to input stimuli: Rg (fingertip radius), Tc (surface reflective coating layer thickness), Tg (surface layer thickness), h (height), Ec (coating Young’s modulus), and Eg (fingertip volume Young’s modulus). If Tc and Tg are too thick or exhibit low compliance a low pass filter effect may be evident on discrete object edges. Similarly, if the object is rich in spatial information and fractal dimension, the fingertip surface may resolve less features due to local gradients from material compliance.
- Particular embodiments may avoid specifying any constraints on the coating thickness layer to best capture small input stimuli while maintaining a suitable parameter range for the general size of the fingertip.
- Developing this layer onto the fingertip surface may involve manual hand painting, airbrushing or dip-coating techniques.
- they may be far from optimal and result in large coating thickness, and inconsistent yield from manufacturing variance.
- the embodiments disclosed herein solve this issue by developing a new chemical deposition technique for the growing of a silver thin film directly onto the surface of the fingertip which produces coating thicknesses far smaller than previous methods and thus achieve better sensitivity.
- Common visuo-tactile sensors may capture input stimuli at a planar surface, use multiple cameras that are difficult to integrate and process together, or default to common off the shelf cameras optimized for human-centered imaging which result in downgraded optical performance in touch.
- Particular embodiments may refrain from the use of standard imagesensor features such as automatic exposure control, automatic white balance, and automatic focus which are designed for responding to changes in the natural environment as the fingertip chamber disclosed herein may be an enclosed and controlled environment.
- a new approach may be required to optimize the capture of the hemispherical surface.
- the imaging system disclosed herein may not limit the performance within the finite element method simulation of the material properties.
- particular embodiments may determine the optical system requirements to best suit capturing images related to tactile sensing with a CMOS pixel size of 1 . 1 urn.
- Parameters may be chosen for converging spot size to increase spatial resolution, intentionally allowing chromatic aberration, introducing shallow depth of field to allow for defocus proportional to object indentation depth, and removing anti-reflective coatings to allow capture and interpretation of reflections and scattering inside the fingertip.
- such parameters may require a non-standard lens. Therefore, particular embodiments may utilize a custom solid immersion hyperfisheye lens to tackle to unique environment of visuo-tactile sensing, rather than an off-the-shelf lens catering towards general purpose imaging thus enabling full control over lens geometry and optical parameters.
- the embodiments disclosed herein describe two metric parameters on the illumination performance within the volume, background uniformity, how evenly is the light distributed, and image to background uniformity contrast, how well do impressions on the surface of the fingertip stand out compared to the background as follows.
- a common approach may be the embodiment of an internal structure which serves as a hemispherical light pipe and provides fingertip rigidity.
- an internal light pipe structure may produce illumination artifacts in the form of glint and hotspots from the convex geometry 7 which contribute to a degradation in image metrics.
- conventional approaches may make use of a textured surface to induce Lambertian scattering of incident light rays.
- Particular embodiments may model the reflective layer surface properties with controlled degrees of scattering from polished to Lambertian, where the entire hemispherical surfaces may act as an integrating sphere, to show that a Lambertian scattering surface may be not the optimal approach to achieve high-performance.
- Particular embodiments may use a rigid solid volume, instead of the more common hollow volume or use of an internal support structure, in conjunction with controlled reflective surface scattering parameters moving away from Lambertian surfaces.
- FIG. 3 A illustrates the ability to determine the volume of water in an opaque container by tapping with one finger and recording the response on the fingers in contact with the container.
- the embodiments disclosed herein further show how this modality may be deconstructed into peak frequency analysis, which may be independent of finger position, whereas the decay time may be dependent on finger placement.
- FIG. 3B illustrates example spectrogram of the surface audio textures recorded for different objects.
- FIG. 3C illustrates example sensitivity 7 of the artificial fingertip towards heat gradients. Using a variable heat source as a control, the artificial fingertip may be sensitive towards heat gradients.
- FIG. 3D illustrates example accuracy of object identification through local gas sensing. The artificial fingertip may provide object state and identification of objects through local gas sensing achieving a 91% accuracy.
- FIG. 3E illustrates example accuracy of object classification through scent.
- the accuracy of object classification through scent may be dependent on the integration time as the artificial fingertip begins approaching the object.
- FIG. 2E shows that 61% accuracy is reached within 6 seconds.
- FIG. 3F illustrates an example localization of finger placement on an object during movement transients with empty and full volumes of liquid. The embodiments disclosed herein measure the effects transients during impulse, resonance during a static hold, and a static hold with no movement.
- Particular embodiments may further evolve the capabilities of the platform to include sensitivity to non-vision based modalities.
- sensitivity when in contact with the environment, dynamic forces and signals may be experienced; swiping the fingertip across a surface, or the very moment a contact transient or slip may occur.
- Particular embodiments may capture this information through in-fingertip audio microphones and pressure MEMS based sensors and show the ability to determine the level of liquid inside an opaque bottle (see FIG. 2A) and understand the nuances of surface texture between different objects at amuch higher frequency (upwards to, e.g., 10 kHz) than visual capture (e.g., 240 Hz) (see FIG. 2B).
- Particular embodiments may further include modalities to understand object state, which is not necessarily a function of contact, but can provide priors in understanding touch through heat and smell. Such priors may estimate if an object may be slippery' due to the presence of water, soap, or butter and if an object may present a danger to human-contact due to its temperature.
- Latency may result from the average time required to process a signal of interest and jitter may be the variation in mean time-based on system overhead which may occur due to host processing or bandwidth constraints.
- the embodiments disclosed herein show that by processing data directly on device through an onboard neural network accelerator, a 2x reduction in latency and jitter towards performing an action may be achieved in an example embodiment.
- a 3D finite element method (FEM) model using Comsol Multiphysics may be utilized for analyzing and characterizing the fingertip material stack-up.
- the 3D FEM model may identify the sensitivity and resolution of the sensor.
- a FEM model may be used to identify the key parameters that presented the largest change in sensitivity and resolution. Since the fingertip may be isotropic and may revolve around the origin, only a quarter of the sensor may be modeled for faster computation, using a multi-layer based model.
- the multi-layer model may comprise the base gel, polymer, and coating layers.
- Particular embodiments may use a particular system to generate nano-mechanical characterization of the fingertip polymer Young’s modulus, E.
- This system may have in-situ high-resolution imaging, dynamic nanoidentation and a high-precision motion stage with high- resolution force sensing tips.
- a 30 pN force may be applied with a 10 um probe tip.
- the FEM models may be updated to correct the simulations.
- Particular embodiments may further employ design-of-experiments techniques used to identify key parameters in affecting sensor sensitivity.
- six different parameters may be used: Rgel (gel radius), Tc (coating layer thickness), Tg (gel layer thickness), h (height), Ec (coating Young’s modulus), Eg (gel Young’s modulus).
- materials associated with the silicone hemispherical dome may be determined based on a plurality of material parameters comprising one or more of gel radius, coating layer thickness, gel layer thickness, heigh, coating Young’s modulus, or gel Young’s modulus.
- Entertaining a full factorial design of 6 parameters may lead to 64 models, thus a quarter-factorial design method may be used to reduce the design into 16 models.
- Analysis of variance and prediction analysis may result in Ec and Eg as main effects and interactions with coating and gel thicknesses.
- the parameters height h and gel radius Rgel may be removed from the model.
- design-of-experiments methods may be used through sweeping Young’s modulus parameters, Ec, Eg, and thickness parameters, Tg and Tc.
- FIG. 4 shows that for decreasing gel and coating thickness, sensitivity increases.
- FIG. 5 illustrates an example visualization of the result of FEM analysis with surface map graphs.
- FIG. 5 shows selecting regions of interest for background and indentation with multi-contact indentations across the surface of the fingertip, where the x-axis shows gel thickness, Tg values and y-axis show coating layer thickness, Tc for each combination of coating and gel Young’s modulus, Ec and Eg.
- FIG. 5 indicates that the material Young’s modulus may contribute significantly to yield a minimum required Tc and Tg for high performance. Similar FEM analysis was performed for the example conventional tactile sensor. A multilayer-based model may be used. The embodiments disclosed herein analyzed Tg (gel thickness), Tc coating thickness, Ec (coating Young’s modulus) and Eg (gel Young's modulus).
- Particular embodiments may determine average and maximum forces which can be applied to the fingertip before the soft hemispherical surface is damaged. Particular embodiments may fix the sensor to a force-torque sensor and apply an increasing force until the fingertip detaches from the body, which may occur at, for example, 40 N and 20 N for normal and shear forces respectably.
- FIG. 6 illustrates example maximum normal and shear force applied to the artificial fingertip surface prior to detachment from the main body. Particular embodiments may apply an incremental normal and shear force to the body of the soft artificial fingertip and record the ground truth force data from a stationary force-torque sensor. When an abrupt change occurs, this may indicate that maximum force has been reached.
- Young’s modulus may be an important parameter in sensor performance to input stimuli, and may require precise controller measurement.
- a set of dynamic mechanical thermal analysis (DMTA) measurements may be performed to obtain the global Young’s modulus of the gel.
- DMTA dynamic mechanical thermal analysis
- particular embodiments may measure the viscoelastic properties of polymers.
- an oscillating force may be applied to the material, and its response may be recorded to calculate the viscosity and stiffness of the material.
- the oscillating stress and strain measurements may be important in determining the viscoelastic properties of the material.
- sinusoidal stress and strain values may be measured.
- the phase difference between sinusoidal stress and strain may provide information about viscous and elastic properties of the material.
- Ideal elastic systems may have a 0° C phase angle while viscous systems may have a phase angle of 90° C.
- elastic response of a material may' be similar to storage energy and may be captured by storage modulus, while the viscous response may be considered as loss of energy, captured by loss modulus.
- the overall modulus of the viscoelastic material may be the combination of elastic and viscous components; in other words, summation of storage modulus and loss modulus.
- Another value, tan 5 may be used to compare the viscous and elastic moduli.
- DMTA may measure the change in the elastic modulus, loss modulus and tan 3 with respect to temperature. As the viscosity of the material is affected by the temperature and time, usually DMTA experiments may be performed at different temperatures and frequencies. Particular embodiments may use a common approach to select the operational conditions of the materials. Hence, for ideal sensitivity the fingertip may be expected to be used at room temperature and low frequency. As such, DMTA measurements may be taken at 25° C with frequency 5 Hz in an example embodiment.
- DMTA measurements may be done at 25 degC with frequency 5 Hz, shown in Table 1. Fingertip materials with lower Young’s modulus may be preferred in order to optimize for higher sensitivity.
- Particular embodiments may select a fabrication using a particular silicone encapsulating rubber using, e.g., a 0.8: 1 (part A to B) ratio for the gel fingertip base material, and another particular cured silicon rubber as thin protection layer for the thin-film layer.
- Table 1 DMTA ir leasurements performed on typical polymers used in fingertip gel manufacturing.
- the silicone hemispherical dome may be generated based on manufacturing a mold from aluminum, finishing the mold with a machine polishing pass, preparing the mold for gel casting through a salinization process in a desiccator, preparing a gel material using a cure silicone rubber compound, combining the gel material in a speed mixer under vacuum, casting the gel material into the mold, curing the casted gel material at a first temperature for a first amount of time, and removing a gel hemispherical dome from the mold once the casted gel material is cured.
- particular embodiments may manufacture the fingertip molds from 6061 aluminum and finish them with a machine polishing pass with a 3 mm diameter tool and a 50 pm step-over.
- the molds may be then prepared for gel casting through a silanization process in a desiccator with 50 pL silane under vacuum for 30 min.
- the gel material may be prepared using a 1: 1 ratio of the aforementioned silicon encapsulating rubber and combined in a speed mixer for 3 minutes under vacuum to release any captured air in the sample.
- the gel material may be then cast into the mold and allowed to cure at 23° C for 12 h. Once cured, the gel fingertip may be removed from the mold using tweezers for transfer to a glass slide.
- the reflective silver-film layer may be generated based on preparing a glucose solution by dissolving a first amount of glucose in a second amount of H2O and adding a third amount of KOH, preparing an AgNOs solution by dissolving a fourth amount of AgNOs in a fifth amount of H2O and adding a sixth amount of NH3, preparing a plating solution by mixing the glucose solution and AgNOs solution, cleaning the gel hemispherical dome using oxygen plasma for a second amount of time, activating the gel hemispherical dome in a solution of a seventh amount of SnCb in an eight amount of H2O for a third amount of time, suspending the gel hemispherical dome into the plating solution for a fourth amount of time, rinsing the gel hemispherical dome with H2O, and air drying the gel hemispherical dome.
- the steps for preparing the thin film metallic reflective layer on the gel fingertip through silver plating may be as follows. First, a glucose solution may be prepared by dissolving 2.035 g glucose in 160 mL H2O and then adding 0.224 g KOH. This may be set aside and the AgNOs solution may be prepared by dissolving 1.02 g AgNO? in 120 mL H2O, and then adding 1.2 g NH3 25%. The plating solution which is used to silver coat the gel fingertip may be then prepared by mixing 2 parts glucose solution, total 80 mL, to 1 part AgNO solution, total 40 mL. The silvering solution may be then set to gently stir. Prior to silver coating, the gel fingertip may be cleaned using oxygen plasma for 3 minutes.
- the gel fingertip may be then activated in a solution of 6.181 g SnCb in 98 mL H2O for 10 s. Once the gel fingertip is activated, it may be suspended into the silvering solution for a total of 3 minutes, then rinsed with H2O and air dried. This process may create a silvered reflective layer with 6 pm thickness.
- particular embodiments may coat the silvered layer in a white or black layer. This layer may be produced by using the aforementioned cured silicon rubber, a mixing ratio of part A to B of 1 : 1 and then adding 3% silicon color pigments to part A of the aforementioned cured silicon rubber.
- Part B of the cured silicon rubber may be then mixed in by weight according to the mixing ratio specified previously and then mixed in the speed mixer for 3 minutes under vacuum.
- the silvered gel fingertip may be then dipped into the pigment of the cured silicon rubber and set to cure for 6 hours.
- Common vision-based tactile sensors may make use of a static illumination configuration whereas some other sensors may use a single light color with colored acrylic to simulate multiple colors.
- Static illumination may be not ideal to promote a modular system. Rather, the illumination system should adapt to the needs of extracting information from the touch surface.
- Some conventional tactile sensors used a gel coated with a Lambertian scattering layer, in which volume illumination may produce an image by means of scattering light off the surface and into the vision system.
- the embodiments disclosed herein determine that Lambertian scattering may be not ideal for producing and optimizing for force and spatial sensitivity.
- the embodiments disclosed herein introduce a dynamic illumination system that provides volume illumination with configurable wavelength, intensity, and positioning.
- the illumination system may comprise 8 fully controllable RGB LEDs that emit Lambertian diffuse light, equally spaced around a circle of radius 9 mm.
- FIG. 7 illustrates an example cross-section of the fingertip gel coating showing the 3 layers: outer, silver and base gel.
- FIGS. 8A-8H illustrate an example hyperfisheye lens designed specifically for capturing omnidirectional tactile sensing images.
- FIG. 8A illustrates example fingertip regions of the hyperfisheye lens. Three regions of interest may be selected for optical system performance, tip, prominent contact surface, and base.
- the fingertip gel in the system disclosed herein may comprise three components, as seen in FIGS. 7-8.
- the outer surface of the base gel may have a reflective silver thin-film coating, which may be coated with a protective colored-diffusive material. To produce an image, the two layers may provide scattering of internally incident light from surface interactions to the vision system.
- Particular embodiments may place the illuminating light-emitting diodes in optical contact to the gel, using an over-molding process.
- the fingertip gels may be manufactured with a smooth and polished surface initially, and through mold texturing one may control and determine how- light is scatered at the interface.
- FIG. 8B illustrates an example lens of the hyperfisheye lens.
- FIG. 8C illustrates an example lens cutaway of the hyperfisheye lens.
- FIG. 8D illustrates an example system layout of solid immersion hyperfisheye lens system.
- the design configuration comprises a 25-mm diameter silicone hyperhemispherical dome in tandem with a 5P lens system.
- This silicone-lens system combination images object points located at or near the dome surface to the sensor plane.
- FIG. 8D also shows the lens system sizes/dimensions.
- FIG. 8E illustrates an example lens layout and lens element plastic material.
- the lens system may utilize EP10000 plastic material for the LI lens element while lens elements L2, L3, L4, and L5 are designed using APEL 5014 plastic material.
- FIG. 8F illustrates an example lens lay out and minimum clear aperture radii.
- the first lens element may be configured to have a maximum clear aperture diameter of 10 millimeters.
- lens element LI may have a maximum clear aperture diameter of 10 mm and in contact (no air space) with the hyper hemispherical silicone dome element.
- FIG. 8F also shows additional example clear aperture data (based on radius in millimeter) as: CIR S4 4.748376, CIR S5 1.707403, CIR S6 1.341748, CIR S7 0.538351, CIR S8 0.151529.
- FIG. 8G illustrate example solid models of silicone plus lens.
- FIG. 8H illustrates example solid models of lens.
- FIG. 9 illustrates example inference latency measurements comparing on-device to host incorporating the tactile data preprocessing and transfer stages.
- FIG. 9 shows that using the on-device CNN accelerator improves inference time and reduces the total latency to producing an action for a MLP deep neural network. A surface texture that minimizes scattering may produce little background illumination.
- the hemispherical surface of the gel fingertip may act as an integrating sphere. While Lambertian scattering provides uniform background illumination, the high scattering illumination from nearby interactions may reduce the overall indentation contrast.
- FIG. 10 illustrates example evaluations of non-uniformity metrics. Minimizing the background non-uniformity with increasing scatter angle may improve the detection of surface indentations.
- the embodiments disclosed herein define a contrast-to-noise (CNR) metric and study three regions of interest on the hemispherical surface for background uniformity noise and indentation contrast. Plotting the calculated CNR across the hemispherical surface for the different scatter angles, it is observed that the CNR is generally higher for less scatter, but CNR is more uniform across the FOV for more scatter. Therefore, the embodiments disclosed herein determine that for a hemispherical fingertip surface the desired texture scattering profile may be constrained between, e.g., half-width-half-max angles of 20° to 25°.
- the embodiments disclosed herein comprise a platform based on modular principles to provide an omnidirectional vision-based, multimodal sensing system, and on- device Al capabilities. Particular embodiments may achieve modularity through isolating each part of the system into an electronic assembly with small surface area. Specifically, these modules may include sensing, optics, vision capture, processing, and communications subsystems shown in FIG. 2. By facilitating re-use of each sub-system, the embodiments disclosed herein present a novel solution to advance the technology in tactile sensor design — reducing the required design, iteration, and manufacturing processes. This may enable one skilled of the art to modify the overall system by adding, removing, and changing sub-systems, rather than having to design a stack of new hardware to support changes to system design.
- this modular system architecture may allow for selecting combinations of sub-systems to facilitate introduction of newer technologies. Feature removal may reduce costs in adapting the system for new mechanical form factors. As well, the possibility' of replacing the sensing fingertip may’ allow adaptability for different environments and tasks, e.g., by using different sensitivities and stiffness in the fingertip material, or by using fingertips with markers to introduce more prominent optical-flow features. Based on simulation to achieve optimal spatial and force resolution, the embodiments disclosed herein introduce a novel process for the design and manufacturing of the sensing fingertip.
- the system disclosed herein may provide communications with the host device over the USB 3.0 standard interface.
- Three separate streams may be provided for data transfer, supporting video, audio, and multimodal data.
- these streams may collectively output at a maximum rate of 148 MBps and below, depending on the configuration sent to the device.
- tactile sensors may be used in open-loop control, providing information to the host device for processing and additional actions to manipulators.
- edge Al may be added at the fingertip for the following reasons. First, edge Al may help create a latent representation of the data and reduce the overall bandwidth sent to the host device. Second, edge Al may help enable fast local decisions for transmitting actions to manipulators.
- edge Al may help improve the overall latency of the system while reducing variance in jitter, which is the change in latency.
- Particular embodiments may model the tactile fingertip system with a manipulator, where both systems may be connected to the host device in a star configuration. This may result in decisions — and actions resulting from tactile information — being processed through the host device and disseminated to the manipulators. For more data-intensive designs, capturing data from multiple fingers at once, this arrangement may result in unstable control schemes where information and action latency cannot be guaranteed.
- a particular neural network accelerator for example, a 9 core RISC-V compute cluster with Al acceleration, for on-device processing of selected data streams.
- the embodiments disclosed herein develop a fast reflex-like control loop using edge Al for local processing.
- Conventional paradigm of transferring the sensory input to a central control computer for processing, then sending back the control signals, may require high bandwidth while introducing communication latency.
- the paradigm disclosed herein may locally process the sensory input inside the fingertip using edge Al. This may allow drastic reductions of the required bandwidth while greatly minimizing communication latency and minimizing jitter.
- the embodiments disclosed herein performed an experimental comparison of these two paradigms, shown in Table 2, by measuring the end-to-end latency of the systems using a PCI- e based precision time measurement tool.
- the embodiments disclosed herein evaluated this experiment on a Linux machine with 64GB memory and a GPU.
- each section of the system was isolated, and samples were collected in repeated trials.
- the embodiments disclosed herein verified these results by subjecting the entire system to repeated measurements and comparing the timing results to the sum of the isolated components. This determined the areas that produce deterministic timing results, as well as highlighting the areas that sustained increased latency and jitter. Furthermore, these results indicated areas of performance improvements and design for future tactile sensors.
- the results for the entire control loop show how the edge Al paradigm results in a reduction of latency from 4 ms to 1 ms with a desirable smaller variance.
- appropriate edge- Al processing may be extended to further exploit the sequential nature of the camera FIFO memory 7 to parallelize the data capture with the processing, thus yielding even lower latency.
- selected horizontal lines may be sent for processing in the configured region of interest. This may be applicable when touch interactions are most likely to appear in certain regions on the omnidirectional fingertip disclosed herein.
- the system disclosed herein may support this region-of-interest data output selection for increased resolution and image-capture frequency.
- Table 2 Normal force prediction error (median) by surface ty pes and regions.
- FIG. 11 illustrates an example data capture pipeline 1100 for the vision system of the disclosed artificial fingertip and touch information from external stimulus.
- External stimulus 1102 may 7 be input to the exposure module 1104 of the image sensor 1106.
- the output of the exposure module 1104 may be input to the image buffer 1108 of the image sensor 1106.
- the output of the image buffer 1108 may be provided to the sub-sampling module 1110 of artificial fingertip (on-device) 1112.
- this disclosure describes an image sensor for processing external stimulus as an example, this disclosure contemplates any suitable modality or any suitable subset of modalities in replacement of the image sensor.
- the output of the sub-sampling module 1110 may be input to the SPI transfer module 11 14.
- the output of the SPI transfer module 11 14 may be input to an on-device inference module 1116 of the host inference module 1118.
- the output of the on-device inference module 1116 may be input to the finger action transfer module 1120 based on I2C to finger transfer.
- the output of the finger action transfer module 1120 may be used to generate a finger action 1122 for an allegro hand 1124a to execute.
- the output of the image buffer 1108 may be also provided to a USB transfer module 1126 of artificial fingertip (host) 1128.
- the output of the USB transfer module 1126 may be input to the sub-sampling module 1130 of the host inference module 1132.
- the output of the sub-sampling module 1130 may be input to the inference module 1134.
- the output of the inference module 1134 may be input to the finger action transfer module 1136 based on USB-CAN.
- the output of the finger action transfer module 1136 may go through a palm processing module 1138 of the allegro hand 1124b.
- the output of the palm processing module 1138 may be input to an I2C palm to finger transfer module 1140, the output of which may be used to generate a finger action 1142 for the allegro hand 1124b to execute.
- FIG. 12 illustrates example data collection and prediction.
- the solid line shows ground-truth shear and normal force trajectory during one indentation.
- the dotted scatter plot shows model-predicted shear and normal force.
- the embodiments disclosed herein establish two pipelines for data transfer and processing: on-device pipelines and host pipelines, shown in FIG. 12. Additionally, a hybrid mode may support transferring data to the host as well as processing it on the device. It is observed that this affects the vision system because it may involve substantially larger amounts of data than does the multimodal data system.
- the limiting parameter for dynamic manipulations in a task is the latency between input data, processing, and action commands. To elicit rapid responses to changes in the environment that are detected through changes in grasp stability, particular embodiments may move toward faster processing and a minimization of latency between input and action.
- the embodiments disclosed herein studied the effects of the vision system because this may be the most common modality used in touch sensors, with impacts on overall system latency for host and on-device configurations.
- the constraining factor on system latency between real-world input and data-processing input is the capture rate of the vision system. This may be limited by the frames per second rate, which may impose a delay of 1 /fps, and the internal processing of the image-signal processor.
- particular embodiments may incorporate a CMOS sensor with 240 fps and a pixel size of 1.1 urn in the system disclosed herein, which may yield a shorter delay of 4. 17 ms as opposed to conventional sensors, which may operate at 60 Hz and thus have delays no fewer than 16.7 ms.
- the embodiments disclosed herein evaluated the inference latency with two deep neural networks, MLP and MobileNetV2, for two scenarios: on-device and host inference.
- the two largest sources of latency arise from the tactile data transfer from device to host, and of action data from host to robotic end effector.
- Tlatency For example, within the available headroom between the differences in pipeline latency, Tlatency, an upper limit of Tlatency ⁇ 2.463 ms was established.
- Tlatency an upper limit of Tlatency ⁇ 2.463 ms was established.
- Tlatency an upper limit of Tlatency ⁇ 2.463 ms was established.
- the embodiments disclosed herein increased the layer depth and observed the latency cost for both scenarios.
- Table 3 shows a ⁇ 4x decrease in action latency for dynamic tasks which involve high velocity movements thereby reducing total time to action to less than 1 ms.
- Table 3 Average data pipeline timing for host and on-device processing.
- FIG. 13 illustrates example simulated performance of the vision system of the artificial fingertip disclosed herein from on-axis to far-field contact for increasing line pairs per millimeters translated to spatial resolution for sagittal and tangential responses.
- FIG. 13 it becomes apparent that using the on-device accelerator without enabling the hardware engine quickly exceeds Tlatency at 10-layer depth.
- enabling hardware acceleration may allow for using MLP models with 60 layers.
- FIG. 14 illustrates example average pipeline latency for a MobileNetV2 network from acquiring tactile data, transferring data, pre-processing, inference to providing actions for varying image and channel width sizes.
- FIG. 15 illustrates an example comparison of touch information from the artificial fingertip disclosed herein tapping a water bottle with varying volumes of water from empty, half filled, and full. While the vision output from the sensor looks nearly identical with only variations in location of touch, multimodal data may provide a larger insight on object properties beyond texture.
- FIG. 14 illustrates example average pipeline latency for a MobileNetV2 network from acquiring tactile data, transferring data, pre-processing, inference to providing actions for varying image and channel width sizes.
- FIG. 15 illustrates an example comparison of touch information from the artificial fingertip disclosed herein tapping a water bottle with varying volumes of water from empty, half filled, and full. While the vision output from the sensor looks nearly identical with only variations in location of touch, multimodal data may provide a larger insight on object properties beyond texture.
- FIG. 14 illustrates example average pipeline
- Tlatency upper boundary may be determined by the output data rate, size of data transfer, and the host system performance.
- the host system may be running a plethora of control and processing applications, with additional overhead of communication between other sensors and devices that introduce overhead to Tlatency of, e.g., 1.2 ms.
- the embodiments disclosed herein observed an overhead of e.g., 4.7 ms.
- These differences may be attributed to the frame rate of the artificial fingertip disclosed herein, e.g., 240 fps, and data transfer over USB 3.0, whereas the example conventional tactile sensor may be limited to 60 fps using USB 2.0.
- the system disclosed herein may enable tactile on-device inference xxi th low latency control with the ability to have reflex-like control of the device to which the system is connected, as well as providing to the host abstractions of lower-level touch signals.
- An example may be training an on-device model to regress force from multimodal data to introduce touch and manipulation force limits to objects.
- Another example may be using the on-device Al capabilities to recognize slip and with low latency provide actions to the robotic end effector to reconfigure grasp.
- FIG. 16 illustrates an example 6-DoF robot indenter for testing tactile sensor force resolution.
- the robot arm and stage setup may be capable of precisely applying measured force onto a target device with controlled contact spatial location and orientation.
- FIGS. 17A-17B illustrate example image snapshots taken by the artificial fingertip disclosed herein from shear force data collection in two key moments. The timestamp t corresponds to the trajectory’ in FIG. 12.
- the overlay arrow field shows the optical flow with respect to the image without any force applied. As shown in FIGS.
- a tactile sensor is mounted on the robot arm to orient the desired test surface down against a probe with precision of 5 pm.
- the probe with a hemisphere tip of 4 mm diameter is mounted on a force sensor measuring the ground-truth contact force with 1 mN accuracy.
- the probe and force sensor assembly are then mounted on a hexapod which can be precisely controlled to translate with 0.1 pm and rotate with 0.05° increment. Due to the rotational symmetry of the artificial fingertip, the embodiments disclosed herein break down the fullsurface force characterization into three representative approximately planar regions. For each region, the embodiments disclosed herein repeat the collection process similarly for normal and shear forces.
- the embodiments disclosed herein start with normal force collection.
- particular embodiments may use the single-axis force sensor that can measure up to 250 mm.
- the robotic indenter may spatially sample 0.5 mm-spaced grid points on the tangential plane.
- the probe may move perpendicular from the plane, pressing into the sensor until the normal force reaches 200 mm.
- normal force Fnorm > 0.2 mN
- both sensor images and measured normal force may be collected synchronously.
- the embodiments disclosed herein collect about 550 image-force pairs per spatial point. For a 7 mm x 6 mm region, the embodiments disclosed herein obtain approximately 12, 000 points. This point data is randomly split into training (70%) and testing set (30%).
- FIG. 18 illustrates an example normal force prediction error distribution by surface types and regions. The dots and error bar show the median and 95-percentile of the error respectively.
- FIG. 18 shows how each shear-force indentation trajectory may be controlled. For example, first, the probe may be moved perpendicular to the contact surface to apply up to 600 mm normal force. Next, the probe may be moved tangential to the surface, loading shear force up to 100 mm. Finally, the probe may be moved back to the previous location, unloading shear force.
- Contact-force prediction on vision-based tactile sensors such as the system disclosed herein may be achieved using an image-to-force regression model.
- the model may be calibrated from reference data. Once calibrated, particular embodiments may evaluate the sensor-model as a system for force-sensing performance on a testing dataset.
- the embodiments disclosed herein collected the dataset for training and evaluating the model to benchmark normal and shear-force-sensing performance.
- Particular embodiments may use a modified ResNet50 deep neural network for the image-to-force regression model.
- the network may take an input image of 224 x 224 x 3 and output 1024-way object-classification probabilities.
- Particular embodiments may replace the classification head with a scalar-output linear layer predicting the force.
- Particular embodiment may use mean-square error as the loss, then optimize with Adam with initial learning rate search.
- the raw images from the sensors are 640 x 480, which may be dow n scaled to 224 x 224 with 20- pixel spatial jitter to improve spatial invariance.
- Particular embodiments may pool training data from all three regions to train a single model and obtain the prediction performance (median error) breakdown by regions as described in FIG. 2A.
- FIG. 19 illustrates an example capture amongst vision-based touch sensors. Objects shown from top to bottom: sandpaper, cloth, oyster and a configure cone. FIG. 19 shows the center-crop of the image captured by the system disclosed herein, where the texture contrast is more evident in the region with strong specular reflections from the LEDs.
- Particular embodiments may establish a modality within the artificial fingertip to determine object state and obtain clues on object classification.
- Particular embodiments may identify tw o key performance metrics for fingertip gas sensing: accuracy and signal acquisition time.
- the embodiments disclosed herein observe 6 different materials from liquid to solid commonly found in a household environment. These materials are coffee powder, liquid coffee, a nondescript rubber material, cheese, and a spread of soap and butter on a surface. All the materials were sampled at room temperature with a robotic arm and the disclosed artificial fingertip approaching the samples to near contact, within 1 cm, for a duration of 90 s.
- the embodiments disclosed herein record multimodal data and isolate the humidity, temperature, pressure, gas oxidation resistance datapoints at the maximum output frequency for each sampling modality.
- the embodiments disclosed herein sample air from the local environment.
- the raw data with the modalities of interest listed above are provided as inputs to a multi-layer perceptron network with a single 64-node hidden layer.
- the embodiments disclosed herein train the network with cross-entropy loss using an Adam optimizer with learning rate 0.1.
- the embodiments disclosed herein show the final accuracy of the model is not sensitive with respect to the size of hidden layers or learning rate. For example, the embodiments disclosed herein show that through these 6 materials a classification accuracy of 91%. Furthermore, as another example, the embodiments disclosed herein show the signal acquisition time to reach 66% accuracy.
- the embodiments disclosed herein evaluate the disclosed artificial fingertip in performance with respect to spatial resolution, shear and normal forces, illumination, vibrations, heat, and local gas sensitivity.
- FIG. 20A illustrates an example analog of the human reflex-arc by quickly processing sensory input within the fingertip, directly controlling the actuators of a robot hand to retract in response to touching an object.
- FIG. 20B illustrates example tactile processing and control paradigm transfer of sensory data to a remote computer for processing. This requires sufficient bandwidth and introduces communication latency.
- FIG. 20C illustrates example local processing for mimicking the reflex-arc with the system fingertip. The system disclosed herein may use the on-device Al neural-network accelerator for local processing to decrease overall latency between event and action.
- FIG. 20D illustrates example mean and standard deviation of the event-to-action latency.
- the on-device local processing and control loop may take 1.2 ms compared to the conventional paradigm, which may take 2.5 ms on the disclosed artificial fingertip and over 6 ms on the example conventional tactile sensor.
- the embodiments disclosed herein model the fingertip surface as a two-layer stack formed by an external diffusive material adhered to an internal reflective thin-film which is grown onto the non-rigid solid silicone body of the fingertip.
- the silicone hemispherical dome may further comprise a protective diffusive layer coated to the reflective silver-film layer.
- the embodiments disclosed herein then explore the effects of the non-rigid solid silicone surface mechanical properties, texture and the degree of controlled light scattering to find an optimal performance metric between background uniformity and image contrast.
- the embodiments disclosed herein show that increasing controlled surface texture scatter from 1 -degree scattering to a Lambertian scatter results in an increase in background illumination uniformity thereby an increase in image impression contrast.
- the embodiments disclosed herein show that the hemispherical sensing surface may act as an integrating sphere, where shadows may cast by direct illumination striking the indentations are wiped out by scattered illumination from other areas and even while imaging may occur on far off-axis angles, their contrast may be low.
- the embodiments disclosed herein introduce a controlled degree of scattering in which an optimized uniform background illumination may be achieved that lends itself well to contrast between indentations and the surrounding surface, furthermore, all indentations are imaged (see FIG. 2).
- the embodiments disclosed herein evaluate the normal force sensitivity and first collect tuples of normal forces applied by a micro-indenter and corresponding outputs from the sensors, and then train a deep-learning model from this dataset.
- the trained model may predict the normal forces applied with a median error of 1.01 mN (Region 1).
- the embodiments disclosed herein first collect tuples of shear forces applied by a micro-indenter and corresponding outputs from the sensors, and then train a deep-leaming model from this dataset.
- the model (see FIG. 2A) is capable of predicting shear forces applied with a median error of 1.27 mN (Region 1).
- the embodiments disclosed herein may directly use the internal texture of the elastomer to measure shear forces.
- the embodiments disclosed herein define the spatial resolution of an artificial fingertip sensor as the minimum feature size that can be resolved with an MTF > 0.5; this may be determined by how well the contrast is preserved quantified by line pairs per millimeter.
- the embodiments disclosed herein first simulate the imaging system from the design which yields that on-axis contacts are resolvable for features of size >6 um for region 1, >8 um for region 2 resolves, and >22 um for region 3.
- the embodiments disclosed herein then validate these results by collecting data with a two-pronged micro-indenter depressed onto the fingertip, varying the distance between the two prongs and observing the taxel intensity line profile; both the visual validation and the inspection of the taxels profile intensity confirmed the embodiments disclosed herein may clearly distinguish features as small as >7 urn for region 1 (see FIG. 2C).
- Multimodal information such as vibrations upwards of 10 kHz, auditory clues, sensitivity to heat and smells may play an important role in human touch.
- typical vision-based tactile sensors may not contain a broad range of multimodal capabilities to capture this information or operate lower sensing frequencies such as 60 Hz.
- the embodiments disclosed herein evaluate capturing vibrations up to 10 kHz which may be enough to distinguish between different materials upon a simple light sliding of the finger.
- the embodiments disclosed herein show that these multimodal features can be used to detect the amount of liquid inside a bottle by simply tapping it with a fingertip (see FIG.
- the embodiments disclosed herein may detect changes in heat gradients which reflect object state, room temperature, warm, hot, dangerous (see FIG. 2C).
- object state there may be a limited amount of information which is captured during visuo-tactile contact.
- the embodiments disclosed herein employ the use of local gas sensing at each fingertip to understand nuances in object state, e.g., determining if the object is slippery or wet. With this modality, the embodiments disclosed herein may sense these parameters during approach and contact (see FIG. 2E).
- the embodiments disclosed herein evaluate contact different samples which not only provide gas signatures, but local environmental information such as humidity and temperature gradients to distinguish between two similarly looking liquids and coffee or coffee grind (see FIG. 2D).
- the use of multimodal sensing may complement the main vision-based sensing modality and enable future work into the importance of the different touch modes for task-specific applications.
- the embodiments disclosed herein demonstrate a fast reflex-like control loop using the on-device Al neural-network accelerator for local processing.
- on-device processing on the artificial fingertip disclosed herein may reduce latency, for example, from 6 ms to 1.2 ms (see FIG. 3D).
- the embodiments disclosed herein may advance the state of artificial fingertip sensing towards digitizing fingertip interactions between the environment and objects.
- the embodiments disclosed herein disclose an artificial fingertip that may be more sensitive in spatial and force sensitivity compared to conventional methods with the additional technical advantage of multimodal sensing features and a local processing ability .
- Experimental results demonstrate the digitization of touch with capabilities that outperform a human fingertip.
- the richness of touch digitized by the disclosed modular platform may open new promising venues into studying the nature of touch in humans and investigating key questions around the digitization and processing of touch as a sensor modality.
- the embodiments disclosed herein may open the doors to a wider adoption of touch sensors beyond conventional niche fields:
- robotics to improve sensing and manipulation capabilities with benefits for applications in manufacturing and logistics, medical robotics, agricultural robotics, and consumer-level robotics;
- artificial intelligence to investigate the learning of appropriate tactile and multimodal representations, and corresponding computational models that can better exploit the active, spatial and temporal nature of touch.
- Further potential applications may include virtual reality and telepresence, prosthesis, and e-commerce.
- arranged to, capable of, configured to, enabled to, operable to, or operative to perform a particular function encompasses that apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, as long as that apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative. Additionally, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide none, some, or all of these advantages.
Landscapes
- Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
In one embodiment, a system includes a silicone hemispherical dome and an omnidirectional optical system. The dome includes a surface including a reflective silver-film layer. The optical system includes a lens including multiple lens elements with the first lens element in direct contact with the hemispherical dome without airgap. The lens is configured to capture scattering of internal incident light generated by the reflective silver-film layer. The optical system also includes an image sensor configured to generate image data from data captured by the lens. The system also includes non-image sensors. The system further includes processors and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to access the image data from the omnidirectional optical system and sensing data from the non-image sensors and generate touch digitization based on the accessed image and sensing data by machine-learning models.
Description
Digitizing Touch with Artificial Robotic Fingertip
PRIORITY
[0001] This application claims the benefit under 35 U.S.C. § 119(e) of U.S. Provisional Patent Application No. 63/383069, filed 09 November 2022, which is incorporated herein by reference.
TECHNICAL FIELD
[0002] This disclosure generally relates to robotics, and in particular relates to hardware and software for smart robotic sensing.
BACKGROUND
[0003] Artificial intelligence (Al) is the intelligence of machines or software. Al technology is widely used throughout industry, government and science, such as advanced web search engines, recommendation systems, understanding human speech, self-driving cars, generative or creative tools, and competing at the highest level in strategic games.
[0004] Robotics is an interdisciplinary branch of electronics and communication, computer science and engineering. Robotics involves the design, construction, operation, and use of robots. The goal of robotics is to design machines that can help and assist humans. Robotics integrates fields of mechanical engineering, electrical engineering, information engineering, mechatronics engineering, electronics, biomedical engineering, computer engineering, control systems engineering, software engineering, mathematics, etc. The field of robotics develops machines that can automate tasks and do various jobs that a human might not be able to do. Certain robots require user input to operate, while other robots function autonomously.
SUMMARY OF PARTICULAR EMBODIMENTS
[0005] Touch is a sensing modality that may provide rich information about object properties and interactions with the physical environment. Humans and robots both benefit from using touch to perceive and interact with the surrounding environment. However, no existing systems may provide rich, multimodal digital touch-sensing capabilities while retaining the form factor of a human finger. The embodiments disclosed herein may improve the digitization of touch with technological advances embodied in an artificial finger-shaped sensor with enhanced sensing capabilities. In particular embodiments, the artificial fingertip may comprise high-resolution sensors (e.g.. approximately 8.3 million taxels) that respond to omnidirectional touch, capture multimodal signals, and use on-device artificial intelligence to process the data in real time. For example, in particular embodiments, evaluations show that the artificial fingertip can resolve spatial features as small as 7 um, sense normal and shear
forces with a resolution between, e.g., 1 mN and 1.3 mN respectively, perceive vibrations up to, e.g., 9-11 kHz, sense odor, and even sense heat. Furthermore, the on-device Al neural- network accelerator may act as a peripheral nervous system on a robot and mimic the reflexarc found in humans. These results demonstrate the embodiments disclosed herein may digitize touch with enhanced performance. The embodiments disclosed herein may be applied in fields including robotics (industrial, medical, agricultural, and consumer-level), virtual reality and telepresence, prosthesis, and e-commerce. Although this disclosure describes digitizing particular modality in a particular manner, this disclosure contemplates digitizing any suitable modality in any suitable manner.
[0006] In particular embodiments, a system for touch digitization may comprise a silicone hemispherical dome comprising a surface comprising a reflective silver-film layer. The system may additionally comprise an omnidirectional optical system comprising a lens comprising a plurality of lens elements and an image sensor configured to generate image data from data captured by the lens. In particular embodiments, a first lens element of the plurality of lens elements may be in direct contact with the hemispherical dome without airgap. The lens may be configured to capture scattering of internal incident light generated by the reflective silver- film layer. The system may also comprise one or more non-image sensors disposed underneath the omnidirectional optical system. The system may further comprise one or more processors and a non-transitory memory coupled to the processors comprising instructions executable by the processors. In particular embodiments, the processors may be operable when executing the instructions to access the image data from the omnidirectional optical system and sensing data from the one or more non-image sensors and generate the touch digitization based on the accessed image and sensing data by one or more machine-learning models.
[0007] In particular embodiments, an artificial fingertip for touch digitization may comprise a silicone hemispherical dome. The artificial fingertip may also comprise an omnidirectional optical system comprising a lens comprising a plurality of lens elements and an image sensor configured to generate image data from data captured by the lens. In particular embodiments, a first lens element of the plurality of lens elements may be in direct contact with the silicone hemispherical dome without airgap. The artificial fingertip may further comprise one or more non-image sensors disposed underneath the omnidirectional optical system.
[0008] Certain embodiments disclosed herein may provide one or more technical advantages. A technical advantage of the embodiments may include improved sensitivity to input stimuli by using dynamic lighting with variable wavelength and direction for reconstruction of touch surface topology signals (images) as the system disclosed herein moves
beyond the traditional Lambertian scattering paradigm towards a controlled degree scattering surface along with a ground up approach on optimizing the material properties for the highest sensitivity to spatial features, which is achieved by developing a new process for chemically growing a thin-film layer of silver film onto the fingertip surface. Another technical advantage of the embodiments may include high spatial resolution in capturing minute details on the surface of the fingertip, which allows for bypassing traditional methods of capturing normal and shear forces through the use of markers. The embodiments disclosed herein utilize a 3- region solid immersion hyperfisheye lens design for touch digitization systems in a non-airgap configuration to a PDMS material. Due to the design and specific requirements for non-human imaging, the embodiments disclosed herein are able to maintain a superior performance of the modulation transfer function (MTF) through the entire field of the lens, thereby ensuring spatial performance along the tip, side and edges of the fingertip. In addition, the omnidirectional surface using a single camera and lens system may achieve full field view of over 200 degrees. Another technical advantage of the embodiments may include high temporal resolution based on interchangeable and modular electronic stack-up for touch digitization (vision capture, multi-modal capture, on-device Al processing), which combines all electronics and sensing elements into a familiar size and shape of a human thumb. Each individual system of the stack- up can be interchanged with a newer design to reduce the hardware development lifecycle, furthermore, components in the stack-up can be excluded to further reduce the size of the sensor based on individual requirements of the application. Another technical advantage of the embodiments may include digitizing touch signals for on-device Al processing for providing human-like reflex arc actions to robotic manipulators as a neural network processor is inside the fingertip for direct inference on input data and a processor can provide direct outputs to control a secondary device such as a robotic end effector. Certain embodiments disclosed herein may provide none, some, or all of the above technical advantages. One or more other technical advantages may be readily apparent to one skilled in the art in view of the figures, descriptions, and claims of the present disclosure.
[0009] The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. Embodiments according to the invention are in particular disclosed in the attached claims directed to a method, a storage medium, a system and a computer program product, wherein any feature mentioned in one claim category, e.g. method, can be claimed in another claim category, e.g. system, as well. The dependencies or references back in the attached claims
are chosen for formal reasons only. However any subject matter resulting from a deliberate reference back to any previous claims (in particular multiple dependencies) can be claimed as well, so that any combination of claims and the features thereof are disclosed and can be claimed regardless of the dependencies chosen in the attached claims. The subject-matter which can be claimed comprises not only the combinations of features as set out in the attached claims but also any other combination of features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein can be claimed in a separate claim and/or in any combination with any embodiment or feature described or depicted herein or with any of the features of the attached claims.
BRIEF DESCRIPTION OF THE DRAWINGS
[0010] FIG. 1 illustrates an example cutaway diagram of the artificial fingertip disclosed herein and its modular electronics.
[0011] FIG. 2A illustrates example evaluations of normal and shear forces in three separate regions of the sensor, from the tip toward the side.
[0012] FIG. 2B illustrates an example prediction of normal and shear forces from visuo- tactile image output.
[0013] FIG. 2C illustrates example evaluations of spatial resolution by using a dual pronged microindenter depressed into the artificial fingertip with varying width.
[0014] FIG. 2D illustrates example methods which create the artificial fingertip volume.
[0015] FIG. 2E illustrates an example simulation of the effects of increasing the surface scattering along the internal reflective layer of the artificial fingertip.
[0016] FIG. 3A illustrates the ability to determine the volume of water in an opaque container by tapping with one finger and recording the response on the fingers in contact with the container.
[0017] FIG. 3B illustrates example spectrogram of the surface audio textures recorded for different objects.
[0018] FIG. 3C illustrates example sensitivity’ of the artificial fingertip towards heat gradients.
[0019] FIG. 3D illustrates example accuracy of object identification through local gas sensing.
[0020] FIG. 3E illustrates example accuracy of object classification through scent.
[0021] FIG. 3F illustrates an example localization of finger placement on an object during movement transients with empty' and full volumes of liquid.
[0022] FIG. 4 illustrates an example touch sensitivity surface map plot of Ec = 5.0 and Eg = 3.0 MPa.
[0023] FIG. 5 illustrates an example visualization of the result of FEM analysis with surface map graphs.
[0024] FIG. 6 illustrates example maximum normal and shear force applied to the artificial fingertip surface prior to detachment from the main body.
[0025] FIG. 7 illustrates an example cross-section of the fingertip gel coating showing the 3 layers: outer, silver and base gel.
[0026] FIGS. 8A-8H illustrate example hyperfisheye lens designed specifically for capturing omnidirectional tactile sensing images.
[0027] FIG. 9 illustrates example inference latency measurements comparing on-device to host incorporating the tactile data pre-processing and transfer stages.
[0028] FIG. 10 illustrates example evaluations of non-uniformity metrics.
[0029] FIG. 11 illustrates an example data capture pipeline for the vision system of the disclosed artificial fingertip and touch information from external stimulus.
[0030] FIG. 12 illustrates example data collection and prediction.
[0031] FIG. 13 illustrates example simulated performance of the vision system of the artificial fingertip disclosed herein from on-axis to far-field contact for increasing line pairs per millimeters translated to spatial resolution for sagittal and tangential responses.
[0032] FIG. 14 illustrates example average pipeline latency for a MobileNetV2 network from acquiring tactile data, transferring data, pre-processing, inference to providing actions for varying image and channel width sizes.
[0033] FIG. 15 illustrates an example comparison of touch information from the artificial fingertip disclosed herein tapping a water bottle with varying volumes of water from empty, half filled, and full.
[0034] FIG. 16 illustrates an example 6-DoF robot indenter for testing tactile sensor force resolution.
[0035] FIGS. 17A-17B illustrate example image snapshots taken by the artificial fingertip disclosed herein from shear force data collection in two key moments.
[0036] FIG. 18 illustrates an example normal force prediction error distribution by surface types and regions.
[0037] FIG. 19 illustrates an example capture amongst vision-based touch sensors.
[0038] FIG. 20A illustrates an example analog of the human reflex-arc by quickly processing sensory input within the fingertip, directly controlling the actuators of a robot hand
to retract in response to touching an object.
[0039] FIG. 20B illustrates example tactile processing and control paradigm transfer of sensory data to a remote computer for processing.
[0040] FIG. 20C illustrates example local processing for mimicking the reflex-arc with the system fingertip.
[0041] FIG. 20D illustrates example mean and standard deviation of the event-to-action latency.
DESCRIPTION OF EXAMPLE EMBODIMENTS
[0042] Touch is a sensing modality that may provide rich information about object properties and interactions with the physical environment. Humans and robots both benefit from using touch to perceive and interact with the surrounding environment. However, no existing systems may provide rich, multimodal digital touch-sensing capabilities while retaining the form factor of a human finger. The embodiments disclosed herein may improve the digitization of touch with technological advances embodied in an artificial finger-shaped sensor with enhanced sensing capabilities. In particular embodiments, the artificial fingertip may comprise high-resolution sensors (e.g., approximately 8.3 million taxels) that respond to omnidirectional touch, capture multimodal signals, and use on-device artificial intelligence to process the data in real time. For example, in particular embodiments, evaluations show that the artificial fingertip can resolve spatial features as small as 7 um, sense normal and shear forces with a resolution between, e.g.. 1 mN and 1.3 mN respectively, perceive vibrations up to, e.g., 9-1 1 kHz, sense odor, and even sense heat. Furthermore, the on-device Al neural- network accelerator may act as a peripheral nervous system on a robot and mimic the reflexarc found in humans. These results demonstrate the embodiments disclosed herein may digitize touch with enhanced performance. The embodiments disclosed herein may be applied in fields including robotics (industrial, medical, agricultural, and consumer-level), virtual reality and telepresence, prosthesis, and e-commerce. Although this disclosure describes digitizing particular modality in a particular manner, this disclosure contemplates digitizing any suitable modality in any suitable manner.
[0043] In particular embodiments, a system for touch digitization may comprise a silicone hemispherical dome comprising a surface comprising a reflective silver-film layer. The system may additionally comprise an omnidirectional optical system comprising a lens comprising a plurality of lens elements and an image sensor configured to generate image data from data captured by the lens. In particular embodiments, a first lens element of the plurality of lens elements may be in direct contact with the silicone hemispherical dome without airgap. The
lens may be configured to capture scattering of internal incident light generated by the reflective silver-film layer. The system may also comprise one or more non-image sensors disposed underneath the omnidirectional optical system. The system may further comprise one or more processors and a non-transitory memory coupled to the processors comprising instructions executable by the processors. In particular embodiments, the processors may be operable when executing the instructions to access the image data from the omnidirectional optical system and sensing data from the one or more non-image sensors and generate the touch digitization based on the accessed image and sensing data by one or more machine-learning models.
[0044] In particular embodiments, an artificial fingertip for touch digitization may comprise a silicone hemispherical dome. The artificial fingertip may also comprise an omnidirectional optical system comprising a lens comprising a plurality of lens elements and an image sensor configured to generate image data from data captured by the lens. In particular embodiments, a first lens element of the plurality of lens elements may be in direct contact with the hemispherical dome without airgap. The artificial fingertip may further comprise one or more non-image sensors disposed underneath the omnidirectional optical system.
[0045] Of all human senses, touch may be the most critical in how humans interact with the world. Touch may enable humans to measure forces and recognize object properties, e.g., shape, weight, density, textures, friction, elasticity. Touch may also play an important role both in social relationships and in cognitive development. By comparison, the earliest efforts to impart that sense to robots came no closer than crude approximations. No solutions have emerged for digitizing touch with the same rich sensorial spectrum humans take for granted. Toward the advancement of robotic in-hand manipulation, the embodiments disclosed herein may mimic familiar features of the human hand: fingers. The touch digitization in the embodiments disclosed herein may enable intelligent systems to discern significantly higher levels of physical information during environmental interaction.
[0046] Digitizing touch may depend on two features including one temporal in nature and one spatial. Temporal features may process information from basic signals of time variation, whereas spatial features may process a discrete multidimensional array of temporal signals. The embodiments disclosed herein combine these methods within a unified platform to improve the capabilities of touch digitization, i.e., a modular, finger-shaped, multimodal tactile sensor with on-device artificial intelligence (Al) capabilities and superhuman performance.
[0047] In particular embodiments, the artificial fingertip may comprise a multimodal modular sensor. The multimodal modular sensor may comprise a main body mechanical
housing. As an example and not by way of limitation, the main body mechanical housing may have a similar shape and size to the human thumb. In particular embodiments, the silicone hemispherical dome, the omnidirectional optical system, the one or more non-image sensors, the one or more processors, and the non-transitory memory may be disposed in the housing. The multimodal modular sensor may also comprise a soft silicone solid body fingertip. As an example and not by way of limitation, the soft silicone may be comprised of a polydimethysiloxane (PDMS) material. In other words, the silicone hemispherical dome may be based on a Poly dimethylsiloxane (PDMS) material. The solid boy fingertip may comprise a chemically grown metallic silver reflective layer and an outer layer to protect the reflective layer. The multimodal modular sensor may additionally comprise multimodal electronics. The multi-modal electronics may extend past the traditional limits of the vision-based modality, which may be limited by the capture rate of the CMOS. Instead, the optical system disclosed herein may capture at a variable frames per second (e.g., 240 fps, depending on the capability of the CMOS) whereas the multimodal inclusion may allow' for capturing temporal data up to 10,000 Hz. This is a system level contribution using off-the-shelf components but assembling them in a manner which allows for directly sampling or capturing signals due to input stimuli on the fingertip surface. In particular embodiments, the multimodal electronics may be enabled by a custom over molding and molding technology' by placing the electronics directly into the mold, injecting silicone, and placing the mold under vacuum to ensure the liquid silicone can entire all portholes and crevices around the off-the-shelf sensors.
[0048] In particular embodiments, the one or more non-image sensors may comprise one or more of an inertial measurement unit (IMU) sensor, a microphone, an environmental sensor, a gas sensor, a pressure sensor, or a temperature sensor. The multimodal modular sensor may comprise inertial measurement units (IMU) in the main housing for processing electronics. The IMU may be rigidly coupled to the enclosure and fingertip. The IMU may be used for sensing vibrations, rotations, and position of the fingertip. The multimodal modular sensor may also comprise MEMS based microphones in the multimodal sensing printed circuit board (PCB), which may be over molded directly to the soft silicone solid body fingertip. In one example embodiment, there may be two microphones with a top porthole, and two with a bottom porthole. Two microphones may be digital, and two microphones may be analog which differ in their sensing frequency ranges to cover a larger bandw idth. These microphones may provide surface audio textures similar to what human’s fingertip would perceive when scratching the surface of an object, sampling object-object or object-environment interactions.
[0049] In particular embodiments, the multimodal modular sensor may additionally
comprise an environmental sensor in the main housing. Internal air flow may be provided by a micro-sized inlet fan, which creates airflow through the device. The environmental sensor may be capable of sampling local air for air pressure, temperature, moisture and humidity. The multimodal modular sensor may also comprise a gas sensor in the main housing. Internal air flow may be provided by a micro-sized inlet fan which creates airflow through the device. The gas sensor may identify chemical compounds and compositions of local air samples to provide object state information and for classification of different objects. The multimodal modular sensor may further comprise pressure/temperature sensor in the multimodal sensing PCB, which may be over molded directly to the soft silicone solid body fingertip. The pressure/temperature sensor may measure absolute values of compression force being applied to the fingertip and capture heat gradients caused by a heat source on the fingertip and heat flow due to the metallic reflective layer.
[0050] In particular embodiments, the multimodal modular sensor may comprise processing electronics. The system may further comprise a stack-up comprising a plurality printed circuit boards for the one or more processors, the omnidirectional optical system, and a data transfer system, wherein the plurality printed circuit boards share a common electrical interface and connector stack. In other words, the processing electronics may be in a stack-up that combines in a limited space five separate and unique PCB's which share a common electrical interface and connector stack. The processing electronics may comprise a microprocessor, a neural-network accelerator, an image capture system, and a data transfer system.
[0051] In particular embodiments, the one or more processors may comprise one or more of a microprocessor or an accelerator. The microprocessor may be responsible for acquiring all the data from the sensors, lightweight processing, and configuration of processing and sampling parameters. In particular embodiments, the one or more machine-learning models may comprise one or more neural-network models. Correspondingly, the one or more processors may comprise one or more neural-network accelerators which are configured for accelerating real-time inference on the accessed image and sensing data by the one or more neural -network models. The neural-network accelerator may have direct access to the microprocessor to obtain a selectable stream of data, whether this be image data or non-image data. The neural-network accelerator may provide neural-network acceleration which can be uploaded based on offline training to perform real-time inference on the input data streams. The neural-network accelerator may also have direct access to providing output control signals to a secondary device. In one example embodiment, the secondary device may comprise a
robotic end effector. As an example and not by way of limitation, the secondary device may be an individual finger on a robotic hand, which may allow for rapid reflexes of the finger based on input data. The image capture system may be responsible for capturing image frames from the complementary metal-oxide semiconductor (CMOS) and configuring the sampling parameters towards different frames per second and resolutions. The data transfer system may provide a USB interface to a host computer which combines all multi-modal information and image/vision information over a single connector, this also powers the device.
[0052] The embodiments disclosed herein may be used in a variety of applications. As an example and not by way of limitation, these applications may include medical applications such as prosthetics, palpation, sensing and localization (e.g., breast, testicular cancers, etc.), remote surgery, etc. As another example and not by way of limitation, these applications may include advertisements such as e-commerce, capturing the feeling of clothes, understanding the feeling of skin and the application of cosmetics, etc. As yet another example and not by way of limitation, these applications may include agriculture such as fruit/vegetable picking, food quality determination, food handling, etc. As yet an example and not by way of limitation, these applications may include haptics such as using artificial fingertips to capture the physical word accurately.
[0053] FIG. 1 illustrates an example cutaway diagram 100 of the artificial fingertip disclosed herein and its modular electronics. Conventional efforts in the field may have involved subsets of sensing modalities, optimized for cost, for iterations in fingertip geometries, or for maximizing a particular design metric. But those choices may have limitations in performance. The main sensing modality of vision-based tactile sensors may capture the geometry of the object being touched. From that geometric data it may be possible to reconstruct normal and shear forces. However, that particular modality may fall short of the multimodal nature of human skin, which uses many different types of receptors (such as mechanoreceptors, thermoreceptors and nocioceptors). In contrast with the conventional efforts, the disclosed artificial fingertip, as shown in FIG. 1, may include 5 printed circuit boards directly within the fingertip. This is a self-contained design. All of these printed circuit boards are custom designed. From top to bottom, the disclosed artificial fingertip comprise a multimodal sensor system, CMOS, an image capture system, processing and neural-network accelerator, and data transfer system.
[0054] In deploying a high-end modular research platform for investigating touch and its digitization, the embodiments disclosed herein introduce a novel approach. The platform disclosed herein, identified as an artificial fingertip, may belong to the family of vision-based
tactile sensors. FIG. 2A illustrates example evaluations of normal and shear forces in three separate regions of the sensor, from the tip toward the side. The dots and error bar show the median and 95-percentile of the error, respectively. For example, in particular embodiments, the median error from the deep-learning model for the three regions is 1.01 mN, 1.09 mN, 1.41 mN for normal forces, and 1.27 mN, 1.48 mN, 1.64 mN for shear forces. FIG. 2B illustrates an example prediction of normal and shear forces from visuo-tactile image output. Particular embodiments may train a deep-leaming model to predict normal and shear forces from visuo-tactile image output. For example, in particular embodiments, normal and shear forces may be predicted with a median error of 1.01 mN and 1.27 mN respectively. Compared to conventional methods, predicting shear force may require the use of markers. However, with increased spatial resolution, far more features may be extracted from the visuo-tactile image which aid in shear force prediction. FIG. 2C illustrates example evaluations of spatial resolution by using a dual pronged microindenter depressed into the artificial fingertip with vary ing width. Visual validation and the inspection of the taxels’ profile intensity confirmed the ability to clearly distinguish features as small as, e.g., 7 m. FIG. 2D illustrates example methods which create the artificial fingertip volume. The top row corresponds to a method based on internal structure and the bottom row corresponds to a method based on solid gel with immersion lens. With an internal structure, illumination artifacts are visible whereas with a solid volume, the resulting image is far higher quality with less illumination artifacts. FIG. 2E illustrates an example simulation of the effects of increasing the surface scattering along the internal reflective layer of the artificial fingertip. From left to right, machine polish of 1 ° to Lambertian scattering, the embodiments disclosed herein optimize for image contrast while constraining the background illumination uniformity'.
[0055] The platform disclosed herein may have an elastomer that sen es as the touchsensing interface, with a subcutaneous camera that measures the deformation of the elastomer though structured light. But in addition to that ability, the platform disclosed herein may be capable of sensing multiple tactile modes (see FIG. 2), including contact intensity' and geometry, static and dynamic forces, surface audio textures and vibrations, heat change, fingertip velocity/acceleration/orientation. and even identification of some airborne chemical compounds. Further, to emulate the human reflex-arc, the embodiments disclosed herein introduce an on-device Al neural-network accelerator that provides next-generation touchprocessing capabilities, enables real-time local processing to minimize reaction delays, and reduces communication bandwidth.
[0056] T o effectively capture the nuances in touch interactions with the world, the artificial
fingertip disclosed herein may be sensitive in both temporal and spatial domains. These domains may be obtained as modal signals encompassed within the artificial fingertip through visual, audio, vibration, pressure, heat and gas sensing. Commonly, conventional vision-based touch sensors using off-the-shelf imaging systems may be bound by slow visual capture rates which reduce the amount of sequential information resultant from frame encoding time and therefore limit the temporal nature of non-static touch interactions encountered during manipulation. Increasing the temporal frequency of the visual system may not be with benefit without an increase in spatial resolution for dynamic movements. The embodiments disclosed herein encompass the touch digitization system into the form of an artificial fingertip with similar geometry to a human finger. The surface of the fingertip may encode touch information from depressions in a reflective layer of which internal light reflections are captured by the camera. A visual tactile system may be utilized to resolve the minimum possible spatial features presented by an object interacting with the fingertip at high temporal rates. The artificial fingertip disclosed herein may advance through the spatial, temporal and multimodal performance through the embodiment of a modular platform for research into the digitization of touch. To achieve the high spatial and temporal performance demonstrated by the artificial fingertip disclosed herein, the embodiments disclosed herein leverage methodological breakthroughs in five different subsystems: elastomer interface, optical system, illumination system, multimodal sensing, and on-device Al processing.
[0057] When the reflective fingertip surface layer is subject to impression stimuli, the surface layer material properties may directly relate to the spatial resolving capabilities. The embodiments disclosed herein developed a design-of-experiment technique to identify these 6 material parameters which affect sensor sensitivity to input stimuli: Rg (fingertip radius), Tc (surface reflective coating layer thickness), Tg (surface layer thickness), h (height), Ec (coating Young’s modulus), and Eg (fingertip volume Young’s modulus). If Tc and Tg are too thick or exhibit low compliance a low pass filter effect may be evident on discrete object edges. Similarly, if the object is rich in spatial information and fractal dimension, the fingertip surface may resolve less features due to local gradients from material compliance. Particular embodiments may avoid specifying any constraints on the coating thickness layer to best capture small input stimuli while maintaining a suitable parameter range for the general size of the fingertip. Developing this layer onto the fingertip surface may involve manual hand painting, airbrushing or dip-coating techniques. However, while producing a touch image, they may be far from optimal and result in large coating thickness, and inconsistent yield from manufacturing variance. The embodiments disclosed herein solve this issue by developing a
new chemical deposition technique for the growing of a silver thin film directly onto the surface of the fingertip which produces coating thicknesses far smaller than previous methods and thus achieve better sensitivity.
[0058] Common visuo-tactile sensors may capture input stimuli at a planar surface, use multiple cameras that are difficult to integrate and process together, or default to common off the shelf cameras optimized for human-centered imaging which result in downgraded optical performance in touch. Particular embodiments may refrain from the use of standard imagesensor features such as automatic exposure control, automatic white balance, and automatic focus which are designed for responding to changes in the natural environment as the fingertip chamber disclosed herein may be an enclosed and controlled environment. For modeling an isotropic representation of similar dimensions to a human fingertip, a new approach may be required to optimize the capture of the hemispherical surface. In optimizing for input stimuli from the touch interaction layer, the imaging system disclosed herein may not limit the performance within the finite element method simulation of the material properties. Hence, for example, particular embodiments may determine the optical system requirements to best suit capturing images related to tactile sensing with a CMOS pixel size of 1 . 1 urn. Parameters may be chosen for converging spot size to increase spatial resolution, intentionally allowing chromatic aberration, introducing shallow depth of field to allow for defocus proportional to object indentation depth, and removing anti-reflective coatings to allow capture and interpretation of reflections and scattering inside the fingertip. However, such parameters may require a non-standard lens. Therefore, particular embodiments may utilize a custom solid immersion hyperfisheye lens to tackle to unique environment of visuo-tactile sensing, rather than an off-the-shelf lens catering towards general purpose imaging thus enabling full control over lens geometry and optical parameters.
[0059] The embodiments disclosed herein describe two metric parameters on the illumination performance within the volume, background uniformity, how evenly is the light distributed, and image to background uniformity contrast, how well do impressions on the surface of the fingertip stand out compared to the background as follows. A common approach may be the embodiment of an internal structure which serves as a hemispherical light pipe and provides fingertip rigidity. However, an internal light pipe structure may produce illumination artifacts in the form of glint and hotspots from the convex geometry7 which contribute to a degradation in image metrics. To reduce these artifacts, conventional approaches may make use of a textured surface to induce Lambertian scattering of incident light rays. Particular embodiments may model the reflective layer surface properties with controlled degrees of
scattering from polished to Lambertian, where the entire hemispherical surfaces may act as an integrating sphere, to show that a Lambertian scattering surface may be not the optimal approach to achieve high-performance. Particular embodiments may use a rigid solid volume, instead of the more common hollow volume or use of an internal support structure, in conjunction with controlled reflective surface scattering parameters moving away from Lambertian surfaces.
[0060] The embodiments disclosed herein disclose a new platform and show that these advancements far outperform conventional visuo-tactile techniques in sensitivity to spatial and force sensitivity. While visual information may provide insight into environmental and object contact, such as textures, and surface deformations, this may only provide a subset of fingertip to object-environment understanding.
[0061] FIG. 3 A illustrates the ability to determine the volume of water in an opaque container by tapping with one finger and recording the response on the fingers in contact with the container. The embodiments disclosed herein further show how this modality may be deconstructed into peak frequency analysis, which may be independent of finger position, whereas the decay time may be dependent on finger placement. FIG. 3B illustrates example spectrogram of the surface audio textures recorded for different objects. FIG. 3C illustrates example sensitivity7 of the artificial fingertip towards heat gradients. Using a variable heat source as a control, the artificial fingertip may be sensitive towards heat gradients. FIG. 3D illustrates example accuracy of object identification through local gas sensing. The artificial fingertip may provide object state and identification of objects through local gas sensing achieving a 91% accuracy. FIG. 3E illustrates example accuracy of object classification through scent. The accuracy of object classification through scent may be dependent on the integration time as the artificial fingertip begins approaching the object. FIG. 2E shows that 61% accuracy is reached within 6 seconds. FIG. 3F illustrates an example localization of finger placement on an object during movement transients with empty and full volumes of liquid. The embodiments disclosed herein measure the effects transients during impulse, resonance during a static hold, and a static hold with no movement.
[0062] Particular embodiments may further evolve the capabilities of the platform to include sensitivity to non-vision based modalities. As an example and not by way of limitation, when in contact with the environment, dynamic forces and signals may be experienced; swiping the fingertip across a surface, or the very moment a contact transient or slip may occur. Particular embodiments may capture this information through in-fingertip audio microphones and pressure MEMS based sensors and show the ability to determine the level of liquid inside
an opaque bottle (see FIG. 2A) and understand the nuances of surface texture between different objects at amuch higher frequency (upwards to, e.g., 10 kHz) than visual capture (e.g., 240 Hz) (see FIG. 2B). Particular embodiments may further include modalities to understand object state, which is not necessarily a function of contact, but can provide priors in understanding touch through heat and smell. Such priors may estimate if an object may be slippery' due to the presence of water, soap, or butter and if an object may present a danger to human-contact due to its temperature.
[0063] For human arc-reflex, quick reaction to input stimuli on the fingertip may benefit from the central nervous system instead of a round trip to the brain. Particular embodiments may utilize a similar local processing response on the artificial fingertip. As an example and not by way of limitation, particular embodiments may include within the form-factor of the fingertip a neural-network accelerator to process the sensory reading, and allow for direct control providing actions to a robotic end effector for controlling the phalanges of a robot finger. While this may be a new era of on-device fingertip processing, the embodiments disclosed herein disclose two main effects that contribute to faster response, i.e., latency and jitter. Latency may result from the average time required to process a signal of interest and jitter may be the variation in mean time-based on system overhead which may occur due to host processing or bandwidth constraints. Compared to conventional methods of using an artificial fingertip with an external host where 2x the round-trip latency is required to perform an action, the embodiments disclosed herein show that by processing data directly on device through an onboard neural network accelerator, a 2x reduction in latency and jitter towards performing an action may be achieved in an example embodiment.
[0064] In particular embodiments, a 3D finite element method (FEM) model using Comsol Multiphysics may be utilized for analyzing and characterizing the fingertip material stack-up. The 3D FEM model may identify the sensitivity and resolution of the sensor. First, a FEM model may be used to identify the key parameters that presented the largest change in sensitivity and resolution. Since the fingertip may be isotropic and may revolve around the origin, only a quarter of the sensor may be modeled for faster computation, using a multi-layer based model. The multi-layer model may comprise the base gel, polymer, and coating layers.
[0065] Particular embodiments may use a particular system to generate nano-mechanical characterization of the fingertip polymer Young’s modulus, E. This system may have in-situ high-resolution imaging, dynamic nanoidentation and a high-precision motion stage with high- resolution force sensing tips. As an example and not by way of limitation, for characterization, a 30 pN force may be applied with a 10 um probe tip. The corresponding force displacement
curve may be measured yielding a Young's modulus of, e.g., E = 2.86 MPa. Using the experimental E value, the FEM models may be updated to correct the simulations. Furthermore, the same force value applied and maximum displacement, Dmax, may be measured, resulting in the verification of simulation, Dmax = 2.1 urn as an example and experimental Dmax = 2.2 um measurement results as an example, with an example error < 5%. Additionally, multiple measurements may be taken across varying samples of the fingertip. For example, an average value for E may be measured at E = 2.6±0.74 MPa. Both Emean and Estd may be used in detailed analysis for the total E range in the FEM models.
[0066] Particular embodiments may further employ design-of-experiments techniques used to identify key parameters in affecting sensor sensitivity. As an example and not by way of limitation, six different parameters may be used: Rgel (gel radius), Tc (coating layer thickness), Tg (gel layer thickness), h (height), Ec (coating Young’s modulus), Eg (gel Young’s modulus). In particular embodiments, materials associated with the silicone hemispherical dome may be determined based on a plurality of material parameters comprising one or more of gel radius, coating layer thickness, gel layer thickness, heigh, coating Young’s modulus, or gel Young’s modulus. Entertaining a full factorial design of 6 parameters may lead to 64 models, thus a quarter-factorial design method may be used to reduce the design into 16 models. Analysis of variance and prediction analysis may result in Ec and Eg as main effects and interactions with coating and gel thicknesses. Hence, the parameters height h and gel radius Rgel may be removed from the model. To analyze the effect of gel and coating thickness, Tg, Tc on the sensor performance, design-of-experiments methods may be used through sweeping Young’s modulus parameters, Ec, Eg, and thickness parameters, Tg and Tc. For example, for the protective fingertip layer’s Young’s modulus, values of Ec,g = 0.5, 1.0, 3.0, 5.0 MPa may be used. As another example, for the values of coating thickness, Tc = 0.1. 0.5, 1.0, 2.0, 3.0 mm and for gel thickness, Tg = 0.5, 1.0, 5.0, 10, 15 mm may be used. FIG. 4 illustrates an example touch sensitivity surface map plot of & = 5.0 and Eg = 3.0 MPa. FIG. 4 shows that for decreasing gel and coating thickness, sensitivity increases. FIG. 5 illustrates an example visualization of the result of FEM analysis with surface map graphs. FIG. 5 shows selecting regions of interest for background and indentation with multi-contact indentations across the surface of the fingertip, where the x-axis shows gel thickness, Tg values and y-axis show coating layer thickness, Tc for each combination of coating and gel Young’s modulus, Ec and Eg. FIG. 5 indicates that the material Young’s modulus may contribute significantly to yield a minimum required Tc and Tg for high performance. Similar FEM analysis was performed for the example conventional tactile sensor. A multilayer-based model may be used. The
embodiments disclosed herein analyzed Tg (gel thickness), Tc coating thickness, Ec (coating Young’s modulus) and Eg (gel Young's modulus). For example , a parametric study was performed with Tg 0.5 to 15 mm with 0.25 steps and Tg 0.1 to 3 mm with coating Young’s modulus Ec 5 MPa and gel’s Young’s modulus Eg 1 MPa. The result of this study shows that gel and coating thickness interact, and the coating and gel thickness combination may affect the overall sensitivity of the sensor.
[0067] Particular embodiments may determine average and maximum forces which can be applied to the fingertip before the soft hemispherical surface is damaged. Particular embodiments may fix the sensor to a force-torque sensor and apply an increasing force until the fingertip detaches from the body, which may occur at, for example, 40 N and 20 N for normal and shear forces respectably. FIG. 6 illustrates example maximum normal and shear force applied to the artificial fingertip surface prior to detachment from the main body. Particular embodiments may apply an incremental normal and shear force to the body of the soft artificial fingertip and record the ground truth force data from a stationary force-torque sensor. When an abrupt change occurs, this may indicate that maximum force has been reached. [0068] Reflecting on FEM simulations. Young’s modulus may be an important parameter in sensor performance to input stimuli, and may require precise controller measurement. In addition to nanoindenter measurement, which is point-based measurement, a set of dynamic mechanical thermal analysis (DMTA) measurements may be performed to obtain the global Young’s modulus of the gel. With this method, particular embodiments may measure the viscoelastic properties of polymers. During DMTA measurements an oscillating force may be applied to the material, and its response may be recorded to calculate the viscosity and stiffness of the material. The oscillating stress and strain measurements may be important in determining the viscoelastic properties of the material.
[0069] When oscillating force is applied, sinusoidal stress and strain values may be measured. The phase difference between sinusoidal stress and strain may provide information about viscous and elastic properties of the material. Ideal elastic systems may have a 0° C phase angle while viscous systems may have a phase angle of 90° C. Additionally, elastic response of a material may' be similar to storage energy and may be captured by storage modulus, while the viscous response may be considered as loss of energy, captured by loss modulus. Thus, the overall modulus of the viscoelastic material may be the combination of elastic and viscous components; in other words, summation of storage modulus and loss modulus. Another value, tan 5, may be used to compare the viscous and elastic moduli. DMTA may measure the change in the elastic modulus, loss modulus and tan 3 with respect to temperature. As the viscosity of
the material is affected by the temperature and time, usually DMTA experiments may be performed at different temperatures and frequencies. Particular embodiments may use a common approach to select the operational conditions of the materials. Hence, for ideal sensitivity the fingertip may be expected to be used at room temperature and low frequency. As such, DMTA measurements may be taken at 25° C with frequency 5 Hz in an example embodiment.
[0070] During fingertip manufacturing, different combinations of polymers with varying shore values may be evaluated. For example, to identify the global Young’s modulus and effect of different gel mixtures, DMTA measurements may be done at 25 degC with frequency 5 Hz, shown in Table 1. Fingertip materials with lower Young’s modulus may be preferred in order to optimize for higher sensitivity. Particular embodiments may select a fabrication using a particular silicone encapsulating rubber using, e.g., a 0.8: 1 (part A to B) ratio for the gel fingertip base material, and another particular cured silicon rubber as thin protection layer for the thin-film layer.
Storage
Polyme Shore Loss Modulus tan 6
Modulus
Sorta Clear 1 12 31.2 2.9 0.09
Sorta Clear 1 18 65.1 8.2 0.12
Solaris 15 38.5 1.8 0.048
Encapso K 33 56.1 6.6 0.11
Ecoflex Gel 32 0.66 0.41 0.61
Ecoflex 0010 10 1.11 0.32 0.29
Ecoflex 0020 20 0.92 0.15 0.16
Ecoflex 0030 30 1.17 0.13 0.11
Ecoflex 0035 35 2.3 0.14 0.06
Ecoflex 0050 50 1.55 0.19 0.12
Ecoflex 0045 45 5.7 0.43 0.07
Solaris A:B N/A 30.7 1.24 0.04
0.9 : 1
Solaris A:B N/A 28.3 1.05 0.037
Table 1: DMTA ir leasurements performed on typical polymers used in fingertip gel manufacturing.
[0071] In parti ci ilar embodiments, the silicone hemispherical dome may be generated
based on manufacturing a mold from aluminum, finishing the mold with a machine polishing pass, preparing the mold for gel casting through a salinization process in a desiccator, preparing a gel material using a cure silicone rubber compound, combining the gel material in a speed mixer under vacuum, casting the gel material into the mold, curing the casted gel material at a first temperature for a first amount of time, and removing a gel hemispherical dome from the mold once the casted gel material is cured. As an example and not by way of limitation, particular embodiments may manufacture the fingertip molds from 6061 aluminum and finish them with a machine polishing pass with a 3 mm diameter tool and a 50 pm step-over. The molds may be then prepared for gel casting through a silanization process in a desiccator with 50 pL silane under vacuum for 30 min. Following this, the gel material may be prepared using a 1: 1 ratio of the aforementioned silicon encapsulating rubber and combined in a speed mixer for 3 minutes under vacuum to release any captured air in the sample. The gel material may be then cast into the mold and allowed to cure at 23° C for 12 h. Once cured, the gel fingertip may be removed from the mold using tweezers for transfer to a glass slide.
[0072] In particular embodiments, the reflective silver-film layer may be generated based on preparing a glucose solution by dissolving a first amount of glucose in a second amount of H2O and adding a third amount of KOH, preparing an AgNOs solution by dissolving a fourth amount of AgNOs in a fifth amount of H2O and adding a sixth amount of NH3, preparing a plating solution by mixing the glucose solution and AgNOs solution, cleaning the gel hemispherical dome using oxygen plasma for a second amount of time, activating the gel hemispherical dome in a solution of a seventh amount of SnCb in an eight amount of H2O for a third amount of time, suspending the gel hemispherical dome into the plating solution for a fourth amount of time, rinsing the gel hemispherical dome with H2O, and air drying the gel hemispherical dome. As an example and not by way of limitation, the steps for preparing the thin film metallic reflective layer on the gel fingertip through silver plating may be as follows. First, a glucose solution may be prepared by dissolving 2.035 g glucose in 160 mL H2O and then adding 0.224 g KOH. This may be set aside and the AgNOs solution may be prepared by dissolving 1.02 g AgNO? in 120 mL H2O, and then adding 1.2 g NH3 25%. The plating solution which is used to silver coat the gel fingertip may be then prepared by mixing 2 parts glucose solution, total 80 mL, to 1 part AgNO solution, total 40 mL. The silvering solution may be then set to gently stir. Prior to silver coating, the gel fingertip may be cleaned using oxygen plasma for 3 minutes. The gel fingertip may be then activated in a solution of 6.181 g SnCb in 98 mL H2O for 10 s. Once the gel fingertip is activated, it may be suspended into the silvering solution for a total of 3 minutes, then rinsed with H2O and air dried. This process may create a
silvered reflective layer with 6 pm thickness. For robotics applications, and for increasing resilience against the intrusion of ambient light, particular embodiments may coat the silvered layer in a white or black layer. This layer may be produced by using the aforementioned cured silicon rubber, a mixing ratio of part A to B of 1 : 1 and then adding 3% silicon color pigments to part A of the aforementioned cured silicon rubber. Part B of the cured silicon rubber may be then mixed in by weight according to the mixing ratio specified previously and then mixed in the speed mixer for 3 minutes under vacuum. The silvered gel fingertip may be then dipped into the pigment of the cured silicon rubber and set to cure for 6 hours.
[0073] Common vision-based tactile sensors may make use of a static illumination configuration whereas some other sensors may use a single light color with colored acrylic to simulate multiple colors. Static illumination may be not ideal to promote a modular system. Rather, the illumination system should adapt to the needs of extracting information from the touch surface. Some conventional tactile sensors used a gel coated with a Lambertian scattering layer, in which volume illumination may produce an image by means of scattering light off the surface and into the vision system. In the case of the monolithic hemispherical gel dome used in the artificial fingertip disclosed herein, the embodiments disclosed herein determine that Lambertian scattering may be not ideal for producing and optimizing for force and spatial sensitivity. Additionally, the embodiments disclosed herein introduce a dynamic illumination system that provides volume illumination with configurable wavelength, intensity, and positioning. As an example and not by way of limitation, the illumination system may comprise 8 fully controllable RGB LEDs that emit Lambertian diffuse light, equally spaced around a circle of radius 9 mm.
[0074] FIG. 7 illustrates an example cross-section of the fingertip gel coating showing the 3 layers: outer, silver and base gel. FIGS. 8A-8H illustrate an example hyperfisheye lens designed specifically for capturing omnidirectional tactile sensing images. FIG. 8A illustrates example fingertip regions of the hyperfisheye lens. Three regions of interest may be selected for optical system performance, tip, prominent contact surface, and base. The fingertip gel in the system disclosed herein may comprise three components, as seen in FIGS. 7-8. The outer surface of the base gel may have a reflective silver thin-film coating, which may be coated with a protective colored-diffusive material. To produce an image, the two layers may provide scattering of internally incident light from surface interactions to the vision system. Particular embodiments may place the illuminating light-emitting diodes in optical contact to the gel, using an over-molding process. The fingertip gels may be manufactured with a smooth and polished surface initially, and through mold texturing one may control and determine how- light
is scatered at the interface. FIG. 8B illustrates an example lens of the hyperfisheye lens. FIG. 8C illustrates an example lens cutaway of the hyperfisheye lens.
[0075] FIG. 8D illustrates an example system layout of solid immersion hyperfisheye lens system. For example, the design configuration comprises a 25-mm diameter silicone hyperhemispherical dome in tandem with a 5P lens system. This silicone-lens system combination images object points located at or near the dome surface to the sensor plane. For example, the imaging system has EFL = 1.57 mm, FOV of 194.5 degrees, and focal ratio of F/3.68 at the used conjugates. The image quality evaluation performance of these designs at 1 = 587.6 nm, in terms of MTFs, spot sizes, relative illumination and wavefront analysis are shown in the following slides. FIG. 8D also shows the lens system sizes/dimensions. FIG. 8E illustrates an example lens layout and lens element plastic material. The lens system may utilize EP10000 plastic material for the LI lens element while lens elements L2, L3, L4, and L5 are designed using APEL 5014 plastic material.
[0076] FIG. 8F illustrates an example lens lay out and minimum clear aperture radii. In particular embodiments, the first lens element may be configured to have a maximum clear aperture diameter of 10 millimeters. As shown in FIG. 8F, lens element LI may have a maximum clear aperture diameter of 10 mm and in contact (no air space) with the hyper hemispherical silicone dome element. FIG. 8F also shows additional example clear aperture data (based on radius in millimeter) as: CIR S4 4.748376, CIR S5 1.707403, CIR S6 1.341748, CIR S7 0.538351, CIR S8 0.151529. CIR S9 0.574821, CIR S10 0.794950, CIR SI 1 0.862950, CIR S12 0.888102, CIR S 13 0.897415, CIR S14 0.866286, CIR S15 0.842312, and CIR S16 0.808351. FIG. 8G illustrate example solid models of silicone plus lens. FIG. 8H illustrates example solid models of lens.
[0077] Using a Gaussian scatter distribution, particular embodiments may model a range of scater. The scatering parameter, o, may be chosen to achieve the half-width-half-max angles, a, of the bi-directional scater distribution (BSDF) function at normal incidence from, e.g., a = 1° to 25°, along with a Lambertian scattering model. FIG. 9 illustrates example inference latency measurements comparing on-device to host incorporating the tactile data preprocessing and transfer stages. FIG. 9 shows that using the on-device CNN accelerator improves inference time and reduces the total latency to producing an action for a MLP deep neural network. A surface texture that minimizes scattering may produce little background illumination. Neither does it produce large shadows created by surface indentations. Additionally, minimizing scatter may produce specular glint reflections, fail to illuminate all indentations equally, and saturate the vision system.
[0078] With a fully Lambertian scattering model, the hemispherical surface of the gel fingertip may act as an integrating sphere. While Lambertian scattering provides uniform background illumination, the high scattering illumination from nearby interactions may reduce the overall indentation contrast. The embodiments disclosed herein optimized for high image contrast while maintaining uniform background illumination, the better to image impressions produced by gel indentations and minimize the amount of glints that would saturate the image sensor.
[0079] Two non-uniformity metrics were evaluated over the fingertip hemispherical surface, Std/Mean and (Max - Min)/Mean. The embodiments disclosed herein demonstrated that low scatter yields images with a large variation in image signal, requiring the camera to handle high dynamic range. If the image is allowed to saturate, to resolve the variations due to the indentations, the saturated areas of the image may be lost. Thus, in these cases the stray light may be more likely to cause objectionable artifacts. The contrast in the image caused by spherical indentations may be in certain areas high, due to bright glint reflections; but areas with large gradients in the background may make the indentations hard to detect. High scatter may give images with low variation in image signal and no areas may be lost due to saturation. In the image caused by spherical indentations, the contrast may be low in certain areas, but the uniform background may make these easier to detect. FIG. 10 illustrates example evaluations of non-uniformity metrics. Minimizing the background non-uniformity with increasing scatter angle may improve the detection of surface indentations.
[0080] The embodiments disclosed herein define a contrast-to-noise (CNR) metric and study three regions of interest on the hemispherical surface for background uniformity noise and indentation contrast. Plotting the calculated CNR across the hemispherical surface for the different scatter angles, it is observed that the CNR is generally higher for less scatter, but CNR is more uniform across the FOV for more scatter. Therefore, the embodiments disclosed herein determine that for a hemispherical fingertip surface the desired texture scattering profile may be constrained between, e.g., half-width-half-max angles of 20° to 25°.
[0081] The embodiments disclosed herein comprise a platform based on modular principles to provide an omnidirectional vision-based, multimodal sensing system, and on- device Al capabilities. Particular embodiments may achieve modularity through isolating each part of the system into an electronic assembly with small surface area. Specifically, these modules may include sensing, optics, vision capture, processing, and communications subsystems shown in FIG. 2. By facilitating re-use of each sub-system, the embodiments disclosed herein present a novel solution to advance the technology in tactile sensor design — reducing
the required design, iteration, and manufacturing processes. This may enable one skilled of the art to modify the overall system by adding, removing, and changing sub-systems, rather than having to design a stack of new hardware to support changes to system design. Furthermore, this modular system architecture may allow for selecting combinations of sub-systems to facilitate introduction of newer technologies. Feature removal may reduce costs in adapting the system for new mechanical form factors. As well, the possibility' of replacing the sensing fingertip may’ allow adaptability for different environments and tasks, e.g., by using different sensitivities and stiffness in the fingertip material, or by using fingertips with markers to introduce more prominent optical-flow features. Based on simulation to achieve optimal spatial and force resolution, the embodiments disclosed herein introduce a novel process for the design and manufacturing of the sensing fingertip.
[0082] In particular embodiments, the system disclosed herein may provide communications with the host device over the USB 3.0 standard interface. Three separate streams may be provided for data transfer, supporting video, audio, and multimodal data. For example, these streams may collectively output at a maximum rate of 148 MBps and below, depending on the configuration sent to the device. In particular embodiments, tactile sensors may be used in open-loop control, providing information to the host device for processing and additional actions to manipulators. In particular embodiments, edge Al may be added at the fingertip for the following reasons. First, edge Al may help create a latent representation of the data and reduce the overall bandwidth sent to the host device. Second, edge Al may help enable fast local decisions for transmitting actions to manipulators. Third, edge Al may help improve the overall latency of the system while reducing variance in jitter, which is the change in latency. Particular embodiments may model the tactile fingertip system with a manipulator, where both systems may be connected to the host device in a star configuration. This may result in decisions — and actions resulting from tactile information — being processed through the host device and disseminated to the manipulators. For more data-intensive designs, capturing data from multiple fingers at once, this arrangement may result in unstable control schemes where information and action latency cannot be guaranteed. To accommodate and expand the terrain of tactile sensing research, particular embodiments may integrate a particular neural network accelerator, for example, a 9 core RISC-V compute cluster with Al acceleration, for on-device processing of selected data streams.
[0083] Following high level abstractions of the human reflex arc, the embodiments disclosed herein develop a fast reflex-like control loop using edge Al for local processing. Conventional paradigm of transferring the sensory input to a central control computer for
processing, then sending back the control signals, may require high bandwidth while introducing communication latency. In contrast, the paradigm disclosed herein may locally process the sensory input inside the fingertip using edge Al. This may allow drastic reductions of the required bandwidth while greatly minimizing communication latency and minimizing jitter. The embodiments disclosed herein performed an experimental comparison of these two paradigms, shown in Table 2, by measuring the end-to-end latency of the systems using a PCI- e based precision time measurement tool. The embodiments disclosed herein evaluated this experiment on a Linux machine with 64GB memory and a GPU. First, to ensure granularity in the measurements, each section of the system was isolated, and samples were collected in repeated trials. Second, the embodiments disclosed herein verified these results by subjecting the entire system to repeated measurements and comparing the timing results to the sum of the isolated components. This determined the areas that produce deterministic timing results, as well as highlighting the areas that sustained increased latency and jitter. Furthermore, these results indicated areas of performance improvements and design for future tactile sensors. The results for the entire control loop show how the edge Al paradigm results in a reduction of latency from 4 ms to 1 ms with a desirable smaller variance. In particular embodiments, appropriate edge- Al processing may be extended to further exploit the sequential nature of the camera FIFO memory7 to parallelize the data capture with the processing, thus yielding even lower latency. In this case, instead of processing the entire image, selected horizontal lines may be sent for processing in the configured region of interest. This may be applicable when touch interactions are most likely to appear in certain regions on the omnidirectional fingertip disclosed herein. The system disclosed herein may support this region-of-interest data output selection for increased resolution and image-capture frequency.
Surface Region 1 [mN] Region 2 [mN] Region 3 [mN] Mean [mN] Specular L01 L09 L41 L 17
Lambertian 1.30 1.77 2.24 1.77
Table 2: Normal force prediction error (median) by surface ty pes and regions.
[0084] FIG. 11 illustrates an example data capture pipeline 1100 for the vision system of the disclosed artificial fingertip and touch information from external stimulus. External stimulus 1102 may7 be input to the exposure module 1104 of the image sensor 1106. The output of the exposure module 1104 may be input to the image buffer 1108 of the image sensor 1106. The output of the image buffer 1108 may be provided to the sub-sampling module 1110 of artificial fingertip (on-device) 1112. Although this disclosure describes an image sensor for processing external stimulus as an example, this disclosure contemplates any suitable modality
or any suitable subset of modalities in replacement of the image sensor.
[0085] The output of the sub-sampling module 1110 may be input to the SPI transfer module 11 14. The output of the SPI transfer module 11 14 may be input to an on-device inference module 1116 of the host inference module 1118. The output of the on-device inference module 1116 may be input to the finger action transfer module 1120 based on I2C to finger transfer. The output of the finger action transfer module 1120 may be used to generate a finger action 1122 for an allegro hand 1124a to execute.
[0086] In particular embodiments, the output of the image buffer 1108 may be also provided to a USB transfer module 1126 of artificial fingertip (host) 1128. The output of the USB transfer module 1126 may be input to the sub-sampling module 1130 of the host inference module 1132. The output of the sub-sampling module 1130 may be input to the inference module 1134. The output of the inference module 1134 may be input to the finger action transfer module 1136 based on USB-CAN. The output of the finger action transfer module 1136 may go through a palm processing module 1138 of the allegro hand 1124b. The output of the palm processing module 1138 may be input to an I2C palm to finger transfer module 1140, the output of which may be used to generate a finger action 1142 for the allegro hand 1124b to execute.
[0087] FIG. 12 illustrates example data collection and prediction. The solid line shows ground-truth shear and normal force trajectory during one indentation. The dotted scatter plot shows model-predicted shear and normal force. The embodiments disclosed herein establish two pipelines for data transfer and processing: on-device pipelines and host pipelines, shown in FIG. 12. Additionally, a hybrid mode may support transferring data to the host as well as processing it on the device. It is observed that this affects the vision system because it may involve substantially larger amounts of data than does the multimodal data system. The limiting parameter for dynamic manipulations in a task is the latency between input data, processing, and action commands. To elicit rapid responses to changes in the environment that are detected through changes in grasp stability, particular embodiments may move toward faster processing and a minimization of latency between input and action.
[0088] The embodiments disclosed herein studied the effects of the vision system because this may be the most common modality used in touch sensors, with impacts on overall system latency for host and on-device configurations. The constraining factor on system latency between real-world input and data-processing input is the capture rate of the vision system. This may be limited by the frames per second rate, which may impose a delay of 1 /fps, and the internal processing of the image-signal processor. For this reason, for example, particular
embodiments may incorporate a CMOS sensor with 240 fps and a pixel size of 1.1 urn in the system disclosed herein, which may yield a shorter delay of 4. 17 ms as opposed to conventional sensors, which may operate at 60 Hz and thus have delays no fewer than 16.7 ms.
[0089] The embodiments disclosed herein evaluated the inference latency with two deep neural networks, MLP and MobileNetV2, for two scenarios: on-device and host inference. The two largest sources of latency arise from the tactile data transfer from device to host, and of action data from host to robotic end effector. For example, within the available headroom between the differences in pipeline latency, Tlatency, an upper limit of Tlatency < 2.463 ms was established. With an MLP-based network, the embodiments disclosed herein increased the layer depth and observed the latency cost for both scenarios. Table 3 shows a ~ 4x decrease in action latency for dynamic tasks which involve high velocity movements thereby reducing total time to action to less than 1 ms.
Table 3: Average data pipeline timing for host and on-device processing.
[0090] FIG. 13 illustrates example simulated performance of the vision system of the artificial fingertip disclosed herein from on-axis to far-field contact for increasing line pairs per millimeters translated to spatial resolution for sagittal and tangential responses. As shown in FIG. 13, it becomes apparent that using the on-device accelerator without enabling the hardware engine quickly exceeds Tlatency at 10-layer depth. However, enabling hardware acceleration may allow for using MLP models with 60 layers.
[0091] Observing a more suitable use case for the tactile research domain, the embodiments disclosed herein deployed a MobileNetV2 model and determine the total system pipeline latency. FIG. 14 illustrates example average pipeline latency for a MobileNetV2 network from acquiring tactile data, transferring data, pre-processing, inference to providing actions for varying image and channel width sizes. FIG. 15 illustrates an example comparison of touch information from the artificial fingertip disclosed herein tapping a water bottle with varying
volumes of water from empty, half filled, and full. While the vision output from the sensor looks nearly identical with only variations in location of touch, multimodal data may provide a larger insight on object properties beyond texture. FIG. 15 shows that using a MobileNetV2 architecture with input size of 64 x 64 is beneficial for reducing Tlatency and applicable to common low-level touch tasks, such as touch detection and classification. Furthermore, the Tlatency upper boundary’ may be determined by the output data rate, size of data transfer, and the host system performance.
[0092] However, in practical robotics environments the host system may be running a plethora of control and processing applications, with additional overhead of communication between other sensors and devices that introduce overhead to Tlatency of, e.g., 1.2 ms. Comparing this to a conventional artificial fingertip, the embodiments disclosed herein observed an overhead of e.g., 4.7 ms. These differences may be attributed to the frame rate of the artificial fingertip disclosed herein, e.g., 240 fps, and data transfer over USB 3.0, whereas the example conventional tactile sensor may be limited to 60 fps using USB 2.0. The system disclosed herein may enable tactile on-device inference xxi th low latency control with the ability to have reflex-like control of the device to which the system is connected, as well as providing to the host abstractions of lower-level touch signals. An example may be training an on-device model to regress force from multimodal data to introduce touch and manipulation force limits to objects. Another example may be using the on-device Al capabilities to recognize slip and with low latency provide actions to the robotic end effector to reconfigure grasp.
[0093] The embodiments disclosed herein design a controllable robot indenter capable of applying with high precision measured 3-axis force on any spatial position of the sensor. FIG. 16 illustrates an example 6-DoF robot indenter for testing tactile sensor force resolution. The robot arm and stage setup may be capable of precisely applying measured force onto a target device with controlled contact spatial location and orientation. FIGS. 17A-17B illustrate example image snapshots taken by the artificial fingertip disclosed herein from shear force data collection in two key moments. The timestamp t corresponds to the trajectory’ in FIG. 12. The overlay arrow field shows the optical flow with respect to the image without any force applied. As shown in FIGS. 17A-17B, for example, a tactile sensor is mounted on the robot arm to orient the desired test surface down against a probe with precision of 5 pm. As another example, the probe with a hemisphere tip of 4 mm diameter is mounted on a force sensor measuring the ground-truth contact force with 1 mN accuracy. As yet another example, the probe and force sensor assembly are then mounted on a hexapod which can be precisely
controlled to translate with 0.1 pm and rotate with 0.05° increment. Due to the rotational symmetry of the artificial fingertip, the embodiments disclosed herein break down the fullsurface force characterization into three representative approximately planar regions. For each region, the embodiments disclosed herein repeat the collection process similarly for normal and shear forces.
[0094] The embodiments disclosed herein start with normal force collection. For example, for high precision, particular embodiments may use the single-axis force sensor that can measure up to 250 mm. As another example, for each region, the robotic indenter may spatially sample 0.5 mm-spaced grid points on the tangential plane. As yet another example, for each point, the probe may move perpendicular from the plane, pressing into the sensor until the normal force reaches 200 mm. During the contact between the probe and gel (defined as normal force, Fnorm > 0.2 mN), both sensor images and measured normal force may be collected synchronously. The embodiments disclosed herein collect about 550 image-force pairs per spatial point. For a 7 mm x 6 mm region, the embodiments disclosed herein obtain approximately 12, 000 points. This point data is randomly split into training (70%) and testing set (30%).
[0095] For shear-force data collection, particular embodiments may select a 3-axis force sensor to simultaneously measure normal force and shear force. Particular embodiments may apply sufficient friction while varying shear force. FIG. 18 illustrates an example normal force prediction error distribution by surface types and regions. The dots and error bar show the median and 95-percentile of the error respectively. FIG. 18 shows how each shear-force indentation trajectory may be controlled. For example, first, the probe may be moved perpendicular to the contact surface to apply up to 600 mm normal force. Next, the probe may be moved tangential to the surface, loading shear force up to 100 mm. Finally, the probe may be moved back to the previous location, unloading shear force. If non-zero residual shear force remained after unloading, slip might have occurred — in which case the data may be discarded. [0096] Contact-force prediction on vision-based tactile sensors such as the system disclosed herein may be achieved using an image-to-force regression model. The model may be calibrated from reference data. Once calibrated, particular embodiments may evaluate the sensor-model as a system for force-sensing performance on a testing dataset. The embodiments disclosed herein collected the dataset for training and evaluating the model to benchmark normal and shear-force-sensing performance. Particular embodiments may use a modified ResNet50 deep neural network for the image-to-force regression model. For example, the network may take an input image of 224 x 224 x 3 and output 1024-way object-classification
probabilities. Particular embodiments may replace the classification head with a scalar-output linear layer predicting the force. Particular embodiment may use mean-square error as the loss, then optimize with Adam with initial learning rate search. In one example embodiment, the raw images from the sensors are 640 x 480, which may be dow n scaled to 224 x 224 with 20- pixel spatial jitter to improve spatial invariance. Particular embodiments may pool training data from all three regions to train a single model and obtain the prediction performance (median error) breakdown by regions as described in FIG. 2A.
[0097] The embodiments disclosed herein evaluated additional normal-force resolution performance with two kinds of gel surface finish: specular and Lambertian. Lambertian surface scattering, typically considered preferable for vision-based tactile sensors, was outperformed by its specular counterpart. This may come from the enhancement of surface texture contrast due to specular reflection, which helps the imaging system track gel deformation. FIG. 19 illustrates an example capture amongst vision-based touch sensors. Objects shown from top to bottom: sandpaper, cloth, oyster and a configure cone. FIG. 19 shows the center-crop of the image captured by the system disclosed herein, where the texture contrast is more evident in the region with strong specular reflections from the LEDs. The embodiments disclosed herein obtained clear optical flow (shown in the figure by arrows) within these texture-rich regions, corresponding to fingertip deformation caused by shear and normal force applied by the probe. [0098] Conventionally, a general view held that some tracking pattern (dots, for example) may be required for shear force measurement. However, the optical flow result reported above suggests that this requirement may be relaxed, due to the increasing resolution and quality of images. Such advancements may facilitate using the natural fingertip surface texture to observe gel deformation, and in turn to perform shear force estimation.
[0099] Particular embodiments may establish a modality within the artificial fingertip to determine object state and obtain clues on object classification. Particular embodiments may identify tw o key performance metrics for fingertip gas sensing: accuracy and signal acquisition time. The embodiments disclosed herein observe 6 different materials from liquid to solid commonly found in a household environment. These materials are coffee powder, liquid coffee, a nondescript rubber material, cheese, and a spread of soap and butter on a surface. All the materials were sampled at room temperature with a robotic arm and the disclosed artificial fingertip approaching the samples to near contact, within 1 cm, for a duration of 90 s. The embodiments disclosed herein record multimodal data and isolate the humidity, temperature, pressure, gas oxidation resistance datapoints at the maximum output frequency for each sampling modality. Over 100 approaches to each material are collected during a 3-hour
sampling period. Between each approach, the embodiments disclosed herein sample air from the local environment. The raw data with the modalities of interest listed above are provided as inputs to a multi-layer perceptron network with a single 64-node hidden layer. The embodiments disclosed herein train the network with cross-entropy loss using an Adam optimizer with learning rate 0.1. The embodiments disclosed herein show the final accuracy of the model is not sensitive with respect to the size of hidden layers or learning rate. For example, the embodiments disclosed herein show that through these 6 materials a classification accuracy of 91%. Furthermore, as another example, the embodiments disclosed herein show the signal acquisition time to reach 66% accuracy.
[0100] The embodiments disclosed herein evaluate the disclosed artificial fingertip in performance with respect to spatial resolution, shear and normal forces, illumination, vibrations, heat, and local gas sensitivity.
[0101] FIG. 20A illustrates an example analog of the human reflex-arc by quickly processing sensory input within the fingertip, directly controlling the actuators of a robot hand to retract in response to touching an object. FIG. 20B illustrates example tactile processing and control paradigm transfer of sensory data to a remote computer for processing. This requires sufficient bandwidth and introduces communication latency. FIG. 20C illustrates example local processing for mimicking the reflex-arc with the system fingertip. The system disclosed herein may use the on-device Al neural-network accelerator for local processing to decrease overall latency between event and action. FIG. 20D illustrates example mean and standard deviation of the event-to-action latency. For example, the on-device local processing and control loop may take 1.2 ms compared to the conventional paradigm, which may take 2.5 ms on the disclosed artificial fingertip and over 6 ms on the example conventional tactile sensor.
[0102] The embodiments disclosed herein model the fingertip surface as a two-layer stack formed by an external diffusive material adhered to an internal reflective thin-film which is grown onto the non-rigid solid silicone body of the fingertip. In other words, the silicone hemispherical dome may further comprise a protective diffusive layer coated to the reflective silver-film layer. The embodiments disclosed herein then explore the effects of the non-rigid solid silicone surface mechanical properties, texture and the degree of controlled light scattering to find an optimal performance metric between background uniformity and image contrast. The embodiments disclosed herein show that increasing controlled surface texture scatter from 1 -degree scattering to a Lambertian scatter results in an increase in background illumination uniformity thereby an increase in image impression contrast. However, with low degrees of scattering, intense hotspots artifacts may dominate the background whereas when
the degree of scattering approaches Lambertian scattering, these artifacts may decrease along with a decrease in image contrast which may directly result in decreased sensitivity to impression stimuli. With little or no scatter with a polished surface, minimal background illumination may be present which motivates the production of shadows created by indentations against the fingertip surface. Furthermore, glint reflections off produced indentations may be minimal to non-existent and may not produce a consistent appearance across the surface. On the contrary, with the conventional method of visual-tactile sensors using Lambertian scattering surfaces, the embodiments disclosed herein show that the hemispherical sensing surface may act as an integrating sphere, where shadows may cast by direct illumination striking the indentations are wiped out by scattered illumination from other areas and even while imaging may occur on far off-axis angles, their contrast may be low. The embodiments disclosed herein introduce a controlled degree of scattering in which an optimized uniform background illumination may be achieved that lends itself well to contrast between indentations and the surrounding surface, furthermore, all indentations are imaged (see FIG. 2).
[0103] The embodiments disclosed herein evaluate the normal force sensitivity and first collect tuples of normal forces applied by a micro-indenter and corresponding outputs from the sensors, and then train a deep-learning model from this dataset. For example, the trained model (see FIG. 2A) may predict the normal forces applied with a median error of 1.01 mN (Region 1). Similarly, to measure the shear force sensitivity, the embodiments disclosed herein first collect tuples of shear forces applied by a micro-indenter and corresponding outputs from the sensors, and then train a deep-leaming model from this dataset. For example, the model (see FIG. 2A) is capable of predicting shear forces applied with a median error of 1.27 mN (Region 1). In contrast to conventional sensors that required the presence of explicit markers, this result demonstrates that with a sufficiently high optical resolution the embodiments disclosed herein may directly use the internal texture of the elastomer to measure shear forces. [0104] To cany' out spatial resolution evaluations, the embodiments disclosed herein define the spatial resolution of an artificial fingertip sensor as the minimum feature size that can be resolved with an MTF > 0.5; this may be determined by how well the contrast is preserved quantified by line pairs per millimeter. The embodiments disclosed herein first simulate the imaging system from the design which yields that on-axis contacts are resolvable for features of size >6 um for region 1, >8 um for region 2 resolves, and >22 um for region 3. The embodiments disclosed herein then validate these results by collecting data with a two-pronged micro-indenter depressed onto the fingertip, varying the distance between the two prongs and
observing the taxel intensity line profile; both the visual validation and the inspection of the taxels profile intensity confirmed the embodiments disclosed herein may clearly distinguish features as small as >7 urn for region 1 (see FIG. 2C).
[0105] Multimodal information such as vibrations upwards of 10 kHz, auditory clues, sensitivity to heat and smells may play an important role in human touch. However, typical vision-based tactile sensors may not contain a broad range of multimodal capabilities to capture this information or operate lower sensing frequencies such as 60 Hz. Even with the fast camera of the disclosed artificial fingertip which operates at, e.g., 240 Hz, highly dynamic movements may not be fully captured. The embodiments disclosed herein evaluate capturing vibrations up to 10 kHz which may be enough to distinguish between different materials upon a simple light sliding of the finger. Furthermore, the embodiments disclosed herein show that these multimodal features can be used to detect the amount of liquid inside a bottle by simply tapping it with a fingertip (see FIG. 2A), like audio and vibratory clues consumed by humans during object interactions. Along audio and vibratory clues, humans evaluate touch interactions based on changes in local heat gradients. The embodiments disclosed herein may detect changes in heat gradients which reflect object state, room temperature, warm, hot, dangerous (see FIG. 2C). In regard to object state, there may be a limited amount of information which is captured during visuo-tactile contact. The embodiments disclosed herein employ the use of local gas sensing at each fingertip to understand nuances in object state, e.g., determining if the object is slippery or wet. With this modality, the embodiments disclosed herein may sense these parameters during approach and contact (see FIG. 2E). Specifically, the embodiments disclosed herein evaluate contact different samples which not only provide gas signatures, but local environmental information such as humidity and temperature gradients to distinguish between two similarly looking liquids and coffee or coffee grind (see FIG. 2D). The use of multimodal sensing may complement the main vision-based sensing modality and enable future work into the importance of the different touch modes for task-specific applications.
[0106] As inspired by the human reflex arc, the embodiments disclosed herein demonstrate a fast reflex-like control loop using the on-device Al neural-network accelerator for local processing. Compared to conventional sensors using an external computer for processing, on- device processing on the artificial fingertip disclosed herein may reduce latency, for example, from 6 ms to 1.2 ms (see FIG. 3D). With the increasing computational power of on-device accelerators, larger surfaces used for sensing touch, and increasing use of Al models for touch processing, the capability of processing data locally and transfer only high-level features may prove crucial for touch processing.
[0107] The embodiments disclosed herein may advance the state of artificial fingertip sensing towards digitizing fingertip interactions between the environment and objects. The embodiments disclosed herein disclose an artificial fingertip that may be more sensitive in spatial and force sensitivity compared to conventional methods with the additional technical advantage of multimodal sensing features and a local processing ability . Experimental results demonstrate the digitization of touch with capabilities that outperform a human fingertip. The richness of touch digitized by the disclosed modular platform may open new promising venues into studying the nature of touch in humans and investigating key questions around the digitization and processing of touch as a sensor modality. Moreover, the embodiments disclosed herein may open the doors to a wider adoption of touch sensors beyond conventional niche fields: In robotics, to improve sensing and manipulation capabilities with benefits for applications in manufacturing and logistics, medical robotics, agricultural robotics, and consumer-level robotics; In artificial intelligence, to investigate the learning of appropriate tactile and multimodal representations, and corresponding computational models that can better exploit the active, spatial and temporal nature of touch. Further potential applications may include virtual reality and telepresence, prosthesis, and e-commerce.
Miscellaneous
[0108] Herein, “or” is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A or B” means “A, B, or both,” unless expressly indicated otherwise or indicated otherwise by context. Moreover, “and” is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A and B” means “A and B, jointly or severally,” unless expressly indicated otherwise or indicated otherwise by context.
[0109] The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend. Furthermore, reference in the appended claims to an apparatus or system or a component of an apparatus or system being adapted to. arranged to, capable of, configured to, enabled to, operable to, or operative to perform a particular
function encompasses that apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, as long as that apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative. Additionally, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide none, some, or all of these advantages.
Claims
1. A system for touch digitization, comprising: a silicone hemispherical dome comprising a surface comprising a reflective silver-film layer; an omnidirectional optical system comprising: a lens comprising a plurality of lens elements, wherein a first lens element of the plurality of lens elements is in direct contact with the silicone hemispherical dome without airgap, and wherein the lens is configured to capture scattering of internal incident light generated by the reflective silver-film layer; and an image sensor configured to generate image data from data captured by the lens; one or more non-image sensors disposed underneath the omnidirectional optical system; one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to: access the image data from the omnidirectional optical system and sensing data from the one or more non-image sensors; and generate, based on the accessed image and sensing data by one or more machine-learning models, the touch digitization.
2. The system of Claim 1, wherein the lens is a solid immersion lens.
3. The system of Claim 1, wherein the lens is a hyperfisheye lens.
4. The system of Claim 1, wherein the one or more non-image sensors comprise one or more of an inertial measurement unit (IMU) sensor, a microphone, an environmental sensor, a gas sensor, a pressure sensor, or a temperature sensor.
5. The system of Claim 1, wherein the first lens element is configured to have a maximum clear aperture diameter of 10 millimeters.
6. The system of Claim 1, wherein materials associated with the silicone hemispherical dome are determined based on a plurality of material parameters comprising one or more of gel radius, coating layer thickness, gel layer thickness, heigh, coating Young’s modulus, or gel Young’s modulus.
7. The system of Claim 1, wherein the silicone hemispherical dome is based on a Poly dimethylsiloxane (PDMS) material.
8. The system of Claim 1, wherein the silicone hemispherical dome further comprises a protective diffusive layer coated to the reflective silver-film layer.
9. The system of Claim 1, further comprising: a housing based on a shape of a human thumb, wherein the silicone hemispherical dome, the omnidirectional optical system, the one or more non-image sensors, the one or more processors, and the non-transitory memory are disposed in the housing.
10. The system of Claim 1, further comprising: a stack-up comprising a plurality printed circuit boards for the one or more processors, the omnidirectional optical system, and a data transfer system, wherein the plurality printed circuit boards share a common electrical interface and connector stack.
11. The system of Claim 1, wherein the one or more processors comprise one or more of a microprocessor or an accelerator.
12. The system of Claim 1, wherein the one or more machine-learning models comprise one or more neural-network models, wherein the one or more processors comprise one or more neural-network accelerators, and wherein the one or more neural-network accelerators are configured for accelerating real-time inference on the accessed image and sensing data by the one or more neural-network models.
13. The system of Claim 1, wherein the processors are further operable when executing the instructions to: provide one or more control signals to a secondary device associated with the system.
14. The system of Claim 13, wherein the secondary device comprises a robotic end effector.
15. The system of Claim 1, wherein the silicone hemispherical dome is generated based on: manufacturing a mold from aluminum; finishing the mold w ith a machine polishing pass; preparing the mold for gel casting through a salinization process in a desiccator; preparing a gel material using a cure silicone rubber compound: combining the gel material in a speed mixer under vacuum; casting the gel material into the mold; curing the casted gel material at a first temperature for a first amount of time; and removing a gel hemispherical dome from the mold once the casted gel material is cured.
16. The system of Claim 15, wherein the reflective silver-film layer is generated based on:
preparing a glucose solution by dissolving a first amount of glucose in a second amount of H2O and adding a third amount of KOH; preparing an AgNOs solution by dissolving a fourth amount of AgNOs in a fifth amount of H2O and adding a sixth amount of NH preparing a plating solution by mixing the glucose solution and AgNOs solution; cleaning the gel hemispherical dome using oxygen plasma for a second amount of time; activating the gel hemispherical dome in a solution of a seventh amount of SnCh in an eight amount of H2O for a third amount of time; suspending the gel hemispherical dome into the plating solution for a fourth amount of time; rinsing the gel hemispherical dome with H2O; and air drying the gel hemispherical dome.
17. The system of Claim 1, further comprising: an illumination system, wherein the illumination system comprises a plurality of controllable light-emitting diodes emitting Lambertian diffuse light, and wherein the illumination system is configured to generate volume illumination with one or more configuration illumination parameters.
18. The system of Claim 17, wherein the one or more configurable illumination parameters comprise one or more of wavelength, intensity, or positioning.
19. An artificial fingertip for touch digitization, comprising: a silicone hemispherical dome; an omnidirectional optical system comprising: a lens comprising a plurality of lens elements, wherein a first lens element of the plurality of lens elements is in direct contact with the silicone hemispherical dome without airgap; and an image sensor configured to generate image data from data captured by the lens; and one or more non-image sensors disposed underneath the omnidirectional optical system.
20. A system for touch digitization, comprising: a silicone hemispherical dome comprising a surface comprising a reflective silver-film layer; an omnidirectional optical system comprising: a lens configured to capture scattering of internal incident light generated by the
reflective silver-film layer; and an image sensor configured to generate image data from data captured by the lens; and one or more non-image sensors disposed underneath the omnidirectional optical system.
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202263383069P | 2022-11-09 | 2022-11-09 | |
US63/383,069 | 2022-11-09 | ||
US18/503,727 US20240149466A1 (en) | 2022-11-09 | 2023-11-07 | Digitizing Touch with Artificial Robotic Fingertip |
US18/503,727 | 2023-11-07 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2024102453A1 true WO2024102453A1 (en) | 2024-05-16 |
Family
ID=89222587
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2023/037113 WO2024102453A1 (en) | 2022-11-09 | 2023-11-09 | Digitizing touch with artificial robotic fingertip |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2024102453A1 (en) |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2859961A2 (en) * | 2006-02-08 | 2015-04-15 | Avent, Inc. | Methods and compositions for metal nanoparticle treated surfaces |
US20170195568A1 (en) * | 2016-01-06 | 2017-07-06 | 360fly, Inc. | Modular Panoramic Camera Systems |
WO2021081084A1 (en) * | 2019-10-21 | 2021-04-29 | The Regents Of The University Of California | Multi-directional high-resolution optical tactile sensors |
-
2023
- 2023-11-09 WO PCT/US2023/037113 patent/WO2024102453A1/en unknown
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2859961A2 (en) * | 2006-02-08 | 2015-04-15 | Avent, Inc. | Methods and compositions for metal nanoparticle treated surfaces |
US20170195568A1 (en) * | 2016-01-06 | 2017-07-06 | 360fly, Inc. | Modular Panoramic Camera Systems |
WO2021081084A1 (en) * | 2019-10-21 | 2021-04-29 | The Regents Of The University Of California | Multi-directional high-resolution optical tactile sensors |
Non-Patent Citations (6)
Title |
---|
CORRADI TADEO ET AL: "Tactile Features: Recognising Touch Sensations with a Novel and Inexpensive Tactile Sensor", 1 September 2014, SAT 2015 18TH INTERNATIONAL CONFERENCE, AUSTIN, TX, USA, SEPTEMBER 24-27, 2015; [LECTURE NOTES IN COMPUTER SCIENCE; LECT.NOTES COMPUTER], SPRINGER, BERLIN, HEIDELBERG, PAGE(S) 163 - 172, ISBN: 978-3-540-74549-5, XP047298394 * |
DO WON KYUNG ET AL: "DenseTact: Optical Tactile Sensor for Dense Shape Reconstruction", 2022 INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION (ICRA), IEEE, 23 May 2022 (2022-05-23), pages 6188 - 6194, XP034146574, DOI: 10.1109/ICRA46639.2022.9811966 * |
LORENZO NATALE ET AL: "Tactile Sensing", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 7 May 2021 (2021-05-07), XP081962040, DOI: 10.1007/978-94-007-7194-9_110-1 * |
PADMANABHA AKHIL ET AL: "OmniTact: A Multi-Directional High-Resolution Touch Sensor", 2020 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION (ICRA), IEEE, 31 May 2020 (2020-05-31), pages 618 - 624, XP033826051, DOI: 10.1109/ICRA40945.2020.9196712 * |
SCHARFF ROB B N ET AL: "Rapid manufacturing of color-based hemispherical soft tactile fingertips", 2022 IEEE 5TH INTERNATIONAL CONFERENCE ON SOFT ROBOTICS (ROBOSOFT), IEEE, 4 April 2022 (2022-04-04), pages 896 - 902, XP034117948, DOI: 10.1109/ROBOSOFT54090.2022.9762136 * |
ZHOU YANGYANG ET AL: "Solid Immersion Maxwell's Fish-Eye Lens Without Drain", PHYSICAL REVIEW APPLIED, vol. 17, no. 3, 1 March 2022 (2022-03-01), US, XP093134182, ISSN: 2331-7019, DOI: 10.1103/PhysRevApplied.17.034039 * |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Lambeta et al. | Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation | |
Ward-Cherrier et al. | The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies | |
Abad et al. | Visuotactile sensors with emphasis on gelsight sensor: A review | |
Shimonomura | Tactile image sensors employing camera: A review | |
Alspach et al. | Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation | |
Yuan et al. | Gelsight: High-resolution robot tactile sensors for estimating geometry and force | |
Trueeb et al. | Towards vision-based robotic skins: a data-driven, multi-camera tactile sensor | |
Patel et al. | Integrated proximity, contact and force sensing using elastomer-embedded commodity proximity sensors | |
Winstone et al. | TACTIP—Tactile fingertip device, challenges in reduction of size to ready for robot hand integration | |
Kampmann et al. | Integration of fiber-optic sensor arrays into a multi-modal tactile sensor processing system for robotic end-effectors | |
Shah et al. | On the design and development of vision-based tactile sensors | |
Cramphorn et al. | Addition of a biomimetic fingerprint on an artificial fingertip enhances tactile spatial acuity | |
Li et al. | F-touch sensor: Concurrent geometry perception and multi-axis force measurement | |
Gomes et al. | Blocks world of touch: Exploiting the advantages of all-around finger sensing in robot grasping | |
Wilson et al. | Design of a fully actuated robotic hand with multiple gelsight tactile sensors | |
JP2013061223A (en) | Six-axis force measuring device, and six-axis force measuring method | |
Andrussow et al. | Minsight: A Fingertip‐Sized Vision‐Based Tactile Sensor for Robotic Manipulation | |
JP2009285737A (en) | Input interface | |
US20230074958A1 (en) | Multi-directional high-resolution optical tactile sensors | |
Abad et al. | HaptiTemp: A next-generation thermosensitive GelSight-like visuotactile sensor | |
Zhang et al. | Vision-based sensing for electrically-driven soft actuators | |
US20240149466A1 (en) | Digitizing Touch with Artificial Robotic Fingertip | |
Sun et al. | Predicting fingertip forces by imaging coloration changes in the fingernail and surrounding skin | |
WO2024102453A1 (en) | Digitizing touch with artificial robotic fingertip | |
Althoefer et al. | A miniaturised camera-based multi-modal tactile sensor |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23825009 Country of ref document: EP Kind code of ref document: A1 |