CN114912598A

CN114912598A - Optical artificial neural network intelligent chip and preparation method thereof

Info

Publication number: CN114912598A
Application number: CN202110172825.1A
Authority: CN
Inventors: 崔开宇; 熊健; 杨家伟; 黄翊东; 张巍; 冯雪; 刘仿
Original assignee: Tsinghua University
Current assignee: Tsinghua University
Priority date: 2021-02-08
Filing date: 2021-02-08
Publication date: 2022-08-16
Also published as: WO2022166189A1

Abstract

The invention provides an optical artificial neural network intelligent chip and a preparation method thereof, wherein an optical filter layer is used as an input layer and a linear layer of an artificial neural network, the filtering effect of the optical filter layer on incident light is used as the connection weight from the input layer to the linear layer, the square detection response of an image sensor is used as a first nonlinear activation function in the nonlinear layer of the artificial neural network, a processor is used as a second nonlinear activation function and an output layer in the full-connection and nonlinear layers of the artificial neural network, so that the filter layer and the image sensor realize the relevant functions of an input layer, a linear layer and a nonlinear activation function in the artificial neural network in a hardware mode, therefore, the subsequent intelligent processing does not need to perform complex signal and algorithm processing corresponding to the input layer and the linear layer, and the power consumption and the time delay during the artificial neural network processing can be greatly reduced.

Description

Optical artificial neural network intelligent chip and preparation method thereof

Technical Field

The invention relates to the technical field of artificial intelligence, in particular to an optical artificial neural network intelligent chip and a preparation method thereof.

Background

The existing intelligent recognition technology generally needs to image a person or an object first, and then input the image into a neural network recognition model for processing, so as to realize the recognition of the person or the object.

Therefore, the current intelligent recognition task generally depends on a neural network recognition model, namely the current intelligent recognition task needs to be imaged firstly and then transmitted to a computer for subsequent neural network recognition model algorithm processing, and large power consumption and time delay are caused by transmission and processing of a large amount of data.

Disclosure of Invention

Aiming at the problems in the prior art, the embodiment of the invention provides an optical artificial neural network intelligent chip and a preparation method thereof.

Specifically, the embodiment of the invention provides the following technical scheme:

in a first aspect, an embodiment of the present invention provides an optical artificial neural network intelligent chip, including: an optical filter layer, an image sensor, and a processor; the optical filter layer corresponds to an input layer and a linear layer of the artificial neural network and the connection weight of the input layer to the linear layer, and the square detection response of the image sensor corresponds to a first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to a full connection and output layer of the artificial neural network, or corresponds to a second nonlinear activation function and an output layer in the full connection and nonlinear layer of the artificial neural network;

The optical filter layer is arranged on the surface of the image sensor and comprises an optical modulation structure, and the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to different position points on the surface of the image sensor;

the image sensor carries out first nonlinear activation processing on incident light carrying information corresponding to different position points after being modulated by the optical filter layer through square detection response, then converts the incident light carrying information into electric signals corresponding to the different position points, and sends the electric signals corresponding to the different position points to the processor;

and the processor performs full connection processing on the electric signals corresponding to the different position points, or performs full connection processing and secondary nonlinear activation processing on the electric signals corresponding to the different position points to obtain an output signal of the artificial neural network.

Further, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

Further, the optical artificial neural network intelligent chip is used for an intelligent processing task of a target object; the intelligent processing task at least comprises one or more of intelligent perception, intelligent identification and intelligent decision task;

Reflected light, transmitted light and/or radiated light of the target object enter a trained optical artificial neural network intelligent chip to obtain an intelligent processing result of the target object; the intelligent processing result at least comprises one or more of an intelligent sensing result, an intelligent recognition result and/or an intelligent decision result;

the trained optical artificial neural network intelligent chip comprises a trained optical modulation structure, an image sensor and a processor;

the trained optical modulation structure, the image sensor and the processor are the optical modulation structure, the image sensor and the processor which meet the training convergence condition and are obtained by training an optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using an input training sample and an output training sample corresponding to the intelligent processing task; or the trained optical modulation structure, image sensor and processor are the optical modulation structure, image sensor and processor which meet the training convergence condition and are obtained by training an optical artificial neural network intelligent chip which comprises different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters by using input training samples and output training samples corresponding to the intelligent processing task.

Further, when training an optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full connection parameters, or training an optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full connection parameters and different second nonlinear activation parameters, the different optical modulation structures are designed and realized in a computer optical simulation design mode.

Further, the light modulating structures in the optical filter layer comprise regular structures and/or irregular structures; and/or the light modulating structures in the optical filter layer comprise discrete structures and/or continuous structures.

Further, the optical filter layer is a single-layer structure or a multi-layer structure.

Further, the light modulation structure in the optical filter layer comprises a unit array consisting of a plurality of micro-nano units, and each micro-nano unit corresponds to one or more pixel points on the image sensor; the structures of the micro-nano units are the same or different.

Further, the micro-nano unit comprises a regular structure and/or an irregular structure; and/or the micro-nano unit comprises a discrete structure and/or a continuous structure.

Further, the micro-nano unit comprises a plurality of groups of micro-nano structure arrays, and the structures of the micro-nano structure arrays are the same or different.

Furthermore, each group of micro-nano structure array has the function of broadband filtering or narrow-band filtering.

Furthermore, each group of micro-nano structure array is a periodic structure array or a non-periodic structure array.

Furthermore, the micro-nano unit comprises one or more groups of hollow structures in a plurality of groups of micro-nano structure arrays.

Further, the micro-nano unit has polarization-independent characteristics.

Further, the micro-nano unit has quadruple rotational symmetry.

Further, the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials and perovskite materials; and/or the filter layer is prepared from one or more of a photonic crystal, a super surface, a random structure, a nano structure, a metal Surface Plasmon Polariton (SPP) micro-nano structure and an adjustable Fabry-Perot resonant cavity.

Further, the semiconductor material comprises one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed according to a preset proportion and a direct band gap compound semiconductor material; and/or the nanostructure comprises one or more of a nanodot two-dimensional material, a nanocolumn two-dimensional material and a nanowire two-dimensional material.

Further, the optical filter layer has a thickness of 0.1 λ to 10 λ, where λ represents a center wavelength of incident light.

Further, the image sensor is any one or more of:

the CMOS image sensor CIS, the charge coupled device CCD, the single photon avalanche diode SPAD array and the focal plane photoelectric detector array.

In a second aspect, an embodiment of the present invention further provides an intelligent device, including: the optical artificial neural network intelligent chip as described in the first aspect.

Further, the intelligent device comprises one or more of a smart phone, a smart computer, an intelligent recognition device, an intelligent perception device and an intelligent decision device.

In a third aspect, an embodiment of the present invention further provides a method for preparing an optical artificial neural network intelligent chip according to the first aspect, including:

preparing an optical filter layer containing an optical modulation structure on the surface of the image sensor;

generating a processor with a function of performing full connection processing on the signal or generating a processor with a function of performing full connection processing and secondary nonlinear activation processing on the signal;

connecting the image sensor and the processor;

the optical filter layer is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure through the optical modulation structure so as to obtain incident light carrying information corresponding to the different position points on the surface of the image sensor; the incident light carrying information comprises light intensity distribution information, spectrum information, angle information of the incident light and phase information of the incident light;

Further, preparing an optical filter layer containing a light modulation structure on the surface of the image sensor includes:

growing one or more layers of preset materials on the surface of the image sensor;

etching the light modulation structure pattern of the one or more layers of preset materials to obtain an optical filter layer containing a light modulation structure;

or the one or more layers of preset materials are subjected to imprinting transfer to obtain an optical filter layer containing an optical modulation structure;

or the one or more layers of preset materials are subjected to additional dynamic modulation to obtain an optical filter layer containing an optical modulation structure;

Or printing the one or more layers of preset materials in a partition mode to obtain an optical filter layer containing an optical modulation structure;

or carrying out partition growth on the one or more layers of preset materials to obtain an optical filter layer containing an optical modulation structure;

or quantum dot transfer is carried out on the one or more layers of preset materials to obtain the optical filter layer containing the optical modulation structure.

Further, when the optical artificial neural network intelligent chip is used for an intelligent processing task of a target object, training the optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full connection parameters by using input training samples and output training samples corresponding to the intelligent processing task to obtain the optical modulation structures, the image sensors and the processors meeting a training convergence condition; or training an optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions.

The embodiment of the invention also provides an optical artificial neural network environment-friendly monitoring chip, which is used for an environment-friendly monitoring intelligent processing task and comprises the following steps: an optical filter layer, an image sensor, and a processor; the optical filter layer corresponds to an input layer and a linear layer of the artificial neural network and the connection weight of the input layer to the linear layer, and the square detection response of the image sensor corresponds to a first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to a full connection and output layer of the artificial neural network, or corresponds to a second nonlinear activation function and an output layer in the full connection and nonlinear layer of the artificial neural network;

the optical filter layer is arranged on the surface of the image sensor and comprises an optical modulation structure, and the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to different position points on the surface of the image sensor; the incident light comprises reflected, transmitted and/or radiated light of the environmental contaminant;

The processor performs full connection processing on the electric signals corresponding to different position points, or performs full connection processing and secondary nonlinear activation processing on the electric signals corresponding to different position points to obtain an environment-friendly monitoring intelligent processing result;

wherein the environmental monitoring intelligent processing task comprises identification and/or qualitative analysis of environmental pollutants; the environment-friendly monitoring intelligent processing result comprises an identification result of the environmental pollutants and/or a qualitative analysis result of the environmental pollution.

Further, the optical artificial neural network environment-friendly monitoring chip comprises a trained optical modulation structure, an image sensor and a processor;

the trained optical modulation structure, the image sensor and the processor are the optical modulation structure, the image sensor and the processor which meet the training convergence condition and are obtained by training an optical artificial neural network environment-friendly monitoring chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using an input training sample and an output training sample corresponding to the environment-friendly monitoring intelligent processing task; or the trained optical modulation structure, image sensor and processor are the optical modulation structure, image sensor and processor meeting the training convergence condition, which are obtained by training an optical artificial neural network environment-friendly monitoring chip of the processor, which comprises different optical modulation structures, image sensors and different full-connection parameters and different second nonlinear activation parameters, by using an input training sample and an output training sample corresponding to the environment-friendly monitoring intelligent processing task;

Wherein the input training sample comprises incident light reflected, transmitted and/or radiated by different environmental pollutants; the output training sample comprises a corresponding environmental pollutant recognition result; and/or, the input training sample comprises incident light reflected, transmitted and/or radiated by different environmental pollutants; and the output training sample comprises a corresponding environmental pollution qualitative analysis result.

Further, when the optical artificial neural network environment-friendly monitoring chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters is trained, or the optical artificial neural network environment-friendly monitoring chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters and different second-time nonlinear activation parameters is trained, the different optical modulation structures are designed and realized in a computer optical simulation design mode.

Further, the micro-nano unit has polarization-independent characteristics.

Further, the micro-nano unit has quadruple rotational symmetry.

Further, the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials and perovskite materials; and/or the filter layer is prepared from one or more of photonic crystals, super surfaces, random structures, nano structures, metal Surface Plasmon Polariton (SPP) micro-nano structures and adjustable Fabry-Perot resonant cavities.

Further, an embodiment of the present invention further provides an environmental protection monitoring device, including the above-mentioned optical artificial neural network environmental protection monitoring chip.

The embodiment of the invention also provides a preparation method of the above-mentioned optical artificial neural network environment-friendly monitoring chip, which comprises the following steps:

connecting the image sensor and the processor;

and the processor performs full connection processing on the electric signals corresponding to different position points, or performs full connection processing and secondary nonlinear activation processing on the electric signals corresponding to different position points to obtain an environment-friendly monitoring intelligent processing result.

Further, the preparation method of the optical artificial neural network environment-friendly monitoring chip further comprises the following steps: the training process of the optical artificial neural network environment-friendly monitoring chip specifically comprises the following steps:

training an optical artificial neural network environment-friendly monitoring chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using input training samples and output training samples corresponding to the environment-friendly monitoring intelligent processing task to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the image sensors and the processors;

Or training an optical artificial neural network environment-friendly monitoring chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters by using input training samples and output training samples corresponding to the environment-friendly monitoring intelligent processing task to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the image sensors and the processors.

The embodiment of the invention also provides an optical artificial neural network fingerprint identification chip, which is used for fingerprint identification processing tasks and comprises the following steps: an optical filter layer, an image sensor, and a processor; the optical filter layer corresponds to an input layer and a linear layer of the artificial neural network and the connection weight of the input layer to the linear layer, and the square detection response of the image sensor corresponds to a first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to a full connection and output layer of the artificial neural network, or corresponds to a second nonlinear activation function and an output layer in the full connection and nonlinear layer of the artificial neural network;

The optical filter layer is arranged on the surface of the image sensor and comprises an optical modulation structure, and the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to different position points on the surface of the image sensor; the incident light comprises reflected light, transmitted light and/or radiated light of a user fingerprint;

and the processor performs full connection processing on the electric signals corresponding to the different position points, or performs full connection processing and secondary nonlinear activation processing on the electric signals corresponding to the different position points to obtain a fingerprint identification processing result.

Further, the optical artificial neural network fingerprint identification chip comprises a trained optical modulation structure, an image sensor and a processor;

the trained optical modulation structure, the image sensor and the processor are the optical modulation structure, the image sensor and the processor which meet the training convergence condition and are obtained by training an optical artificial neural network fingerprint identification chip comprising different optical modulation structures, image sensors and processors with different full connection parameters by using an input training sample and an output training sample corresponding to the fingerprint identification processing task; or the trained optical modulation structure, image sensor and processor are the optical modulation structure, image sensor and processor which meet the training convergence condition and are obtained by training an optical artificial neural network fingerprint identification chip of the processor which comprises different optical modulation structures, image sensors and different full-connection parameters and different second nonlinear activation parameters by using an input training sample and an output training sample corresponding to the fingerprint identification processing task;

wherein the input training samples comprise incident light reflected, transmitted and/or radiated by different human fingerprints; the output training samples comprise corresponding fingerprint recognition results.

Further, when training an optical artificial neural network fingerprint recognition chip comprising different optical modulation structures, image sensors and processors with different full connection parameters, or training an optical artificial neural network fingerprint recognition chip comprising different optical modulation structures, image sensors and processors with different full connection parameters and different second nonlinear activation parameters, the different optical modulation structures are designed and realized in a computer optical simulation design mode.

Further, the micro-nano unit has polarization-independent characteristics.

Further, the micro-nano unit has quadruple rotational symmetry.

The embodiment of the invention also provides fingerprint identification equipment which comprises the optical artificial neural network fingerprint identification chip.

The embodiment of the invention also provides a preparation method of the optical artificial neural network fingerprint identification chip, which comprises the following steps:

connecting the image sensor and the processor;

the optical filter layer is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure through the optical modulation structure so as to obtain incident light carrying information corresponding to the different position points on the surface of the image sensor;

Further, the method for preparing the optical artificial neural network fingerprint identification chip further comprises the following steps: the training process of the optical artificial neural network fingerprint identification chip specifically comprises the following steps:

training an optical artificial neural network fingerprint identification chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using input training samples and output training samples corresponding to the fingerprint identification processing tasks to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the image sensors and the processors;

Or training an optical artificial neural network fingerprint identification chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters by using input training samples and output training samples corresponding to the fingerprint identification processing task to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the trained image sensors and the trained processors.

The embodiment of the invention also provides an optical artificial neural network face recognition chip, which is used for a face recognition processing task and comprises the following steps: an optical filter layer, an image sensor, and a processor; the optical filter layer corresponds to an input layer and a linear layer of the artificial neural network and the connection weight of the input layer to the linear layer, and the square detection response of the image sensor corresponds to a first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to a full connection and output layer of the artificial neural network, or corresponds to a second nonlinear activation function and an output layer in the full connection and nonlinear layer of the artificial neural network;

The optical filter layer is arranged on the surface of the image sensor and comprises an optical modulation structure, and the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to different position points on the surface of the image sensor; the incident light comprises reflected light, transmitted light and/or radiated light of the face of the user;

and the processor performs full connection processing on the electric signals corresponding to the different position points, or performs full connection processing and secondary nonlinear activation processing on the electric signals corresponding to the different position points to obtain a face recognition processing result.

Further, the optical artificial neural network face recognition chip comprises a trained optical modulation structure, an image sensor and a processor;

the trained optical modulation structure, the image sensor and the processor are the optical modulation structure, the image sensor and the processor which meet the training convergence condition and are obtained by training an optical artificial neural network face recognition chip comprising different optical modulation structures, image sensors and processors with different full connection parameters by using an input training sample and an output training sample corresponding to the face recognition processing task; or the trained optical modulation structure, image sensor and processor are the optical modulation structure, image sensor and processor which meet the training convergence condition and are obtained by training an optical artificial neural network face recognition chip which comprises different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters by using input training samples and output training samples corresponding to the face recognition processing task;

wherein the input training samples comprise incident light reflected, transmitted and/or radiated by different human faces; the output training samples include corresponding face recognition results.

Further, when the optical artificial neural network face recognition chip comprising different optical modulation structures, image sensors and processors with different full connection parameters is trained, or the optical artificial neural network face recognition chip comprising different optical modulation structures, image sensors and processors with different full connection parameters and different second nonlinear activation parameters is trained, the different optical modulation structures are designed and realized in a computer optical simulation design mode.

Further, the micro-nano unit has polarization-independent characteristics, and particularly, the micro-nano unit has quadruple rotational symmetry.

Further, the optical filter layer is composed of one or more layers of structures;

the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials and perovskite materials; and/or the filter layer is prepared from one or more of a photonic crystal, a super surface, a random structure, a nano structure, a metal Surface Plasmon Polariton (SPP) micro-nano structure and an adjustable Fabry-Perot resonant cavity.

The embodiment of the invention also provides face recognition equipment which comprises the optical artificial neural network face recognition chip.

The embodiment of the invention also provides a preparation method of the optical artificial neural network face recognition chip, which comprises the following steps:

connecting the image sensor and the processor;

Further, the method for preparing the optical artificial neural network face recognition chip further comprises the following steps: the training process of the optical artificial neural network face recognition chip specifically comprises the following steps:

training an optical artificial neural network face recognition chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using input training samples and output training samples corresponding to the face recognition processing tasks to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the image sensors and the processors;

Or training an optical artificial neural network face recognition chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters by using input training samples and output training samples corresponding to the face recognition processing task to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the trained image sensors and the trained processors.

The embodiment of the invention also provides an optical artificial neural network machine vision enhancement chip, which is used for machine vision intelligent processing tasks and comprises the following steps: an optical filter layer, an image sensor, and a processor; the optical filter layer corresponds to an input layer and a linear layer of the artificial neural network and the connection weight of the input layer to the linear layer, and the square detection response of the image sensor corresponds to a first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to a full connection and output layer of the artificial neural network, or corresponds to a second nonlinear activation function and an output layer in the full connection and nonlinear layer of the artificial neural network;

The optical filter layer is arranged on the surface of the image sensor and comprises an optical modulation structure, and the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to different position points on the surface of the image sensor; the incident light comprises reflected light, transmitted light and/or radiated light of a target object in a machine vision scene;

the processor performs full connection processing on the electric signals corresponding to different position points, or performs full connection processing and secondary nonlinear activation processing on the electric signals corresponding to different position points to obtain a machine vision intelligent processing result;

the machine vision intelligent processing task comprises identification and/or qualitative analysis of a target object in a machine vision scene; the machine vision intelligent processing result comprises an identification result and/or a qualitative analysis result of a target object in a machine vision scene.

Further, the optical artificial neural network machine vision enhancement chip comprises a trained optical modulation structure, an image sensor and a processor;

the trained optical modulation structure, image sensor and processor are the optical modulation structure, image sensor and processor which meet the training convergence condition and are obtained by training an optical artificial neural network machine vision enhancement chip comprising different optical modulation structures, image sensors and processors with different full connection parameters by using an input training sample and an output training sample corresponding to the machine vision intelligent processing task; or the trained optical modulation structure, image sensor and processor are the optical modulation structure, image sensor and processor which meet the training convergence condition and are obtained by training an optical artificial neural network machine vision enhancement chip of the processor which comprises different optical modulation structures, image sensors and different full-connection parameters and different second nonlinear activation parameters by using an input training sample and an output training sample corresponding to the machine vision intelligent processing task;

Wherein the input training samples comprise incident light reflected, transmitted and/or radiated by target objects in a particular machine vision scene, and the output training samples comprise target object recognition results in the particular machine vision scene; and/or the input training samples comprise incident light reflected, transmitted and/or radiated by target objects in a specific machine vision scene, and the output training samples comprise the results of qualitative analysis of the target objects in the specific machine vision scene.

Further, when training an optical artificial neural network machine vision enhancement chip comprising different optical modulation structures, image sensors and processors with different full connection parameters, or training an optical artificial neural network machine vision enhancement chip comprising different optical modulation structures, image sensors and processors with different full connection parameters and different second nonlinear activation parameters, the different optical modulation structures are designed and realized by adopting a computer optical simulation design mode.

Further, the micro-nano unit has polarization-independent characteristics.

Furthermore, the micro-nano unit has quadruple rotational symmetry.

Further, the filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials and perovskite materials; and/or the filter layer is prepared from one or more of a photonic crystal, a super surface, a random structure, a nano structure, a metal Surface Plasmon Polariton (SPP) micro-nano structure and an adjustable Fabry-Perot resonant cavity.

The embodiment of the invention also provides an enhanced machine vision system which comprises a control mechanism and the optical artificial neural network machine vision enhancement chip.

The embodiment of the invention also provides a preparation method of the optical artificial neural network machine vision enhancement chip, which comprises the following steps:

connecting the image sensor and the processor;

and the processor performs full connection processing on the electric signals corresponding to the different position points, or performs full connection processing and secondary nonlinear activation processing on the electric signals corresponding to the different position points to obtain a machine vision intelligent processing result.

Further, the method for preparing the optical artificial neural network machine vision enhancement chip further comprises the following steps: the training process of the optical artificial neural network machine vision enhancement chip specifically comprises the following steps:

training an optical artificial neural network machine vision enhancement chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using input training samples and output training samples corresponding to the machine vision intelligent processing tasks to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the trained image sensors and the trained processors;

Or training an optical artificial neural network machine vision enhancement chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters by using input training samples and output training samples corresponding to the machine vision intelligent processing task to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the trained image sensors and the trained processors.

The embodiment of the invention provides an optical artificial neural network intelligent chip and a preparation method thereof, which realize a brand new intelligent chip capable of realizing the function of an artificial neural network, wherein in the intelligent chip, an optical filter layer corresponds to an input layer and a linear layer of the artificial neural network, and an image sensor corresponds to a part of a nonlinear layer of the artificial neural network; the processor corresponds to another portion of the non-linear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, the optical filter layer comprises an optical modulation structure, the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to the different position points on the surface of the image sensor, and in the embodiment of the invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. Meanwhile, in the embodiment of the invention, the image sensor carries out the first nonlinear activation processing on the incident light carrying information corresponding to different position points after being modulated by the optical filter layer through square detection response and then converts the incident light carrying information into electric signals corresponding to different position points, and the electric signals corresponding to the different position points are sent to the processor, the processor carries out full connection processing on the electric signals corresponding to the different position points, or the processor carries out full connection processing and secondary nonlinear activation processing on the electric signals corresponding to different position points to obtain the output signal of the artificial neural network, so that, in the intelligent chip, the optical filter layer corresponds to an input layer, a linear layer and a connection weight of the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to a first-time nonlinear activation function in a nonlinear layer of the artificial neural network; the processor corresponds to the full connection and output layer of the artificial neural network, or the processor corresponds to the full connection and output layer of the artificial neural network, the second nonlinear activation function and output layer in the nonlinear layer, namely, the optical filter layer and the image sensor in the intelligent chip realize the related functions of the input layer, the linear layer and part of the nonlinear activation function in the artificial neural network, namely, the embodiment of the invention peels off the input layer, the linear layer and part or all of the nonlinear activation function in the artificial neural network realized by software in the prior art, and realizes the structures of the input layer, the linear layer and part or all of the nonlinear activation function in the artificial neural network by using a hardware mode, so that the intelligent processing with the input layer is not required to be carried out subsequently when the intelligent chip is used for carrying out the intelligent processing of the artificial neural network, The linear layer and the complex signal processing and algorithm processing corresponding to a part or all of the nonlinear activation functions are only required to be carried out by the processor in the intelligent chip, the signal processing or full connection with the electric signal and the second nonlinear activation processing, so that the power consumption and the time delay during the artificial neural network processing can be greatly reduced.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a schematic structural diagram of an optical artificial neural network intelligent chip according to a first embodiment of the present invention;

FIG. 2 is a schematic diagram illustrating the recognition principle of an optical artificial neural network intelligent chip according to an embodiment of the present invention;

fig. 3 is a schematic diagram illustrating a disassembled optical artificial neural network intelligent chip according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a target object identification process according to an embodiment of the present invention;

FIG. 5 is a top view of an optical filter layer according to an embodiment of the present invention;

FIG. 6 is a top view of another optical filter layer provided in accordance with an embodiment of the present invention;

FIG. 7 is a top view of yet another optical filter layer provided in accordance with an embodiment of the present invention;

FIG. 8 is a top view of yet another optical filter layer according to an embodiment of the present invention;

FIG. 9 is a top view of yet another optical filter layer provided in accordance with an embodiment of the present invention;

FIG. 10 is a top view of yet another optical filter layer provided in accordance with an embodiment of the present invention;

fig. 11 is a schematic diagram of a micro-nano structure broadband filtering effect according to an embodiment of the present invention;

fig. 12 is a schematic diagram of a narrow-band filtering effect of a micro-nano structure according to an embodiment of the present invention;

FIG. 13 is a schematic diagram of a front-illuminated image sensor according to an embodiment of the present invention;

FIG. 14 is a schematic diagram of a back-illuminated image sensor according to an embodiment of the present invention;

FIG. 15 is a flowchart illustrating a method for manufacturing an intelligent chip of an optical artificial neural network according to a third embodiment of the present invention;

FIG. 16 is a schematic view of a contaminant sample identification process provided in accordance with an embodiment of the present invention;

FIG. 17 is a schematic diagram of a fingerprint identification process provided by an embodiment of the present invention;

FIG. 18 is a schematic diagram of a face recognition process according to an embodiment of the present invention;

fig. 19 is a schematic diagram of a machine vision-enhanced recognition process according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The existing intelligent recognition technology generally needs to image a person or an object, and then input the image into a neural network recognition model for processing, so as to realize recognition of the person or the object. Therefore, the current intelligent recognition task generally depends on a neural network recognition model, namely the current intelligent recognition task needs to be imaged first and then transmitted to a computer for subsequent neural network recognition model algorithm processing, and transmission and processing of a large amount of data cause large power consumption and time delay. Based on this, the embodiment of the present invention provides an optical artificial neural network intelligent chip, where an optical filter layer in the intelligent chip corresponds to an input layer and a linear layer of an artificial neural network and a connection weight from the input layer to the linear layer, and a square detection response of an image sensor corresponds to a first-order nonlinear activation function in a nonlinear layer of the artificial neural network; the processor corresponds to a full connection and output layer of the artificial neural network, or corresponds to a second nonlinear activation function and an output layer in the full connection and nonlinear layer of the artificial neural network, the optical filter layer and the image sensor are used for projecting the spatial spectrum information of the target object into an electric signal, and then the full connection processing or the full connection processing and the second nonlinear activation processing of the electric signal are realized in the processor, so that the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and part or all of the nonlinear activation functions in the prior art can be omitted. The embodiment of the invention strips an input layer, a linear layer and a part or all of nonlinear activation functions in an artificial neural network realized by software in the prior art, and realizes the structures of the input layer, the linear layer and the part or all of nonlinear activation functions in the artificial neural network by using a hardware mode, so that the subsequent intelligent processing of the artificial neural network by using the intelligent chip does not need to perform complex signal processing and algorithm processing corresponding to the input layer, the linear layer and the part or all of nonlinear activation functions, and only needs to perform full connection processing or full connection with electric signals and secondary nonlinear activation processing by a processor in the intelligent chip, thereby greatly reducing the power consumption and the time delay during the processing of the artificial neural network. The present invention will be explained and illustrated in detail by specific examples.

As shown in fig. 1, an optical artificial neural network intelligent chip according to a first embodiment of the present invention includes: an optical filter layer 1, an image sensor 2 and a processor 3; the optical filter layer 1 corresponds to an input layer and a linear layer of the artificial neural network and the connection weight of the input layer to the linear layer, and the square detection response of the image sensor 2 corresponds to a first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor 3 corresponds to a full connection and output layer of the artificial neural network, or the processor corresponds to a second nonlinear activation function and output layer in the full connection and nonlinear layer of the artificial neural network;

the optical filter layer 1 is arranged on the surface of the image sensor or the surface of a photosensitive area of the image sensor, the optical filter layer 1 comprises an optical modulation structure, and the optical filter layer 1 is used for respectively performing spectrum modulation with intensity modulation along with wavelength variation on incident light entering different positions of the optical modulation structure through the optical modulation structure, namely performing different intensity modulation on the incident light with different wavelengths so as to obtain incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light carrying information includes image information and/or various optical spatial information of a target object to be processed by the optical artificial neural network intelligent chip, for example, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light;

In this embodiment, the square detection response of the image sensor 2 means that the image sensor detects intensity information of an incident light field, and the intensity information of the incident light field is a square of a light field signal modulus, that is, the image sensor 2 performs first nonlinear activation processing on incident light carrying information corresponding to different position points after being modulated by the optical filter layer 1 through the square detection response, converts the incident light carrying information into electric signals corresponding to the different position points, and sends the electric signals corresponding to the different position points to the processor 3; the electric signal is an image signal modulated by the optical filter layer;

the processor 3 is used for carrying out full connection processing on the electric signals corresponding to the different position points, or carrying out full connection processing and secondary nonlinear activation processing on the electric signals corresponding to the different position points by the processor to obtain output signals of the artificial neural network.

In this embodiment, the optical filter layer 1 is disposed on the surface of the image sensor, the optical filter layer 1 includes an optical modulation structure, and the optical filter layer 1 is configured to perform different spectrum modulations on incident light entering different position points of the optical modulation structure through the optical modulation structure, so as to obtain information carried by the modulated incident light corresponding to the different position points on the surface of the image sensor. It follows that in an embodiment, the modulation effect of the light modulating structures on the optical filter layer on the incident light can be seen as the connection weights of the input layer to the linear layer;

In this embodiment, when the image sensor 2 performs photoelectric conversion on the modulated incident light carrying information, since the image sensor 2 can detect the intensity information of light, the electrical signal obtained by processing the light field distribution signal is proportional to the square of the mode of the light field distribution signal, and therefore, the image sensor 2 has a square detection response, so that the image sensor 2 can be regarded as a part of the nonlinear layer of the artificial neural network, that is, the square detection response of the image sensor 2 can be regarded as the first nonlinear activation function of the artificial neural network.

In this embodiment, the image sensor 2 performs a first nonlinear activation process on the incident light carrying information corresponding to different position points modulated by the optical filter layer 1 through a square detection response, and then converts the incident light carrying information into electrical signals corresponding to the different position points, that is, image signals modulated by the optical filter layer, and meanwhile, the processor 3 connected to the image sensor 2 is configured to perform a full connection process or a full connection and a second nonlinear activation process on the electrical signals corresponding to the different position points, so as to obtain an output signal of the artificial neural network.

In this embodiment, the optical filter layer 1 includes an optical modulation structure, and performs spectrum modulation with different intensities on incident light (for example, reflected light, transmitted light, radiated light, and other related action light of an object to be identified) entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain incident light carrying information corresponding to the different positions on the surface of the image sensor 2.

In this embodiment, it is understood that the modulation intensity is related to the specific structural form of the light modulation structure, for example, different modulation intensities can be realized by designing different light modulation structures (e.g., changing the shape and/or size parameters of the light modulation structure).

In this embodiment, it can be understood that the light modulation structures at different positions on the optical filter layer 1 have different spectrum modulation effects on the incident light, and the modulation intensities of the light modulation structures on different wavelength components of the incident light correspond to the connection intensities of the linear layers of the artificial neural network, that is, correspond to the input layers and the connection weights of the input layers to the linear layers. It should be noted that the optical filter layer 1 is composed of a plurality of optical filter units, and the optical modulation structures at different positions in each optical filter unit are different, so that the optical filter layer has different spectrum modulation effects on incident light; the light modulating structures at different locations between the optical filter cells may be the same or different and thus have the same or different spectral modulation effect on the incident light.

In this embodiment, the image sensor 2 performs a first nonlinear activation process on the incident light carrying information corresponding to different position points modulated by the optical filter layer 1 through a square detection response, and then converts the incident light carrying information into electrical signals corresponding to the different position points, and sends the electrical signals corresponding to the different position points to the processor 3, where the image sensor 2 corresponds to a part of a nonlinear layer of the neural network.

In this embodiment, the processor 3 performs full-connection processing on the electrical signals at different position points, or the processor 3 performs full-connection processing and second nonlinear activation processing on the electrical signals at different position points, so as to obtain an output signal of the artificial neural network.

It is understood that, in the present embodiment, the image sensor 2 corresponds to a part of the nonlinear layer of the neural network, and the processor 3 corresponds to another part of the nonlinear layer of the neural network and the output layer, and it is also understood to correspond to the remaining layers (all other layers) of the neural network except the first nonlinear activation function in the input layer, the linear layer and the nonlinear layer.

In the present embodiment, it should be noted that the square detection response of the image sensor 2 corresponds to the first nonlinear activation function in the nonlinear layer of the neural network, in this case, only the full connection processing may be performed in the processor, and the second nonlinear activation processing is not performed, or both the full connection processing and the second nonlinear activation processing may be performed in the processor. The method may be determined according to an actual application scenario of the chip, which is not limited in this embodiment.

In addition, it should be added that the processor 3 may be disposed in the smart chip, that is, the processor 3 may be disposed in the smart chip together with the filter layer 1 and the image sensor 2, or may be disposed outside the smart chip separately and connected to the image sensor 2 in the smart chip through a data line or a connection device, which is not limited in this embodiment.

In addition, it should be noted that the processor 3 may be implemented by a computer, may also be implemented by an ARM or FPGA circuit board having a certain operation capability, and may also be implemented by a microprocessor, which is not limited in this embodiment. Furthermore, as mentioned above, the processor 3 may be integrated within the smart chip or may be provided separately from the smart chip. When the processor 3 is independent of the smart chip, the electrical signal in the image sensor 2 can be read out to the processor 3 through the signal reading circuit, and then the processor 3 performs full connection processing and nonlinear activation processing on the read electrical signal.

In this embodiment, it is understood that, when performing the second nonlinear activation processing, the processor 3 may be implemented by using a nonlinear activation function, for example, a Sigmoid function, a Tanh function, a ReLU function, and the like, which is not limited in this embodiment.

In this embodiment, the optical filter layer 1 corresponds to an input layer, a linear layer, and a connection weight from the input layer to the linear layer of the artificial neural network, the image sensor 2 corresponds to a part of a nonlinear layer of the artificial neural network, that is, a square detection response of the image sensor 2 corresponds to a first nonlinear activation function of the artificial neural network, the image sensor 2 is configured to perform nonlinear activation processing on incident light carrying information at different spatial location points through the square detection response and further convert the incident light carrying information into an electrical signal, and the processor 3 corresponds to the remaining layers of the artificial neural network and fully connects the electrical signals at different spatial locations, or further obtains an output signal of the artificial neural network through a second nonlinear activation function, thereby implementing intelligent perception, identification, and/or decision of a specific target.

As shown in the left side of fig. 2, the optical artificial neural network intelligent chip includes an optical filter layer 1, an image sensor 2, and a processor 3, and in fig. 2, the processor 3 is implemented using a signal readout circuit and a computer. As shown in the right side of fig. 2, the optical filter layer 1 in the optical artificial neural network intelligent chip corresponds to the input layer and the linear layer of the artificial neural network, the image sensor 2 corresponds to a part of the nonlinear layer of the artificial neural network, the processor 3 corresponds to another part of the nonlinear layer and the output layer of the artificial neural network, the filtering effect of the optical filter layer 1 on the incident light entering the optical filter layer 1 corresponds to the connection weight from the input layer to the linear layer, and the square detection response of the image sensor 2 corresponds to the first nonlinear activation function of the artificial neural network, so that it can be seen that the optical filter layer and the image sensor in the intelligent chip provided by this embodiment realize the related functions of the input layer, the linear layer and a part or all of the nonlinear activation functions in the artificial neural network in a hardware manner, so that the complicated signal processing and algorithm processing corresponding to the input layer and the linear layer are not needed to be performed when the intelligent processing is performed subsequently using the intelligent chip: (the optical filter layer and the image sensor are not needed to perform the complicated signal processing and algorithm processing corresponding to the input layer and the linear layer) and the input layer E.g., omitting calculations such as input layer to linear layer connection weights), which can significantly reduce power consumption and latency in artificial neural network processing.

As shown in the right side of FIG. 2, the optical filter layer 1 has different broadband spectrum modulation effects on the incident light, and will correspond to the incident light spectrum P at the cell location _λ Projecting/connecting to the emergent light field E _N The above step (1); the square detection response of the image sensor 2 corresponds to the nonlinear activation function of the optical artificial neural network, and the emergent light field E of the optical filter layer 1 _N Conversion to photocurrent response of image sensor I _N The above. The processor 3 comprises a signal readout circuit and a computer, the signal readout circuit in the processor 3 reads out the photocurrent response I _N The signal is transmitted to a computer, the computer carries out full connection processing of the electric signal or carries out nonlinear activation processing again, and finally, the result is output.

As shown in fig. 3, the optical modulation structure on the optical filter layer 1 is integrated on the surface of the image sensor 2, modulates the incident light, projects/connects the spectrum information of the incident light to different pixels of the image sensor 2 to obtain an electrical signal containing information carried by the incident light, i.e., after the incident light passes through the optical filter layer 1, the incident light is nonlinearly activated by the square detection response of the image sensor 2 and then converted into an electrical signal to form an image containing the spectrum information of the incident light, and finally, the electrical signal containing the spectrum information and the image information of the incident light is processed by the processor 3 connected to the image sensor 2 to obtain an output result.

In this embodiment, the incident light carrying information may include one or more (including two) of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

For example, in one implementation, the incident light carrying information may include light intensity distribution information, and in other implementations, multiple information of image information, spectrum information, an angle of incident light, and phase information of incident light of the target object may be used to identify the target object, so that intelligent identification of the target object may be more accurately implemented.

Therefore, the optical artificial neural network chip provided by this embodiment can actually utilize the image information, the spectrum information, the angle of the incident light and the phase information of the incident light of the target object at the same time, that is, the incident light at different points in space carries information, and the artificial neural network is embedded in hardware, and information such as material components, image shapes, three-dimensional depths and the like can be further extracted from the spatial image, the spectrum, the angle and the phase information, so that the problem that it is difficult to ensure the accuracy of identification, for example, to distinguish a real person from a picture, by using the two-dimensional image information of the target object mentioned in the background technology section can be solved, the intelligent sensing, identification and/or decision functions facing different application fields can be realized, and the spectral optical artificial neural network intelligent chip with low power consumption, low delay and high accuracy can be realized.

The embodiment of the invention provides an optical artificial neural network intelligent chip and a preparation method thereof, which realize a brand-new intelligent chip capable of realizing the function of an artificial neural network, wherein in the intelligent chip, an optical filter layer corresponds to an input layer and a linear layer of the artificial neural network, and an image sensor corresponds to a part of a nonlinear layer of the artificial neural network; the processor corresponds to another portion of the non-linear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, the optical filter layer comprises an optical modulation structure, the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to the different position points on the surface of the image sensor, and in the embodiment of the invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. Meanwhile, in the embodiment of the invention, the image sensor carries out the first nonlinear activation processing on the incident light carrying information corresponding to different position points after being modulated by the optical filter layer through square detection response and then converts the incident light carrying information into electric signals corresponding to different position points, and the electric signals corresponding to the different position points are sent to the processor, the processor carries out full connection processing on the electric signals corresponding to the different position points, or the processor carries out full connection processing and secondary nonlinear activation processing on the electric signals corresponding to different position points to obtain the output signal of the artificial neural network, so that, in the intelligent chip, the optical filter layer corresponds to an input layer, a linear layer and a connection weight of the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to a first-time nonlinear activation function in a nonlinear layer of the artificial neural network; the processor corresponds to the full connection and output layer of the artificial neural network, or the processor corresponds to the full connection and output layer of the artificial neural network, the second nonlinear activation function and output layer in the nonlinear layer, namely, the optical filter layer and the image sensor in the intelligent chip realize the related functions of the input layer, the linear layer and part of the nonlinear activation function in the artificial neural network, namely, the embodiment of the invention peels off the input layer, the linear layer and part or all of the nonlinear activation function in the artificial neural network realized by software in the prior art, and realizes the structures of the input layer, the linear layer and part or all of the nonlinear activation function in the artificial neural network by using a hardware mode, so that the intelligent processing with the input layer is not required to be carried out subsequently when the intelligent chip is used for carrying out the intelligent processing of the artificial neural network, The linear layer and the complex signal processing and algorithm processing corresponding to a part or all of the nonlinear activation functions are only required to be carried out by the processor in the intelligent chip, the signal processing or full connection with the electric signal and the second nonlinear activation processing, so that the power consumption and the time delay during the artificial neural network processing can be greatly reduced.

In addition, it should be noted that, in the prior art, only two-dimensional image information of a person or an object is used for identifying the person or the object, however, the two-dimensional image information is difficult to ensure the accuracy of identification, for example, it is difficult to distinguish a real face from a face picture. Therefore, based on this, in one implementation manner, the incident light carrying information may include light intensity distribution information and spectrum information, so that when an intelligent recognition task is executed by using the optical artificial neural network intelligent chip provided in this application, the light intensity distribution information and the spectrum information of an object to be recognized may be simultaneously used, and thus it is visible that since the incident light carrying information covers the image, component, shape, three-dimensional depth, structure and other information of the target object, when recognition processing is performed according to the incident light carrying information at different points in the target object space, multi-dimensional information of the image, component, shape, three-dimensional depth, structure and other information of the target object may be covered, so that the mentioned problem that it is difficult to ensure the accuracy of recognition by using two-dimensional image information of the target object, for example, it is difficult to distinguish whether it is a real person or a picture, may be solved, therefore, the identification of the object to be identified can be realized more accurately. In addition, in another implementation manner, the incident light carrying information may further include light intensity distribution information, spectrum information, and angle information of the incident light, so that information such as an image, a component, a shape, a three-dimensional depth, a three-dimensional structure, and the like of the target object can be captured more comprehensively, and thus, the identification of the object to be identified can be achieved more accurately. In addition, in another implementation manner, the incident light carrying information may further include light intensity distribution information, spectrum information, incident light angle information, and incident light phase information, so that information such as an image, a component, a shape, a three-dimensional depth, a three-dimensional structure, and the like of the target object can be captured more comprehensively, and thus, the identification of the object to be identified can be achieved more accurately. In the optical artificial neural network intelligent chip provided by the embodiment of the invention, an optical filter layer corresponds to an input layer and a linear layer of an artificial neural network, and an image sensor corresponds to a part of a nonlinear layer of the artificial neural network; the processor corresponds to another portion of the non-linear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, the optical filter layer comprises an optical modulation structure, the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to the different position points on the surface of the image sensor, and in the embodiment of the invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. Meanwhile, in the embodiment of the invention, the image sensor carries out the first nonlinear activation processing on the incident light carrying information corresponding to different position points after being modulated by the optical filter layer through square detection response and then converts the incident light carrying information into electric signals corresponding to different position points, and the electric signals corresponding to the different position points are sent to the processor, the processor carries out full connection processing on the electric signals corresponding to the different position points, or the processor carries out full connection processing and secondary nonlinear activation processing on the electric signals corresponding to different position points to obtain the output signal of the artificial neural network, so that, in the intelligent chip, the optical filter layer corresponds to an input layer, a linear layer and a connection weight of the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to a first-order nonlinear activation function in a nonlinear layer of the artificial neural network; the processor corresponds to the full connection and output layer of the artificial neural network, or the processor corresponds to the full connection and output layer of the artificial neural network, the second nonlinear activation function and output layer in the nonlinear layer, namely, the optical filter layer and the image sensor in the intelligent chip realize the related functions of the input layer, the linear layer and part of the nonlinear activation function in the artificial neural network, namely, the embodiment of the invention peels off the input layer, the linear layer and part or all of the nonlinear activation function in the artificial neural network realized by software in the prior art, and realizes the structures of the input layer, the linear layer and part or all of the nonlinear activation function in the artificial neural network by using a hardware mode, so that the intelligent processing with the input layer is not required to be carried out subsequently when the intelligent chip is used for carrying out the intelligent processing of the artificial neural network, The linear layer and the complex signal processing and algorithm processing corresponding to a part or all of the nonlinear activation functions are only required to be carried out by the processor in the intelligent chip, the signal processing or full connection with the electric signal and the second nonlinear activation processing, so that the power consumption and the time delay during the artificial neural network processing can be greatly reduced. Therefore, the embodiment of the invention takes the optical filter layer as the input layer and the linear layer of the artificial neural network and the connection weight of the input layer to the linear layer, and takes the square detection response of the image sensor as the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor is used as a full connection and output layer of the artificial neural network, or the processor is used for corresponding to a second nonlinear activation function and an output layer in the full connection and nonlinear layer of the artificial neural network, so that the embodiment of the invention not only can omit complex signal processing and algorithm processing corresponding to an input layer, a linear layer and a part of nonlinear activation function in the prior art, but also actually simultaneously utilizes image information, spectrum information, incident light angle and incident light phase information of a target object, namely, incident light carrying information at different points of the target object space, so that the embodiment of the invention can be seen, as the incident light carrying information at different points of the target object space covers information of image, composition, shape, three-dimensional depth, structure and the like of the target object, when identification processing is carried out according to the incident light carrying information at different points of the target object space, the method can cover multi-dimensional information such as images, components, shapes, three-dimensional depths, structures and the like of the target object, and therefore the problem that identification accuracy is difficult to guarantee, for example, real persons or pictures are difficult to distinguish by adopting two-dimensional image information of the target object, which is mentioned in the background technology section, can be solved.

Based on the content of the above embodiment, in this embodiment, the optical artificial neural network intelligent chip is used for an intelligent processing task of a target object; the intelligent processing task at least comprises one or more of intelligent perception, intelligent identification and intelligent decision task;

the trained optical artificial neural network intelligent chip is an optical artificial neural network intelligent chip comprising a trained optical modulation structure, an image sensor and a processor; the trained optical modulation structure, the image sensor and the processor are the optical modulation structure, the image sensor and the processor which meet the training convergence condition and are obtained by training an optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using input training samples and output training samples corresponding to the intelligent processing task; or, the trained optical modulation structure, image sensor and processor refer to the optical modulation structure, image sensor and processor which satisfy the training convergence condition and are obtained by training an optical artificial neural network intelligent chip of the processor which comprises different optical modulation structures, image sensors and different full-link parameters and different second nonlinear activation parameters by using an input training sample and an output training sample corresponding to the intelligent processing task.

In this embodiment, the input training samples comprise incident light reflected, transmitted and/or radiated by a target object in a respective smart processing task; the output training sample comprises an intelligent processing result (such as a recognition result, a perception result, a decision result or a qualitative analysis result) of the target object.

In this embodiment, the optical artificial neural network intelligent chip may be used for intelligent processing tasks of the target object, for example, including one or more of intelligent perception, intelligent recognition and intelligent decision tasks.

In this embodiment, it is understood that the intelligent sensing refers to mapping signals of the physical world to the digital world through hardware devices of cameras, microphones or other sensors by means of leading-edge technologies such as voice recognition, image recognition, etc., and further promoting the digital information to recognizable levels, such as memory, understanding, planning, decision making, etc. The intelligent recognition is a technology for processing, analyzing and understanding images by using a computer to recognize targets and objects in different modes, and the intelligent recognition technology at the present stage is generally divided into face recognition and commodity recognition, wherein the face recognition is mainly applied to security inspection, identity verification and mobile payment, and the commodity recognition is mainly applied to the commodity circulation process, in particular to the unmanned retail fields such as unmanned shelves and intelligent retail cabinets. The intelligent decision is to solve the problem that a computer automatically organizes and coordinates the operation of multiple models, access and process data in a large number of databases, and perform corresponding data processing and numerical calculation.

In this embodiment, the reflected light, the transmitted light, and/or the radiated light of the target object enter the trained optical artificial neural network intelligent chip to obtain an intelligent processing result of the target object.

In this embodiment, an identification task of a target object is taken as an example for description, and it can be understood that, when the intelligent chip is used for performing an identification task, the optical artificial neural network intelligent chip needs to be trained first, where training the optical artificial neural network intelligent chip refers to determining, through training, an optical modulation structure suitable for a current identification task, and a full connection parameter and a nonlinear activation parameter suitable for the current identification task.

It can be understood that, because the filtering effect of the optical filter layer on the incident light entering the optical filter layer corresponds to the connection weight from the artificial neural network input layer to the linear layer, during training, changing the optical modulation structure in the optical filter layer is equivalent to changing the connection weight from the artificial neural network input layer to the linear layer, and by training the convergence condition, the optical modulation structure suitable for the current recognition task, and the full connection parameter and the nonlinear activation parameter suitable for the current recognition task are determined, so that the training of the intelligent chip is completed.

It will be appreciated that after training the smart chip, the smart chip may be used to perform recognition tasks. Specifically, after the incident light carrying the image information and the spatial spectrum information of the target object enters the optical filter layer 1 of the trained intelligent chip, the light modulation structure in the optical filter layer 1 modulates the incident light, the intensity of the modulated light signal is detected by the image sensor 2 and converted into an electric signal, and then the processor 3 performs full connection processing or performs full connection and secondary nonlinear activation processing simultaneously, so that the recognition result of the target object can be obtained.

As shown in fig. 4, the complete flow for target object identification is: the wide-spectrum light source 100 irradiates on the target object 200, and then reflected light or transmitted light of the target object is collected by the optical artificial neural network intelligent chip 300, or light directly radiated outwards by the target object is collected by the optical artificial neural network intelligent chip 300 and processed by an optical filter layer, an image sensor and a processor in the intelligent chip, so that a recognition result can be obtained.

The trained optical artificial neural network intelligent chip comprises a trained optical modulation structure, an image sensor and a processor; the trained optical modulation structure, the image sensor and the processor are the optical modulation structure, the image sensor and the processor which meet the training convergence condition and are obtained by training an optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters by using input training samples and output training samples corresponding to the intelligent processing task.

For example, for the smart recognition task, the input training sample corresponding to the smart recognition task is a recognition object sample, and the output training sample corresponding to the smart recognition task is a recognition result of the recognition object sample. It can be understood that, for the recognition task, since the smart chip provided by this embodiment is also advantageous in that it can acquire image information, spectrum information, angle information of incident light, and phase information of incident light at different points in the recognition object space, in order to fully utilize this advantage, it is preferable to use a real recognition object for the recognition object sample as the input training sample, rather than a two-dimensional image of the recognition object. Of course, this does not mean that the two-dimensional image may not be used as the recognition object sample.

In addition, the optical artificial neural network intelligent chip provided by the embodiment can also be used for other intelligent processing tasks of the target object, such as intelligent perception, intelligent decision and other tasks.

In the present embodiment, the optical filter layer 1 serves as an input layer and a linear layer of the neural network, the image sensor 2 serves as a part of a nonlinear layer of the neural network (that is, the square detection response of the image sensor 2 serves as a first nonlinear activation function of the neural network), and in order to minimize a loss function of the neural network, the modulation intensities of different wavelength components in incident light of a target object by the optical modulation structure in the optical filter layer are used as connection weights of the input layer to the linear layer of the neural network, and the modulation intensities of different wavelength components in the incident light of the target object can be adjusted by adjusting the structure of the filter, so that the adjustment of the connection weights of the input layer to the linear layer is realized, and the training of the neural network is further optimized.

Therefore, in this embodiment, the optical modulation structure is obtained based on neural network training, a computer performs optical simulation on a training sample to obtain sample modulation intensities of the optical modulation structure on different wavelength components of incident light of a target object in an intelligent processing task in the training sample, the sample modulation intensities are used as connection weights from an input layer to a linear layer of the neural network to perform nonlinear activation, and the training sample corresponding to the intelligent processing task is used for neural network training until the neural network converges, and the corresponding training sample optical modulation structure is used as an optical filter layer corresponding to the intelligent processing task.

As can be seen from this, in the present embodiment, by implementing the input layer and the linear layer (optical filter layer) of the neural network and a part of the nonlinear layer (square detection response of the image sensor 2 as the first nonlinear activation function of the neural network) at the physical layer, not only can complicated signal processing and algorithm processing corresponding to the input layer, the linear layer, and a part or all of the nonlinear activation functions in the related art be omitted. Meanwhile, the embodiment of the invention actually utilizes the image information, the spectrum information, the angle information of the incident light and the phase information of the incident light at different points of the target object space, namely the incident light carrying information at different points of the target object space, so that the embodiment of the invention can see that the incident light carrying information at different points of the target object space covers the image, the composition, the shape, the three-dimensional depth, the structure and other information of the target object, so that when the identification processing is carried out according to the incident light carrying information at different points of the target object space, the multi-dimensional information of the image, the composition, the shape, the three-dimensional depth, the structure and the like of the target object can be covered, and the problem that the adoption of the two-dimensional image information of the target object, which is mentioned in the background technology, is difficult to ensure the identification accuracy, such as difficult to distinguish whether the real person or the picture, is solved, therefore, the invention provides the optical artificial neural network chip, the method can not only realize the effects of low power consumption and low time delay, but also realize the effect of high accuracy, thereby being capable of preparing for intelligent processing tasks such as intelligent perception, identification and/or decision making.

Based on the content of the foregoing embodiment, in this embodiment, when training an optical artificial neural network intelligent chip including different optical modulation structures, image sensors, and processors with different full connection parameters, or training an optical artificial neural network intelligent chip including different optical modulation structures, image sensors, and processors with different full connection parameters and different second nonlinear activation parameters, the different optical modulation structures are designed and implemented by using a computer optical simulation design.

In this embodiment, the optical simulation enables a user to experience the product through a digital environment before making a physical prototype, for example, for an automobile, because illumination and reflected light may interfere with the attention of a driver, especially in the case of driving at night, a suitable optical simulation solution may not only effectively help the user to improve the design efficiency, but also simulate the interaction between light and materials, so as to know the display effect of the product under the real condition. Therefore, in the embodiment, the optical modulation structure is designed through computer optical simulation, and the optical modulation structure is adjusted through optical simulation until the corresponding optical modulation structure is determined to be the size of the optical modulation structure to be manufactured finally when the neural network converges, so that the prototype manufacturing time and cost are saved, the product efficiency is improved, and the complex optical problem is solved easily.

For example, the light modulation structure can be simulated and designed through FDTD software, and the light modulation structure is changed in optical simulation, so that the modulation intensity of the light modulation structure on different incident lights can be accurately predicted and used as the connection weight of a neural network input layer and a linear layer to train an optical artificial neural network intelligent chip and accurately obtain the light modulation structure.

Therefore, the light modulation structure is designed in a computer optical simulation design mode, so that the time and the cost for manufacturing the prototype of the light modulation structure are saved, and the product efficiency is improved.

Based on the content of the above embodiments, in the present embodiment, the light modulation structure in the optical filter layer includes a regular structure and/or an irregular structure; and/or the light modulating structures in the optical filter layer comprise discrete structures and/or continuous structures.

In this embodiment, the light modulation structure in the optical filter layer may only include a regular structure, may also only include an irregular structure, and may also include both a regular structure and an irregular structure.

In this embodiment, where the light modulation structure includes a regular structure, it may mean: the minimum modulation units included in the light modulation structure are regular structures, for example, the minimum modulation units may be in regular patterns such as rectangles, squares, and circles. Further, where the light modulating structure comprises a regular structure, it may also refer to: the arrangement of the minimum modulation units included in the light modulation structure is regular, for example, the arrangement may be in a regular array form, a circular form, a trapezoidal form, a polygonal form, and the like. Further, where the light modulating structure comprises a regular structure, it may also mean: the minimum modulation units included in the light modulation structure are regular structures, and the arrangement mode of the minimum modulation units is also regular.

In this embodiment, the light modulation structure including the irregular structure may refer to: the light modulation structure comprises a minimum modulation unit which is an irregular structure, for example, the minimum modulation unit can be an irregular figure such as an irregular polygon, a random shape and the like. Further, where the light modulating structure includes an irregular structure, it may also mean: the arrangement of the minimum modulation units included in the light modulation structure is irregular, for example, the arrangement may be in an irregular polygon form, a random arrangement form, or the like. Further, where the light modulating structure includes an irregular structure, it may also mean: the minimum modulation units included in the light modulation structure are irregular structures, and the arrangement mode of the minimum modulation units is also irregular.

In this embodiment, the optical modulation structure in the optical filter layer may include a discrete structure, a continuous structure, or both a discrete structure and a continuous structure.

In this embodiment, where the light modulation structure includes a continuous type structure, it may mean: the light modulation structure is formed by continuous modulation patterns; where the light modulating structure comprises a discrete structure may refer to: the light modulating structure is formed of discrete modulation patterns.

It is understood that the continuous modulation pattern may refer to a rectilinear pattern, a wavy pattern, a polygonal pattern, and the like.

It is to be understood that a discrete modulation pattern herein may refer to a modulation pattern formed by discrete patterns (e.g., discrete dots, discrete triangles, discrete stars, etc.).

In this embodiment, it should be noted that the optical modulation structure has different modulation effects on light with different wavelengths, and specific modulation methods include, but are not limited to, scattering, absorption, interference, surface plasmon, resonance enhancement, and the like. By designing different filter structures, corresponding transmission spectrums are different after light passes through different groups of filter structures.

Based on the content of the above embodiments, in the present embodiment, the optical filter layer is a single-layer structure or a multi-layer structure.

In this embodiment, the optical filter layer may have a single-layer filter structure, or may have a multi-layer filter structure, for example, a multi-layer structure including two, three, or four layers.

In the present embodiment, as shown in fig. 1, the optical filter layer 1 is a single-layer structure, the thickness of the optical filter layer 1 is related to the target wavelength range, and the thickness of the grating structure may be 50nm to 5 μm for wavelengths of 400nm to 10 μm.

It is understood that since the optical filter layer 1 serves to spectrally modulate incident light, it is preferable to fabricate materials with high refractive index and low loss, such as silicon, germanium, silicon-germanium materials, silicon compounds, germanium compounds, III-V materials, and the like, wherein silicon compounds include, but are not limited to, silicon nitride, silicon dioxide, silicon carbide, and the like.

In addition, it should be noted that, in order to form more or more complicated connection weights between the input layer and the linear layer, preferably, the optical filter layer 1 may be set to have a multilayer structure, and the optical modulation structures corresponding to the respective layers may be set to have different structures, so as to increase the spectral modulation capability of the optical filter layer on the incident light, so as to form more or more complicated connection weights between the input layer and the linear layer, thereby improving the accuracy of the smart chip in processing the smart task.

In addition, for the filter layer including a multi-layer structure, the material of each layer structure may be the same or different, for example, for the optical filter layer 1 having two layers, the first layer may be a silicon layer, and the second layer may be a silicon nitride layer.

The thickness of the optical filter layer 1 is related to the target wavelength range, and the total thickness of the multilayer structure may be 50nm to 5 μm for wavelengths of 400nm to 10 μm.

Based on the content of the above embodiment, in this embodiment, the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, and each micro-nano unit corresponds to one or more pixel points on the image sensor; the structures of the micro-nano units are the same or different.

In this embodiment, in order to obtain connection weights (used for connecting connection weights between the input layer and the linear layer) distributed in an array so as to facilitate the processor to perform subsequent full-connection and nonlinear activation processing, preferably, in this embodiment, the light modulation structure is in an array structure form, specifically, the light modulation structure includes a unit array composed of a plurality of micro-nano units, and each micro-nano unit corresponds to one or more pixel points on the image sensor. It should be noted that the structures of the micro-nano units may be the same or different. In addition, the structure of each micro-nano unit may be periodic or non-periodic. In addition, it should be noted that each micro-nano unit may further include multiple groups of micro-nano structure arrays, and the structures of the micro-nano structure arrays of each group are the same or different.

As illustrated in fig. 5 to 9, in this embodiment, as shown in fig. 5, the optical filter layer 1 includes a plurality of repeated continuous or discrete micro-nano units, such as 11, 22, 33, 44, 55, and 66, each micro-nano unit has the same structure (and each micro-nano unit has a non-periodic structure), and each micro-nano unit corresponds to one or more pixel points on the image sensor 2; as shown in fig. 6, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each micro-nano unit has the same structure (different from fig. 5 in that each micro-nano unit in fig. 6 is a periodic structure), and each micro-nano unit corresponds to one or more pixel points on the image sensor 2; as shown in fig. 7, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each micro-nano unit has the same structure (and each micro-nano unit has a periodic structure), each micro-nano unit corresponds to one or more pixel points on the image sensor 2, and the difference from fig. 6 is that the unit shape of the periodic array in each micro-nano unit in fig. 7 has quadruple rotational symmetry; as shown in fig. 8, the optical filter layer 1 includes a plurality of micro-nano units, such as 11, 22, 33, 44, 55, and 66, and the difference from fig. 6 lies in that each micro-nano unit has a different structure, and each micro-nano unit corresponds to one or more pixel points on the image sensor 2, in this embodiment, the optical filter layer 1 includes a plurality of micro-nano units that are different from each other, that is, different regions on the smart chip have different modulation effects on incident light, so that the degree of freedom of design is improved, and the accuracy of identification can be further improved. As shown in fig. 9, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each of which has the same structure, and is different from fig. 5 in that each of the micro-nano units is composed of a discrete non-periodic array structure, and each of the micro-nano units corresponds to one or more pixels on the image sensor 2.

In this embodiment, the micro-nano unit has different modulation effects on light with different wavelengths, and specific modulation methods include, but are not limited to, scattering, absorption, interference, surface plasmons, resonance enhancement, and the like. By designing different filter structures, corresponding transmission spectrums are different after light passes through different groups of filter structures.

Based on the content of the above embodiment, in this embodiment, the micro-nano unit includes a regular structure and/or an irregular structure; and/or the micro-nano unit comprises a discrete structure and/or a continuous structure.

In this embodiment, the micro-nano unit may only include a regular structure, may also only include an irregular structure, and may also include both a regular structure and an irregular structure.

In this embodiment, the micro-nano unit including a regular structure may refer to: the micro-nano unit comprises a minimum modulation unit which is of a regular structure, for example, the minimum modulation unit can be a regular pattern such as a rectangle, a square and a circle. In addition, the micro-nano unit including a regular structure here can also mean: the arrangement mode of the minimum modulation units contained in the micro-nano units is regular, for example, the arrangement mode can be a regular array form, a circular form, a trapezoidal form, a polygonal form and the like. In addition, the micro-nano unit including a regular structure here can also mean: the minimum modulation units contained in the micro-nano units are of a regular structure, and the arrangement mode of the minimum modulation units is also regular.

In this embodiment, the micro-nano unit including an irregular structure may refer to: the minimum modulation unit contained in the micro-nano unit is of an irregular structure, and for example, the minimum modulation unit can be irregular patterns such as irregular polygons, random shapes and the like. In addition, the micro-nano unit comprising an irregular structure here can also mean: the arrangement mode of the minimum modulation units contained in the micro-nano units is irregular, for example, the arrangement mode can be an irregular polygon form, a random arrangement form and the like. In addition, the micro-nano unit including an irregular structure here can also mean: the minimum modulation units contained in the micro-nano units are of irregular structures, and meanwhile, the arrangement mode of the minimum modulation units is also irregular.

In this embodiment, the micro-nano unit in the optical filter layer may include a discrete structure, may also include a continuous structure, and may also include both a discrete structure and a continuous structure.

In this embodiment, the micro-nano unit including a continuous structure may refer to: the micro-nano unit is formed by continuous modulation patterns; here, the micro-nano unit including a discrete structure may mean: the micro-nano unit is formed by discrete modulation patterns.

In this embodiment, it should be noted that different micro-nano units have different modulation effects on light with different wavelengths, and specific modulation methods include, but are not limited to, scattering, absorption, interference, surface plasmons, resonance enhancement, and the like. By designing different micro-nano units, corresponding transmission spectrums are different after light passes through different groups of micro-nano units.

Based on the content of the above embodiment, in this embodiment, the micro-nano unit includes multiple sets of micro-nano structure arrays, and the structures of the micro-nano structure arrays of the sets are the same or different.

In this embodiment, as shown in fig. 5, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each micro-nano unit includes a plurality of micro-nano structure arrays, for example, the micro-nano unit 11 includes 4 different

micro-nano structure arrays

110, 111, 112, and 113, and the filter unit 44 includes 4 different

micro-nano structure arrays

440, 441, 442, and 443. As shown in fig. 10, the optical filter layer 1 includes a plurality of micro-nano units, such as 11, 22, 33, 44, 55, 66, each micro-nano unit includes a plurality of micro-nano structure arrays, for example, the micro-nano unit 11 includes 4 identical

micro-nano structure arrays

110, 111, 112, and 113.

It should be noted that, here, the micro-nano units including four groups of micro-nano structure arrays are only used for illustration, and do not play a limiting role, and in practical application, the micro-nano units including six groups, eight groups, or other number of groups of micro-nano structure arrays may also be set as needed.

In this embodiment, each micro-nano structure array in the micro-nano unit has different modulation effects on light with different wavelengths, and the modulation effects on input light between each group of filtering structures are also different, and specific modulation modes include, but are not limited to, scattering, absorption, interference, surface plasmons, resonance enhancement, and the like. By designing different micro-nano structure arrays, corresponding transmission spectrums are different after light passes through different groups of micro-nano structure arrays.

Based on the content of the above embodiment, in this embodiment, each group of micro-nano structure arrays has a function of broadband filtering or narrowband filtering.

In this embodiment, in order to obtain the modulation intensities of different wavelength components of the incident light of the target object as the connection weights of the neural network input layer and the linear layer, the broadband filtering and the narrowband filtering are implemented by using different micro-nano structure arrays, and therefore, in this embodiment, the micro-nano structure arrays perform the broadband filtering or the narrowband filtering on the incident light of the target object, so as to obtain the modulation intensities of the different wavelength components of the incident light of the target object. As shown in fig. 11 and 12, each group of micro-nano structure array in the optical filter layer has a broadband filtering or narrowband filtering effect.

It can be understood that each group of micro-nano structure arrays can have a broadband filtering function, can also have a narrowband filtering function, can also partially have a broadband filtering function, and partially have a narrowband filtering function. In addition, the wide band filtering range and the narrow band filtering range of each group of micro-nano structure array can be the same or different. For example, by designing the dimensional parameters such as the period, duty ratio, radius, side length and the like of each group of micro-nano structures in the micro-nano unit, the micro-nano unit has a narrow-band filtering effect, that is, only light with one (or a few) wavelength can pass through the micro-nano unit. For another example, by designing the dimensional parameters such as the period, duty ratio, radius, side length and the like of each group of micro-nano structures in the micro-nano unit, the micro-nano unit has a broadband filtering effect, that is, light with more wavelengths or all wavelengths can be allowed to pass through.

It can be understood that, in specific use, the filtering state of each group of micro-nano structure array can be determined in a mode of performing broadband filtering, narrowband filtering or a combination thereof according to an application scene.

Based on the content of the foregoing embodiment, in this embodiment, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

In this embodiment, each micro-nano structure array may be a periodic structure array, or may be a non-periodic structure array, or may be a partial periodic structure array and a partial non-periodic structure array. The periodic structure array is easy to carry out optical simulation design, and the non-periodic structure array can realize more complex modulation action.

In this embodiment, as shown in fig. 5, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each micro-nano unit is composed of a plurality of micro-nano structure arrays, the structures of the micro-nano structure arrays are different from each other, and the micro-nano structure arrays are aperiodic structures. The aperiodic structure refers to that the shapes of the modulation holes on the micro-nano structure array are arranged according to a non-periodic arrangement mode. As shown in fig. 5, the micro-nano unit 11 includes 4 different

aperiodic structure arrays

110, 111, 112, and 113, the micro-nano unit 44 includes 4 different

aperiodic structure arrays

440, 441, 442, and 443, and the micro-nano structure array with aperiodic structure is designed by training neural network data aiming at an intelligent processing task in the previous stage, and is usually an irregularly shaped structure. As shown in fig. 6, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66, each of which is composed of a plurality of sets of micro-nano structure arrays, the structures of the micro-nano structure arrays are different from each other, and unlike fig. 5, the micro-nano structure array is a periodic structure. The periodic structure refers to that the shapes of modulation holes on the micro-nano structure array are arranged according to a periodic arrangement mode, and the size of the period is usually 20 nm-50 mu m. As shown in fig. 6, the micro-nano unit 11 includes 4 different

periodic structure arrays

110, 111, 112, and 113, the micro-nano unit 44 includes 4 different

periodic structure arrays

440, 441, 442, and 443, and the filter structure of the periodic structure is designed by training neural network data for an intelligent processing task in an early stage, and is usually an irregularly shaped structure. As shown in fig. 7, the optical filter layer 1 includes a plurality of micro-nano units, such as 11, 22, 33, 44, 55, and 66, which are different from each other, each micro-nano unit is composed of a plurality of micro-nano structure arrays, the micro-nano structure arrays are different from each other, and the micro-nano structure arrays are periodic structures. The periodic structure refers to the shape of the filter structure arranged in a periodic arrangement, and the period is usually 20nm to 50 μm. As shown in fig. 7, the micro-nano structure arrays of the micro-nano unit 11 and the micro-nano unit 12 are different from each other, the micro-nano unit 11 includes 4 different

periodic structure arrays

110, 111, 112, and 113, the micro-nano unit 44 includes 4 different

periodic structure arrays

440, 441, 442, and 443, and the micro-nano structure array of the periodic structure is designed by training neural network data aiming at an intelligent processing task in an early stage, and is usually an irregular structure.

It should be noted that each micro-nano unit in fig. 5 to 9 includes four micro-nano structure arrays, the four micro-nano structure arrays are formed by four modulation holes with different shapes, and the four micro-nano structure arrays are used for modulating incident light differently. It should be noted that, here, the micro-nano units including four groups of micro-nano structure arrays are only used for illustration, and do not play a limiting role, and in practical application, the micro-nano units including six groups, eight groups, or other number of groups of micro-nano structure arrays may also be set as needed. In the present embodiment, the four different shapes may be (without limitation) a circle, a cross, a regular polygon, and a rectangle.

In this embodiment, each group of micro-nano structure arrays in the micro-nano unit has different modulation effects on light with different wavelengths, and the modulation effects on input light between each group of micro-nano structure arrays are also different, and specific modulation modes include, but are not limited to, scattering, absorption, interference, surface plasmons, resonance enhancement, and the like. By designing different micro-nano structure arrays, corresponding transmission spectrums are different after light passes through different groups of micro-nano structure arrays.

Based on the content of the foregoing embodiment, in this embodiment, the micro-nano unit includes one or more groups of hollow structures in a plurality of groups of micro-nano structure arrays.

For example, referring to an example shown in fig. 9, in this embodiment, as shown in fig. 9, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each micro-nano unit is composed of a plurality of micro-nano structure arrays, structures corresponding to the plurality of micro-nano structure arrays are different from each other, the micro-nano structure arrays are periodic structures, and different from the above embodiment, for any micro-nano unit, one or more groups of empty structures are included, and the empty structures are used for directly passing through incident light. It can be understood that when the micro-nano structure arrays comprise one or more groups of hollow structures, a richer spectrum modulation effect can be formed, so that the spectrum modulation requirement under a specific scene (or the specific connection weight requirement between the input layer and the linear layer under the specific scene) is met.

As shown in fig. 9, each micro-nano unit includes a group of micro-nano structure arrays and three groups of hollow structures, the micro-nano unit 11 includes 1 aperiodic structure array 111, the micro-nano unit 22 includes 1 aperiodic structure array 221, the micro-nano unit 33 includes 1 aperiodic structure array 331, the micro-nano unit 44 includes 1 aperiodic structure array 441, the micro-nano unit 55 includes 1 aperiodic structure array 551, and the micro-nano unit 66 includes 1 aperiodic structure array 661, where the micro-nano structure arrays are used for performing different modulations on incident light. It should be noted that, here, the example is only given by including a group of micro-nano structure arrays and three groups of hollow structures, and does not play a limiting role, and in practical application, micro-nano units including a group of micro-nano structure arrays and five groups of hollow structures or other numbers of groups of micro-nano structure arrays may also be set as required. In this embodiment, the micro-nano structure array may be made of modulation holes having a circular shape, a cross shape, a regular polygon shape, and a rectangular shape (not limited thereto).

It should be noted that, the micro-nano units may not include empty structures in the multiple sets of micro-nano structure arrays, that is, the multiple sets of micro-nano structure arrays may be non-periodic structure arrays or periodic structure arrays.

Based on the content of the foregoing embodiment, in this embodiment, the micro-nano unit has a polarization-independent characteristic.

In the embodiment, the micro-nano unit has a polarization-independent characteristic, so that the optical filter layer is insensitive to the polarization of incident light, and the optical artificial neural network intelligent chip insensitive to an incident angle and polarization is realized. The optical artificial neural network intelligent chip provided by the embodiment of the invention is insensitive to the incident angle and the polarization characteristic of incident light, namely, the measurement result is not influenced by the incident angle and the polarization characteristic of the incident light, so that the stability of the spectral measurement performance can be ensured, and the stability of intelligent processing, such as the stability of intelligent perception, the stability of intelligent recognition, the stability of intelligent decision and the like, can be further ensured. The micro-nano unit may also have polarization dependent properties.

Based on the content of the foregoing embodiment, in this embodiment, the micro-nano unit has quadruple rotational symmetry.

In this embodiment, it should be noted that the quadruple rotational symmetry belongs to a specific case of the polarization-independent characteristic, and the requirement of the polarization-independent characteristic can be satisfied by designing the micro-nano unit to have a structure with quadruple rotational symmetry.

As illustrated below with reference to the example shown in fig. 7, in this embodiment, as shown in fig. 7, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each micro-nano unit is composed of a plurality of micro-nano structure arrays, the structures corresponding to the plurality of micro-nano structure arrays are different from each other, the micro-nano structure arrays are periodic structures, and different from the above embodiments, the structure corresponding to each micro-nano structure array may be a structure having quadruple rotational symmetry, such as a circle, a cross, a regular polygon, and a rectangle, that is, after the structure is rotated by 90 °, 180 °, and 270 °, the structure is overlapped with the original structure, so that the structure has a polarization-independent characteristic, and the same intelligent identification effect can be obtained when different polarized light is incident.

Based on the content of the above embodiments, in the present embodiment, the optical filter layer is composed of one or more layers;

the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials and perovskite materials; and/or the filter layer is prepared from one or more of a photonic crystal, a super Surface, a random structure, a nano structure, a metal Surface Plasmon Polaritons (SPP) micro-nano structure and a tunable Fabry-Perot Cavity (FP Cavity).

The semiconductor material comprises one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed according to a preset proportion and direct band gap compound semiconductor materials; and/or the nanostructure comprises one or more of a nanodot two-dimensional material, a nanocolumn two-dimensional material and a nanowire two-dimensional material.

The photonic crystal, the super surface and the random structure combination can adopt CMOS (complementary metal oxide semiconductor) process compatibility, and have better modulation effect, and micro-nano modulation structure micropores can also be filled with other materials for surface smoothing; the quantum dots and the perovskite can utilize the spectral modulation characteristics of the material to minimize the volume of a single modulation structure; the SPP is small in size, and polarization-related light modulation can be realized; the liquid crystal can be dynamically regulated and controlled by voltage, so that the spatial resolution is improved; the adjustable Fabry-Perot resonant cavity can be dynamically adjusted and controlled, and the spatial resolution is improved.

Based on the contents of the above embodiments, in the present embodiment, the thickness of the optical filter layer is 0.1 λ to 10 λ, where λ denotes the center wavelength of incident light.

In this embodiment, it should be noted that if the thickness of the optical filter layer is much smaller than the central wavelength of the incident light, the effective spectrum modulation effect cannot be achieved; if the thickness of the optical filter layer is much larger than the center wavelength of the incident light, it is difficult to fabricate in the process and a large optical loss is introduced. Therefore, in the present embodiment, in order to reduce optical loss, facilitate preparation, and ensure effective spectrum modulation effect, the overall size (area) of each micro-nano unit in the optical filter layer 1 is generally λ ² ～10 ⁵ λ ² The thickness is usually 0.1 λ to 10 λ (λ represents the center wavelength of incident light of the target object). As shown in FIG. 5, the overall size of each micro-nano unit is 0.5 μm ² ～40000μm ² The dielectric material in the optical filter layer 1 is polysilicon with a thickness of 50 nm-2 μm.

Based on the content of the above embodiments, in the present embodiment, the image sensor is any one or more of the following:

a CMOS Image Sensor (CIS), a Charge Coupled Device (CCD), a Single Photon Avalanche Diode (SPAD) array, and a focal plane photodetector array.

In this embodiment, it should be noted that, by using the wafer-level CMOS image sensor CIS, monolithic integration is implemented at the wafer level, which may reduce the distance between the image sensor and the optical filter layer to the greatest extent, and is beneficial for reducing the size of the cell and reducing the device volume and packaging cost, the SPAD may be used for weak light detection, and the CCD may be used for strong light detection.

In this embodiment, the optical filter layer and the image sensor may be manufactured by a Complementary Metal Oxide Semiconductor (CMOS) integration process, which is beneficial to reducing the failure rate of the device, improving the yield of the device, and reducing the cost. For example, the optical filter layer can be prepared by growing one or more layers of dielectric material directly on the image sensor, etching, depositing a metal material before removing the sacrificial layer used for etching, and finally removing the sacrificial layer.

Based on the content of the above embodiment, in the present embodiment, the types of the artificial neural network include: a feed-forward neural network.

In this embodiment, a feed Forward Neural Network (FNN), also called a Deep feed forward Network (DFN), and a Multi-Layer Perceptron (MLP) are the simplest Neural networks, and each neuron is arranged in a Layer. Each neuron is connected to only the neuron in the previous layer. And receiving the output of the previous layer and outputting the output to the next layer, wherein no feedback exists between the layers. The feedforward neural network has simple structure, easy realization on hardware and wide application, can approach any continuous function and square integrable function with any precision, and can accurately realize any finite training sample set. The feed forward network is a static non-linear mapping. Complex non-linear processing capabilities can be obtained by complex mapping of simple non-linear processing units.

Based on the content of the foregoing embodiments, in this embodiment, a light-transmitting medium layer is disposed between the optical filter layer and the image sensor.

In this embodiment, it should be noted that, by providing a light-transmitting medium layer between the optical filter layer and the image sensor, the optical filter layer and the image sensor layer can be effectively separated from each other, so as to avoid mutual interference between the optical filter layer and the image sensor layer.

Based on the content of the above embodiments, in the present embodiment, the image sensor is a front-illuminated type, including: the optical filter layer is integrated on one surface, far away from the optical detection layer, of the metal wire layer; or the like, or, alternatively,

the image sensor is of a back-illuminated type, including: the optical filter comprises an optical detection layer and a metal wire layer which are arranged from top to bottom, wherein an optical filter layer is integrated on one surface of the optical detection layer, which is far away from the metal wire layer.

In the present embodiment, as shown in fig. 13, which is a front-illuminated image sensor, the silicon detection layer 21 is below the metal line layer 22, and the optical filter layer 1 is directly integrated onto the metal line layer 22.

In the present embodiment, unlike fig. 13, fig. 14 shows a back-illuminated image sensor, in which a silicon detection layer 21 is above a metal wire layer 22 and an optical filter layer 1 is directly integrated onto the silicon detection layer 21.

It should be noted that, for the back-illuminated image sensor, the silicon detection layer 21 is above the metal wire layer 22, so that the influence of the metal wire layer on the incident light can be reduced, and the quantum efficiency of the device can be improved.

From the above, it can be seen that the present embodiment uses the optical filter layer as the input layer and the linear layer of the artificial neural network, uses the image sensor as a part of the nonlinear layer of the artificial neural network (uses the square detection response of the image sensor as the first nonlinear activation function of the artificial neural network), and uses the filtering effect of the optical filter layer on the incident light entering the optical filter layer as the connection weight from the input layer to the linear layer, and the optical filter layer and the image sensor in the intelligent chip of the optical artificial neural network provided by the present embodiment implement the relevant functions of the input layer, the linear layer and the part of the nonlinear activation function in the artificial neural network in a hardware manner, so that the complicated signal processing and algorithm processing corresponding to the input layer, the linear layer and the part of the nonlinear activation function are not required to be performed when the intelligent chip is used for the subsequent intelligent processing, therefore, the power consumption and the time delay during the artificial neural network processing can be greatly reduced. In addition, in the embodiment, the image information, the spectrum information, the angle information of the incident light and the phase information of the incident light at different points in the target object space are simultaneously utilized, so that the intelligent processing of the target object can be more accurately realized.

Therefore, in the embodiment of the invention, the optical filter layer is used as the input layer and the linear layer of the artificial neural network, the image sensor is used as a part of the nonlinear layer of the artificial neural network, the spatial spectrum information of the object is projected into the photocurrent response of the detector, the full connection and the secondary nonlinear activation of the electric signal are realized in the processor, and the functions of intelligent sensing, identification and/or decision making with low power consumption, low time delay and high accuracy are realized. The optical artificial neural network intelligent chip based on the optical filter and the image sensor in the embodiment of the invention has the following effects: the artificial neural network is partially embedded into an image sensor comprising various optical filter layers, so that the functions of quick and accurate intelligent perception, identification and/or decision are realized. In addition, the embodiment of the invention can realize monolithic integration at the wafer level, thereby reducing the distance between the sensor and the optical filter layer to the greatest extent, being beneficial to reducing the size of a unit and reducing the volume and the packaging cost of a device.

Based on the same inventive concept, another embodiment of the present invention provides an intelligent device, including: the optical artificial neural network intelligent chip is described in the above embodiments. The intelligent equipment comprises one or more of a smart phone, a smart computer, intelligent recognition equipment, intelligent perception equipment and intelligent decision-making equipment.

Since the intelligent device provided in this embodiment includes the optical artificial neural network intelligent chip described in the above embodiment, the intelligent device provided in this embodiment has all the beneficial effects of the optical artificial neural network intelligent chip described in the above embodiment, and since the above embodiment has been described in detail, this embodiment is not described again.

Based on the same inventive concept, another embodiment of the present invention provides a method for manufacturing an optical artificial neural network intelligent chip according to the above embodiment, as shown in fig. 15, specifically including the following steps:

step 1510, preparing an optical filter layer including an optical modulation structure on the surface of the image sensor;

step 1520, generating a processor with the function of fully connecting the signals or generating a processor with the functions of fully connecting the signals and performing a second nonlinear activation process;

step 1530, connecting the image sensor and the processor;

In this embodiment, the method further includes a training process of the optical artificial neural intelligence chip, specifically including:

training an optical artificial neural intelligent chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using input training samples and output training samples corresponding to the intelligent processing tasks to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the trained image sensors and the trained processors;

Or, training an optical artificial neural intelligent chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters by using input training samples and output training samples corresponding to the intelligent processing tasks to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the trained image sensors and the trained processors.

It can be understood that when training an optical artificial neural intelligence chip comprising different optical modulation structures, image sensors and processors having different full-link parameters, or training an optical artificial neural intelligence chip comprising different optical modulation structures, image sensors and processors having different full-link parameters and different second-order nonlinear activation parameters, the different optical modulation structures are designed by adopting a computer optical simulation design mode.

In this embodiment, preparing an optical filter layer including a light modulation structure on a surface of a photosensitive region of the image sensor includes:

performing dry etching on the one or more layers of preset materials to obtain an optical filter layer containing an optical modulation structure;

or through carrying out additional dynamic regulation and control on the one or more layers of preset materials, obtaining an optical filter layer containing an optical modulation structure;

or carrying out partition material growth on the one or more layers of preset materials to obtain an optical filter layer containing an optical modulation structure;

When the optical artificial neural network intelligent chip is used for an intelligent processing task of a target object, training the optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using input training samples and output training samples corresponding to the intelligent processing task to obtain the optical modulation structures, the image sensors and the processors which meet training convergence conditions; or training an optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions.

In this embodiment, it should be noted that, as shown in fig. 1, the optical filter layer 1 may be prepared by directly growing one or more layers of dielectric materials on the image sensor 2, etching, depositing a metal material before removing the sacrificial layer for etching, and finally removing the sacrificial layer. By designing the size parameters of the light modulation structure, each unit can have different modulation effects on light with different wavelengths in a target range, and the modulation effects are insensitive to incident angles and polarization. Each cell in the optical filter layer 1 corresponds to one or more pixels on the image sensor 2. 1 is prepared directly on 2.

In this embodiment, it should be noted that, as shown in fig. 14, assuming that the image sensor 2 is a backside-illuminated structure, the optical filter layer 1 may be prepared by directly etching on the silicon detector layer 21 of the backside-illuminated image sensor and then depositing metal.

In addition, it should be noted that the optical modulation structure on the optical filter layer may be dry-etched by performing a pattern of the optical modulation structure on one or more layers of the preset materials, where the dry-etching is to directly remove an unnecessary portion of the one or more layers of the preset materials on the surface of the photosensitive area of the image sensor, so as to obtain the optical filter layer including the optical modulation structure; or one or more layers of preset materials are subjected to imprinting transfer, the imprinting transfer is to prepare a required structure on other substrates through etching, and the structure is transferred to a photosensitive area of the image sensor through PDMS and other materials to obtain an optical filter layer containing an optical modulation structure; or one or more layers of preset materials are subjected to external dynamic regulation and control, wherein the external dynamic regulation and control adopts active materials, and then an external electrode is used for regulating and controlling the light modulation characteristics of corresponding areas by changing voltage to obtain an optical filter layer containing a light modulation structure; or one or more layers of preset materials are printed in a partition mode, and the partition printing is to obtain an optical filter layer containing the light modulation structure by adopting a printing technology in the partition mode; or carrying out partition material growth on one or more layers of preset materials to obtain an optical filter layer containing an optical modulation structure; or quantum dot transfer is carried out on one or more layers of preset materials to obtain the optical filter layer containing the optical modulation structure.

In addition, it should be noted that, because the preparation method provided in this embodiment is the preparation method of the optical artificial neural network intelligent chip in the foregoing embodiment, for details of some principles, structures, and other aspects, reference may be made to the description of the foregoing embodiment, and details of this embodiment are not repeated here.

Based on this, the embodiment of the present invention provides an optical artificial neural network intelligent chip, where an optical filter layer in the intelligent chip corresponds to an input layer and a linear layer of an artificial neural network and a connection weight from the input layer to the linear layer, and a square detection response of an image sensor corresponds to a first nonlinear activation function in a nonlinear layer of the artificial neural network; the processor corresponds to a full connection and output layer of the artificial neural network, or corresponds to a second nonlinear activation function and an output layer in the full connection and nonlinear layer of the artificial neural network, the optical filter layer and the image sensor are used for projecting the spatial spectrum information of the target object into an electric signal, and then the full connection processing or the full connection processing and the second nonlinear activation processing of the electric signal are realized in the processor, so that the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and part or all of the nonlinear activation functions in the prior art can be omitted. The embodiment of the invention strips an input layer, a linear layer and a part or all of nonlinear activation functions in an artificial neural network realized by software in the prior art, and realizes the structures of the input layer, the linear layer and the part or all of nonlinear activation functions in the artificial neural network by using a hardware mode, so that the following intelligent processing of the artificial neural network by using the intelligent chip does not need to perform complex signal processing and algorithm processing corresponding to the input layer, the linear layer and the part or all of nonlinear activation functions, and only needs to perform full connection processing or full connection with an electric signal and secondary nonlinear activation processing by using a processor in the intelligent chip, thereby greatly reducing the power consumption and the time delay during the processing of the artificial neural network, and simultaneously utilizing the image information, the image information and the image information of a target object, Spectral information, angle of incident light, and phase information of the incident light, i.e., the incident light at different points in the target object space carries information, and thus, because the incident light at different points of the target object space carries information which covers the information of the image, the composition, the shape, the three-dimensional depth, the structure and the like of the target object, so that when the identification processing is carried out according to the incident light carrying information at different points of the target object space, can cover multi-dimensional information such as image, composition, shape, three-dimensional depth, structure and the like of the target object, therefore, the accuracy of intelligent processing (such as intelligent identification) can be improved, and therefore, the optical artificial neural network chip provided by the embodiment of the invention not only can realize the effects of low power consumption and low time delay, but also can improve the accuracy of intelligent processing, therefore, the method can be better applied to the intelligent processing fields of intelligent perception, identification and/or decision making and the like.

The environmental problem is the main problem faced by the sustainable development of China, and the environment detection work can realize effective protection and treatment on the environment. The environment detection comprises real-time monitoring of air, water quality, soil and the like, scientific, accurate and effective monitoring is provided for environment management, and reasonable solution strategies are formulated according to the scientific, accurate and effective monitoring. Conventional environmental pollution monitoring is based on wet chemistry techniques and experimental analysis after aspiration sampling. The rapid development of analytical instruments in recent years has served the need for many environmental pollution monitoring, but these instruments are generally limited to single point measurements. In contrast, optical and spectroscopic techniques are ideal tools for monitoring environmental contamination in a large-scale, multi-component detection, time-resolved, continuous, real-time monitoring manner. Therefore, the chip for realizing the real-time environmental pollution detection with large range, high resolution, small volume, low cost, safety and reliability has important significance for environmental management and protection.

Therefore, based on the intelligent chip of the optical artificial neural network introduced in the foregoing embodiment, the present embodiment provides a novel optoelectronic chip for multicomponent environment detection, where the chip includes an input layer and a linear layer of the optical artificial neural network formed by an optical filter layer, and a first nonlinear activation function of the optical artificial neural network formed by an image sensor, and by collecting image information and spectral information of an environment sample to be detected, it is possible to realize fast, accurate, safe and reliable identification and qualitative analysis of environmental pollutants in samples such as air, water, and soil. The contents of the present embodiment will be specifically explained and explained below.

The embodiment of the invention also provides an optical artificial neural network environment-friendly monitoring chip, which is used for an environment-friendly monitoring intelligent processing task and comprises the following steps: an optical filter layer, an image sensor, and a processor; the optical filter layer corresponds to an input layer and a linear layer of the artificial neural network and a connection weight of the input layer to the linear layer, and the square detection response of the image sensor corresponds to a first nonlinear activation function in a nonlinear layer of the artificial neural network; the processor corresponds to a full connection and output layer of the artificial neural network, or corresponds to a second nonlinear activation function and an output layer in the full connection and nonlinear layer of the artificial neural network;

the image sensor carries out first nonlinear activation processing on incident light carrying information corresponding to different position points after being modulated by the optical filter layer through square detection response, converts the incident light carrying information into electric signals corresponding to the different position points, and sends the electric signals corresponding to the different position points to the processor;

The processor performs full connection processing on the electric signals corresponding to the different position points, or performs full connection processing and secondary nonlinear activation processing on the electric signals corresponding to the different position points to obtain an environment-friendly monitoring intelligent processing result;

wherein the environmental monitoring intelligent processing task comprises identification and/or qualitative analysis of environmental pollutants; the environment-friendly monitoring intelligent processing result comprises an identification result of the environmental pollutants and/or a qualitative analysis result of the environmental pollutants.

In this embodiment, the incident light carrying information includes image information and/or various optical spatial information of a target object (such as an environmental pollutant) to be processed by the optical artificial neural network intelligent chip, for example, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light;

the embodiment of the invention realizes a novel optical artificial neural network environment-friendly monitoring chip capable of realizing the function of an artificial neural network, which is used for an environment-friendly monitoring task, wherein the artificial neural network is embedded on a hardware chip, an optical filter layer on the hardware chip is used as an input layer and a linear layer of the artificial neural network, the filtering action of the optical filter layer on the hardware chip on incident light is used as the connection weight from the input layer to the linear layer, the square detection response of an image sensor on the hardware chip is used as a first nonlinear activation function in the nonlinear layer of the artificial neural network, the embodiment of the invention inputs the spatial spectrum information of environmental pollutants into a pre-trained hardware chip, and the hardware chip carries information on the incident light at different points of the environmental pollutants, such as one or more of the image information, the spectrum information, the angle information of the incident light and the phase information of the incident light, so as to obtain the artificial neural network analysis According to the environment-friendly monitoring result, the embodiment of the invention realizes the quick and accurate environment-friendly monitoring with low power consumption, safety and reliability.

It can be understood that, in the optical artificial neural network environmental protection monitoring chip, the hardware structure-optical filter layer thereon corresponds to the input layer and the linear layer of the artificial neural network, and the hardware structure-image sensor thereon corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to another portion of the non-linear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, the optical filter layer comprises an optical modulation structure, the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to the different position points on the surface of the image sensor, and in the embodiment of the invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. Meanwhile, in the embodiment of the invention, the image sensor performs a first nonlinear activation process on incident light carrying information corresponding to different position points after being modulated by the optical filter layer through square detection response, and then converts the incident light carrying information into electric signals corresponding to the different position points, and sends the electric signals corresponding to the different position points to the processor, the processor performs a full connection process on the electric signals corresponding to the different position points, or the processor performs a full connection process and a second nonlinear activation process on the electric signals corresponding to the different position points, so as to obtain an output signal of the artificial neural network, therefore, in the optical artificial neural network environment-friendly monitoring chip, the optical filter layer and the image sensor which are realized in a hardware mode replace or realize related functions of an input layer, a linear layer and a part of nonlinear activation functions in the existing artificial neural network, that is, the embodiment of the present invention strips the input layer, the linear layer and a part or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the structures of the input layer, the linear layer and a part or all of the nonlinear activation functions in the artificial neural network by using a hardware method, therefore, the subsequent complex signal processing and algorithm processing corresponding to the input layer, the linear layer and a part or all of the nonlinear activation functions is not required when the optical artificial neural network environment-friendly monitoring chip is used for carrying out artificial neural network intelligent processing, and only the processor in the optical artificial neural network environment-friendly monitoring chip is required to carry out full-connection processing or full-connection processing with an electric signal and secondary nonlinear activation processing, so that the power consumption and the time delay during artificial neural network processing can be greatly reduced. Therefore, the embodiment of the invention takes the optical filter layer as the input layer and the linear layer of the artificial neural network and the connection weight of the input layer to the linear layer, and takes the square detection response of the image sensor as the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor is used as a full connection and output layer of the artificial neural network, or the processor corresponds to a second nonlinear activation function and an output layer in the full connection and nonlinear activation layers of the artificial neural network, so that the embodiment of the invention can save complex signal processing and algorithm processing corresponding to an input layer, a linear layer and a part of nonlinear activation functions in the prior art, and can greatly reduce the power consumption and the time delay during the processing of the artificial neural network.

Further, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light. Therefore, the embodiment of the invention can utilize one or more of the image information, the spectrum information, the angle of the incident light and the phase information of the incident light of the environmental pollutants, thereby improving the identification accuracy. It should be noted that, in the prior art, only the two-dimensional image information of the environmental pollutants is used when the environmental pollutants are identified, but the accuracy of identification is difficult to be ensured by the two-dimensional image information. Therefore, based on this, in one implementation, the incident light carrying information may include light intensity distribution information and spectrum information, so that when the optical artificial neural network intelligent chip provided in the present application is used to perform an intelligent recognition task, the light intensity distribution information and the spectrum information of an object to be recognized may be simultaneously used, and thus it is visible that since the incident light carrying information covers information of an image, a component, a shape, a three-dimensional depth, a structure, and the like of an environmental pollutant (hereinafter also referred to as a target object), when recognition processing is performed according to the incident light carrying information at different points of a target object space, multi-dimensional information of the image, the component, the shape, the three-dimensional depth, the structure, and the like of the target object may be covered, so that the problem that it is difficult to ensure recognition accuracy, for example, it is difficult to distinguish a real object from a picture, by using the two-dimensional image information of the target object mentioned above may be solved, therefore, the recognition of the object to be recognized can be realized more accurately. In addition, in another implementation manner, the incident light carrying information may further include light intensity distribution information, spectrum information, and angle information of the incident light, so that information such as an image, a component, a shape, a three-dimensional depth, a three-dimensional structure, and the like of the target object can be captured more comprehensively, and thus, the identification of the object to be identified can be achieved more accurately. In addition, in another implementation manner, the incident light carrying information may further include light intensity distribution information, spectrum information, incident light angle information, and incident light phase information, so that information such as an image, a component, a shape, a three-dimensional depth, a three-dimensional structure, and the like of the target object can be captured more comprehensively, and thus, the identification of the object to be identified can be achieved more accurately.

Therefore, the embodiment of the invention can fully utilize the incident light carrying information at different points of the space of the environmental pollutant, and the incident light carrying information at different points of the space of the environmental pollutant covers the image, composition, shape, three-dimensional depth, structure and other information of the environmental pollutant, so that when the identification processing is carried out according to the incident light carrying information at different points of the space of the environmental pollutant, the chip can cover the multi-dimensional information of the image, composition, shape, three-dimensional depth, structure and the like of the environmental pollutant, thereby accurately measuring the information of the environmental pollutant, and in addition, the chip can become an ideal tool for monitoring the environmental pollution due to the advantages of large range, multi-composition detection, good time resolution, continuous real-time monitoring and the like of the spectral information, thereby solving the problem that the current environmental monitoring instrument is only suitable for single-point monitoring, the chip provided by the embodiment can realize large-range, high-resolution, small-volume, low-cost, safe and reliable real-time environmental pollution detection, and has important significance for environmental management and protection.

In this embodiment, a large number of environmental pollutant samples may be collected first, and the weight of the linear layer, i.e., the system function of the optical filter layer, may be obtained through data training, so that the required optical filter layer may be designed in reverse and integrated above the image sensor. In the actual environment detection process, the output of the manufactured optical filter layer is reused, the weight of the electric signal full-connection layer is further trained and optimized, the high-accuracy optical artificial neural network can be realized, and the rapid and accurate identification and qualitative analysis of the environmental pollutant sample are completed. Therefore, the chip actually utilizes the image information, the spectrum information, the angle information of the incident light and the phase information of the incident light at different points of the environment pollutant space at the same time, improves the accuracy and the diversity of the environment pollution detection, partially realizes an artificial neural network on hardware, and improves the speed of the environment pollution detection. In addition, the chip scheme can realize mass production by utilizing the existing CMOS process, and the volume, the power consumption and the cost of the device are reduced.

Further, the micro-nano unit has polarization-independent characteristics.

Further, the micro-nano unit has quadruple rotational symmetry.

The embodiment of the invention also provides environment-friendly monitoring equipment which comprises the optical artificial neural network environment-friendly monitoring chip. The environment-friendly monitoring equipment can be an environment condition detector, a pollutant content analyzer and the like.

connecting the image sensor and the processor;

and the processor performs full connection processing on the electric signals corresponding to the different position points, or performs full connection processing and secondary nonlinear activation processing on the electric signals corresponding to the different position points to obtain an environment-friendly monitoring intelligent processing result.

In this embodiment, it should be noted that the micro-nano modulation structure is directly prepared on the surface of the photosensitive region of the image sensor, a plurality of discrete or continuous micro-nano structures form a unit, and the micro-nano modulation structures at different positions have different spectrum modulation effects on incident light, thereby forming an optical filter layer together. The modulation intensity of the micro-nano modulation structures on different wavelength components of incident light corresponds to the connection intensity (linear layer weight) of the artificial neural network. The optical filter layer is used for weighting an input signal on a frequency spectrum, converting the weighted signal into an electric signal by the image sensor (the part processed by the image sensor is equivalent to a first nonlinear activation function), then fully connecting the electric signals output by the image sensors at different positions by the processor, and realizing the complete optical artificial neural network by the second nonlinear activation function. For the environmental pollution detection, a large number of environmental pollutant samples can be collected firstly, the weight of a linear layer, namely the system function of an optical filter layer, can be obtained through data training, so that the required optical filter layer can be designed reversely, the optical filter layer is integrated above an image sensor, in the actual environmental detection process, the output of the manufactured optical filter layer is reused, the weight of an electric signal full-connection layer is further trained and optimized, a high-accuracy optical artificial neural network can be realized, and the rapid and accurate identification and qualitative analysis of the environmental pollutant samples are completed.

It can be understood that, for environmental pollution detection, the modulation intensity (transmittance) of the modulation structure to different wavelength components of incident light can be obtained by performing optical simulation on the micro-nano modulation structure on a computer, the modulation intensity (transmittance) is used as the connection weight from the input layer to the linear layer of the artificial neural network, a nonlinear activation function is realized in a processor, a large number of environmental pollutant samples are collected in advance and subjected to data training, the required micro-nano modulation structure can be designed and prepared, and the input layer, the linear layer and a part of the nonlinear activation function of the artificial neural network are realized on a chip.

Further, it is understood that, referring to fig. 5, the image sensor 2 may employ a CIS wafer on which the optical filter layer 1 is directly fabricated. The optical filter layer 1 comprises a plurality of repeated modulation units, each modulation unit comprises 4 different continuous non-periodic structure arrays, and the basic units of the non-periodic structure arrays are obtained by collecting a large number of different environmental pollutant samples in the early stage and training and designing artificial neural network dataTypically irregularly shaped structures. Each non-periodic structure array has different wide-spectrum modulation effects on incident light, and the overall size of each modulation unit is 0.5 mu m ² ～40000μm ² . The dielectric material in the optical filter layer 1 is polysilicon, and the thickness is 50 nm-2 μm. As can be understood, the CIS wafer includes a silicon detector layer and a metal wire layer, and the response range is the visible-near infrared band; the CIS wafer was bare and no bayer filter array and microlens array were prepared. Each modulation unit corresponds to a plurality of sensor units on the CIS wafer.

The complete process for detecting pollutants by the environment detection chip comprises the following steps: as shown in fig. 16, a light source under the detection instrument irradiates on the detection sample, and then reflected light is collected by the chip and processed to obtain a recognition result. The optical filter layer and the image sensor can be manufactured by a semiconductor CMOS (complementary metal oxide semiconductor) integration process, monolithic integration is realized at a wafer level, the distance between the sensor and the optical filter layer is favorably reduced, the size of a device is reduced, the packaging cost is reduced, and meanwhile portable detection can be realized.

The environment-friendly monitoring chip of the optical artificial neural network based on the micro-nano modulation structure and the image sensor has the following effects:

A. the artificial neural network part is embedded into an image sensor comprising various optical filter layers, so that safe, reliable, rapid and accurate environmental pollution detection is realized; B. the method has the advantages that samples including but not limited to air, water and soil can be detected, artificial neural network training is introduced to identify pollutants, the detection range is large, the samples are rich, the identification accuracy is high, and the qualitative analysis is accurate; C. the chip can be prepared by one-time chip flow of the CMOS process, so that the failure rate of the device is reduced, the finished product yield of the device is improved, and the cost is reduced. D. The monolithic integration is realized at the wafer level, the distance between the sensor and the optical filter layer can be reduced to the greatest extent, the size of a unit is reduced, and the size and the packaging cost of a device are reduced.

It should be noted that, for a detailed structural description of the optical artificial neural network environmental monitoring chip provided in this embodiment, reference may be made to the description of the optical artificial neural network chip in the foregoing embodiment, and for avoiding redundant description, the description is not repeated here. In addition, for a detailed description of the method for manufacturing the environmental monitoring chip for the optical artificial neural network, reference may also be made to the description of the method for manufacturing the chip for the optical artificial neural network in the foregoing embodiments, and details are not repeated herein.

It can be understood that the fingerprint identification technology is a biometric identification technology, and is widely applied to the fields of smart phone unlocking, access control systems, bank password verification and the like. The main processes of fingerprint identification comprise fingerprint acquisition, fingerprint preprocessing, fingerprint feature extraction and comparison.

The current fingerprint identification task generally depends on a neural network identification model, namely the current fingerprint identification task needs to be processed by a fingerprint imaging algorithm and then transmitted to a computer for subsequent neural network identification model algorithm processing, and the transmission and processing of a large amount of data cause larger power consumption and time delay.

In addition, the collection of the fingerprint image is the key of fingerprint identification, and the modes for acquiring the fingerprint image mainly comprise optical collection, capacitance sensor collection, thermosensitive sensor collection, ultrasonic collection and the like. Wherein, capacitive sensor lays in cell-phone back below collection fingerprint usually, when combining heat sensor, can also realize live body detection, nevertheless can not be used to fingerprint identification under the screen, and this is because the thickness of screen module has restricted capacitive sensor's signal acquisition. The existing technology for identifying fingerprints under a screen mainly comprises two schemes of optical acquisition and ultrasonic acquisition, but the two schemes only acquire the texture image information of fingerprints, so that the accuracy of fingerprint identification is limited, and the size and the power consumption of a device are still large. In addition, some people have fewer fingerprint features and are not easy to identify; fingerprints between relatives have similarity, so that identification errors are easily caused; the fingerprint information left on the surface of the object can be stolen, and the safety is not high. Therefore, there is a need to combine fingerprint information with other information to improve the accuracy and security of identification. In summary, it is of great significance to realize a fast underscreen fingerprint identification device with high accuracy, small volume, low power consumption, safety and reliability.

Therefore, based on the intelligent chip of the optical artificial neural network introduced in the foregoing embodiment, the embodiment provides a novel photoelectric chip for accurately identifying a fingerprint, the chip comprises an input layer and a linear layer of the optical artificial neural network, an image sensor forms a first nonlinear activation function of the optical artificial neural network, and the fingerprint identification can be quickly, accurately, safely and reliably realized by collecting image information and spectral information of a fingerprint to be identified. The contents of the present embodiment will be specifically explained and illustrated below.

In this embodiment, the incident light carrying information includes image information of a target object (fingerprint) to be processed by the optical artificial neural network intelligent chip and/or various optical spatial information, for example, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light;

Therefore, the embodiment of the invention realizes a novel optical artificial neural network fingerprint identification chip capable of realizing the function of an artificial neural network, which is used for a fingerprint identification task, the embodiment of the invention embeds the artificial neural network on a hardware chip, takes an optical filter layer on the hardware chip as an input layer and a linear layer of the artificial neural network, takes the filtering action of the optical filter layer on the hardware chip on incident light as the connection weight from the input layer to the linear layer, takes the square detection response of an image sensor on the hardware chip as a first nonlinear activation function in the nonlinear layer of the artificial neural network, the embodiment of the invention leads the incident light carrying information at different points of a fingerprint space to be incident into the hardware chip which is trained in advance, and carries out artificial neural network analysis on the space spectrum information of the fingerprint through the hardware chip to obtain a fingerprint identification result, it should be noted that the embodiment of the invention realizes fast and accurate fingerprint identification with low power consumption, safety and reliability.

It can be understood that, in the optical artificial neural network fingerprint identification chip, the hardware structure-optical filter layer thereon corresponds to the input layer and the linear layer of the artificial neural network, and the hardware structure-image sensor thereon corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to another portion of the non-linear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, the optical filter layer comprises an optical modulation structure, the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to the different position points on the surface of the image sensor, and in the embodiment of the invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. Meanwhile, in the embodiment of the invention, the image sensor carries out first nonlinear activation processing on incident light carrying information corresponding to different position points after being modulated by the optical filter layer through square detection response, and then converts the incident light carrying information into electric signals corresponding to different position points, and sends the electric signals corresponding to the different position points to the processor, the processor carries out full connection processing on the electric signals corresponding to the different position points, or the processor carries out full connection processing and second nonlinear activation processing on the electric signals corresponding to the different position points to obtain output signals of the artificial neural network, therefore, in the fingerprint identification chip of the optical artificial neural network, the optical filter layer and the image sensor which are realized in a hardware mode replace or realize related functions of an input layer, a linear layer and a part of nonlinear activation functions in the existing artificial neural network, that is, the embodiment of the present invention strips the input layer, the linear layer and a part or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the structures of the input layer, the linear layer and a part or all of the nonlinear activation functions in the artificial neural network by using a hardware method, therefore, the follow-up process of carrying out artificial neural network intelligent processing by using the optical artificial neural network fingerprint identification chip does not need to carry out complex signal processing and algorithm processing corresponding to an input layer, a linear layer and a part or all of nonlinear activation functions, and only the processor in the optical artificial neural network fingerprint identification chip needs to carry out full connection processing or full connection with an electric signal and secondary nonlinear activation processing, so that the power consumption and the time delay of the artificial neural network processing can be greatly reduced. Therefore, the embodiment of the invention can save complex signal processing and algorithm processing corresponding to the input layer, the linear layer and a part of nonlinear activation function in the prior art, thereby greatly reducing the power consumption and the time delay during the artificial neural network processing.

Further, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light. Therefore, the embodiment of the invention can utilize one or more information of the image information, the spectrum information, the angle of the incident light and the phase information of the incident light of the fingerprint, thereby improving the identification accuracy. In this embodiment, in an implementation manner, the incident light carrying information may include light intensity distribution information and spectrum information, so that when the optical artificial neural network intelligent chip provided in the present application is used to perform an intelligent recognition task, the light intensity distribution information and the spectrum information of an object to be recognized may be simultaneously used, and thus it is visible that since the incident light carrying information covers information such as an image, a component, a shape, a three-dimensional depth, and a structure of a fingerprint (hereinafter, may also be referred to as a target object), when recognition processing is performed according to the incident light carrying information at different points in a target object space, multi-dimensional information such as an image, a component, a shape, a three-dimensional depth, and a structure of the target object may be covered, and thus recognition of the object to be recognized may be more accurately achieved. In addition, in another implementation manner, the incident light carrying information may further include light intensity distribution information, spectrum information, and angle information of the incident light, so that information such as an image, a component, a shape, a three-dimensional depth, a three-dimensional structure, and the like of the target object can be captured more comprehensively, and thus, the identification of the object to be identified can be achieved more accurately. In addition, in another implementation manner, the incident light carrying information may further include light intensity distribution information, spectrum information, incident light angle information, and incident light phase information, so that information such as an image, a component, a shape, a three-dimensional depth, a three-dimensional structure, and the like of the target object can be captured more comprehensively, and thus, the identification of the object to be identified can be achieved more accurately.

Therefore, the embodiment of the invention can fully utilize the incident light carrying information at different points of the fingerprint space, and the incident light carrying information at different points of the fingerprint space covers the information of the image, the component, the shape, the three-dimensional depth, the structure and the like of the fingerprint, so that when the identification processing is carried out according to the incident light carrying information at different points of the fingerprint space, the embodiment of the invention can cover the multi-dimensional information of the image, the component, the shape, the three-dimensional depth, the structure and the like of the fingerprint, thereby accurately carrying out the fingerprint identification.

In this embodiment, for fingerprint recognition under the screen, a large number of fingerprints of people can be collected first, and the weight from the input layer to the linear layer, i.e., the system function of the optical filter layer, is obtained through data training, so that the required optical filter layer can be designed in a reverse manner and integrated above the image sensor. In actual use, in the process of inputting a fingerprint by a user, the output of the manufactured optical filter layer is utilized to further train and optimize the weight of the electric signal full-connection layer, so that the optical artificial neural network with high accuracy can be realized, and the fingerprint of the user can be quickly and accurately identified.

It is understood that the specific modulation pattern of the modulation structure on the optical filter layer is designed by training artificial neural network data through collecting fingerprints of a large number of people in the early stage, and is usually an irregular structure, although a regular structure is also possible.

As shown in fig. 17, the complete flow for the underscreen fingerprint identification is: a light source under a mobile phone screen irradiates a finger of a user, reflected light is collected by a chip, and an identification result is obtained after internal processing.

It can be understood that the chip actually utilizes the image information and the spectrum information of the fingerprint at the same time, and the accuracy and the safety of fingerprint identification are improved. Meanwhile, the chip partially realizes an artificial neural network on hardware, and improves the speed of fingerprint identification. In addition, the chip scheme can realize mass production by utilizing the existing CMOS process, and the volume, the power consumption and the cost of the device are reduced.

Furthermore, the light modulation structure in the optical filter layer comprises a unit array consisting of a plurality of micro-nano units, and each micro-nano unit corresponds to one or more pixel points on the image sensor; the structures of the micro-nano units are the same or different.

Furthermore, one or more groups of hollow structures are arranged in a plurality of groups of micro-nano structure arrays contained in the micro-nano unit.

Further, the micro-nano unit has polarization-independent characteristics.

Further, the micro-nano unit has quadruple rotational symmetry.

The embodiment of the invention also provides fingerprint identification equipment which comprises the optical artificial neural network fingerprint identification chip. The fingerprint identification device can be a portable fingerprint identification device or a fingerprint identification device installed at a fixed position.

Connecting the image sensor and the processor;

Or training an optical artificial neural network fingerprint identification chip of the processor, which comprises different optical modulation structures, image sensors and different full-connection parameters and different second nonlinear activation parameters, by using input training samples and output training samples corresponding to the fingerprint identification processing task to obtain the optical modulation structures, the image sensors and the processor which meet the training convergence condition, and taking the optical modulation structures, the image sensors and the processor which meet the training convergence condition as the trained optical modulation structures, the trained image sensors and the trained processor.

It should be noted that the optical artificial neural network screen lower fingerprint identification chip based on the micro-nano modulation structure and the image sensor provided by the embodiment has the following effects: A. the artificial neural network part is embedded into a hardware chip, so that the safe, reliable, rapid and accurate identification of the fingerprint under the screen is realized. B. The preparation of the chip can be completed through one-time chip flow of a CMOS process, the failure rate of a device is reduced, the finished product yield of the device is improved, and the cost is reduced. C. The monolithic integration is realized at the wafer level, the distance between the sensor and the optical filter layer can be reduced to the greatest extent, the size of a unit is reduced, and the size and the packaging cost of a device are reduced.

It should be noted that, for the detailed structural description of the optical artificial neural network fingerprint identification chip provided in this embodiment, reference may be made to the description of the optical artificial neural network chip in the foregoing embodiment, and for avoiding redundant description, the description is not repeated here. In addition, for a detailed description of the method for preparing the optical artificial neural network fingerprint identification chip, reference may also be made to the description of the method for preparing the optical artificial neural network chip in the foregoing embodiment, and details are not repeated here.

It can be understood that the face recognition technology is a biological feature recognition technology, and is widely applied to the fields of access control and attendance systems, criminal investigation systems, electronic commerce and the like. The main process of face recognition comprises the steps of face image acquisition, preprocessing, feature extraction, matching and recognition.

The current face recognition task generally depends on a neural network recognition model, namely the current face recognition task needs to image a human face first and then transmit the human face to a computer for subsequent neural network recognition model algorithm processing, and large power consumption and time delay are caused by transmission and processing of a large amount of data.

In addition, at present, when face recognition is performed, only image information of a face is generally used, however, it is difficult to ensure accuracy of recognition only by using image information of the face, for example, it is difficult to distinguish a real face from a face photograph; even with depth information, it is difficult to distinguish between face models and real faces. Therefore, the method has important significance in mining more face information and realizing high-accuracy, safe and reliable rapid face recognition.

Therefore, based on the intelligent chip of the optical artificial neural network introduced in the foregoing embodiment, the embodiment provides a novel photoelectric chip for accurately recognizing a human face, the chip forms an input layer and a linear layer of the optical artificial neural network by an optical filter layer, forms a first nonlinear activation function of the optical artificial neural network by an image sensor, and can realize rapid, accurate, safe and reliable human face recognition by collecting information carried by incident light at different points of a human face space to be recognized. The contents of the present embodiment will be specifically explained and explained below.

In this embodiment, the incident light carrying information includes image information and/or various optical spatial information of a target object (such as a human face) to be processed by the optical artificial neural network intelligent chip, for example, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

Therefore, the embodiment of the invention realizes a novel optical artificial neural network face recognition chip capable of realizing the function of an artificial neural network, which is used for a face recognition task, the embodiment of the invention embeds the artificial neural network on a hardware chip, an optical filter layer on the hardware chip is used as an input layer and a linear layer of the artificial neural network, the filtering effect of the optical filter layer on incident light on the hardware chip is used as the connection weight from the input layer to the linear layer, the square detection response of an image sensor on the hardware chip is used as a first nonlinear activation function in the nonlinear layer of the artificial neural network, the embodiment of the invention inputs the spatial spectrum information of the face into the hardware chip trained in advance, and the hardware chip carries information on the incident light at different points of the face space, such as one or more of the image information, the spectrum information, the angle information of the incident light and the phase information of the incident light, so as to obtain the artificial neural network analysis The embodiment of the invention realizes the quick and accurate face recognition with low power consumption, safety and reliability.

It can be understood that, in the optical artificial neural network face recognition chip, the hardware structure-optical filter layer thereon corresponds to the input layer and the linear layer of the artificial neural network, and the hardware structure-image sensor thereon corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to another portion of the non-linear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, the optical filter layer comprises an optical modulation structure, the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light entering different position points of the optical modulation structure so as to obtain incident light carrying information corresponding to the different position points on the surface of the image sensor, and in the embodiment of the invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. Meanwhile, in the embodiment of the invention, the image sensor performs a first nonlinear activation process on incident light carrying information corresponding to different position points after being modulated by the optical filter layer through square detection response, and then converts the incident light carrying information into electric signals corresponding to the different position points, and sends the electric signals corresponding to the different position points to the processor, the processor performs a full connection process on the electric signals corresponding to the different position points, or the processor performs a full connection process and a second nonlinear activation process on the electric signals corresponding to the different position points to obtain an output signal of the artificial neural network, so that in the optical artificial neural network face recognition chip, the optical filter layer and the image sensor which are realized in a hardware mode replace or realize related functions of an input layer, a linear layer and a part of nonlinear activation functions in the existing artificial neural network, that is, the embodiment of the present invention strips the input layer, the linear layer and a part or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the structures of the input layer, the linear layer and a part or all of the nonlinear activation functions in the artificial neural network by using a hardware method, therefore, when the optical artificial neural network face recognition chip is used for carrying out artificial neural network intelligent processing subsequently, complex signal processing and algorithm processing corresponding to the input layer, the linear layer and a part or all of nonlinear activation functions do not need to be carried out, and only the processor in the optical artificial neural network face recognition chip is required to carry out full connection processing or full connection with an electric signal and secondary nonlinear activation processing, so that the power consumption and the time delay during artificial neural network processing can be greatly reduced. Therefore, the embodiment of the invention can save complex signal processing and algorithm processing corresponding to the input layer, the linear layer and a part of nonlinear activation function in the prior art, thereby greatly reducing the power consumption and the time delay during the artificial neural network processing.

Further, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light. Therefore, the embodiment of the invention can utilize one or more information of image information, spectrum information, incident light angle and incident light phase information of the human face, thereby improving the identification accuracy. It should be noted that, in the prior art, only two-dimensional image information of a human face is used when the human face is recognized, but the two-dimensional image information is difficult to ensure the accuracy of recognition (for example, a human face picture is used). Therefore, based on this, in one implementation, the incident light carrying information may include light intensity distribution information and spectrum information, so that when the optical artificial neural network intelligent chip provided in the present application is used to perform an intelligent recognition task, the light intensity distribution information and the spectrum information of an object to be recognized may be simultaneously used, and thus it is visible that, since the incident light carrying information covers information of an image, a component, a shape, a three-dimensional depth, a structure, and the like of a human face (hereinafter, may also be referred to as a target object), when recognition processing is performed according to the incident light carrying information at different points in a target object space, multi-dimensional information of the image, the component, the shape, the three-dimensional depth, the structure, and the like of the target object may be covered, so that the mentioned problem that it is difficult to ensure recognition accuracy using two-dimensional image information of the target object, for example, it is difficult to distinguish a real person from a picture, therefore, the identification of the object to be identified can be realized more accurately. In addition, in another implementation manner, the incident light carrying information may further include light intensity distribution information, spectrum information, and angle information of the incident light, so that information such as an image, a component, a shape, a three-dimensional depth, a three-dimensional structure, and the like of the target object can be captured more comprehensively, and thus, the identification of the object to be identified can be achieved more accurately. In addition, in another implementation manner, the incident light carrying information may further include light intensity distribution information, spectrum information, incident light angle information, and incident light phase information, so that information such as an image, a component, a shape, a three-dimensional depth, a three-dimensional structure, and the like of the target object can be captured more comprehensively, and thus, the identification of the object to be identified can be achieved more accurately.

Therefore, the embodiment of the invention can fully utilize the incident light carrying information at different points of the face space, and the incident light carrying information at different points of the face space covers the information of the image, the component, the shape, the three-dimensional depth, the structure and the like of the face, so that when the identification processing is carried out according to the incident light carrying information at different points of the face space, the multi-dimensional information of the image, the component, the shape, the three-dimensional depth, the structure and the like of the face can be covered, and the face identification can be accurately carried out.

the trained optical modulation structure, the image sensor and the processor are the optical modulation structure, the image sensor and the processor which meet the training convergence condition and are obtained by training an optical artificial neural network face recognition chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using input training samples and output training samples corresponding to the face recognition processing task; or the trained optical modulation structure, image sensor and processor are the optical modulation structure, image sensor and processor which meet the training convergence condition and are obtained by training an optical artificial neural network face recognition chip which comprises different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters by using input training samples and output training samples corresponding to the face recognition processing task;

In the embodiment, due to the addition of the spectrum modulation of light, the artificial neural network photoelectric chip which inputs the object image and the spectrum thereof can be realized, and the living body face recognition can be realized quickly, accurately, safely and reliably.

In this embodiment, for face recognition, a large number of people's faces may be collected first, and the weight from the input layer to the linear layer, i.e., the system function of the optical filter layer, is obtained through data training, so that the required optical filter layer may be designed in reverse and integrated above the image sensor. During actual training, a human face sample to be recognized is utilized, the output of the manufactured optical filter layer is utilized, the weight of the electric signal full-connection layer is further trained and optimized, the optical artificial neural network with high accuracy can be realized, and the human face of the user is rapidly and accurately recognized.

It is understood that the specific modulation pattern of the modulation structure on the optical filter layer is designed by training artificial neural network data through collecting human faces of a large number of people in the early stage, and is usually an irregular-shaped structure, although a regular-shaped structure is also possible.

As shown in fig. 18, the complete flow for face recognition is: ambient light or other light sources irradiate the face of the user, then reflected light is collected by the chip, and an identification result is obtained after internal processing.

It can be understood that the chip actually utilizes the image information and the spectrum information of the human face at the same time, improves the accuracy and the safety of human face recognition, and can accurately exclude the human face model especially for the non-living human face model. Meanwhile, the chip partially realizes an artificial neural network on hardware, and improves the speed of fingerprint identification. In addition, the chip scheme can realize mass production by utilizing the existing CMOS process, and the volume, the power consumption and the cost of the device are reduced.

Further, the micro-nano unit comprises a plurality of groups of micro-nano structure arrays, and the structures of the micro-nano structure arrays in each group are the same or different.

The embodiment of the invention also provides face recognition equipment, which comprises the optical artificial neural network face recognition chip. The face recognition device can be a portable face recognition device, and can also be a face recognition device installed at a fixed position. Because the face recognition equipment has the similar beneficial effect with the face recognition chip of the optical artificial neural network, the description is omitted here.

Connecting the image sensor and the processor;

It should be noted that the optical artificial neural network face recognition chip based on the micro-nano modulation structure and the image sensor provided in this embodiment has the following effects: A. the artificial neural network is partially embedded into a hardware chip, so that safe, reliable, rapid and accurate face recognition is realized. B. The chip can be prepared by one-time chip flow of the CMOS process, so that the failure rate of the device is reduced, the finished product yield of the device is improved, and the cost is reduced. C. Monolithic integration is realized at a wafer level, so that the distance between the sensor and the optical filter layer can be reduced to the greatest extent, the size of a unit is reduced, and the volume and the packaging cost of a device are reduced.

It should be noted that, for the detailed structural description of the optical artificial neural network face recognition chip provided in this embodiment, reference may be made to the description of the optical artificial neural network chip in the foregoing embodiment, and for avoiding redundant description, the description is not repeated here. In addition, for a detailed description of the method for preparing the optical artificial neural network face recognition chip, reference may also be made to the description of the method for preparing the optical artificial neural network chip in the foregoing embodiment, and details are not repeated here.

The machine vision technology is a branch technology of artificial intelligence, and the machine replaces human eyes to observe and judge, so that the machine vision technology is widely applied to the fields of industrial production, quality detection, express sorting, unmanned driving and the like. A typical machine vision system includes an imaging system, an image processing system, a communication and IO system, and a linkage. The imaging system is responsible for acquiring image information of a target object, and is the key of a machine vision technology.

In the current machine vision system, an image processing system generally depends on a neural network recognition model, namely, the current machine vision intelligent recognition task needs to be imaged first and then transmitted to a computer for subsequent neural network recognition model algorithm processing, and transmission and processing of a large amount of data cause larger power consumption and time delay.

In addition, the current machine vision technology only utilizes the image information of the object, and the accuracy and reliability in measurement and identification are still to be improved. Therefore, the method has important significance in realizing the enhanced machine vision with higher accuracy and reliability by utilizing the information of other dimensions of the object.

Therefore, based on the optical artificial neural network intelligent chip introduced in the foregoing embodiment, this embodiment provides a novel optoelectronic chip for enhancing machine vision, where the chip includes an input layer and a linear layer of an optical artificial neural network formed by an optical filter layer, and a first-time nonlinear activation function of the optical artificial neural network formed by an image sensor, and by collecting image information and spectral information of an object target to be identified, object target identification can be implemented quickly, accurately, safely and reliably, so as to implement enhanced machine vision with higher accuracy and reliability. The contents of the present embodiment will be specifically explained and explained below.

The embodiment of the invention also provides an optical artificial neural network machine vision enhancement chip, which is used for machine vision intelligent processing tasks and comprises the following steps: an optical filter layer, an image sensor, and a processor; the optical filter layer corresponds to an input layer and a linear layer of the artificial neural network and a connection weight of the input layer to the linear layer, and the square detection response of the image sensor corresponds to a first nonlinear activation function in a nonlinear layer of the artificial neural network; the processor corresponds to the full connection and output layer of the artificial neural network, or the processor corresponds to the second nonlinear activation function and output layer in the full connection and nonlinear layer of the artificial neural network;

In this embodiment, the incident light carrying information includes image information and/or various optical spatial information of a target object (a target object in a machine vision scene) to be processed by the optical artificial neural network intelligent chip, for example, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light;

therefore, the embodiment of the invention realizes a novel optical artificial neural network enhanced machine vision recognition chip capable of realizing the function of an artificial neural network, which is used for target object recognition tasks in various machine vision application scenes, the embodiment of the invention embeds the artificial neural network on a hardware chip, an optical filter layer on the hardware chip is used as an input layer and a linear layer of the artificial neural network, the filtering function of the optical filter layer on the hardware chip on incident light is used as the connection weight from the input layer to the linear layer, the square detection response of an image sensor on the hardware chip is used as a first-time nonlinear activation function in the nonlinear layer of the artificial neural network, the embodiment of the invention enables image information, spectrum information, incident light angle information and incident light phase information at different points of a target object space in the machine vision application scenes to be incident to a pre-trained hardware chip, the hardware chip is used for carrying out artificial neural network analysis on the image information, the spectrum information, the incident light angle information and the incident light phase information at different points of the target object space to obtain the identification result of the target object.

It can be understood that, in the optical artificial neural network enhanced machine vision chip, the hardware structure-optical filter layer thereon corresponds to the input layer and the linear layer of the artificial neural network, and the hardware structure-image sensor thereon corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to another portion of the non-linear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, the optical filter layer comprises an optical modulation structure, the optical modulation structure is used for respectively carrying out different spectrum modulation on incident light carrying information entering different position points of the optical modulation structure so as to obtain the incident light carrying information corresponding to the different position points on the surface of the image sensor, and in the embodiment of the invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. Meanwhile, in the embodiment of the invention, the image sensor performs a first nonlinear activation process on incident light carrying information corresponding to different position points after being modulated by the optical filter layer through square detection response, and then converts the incident light carrying information into electric signals corresponding to the different position points, and sends the electric signals corresponding to the different position points to the processor, the processor performs a full connection process on the electric signals corresponding to the different position points, or the processor performs a full connection process and a second nonlinear activation process on the electric signals corresponding to the different position points to obtain output signals of the artificial neural network, so that in the optical artificial neural network enhanced machine vision chip, the optical filter layer and the image sensor which are realized in a hardware mode replace or realize related functions of an input layer, a linear layer and a part of nonlinear activation functions in the existing artificial neural network, that is, the embodiment of the present invention strips the input layer, the linear layer and a part or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the structures of the input layer, the linear layer and a part or all of the nonlinear activation functions in the artificial neural network by using a hardware method, therefore, when the chip is used for carrying out machine vision intelligent processing based on the artificial neural network in the follow-up process, complex signal processing and algorithm processing corresponding to an input layer, a linear layer and a part or all of nonlinear activation functions are not needed, and only the processor in the optical artificial neural network enhanced machine vision chip is needed to carry out full connection processing or full connection with an electric signal and secondary nonlinear activation processing, so that the power consumption and the time delay during the artificial neural network processing can be greatly reduced. Therefore, the embodiment of the invention can save complex signal processing and algorithm processing corresponding to the input layer, the linear layer and a part of nonlinear activation function in the prior art, thereby greatly reducing the power consumption and the time delay during the artificial neural network processing.

Further, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light. Therefore, the embodiment of the invention can utilize one or more of image information, spectrum information, incident light angle and incident light phase information of the target object in the machine vision scene, thereby improving the identification accuracy. In an implementation manner of this embodiment, the incident light carrying information may include light intensity distribution information and spectrum information, so that when the optical artificial neural network intelligent chip provided by the present application is used to perform an intelligent recognition task, the light intensity distribution information and the spectrum information of an object to be recognized may be simultaneously utilized, and thus it is visible that since the incident light carrying information covers information of an image, a component, a shape, a three-dimensional depth, a structure, and the like of a target object, when the incident light carrying information at different points in a space of the target object is recognized, multi-dimensional information of the image, the component, the shape, the three-dimensional depth, the structure, and the like of the target object may be covered, and thus recognition of the object to be recognized may be more accurately achieved. In addition, in another implementation manner, the incident light carrying information may further include light intensity distribution information, spectral information, and angle information of the incident light, so that information such as an image, a composition, a shape, a three-dimensional depth, and a three-dimensional structure of the target object can be captured more comprehensively, and thus the target object can be identified more accurately. In addition, in another implementation manner, the incident light carrying information may further include light intensity distribution information, spectral information, incident light angle information, and incident light phase information, so that information such as an image, a composition, a shape, a three-dimensional depth, a three-dimensional structure, and the like of the target object can be captured more comprehensively, and thus the target object can be identified more accurately.

Therefore, the embodiment of the invention can fully utilize the incident light carrying information at different points of the target object in the machine vision application scene, and the incident light carrying information at different points of the target object space covers the information of the image, the composition, the shape, the three-dimensional depth, the structure and the like of the target object, so that when the identification processing is carried out according to the incident light carrying information at different points of the target object space, the multi-dimensional information of the image, the composition, the shape, the three-dimensional depth, the structure and the like of the target object can be covered, and the target object can be accurately identified.

In the embodiment, due to the addition of the spectrum modulation of light, the artificial neural network photoelectric chip which inputs the object image and the spectrum thereof can be realized, and the accurate, safe and reliable identification of the machine vision target object can be realized.

In this embodiment, for a specific machine vision application scene, the connection weight from the input layer to the linear layer, that is, the system function of the optical filter layer, can be obtained through data training, so that the required optical filter layer can be reversely designed and integrated above the image sensor, and then the optical artificial neural network enhanced machine vision chip capable of rapidly and accurately identifying and judging in the machine vision application scene can be prepared.

It is understood that the specific modulation pattern of the modulation structure on the optical filter layer is designed by training artificial neural network data by acquiring a large number of target objects in the corresponding machine vision application scene in the early stage, and is usually an irregular-shaped structure, although a regular-shaped structure is also possible.

As shown in fig. 19, the complete flow for the enhanced machine vision application is: the light source irradiates on the detected object, then the reflected light is collected by the chip, the processor carries out algorithm processing, the recognition result can be obtained, and finally the control mechanism carries out corresponding operation.

It can be understood that the chip actually utilizes the image information and the spectrum information of the detected object at the same time, so that the accuracy and the safety of the identification of the detected object are improved, meanwhile, the chip partially realizes an artificial neural network on hardware, and the speed of the identification of the detected object is improved. In addition, the chip scheme can realize mass production by utilizing the existing CMOS process, and the volume, the power consumption and the cost of the device are reduced.

Further, the micro-nano unit has polarization-independent characteristics.

Further, the micro-nano unit has quadruple rotational symmetry.

The embodiment of the invention also provides an enhanced machine vision system which comprises a control mechanism and the optical artificial neural network enhanced machine vision chip. The control mechanism is connected with the optical artificial neural network enhanced machine vision chip and correspondingly controlled according to the recognition result of the artificial neural network enhanced machine vision chip, so that the application target in the machine vision scene is completed. Here, the control mechanism may be a robot arm, an intelligent control button, or the like, and this embodiment is not limited thereto. The control structure can correspondingly control according to the recognition result of the optical artificial neural network enhanced machine vision chip and preset control logic.

The machine vision technology is a branch technology of artificial intelligence, observation and judgment are carried out by replacing human eyes through a machine, and the machine vision technology is widely applied to the fields of industrial production, quality detection, express sorting, unmanned driving and the like. A typical machine vision system includes an imaging system, an image processing system, a communication and IO system, and a linkage. The imaging system is responsible for acquiring image information of a target object, and is the key of a machine vision technology. At present, the machine vision technology only utilizes the image information of an object, and the accuracy and the reliability of measurement and identification are still to be improved. The embodiment comprehensively utilizes object spectrum information, so that the enhanced machine vision system with higher accuracy and reliability can be realized.

Connecting the image sensor and the processor;

training an optical artificial neural network machine vision enhancement chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using input training samples and output training samples corresponding to the machine vision intelligent processing tasks to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions, and taking the optical modulation structures, the image sensors and the processors meeting the training convergence conditions as the trained optical modulation structures, the image sensors and the processors;

For example, in the case of a machine vision system such as unmanned vehicle, it is necessary to automatically identify the type of obstacle in front of the vehicle, so as to perform accurate and rapid automatic control. For another example, for a machine vision system for quality inspection, it is necessary to be able to accurately and quickly identify the quality defects of the target object, so as to ensure the quality of the quality inspection and avoid missing inspection or false inspection. For another example, for a machine vision system of surgical navigation, the requirement on real-time performance is very high, and the problem of real-time performance can be well solved by using the chip provided by the embodiment.

It can be understood that, when designing the structure of the optical filter layer on the chip, a large number of target objects in the corresponding machine vision application scene need to be collected first, and the weight of the linear layer, i.e., the system function of the optical filter layer, is obtained through data training, so that the required optical filter layer can be reversely designed and integrated above the image sensor.

It should be noted that the optical artificial neural network enhanced machine vision chip based on the micro-nano modulation structure and the image sensor provided by the embodiment has the following effects: A. the artificial neural network is partially embedded into a hardware chip, so that the real-time performance and the reliability of the machine vision related application are improved. B. The chip can be prepared by one-time chip flow of the CMOS process, so that the failure rate of the device is reduced, the finished product yield of the device is improved, and the cost is reduced. C. The monolithic integration is realized at the wafer level, the distance between the sensor and the optical filter layer can be reduced to the greatest extent, the size of a unit is reduced, and the size and the packaging cost of a device are reduced.

It should be noted that, for a detailed structural description of the optical artificial neural network enhanced machine vision chip provided in this embodiment, reference may be made to the description of the optical artificial neural network chip in the foregoing embodiment, and for avoiding redundant description, the description is not repeated here. In addition, for a detailed description of the method for preparing the optical artificial neural network enhanced machine vision chip, reference may also be made to the description of the method for preparing the optical artificial neural network chip in the foregoing embodiment, and details are not repeated here.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. An optical artificial neural network intelligent chip, comprising: an optical filter layer, an image sensor, and a processor; the optical filter layer corresponds to an input layer and a linear layer of the artificial neural network and the connection weight of the input layer to the linear layer, and the square detection response of the image sensor corresponds to a first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to a full connection and output layer of the artificial neural network, or corresponds to a second nonlinear activation function and an output layer in the full connection and nonlinear layer of the artificial neural network;

2. The photonic artificial neural network intelligent chip according to claim 1, wherein the incident light carrying information includes at least one of light intensity distribution information, spectral information, angle information of the incident light, and phase information of the incident light.

3. The optical artificial neural network intelligence chip of claim 1, wherein the optical artificial neural network intelligence chip is used for intelligent processing tasks of target objects; the intelligent processing task at least comprises one or more of intelligent perception, intelligent identification and intelligent decision task;

the trained optical modulation structure, the image sensor and the processor are the optical modulation structure, the image sensor and the processor which meet the training convergence condition and are obtained by training an optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters by using input training samples and output training samples corresponding to the intelligent processing task; or the trained optical modulation structure, image sensor and processor are the optical modulation structure, image sensor and processor which meet the training convergence condition and are obtained by training an optical artificial neural network intelligent chip which comprises different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters by using input training samples and output training samples corresponding to the intelligent processing task.

4. The photonic artificial neural network intelligence chip of claim 3, wherein the different photonic modulation structures are designed by using a computer optical simulation design when training a photonic artificial neural network intelligence chip comprising different photonic modulation structures, image sensors and processors with different full-link parameters, or training a photonic artificial neural network intelligence chip comprising different photonic modulation structures, image sensors and processors with different full-link parameters and different second nonlinear activation parameters.

5. The optical artificial neural network intelligent chip according to any one of claims 1 to 4, wherein the optical modulation structure in the optical filter layer comprises a regular structure and/or an irregular structure; and/or the light modulating structures in the optical filter layer comprise discrete structures and/or continuous structures.

6. The photonic artificial neural network intelligent chip according to any one of claims 1 to 4, wherein the optical filter layer has a single-layer structure or a multi-layer structure.

7. The intelligent chip of the optical artificial neural network according to any one of claims 1 to 4, wherein the optical modulation structure in the optical filter layer comprises a unit array consisting of a plurality of micro-nano units, and each micro-nano unit corresponds to one or more pixel points on the image sensor; the structures of the micro-nano units are the same or different.

8. The intelligent chip of an optical artificial neural network according to claim 7, wherein the micro-nano units comprise regular structures and/or irregular structures; and/or the micro-nano unit comprises a discrete structure and/or a continuous structure.

9. The intelligent chip of the optical artificial neural network according to claim 7, wherein the micro-nano units comprise a plurality of groups of micro-nano structure arrays, and the structures of the micro-nano structure arrays in each group are the same or different.

10. The intelligent chip of optical artificial neural network according to claim 9, wherein each group of micro-nano structure array has the function of broadband filtering or narrowband filtering.

11. The intelligent chip of optical artificial neural network according to claim 9, wherein each micro-nano structure array is a periodic structure array or a non-periodic structure array.

12. The intelligent chip of an optical artificial neural network according to claim 9, wherein the micro-nano units comprise one or more groups of micro-nano structure arrays with one or more groups of hollow structures.

13. The intelligent chip of the optical artificial neural network according to claim 9, wherein the micro-nano units have polarization-independent characteristics.

14. The intelligent chip of an optical artificial neural network according to claim 13, wherein the micro-nano units have four-fold rotational symmetry.

15. The optical artificial neural network intelligence chip of claim 1, wherein the optical filter layer is composed of one or more layers;

16. The optical artificial neural network smart chip of claim 15, wherein the semiconductor material comprises one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed according to a preset ratio, and a direct bandgap compound semiconductor material; and/or the nanostructure comprises one or more of a nanodot two-dimensional material, a nanorod two-dimensional material, and a nanowire two-dimensional material.

17. The optical artificial neural network smart chip of claim 1, wherein the thickness of the optical filter layer is 0.1 λ -10 λ, where λ represents a center wavelength of the incident light.

18. The optical artificial neural network intelligent chip of claim 1, wherein the image sensor is any one or more of the following:

the CMOS image sensor comprises a CMOS image sensor CIS, a charge coupled device CCD, a single photon avalanche diode SPAD array and a focal plane photoelectric detector array.

19. A smart device, comprising: the optical artificial neural network intelligence chip of any one of claims 1-18.

20. The smart device of claim 19, wherein the smart device comprises one or more of a smart phone, a smart computer, a smart identification device, a smart sensing device, and a smart decision device.

21. A method for preparing an optical artificial neural network intelligent chip according to any one of claims 1 to 18, comprising:

connecting the image sensor and the processor;

22. The method for manufacturing an optical artificial neural network intelligent chip according to claim 21, wherein manufacturing an optical filter layer including an optical modulation structure on a surface of the image sensor includes:

23. The method for manufacturing an optical artificial neural network intelligent chip according to claim 21, wherein when the optical artificial neural network intelligent chip is used for an intelligent processing task of a target object, the optical artificial neural network intelligent chip including different optical modulation structures, image sensors and processors having different full connection parameters is trained by using input training samples and output training samples corresponding to the intelligent processing task, so as to obtain the optical modulation structures, the image sensors and the processors satisfying a training convergence condition; or training an optical artificial neural network intelligent chip comprising different optical modulation structures, image sensors and processors with different full-connection parameters and different second nonlinear activation parameters to obtain the optical modulation structures, the image sensors and the processors meeting the training convergence conditions.