AU620959B2 - Neural network signal processor - Google Patents

Description

620959 COMMONWEALTH OF AUSTRALIA PATENTS ACT 1952 Form COMPLETE SPECIFICATION FOR OFFICE USE Short Title: Int. Cl: Application Number: Lodged: Complete Specification-Lodged: Accepted: Lapsed: SPublished: Priority: Related Art: TO BE COMPLETED BY APPLICANT e C S Name of Applicant: HUGHES AIRCRAFT COMPANY Address of Applicant: 7200 Hughes Terrace, Los Angeles, California 90046-0066, United States of America S Actual Inventor: PATRICK F, CASTELAZ and DWIGHT E.

MILLS

Address for Service: GRIFFITH HACK CO.

71 YORK STREET SYDNEY NSW 2000

AUSTRALIA

Complete Specification for the invention entitled: NEURAL NETWORK SIGNAL PROCESSOR Tho following statement is a full description of this invention, including the best method of performing it known to us:- 3782-JV jT/DV i C_~1 ~P NEURAL NETWORK SIGNAL PROCESSOR 1 BACKGROUND OF THE INVENTION 1. Technical Field This invention relates to feature extraction and pattern recognition devices, and in particular, to a neural network signal 5 processor that can internally develop, or "learn" the algorithms required for identification of features directly from input sensor signals.

2. Discussion 1 10 The ability to recognize patterns is a major step towards the development of artificial systems that are capable of performing perceptual tasks that currently only biological systems can perform. Speech and visual pattern recognition are two areas in which conventional computers are seriously deficient. In an effort to develop artificial systems that can perform these and other tasks, a number of signal processing techniques have been developed to extract features from signals. These techniques typically involve extensive preprocessing. Such preprocessing may require, for example, measuring pulse width, amplitude, rise and 20 fall times, frequency, etc. Once these features are extracted to, they can be matched with stored patterns for classification and identification of the signal. The software required to accomplish these steps is often complex and is time consum,ng to develop.

Moreover, conventional digital signal processors are not able to tolerate certain variations in the input signal, such as changes 2- 0444 0 4o *0 0 #400 4 0400 Oe 4 *0 44 0 0010 0404 1 in orientation of a visual pattern, or differences in speakers, in the case of speech recognition, In recent years it has been realized that conventional Von Neumann comnputers, which operate serially, bear little resemblance to the parallel processing that takes place in biological systems such as the brain. It is not surprising, therefore, that conventional signal processing techniques should fail to adequately perfoini the tasks involved in human perception.

Consequently, new methods based on neural models of the brain are Jhing developed to perform perceptual tasks. These systems are known variously as neural networks, neuxornorphic systems, learning machines, parallel distributed processors, self-organizing systems, or adaptive logic systems. Whatever the name, these models utilize numerous nonlinear coputational elements operating 15 in parallel and arranged in patterns reminiscent of biological neural networks. Each computational element or "neuron" is connected via weights or "synapses" that typically are adapted during training to improve performance. Thus, these systems exhibit self-learning by changing their synaptic weights until the correct output is achieved in response to a particular input.

once trained, neural nets are capable of recognizing a target and producing a desired output even where the input is inccoplete or hidden in background noise. Also, ieural nets exhibit greater robustness, or fault tolerance, than Von Neumann sequential computers because there are many more processing nodes, each with primarily local connections. Damage to a few nodes or links need not impair overall perfo~rmance significantly.

There is a wide vaziety of neural net models utilizing various topologies, neuron characteristics, and training or learning rules. Learning rules specify an internal set of weights and indicate how weights, should be adapted during use, or training, to improve performance. By way of illustration, sane of these neural net models include the Perceptron, described in U. S.

Patent No. 3,287,649 issued to F. Rosenblatt; the Hopfield Net, described in U. S. Patent Nos. 4,660,166 and 4,719,591 issued to i -3- 1 J. Hopfield; the Hanming Net and Kohohonen self-organizing maps, described in R. Lippman, "An Introduction to Computing with Neural Nets", 1EEE ASSP Magazine, April 1987, pages 4-22; and the Generalized Delta Rule for Multilayered Perceptrons, described in Rumelhart, Hinton, and Williams, "Learning Internal Representations by Error Propagation", in D. E. Rumelhart and J.

L. McClelland Parallel Distributed Processing; Explorations in the Microstructure of Cognition. Vol. 1: Foundations. MIT Press (1986).

While each of these models achieves varying degrees of success at the particular perceptual tasks to which it is are best suited, the parallel inputs required by these systems are thought to necessitate special purpose preprocessors for real time hardware implementations. (See the above-mentioned article by R.

Lippman.) For example, in Rosenblatt's Perceptron, S. Patent 3,287,649) each input receives a separate frequency band of an analog audio signal. More recent physiologically based preprocessor algorithms for speech recognition attempt to provide information similar to that available on the auditory nerve.

Thus, while neural nets offer the distinct advantage of self-learning and elimination of software, a certain amount of preprocessing has still been required in prior neural nets. While prior neural nets may require less preprocessing than digital signal, processors, they still share this disadvantage with conventional digital signal processors. Accordingly, it would be desirable to provide a neural network capable of real time signal processing that eliminates, or significantly reduces the Spreprocessing required.

SUMMARY OF TIE INVENTION Pursuant to the present invention, a neural network signal processor (NSP) is provided which does not require preprocessors but instead, can receive and analyze raw sensor signal data. This data may canprise one dimensional or multi-dimensional optical, audio or other types of data. The NSP is a feature extraction and

I

y -4pattern recognition device that can accept raw sensor signals as input and identify target signatures by using features in algorithms it has previously learned by example.

The invention is defined in alternative forms in the claims concluding the specification. The invention can be described in alternative forms and with alternative language but for ease of general understanding, the invention will now be further described in a general illustrative manner.

In one broad form, the invention consists in a neural network having a plurality of neurons adapted to receive .re an input signal and adapted to present an output, said input signal being an external input signal or a training input signal, a plurality of connective synapses providing a weighted coupling between said neurons, said neural network being capable of adapting itself to produce a desired output in response to a training input :signal by changing the value of said weighted coupling, 20 the neural network comprising: a plurality of input neurons adapted to receive the ,input signal; means for directing selected, consecutive portions i •of said input signal directly into said input neurons; means for advancing said input signal so that the entire input signal from beginning to end is directed to each of said input neurons; and means for changing said weighted coupling to produce said desired output during a training procedure each time a portion of a training input signal is directed to said input neurons, whereby after a plurality of said training procedures, said neural network will respond with said desired output to an external input signal that contains some form of said training input signal.

I~n~lU_ -4a- In at least one form, the NSP comprises a layer of input processing units, or "neurons", connected to other layers of similar neurons. The neurons are interconnected through weighted connections, or "synapses", in accordance with the particular neural network model employed. The neurons in the input layer are connected to a sensor or other signal producing means to receive the signal to be analyzed. This signal may contain a target signal that is to be identified, or it may contain a target signal that has been obscured in the raw signal, for example, due to extraneous signals, an incomplete target signal, or alterations in the target signal.

15 In order to "train" the neural net to "recognize", or respond with the correct output in response to a target input, a target, or training input is repeatedly ;r o.fed into the NSP and the net adapts the interconnections 0 between the neurons until it responds in the desired manner. The particular training algorithm used will be o^ one appropriate for the particular neural net model employed. The desired output may be, for example, a 0. binary response where certain output neurons respond when 0 the target signal is present and do not respond when it is absent. It should be noted that the NSP can be j trained to respond to more than one target signal. Since the "knowledge" in the NSP is in the connections, if there are sufficient interconnections, in a given training session, the net may be trained with a number of different target classes. Once trained, the net will respond with the correct output when an input signal containing the target is fed to it, even where the input contains noise or is otherwise obscured.

In use of embodiments of the invention, during training and during identification of an unknown signal, the input signal is fed directly to the input neurons of the NSP. This may be accomplished by moving the input %rY signal across the input neurons 1 in a stepwise fashion. A circuit which employs a storage means, such as a sample and hold circuit, is used to "stop" the signal for each discrete step. Rather than employing a training scheme in which a single or static set of inputs are used to train the net to produce a single output, in the present invention, the net is trained to produce a single output state in response to a changing input comprising successive portions of the input waveform. Thus, a kind of dynamic learning is employed, since a changing set of inputs teaches the net to produce a single output response.

In the preferred embodiment, the training procedure is as follows. Initially, the system is at rest and the weights between the neurons are set to various small preferably random values, I ,Next, the initial portion of the input signal is fed to the first input neuron. Input neurons produce a response that is some function of the input signal. This response is directed to the neurons in successive layers through weighted connections. The signal is propagated through the NSP until an output is produced by the neurons in the output layer. In the "typical" neural net model, the learning algorithm will attempt to minimize the difference between the actual and the desired output by effecting a change in the synaptic weights between the neurons.

Next, the input signal will be advanced one step through the sampling circuit, so that the first output neuron will receive 25 a second portion of the input signal. The signal is again propagated through the net, a new output is produced, and the weights are again adjusted to reduce the difference between the S actual and correct output. In this way, the weights are adjusted after each step of the input signal across the input neurons.

At some point after entering the first input neuron, the input signal will progress to the second and successive input neurons. Thus, after a certain number of steps, the first part of the input signal will be fed to the second input neuron and a later portion will be fed to the first input neuron. At this stage, the signals introduced into the first and second input -6 1 neurons are processed through the net simultaneously. Depending on the length of the input signal and the number of input neurons in the input layer, at a later stage a different consecutive portion of -the input signal may be fed to each of the input neurons sinmultaneously, whereupon each portion of the signal will be processed through the net at -the same time. Once the entire signal has moved through each input neuron in a stepwise manner, the first training cycle is ccorplete. It should he noted that many training cycles are normally needed before the NSP is adequately trained to produce 'the correct response. Additional training cycles may then be employed for other target signals that are desired to be classified by -the NSP. In this way, the NSP will b9 trained to produce a different response for each different II target signal. The number of training cycles required will depend on a number of factors including the type of neural net model, th~e o number of neurons, the complexity of the target signal, etc.

Once all training is complete, an unknown signal of a particular length may be fed to the NSP. The input mechanism may be identical to that employed during training. That is, the signal passes through each neuron in the input layer in a series of discrete steps until the entire input signal passes through each input neuron. However, weights are normally only changed during the training cycles. Alternatively, it [nay be possible to feed the unknown signal continuously, omitting the discrete steps used during training.

If the unknown signal contains some form of the target if, signal, the net will respond with the particular output that.

p icorresponded to that target during training. The neural net will respond even if noise is present or there is an alteration from the pure target signal. The NSP thus employs a structure that Is independent of any explicit signal processing algorithms. it requires no preprocessing, no software, is inherently fault tolerant, and its regular architecture results in a high speed, low complexity, relativel.y low cost implementation.

_i_7 BRIEF DESCRIPTION OF THE DRAWINGS The various advantages of the present invention will become apparent to one skilled in the art by reading the following specifications and by reference to the following drawings in which; FIG. 1 is a block diagram of a conventional digital signal processor; FIG. 2 is a block diagram of the neural net signal processor in accordance with an embodiment of the present invention; FIG. 3 is a graph of a typical sigmoid activation function in accordance with an embodiment of the present invention; FIG. 4 is a diagram of a three layer neural network having three neurons per layer in accordance with an 8 embodiment of the present invention; 8 FIG. 5 is a flowchart showing the steps of the backward error propagation technique for training in accordance with an embodiment of the present invention; and FIG. 6 shows the results of a computer simulation of "the neural network signal processor indicating the identification of the two target signals.

DESCRIPTION OF THE PREFERRED EMBODIMENT Referring to FIG. I, a block diagram of a conventional signal processor 10 is shown. A raw analog signal 12, which may comprise the output of a sensor or other signal producing device, is fed to the signal processor 10. A number of operations then are performed to extract certain features from the signal 12 in a feature extraction stage 14. These steps may involve a measurement of the pulse width, the amplitude, rise and fall time, frequency, etc. The results of the feature extraction process are then analyzed in a pattern recognition stage 16, where stored features are compared to the extracted features and the signal processor searches for a match. If a match is found the processor responds with the correct classification.

I78

J

3782-VV

I

I Such conventional signal processors have a number of drawbacks. These include the requirements that the problem be well understood and that explicit and complex algorithms must be developed. Also, the hardware needed to accomplish conventional signal processing is costly and complex, especially for real time processing. The present invention provides a neural net signal processor (NSP) which requires no software, because the algorithms are developed by the processor itself through training. A block diagrami of the preferred embodiment of the NSP 20 is shown in FIG.

2. The NSP 20 comprises a plurality of rows of individual processors or "neurons" arranged in a configuration of the general class known as a Multilayer Perceptron. Wn a Multilayer Perceptron the neurons are arranged in three or more layers. Each neuron produces a output which is some predetermined function of its input. The first, or input, layer comprises neurons that are o called the input neurons 22, and the neurons in the last layer are called output neurons 24. The neurons 22, 24 may be constructed from a variety of conventional digital or analog devices. For example, op amps may be used for the neu.,ons 22, 24.

One or more inner layers comnprise neurons that are called hidden neurons 26. While only three neurons are shown in each layer in its, FIG. 2, it will be understood that any number of neurons may be Vemployed depending on the particular problem to be solved. Each neuron in each layer is connected to each neuron in each adjacent layer. That is, each input nn~ron 22 is connected to each hidden neuron 26 in the adjacent inner layer. Likewise, each inner neuron 26 is connected to each neuron in the next adjacent layer.

This next layer may comprise additional inner neurons 26, or, as shown in FIG. 2, the next layer may comprise the output neurons 24. It should be noted that in a Perceptron, neurons are not connected to other neurons in the same layer.

E~ach of the connections 27 between the neurons contain weights or "synapses" 28 (only some of the connections 27 and synapses 28 are labeled in FIU. 2 to avoid confusion; however, numerals 27 and 28 are meant 'to include all connections 27 and synapses 28 shown). These synapses 28 may be iinplernen ted with 9- 1 variable resistances, or with amplifiers with variable gains, or with FET connection control devices utilizing capacitors, The synapses 28 serve to reduce or increase the strength of the connection between the neurons. While the connections 27 are shown with single lirnes, it will, be understood that two individual lines may be employed to provide signal transmission in two directions, since this will be required during the training procedure. The value of the connection strength of each synapse 28 may vary from some predetermined maximum value to zero, When the weight is zero there is in effect, no connection between the two neurons.

The process of training the NSP 20 to recognize a particular signal involves adjusting the connection strengths of each synapse 28 in a repetitive fashion until the desired output is produced in response to a particular input. More specifically, during training a raw signal containing a known waveform, or .44. target 30, is fed to the input layer neurons 22. The particular input mechanism, 4 -e e c w'.ith the present 1 a-4e-l will be described in more detail below. This target signal 30 is fed to 1 0 20 each neuron 22 in the input layer and a particular output is too: produced which is a function of the processing by each neuron and the weighting value of each synapse 28. The output of the output neurons 24 is compared with a desired output and the difference between the actual and desired output is computed. B~ased on this difference, an error signal is then produced which is used to 4 adjust the weights in each synapse 28 in a way that reduces the value of the output error.

The above training procedure is repeated until the error signal is reduced to an acceptable level, and the NSP 20 produices the desired output in response to the target input 30. Once trained, a raw signal to be identified or classified 32 is fed to 4 the input neurons 22 In a way similar to the manner in which the training signal 30 was introduced. The signal to be identified 32 may or may not contain the target signal 30 or it may contain a degraded or a noisy version of the target signal 30. It the target Is present in soume form, the NSP 20 will respond with the output that corresponded to that target signal 30 during training. If the target signal is not present, a different response or no response will be produced.

The NSP 20 may be trained to recognize more than one target 30. The number of targets the NbP 20 is capable of recognizing will depend on various factors such as the number of neurons, the number of layers and the complexity of the target signal. The NSP 20 can recognize multiple targets because, as the NSP 20 is trained with successive targets, the effective "path" of each target through the neurons will differ due to the different weights connecting the neurons. However, there is a limit to the number of targets because as more targets are introduced, the subsequent training procedures will alter weights from previous training to partially erase or degrade the NSP's ability to recognize the earlier target.

S in Referring now to FIG. 4, the operation of the NSP *in accordance with the present invention, will be discussed in more detail. The input 22, hidden 26, and 20 output 24 neurons each comprise similar processing units which have one or more inputs and produce a single output signal. In accordance with the preferred embodiment, a S, modified version of the Back Propagation training algorithm described in the above-mentioned article by Rumelhart is employed. This algorithm requires that each neuron produce an output that is continuous differentiable nonlinear or semilineat function of its output. It is preferred that this function, called an activation function, be a sigmoid logistic nonlinearity of the general form e "(Ejiw where y is the output of neuron j in layer i, Ejwjy(yJ) is the sum of the inputs to neuron j from the previous layer, yi0) is the output of a neuron in the previous layer to neuron j, wj is the weight associated with a synapse connecting a neuron in the previous layer to neuron j, and 0i is a bias similar in function to a threshold. The Ad general shape of this sigmoid function y(ij) 11 1 is shown in FIG. 3. The derivative of this function y(ij) with respect to its total input, net ij wijy(ij) +8j is given by 3 y(ij) Y(ij(-y(ij)).

net-, Ij Thus, the requirement that the activation function be differentiable is met, During training, the activation function Y(ij) remains the same for each neuron but the weights of each synapse 28 are modified. Thus, the patterns of connectivity are modified as a function of experience. The weights on each synapse 28 are modified according to A w ii (ik)Y(i) where 6 (ik) is an error signal available to the neuron receiving input along that line, y(ij) is the output of the unit sending 0o activation along that line, and rl is a constant of proportionality 000 o also called the learning rate.

The determination of the error signal is a recursive o 0* 020 process that starts with the output units. First, a target signal is transmitted to the input neurons 22. This will cause a signal to be propagated through the NbP 20 until an output signal is produced. This output is then compared with the output that is desired. For example, a binary output may be desired where, in response to a particular target signal 30, certain ones of the output neurons 24 are "on" and the others are "off". It should be S°o noted that the activation function Y(ij) cannot reach the extreme 0 ad values of one or zero without infinitely large weights, so that as a practical matter where the desired outputs are zero or one, 0 30 values of .1 and .9 can be used as target values. The actual output produced by each output neuron 24 is compared with the o: desired output and the error signal is calculated from this difference. For output units 6'ij (tij-Yij)(Yij ("netij) Fran equation then -12- 6j (ti-Y (Yi l-Yj For hidden neurons 26 there is no specified target so the error signal is determined recursively in terms of the error signals in the neurons to which it directly connects and the weights of those connections. Thu-, for non-output neurons 6 iJ y(ij)(l-yiJ) Z k 6 ikW(jk) From equation 3 it can be seen that the learning rate flwill effect how much the weights are changed each time the error signal is propagated. The larger n the larger the changes in the weights and the faster the learning rate. If, however, the learning rate is made too large the system can oscillate.

Oscillation can be avoided even with large learning rates by using a momentum term a Thus, A w(i,j,k+l)=3 n6 W(ij,k) The constant a determines the effect of past weight changes on the current dirrction of movement in weight space, providing a kind of momentum in weight space that effectively filters out high frequency variations in the weight space.

A summary of the back propagation training algorithm is shown in FIG. 5. First, the weight w and neuron offsets are set to small random values. (Step 34). A portion of the target input signal 30 is then presented to the input neuron and the desired output is specified. (Step 36). After the input signal 30 is propagated through each layer of neurons an output value is 30 eventually calculated for each output neuron 24 based on the the si, sigmoid activation function Y(J) described in equation Next, the actual output is generated (Step 38) and compared to the desired output for each output neuron 24, and the error signalS in equation is computed (Step 40). The error signal is then compared to a preset tolerance (Step 42). If the error is larger 13 4444 0 44 0~ 0 0 4 0 44*0 .4 6 04 .4

Q

0.4 0404 4 0404 1 than the tolerance, the error' signal makes a backward pass through the network and each weight is changed by an amount w as defined by equation (Step 44). The target signal 30 is again presented and the weights adjusted again (Steps 36-44) repeatedly until the error is reduced to an acceptable level. When th,,e error signal is smaller than the preset tolerance (Step 42), the training procedure for that target signal 30 is complete. (Step 46). '1he NSP 20 can then be retrained with a new target signal.

Once training for all the target signals is complete, an unknown signal 32 is then presented to the input neurons 22. (Step 48).

After the signal is propagated through the network, the output n~euron~s will produce an output signal. If the -target signal is present in the input, the NSP 20 will produce the desired response to correctly identify the tarm esina. te 15 In accordance with( the present invention, the particular input mechanism of input signals into input neurons 22 will now be described. Referring the FIG. 4, an input signal, which may comprise the target Inlput 30 or an unk~nown signal 32 to be classified, is directed to an input line 51, The input signal Is 20 an analog signal which may originate from a variety of sources.

For exampie, the input signal may comprise one dimensional information derived from one of a number of signal producing sensors such as infrared, radar, sonar, and speech sensors. or, the signal may comprise two dimensional data for solving such 25 problems as object detection and identification in surveillance, reconnaissance, etc.

In some cases this input signal will comprise a raw signal directly frrxn a sensor, in other cases it may be desirable to perform some type of preprocessing, such as the use of doppler data derived from sensors% In any event, the input data signal is transmitted to the first of a series of samipling circuits 52, 54, 56. It will be appreciated that various types of sampling circuits, such as sample and hold circuits, can be constructed to accomplish the desired function. The purpose of these sampling circuits is to 4,4, 6 00 4, 0 44 64 0 40 0 4444** 4 4 0 4440 4 I 14 1 present the input neurons 22 with a window of sampled signal data that contains as many samples as there are input neurons 22.

Initially, a sample of the leading edge of the input signal is entered into the first sampling circuit 52. This input signal is propagated through the NSP 20 until an output is produced by the output neurons 24. In the next step, the next portion of the input signal is sampled by the first sampling circuit 52, and the portion that was received by sample circuit 52 in the previous step is passed to the next sampling circuit 54. Thus, two signals originating from both sampling circuits 52 and 54 are propagated simultaneously through the NSP 20. The sampling circuits are synchronized by pulses from a clock circuit 58. Later, all of the input neurons 22 are sinultaneously receiving different portions of the input signal and the NSP 20 produces a single output state 15 for each step. Eventually, the entire input signal propagates in a stepwise fashion from right to left through the NSP 20 in FIG.

4. In the training mode, the training algorithm will adjust the weights after each step until the output is brought to a I particular desired state. When not in the training mode, the NSP S 20 20 will not adjust the weights but will produce an output state that corresponds to the input signal 32. Since no adjustment in weights is necessary for analyzing an unknown signal 32, the signal may alternatively be input using smaller steps, or even in a continuous manner by the sampling circuits 52, 54, and 56.

A number of variables such as the signal width and amplitude, the width of acch sampled step, and the number of input 22, hidden 26 and output 24 neurons will vary with the particular type of signal to be analyzed. The results of a software simulation of the NSP 20, is illustrated graphically in FIG. demonstrating the identification of two target patterns by a three layer neural network signal processor It should be noted that beyond solving one and two dimensional problems as mentioned above, the NSP 20 is adaptable to multi-dimensional problems such as predetection data fusion, natural language processing, real time synthetic expert (not 15 i 1 requiring an expert) systems, multi-dimensional optimization classes of problems, and other classical pattern recognition problems. It will be appreciated that the basic components of the NSP 20 may be implemented with conventional analog or digital electrical circuits, as well as with analog VLSI circuitry. Also, optical devices may be used for some or all of the functions of the NSP 20. An optical embodiment has been made feasible due to recent advances in such areas as holographic storage, phase conjugate optics, and wavefront modulation and mixing. In addition, once the NSP 20 has been trained to recognize a particular waveform, an NSP could then be reproduced an unlimited number of times by making an exact copy of the trained NSP having the same but fixed synaptic weight values as the trained NSP 20. In this way, mass production of NSP's 20 is possible without repeating the training process.

From the foregoing description it can be appreciated that the present invention provides a high speed neural network signal processor 20 that is capable of self-learning and can be implemented with noncomplex, low cost components and without 20 software. It isn't as susceptible to damage as conventional signal processors and can perform target identification in a robust manner. Once trained, the NSP 20 can be subsequently re-trainable for whole new classes of targets. Those skilled in the art can appreciate that other advantages can be obtained from 25 the use of this invention and that modifications can be made without departing from the true spirit of the invention after studying the specification, drawings and following claims.

L 0 0 a I 09 a 0 069 09 O* g 0 00 o 0 0 1

Claims (15)

1. A neural network having a plurality of neurons adapted to receive an input signal and adapted to present an output, said input signal being an external input signal or a training input signal, a plurality of connective synapses providing a weighted coupling between said neurons, said neural network being capable of adapting itself to produce a desired output in response to a training input signal by changing the value of said weighted coupling, the neural network comprising: a plurality of inpuc neurons adapted to receive the input signal; means for directing selected, consecutive portions of said input signal directly into said input neurons; S 15 means for advancing said input signal so that the entire input signal from beginning to end is directed to each of said input neurons; and means for changing said weighted coupling to produce said desired output during a training procedure each time 20 a portion of a training input signal is directed to said ::":input neurons, whereby after a plurality of said training i procedures, said neural network will respond with said desired output to an external input signal that contains some form of said training input signal. 0 t0

2. The neural network of claim i, wherein a plurality of said input neurons receive different portions of said input signal simultaneously. 3, The neural network of claim 1 or claim 2, wherein said means for directing further comprises a plurality of sampling circuits, each connected to one input neuron for directing a portion of said input signal into said input neuron. 3782JV 17

4. The neural network of claim 3, wherein said means for advancing said input signal further comprises: means for transferring the output signal of each sampling circuit to the input of the next successive sampling circuit; and timing means for synchronizing the transferring of the output signal from each sampling circuit. The neural network of any one of the preceding claims, wherein said means for changing said weights further comprises: means for computing the difference between said desired output and the actual output during training; and means for minimising the difference between said desired output and the actual output.

6. The neural network of any one of the preceding claims, wherein the network is adapted to receive input toi signals comprising analog waveforms originating from a sensor.

7. The neural network of any one of the preceding 20 claims, wherein the network is arranged to provide the Sdesired output in the form of a binary signal produced by a plurality of said neurons. 8, The neural network of any one of the preceding claims wherein the input neurons provide a layer of input neurons, and said neurons of the network further include at least one layer of inner neurons, and a layer of output neurons to provide said output, wherein said synapses provide a weighted coupling between each neuron and every other neuron in the or each adjacent layer. 3782JV

18- 9. The neural network of claim 8 wherein each of said neurons other than the input neurons produces an output that depends upon an activation function which takes the form, for neuron j, 1 1 e W i)Y (ui) '0) where Y'(ij) is the output of each neuron in the previous layer, wGi) is the weight associated with each synapse connecting the neurons in the previous layer to neuron j, and 0j is a fixed bias. :S 10. The neural network of claim 5 wherein said means for computing the difference between said desired output "and the actual output generates an error signal which is propagated to each neuron, said error signal taking the S 15 form I ij (tij Yj) Y(ij)( y() and wherein said means for minimizing the difference between said desired output and the actual output adjusts weights by an amount A w that is calculated according to A w 5 (ij) cAw(ijk) 11. A neural network for producing a desired output in response to a particular input signal which is either an external input signal or a training input signal, the network comprising: a plurality of neurons each adapted to receive a signal and adapted to produce an output; a plurality of connective synapses providing a weighted coupling between some of said neurons and other of said neurons; s 3782JV 19 said neural network being capable of adapting itself during a training procedure to produce a desired output in response to a training input by changing the strength of said weighted couplings; selected ones of said neurons, designated input neurons, being adapted to receive input signals; means for directing selected consecutive portions of said input signal directly into said input neurons; means for advancing said input signal so that the entire input signal fLom beginning to end is directed to each of said input neurons; and means for changing said weighted couplings to produce said desired output during a training procedure S. while a training input signal advances through said input neurons, whereby after said training procedure said neural network will respond with said desired response to an external input signal containing some form of said training input signal. 12. The neural network of claim 11 wherein a plurality 20 of said input neurons receive different portions of said stil input signal simultaneously. i t 13. The neural network of claim 11 or claim 12 wherein said means for directing further comprises a plurality sampling circuits each connected to one input neuzon for directing a portion of said input signal into said input neuron. 14. The neural network of claim 13 wherein said means for advancing said input signal further comprises: means for transferring the output signal of each sampling circuit to the input of the next successive sampling circuit; and timing means for synchronizing the transferring of the output signal from each sampling circuit. 37820V 7 The neural network of any one of claims 11-13, wherein said means for changing said weights further comprises: means for computing the difference between said desired output and the actual output during training; and means for minimizing the difference between said desired output and the actual output. 16. The neural network of any one of claims 11-15, wherein the network is adapted to receive input signals comprising analog waveforms originating from a sensor. 17. The neural network of any one of claims 11-15 wherein the network is adapted to provide the desired output in the form of a binary signal produced by a plurality of said neurons. 15 18. The neural network of any one of claims 11-15 wherein the input neurons provide a layer of input neurons, and said neurons of the network include at least ,;oao one layer of inner neurons, and a layer of output neurons to provide said desired output, wherein said synapses 04. 20 provide a weighted coupling between each neuron and every other neuron in the or each adjacent layer. 9 4

19. The neural network of claim 18 wherein each of said neurons other than the input neurons produces an output that depends upon an activation function which takes the form, for neuron j, 1 1 e i)y where y'l(j) is the output of each neuron in the previous layer, wjl) is the weight associated with each synapse connecting the neurons in the previous layer to neuron j, and 0j is a fixed bias. f c 3782JV -i -i l 21 The neural network of claim 15 wherein said means for computing the difference between said desired output and the actual output generates an error signal which is propagated to each neuron, said error signal taking the form j (tij yij) y(ij) and wherein said means for minimizing the difference between said desired output and the actual output adjusts weights by an amount A w that is calculated according to A w 5(ij)+ Aw(ijk)

21. A multilayered perceptron for classifying an input waveform comprising: a plurality of input neurons adapted to receive said input waveform and to produce an output signal that is a sigmoid function of said input, waveform; a plurality of inner neurons adapted to receive said output signal from said input neurons and adapted to produce an output signal that is a sigmoid function of 20 said received output signal; S°o a plurality of output neurons adapted to receive o. said output signal from said inner neurons that is a S; sigmoid function of said received output signal; t a plurality of sampling circuits each connected to one input neuron for directing selected, consecutive portions of said input waveform into said input neurons; means for transferring the output of each sampling circuit to the input of the next successive sampling circuit so that the entire input waveform from beginning to end is directed to each of said input neurons in discrete steps; timing means for synchronizing the transferring of the output signal from each sampling circuit; 3782JV -I 22 4444 0* 4 44 4, 4 4 4449 44 o 44 .4 4 44 4 4* 4 4 4 41S004# 4 4, 44 4 44 4 4 4 4, means f or training said perceptron to produce a desired output each time a portion of a training input wavef orm is directed to said input neurons, including a means for computing the difference between said desired output and the actual output during a training procedure and means for minimizing the difference between said desired output and the actual output, whereby after a plurality of said training procedures, said perceptron will respond with said desired output to an external input waveform that contains some form of said t.Laining input waveform.

22. A method for classifying an input signal having a characterist~ic waveform, said method comprising the steps of: 15 receiving said input signal by a network of processing units; sampling simultaneously a plurality of consecutive portions of said input signal; directing said sampled portions of said input signal 20 to a plurality of said processing units; advancing the sampled portions of the input signal through consecutive ones of said plurality of said processing units until the entire input signal is sampled; 25 producing a plurality of intermediate signals by said plurality of said processing units each of which is a function of said sampled portions of the input signal and an associated weighting function; producing an output response that is dependent upon at least one of said intermediate signals; training saidi network by comparing said output produced in response to a known input signal to a desired output and modifying said weighting functions to reduce the difference between the output produced and said 3S desired output; and 37 832 JV -i i 4 re~-^-YF.F1Eirr~-~~. 23 comparing the output produced in response to an unknown input with said desired output, wherein said unknown signal can be classified when said output produced matches said desired output.

23. The method of claim 22 or claim 23 wherein said step of advancing said input is accomplished in discrete steps and said weighting functions are modified after each discrete step during the training step.

24. The method of claim 22 or claim 23 wherein the step of producing a plurality of intermediate signals and the step of producing an output response both further comprise the step of producing signals in accordance with the activation functioin, for neuron j, S 15 1 e i)Y (Uj) ywhere is the output of each neuron in the previous layer, w0j) is the weight associated with each synapse connecting the neurons in the previous layer to neuron j, and 0J is a fixed bias.

25. The method of any one of claim 22-24 wherein said step of modifying said weighting functions comprising generating an error signal which is propagated to each neuron, said error signal taking the form 6 1j (t j YiJ) Y(iu) and changing the weighting functions by an amount A w that is calculated according to A w V 6 aAw(lk)

26. An information processor for classifying an input signal made in accordance with a method comprising: receiving said input signal by a network of processing units; sampling simultaneously a plurality of consecutive portions of said input signal; 3782JV 24 directing said sampled portions of said input signal to a plurality of said processing units; advancing the sampled portions of the input signal through consecutive ones of said plurality of said processing units until the entire input signal is sampled; producing a plurality of intermediate signals by said plurality of said processing units each of which is a function of said sampled portions of the input signal and an associated weighting function; producing an output response that is dependent upon at least one of said intermediate signals; training said network by comparing said output response produced in response to a known input signal to a desired output and modifying said weighting functions to reduce the difference between the output response produced and said output; and comparing the output produced in response to an i o" unknown input with said desired output, wherein said unknown signal can be classified when said output produced matches said desired output.

27. A method for classifying an input signal said method ;comprising the steps of: receiving said input signal by a network of 25 processing units; sampling simultaneously a plurality of consecutive portions of said input signal; directing said sampled portions of said input signal to a plwirality of said processing units; advancing the sampled portions of the input signal through consecutive ones of said plurality of processing units until the entire input signal is sampled; producing a plurality of intermediate signals by said plurality of processing units each of which is a function of said sampled portions of the input signal and an associated weighting function; 3782JV producing an output response that is dependent upon at least one of said intermediate signals; training said network by comparing said output produced in response to a known input signal to a desired output and modifying said weighting functions to reduce the difference between the output produced and said desired output; comparing the output produced in response to an unknown input with said desired output, wherein said unknown signal can be classified when said output produced matches said desired output; and setting weighting functions in a second network to be the same as those weighting functions in a trained network, whereby an unlimited number of trained networks I 15 may be produced from a single trained network.

28. A neural network signal processor substantially as herein described with reference to Figures 2-6 of the accompanying drawings. Dated this 16th day of December 1991 HUGHES AIRCRAFT COMPANY By its Patent Attorney S. GRIFFITH HACK CO i S3782V