CN117436541A - Runtime data collection and monitoring in machine learning systems - Google Patents

Abstract

Runtime data collection and monitoring in a machine learning system. A test and measurement system comprising: a test and measurement instrument configured to receive waveform data from a Device Under Test (DUT) on a production line; a machine learning system connected to the test and measurement instrument; and one or more processors configured to execute code that causes the one or more processors to: after tuning the DUT on the production line, collecting an optimal tuning parameter data set from the DUT; determining one or more parameter data sets from the optimal tuning parameter data; loading one or more parameter data sets into the DUT; collecting waveform data for one or more parameter data sets from the DUT as a training data set; training a machine learning system using the training data set; and using a machine learning system to generate an output related to the DUT after training.

Description

Runtime data collection and monitoring in machine learning systems

Cross Reference to Related Applications

The present disclosure claims the benefit of U.S. provisional application No. 63/391,269, entitled "LIVE LISTENER FOR DATA COLLECTION MONITORING", filed on 7.21 of 2022, and U.S. provisional application No. 63/406,644, entitled RUNTIME TRAINING DATA COLLECTION FOR MACHINE-LEARNING-BASED TUNING SYSTEM, filed on 14 of 2022, the disclosures of both of which are incorporated herein by reference in their entirety.

Technical Field

The present disclosure relates to test and measurement systems, and more particularly to machine learning modules and algorithms for use in test and measurement systems.

Background

Machine Learning (ML) algorithms require a large amount of data to train and validate the algorithm/network. The amount of data required is often so large that it is a burden on the customer to collect, and whatever data may be collected from the customer is often not comprehensive enough to train a robust ML algorithm. Furthermore, the physical task of instructing the customer to collect the correct data may lead to frustration and delays. Typically, operators performing data collection draw their body from performing daily tasks to obtain accurate data for outside companies, meaning that there is always pressure to return to normal operation as soon as possible.

Drawings

FIG. 1 illustrates an embodiment of a user interface for a Device Under Test (DUT) tuning software application.

FIG. 2 illustrates a state diagram of an embodiment of a method of collecting data and training a machine learning system.

Fig. 3 illustrates an embodiment of a system for collecting neural network training data during a manufacturing cycle without shutting down the (shutdown) production line.

Fig. 4 shows a flow chart of an embodiment of a method of controlling data collection.

Fig. 5 illustrates an embodiment of a method of a real-time listener tool (live listener tool).

Fig. 6 illustrates an embodiment of a workflow of a real-time listener in a method of collecting machine learning training waveforms for performance measurements.

Fig. 7 illustrates an embodiment of a user interface of a real-time listener tool.

FIG. 8 illustrates an embodiment of a method for retraining a machine learning system using a real-time listener.

Detailed Description

Embodiments herein perform collection of waveform data and metadata for training a deep learning network for tuning and verifying a Device Under Test (DUT). Examples discussed below relate to tuning and verification of an optical transmitter, but are applicable to other types of electronic devices as well. The described embodiment enables collection of waveform data with minimal slowdown of existing production lines and avoids having to shut down the production line completely. When the system has collected enough data to allow training of the machine learning system, embodiments may notify the user so that the machine learning system may predict verification of optimal tuning parameters and performance measurements. The described embodiments also allow the system to revert to a conventional tuning process to collect additional training data or other problems. The embodiments may also collect and calculate statistics and histograms of predicted optimal tuning parameters and validation results to monitor system performance.

Furthermore, embodiments for monitoring data collection may extend beyond machine learning data of light emitters and take the form of a "real-time listener" component for monitoring collection and quality of data. Like machine learning data collection, the real-time listener component passively monitors the collected data, thereby avoiding any delays or shutdowns in the manufacturing system. While the following discussion initially discusses data collection for machine learning purposes, the real-time listener may be directed to other types of data, as will be discussed later.

FIG. 1 illustrates an example of a user interface on a test and measurement instrument within a test and measurement system including a manufacturing system and a machine learning system. The user interface refers to the name OptaML Pro, i.e. the name of the machine learning light emitter tuning software. The term "manufacturing system" as used herein includes manufacturing systems for electronic devices such as light emitters, and may also be referred to as a "production line". The term "machine learning system" as used herein includes many different architectures of machine learning models, including neural networks. The following discussion focuses on embodiments employing a deep learning network, which generally means a network with an input layer, several hidden layers, and an output layer, although other architectures may be used.

As mentioned above, examples for the presentation of embodiments focus on the testing of light emitters. In fig. 1, the system tab menu depicts a high-level view of the entire light tuning system. It is simplified into four modules, including: the user tests automation SW, TX, scope (Scope) and OptaML Pro. The user test automation SW in this example refers to manufacturing system software that controls the user's production line. It controls ovens, light switches, instrumentation and machine learning systems. TX refers to the transmitter to be tuned, while the observer refers to the oscilloscope used to capture the waveform. It should be noted that the test and measurement instrument may be an oscilloscope as in this example, or in other examples it may be a different type of test equipment.

The machine learning system requires a training period during which the customer and possibly an engineer, or system of customers, collect data, format and store the data required to train the neural network to make tuning parameter predictions based on the input waveforms from the transmitters.

In an overview of the system in embodiments involving light emitters, once the neural network is trained, the user is required to automate SW to store three sets of reference parameters in the emitter to be tuned and to collect three waveforms, one at a time. These three waveforms will go into the tensor builder block discussed in fig. 3, which will create an RGB image representation of the pruned (pruned) data within the acquired waveforms. This will be input to the neural network and the optimal set of tuning parameters for the transmitter will be output from the neural network.

The user automation SW will then load the predicted tuning parameters to the transmitter and control the observer to acquire a waveform. The system will then test the waveform using measurements such as TDECQ (transmitter dispersive eye closure four term (transmitter dispersion eye closure quaternary)), OMA (optical modulation amplitude), and others to determine pass or fail. The discussion may refer to this as "authentication".

If the ML prediction fails, meaning that the light emitter operating with the predicted tuning parameters fails, the system makes it easier for the user to run his "conventional" tuning algorithm to attempt tuning. If that fails, the device is rejected.

Fig. 2 shows a state diagram of various states within the system. The state represents a state that occurs on the production line when the user's production line tunes the light emitters in preparation for transporting them. The default state in the state transition diagram represents the state in which the user's own tuning algorithm tunes the transmitter. If the machine learning application is not in use, the system will use this process. The gray state is a different state required to maintain the user tuning, but at the same time goes through several states where data for tuning is collected and set. The dashed ML tuning state is used to replace the user tuning state to nominally speed up the tuning process 75 times. As shown herein, during the time that waveform and metadata tuning data is collected, the manufacturing system still uses the user tuning algorithm. The system logic is designed so that the user automated SW can choose to operate only in this state at any time.

In the optimally tuned array state, where the user tunes and gathers metadata, the system gathers optimal tuning parameters into an array of optical tuning parameter structures each time the user tunes the transmitter.

When the previous state has collected enough optimally tuned samples, the system enters the create reference parameter state. The array of optimal tuning data is averaged to obtain the meanPar reference parameter set. The system derives other sets from meanPar by changing meanPar by a certain increment to obtain the deltaPar1 reference parameter set and another increment (which may be opposite to the first increment (+/-)) to obtain the deltaPar2 reference tuning parameter set. Once these reference parameters have been determined, the system transitions to the next state to collect training waveform data and metadata.

In the user tuning and collecting Wfm and metadata training data state, each transmitter is tuned using the user's tuning algorithm. The system maintains optimal tuning settings when tuning the transmitter. In addition, the manufacturing system loads each of the three reference parameter sets into the transmitter and collects about 5 waveforms for each set. This process slows the overall manufacturing throughput by only about 4% and may take about 6.25 days to complete. This process avoids a complete shut down of the production line.

The ML training state represents the completion of data collection, and wherein a user or system engineer can use the collected data to train points of multiple neural networks in the system in an ML application. Once this training is complete, the system saves the trained model settings and can now use it to predict the optimal tuning parameters and TDECQ measurements.

ML tuning uses a trained model and predicts the optimal tuning parameters for the transmitter. It requires the user test automation SW to load three reference parameter sets into the transmitter and acquire one waveform for each parameter set. It then processes these waveforms through the tensor builder block shown in fig. 3 to be used as input to the neural network. The output of the neural network may be the optimal tuning parameter set when the system is in this time, or the output of the neural network may be TDECQ or other measurement when the system is in this part of the process. As described above, this state is nominally 75 times faster than the user's default state.

The output tuning transmitter status represents the task required to output and process a tuning transmitter that has passed the verification test. When completed, the state transitions back through the transmitter, as shown by the ellipses (such as 10) in the various transmit and return paths.

Discarding failed transmitter states represents the task required to handle a failed transmitter. It is a temporary state and when completed, the flow returns to a state where the transmitter failed.

Fig. 3 shows a more detailed example of the system of the embodiment. FIG. 3 illustrates various principal states of the system shown in FIG. 2 by a conditional block that provides the necessary logic to control state flow transitions. Thus, each condition block is intended to represent not only the conditions that result in the state transition, but also the necessary algorithms and code to test the conditions. Control of these states resides in the user automation SW so that the overall system operates to control the process on the production line.

Thus, the system diagram of FIG. 3 facilitates implementing the state flow diagram shown in FIG. 2. To achieve this, the user-automated SW is required to program the condition blocks in the system flow diagram required to cause a state transition to occur. These condition blocks facilitate the use of only the user's tuning application or the ML tuning application. They also help collect data without stopping the MFG line tuning transmitter. A nominal 4% slowdown was expected to occur when training data was collected. It would nominally take 14 days to collect data and train.

In an embodiment for an optical transmitter, the first state or stage of data collection, and thus the system may create three sets of reference tuning parameter sets. For other embodiments, the system may use only one set of reference tuning parameters, or other numbers of reference tuning parameter sets. In a second state or stage of collecting data, the system loads a reference parameter set to allow waveforms to be collected for use in training. Once the data collection is complete, the next stage involves training the deep learning network. Once trained, the system will move to the ML tuning state as a faster way to tune the transmitter, rather than by using a user default tuning algorithm. As mentioned above, the system includes some type of switch that allows the user to select the ML tuning method or return to their default state.

Fig. 3 shows several components of the overall system and process. The new TX 20 represents each new transmitter to be tuned. The new transmitter receives tuning parameters from different states as inputs. For example, when the system is operating in one of the states running the user's tuning algorithm, it may receive the tuning parameters that are scanned. The transmitter may receive as input a reference parameter when the system is in a training state, or when the system is in an ML tuning state. When the transmitter is in the ML tuning state, it can receive the predicted optimal parameter set that it will use when performing verification.

The optoelectronic converter (O/E) 22 includes optional components that are used only when handling the tuning light emitters.

The scope 24 collects the electrical signals into memory for processing. As mentioned above, oscilloscopes represent only one possible test and measurement instrument that may be used herein. A switch 26 in communication with the instrument/scope 24 directs the flow of the received waveform to the appropriate process in the appropriate state. The switch may take the form of a software switch or a hardware switch.

The gray condition block represents the condition of the state transition and represents the code and algorithm required to determine the condition. The user automates the SW application to set up and control these modules.

Block 28 collects 1 WFM for each of the 3 reference tunes, clearly indicating that three waveforms must be acquired using each of the three reference parameter tuning settings. These tuning parameter sets are loaded into the TX by the user automation application. As described above, three reference sets are suitable for use in the light-tuned emitter example of an embodiment. In other examples, the system may use more or fewer reference parameter sets.

The RGB image tensor builder block 30 receives the three waveforms and processes each waveform into a different color channel of the RGB image. U.S. patent application Ser. No. 17/747,954, "SHORT PATTERN WAVEFORM DATABASE BASED MACHINE LEARNING FOR MEASUREMENT," filed 5/18 a 2022, AND U.S. patent application Ser. No. 17/592,437, "EYE CLASSES SEPARATOR WITH OVERLAY, AND COMPOSITE, AND DYNAMIC EYE-TRIGGER FOR HUMANS AND MACHINE LEARNING," filed 2/3 a 2022, are discussed tensor constructs AND are incorporated herein by reference in their entirety.

Tensor builder 30 may also create monochromatic images from parameter inputs such as noise and temperature and may incorporate bar charts as discussed in U.S. patent application Ser. No. 18/208,562, "SEPARATING NOISE TO INCREASE MACHINE LEARNING PREDICTION ACCURACY IN A TEST AND MEASUREMENT SYSTEM," filed on 6-month 12 2023, which is incorporated herein by reference in its entirety. When predictive tuning parameters are being performed or performance measurements such as TDECQ are made, the resulting image is used as an input to a trained neural network. An array of these images is generated from the training data and used as input during the training process of the deep learning network.

The deep learning network 32 represents one or more deep learning networks that make up an ML system. These are trained to correlate the RGB color image or monochrome image of the waveform with optimal tuning parameters or with measurements.

The user TDECQ and verification block 34 represents verification of the predicted tuning parameter set from the ML system. The system loads the set into the transmitter and obtains a waveform. Performing TDECQ and/or other performance measurements on the waveform determines whether the transmitter tuning passed or failed. The output of this module is a tuned transmitter tuned using ML.

User 1 tuning process plus validation block 36 indicates that in the user tuning default state, the customer tuning algorithm and validation algorithm are used without machine learning. The output of this state is a tuned transmitter using a customer tuning algorithm. Similarly, the user 2 tuning process plus verification block 38 represents using only the customer tuning algorithm and verification algorithm. This uses the same tuning algorithm as the user tunes the default state. One output of this state is a tuned transmitter using a customer tuning algorithm. The second output of this state is the optimal tuning parameters into the array to collect them for later use in training the deep learning network.

When the first stage of training is completed from the state user tuning and collecting the optimal tuning array of metadata as shown in fig. 2, a block determination of three reference tuning 40 is performed. The user must average all the collected tuning parameters in the save array to create the first reference parameter set meanPar. In one embodiment, the system or user adds an increment to a parameter in meanPar to obtain delta1Par. Different increments were added to meanPar to give delta2Par.

In the collection of M groups of 3 Wfm blocks 42 for each reference tuning, the value of M is typically equal to 5. This means that the system must collect 5 sets of three waveforms for each tuned transmitter. Three waveforms are obtained by placing three reference parameter sets into the transmitter and collecting one waveform for each parameter set. This results in a total of 15 waveforms from each transmitter being saved as training data.

In a notification engineer or user execution training block 44, the system provides a notification to a user or system engineer running the system indicating that the system is ready for training. This means that the array of waveform data collection and metadata will be input to the training input of the deep learning network block. The super parameter values required for the command plus training will be sent to the deep learning network. After training, the network will be ready for use on the production line for predicting optimal tuning parameters.

In a histogram and statistics block 46 that creates optimal tuning parameters, a histogram is made for each tuning parameter in the optimal tuning parameter set that each transmitter obtained as it is trained. The data may be used for ML tuning only, it may be used for user tuning, it may be used for a combination of both, or both may be saved as separate data. The data may be available for viewing in the UI of the application to monitor performance. This data may also be used to find trends that will signal to the user that the manufactured component is changing characteristics and that a new set of training data may need to be collected and used to retrain the system.

Determining whether more tuning blocks 48 are needed contains algorithms for analyzing statistics from previous blocks and then determining whether the system needs retraining. If it is determined that more tuning is needed, the block informs the user to reset the active training data collection 50 to send a notification to the user automation SW indicating that the system state must be changed to begin collecting more training data. This will tell the user that they should look at the collected statistics and evaluate what has changed on their MFG line. The user may decide to slow their line down by 4% and begin collecting more training data.

Fig. 4 shows an example of a flow chart illustrating some of the logic required in the state transition diagram shown in fig. 2. There are two parts of this process. Beginning at 90, an oven in which the emitter is to be tested is set to a first temperature. The variable k defines the number of temperature settings. In one embodiment, the number of temperatures is equal to 5, so k=5. Transmitter n or some first group of transmitters undergoes tuning at 94 using a customer's conventional tuning system. Once tuned, the system stores the optimal parameters of the transmitter at 94. Then, at 96, the transmitter count n is incremented. This process continues until, at 98, the transmitter count N equals the total number of transmitters N. The temperature count k is then incremented and the process is repeated at the new temperature, with the number of emitters acting as an inner loop. Once all transmitters are tuned at the current temperature at 100, the system uses the optimal parameters to determine the average parameter set meanPar and the two delta parameter sets deltaPar1 and deltaPar2 at 102. At 104, once all transmitters are tuned for all temperatures, this portion of the process ends. This portion of the process of creating the reference parameter set may take 150 hours, or 6.25 days.

Once the process has completed the development of the reference parameter set, the system may collect waveforms for each parameter set of a set of transmitters to construct a training data set. At 100, the system sets a temperature corresponding to a temperature count k. At 112, the transmitter undergoes conventional tuning, and the system stores the optimal parameters at 114. Beginning at 116, the process loops through loading the transmitter with different reference sets at 116, acquiring waveforms at 118, and then repeating all waveforms that have been acquired for all reference sets at 120. Once all sets are completed for the DUT, the DUT count is incremented at 122 and the waveform is stored at 124. Once all DUTs have been trained at 126, the system sets the temperature to the temperature corresponding to the next count k. At 130, the process is repeated until all DUTs have generated waveforms for all parameter sets at all temperatures. At this point, the system has enough data to train at 132. The process was estimated to take 150 hours, 5 hours, or 6.5 days. During this process, the wire will remain operating at 96% of its capacity.

In general, training the DUT using the user process takes 2700 seconds, or 45 minutes. Once trained, the model took 40 minutes on average.

The above-described embodiments present a novel method for collecting waveforms and metadata for a tuning system that uses the Tektronix pending patent method to provide three reference parameter sets for an optical transmitter and is scalable to other Devices Under Test (DUTs). The embodiment allows the operating production line to be slowed down by only 4% while collecting the necessary data to then trigger a training period after a collection time of about 14 days. Once training is complete, the system also specifies the necessary conditional logic blocks to implement the state flow diagram as shown in fig. 2, and fig. 3 shows the data flow in the system. This gives the user control over the state so that they can choose to tune using only their original tuning algorithm, or they can choose to tune using their original algorithm and at the same time collect data for a deep learning network that trains the machine learning system. After training has been completed, the user may choose to tune using the deep learning network. If the machine learning tuning fails, the system facilitates the last attempt to tune the transmitter using the customer tuning algorithm. The system also includes a block representing a set of tuning statistics and algorithms that look at the statistics and determine if a change in the characteristics of the manufactured device occurred to the extent that a new training period needs to be started.

The various blocks and algorithms discussed above may take the form of executable code running on one or more processors. The processor may reside in a user manufacturing system, a test and measurement instrument such as an oscilloscope, or in a separate computing device containing a machine learning system, distributed between two of the three, or between all three.

As mentioned above, the module of the monitoring and analysis data collection system may be extended beyond the data collection discussed above for the tuning system. The following discussion refers to this as a "real-time listener". The described embodiments alleviate the burden of data collection by automating the process with minimal to no interruption in its existing process. The described embodiments ensure that sufficient data is collected and that the data is of high quality. The real-time monitor module comprises a timing inquiry module, a progress monitor, a statistical analysis module and an automatic feedback module based on statistical analysis.

A core design for such a "real-time listener" is shown below in fig. 5, according to an example embodiment of the present disclosure. The collection objective, timer interval, and desired statistical analysis will be determined based on the desired results of the user. The timer will periodically query the data save location at 140 to obtain the newly arrived data since the last analysis. If the collected data meets the statistical criteria at 142, the loop will continue until a predetermined collection target is reached at 144, and then the user is prompted to stop collecting at 146. If the data does not meet the desired criteria, it will automatically diagnose based on the analysis and prompt the user to update the real-time session settings at 146.

Some non-exclusive examples of the use of such real-time listener modules, according to various embodiments of the present disclosure, include the following. In a first example, collecting a large data set from a real Device Under Test (DUT) is often a prerequisite to building a reliable machine learning algorithm, as discussed in the above embodiments. It is often a great burden on the customer. Customers may have to shut down their production line and they may not know whether the training data they have collected is of high quality. Some embodiments of the present disclosure may address this customer problem. With an example of measuring transmitter dispersive eye closure four Terms (TDECQ) of an optical transceiver or transmitter using machine learning, fig. 6 illustrates a workflow in accordance with some embodiments of the present disclosure. It should be noted that other types of performance measurements for other types of DUTs may also use the system.

The user provides a data save location 150 where the listener can query at 152 on a timer, with each new addition of waveforms, the progress monitor 154 slowly targeting the receipt of N training waveform files. After the goal is reached, the system measures the TDECQ or other performance measure of the random sample subset at 156 and constructs a statistical distribution. Based on the distribution, the listener will prompt the user to collect more data within a particular range of the TDECQ, or to complete if the distribution is satisfactory, 158.

Fig. 7 illustrates an example user interface of a real-time listener module, which in this example is implemented in Matlab (although other program implementations may be used).

The workflow described above may be applied to many other machine learning training data collection processes with limited modifications. This provides a low profile, low disruption blueprint for future ML projects that gathers training data.

According to other embodiments, the above-described capabilities may be extended to longer-term monitoring use cases. Instead of using listeners for specific data collection events, clients may link listeners to their long-term data store or real-time test instrumentation. The data storage or real-time test instrument will be referred to herein as a data source. The listener will passively query at a reduced frequency and provide statistical analysis of the quality metrics of the historical data and/or real-time data. It can generate long-term manufacturing health reports for users/businesses. It may also automatically generate a warning when it detects long-term degradation of any bad lot of product or station, similar to fig. 6. Fig. 8 shows an example of this process, which is similar to fig. 6, except that the timer 152 of fig. 6 becomes a long-term, low frequency time at 160.

In a typical support scenario, and an application/system engineer (AE) or other member requests data to reproduce the problem or confirm that the proposed solution is functioning and valid. It may be difficult to obtain the correct data for both cases because the parties to both may be different from those who find the problem. According to some embodiments, using a real-time listener in conjunction with a similar remote assistance tool will allow engineers to have a deeper understanding of the root cause of the problem. One possibility involves an engineer entering remotely into the customer's system and directing a real-time listener at the affected area. This allows engineers to determine the quality of the data produced. They can then compare historical criteria, simulated expected results, or some other golden reference to diagnose problems more quickly.

According to yet other embodiments of the present disclosure, in accordance with GPDR, EULA, and other regulations, some of the metadata held from the real-time listener may also include the type of device under test, the quality of the data (TDECQ or other metrics), the amount of test data, the high and low of the data set, and other statistical models, as well as the configuration of the settings (test equipment vendor/model/serial number/etc., cable, probe, etc.). This data can be used to generate/enhance predictive interoperability modeling, where clients can be shown known good or bad status, have them set with confidence, or have previously avoided issues with compatibility. For example, if a real-time listener is deployed on two or more production lines performing similar tasks, it would be possible to compare the quality of the configuration and testing for the benefit of the line.

Aspects of the disclosure may operate on specially created hardware, on firmware, digital signal processors, or on specially programmed general-purpose computers, including processors operating according to programmed instructions. The term controller or processor as used herein is intended to include microprocessors, microcomputers, application Specific Integrated Circuits (ASICs), and special purpose hardware controllers. One or more aspects of the present disclosure may be implemented in computer-usable data and computer-executable instructions (such as in one or more program modules) that are executed by one or more computers (including a monitoring module) or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types when executed by a processor in a computer or other device. Computer-executable instructions may be stored on non-transitory computer-readable media such as hard disks, optical disks, removable storage media, solid state memory, random Access Memory (RAM), and the like. As will be appreciated by those skilled in the art, the functionality of the program modules may be combined or distributed as desired in various aspects. Furthermore, the functions may be implemented in whole or in part in firmware or hardware equivalents such as integrated circuits, FPGAs, and the like. Particular data structures may be used to more efficiently implement one or more aspects of the present disclosure, and such data structures are contemplated to be within the scope of computer-executable instructions and computer-usable data described herein.

In some cases, the disclosed aspects may be implemented in hardware, firmware, software, or any combination thereof. The disclosed aspects may also be implemented as instructions carried by or stored on one or more non-transitory computer-readable media, which may be read and executed by one or more processors. Such instructions may be referred to as a computer program product. Computer-readable media as discussed herein means any medium that can be accessed by a computing device. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media.

Computer storage media means any medium that can be used to store computer readable information. By way of example, and not limitation, computer storage media may include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital Video Disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, and any other volatile or non-volatile, removable or non-removable media implemented in any technology. Computer storage media excludes signals themselves and transitory forms of signal transmission.

Communication media means any medium that can be used for communication of computer readable information. By way of example, and not limitation, communication media may include coaxial cables, fiber-optic cables, air, or any other medium suitable for communication of electrical, optical, radio Frequency (RF), infrared, acoustic, or other types of signals.

Furthermore, the written description references specific features. It is to be understood that the disclosure in this specification includes all possible combinations of those particular features. For example, where a particular feature is disclosed in the context of a particular aspect, that feature may also be used in the context of other aspects, insofar as possible.

Furthermore, when reference is made in this application to a method having two or more defined steps or operations, the defined steps or operations may be performed in any order or simultaneously unless the context excludes those possibilities.

The previously described versions of the disclosed subject matter have many advantages that are described or will be apparent to one of ordinary skill. Even so, such advantages or features are not required in all versions of the disclosed apparatus, systems or methods.

All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and/or steps are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Example

Illustrative examples of the disclosed technology are provided below. Embodiments of the technology may include one or more of the examples described below, and any combination thereof.

Example 1 is a test and measurement system, comprising: a test and measurement instrument configured to receive waveform data from a Device Under Test (DUT) on a production line; a machine learning system connected to the test and measurement instrument; and one or more processors configured to execute code that causes the one or more processors to: after tuning the DUT on the production line, collecting an optimal tuning parameter data set from the DUT; determining one or more parameter data sets from the optimal tuning parameter data; loading one or more parameter data sets into the DUT; collecting waveform data from the DUT as a training data set for one or more parameter data sets; training a machine learning system using the training data set; and using a machine learning system to generate an output related to the DUT after training.

Example 2 is the test and measurement system of claim 1, further comprising a switch configured to select between a DUT tuned by the production line, a DUT tuned by the machine learning system, and a DUT that is verified.

Example 3 is the test and measurement system of any of examples 1 or 2, wherein the code that causes the one or more processors to determine the one or more parameter data sets includes code that causes the one or more processors to determine an average parameter set and derive at least one additional parameter data set from the average parameter data set.

Example 4 is the test and measurement system of examples 1-3, wherein the output related to the DUT includes one of a validation measurement or a predicted optimal tuning parameter set.

Example 5 is the test and measurement system of examples 1-4, wherein the one or more processors are further configured to execute code that causes the one or more processors to monitor data collection of waveform data and at least one of the optimal tuning parameter data sets from the DUT.

Example 6 is the test and measurement system of example 5, wherein the code to cause the one or more processors to monitor data collection causes the one or more processors to generate a histogram of each tuning parameter in the optimal tuning parameter set.

Example 7 is the test and measurement system of example 6, wherein the code that causes the one or more processors to generate a histogram for each tuning parameter includes code that causes the one or more processors to analyze the histogram to determine whether the machine learning system requires more training.

Example 8 is the test and measurement system of examples 1-7, wherein the one or more processors are further configured to monitor data collection by executing code that causes the one or more processors to: querying a location at which data is stored; analyzing the latest data stored in the location; and continuing the collection of data when the most recent optimal tuning parameter data meets one or more criteria.

Example 9 is the test and measurement system of example 8, wherein the one or more processors are further configured to execute code that causes the one or more processors to: the reason why the diagnostic data does not meet one or more criteria; and providing the update to the user to update the collection of data.

Example 10 is the test and measurement system of example 9, wherein the one or more processors are further configured to notify the user when the machine learning system requires more training.

Example 11 is the test and measurement system of examples 1-9, wherein the code that causes the one or more processors to collect waveform data from the DUT includes code that causes the one or more processors to notify a user when sufficient waveform data has been collected to allow training of the machine learning system.

Example 12 is the test and measurement system of examples 1-11, wherein the one or more processors reside in a test and measurement instrument, a computing device on a production line, a machine learning system, or any combination thereof.

Example 13 is a method of collecting and using data for a machine learning system, comprising: after tuning a Device Under Test (DUT) on a production line, collecting an optimal tuning parameter data set from the DUT; determining one or more parameter data sets from the optimal tuning parameter data; loading one or more parameter data sets into the DUT; collecting waveform data from the DUT as a training data set for one or more parameter data sets; training a machine learning system using the training data set; and using a machine learning system to generate an output related to the DUT after training.

Example 14 is the method of example 13, wherein determining one or more parameter data sets includes determining an average parameter set and deriving at least one additional parameter data set from the average parameter data set.

Example 15 is the method of any of examples 13 or 14, wherein the output related to the DUT includes one of a validation measurement or a prediction of an optimal tuning parameter set.

Example 16 is the method of claim 13, further comprising monitoring data collection of waveform data and at least one of the optimal tuning parameter data sets from the DUT.

Example 17 is a method of monitoring data collected in a system, comprising: periodically querying a data source in a system for testing a Device Under Test (DUT); performing a statistical analysis on the newly added data at the data source; and generating an output identifying any problems with the data.

Example 18 is the method of example 17, wherein the data source includes at least one of a data storage device and a test and measurement instrument.

Example 19 is the method of example 17 or 18, wherein the data source is a storage device to collect waveform data for use as training data in a machine learning system.

Example 20 is the method of any one of examples 17 to 19, wherein the output identifies at least one of: having enough data to train the machine learning system, recent data showing multiple DUT failures in the test, and more data needed within a particular performance measurement range.

Although specific examples of the invention have been illustrated and described herein for purposes of description, it will be appreciated that various modifications may be made without deviating from the spirit and scope of the invention. Accordingly, the invention should not be limited except as by the appended claims.

Claims (20)

1. A test and measurement system comprising:

a test and measurement instrument configured to receive waveform data from a Device Under Test (DUT) on a production line;

a machine learning system connected to the test and measurement instrument; and

one or more processors configured to execute code that causes the one or more processors to:

after tuning the DUT on the production line, collecting an optimal tuning parameter data set from the DUT;

determining one or more parameter data sets from the optimal tuning parameter data;

loading one or more parameter data sets into the DUT;

collecting waveform data from the DUT as a training data set for one or more parameter data sets;

training a machine learning system using the training data set; and

a machine learning system is used after training to produce output related to the DUT.

2. The test and measurement system of claim 1, further comprising a switch configured to select between DUTs tuned by the production line, DUTs tuned by the machine learning system, and DUTs verified.

3. The test and measurement system of claim 1, wherein the code that causes the one or more processors to determine the one or more parameter data sets includes code that causes the one or more processors to determine an average parameter set and derive at least one additional parameter data set from the average parameter data set.

4. The test and measurement system of claim 1, wherein the output associated with the DUT comprises one of a validation measurement or a predictive optimal tuning parameter set.

5. The test and measurement system of claim 1, wherein the one or more processors are further configured to execute code that causes the one or more processors to monitor data collection of waveform data and at least one of the optimal tuning parameter data sets from the DUT.

6. The test and measurement system of claim 5, wherein the code that causes the one or more processors to monitor data collection causes the one or more processors to generate a histogram of each tuning parameter in the optimal tuning parameter set.

7. The test and measurement system of claim 6, wherein the code that causes the one or more processors to generate a histogram for each tuning parameter includes code that causes the one or more processors to analyze the histogram to determine whether the machine learning system requires more training.

8. The test and measurement system of claim 1, wherein the one or more processors are further configured to monitor data collection by executing code that causes the one or more processors to:

Querying a location at which data is stored;

analyzing the latest data stored in the location; and

when the most recent optimal tuning parameter data meets one or more criteria, the collection of data continues.

9. The test and measurement system of claim 8, wherein the one or more processors are further configured to execute code that causes the one or more processors to:

the reason why the diagnostic data does not meet one or more criteria; and

updates are provided to the user to update the collection of data.

10. The test and measurement system of claim 9, wherein the one or more processors are further configured to notify a user when the machine learning system requires more training.

11. The test and measurement system of claim 1, wherein the code that causes the one or more processors to collect waveform data from the DUT comprises code that causes the one or more processors to notify a user when sufficient waveform data has been collected to allow training of the machine learning system.

12. The test and measurement system of claim 1, wherein the one or more processors reside in a test and measurement instrument, a computing device on a production line, a machine learning system, or any combination thereof.

13. A method of collecting and using data for a machine learning system, comprising:

after tuning a Device Under Test (DUT) on a production line, collecting an optimal tuning parameter data set from the DUT;

determining one or more parameter data sets from the optimal tuning parameter data;

loading one or more parameter data sets into the DUT;

collecting waveform data from the DUT as a training data set for one or more parameter data sets;

training a machine learning system using the training data set; and

a machine learning system is used after training to produce output related to the DUT.

14. The method of claim 13, wherein determining one or more parameter data sets comprises determining an average parameter set and deriving at least one additional parameter data set from the average parameter data set.

15. The method of claim 13, wherein the output related to the DUT comprises one of a validation measurement or a prediction of an optimal tuning parameter set.

16. The method of claim 13, further comprising monitoring data collection of waveform data and at least one of the optimal tuning parameter data sets from the DUT.

17. A method of monitoring data collected in a system, comprising:

Periodically querying a data source in a system for testing a Device Under Test (DUT);

performing a statistical analysis on the newly added data at the data source; and

an output is generated that identifies any problems with the data.

18. The method of claim 17, wherein the data source comprises at least one of a data storage device and a test and measurement instrument.

19. The method of claim 17, wherein the data source is a storage device for collecting waveform data for use as training data in a machine learning system.

20. The method of claim 17, wherein the output identifies at least one of: having enough data to train the machine learning system, recent data showing multiple DUT failures in the test, and more data needed within a particular performance measurement range.