US20230334367A1 - Automatic machine learning model generation - Google Patents
Automatic machine learning model generation Download PDFInfo
- Publication number
- US20230334367A1 US20230334367A1 US18/196,963 US202318196963A US2023334367A1 US 20230334367 A1 US20230334367 A1 US 20230334367A1 US 202318196963 A US202318196963 A US 202318196963A US 2023334367 A1 US2023334367 A1 US 2023334367A1
- Authority
- US
- United States
- Prior art keywords
- machine learning
- dataset
- data
- predictive machine
- predictive
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010801 machine learning Methods 0.000 title claims abstract description 153
- 238000000034 method Methods 0.000 claims abstract description 119
- 230000008569 process Effects 0.000 claims abstract description 39
- 238000012549 training Methods 0.000 claims description 24
- 238000011156 evaluation Methods 0.000 claims description 13
- 238000005070 sampling Methods 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 11
- 238000007619 statistical method Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims 3
- 238000004458 analytical method Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 23
- 230000003993 interaction Effects 0.000 description 15
- 238000004891 communication Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 7
- 238000003860 storage Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 239000002245 particle Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012946 outsourcing Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000035899 viability Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
Definitions
- the present disclosure relates generally to database systems and data processing, and more specifically to automatic machine learning model generation.
- a cloud platform (i.e., a computing platform for cloud computing) may be employed by many users to store, manage, and process data using a shared network of remote servers. Users may develop applications on the cloud platform to handle the storage, management, and processing of data. In some cases, the cloud platform may utilize a multi-tenant database system. Users may access the cloud platform using various user devices (e.g., desktop computers, laptops, smartphones, tablets, or other computing systems, etc.).
- various user devices e.g., desktop computers, laptops, smartphones, tablets, or other computing systems, etc.
- the cloud platform may support customer relationship management (CRM) solutions. This may include support for sales, service, marketing, community, analytics, applications, and the Internet of Things.
- CRM customer relationship management
- a user may utilize the cloud platform to help manage contacts of the user. For example, managing contacts of the user may include analyzing data, storing and preparing communications, and tracking opportunities and sales.
- a user of a CRM application may wish to predict values for certain fields of data using a predictive model.
- Such systems may include a user interface with a number of fields to manually enter data, identifiers, or any content associated with the prediction sought by the user.
- the user interface may include fields to manually select certain parameters related to the predictive model, such as a definition of the relationships within the data, a selection of appropriate algorithms to use, tuning parameters of the algorithms, etc.
- the system may require that the data to be used for building the predictive model be manually tailored such that it is suitable for building a model. Without specialized statistical training, knowledge of such parameters may be unknown to a user. The system may require that these fields be filled in before a prediction will be created.
- the user may manually enter data into the input fields that may be incorrect or incomplete, and may result in the system providing an inaccurate prediction or, in some cases, the system may be unable to provide a prediction altogether, given the input.
- FIG. 1 illustrates an example of a system for generating a predictive machine learning model that supports automatic machine learning model generation in accordance with aspects of the present disclosure.
- FIG. 2 illustrates an example of a database system that supports automatic machine learning model generation in accordance with aspects of the present disclosure.
- FIG. 3 illustrates an example of a user interface that supports automatic machine learning model generation in accordance with aspects of the present disclosure.
- FIG. 4 illustrates an example of a process flow diagram performed by a database system that supports automatic machine learning model generation in accordance with aspects of the present disclosure.
- FIGS. 5 - 7 illustrate additional examples of database systems that support automatic machine learning model generation in accordance with aspects of the present disclosure.
- FIGS. 8 through 9 show block diagrams of a device that supports automatic machine learning model generation in accordance with aspects of the present disclosure.
- FIG. 10 illustrates a block diagram of a system including a database server that supports automatic machine learning model generation in accordance with aspects of the present disclosure.
- FIGS. 11 through 15 illustrate methods for automatic machine learning model generation in accordance with aspects of the present disclosure.
- the prediction is usually generated based on several user-defined inputs.
- the accuracy of the prediction may be wide-ranging.
- the accuracy of a prediction may be directly correlated to the quantity and accuracy of the received input.
- a typical user without specialized statistical training e.g., an administrator, a sales manager, etc. may be unable to build accurate predictive models because of the manual selection process required.
- a system may be configured to automatically generate a predictive model by choosing the necessary parameters based on an automatic analysis of the data (e.g., based on metadata associated with the data), thereby reducing or eliminating the need for the user to generate any code or to understand the data science being used to automatically select the parameters.
- a predictive machine learning model may be generated.
- a database server may receive a selection of a data set that may include a plurality of fields.
- the data set may include an object—a representation of a type of structured data.
- the plurality of fields may represent subsets of the structured data (e.g., rows in a table).
- the database server may subsequently receive a selection of a prediction field for use in generating the prediction.
- the database server may automatically generate a plurality of features from the data set and then may automatically remove certain features that cause inaccuracies in the predictive model. This automatic feature generation and selection process may be performed by the database server based on metadata associated with the data (e.g., knowing that a string of characters is actually an email address).
- the database server may then generate a predictive learning model based on the plurality of features and may transmit an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model.
- a predictive machine learning model may be generated by first receiving, at a database server, a selection of a data set that may include a plurality of fields. The database server may subsequently receive a selection of one or more prediction fields for use in generating the prediction. The database server may generate a plurality of features from the data set and may then generate a predictive learning model based on the plurality of features. The database server may then evaluate a statistical relationship (e.g., a correlation or a Cramer's V calculation) between the plurality of predicted values generated by the predictive learning model and, in some examples, remove at least one feature of the subset of features.
- a statistical relationship e.g., a correlation or a Cramer's V calculation
- the evaluation may be based on a correlation between the data set and the prediction field. In other examples, the evaluation may be based on a nominal association, such as Cramer's V. In some examples, the removal of certain features may result in a more-accurate prediction by not considering potentially extraneous relationships in the determination of the predicted values.
- the database server may transmit an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model.
- a predictive machine learning model may be generated by first receiving, at a database server, a selection of a data set that may include a plurality of fields.
- the database server may generate a plurality of features from the data set and may then generate a predictive learning model based on the plurality of features.
- the database server may generate a predictive learning model based on training a plurality of candidate machine learning models.
- the database server may evaluate a plurality of machine learning models based in part on a predictive accuracy of the model.
- the database may then, for example, select the predictive machine learning model based in part on the evaluation. This may result in a more-accurate determination of the predicted values.
- the database server may transmit an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model.
- a predictive machine learning model may be generated by first receiving, at a database server, a selection of a data set that may include a plurality of fields.
- the database server may generate a plurality of features from the data set and may then generate a predictive learning model based on the plurality of features.
- the database server may determine a plurality of scores for the prediction field based in part on generating the model.
- the database server may receive a new record associated with the data set and update the scores for the prediction field based in part on the new record. This process may be done in real time and may lead to updated determinations of the predicted values.
- the database server may transmit an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model.
- the predictive machine learning model may improve one or more business processes or workflows within a company, by making the processes more intelligent or more efficient. For example, employees may need more insights into the future and may need to deflect repetitive tasks that take up their days. However, companies may be hindered by the infrastructure costs, lack of expertise, and the resources required to optimize their workflow with AI.
- the employees who are actually managing and driving business processes may have the power to build and customize AI apps to fit their specific needs.
- the predictive machine learning model may automate the model building and data scoring process, and custom predictive models and bots may be easily embedded directly into workflows. The models may automatically learn and improve as they're used, delivering accurate, personalized recommendations and predictions in the context of business.
- the predictive machine learning model may allow users to create custom AI models on any field or object—custom or standard—to predict business outcomes, such as a customer's likelihood to churn or the lifetime value of an account.
- an admin at a financial services company may build a predictive attrition model to understand which customers are most likely to churn based on any fields related to the account, such as the number of external accounts they link to their checking account and history of customer support calls.
- users may define the prediction, identify the field to build the model on, and select which data to use.
- the attrition score may be based on these signals, and may be directly embedded into a customer account page and create a task, automatically alerting users to high-risk customers, so the user may deliver personalized, one-on-one interactions to reduce attrition risk.
- aspects of the disclosure are initially described in the context of an environment supporting an on-demand database service. Further aspects of the disclosure are described with respect to systems that support automatic machine learning model generation, such as at a database server. Aspects of the disclosure are further illustrated by and described with reference to apparatus diagrams, system diagrams, and flowcharts that relate to automatic machine learning model generation.
- FIG. 1 illustrates an example of a system 100 for cloud computing that supports automatic machine learning model generation in accordance with various aspects of the present disclosure.
- the system 100 includes cloud clients 105 , contacts 110 , cloud platform 115 , and data center 120 .
- Cloud platform 115 may be an example of a public or private cloud network.
- a cloud client 105 may access cloud platform 115 over network connection 135 .
- the network may implement transfer control protocol and internet protocol (TCP/IP), such as the Internet, or may implement other network protocols.
- TCP/IP transfer control protocol and internet protocol
- a cloud client 105 may be an example of a user device, such as a server (e.g., cloud client 105 - a ), a smartphone (e.g., cloud client 105 - b ), or a laptop (e.g., cloud client 105 - c ).
- a cloud client 105 may be a desktop computer, a tablet, a sensor, or another computing device or system capable of generating, analyzing, transmitting, or receiving communications.
- a cloud client 105 may be operated by a user that is part of a business, an enterprise, a non-profit, a startup, or any other organization type.
- a cloud client 105 may interact with multiple contacts 110 .
- the interactions 130 may include communications, opportunities, purchases, sales, or any other interaction between a cloud client 105 and a contact 110 .
- Data may be associated with the interactions 130 .
- a cloud client 105 may access cloud platform 115 to store, manage, and process the data associated with the interactions 130 .
- the cloud client 105 may have an associated security or permission level.
- a cloud client 105 may have access to certain applications, data, and database information within cloud platform 115 based on the associated security or permission level, and may not have access to others.
- Contacts 110 may interact with the cloud client 105 in person or via phone, email, web, text messages, mail, or any other appropriate form of interaction (e.g., interactions 130 - a , 130 - b , 130 - c , and 130 - d ).
- the interaction 130 may be a business-to-business (B2B) interaction or a business-to-consumer (B2C) interaction.
- a contact 110 may also be referred to as a customer, a potential customer, a lead, a client, or some other suitable terminology.
- the contact 110 may be an example of a user device, such as a server (e.g., contact 110 - a ), a laptop (e.g., contact 110 - b ), a smartphone (e.g., contact 110 - c ), or a sensor (e.g., contact 110 - d ).
- the contact 110 may be another computing system.
- the contact 110 may be operated by a user or group of users. The user or group of users may be associated with a business, a manufacturer, or any other appropriate organization.
- Cloud platform 115 may offer an on-demand database service to the cloud client 105 .
- cloud platform 115 may be an example of a multi-tenant database system.
- cloud platform 115 may serve multiple cloud clients 105 with a single instance of software.
- other types of systems may be implemented, including—but not limited to—client-server systems, mobile device systems, and mobile network systems.
- cloud platform 115 may support CRM solutions. This may include support for sales, service, marketing, community, analytics, applications, and the Internet of Things.
- Cloud platform 115 may receive data associated with contact interactions 130 from the cloud client 105 over network connection 135 , and may store and analyze the data.
- cloud platform 115 may receive data directly from an interaction 130 between a contact 110 and the cloud client 105 .
- the cloud client 105 may develop applications to run on cloud platform 115 .
- Cloud platform 115 may be implemented using remote servers.
- the remote servers may be located at one or more data centers 120 .
- Data center 120 may include multiple servers. The multiple servers may be used for data storage, management, and processing. Data center 120 may receive data from cloud platform 115 via connection 140 , or directly from the cloud client 105 or an interaction 130 between a contact 110 and the cloud client 105 . Data center 120 may utilize multiple redundancies for security purposes. In some cases, the data stored at data center 120 may be backed up by copies of the data at a different data center (not pictured).
- Subsystem 125 may include cloud clients 105 , cloud platform 115 , and data center 120 .
- data processing may occur at any of the components of subsystem 125 , or at a combination of these components.
- servers may perform the data processing.
- the servers may be a cloud client 105 or located at data center 120 .
- the cloud clients 105 of subsystem 125 may be examples of one or more computing devices and data center 120 may be an example of or include, as a component, a database server to which a cloud client 105 may attempt to connect with.
- the data center 120 may be configured to automatically build or generate a predictive model based on inputs selected by a user via a could client 105 .
- data center 120 may receive a selection of a data set, which may include a plurality of fields.
- the data set may be received in response to a customer seeking a prediction of an outcome, given a particular data set.
- the data center 120 may receive a selection of a prediction field from the plurality of fields.
- the selection may be made, for example, at a user interface of cloud client 105 .
- the data center 120 may then generate a plurality of features from the data set. In some examples, the features may be generated automatically based on metadata associated with the data set.
- the data center 120 may generate a predictive learning model—for use in generating a prediction-based in part on the plurality of features.
- the data center 120 may then transmit, for example to cloud client 105 —an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model.
- the generation of the predictive model by data center 120 may be facilitated by a user interface at a cloud client 105 (e.g., a wizard).
- FIG. 2 illustrates an example of a system 200 that supports generating a predictive machine learning model in accordance with various aspects of the present disclosure.
- the system 200 may include a computing device 205 , which may be an example of a cloud client 105 as described with reference to FIG. 1 ; a database server 210 , which may be an example of the data center 120 as described with reference to FIG. 1 ; and a database 215 .
- the database server 210 may be an example of components of a data center 120 , a cloud platform 115 , or some combination of these, as described with reference to FIG. 1 .
- the system 200 may also include one or more connections, such as connection 220 and connection 225 .
- the computing device 205 may include display 230 and user interface 235 .
- the database server 210 may include model builder 240 , which may include a plurality of modules—including data set selector 245 , prediction field selector 250 , feature generator 255 , learning model generator 260 , and predicted value transmitter 265 .
- the database server 210 may be a multi-tenant database server. In other examples, the database server 210 may be an example of or include a predictive model platform.
- the computing device 205 may display a user interface 235 via a display 230 .
- the user interface 235 may contain one or more fields to receive user input.
- the fields may receive user input that includes a prediction name, a prediction type, an Application Program Interface (API) name, an object prediction, or a field prediction, among others.
- the fields may be associated with receiving a selection of a data set or receiving a selection of a prediction field.
- the fields may be associated with receiving an indication of one or more objects.
- a user may input each of the prediction name, the prediction type, the API name, the object prediction, or the field prediction.
- the user may input the object prediction or the field prediction by selecting one or more objects or fields to predict from, for example, a populated list of objects or fields.
- database server 210 may receive a selection of a data set.
- the data set may include data stored on database 215 that is associated with a CRM account.
- the data set may include a plurality of fields.
- the database server 210 may receive the data set, for example, at data set selector 245 .
- the data set received by database server 210 may, be structured as one or more objects.
- the data set, or the object may be a representation of a type of structured data.
- the plurality of fields included in the data set may be or may be indicative of specific types or subsets of structured data.
- selecting objects e.g., at data set selector 245
- selecting additional objects may filter the amount of features capable of being generated.
- An example of an object may be “account,” “contact,” or “case.” By selecting multiple objects (e.g., “account” and “contact”), a more accurate listing of features may be generated, than if only one object (e.g., “case”) was selected.
- the database server 210 may receive an indication of one or more additional objects (e.g., a related object). In either example, the database server 210 may receive the selection of the data set from computing device 205 via connection 220 . The selection of the object or data set may be facilitated by the user interface 235 such that a user can click or otherwise select an existing object within the data set.
- user input may allow for a data set to be segmented.
- a user may wish to generate a predictive machine learning model based on a specific set of data.
- database server 210 may receive a selection of one or more fields (e.g., an indication of a specific data set), which may be indicative of specific types or subsets of structured data.
- a user may wish to segment the data based on a field such as “contact opportunities” (e.g., leads) that are “worth more than $1 million.”
- this parameter may be taken into account when generating the predictive machine learning model.
- the data set segmentation may occur at data segmentation field selector 270 .
- database server 210 may receive a selection of a prediction field from the plurality of fields.
- a prediction field may refer to the field of data for which the user wishes the database server 210 to perform a prediction.
- the plurality of fields included in the data set may be or may be indicative of specific types or subsets of structured data.
- an example of an associated field may be “e-mail address.”
- the selection of the prediction field may occur at prediction field selector 250 .
- user input may allow for an indication of example data for use in generating a predictive machine learning model.
- a user may indicate which fields include accurate data, and which fields to predict.
- a predictive machine learning model may be based on scenarios that previously occurred (e.g., using the accurate data). For example, a user may indicate that the fields “contact opportunities” and “worth more than $1 million.” The user may indicate one or more prior contacts that closed with sales of greater than $1 million (e.g., accurate data). The user may also indicate that he or she wishes to use these examples (e.g., prior examples) as example data for use in predicting a lead (e.g., a suitable client). Thus the user may also indicate that he or she wishes to use the “contact opportunities” field to predict the lead.
- the accurate data may be taken into account when generating the predictive machine learning model. This data selection may occur at example data field selector 275 .
- the database server 210 may, in some examples, automatically generate a plurality of features from the data set after receiving the selection of the prediction field. The generation of the plurality of features may occur at feature generator 255 .
- the database server 210 may generate the plurality of features automatically based in part on metadata associated with the data set.
- the metadata may include a characteristic of the data and a data type.
- the type of data may be a string and the characteristic of the data may be that the string is an email address.
- the data type of the data may be a number of integer and the characteristic of the data may be that the number is a phone number.
- the metadata may be stored on and received by the database server 210 , for example, from database 215 via connection 225 .
- the metadata stored in database 215 may be available because the data may be stored in a CRM application that analyzes and understands these characteristics of the data.
- database 215 may be an example of a multi-tenant database.
- the database server 210 may then automatically evaluate and potentially remove certain features that where automatically generated from the predictive model. For example, the database server 210 may evaluate a statistical relationship between the plurality of predicted values for the prediction field and a subset of the plurality of features. The database server 210 may remove at least one feature of the subset of the plurality of features from the predictive machine learning model if the statistical relationship exceeds a threshold that indicates that the feature is unsuitable for inclusion in the model. Stated alternatively, the database server 210 may determine if any of the features are outliers or are too predictive of the predictive value. This determination may be based on a threshold value—for example, whether too high or too low of a correlation between the predicted values and the subset of features exist. In such an instance, for example, the database server 210 may remove the feature of the subset from the predictive learning machine, which may result in fewer data outliers and, subsequently, a more-accurate prediction model.
- a threshold value for example, whether too high or too low of a correlation between the predicted
- the database server 210 may adjust a sampling rate of the data to balance out relative representations of the data. For example, the database server 210 may oversample data that predicts underrepresented predictive value. For example, if a relatively small set of the data set is predictive of one predictive outcome, then this small set of data may be oversampled with respect to other data in the data set such that the predictive model is not biased towards the predictive outcome represented by the relatively larger set of data. This automatic balancing of the data may result in a more accurate predictive model.
- the database server 210 may automatically generate a predictive learning model based in part on the plurality of features. For example, the database server 210 may generate the predictive learning model at learning model generator 260 . In other examples, the database server 210 may generate the predictive learning model based on training a plurality of candidate machine learning models. The plurality of candidate machine learning models may include any number of machine learning algorithms used in predictive model building. The database server 210 may then evaluate the plurality of candidate machine learning models. This evaluation may be based in part on a predictive accuracy of each of the machines. The database server 210 may subsequently select the predictive machine learning model based in part on the evaluation (e.g., based on which model is most accurate, or is otherwise best suited for the selected data set or the desired predictive value).
- the database server 210 may generate scores based on the model (e.g., predictive values for selected field). In some examples, the database server 210 may also generate an explanation as to why a particular score was generated. For example, a score may relate to a risk of attrition and the explanation (e.g., reasoning) for the given score may be that “the individual has opened bank accounts with external companies.” The database server 210 may then transmit an indication of the predicted values for the prediction field based in part on the predictive machine learning model. In some examples, the database server 210 may perform one or more statistical analyses on the predictive machine learning model prior to transmitting the indication of the values. For example, the statistical analyses may aid in a determination of the viability of the model.
- the statistical analyses may aid in a determination of the viability of the model.
- the database server 210 may determine whether the model contains requisite information to generate accurate predictive scores.
- the database server 210 may transmit the plurality of predicted values by predicted value transmitter 265 , by connection 220 , to computing device 205 .
- the score may be determined based in part on selecting the predictive learning model from the plurality of candidate machine learning models.
- the database server 210 may receive a new record associated with the data set. For example, additional data may be received from database 215 .
- the database server 210 may then update the score for the prediction field based in part on the new record. This score may be stored at, for example, the database server 210 or the database 215 .
- the plurality of predicted scores, the plurality of predicted values, or both may be saved to the database server 210 or the database 215 based in part on transmitting the indication of the plurality of predicted values.
- FIG. 3 illustrates an example of a system 300 that supports generating a predictive machine learning model in accordance with various aspects of the present disclosure.
- the system 300 may include a computing device 305 , which may be an example of a computing device 205 as described with reference to FIG. 2 .
- the computing device 305 of system 200 may include a user interface 310 , which may be an example of the user interface 235 as described with reference to FIG. 2 .
- the user interface 310 may include input fields 315 , 320 , 325 , 330 , and 345 , 355 , and 360 which may correspond to input fields for a prediction name, an API name, object prediction, field prediction, a data filter, a data segmentation field selector, and an example data field selector, respectively.
- the user interface 310 may also include display fields 335 and 350 , which may correspond to a confidence prediction and a prediction display, respectively.
- the user interface 310 may also include additional input fields or display fields that are not shown.
- the computing device 305 via the user interface 310 —may receive a selection of a data set and a selection of a prediction field.
- the user interface 310 may be or may be an example of a wizard and may direct a user in a step-by-step input process.
- the user may first input a prediction name 315 , an API name 320 , or both to identify the prediction.
- the user may input a selection of a data set that may include, for example, an object prediction 325 , a field prediction 330 , or both. This may be an example of a user inputting a selection of a data set, a selection of a prediction field from a plurality of fields, or both.
- the user may manually input these fields, or may select input from a pre-populated list.
- a user may be automatically navigated through multiple interfaces, each containing one or more input field.
- each subsequent input field in the wizard may depend on the input received in the previous field. For example, a user may select a data set and, at a subsequent stage, may select a prediction field from a pre-populated list based on the data set selected.
- a user may input a prediction save location 340 and may also choose to filter 345 the data to be received. For example, a user may select a particular data set, a particular location (e.g., local to computing device 305 ) or a database, or some other location to save the prediction to. In the instance that the prediction is saved to a particular data set, future predictions using the same data set may be improved. Stated alternatively, the predictive machine learning model may utilize the prediction or data associated with the prediction to improve its accuracy in generating a future prediction.
- the input may be received at a database server (e.g., database server 210 as described with reference to FIG. 2 ).
- the database server may, for example, generate a plurality of features from the data set.
- the database server may generate the plurality of features automatically based in part on metadata associated with the data set.
- the database server may generate a predictive learning model based in part on the plurality of features.
- the database sever may then transmit an indication of a plurality of predicted values to the computing device 305 .
- the indication may be displayed at prediction display 350 .
- the database server may transmit a confidence prediction 335 , which may indicate to a user a confidence level associated with the prediction.
- FIG. 4 illustrates an example of a process flow 400 for generating a predictive machine learning model in accordance with various aspects of the present disclosure.
- the steps of process flow 400 may be performed by a database server or components of the database server, such as a predictive model platform.
- the process flow 400 may include receiving a data set 405 , receiving a selection of a prediction field 410 , generating a plurality of features 415 , generating a learning model 420 , and transmitting predicted values 425 .
- process flow 400 may also include receiving a selection of a plurality of fields 430 , removing a subset of data 435 , evaluation 440 , removing one or more features 445 , training learning models 450 , evaluating learning models 455 , selecting learning models 460 , determining scores 465 , and saving the determined scores 470 .
- a database server may receive a selection of a data set 405 .
- the data set may be organized as an object, and may include an indication of one or more fields.
- the database server may receive a selection of a prediction field 410 from the plurality of fields.
- the selection of a prediction field 410 may be, for example, based on the selected data set.
- the prediction field 410 may refer to the field in the data set for which the user is building the predictive model.
- the database server may receive a selection of fields 430 that the user selects to be included in the model building process.
- the user may select all available fields for the object or may select a subset of the available fields. For example, if the user wishes to not include certain data from the prediction (e.g., not include gender data in the prediction of a credit score), the user may select just a subset of the plurality of fields 430 . However, as described with reference to FIG. 2 , even if the user selects fields that eventually turn out to be causing inaccuracies in the predictive model, the database system may determine to not use those fields in the process of building the model.
- the database server may balance the training data by removing a subset of the data 435 or by adjusting a sampling rate of the data. In some examples, this step may be performed after the feature generation and removal steps discussed below. In other examples, this step may be performed before generating a plurality of features 415 , as currently depicted in FIG. 4 .
- the data within certain fields e.g., prediction fields
- the data may cause an overrepresentation in the predictive nature of the model. As such, the data may be over-sampled or under-sampled with respect to other portions of the data to yield a more balanced ratio of positive and negative examples. This process may be referred to as balancing the data. In general, balancing the data may include adjusting a sampling rate of certain features or sets of data.
- the database server may generate a plurality of features 415 , such as a plurality of relationships, functions, groupings or other associations that can be formed based on the data set. Such a procedure may be referred to as feature selection and engineering.
- the generation of the plurality of features 415 may be an example of generating a plurality of features from the data set.
- the database server may generate the plurality of features automatically based in part on metadata associated with the data set. For example, the database server may recognize that a field of string data is associated with email addresses, and generate one or more features associated with the email addresses.
- a richer feature set may be generated than if only the raw data were provided. For example, the database server may be able to determine if the email addresses are valid or if the name in the email addresses match a name in some other field of the data set.
- the database server may evaluate the plurality of generated features and may remove some features from inclusion in the predicted model. Such a process may be referred to as sanity checking the feature generation.
- the database server may evaluate a correlation 440 (or some other statistical relationship) between the plurality of predicted values (e.g., the prediction(s)) for the prediction field and a subset of the plurality of features. Upon conducting the evaluation 440 , the database server may remove at least one feature 445 of the subset of the plurality of features from the predictive machine learning model if the correlation exceeds a threshold.
- the threshold may be an indication of the accuracy of the feature (or how the feature impacts the accuracy of the model) or the threshold may be indicative of how predictive the feature for the prediction field (e.g., if the feature is too predictive, the feature may bias the model towards the one feature).
- the threshold may be an indication of the accuracy of the feature (or how the feature impacts the accuracy of the model) or the threshold may be indicative of how predictive the feature for the prediction field (e.g., if the feature is too predictive, the feature may bias the model towards the one feature).
- the database server may then generate a predictive learning model 420 based in part on the plurality of features.
- the predictive learning model may utilize the received data set 405 , the received selection of the prediction field 410 , and the plurality of features 415 to generate the model 420 .
- a plurality of candidate machine learning models may be trained 450 .
- the plurality of candidate machine learning models 450 may be trained, for example, based on one or more of the data set or the prediction field received by the database server.
- the models may be evaluated based in part on a predictive accuracy of each of the plurality of models 455 .
- the models may be evaluated to determine a threshold level of accuracy given the input received by the database server. Accordingly, for example, a predictive learning machine model may be selected 460 based in part on the evaluating. Thus, in some examples, the predictive machine learning model that would result in a most-accurate prediction given the input received by the database server may be selected.
- the database server may perform the predictions and transmit the predicted 425 values to, for example, a computing device. This may be an example of transmitting an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model.
- a score may be determined for the prediction field 465 . This score may be based in part on generating the predictive learning model 420 . Thus, in some examples, the transmission of the predicted values 425 may be based on the score.
- the determined score may be based on a number of characteristics. For example, the score may be based on a number of predicted values or a correlation between a particular predicted value and the data set (e.g., the accuracy of the prediction).
- the score may be saved 470 to the database server, which may be used in determining future predictions using a same or similar data set, prediction field(s), or both.
- FIG. 5 illustrates an example of a system 500 that supports generating a predictive machine learning model in accordance with various aspects of the present disclosure.
- the system 500 may include Team City 505 , which may be an example of a continuous integration server; EP Provisioning 510 , which may be an example of a server or, more specifically, a business process outsourcing (BPO) server; creation component 520 , which may create a set of objects to utilize in generating a predictive machine learning model; updating and scheduling component 525 , which may schedule and update system workflows; and objects 530 , 535 , and 540 , which may be or may represent specific data or data sets.
- Team City 505 may be an example of a continuous integration server
- EP Provisioning 510 which may be an example of a server or, more specifically, a business process outsourcing (BPO) server
- creation component 520 which may create a set of objects to utilize in generating a predictive machine learning model
- updating and scheduling component 525 which may schedule and update system workflows
- EP Provisioning database 510 may obtain a list of predictive models to be generated. The EP Provisioning database 510 may also obtain requisite data to generate one or more predictive models.
- Team City 505 may run hourly to pull BPO configurations for all Builder tenants (e.g., of a multi-tenant database system) to get current and disabled configurations.
- a whitelist is constructed and stored in the EP Provisioning database 510 .
- Team City 505 may run at a pre-defined time interval or at random time intervals to obtain configurations for one or more tenants.
- the configurations may be associated with one or more specific tenants.
- the whitelist created based on the configurations may be stored in the EP Provisioning database 510 .
- the data puller job is executed hourly on all prediction objects to get all incremental changes to them since the last successful pull.
- Team City 505 may be in communication with EP Provisioning database 510 via connection 515 .
- Team City 505 may receive data from EP Provisioning database 510 more or less frequently in order to record incremental changes to the database.
- a daily snapshotting job may be executed for all prediction objects.
- Team City 505 may monitor EP Provisioning database 510 at certain time intervals to ensure that any changes, no matter how incremental, are recorded. Team City 505 may not be limited to a daily snapshotting job.
- Team City 505 may create and organize any number of jobs. For example, Team City 505 may determine and schedule one or more mathematical calculations to be performed on the data used to generate a predictive model. Team City 505 may then monitor the jobs (e.g., calculations) and schedule subsequent jobs accordingly.
- any number of data flows may be set up.
- the data flows may, for example, correspond to one or more prediction objects.
- any one of creation component 520 , scheduling component 525 , and objects 530 , 535 , and 540 may be utilized to generate a predictive machine learning model as described above with reference to FIGS. 1 through 4 .
- FIG. 6 illustrates an example of a system 600 that supports generating a predictive machine learning model in accordance with various aspects of the present disclosure.
- the system 600 may include Team City 605 , which may be an example of Team City as described with reference to FIG. 5 ; EP Provisioning 610 , which may be an example of EP Provisioning 510 as described with reference to FIG. 5 ; configuration component 620 , which may create scoring workflows for configurations; data store 625 , which may be an example of database server 210 as described with reference to FIG. 2 ; updating and scheduling component 525 , which may be an example of updating and scheduling component 525 as described with reference to FIG. 5 ; and training models 640 , 645 , and 650 .
- Team City 605 may be in communication with EP Provisioning database via connection 615 and data store 625 may transmit data or communications to configuration component 620 via transmission 630 .
- FIG. 6 may illustrate model training flows for new configurations in the EP Provisioning database 610 .
- FIG. 6 may illustrate the monthly model retrain process for all configurations.
- FIG. 6 may illustrate that all model training runs operate off of the latest snapshot of the prediction object.
- any one of Team City 605 , EP Provisioning database 610 , configurations component 620 , data store 625 , updating and scheduling component 635 , and training models 640 , 645 , and 650 may be utilized to generate a predictive machine learning model as described above with reference to FIGS. 1 through 5 .
- FIG. 7 illustrates an example of a system 700 that supports generating a predictive machine learning model in accordance with various aspects of the present disclosure.
- the system 700 may include Team City 705 , which may be an example of Team City as described with reference to FIG. 5 ; EP Provisioning 710 , which may be an example of EP Provisioning 510 as described with reference to FIG. 5 ; scoring creation and workflow component 720 ; data store 725 , which may be an example of the data store 625 as described with reference to FIG. 6 ; updating and scheduling component 730 , which may be an example of updating and scheduling component 525 as described with reference to FIG.
- training models 735 and 740 which may be an example of one or more of training models 640 , 645 , and 650 as described with reference to FIG. 6 ; new score components 745 and 750 ; and server 745 , which may be an example of database server 210 or database 215 as described with reference to FIG. 2 .
- Team City 605 may be in communication with EP Provisioning database via connection 715 and data store 625 may transmit data or communications to training models 735 and 740 .
- FIG. 7 may illustrate scoring and data pushback flows.
- scoring jobs may run on an hourly cadence scoring all increments pulled since the last dataset that was successfully scored.
- scoring jobs may run periodically, randomly, or on a predetermined time frame to score any increments pulled since the dataset was last successfully scored.
- data pushback jobs are chained off of scoring jobs and push back the last two scored datasets, in order, for redundancy in case data pushback failed the previous time.
- data pushback jobs may be chained off of scoring jobs and may push back any number of scored datasets, in any order.
- the data pushback jobs may be based on previously-successful data pushback.
- any one of Team City 705 , EP Provisioning database 710 , scoring creation and workflow component 720 , data store 725 , updating and scheduling component 730 , training models 735 and 740 , new score components 745 and 750 , and server 745 may be utilized to generate a predictive machine learning model as described above with reference to FIGS. 1 through 6 .
- FIG. 8 shows a block diagram 800 of an apparatus 805 that supports automatic machine learning model generation in accordance with aspects of the present disclosure.
- Apparatus 805 may include input module 810 , predictive model builder 815 , and output module 820 .
- Apparatus 805 may also include a processor. Each of these components may be in communication with one another (e.g., via one or more buses).
- apparatus 805 may be an example of a user terminal, a database server, or a system containing multiple computing devices.
- Predictive model builder 815 may be an example of aspects of the predictive model builder 1015 described with reference to FIG. 10 .
- Predictive model builder 815 and/or at least some of its various sub-components may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions of the predictive model builder 815 and/or at least some of its various sub-components may be executed by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described in the present disclosure.
- DSP digital signal processor
- ASIC application-specific integrated circuit
- FPGA field-programmable gate array
- the predictive model builder 815 and/or at least some of its various sub-components may be physically located at various positions, including being distributed such that portions of functions are implemented at different physical locations by one or more physical devices.
- predictive model builder 815 and/or at least some of its various sub-components may be a separate and distinct component in accordance with various aspects of the present disclosure.
- predictive model builder 815 and/or at least some of its various sub-components may be combined with one or more other hardware components, including but not limited to an I/O component, a transceiver, a network server, another computing device, one or more other components described in the present disclosure, or a combination thereof in accordance with various aspects of the present disclosure.
- Predictive model builder 815 may also include data selector 825 , field selector 830 , feature generator 835 , model builder 840 , and scoring component 845 .
- Data selector 825 may receive, at a database server, a selection of a data set, where the data set includes a set of fields and receive an indication of one or more additional objects, where the indication of the set of predicted values is based on receiving the indication of the one or more additional objects.
- the receiving the selection of the data set includes receiving an indication of an object, where the indication of the set of predicted values is based on receiving the indication of the object.
- the database server includes a multi-tenant database server.
- Field selector 830 may receive, at the database server, a selection of a prediction field from the set of fields and receive a selection of prediction fields from the set of fields, where the set of features from the data set are generated based on the selection of prediction fields.
- Feature generator 835 may generate, by the database server, a set of features from the data set, where the set of features are generated automatically based on metadata associated with the data set.
- the metadata associated with the data set includes a classification of a field and a data type of the field.
- Model builder 840 may generate, by the database server, the predictive machine learning model based on the set of features.
- Scoring component 845 may determine a score for the prediction field based on generating the predictive machine learning model, where transmitting the indication of the set of predicted values is based on the determined score, transmit an indication of a set of predicted values for the prediction field based on the predictive machine learning model, receive a new record associated with the data set, update the score for the prediction field based on the new record, and store the score in the database server.
- FIG. 9 shows a block diagram 900 of a predictive model builder 915 that supports automatic machine learning model generation in accordance with aspects of the present disclosure.
- the predictive model builder 915 may be an example of aspects of a predictive model builder 1015 described with reference to FIGS. 7 , 8 , and 10 .
- the predictive model builder 915 may include data selector 920 , field selector 925 , feature generator 930 , model builder 935 , scoring component 940 , feature checking component 945 , model selection component 950 , data balancing component 955 , and score output component 960 . Each of these modules may communicate, directly or indirectly, with one another (e.g., via one or more buses).
- Data selector 920 may receive, at a database server, a selection of a data set, where the data set includes a set of fields and receive an indication of one or more additional objects, where the indication of the set of predicted values is based on receiving the indication of the one or more additional objects.
- the receiving the selection of the data set includes receiving an indication of an object, where the indication of the set of predicted values is based on receiving the indication of the object.
- the database server includes a multi-tenant database server.
- Field selector 925 may receive, at the database server, a selection of a prediction field from the set of fields and receive a selection of prediction fields from the set of fields, where the set of features from the data set are generated based on the selection of prediction fields.
- Feature generator 930 may generate, by the database server, a set of features from the data set, where the set of features are generated automatically based on metadata associated with the data set.
- the metadata associated with the data set includes a classification of a field and a data type of the field.
- Model builder 935 may generate, by the database server, the predictive machine learning model based on the set of features.
- Scoring component 940 may determine a score for the prediction field based on generating the predictive machine learning model, where transmitting the indication of the set of predicted values is based on the determined score, transmit an indication of a set of predicted values for the prediction field based on the predictive machine learning model, receive a new record associated with the data set, update the score for the prediction field based on the new record, and store the score in the database server.
- Feature checking component 945 may evaluate a statistical relationship between the set of predicted values for the prediction field and a subset of the set of features and remove at least one feature of the subset of the set of features from the predictive machine learning model if the statistical relationship exceeds a threshold.
- the statistical relationship includes a correlation calculation, a Cramer's V calculation, or a combination thereof.
- Model selection component 950 may train a set of candidate machine learning models, evaluate the set of candidate machine learning models based on a predictive accuracy of each of the set of candidate machine learning models, and select the predictive machine learning model based on the evaluating.
- Data balancing component 955 may adjust a sampling rate of the data set based on a representation rate of the set of predicted values.
- Score output component 960 may save the set of predicted values to the prediction field based on transmitting the indication of the set of predicted values.
- FIG. 10 shows a diagram of a system 1000 including a device 1005 that supports automatic machine learning model generation in accordance with aspects of the present disclosure.
- Device 1005 may be an example of or include the components of data center 120 or database server 210 as described above, e.g., with reference to FIG. 1 .
- Device 1005 may include components for bi-directional data communications including components for transmitting and receiving communications, including predictive model builder 1015 , processor 1020 , memory 1025 , database controller 1030 , database 1035 , and I/O controller 1040 . These components may be in electronic communication via one or more buses (e.g., bus 1010 ).
- buses e.g., bus 1010
- Processor 1020 may include an intelligent hardware device, (e.g., a general-purpose processor, a DSP, a central processing unit (CPU), a microcontroller, an ASIC, an FPGA, a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof).
- processor 1020 may be configured to operate a memory array using a memory controller.
- a memory controller may be integrated into processor 1020 .
- Processor 1020 may be configured to execute computer-readable instructions stored in a memory to perform various functions (e.g., functions or tasks supporting automatic machine learning model generation).
- Memory 1025 may include random access memory (RAM) and read only memory (ROM).
- the memory 1025 may store computer-readable, computer-executable software 1030 including instructions that, when executed, cause the processor to perform various functions described herein.
- the memory 1025 may contain, among other things, a basic input/output system (BIOS) which may control basic hardware or software operation such as the interaction with peripheral components or devices.
- BIOS basic input/output system
- Database controller 1030 may manage data storage and processing in database 1035 . In some cases, a user may interact with database controller 1030 . In other cases, database controller 1030 may operate automatically without user interaction.
- Database 1035 may be an example of a single database, a distributed database, multiple distributed databases, or an emergency backup database.
- I/O controller 1040 may manage input and output signals for device 1005 . I/O controller 1040 may also manage peripherals not integrated into device 1005 . In some cases, I/O controller 1040 may represent a physical connection or port to an external peripheral. In some cases, I/O controller 1040 may utilize an operating system such as iOS®, ANDROID®, MS-DOS®, MS-WINDOWS®, OS/2®, UNIX®, LINUX®, or another known operating system. In other cases, I/O controller 1040 may represent or interact with a modem, a keyboard, a mouse, a touchscreen, or a similar device. In some cases, I/O controller 1040 may be implemented as part of a processor. In some cases, a user may interact with device 1005 via I/O controller 1040 or via hardware components controlled by I/O controller 1040 .
- I/O controller 1040 may manage input and output signals for device 1005 . I/O controller 1040 may also manage peripherals not integrated into device 1005 . In some cases, I
- FIG. 11 shows a flowchart illustrating a method 1100 for automatic machine learning model generation in accordance with aspects of the present disclosure.
- the operations of method 1100 may be implemented by a data center 120 or database server 210 or its components as described herein.
- the operations of method 1100 may be performed by a predictive model builder as described with reference to FIGS. 8 through 10 .
- a data center 120 or database server 210 may execute a set of codes to control the functional elements of the device to perform the functions described below. Additionally or alternatively, the data center 120 or database server 210 may perform aspects of the functions described below using special-purpose hardware.
- the data center 120 or database server 210 may receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields.
- the operations of 1105 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1105 may be performed by a data selector as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may receive, at the database server, a selection of a prediction field from the plurality of fields.
- the operations of 1110 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1110 may be performed by a field selector as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set.
- the operations of 1115 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1115 may be performed by a feature generator as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may generate, by the database server, the predictive machine learning model based at least in part on the plurality of features.
- the operations of 1120 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1120 may be performed by a model builder as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- the operations of 1125 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1125 may be performed by a scoring component as described with reference to FIGS. 8 through 10 .
- FIG. 12 shows a flowchart illustrating a method 1200 for automatic machine learning model generation in accordance with aspects of the present disclosure.
- the operations of method 1200 may be implemented by a data center 120 or database server 210 or its components as described herein.
- the operations of method 1200 may be performed by a predictive model builder as described with reference to FIGS. 8 through 10 .
- a data center 120 or database server 210 may execute a set of codes to control the functional elements of the device to perform the functions described below. Additionally or alternatively, the data center 120 or database server 210 may perform aspects of the functions described below using special-purpose hardware.
- the data center 120 or database server 210 may receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields.
- the operations of 1205 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1205 may be performed by a data selector as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may receive, at the database server, a selection of a prediction field from the plurality of fields.
- the operations of 1210 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1210 may be performed by a field selector as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set.
- the operations of 1215 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1215 may be performed by a feature generator as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may evaluate a statistical relationship between the plurality of predicted values for the prediction field and a subset of the plurality of features.
- the operations of 1220 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1220 may be performed by a feature checking component as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may remove at least one feature of the subset of the plurality of features from the predictive machine learning model if the statistical relationship exceeds a threshold.
- the operations of 1225 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1225 may be performed by a feature checking component as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may generate, by the database server, the predictive machine learning model based at least in part on the plurality of features.
- the operations of 1230 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1230 may be performed by a model builder as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- the operations of 1235 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1235 may be performed by a scoring component as described with reference to FIGS. 8 through 10 .
- FIG. 13 shows a flowchart illustrating a method 1300 for automatic machine learning model generation in accordance with aspects of the present disclosure.
- the operations of method 1300 may be implemented by a data center 120 or database server 210 or its components as described herein.
- the operations of method 1300 may be performed by a predictive model builder as described with reference to FIGS. 8 through 10 .
- a data center 120 or database server 210 may execute a set of codes to control the functional elements of the device to perform the functions described below. Additionally or alternatively, the data center 120 or database server 210 may perform aspects of the functions described below using special-purpose hardware.
- the data center 120 or database server 210 may receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields.
- the operations of 1305 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1305 may be performed by a data selector as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may receive, at the database server, a selection of a prediction field from the plurality of fields.
- the operations of 1310 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1310 may be performed by a field selector as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set.
- the operations of 1315 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1315 may be performed by a feature generator as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may train a plurality of candidate machine learning models.
- the operations of 1320 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1320 may be performed by a model selection component as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may evaluate the plurality of candidate machine learning models based at least in part on a predictive accuracy of each of the plurality of candidate machine learning models.
- the operations of 1325 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1325 may be performed by a model selection component as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may select the predictive machine learning model based at least in part on the evaluating.
- the operations of 1330 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1330 may be performed by a model selection component as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may generate, by the database server, the predictive machine learning model based at least in part on the plurality of features.
- the operations of 1335 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1335 may be performed by a model builder as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- the operations of 1340 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1340 may be performed by a scoring component as described with reference to FIGS. 8 through 10 .
- FIG. 14 shows a flowchart illustrating a method 1400 for automatic machine learning model generation in accordance with aspects of the present disclosure.
- the operations of method 1400 may be implemented by a data center 120 or database server 210 or its components as described herein.
- the operations of method 1400 may be performed by a predictive model builder as described with reference to FIGS. 8 through 10 .
- a data center 120 or database server 210 may execute a set of codes to control the functional elements of the device to perform the functions described below. Additionally or alternatively, the data center 120 or database server 210 may perform aspects of the functions described below using special-purpose hardware.
- the data center 120 or database server 210 may receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields.
- the operations of 1405 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1405 may be performed by a data selector as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may receive, at the database server, a selection of a prediction field from the plurality of fields.
- the operations of 1410 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1410 may be performed by a field selector as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set.
- the operations of 1415 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1415 may be performed by a feature generator as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may adjust a sampling rate of the data set based at least in part on a representation rate of the plurality of predicted values.
- the operations of 1420 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1420 may be performed by a data balancing component as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may generate, by the database server, the predictive machine learning model based at least in part on the plurality of features.
- the operations of 1425 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1425 may be performed by a model builder as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- the operations of 1430 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1430 may be performed by a scoring component as described with reference to FIGS. 8 through 10 .
- FIG. 15 shows a flowchart illustrating a method 1500 for automatic machine learning model generation in accordance with aspects of the present disclosure.
- the operations of method 1500 may be implemented by a data center 120 or database server 210 or its components as described herein.
- the operations of method 1500 may be performed by a predictive model builder as described with reference to FIGS. 8 through 10 .
- a data center 120 or database server 210 may execute a set of codes to control the functional elements of the device to perform the functions described below. Additionally or alternatively, the data center 120 or database server 210 may perform aspects of the functions described below using special-purpose hardware.
- the data center 120 or database server 210 may receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields.
- the operations of 1505 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1505 may be performed by a data selector as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may receive, at the database server, a selection of a prediction field from the plurality of fields.
- the operations of 1510 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1510 may be performed by a field selector as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set.
- the operations of 1515 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1515 may be performed by a feature generator as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may determine a score for the prediction field based at least in part on generating the predictive machine learning model, wherein transmitting the indication of the plurality of predicted values is based at least in part on the determined score.
- the operations of 1520 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1520 may be performed by a scoring component as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may generate, by the database server, the predictive machine learning model based at least in part on the plurality of features.
- the operations of 1525 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1525 may be performed by a model builder as described with reference to FIGS. 8 through 10 .
- the data center 120 or database server 210 may transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- the operations of 1530 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1530 may be performed by a scoring component as described with reference to FIGS. 8 through 10 .
- a method of generating a predictive machine learning model may include receiving, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields, receiving, at the database server, a selection of a prediction field from the plurality of fields, generating, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set, generating, by the database server, the predictive machine learning model based at least in part on the plurality of features, and transmitting an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- the apparatus may include means for receiving, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields, means for receiving, at the database server, a selection of a prediction field from the plurality of fields, means for generating, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set, means for generating, by the database server, the predictive machine learning model based at least in part on the plurality of features, and means for transmitting an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- the apparatus may include a processor, memory in electronic communication with the processor, and instructions stored in the memory.
- the instructions may be operable to cause the processor to receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields, receive, at the database server, a selection of a prediction field from the plurality of fields, generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set, generate, by the database server, the predictive machine learning model based at least in part on the plurality of features, and transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- a non-transitory computer-readable medium for generating a predictive machine learning model may include instructions operable to cause a processor to receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields, receive, at the database server, a selection of a prediction field from the plurality of fields, generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set, generate, by the database server, the predictive machine learning model based at least in part on the plurality of features, and transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- the statistical relationship comprises a correlation calculation, a Cramer's V calculation, or a combination thereof.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for training a plurality of candidate machine learning models. Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for evaluating the plurality of candidate machine learning models based at least in part on a predictive accuracy of each of the plurality of candidate machine learning models. Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for selecting the predictive machine learning model based at least in part on the evaluating.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for adjusting a sampling rate of the data set based at least in part on a representation rate of the plurality of predicted values.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for determining a score for the prediction field based at least in part on generating the predictive machine learning model, wherein transmitting the indication of the plurality of predicted values may be based at least in part on the determined score.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for receiving a new record associated with the data set. Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for updating the score for the prediction field based at least in part on the new record.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for storing the score in the database server.
- the receiving the selection of the data set comprises receiving an indication of an object, wherein the indication of the plurality of predicted values may be based at least in part on receiving the indication of the object.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for receiving an indication of one or more additional objects, wherein the indication of the plurality of predicted values may be based at least in part on receiving the indication of the one or more additional objects.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for receiving a selection of prediction fields from the plurality of fields, wherein the plurality of features from the data set may be generated based at least in part on the selection of prediction fields.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for saving the plurality of predicted values to the prediction field based at least in part on transmitting the indication of the plurality of predicted values.
- the database server comprises a multi-tenant database server.
- the metadata associated with the data set comprises a classification of a field and a data type of the field.
- Information and signals described herein may be represented using any of a variety of different technologies and techniques.
- data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
- a general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
- a processor may also be implemented as a combination of computing devices (e.g., a combination of a digital signal processor (DSP) and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration).
- DSP digital signal processor
- the functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Other examples and implementations are within the scope of the disclosure and appended claims. For example, due to the nature of software, functions described above can be implemented using software executed by a processor, hardware, firmware, hardwiring, or combinations of any of these. Features implementing functions may also be physically located at various positions, including being distributed such that portions of functions are implemented at different physical locations.
- “or” as used in a list of items indicates an inclusive list such that, for example, a list of at least one of A, B, or C means A or B or C or AB or AC or BC or ABC (i.e., A and B and C).
- the phrase “based on” shall not be construed as a reference to a closed set of conditions. For example, an exemplary step that is described as “based on condition A” may be based on both a condition A and a condition B without departing from the scope of the present disclosure.
- the phrase “based on” shall be construed in the same manner as the phrase “based at least in part on.”
- Computer-readable media includes both non-transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.
- a non-transitory storage medium may be any available medium that can be accessed by a general purpose or special purpose computer.
- non-transitory computer-readable media can comprise RAM, ROM, electrically erasable programmable read only memory (EEPROM), compact disk (CD) ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor.
- RAM random access memory
- ROM read only memory
- EEPROM electrically erasable programmable read only memory
- CD compact disk
- magnetic disk storage or other magnetic storage devices or any other non-transitory medium that can be used to carry or store desired program code means in the form of instructions or data structures
- any connection is properly termed a computer-readable medium.
- the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave
- the coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave are included in the definition of medium.
- Disk and disc include CD, laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of computer-readable media.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Medical Informatics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A system may automatically generate a predictive machine learning model by automatically performing various processes based on an analysis of the data as well as metadata associated with the data. The system may accept a selection of data and a prediction field from the data. The system may automatically generate a set of features based on the data and may automatically remove certain features that cause inaccuracies in the model. The system may balance the data based on a representation rate of certain outcomes. The system may train and select a model based on several candidate models. The system may then perform the predictions based on the selected model and send an indication of the predictions to a user.
Description
- The present application for patent is a continuation of U.S. patent application Ser. No. 15/884,878 by Asher et al., entitled “AUTOMATIC MACHINE LEARNING MODEL GENERATION,” filed Jan. 31, 2018, which claims priority to U.S. Patent Application No. 62/581,624 by Asher et al., entitled “AUTOMATIC MACHINE LEARNING MODEL GENERATION,” filed Nov. 3, 2017, assigned to the assignee hereof, and expressly incorporated by reference herein.
- The present disclosure relates generally to database systems and data processing, and more specifically to automatic machine learning model generation.
- A cloud platform (i.e., a computing platform for cloud computing) may be employed by many users to store, manage, and process data using a shared network of remote servers. Users may develop applications on the cloud platform to handle the storage, management, and processing of data. In some cases, the cloud platform may utilize a multi-tenant database system. Users may access the cloud platform using various user devices (e.g., desktop computers, laptops, smartphones, tablets, or other computing systems, etc.).
- In one example, the cloud platform may support customer relationship management (CRM) solutions. This may include support for sales, service, marketing, community, analytics, applications, and the Internet of Things. A user may utilize the cloud platform to help manage contacts of the user. For example, managing contacts of the user may include analyzing data, storing and preparing communications, and tracking opportunities and sales.
- A user of a CRM application may wish to predict values for certain fields of data using a predictive model. Such systems may include a user interface with a number of fields to manually enter data, identifiers, or any content associated with the prediction sought by the user. For example, the user interface may include fields to manually select certain parameters related to the predictive model, such as a definition of the relationships within the data, a selection of appropriate algorithms to use, tuning parameters of the algorithms, etc. Furthermore, the system may require that the data to be used for building the predictive model be manually tailored such that it is suitable for building a model. Without specialized statistical training, knowledge of such parameters may be unknown to a user. The system may require that these fields be filled in before a prediction will be created. In some cases, the user may manually enter data into the input fields that may be incorrect or incomplete, and may result in the system providing an inaccurate prediction or, in some cases, the system may be unable to provide a prediction altogether, given the input.
-
FIG. 1 illustrates an example of a system for generating a predictive machine learning model that supports automatic machine learning model generation in accordance with aspects of the present disclosure. -
FIG. 2 illustrates an example of a database system that supports automatic machine learning model generation in accordance with aspects of the present disclosure. -
FIG. 3 illustrates an example of a user interface that supports automatic machine learning model generation in accordance with aspects of the present disclosure. -
FIG. 4 illustrates an example of a process flow diagram performed by a database system that supports automatic machine learning model generation in accordance with aspects of the present disclosure. -
FIGS. 5-7 illustrate additional examples of database systems that support automatic machine learning model generation in accordance with aspects of the present disclosure. -
FIGS. 8 through 9 show block diagrams of a device that supports automatic machine learning model generation in accordance with aspects of the present disclosure. -
FIG. 10 illustrates a block diagram of a system including a database server that supports automatic machine learning model generation in accordance with aspects of the present disclosure. -
FIGS. 11 through 15 illustrate methods for automatic machine learning model generation in accordance with aspects of the present disclosure. - When a user seeks to generate a prediction using machine learning, the prediction is usually generated based on several user-defined inputs. However, given any one user's familiarity with data science, the accuracy of the prediction may be wide-ranging. Further, the accuracy of a prediction may be directly correlated to the quantity and accuracy of the received input. Thus a typical user without specialized statistical training (e.g., an administrator, a sales manager, etc.) may be unable to build accurate predictive models because of the manual selection process required. However, in accordance with aspects of the present disclosure, by receiving a selection of a data set and a selection of a prediction field—for example, via a wizard or a simplified user interface—a system may be configured to automatically generate a predictive model by choosing the necessary parameters based on an automatic analysis of the data (e.g., based on metadata associated with the data), thereby reducing or eliminating the need for the user to generate any code or to understand the data science being used to automatically select the parameters.
- In a first example, a predictive machine learning model may be generated. A database server may receive a selection of a data set that may include a plurality of fields. In some examples, the data set may include an object—a representation of a type of structured data. Thus the plurality of fields may represent subsets of the structured data (e.g., rows in a table). The database server may subsequently receive a selection of a prediction field for use in generating the prediction. The database server may automatically generate a plurality of features from the data set and then may automatically remove certain features that cause inaccuracies in the predictive model. This automatic feature generation and selection process may be performed by the database server based on metadata associated with the data (e.g., knowing that a string of characters is actually an email address). The database server may then generate a predictive learning model based on the plurality of features and may transmit an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model.
- In another example, a predictive machine learning model may be generated by first receiving, at a database server, a selection of a data set that may include a plurality of fields. The database server may subsequently receive a selection of one or more prediction fields for use in generating the prediction. The database server may generate a plurality of features from the data set and may then generate a predictive learning model based on the plurality of features. The database server may then evaluate a statistical relationship (e.g., a correlation or a Cramer's V calculation) between the plurality of predicted values generated by the predictive learning model and, in some examples, remove at least one feature of the subset of features. This process may be referred to as a “sanity checker.” In some examples the evaluation may be based on a correlation between the data set and the prediction field. In other examples, the evaluation may be based on a nominal association, such as Cramer's V. In some examples, the removal of certain features may result in a more-accurate prediction by not considering potentially extraneous relationships in the determination of the predicted values. In some examples, the database server may transmit an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model.
- In yet another example, a predictive machine learning model may be generated by first receiving, at a database server, a selection of a data set that may include a plurality of fields. The database server may generate a plurality of features from the data set and may then generate a predictive learning model based on the plurality of features. In some examples, the database server may generate a predictive learning model based on training a plurality of candidate machine learning models. For example, the database server may evaluate a plurality of machine learning models based in part on a predictive accuracy of the model. The database may then, for example, select the predictive machine learning model based in part on the evaluation. This may result in a more-accurate determination of the predicted values. In some examples, the database server may transmit an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model.
- In another example, a predictive machine learning model may be generated by first receiving, at a database server, a selection of a data set that may include a plurality of fields. The database server may generate a plurality of features from the data set and may then generate a predictive learning model based on the plurality of features. After determining the predictive learning model, in some examples, the database server may determine a plurality of scores for the prediction field based in part on generating the model. The database server may receive a new record associated with the data set and update the scores for the prediction field based in part on the new record. This process may be done in real time and may lead to updated determinations of the predicted values. In some examples, the database server may transmit an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model.
- In another example, the predictive machine learning model may improve one or more business processes or workflows within a company, by making the processes more intelligent or more efficient. For example, employees may need more insights into the future and may need to deflect repetitive tasks that take up their days. However, companies may be hindered by the infrastructure costs, lack of expertise, and the resources required to optimize their workflow with AI. With a predictive machine learning model, the employees who are actually managing and driving business processes may have the power to build and customize AI apps to fit their specific needs. By using selections—such as clicks—the declarative setup guide may walk users through building, training, and deploying AI models using structured and unstructured data, such as metadata. The predictive machine learning model may automate the model building and data scoring process, and custom predictive models and bots may be easily embedded directly into workflows. The models may automatically learn and improve as they're used, delivering accurate, personalized recommendations and predictions in the context of business.
- For example, the predictive machine learning model may allow users to create custom AI models on any field or object—custom or standard—to predict business outcomes, such as a customer's likelihood to churn or the lifetime value of an account. For example, an admin at a financial services company may build a predictive attrition model to understand which customers are most likely to churn based on any fields related to the account, such as the number of external accounts they link to their checking account and history of customer support calls. Using a declarative, point-and-click setup tool, users may define the prediction, identify the field to build the model on, and select which data to use. The attrition score may be based on these signals, and may be directly embedded into a customer account page and create a task, automatically alerting users to high-risk customers, so the user may deliver personalized, one-on-one interactions to reduce attrition risk.
- Aspects of the disclosure are initially described in the context of an environment supporting an on-demand database service. Further aspects of the disclosure are described with respect to systems that support automatic machine learning model generation, such as at a database server. Aspects of the disclosure are further illustrated by and described with reference to apparatus diagrams, system diagrams, and flowcharts that relate to automatic machine learning model generation.
-
FIG. 1 illustrates an example of asystem 100 for cloud computing that supports automatic machine learning model generation in accordance with various aspects of the present disclosure. Thesystem 100 includescloud clients 105,contacts 110,cloud platform 115, anddata center 120.Cloud platform 115 may be an example of a public or private cloud network. Acloud client 105 may accesscloud platform 115 overnetwork connection 135. The network may implement transfer control protocol and internet protocol (TCP/IP), such as the Internet, or may implement other network protocols. Acloud client 105 may be an example of a user device, such as a server (e.g., cloud client 105-a), a smartphone (e.g., cloud client 105-b), or a laptop (e.g., cloud client 105-c). In other examples, acloud client 105 may be a desktop computer, a tablet, a sensor, or another computing device or system capable of generating, analyzing, transmitting, or receiving communications. In some examples, acloud client 105 may be operated by a user that is part of a business, an enterprise, a non-profit, a startup, or any other organization type. - A
cloud client 105 may interact withmultiple contacts 110. Theinteractions 130 may include communications, opportunities, purchases, sales, or any other interaction between acloud client 105 and acontact 110. Data may be associated with theinteractions 130. Acloud client 105 may accesscloud platform 115 to store, manage, and process the data associated with theinteractions 130. In some cases, thecloud client 105 may have an associated security or permission level. Acloud client 105 may have access to certain applications, data, and database information withincloud platform 115 based on the associated security or permission level, and may not have access to others. -
Contacts 110 may interact with thecloud client 105 in person or via phone, email, web, text messages, mail, or any other appropriate form of interaction (e.g., interactions 130-a, 130-b, 130-c, and 130-d). Theinteraction 130 may be a business-to-business (B2B) interaction or a business-to-consumer (B2C) interaction. Acontact 110 may also be referred to as a customer, a potential customer, a lead, a client, or some other suitable terminology. In some cases, thecontact 110 may be an example of a user device, such as a server (e.g., contact 110-a), a laptop (e.g., contact 110-b), a smartphone (e.g., contact 110-c), or a sensor (e.g., contact 110-d). In other cases, thecontact 110 may be another computing system. In some cases, thecontact 110 may be operated by a user or group of users. The user or group of users may be associated with a business, a manufacturer, or any other appropriate organization. -
Cloud platform 115 may offer an on-demand database service to thecloud client 105. In some cases,cloud platform 115 may be an example of a multi-tenant database system. In this case,cloud platform 115 may servemultiple cloud clients 105 with a single instance of software. However, other types of systems may be implemented, including—but not limited to—client-server systems, mobile device systems, and mobile network systems. In some cases,cloud platform 115 may support CRM solutions. This may include support for sales, service, marketing, community, analytics, applications, and the Internet of Things.Cloud platform 115 may receive data associated withcontact interactions 130 from thecloud client 105 overnetwork connection 135, and may store and analyze the data. In some cases,cloud platform 115 may receive data directly from aninteraction 130 between acontact 110 and thecloud client 105. In some cases, thecloud client 105 may develop applications to run oncloud platform 115.Cloud platform 115 may be implemented using remote servers. In some cases, the remote servers may be located at one ormore data centers 120. -
Data center 120 may include multiple servers. The multiple servers may be used for data storage, management, and processing.Data center 120 may receive data fromcloud platform 115 viaconnection 140, or directly from thecloud client 105 or aninteraction 130 between acontact 110 and thecloud client 105.Data center 120 may utilize multiple redundancies for security purposes. In some cases, the data stored atdata center 120 may be backed up by copies of the data at a different data center (not pictured). -
Subsystem 125 may includecloud clients 105,cloud platform 115, anddata center 120. In some cases, data processing may occur at any of the components ofsubsystem 125, or at a combination of these components. In some cases, servers may perform the data processing. The servers may be acloud client 105 or located atdata center 120. - The
cloud clients 105 ofsubsystem 125 may be examples of one or more computing devices anddata center 120 may be an example of or include, as a component, a database server to which acloud client 105 may attempt to connect with. In accordance with aspects of the present disclosure, thedata center 120 may be configured to automatically build or generate a predictive model based on inputs selected by a user via a couldclient 105. For example,data center 120 may receive a selection of a data set, which may include a plurality of fields. The data set may be received in response to a customer seeking a prediction of an outcome, given a particular data set. In some examples, thedata center 120 may receive a selection of a prediction field from the plurality of fields. The selection may be made, for example, at a user interface ofcloud client 105. Thedata center 120 may then generate a plurality of features from the data set. In some examples, the features may be generated automatically based on metadata associated with the data set. Upon generating the plurality of features, thedata center 120 may generate a predictive learning model—for use in generating a prediction-based in part on the plurality of features. Thedata center 120 may then transmit, for example to cloudclient 105—an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model. As described in more detail below, the generation of the predictive model bydata center 120 may be facilitated by a user interface at a cloud client 105 (e.g., a wizard). -
FIG. 2 illustrates an example of asystem 200 that supports generating a predictive machine learning model in accordance with various aspects of the present disclosure. Thesystem 200 may include acomputing device 205, which may be an example of acloud client 105 as described with reference toFIG. 1 ; adatabase server 210, which may be an example of thedata center 120 as described with reference toFIG. 1 ; and adatabase 215. In some cases, thedatabase server 210 may be an example of components of adata center 120, acloud platform 115, or some combination of these, as described with reference toFIG. 1 . Thesystem 200 may also include one or more connections, such asconnection 220 andconnection 225. In some examples, thecomputing device 205 may includedisplay 230 and user interface 235. Thedatabase server 210 may includemodel builder 240, which may include a plurality of modules—including data setselector 245, prediction field selector 250,feature generator 255, learning model generator 260, and predicted value transmitter 265. In some examples, thedatabase server 210 may be a multi-tenant database server. In other examples, thedatabase server 210 may be an example of or include a predictive model platform. - In some examples, the
computing device 205 may display a user interface 235 via adisplay 230. The user interface 235 may contain one or more fields to receive user input. For example, the fields may receive user input that includes a prediction name, a prediction type, an Application Program Interface (API) name, an object prediction, or a field prediction, among others. In some examples, the fields may be associated with receiving a selection of a data set or receiving a selection of a prediction field. In other examples, the fields may be associated with receiving an indication of one or more objects. Additionally or alternatively, for example, a user may input each of the prediction name, the prediction type, the API name, the object prediction, or the field prediction. In other examples, the user may input the object prediction or the field prediction by selecting one or more objects or fields to predict from, for example, a populated list of objects or fields. - Upon receiving the user input,
database server 210 may receive a selection of a data set. The data set may include data stored ondatabase 215 that is associated with a CRM account. In some examples, the data set may include a plurality of fields. Thedatabase server 210 may receive the data set, for example, atdata set selector 245. The data set received bydatabase server 210 may, be structured as one or more objects. For example, the data set, or the object, may be a representation of a type of structured data. Thus the plurality of fields included in the data set may be or may be indicative of specific types or subsets of structured data. In some examples, selecting objects (e.g., at data set selector 245) may facilitate feature generation. For example, selecting additional objects may filter the amount of features capable of being generated. An example of an object may be “account,” “contact,” or “case.” By selecting multiple objects (e.g., “account” and “contact”), a more accurate listing of features may be generated, than if only one object (e.g., “case”) was selected. In other examples, thedatabase server 210 may receive an indication of one or more additional objects (e.g., a related object). In either example, thedatabase server 210 may receive the selection of the data set fromcomputing device 205 viaconnection 220. The selection of the object or data set may be facilitated by the user interface 235 such that a user can click or otherwise select an existing object within the data set. - In some examples, user input may allow for a data set to be segmented. For example, a user may wish to generate a predictive machine learning model based on a specific set of data. Thus
database server 210 may receive a selection of one or more fields (e.g., an indication of a specific data set), which may be indicative of specific types or subsets of structured data. For example, a user may wish to segment the data based on a field such as “contact opportunities” (e.g., leads) that are “worth more than $1 million.” Thus, this parameter may be taken into account when generating the predictive machine learning model. The data set segmentation may occur at data segmentation field selector 270. - Upon receiving the selection of the data set,
database server 210 may receive a selection of a prediction field from the plurality of fields. A prediction field may refer to the field of data for which the user wishes thedatabase server 210 to perform a prediction. As described above, the plurality of fields included in the data set may be or may be indicative of specific types or subsets of structured data. Thus, in the example of the object “contact,” an example of an associated field may be “e-mail address.” The selection of the prediction field may occur at prediction field selector 250. - In other examples, user input may allow for an indication of example data for use in generating a predictive machine learning model. Stated another way, a user may indicate which fields include accurate data, and which fields to predict. Thus, a predictive machine learning model may be based on scenarios that previously occurred (e.g., using the accurate data). For example, a user may indicate that the fields “contact opportunities” and “worth more than $1 million.” The user may indicate one or more prior contacts that closed with sales of greater than $1 million (e.g., accurate data). The user may also indicate that he or she wishes to use these examples (e.g., prior examples) as example data for use in predicting a lead (e.g., a suitable client). Thus the user may also indicate that he or she wishes to use the “contact opportunities” field to predict the lead. The accurate data may be taken into account when generating the predictive machine learning model. This data selection may occur at example data field selector 275.
- The
database server 210 may, in some examples, automatically generate a plurality of features from the data set after receiving the selection of the prediction field. The generation of the plurality of features may occur atfeature generator 255. Thedatabase server 210 may generate the plurality of features automatically based in part on metadata associated with the data set. The metadata may include a characteristic of the data and a data type. For example, the type of data may be a string and the characteristic of the data may be that the string is an email address. As another example, the data type of the data may be a number of integer and the characteristic of the data may be that the number is a phone number. The metadata may be stored on and received by thedatabase server 210, for example, fromdatabase 215 viaconnection 225. The metadata stored indatabase 215 may be available because the data may be stored in a CRM application that analyzes and understands these characteristics of the data. In some examples,database 215 may be an example of a multi-tenant database. - The
database server 210 may then automatically evaluate and potentially remove certain features that where automatically generated from the predictive model. For example, thedatabase server 210 may evaluate a statistical relationship between the plurality of predicted values for the prediction field and a subset of the plurality of features. Thedatabase server 210 may remove at least one feature of the subset of the plurality of features from the predictive machine learning model if the statistical relationship exceeds a threshold that indicates that the feature is unsuitable for inclusion in the model. Stated alternatively, thedatabase server 210 may determine if any of the features are outliers or are too predictive of the predictive value. This determination may be based on a threshold value—for example, whether too high or too low of a correlation between the predicted values and the subset of features exist. In such an instance, for example, thedatabase server 210 may remove the feature of the subset from the predictive learning machine, which may result in fewer data outliers and, subsequently, a more-accurate prediction model. - In some examples, the
database server 210 may adjust a sampling rate of the data to balance out relative representations of the data. For example, thedatabase server 210 may oversample data that predicts underrepresented predictive value. For example, if a relatively small set of the data set is predictive of one predictive outcome, then this small set of data may be oversampled with respect to other data in the data set such that the predictive model is not biased towards the predictive outcome represented by the relatively larger set of data. This automatic balancing of the data may result in a more accurate predictive model. - In other examples, the
database server 210 may automatically generate a predictive learning model based in part on the plurality of features. For example, thedatabase server 210 may generate the predictive learning model at learning model generator 260. In other examples, thedatabase server 210 may generate the predictive learning model based on training a plurality of candidate machine learning models. The plurality of candidate machine learning models may include any number of machine learning algorithms used in predictive model building. Thedatabase server 210 may then evaluate the plurality of candidate machine learning models. This evaluation may be based in part on a predictive accuracy of each of the machines. Thedatabase server 210 may subsequently select the predictive machine learning model based in part on the evaluation (e.g., based on which model is most accurate, or is otherwise best suited for the selected data set or the desired predictive value). - Upon generating the predictive learning model, as described above, the
database server 210 may generate scores based on the model (e.g., predictive values for selected field). In some examples, thedatabase server 210 may also generate an explanation as to why a particular score was generated. For example, a score may relate to a risk of attrition and the explanation (e.g., reasoning) for the given score may be that “the individual has opened bank accounts with external companies.” Thedatabase server 210 may then transmit an indication of the predicted values for the prediction field based in part on the predictive machine learning model. In some examples, thedatabase server 210 may perform one or more statistical analyses on the predictive machine learning model prior to transmitting the indication of the values. For example, the statistical analyses may aid in a determination of the viability of the model. Based on the statistical analyses, thedatabase server 210 may determine whether the model contains requisite information to generate accurate predictive scores. Thedatabase server 210 may transmit the plurality of predicted values by predicted value transmitter 265, byconnection 220, tocomputing device 205. In some examples, the score may be determined based in part on selecting the predictive learning model from the plurality of candidate machine learning models. In some examples, thedatabase server 210 may receive a new record associated with the data set. For example, additional data may be received fromdatabase 215. Thedatabase server 210 may then update the score for the prediction field based in part on the new record. This score may be stored at, for example, thedatabase server 210 or thedatabase 215. In some examples, the plurality of predicted scores, the plurality of predicted values, or both may be saved to thedatabase server 210 or thedatabase 215 based in part on transmitting the indication of the plurality of predicted values. -
FIG. 3 illustrates an example of asystem 300 that supports generating a predictive machine learning model in accordance with various aspects of the present disclosure. Thesystem 300 may include acomputing device 305, which may be an example of acomputing device 205 as described with reference toFIG. 2 . Thecomputing device 305 ofsystem 200 may include a user interface 310, which may be an example of the user interface 235 as described with reference toFIG. 2 . The user interface 310 may include input fields 315, 320, 325, 330, and 345, 355, and 360 which may correspond to input fields for a prediction name, an API name, object prediction, field prediction, a data filter, a data segmentation field selector, and an example data field selector, respectively. The user interface 310 may also includedisplay fields - As described above, the
computing device 305—via the user interface 310—may receive a selection of a data set and a selection of a prediction field. In some examples, the user interface 310 may be or may be an example of a wizard and may direct a user in a step-by-step input process. For example, the user may first input aprediction name 315, anAPI name 320, or both to identify the prediction. Subsequently, for example, the user may input a selection of a data set that may include, for example, anobject prediction 325, afield prediction 330, or both. This may be an example of a user inputting a selection of a data set, a selection of a prediction field from a plurality of fields, or both. In some examples, the user may manually input these fields, or may select input from a pre-populated list. When using a wizard for example, a user may be automatically navigated through multiple interfaces, each containing one or more input field. In such an example, each subsequent input field in the wizard may depend on the input received in the previous field. For example, a user may select a data set and, at a subsequent stage, may select a prediction field from a pre-populated list based on the data set selected. - In other examples, a user may input a prediction save
location 340 and may also choose to filter 345 the data to be received. For example, a user may select a particular data set, a particular location (e.g., local to computing device 305) or a database, or some other location to save the prediction to. In the instance that the prediction is saved to a particular data set, future predictions using the same data set may be improved. Stated alternatively, the predictive machine learning model may utilize the prediction or data associated with the prediction to improve its accuracy in generating a future prediction. - Upon receiving the user-defined input, the input may be received at a database server (e.g.,
database server 210 as described with reference toFIG. 2 ). The database server may, for example, generate a plurality of features from the data set. In some examples the database server may generate the plurality of features automatically based in part on metadata associated with the data set. In some examples, the database server may generate a predictive learning model based in part on the plurality of features. The database sever may then transmit an indication of a plurality of predicted values to thecomputing device 305. The indication may be displayed atprediction display 350. In other examples, the database server may transmit aconfidence prediction 335, which may indicate to a user a confidence level associated with the prediction. -
FIG. 4 illustrates an example of aprocess flow 400 for generating a predictive machine learning model in accordance with various aspects of the present disclosure. As described with reference toFIG. 2 , the steps ofprocess flow 400 may be performed by a database server or components of the database server, such as a predictive model platform. Theprocess flow 400 may include receiving adata set 405, receiving a selection of aprediction field 410, generating a plurality offeatures 415, generating alearning model 420, and transmitting predicted values 425. In some examples, process flow 400 may also include receiving a selection of a plurality offields 430, removing a subset ofdata 435,evaluation 440, removing one ormore features 445,training learning models 450, evaluatinglearning models 455, selectinglearning models 460, determiningscores 465, and saving the determined scores 470. - As described above with reference to
FIG. 2 , a database server (e.g., database server 210) may receive a selection of adata set 405. In some examples, the data set may be organized as an object, and may include an indication of one or more fields. Subsequently, for example, the database server may receive a selection of aprediction field 410 from the plurality of fields. The selection of aprediction field 410 may be, for example, based on the selected data set. Theprediction field 410 may refer to the field in the data set for which the user is building the predictive model. - In some examples, the database server may receive a selection of
fields 430 that the user selects to be included in the model building process. The user may select all available fields for the object or may select a subset of the available fields. For example, if the user wishes to not include certain data from the prediction (e.g., not include gender data in the prediction of a credit score), the user may select just a subset of the plurality offields 430. However, as described with reference toFIG. 2 , even if the user selects fields that eventually turn out to be causing inaccuracies in the predictive model, the database system may determine to not use those fields in the process of building the model. - In other examples, after receiving one or a plurality of
fields 430, the database server may balance the training data by removing a subset of thedata 435 or by adjusting a sampling rate of the data. In some examples, this step may be performed after the feature generation and removal steps discussed below. In other examples, this step may be performed before generating a plurality offeatures 415, as currently depicted inFIG. 4 . As discussed above, the data within certain fields (e.g., prediction fields) may cause an overrepresentation in the predictive nature of the model. As such, the data may be over-sampled or under-sampled with respect to other portions of the data to yield a more balanced ratio of positive and negative examples. This process may be referred to as balancing the data. In general, balancing the data may include adjusting a sampling rate of certain features or sets of data. - Upon receiving the selection of the
prediction field 410, the database server may generate a plurality offeatures 415, such as a plurality of relationships, functions, groupings or other associations that can be formed based on the data set. Such a procedure may be referred to as feature selection and engineering. The generation of the plurality offeatures 415 may be an example of generating a plurality of features from the data set. Thus, for example, the database server may generate the plurality of features automatically based in part on metadata associated with the data set. For example, the database server may recognize that a field of string data is associated with email addresses, and generate one or more features associated with the email addresses. Furthermore, because the metadata is available to the database server, a richer feature set may be generated than if only the raw data were provided. For example, the database server may be able to determine if the email addresses are valid or if the name in the email addresses match a name in some other field of the data set. - In some examples, the database server may evaluate the plurality of generated features and may remove some features from inclusion in the predicted model. Such a process may be referred to as sanity checking the feature generation. In some examples, the database server may evaluate a correlation 440 (or some other statistical relationship) between the plurality of predicted values (e.g., the prediction(s)) for the prediction field and a subset of the plurality of features. Upon conducting the
evaluation 440, the database server may remove at least onefeature 445 of the subset of the plurality of features from the predictive machine learning model if the correlation exceeds a threshold. The threshold may be an indication of the accuracy of the feature (or how the feature impacts the accuracy of the model) or the threshold may be indicative of how predictive the feature for the prediction field (e.g., if the feature is too predictive, the feature may bias the model towards the one feature). Upon determining that the threshold has been exceeded, one or more features may be removed and the learning model may be re-generated to determine a plurality of predicted values. This process may repeat until the threshold is not exceeded, which may ultimately result in more-accurate predicted values. - In some examples, the database server may then generate a
predictive learning model 420 based in part on the plurality of features. The predictive learning model may utilize the receiveddata set 405, the received selection of theprediction field 410, and the plurality offeatures 415 to generate themodel 420. In some instances, upon generating thelearning model 420, a plurality of candidate machine learning models may be trained 450. The plurality of candidatemachine learning models 450 may be trained, for example, based on one or more of the data set or the prediction field received by the database server. Upon training thecandidate machine models 450, the models may be evaluated based in part on a predictive accuracy of each of the plurality ofmodels 455. Stated alternatively, the models may be evaluated to determine a threshold level of accuracy given the input received by the database server. Accordingly, for example, a predictive learning machine model may be selected 460 based in part on the evaluating. Thus, in some examples, the predictive machine learning model that would result in a most-accurate prediction given the input received by the database server may be selected. - Upon generating the
predictive learning model 420, the database server may perform the predictions and transmit the predicted 425 values to, for example, a computing device. This may be an example of transmitting an indication of a plurality of predicted values for the prediction field based in part on the predictive machine learning model. In some examples, a score may be determined for theprediction field 465. This score may be based in part on generating thepredictive learning model 420. Thus, in some examples, the transmission of the predicted values 425 may be based on the score. The determined score may be based on a number of characteristics. For example, the score may be based on a number of predicted values or a correlation between a particular predicted value and the data set (e.g., the accuracy of the prediction). In some examples, the score may be saved 470 to the database server, which may be used in determining future predictions using a same or similar data set, prediction field(s), or both. -
FIG. 5 illustrates an example of asystem 500 that supports generating a predictive machine learning model in accordance with various aspects of the present disclosure. Thesystem 500 may includeTeam City 505, which may be an example of a continuous integration server;EP Provisioning 510, which may be an example of a server or, more specifically, a business process outsourcing (BPO) server;creation component 520, which may create a set of objects to utilize in generating a predictive machine learning model; updating andscheduling component 525, which may schedule and update system workflows; and objects 530, 535, and 540, which may be or may represent specific data or data sets. - In some examples,
EP Provisioning database 510 may obtain a list of predictive models to be generated. TheEP Provisioning database 510 may also obtain requisite data to generate one or more predictive models. - In some examples,
Team City 505 may run hourly to pull BPO configurations for all Builder tenants (e.g., of a multi-tenant database system) to get current and disabled configurations. In some examples, a whitelist is constructed and stored in theEP Provisioning database 510. In other examples,Team City 505 may run at a pre-defined time interval or at random time intervals to obtain configurations for one or more tenants. In some examples, the configurations may be associated with one or more specific tenants. In other examples the whitelist created based on the configurations may be stored in theEP Provisioning database 510. In other examples, the data puller job is executed hourly on all prediction objects to get all incremental changes to them since the last successful pull. In some examples,Team City 505 may be in communication withEP Provisioning database 510 viaconnection 515. In some examples,Team City 505 may receive data fromEP Provisioning database 510 more or less frequently in order to record incremental changes to the database. - Additionally or alternatively, for example a daily snapshotting job may be executed for all prediction objects. For example,
Team City 505 may monitorEP Provisioning database 510 at certain time intervals to ensure that any changes, no matter how incremental, are recorded.Team City 505 may not be limited to a daily snapshotting job. In other examples,Team City 505 may create and organize any number of jobs. For example,Team City 505 may determine and schedule one or more mathematical calculations to be performed on the data used to generate a predictive model.Team City 505 may then monitor the jobs (e.g., calculations) and schedule subsequent jobs accordingly. - In other examples, even if a same prediction object is referenced multiple times across different configurations, only a single set of data flows is set up. Additionally or alternatively, for example, even if a same prediction object is reference multiple times across different configurations, any number of data flows may be set up. The data flows may, for example, correspond to one or more prediction objects. In other examples, any one of
creation component 520,scheduling component 525, and objects 530, 535, and 540 may be utilized to generate a predictive machine learning model as described above with reference toFIGS. 1 through 4 . -
FIG. 6 illustrates an example of asystem 600 that supports generating a predictive machine learning model in accordance with various aspects of the present disclosure. Thesystem 600 may includeTeam City 605, which may be an example of Team City as described with reference toFIG. 5 ;EP Provisioning 610, which may be an example ofEP Provisioning 510 as described with reference toFIG. 5 ; configuration component 620, which may create scoring workflows for configurations;data store 625, which may be an example ofdatabase server 210 as described with reference toFIG. 2 ; updating andscheduling component 525, which may be an example of updating andscheduling component 525 as described with reference toFIG. 5 ; andtraining models Team City 605 may be in communication with EP Provisioning database viaconnection 615 anddata store 625 may transmit data or communications to configuration component 620 viatransmission 630. - In some examples,
FIG. 6 may illustrate model training flows for new configurations in theEP Provisioning database 610. In other examples,FIG. 6 may illustrate the monthly model retrain process for all configurations. In other examples,FIG. 6 may illustrate that all model training runs operate off of the latest snapshot of the prediction object. In other examples, any one ofTeam City 605,EP Provisioning database 610, configurations component 620,data store 625, updating andscheduling component 635, andtraining models FIGS. 1 through 5 . -
FIG. 7 illustrates an example of asystem 700 that supports generating a predictive machine learning model in accordance with various aspects of the present disclosure. Thesystem 700 may includeTeam City 705, which may be an example of Team City as described with reference toFIG. 5 ;EP Provisioning 710, which may be an example ofEP Provisioning 510 as described with reference toFIG. 5 ; scoring creation andworkflow component 720;data store 725, which may be an example of thedata store 625 as described with reference toFIG. 6 ; updating andscheduling component 730, which may be an example of updating andscheduling component 525 as described with reference toFIG. 5 ;training models training models FIG. 6 ;new score components server 745, which may be an example ofdatabase server 210 ordatabase 215 as described with reference toFIG. 2 . In some examples,Team City 605 may be in communication with EP Provisioning database viaconnection 715 anddata store 625 may transmit data or communications totraining models - In some examples,
FIG. 7 may illustrate scoring and data pushback flows. For example, scoring jobs may run on an hourly cadence scoring all increments pulled since the last dataset that was successfully scored. In other examples, scoring jobs may run periodically, randomly, or on a predetermined time frame to score any increments pulled since the dataset was last successfully scored. In other examples, data pushback jobs are chained off of scoring jobs and push back the last two scored datasets, in order, for redundancy in case data pushback failed the previous time. In other examples, data pushback jobs may be chained off of scoring jobs and may push back any number of scored datasets, in any order. In some examples, the data pushback jobs may be based on previously-successful data pushback. In other examples, any one ofTeam City 705,EP Provisioning database 710, scoring creation andworkflow component 720,data store 725, updating andscheduling component 730,training models new score components server 745 may be utilized to generate a predictive machine learning model as described above with reference toFIGS. 1 through 6 . -
FIG. 8 shows a block diagram 800 of anapparatus 805 that supports automatic machine learning model generation in accordance with aspects of the present disclosure.Apparatus 805 may includeinput module 810,predictive model builder 815, andoutput module 820.Apparatus 805 may also include a processor. Each of these components may be in communication with one another (e.g., via one or more buses). In some cases,apparatus 805 may be an example of a user terminal, a database server, or a system containing multiple computing devices. -
Predictive model builder 815 may be an example of aspects of thepredictive model builder 1015 described with reference toFIG. 10 . -
Predictive model builder 815 and/or at least some of its various sub-components may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions of thepredictive model builder 815 and/or at least some of its various sub-components may be executed by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described in the present disclosure. Thepredictive model builder 815 and/or at least some of its various sub-components may be physically located at various positions, including being distributed such that portions of functions are implemented at different physical locations by one or more physical devices. In some examples,predictive model builder 815 and/or at least some of its various sub-components may be a separate and distinct component in accordance with various aspects of the present disclosure. In other examples,predictive model builder 815 and/or at least some of its various sub-components may be combined with one or more other hardware components, including but not limited to an I/O component, a transceiver, a network server, another computing device, one or more other components described in the present disclosure, or a combination thereof in accordance with various aspects of the present disclosure. -
Predictive model builder 815 may also includedata selector 825,field selector 830,feature generator 835,model builder 840, andscoring component 845. -
Data selector 825 may receive, at a database server, a selection of a data set, where the data set includes a set of fields and receive an indication of one or more additional objects, where the indication of the set of predicted values is based on receiving the indication of the one or more additional objects. In some cases, the receiving the selection of the data set includes receiving an indication of an object, where the indication of the set of predicted values is based on receiving the indication of the object. In some cases, the database server includes a multi-tenant database server. -
Field selector 830 may receive, at the database server, a selection of a prediction field from the set of fields and receive a selection of prediction fields from the set of fields, where the set of features from the data set are generated based on the selection of prediction fields. -
Feature generator 835 may generate, by the database server, a set of features from the data set, where the set of features are generated automatically based on metadata associated with the data set. In some cases, the metadata associated with the data set includes a classification of a field and a data type of the field. -
Model builder 840 may generate, by the database server, the predictive machine learning model based on the set of features. - Scoring
component 845 may determine a score for the prediction field based on generating the predictive machine learning model, where transmitting the indication of the set of predicted values is based on the determined score, transmit an indication of a set of predicted values for the prediction field based on the predictive machine learning model, receive a new record associated with the data set, update the score for the prediction field based on the new record, and store the score in the database server. -
FIG. 9 shows a block diagram 900 of apredictive model builder 915 that supports automatic machine learning model generation in accordance with aspects of the present disclosure. Thepredictive model builder 915 may be an example of aspects of apredictive model builder 1015 described with reference toFIGS. 7, 8, and 10 . Thepredictive model builder 915 may includedata selector 920,field selector 925,feature generator 930,model builder 935, scoringcomponent 940,feature checking component 945,model selection component 950,data balancing component 955, and scoreoutput component 960. Each of these modules may communicate, directly or indirectly, with one another (e.g., via one or more buses). -
Data selector 920 may receive, at a database server, a selection of a data set, where the data set includes a set of fields and receive an indication of one or more additional objects, where the indication of the set of predicted values is based on receiving the indication of the one or more additional objects. In some cases, the receiving the selection of the data set includes receiving an indication of an object, where the indication of the set of predicted values is based on receiving the indication of the object. In some cases, the database server includes a multi-tenant database server. -
Field selector 925 may receive, at the database server, a selection of a prediction field from the set of fields and receive a selection of prediction fields from the set of fields, where the set of features from the data set are generated based on the selection of prediction fields. -
Feature generator 930 may generate, by the database server, a set of features from the data set, where the set of features are generated automatically based on metadata associated with the data set. In some cases, the metadata associated with the data set includes a classification of a field and a data type of the field. -
Model builder 935 may generate, by the database server, the predictive machine learning model based on the set of features. - Scoring
component 940 may determine a score for the prediction field based on generating the predictive machine learning model, where transmitting the indication of the set of predicted values is based on the determined score, transmit an indication of a set of predicted values for the prediction field based on the predictive machine learning model, receive a new record associated with the data set, update the score for the prediction field based on the new record, and store the score in the database server. -
Feature checking component 945 may evaluate a statistical relationship between the set of predicted values for the prediction field and a subset of the set of features and remove at least one feature of the subset of the set of features from the predictive machine learning model if the statistical relationship exceeds a threshold. In some cases, the statistical relationship includes a correlation calculation, a Cramer's V calculation, or a combination thereof. -
Model selection component 950 may train a set of candidate machine learning models, evaluate the set of candidate machine learning models based on a predictive accuracy of each of the set of candidate machine learning models, and select the predictive machine learning model based on the evaluating. -
Data balancing component 955 may adjust a sampling rate of the data set based on a representation rate of the set of predicted values. -
Score output component 960 may save the set of predicted values to the prediction field based on transmitting the indication of the set of predicted values. -
FIG. 10 shows a diagram of asystem 1000 including adevice 1005 that supports automatic machine learning model generation in accordance with aspects of the present disclosure.Device 1005 may be an example of or include the components ofdata center 120 ordatabase server 210 as described above, e.g., with reference toFIG. 1 .Device 1005 may include components for bi-directional data communications including components for transmitting and receiving communications, includingpredictive model builder 1015,processor 1020,memory 1025,database controller 1030,database 1035, and I/O controller 1040. These components may be in electronic communication via one or more buses (e.g., bus 1010). -
Processor 1020 may include an intelligent hardware device, (e.g., a general-purpose processor, a DSP, a central processing unit (CPU), a microcontroller, an ASIC, an FPGA, a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases,processor 1020 may be configured to operate a memory array using a memory controller. In other cases, a memory controller may be integrated intoprocessor 1020.Processor 1020 may be configured to execute computer-readable instructions stored in a memory to perform various functions (e.g., functions or tasks supporting automatic machine learning model generation). -
Memory 1025 may include random access memory (RAM) and read only memory (ROM). Thememory 1025 may store computer-readable, computer-executable software 1030 including instructions that, when executed, cause the processor to perform various functions described herein. In some cases, thememory 1025 may contain, among other things, a basic input/output system (BIOS) which may control basic hardware or software operation such as the interaction with peripheral components or devices. -
Database controller 1030 may manage data storage and processing indatabase 1035. In some cases, a user may interact withdatabase controller 1030. In other cases,database controller 1030 may operate automatically without user interaction. -
Database 1035 may be an example of a single database, a distributed database, multiple distributed databases, or an emergency backup database. - I/
O controller 1040 may manage input and output signals fordevice 1005. I/O controller 1040 may also manage peripherals not integrated intodevice 1005. In some cases, I/O controller 1040 may represent a physical connection or port to an external peripheral. In some cases, I/O controller 1040 may utilize an operating system such as iOS®, ANDROID®, MS-DOS®, MS-WINDOWS®, OS/2®, UNIX®, LINUX®, or another known operating system. In other cases, I/O controller 1040 may represent or interact with a modem, a keyboard, a mouse, a touchscreen, or a similar device. In some cases, I/O controller 1040 may be implemented as part of a processor. In some cases, a user may interact withdevice 1005 via I/O controller 1040 or via hardware components controlled by I/O controller 1040. -
FIG. 11 shows a flowchart illustrating amethod 1100 for automatic machine learning model generation in accordance with aspects of the present disclosure. The operations ofmethod 1100 may be implemented by adata center 120 ordatabase server 210 or its components as described herein. For example, the operations ofmethod 1100 may be performed by a predictive model builder as described with reference toFIGS. 8 through 10 . In some examples, adata center 120 ordatabase server 210 may execute a set of codes to control the functional elements of the device to perform the functions described below. Additionally or alternatively, thedata center 120 ordatabase server 210 may perform aspects of the functions described below using special-purpose hardware. - At 1105 the
data center 120 ordatabase server 210 may receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields. The operations of 1105 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1105 may be performed by a data selector as described with reference toFIGS. 8 through 10 . - At 1110 the
data center 120 ordatabase server 210 may receive, at the database server, a selection of a prediction field from the plurality of fields. The operations of 1110 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1110 may be performed by a field selector as described with reference toFIGS. 8 through 10 . - At 1115 the
data center 120 ordatabase server 210 may generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set. The operations of 1115 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1115 may be performed by a feature generator as described with reference toFIGS. 8 through 10 . - At 1120 the
data center 120 ordatabase server 210 may generate, by the database server, the predictive machine learning model based at least in part on the plurality of features. The operations of 1120 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1120 may be performed by a model builder as described with reference toFIGS. 8 through 10 . - At 1125 the
data center 120 ordatabase server 210 may transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model. The operations of 1125 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1125 may be performed by a scoring component as described with reference toFIGS. 8 through 10 . -
FIG. 12 shows a flowchart illustrating amethod 1200 for automatic machine learning model generation in accordance with aspects of the present disclosure. The operations ofmethod 1200 may be implemented by adata center 120 ordatabase server 210 or its components as described herein. For example, the operations ofmethod 1200 may be performed by a predictive model builder as described with reference toFIGS. 8 through 10 . In some examples, adata center 120 ordatabase server 210 may execute a set of codes to control the functional elements of the device to perform the functions described below. Additionally or alternatively, thedata center 120 ordatabase server 210 may perform aspects of the functions described below using special-purpose hardware. - At 1205 the
data center 120 ordatabase server 210 may receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields. The operations of 1205 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1205 may be performed by a data selector as described with reference toFIGS. 8 through 10 . - At 1210 the
data center 120 ordatabase server 210 may receive, at the database server, a selection of a prediction field from the plurality of fields. The operations of 1210 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1210 may be performed by a field selector as described with reference toFIGS. 8 through 10 . - At 1215 the
data center 120 ordatabase server 210 may generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set. The operations of 1215 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1215 may be performed by a feature generator as described with reference toFIGS. 8 through 10 . - At 1220 the
data center 120 ordatabase server 210 may evaluate a statistical relationship between the plurality of predicted values for the prediction field and a subset of the plurality of features. The operations of 1220 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1220 may be performed by a feature checking component as described with reference toFIGS. 8 through 10 . - At 1225 the
data center 120 ordatabase server 210 may remove at least one feature of the subset of the plurality of features from the predictive machine learning model if the statistical relationship exceeds a threshold. The operations of 1225 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1225 may be performed by a feature checking component as described with reference toFIGS. 8 through 10 . - At 1230 the
data center 120 ordatabase server 210 may generate, by the database server, the predictive machine learning model based at least in part on the plurality of features. The operations of 1230 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1230 may be performed by a model builder as described with reference toFIGS. 8 through 10 . - At 1235 the
data center 120 ordatabase server 210 may transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model. The operations of 1235 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1235 may be performed by a scoring component as described with reference toFIGS. 8 through 10 . -
FIG. 13 shows a flowchart illustrating amethod 1300 for automatic machine learning model generation in accordance with aspects of the present disclosure. The operations ofmethod 1300 may be implemented by adata center 120 ordatabase server 210 or its components as described herein. For example, the operations ofmethod 1300 may be performed by a predictive model builder as described with reference toFIGS. 8 through 10 . In some examples, adata center 120 ordatabase server 210 may execute a set of codes to control the functional elements of the device to perform the functions described below. Additionally or alternatively, thedata center 120 ordatabase server 210 may perform aspects of the functions described below using special-purpose hardware. - At 1305 the
data center 120 ordatabase server 210 may receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields. The operations of 1305 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1305 may be performed by a data selector as described with reference toFIGS. 8 through 10 . - At 1310 the
data center 120 ordatabase server 210 may receive, at the database server, a selection of a prediction field from the plurality of fields. The operations of 1310 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1310 may be performed by a field selector as described with reference toFIGS. 8 through 10 . - At 1315 the
data center 120 ordatabase server 210 may generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set. The operations of 1315 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1315 may be performed by a feature generator as described with reference toFIGS. 8 through 10 . - At 1320 the
data center 120 ordatabase server 210 may train a plurality of candidate machine learning models. The operations of 1320 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1320 may be performed by a model selection component as described with reference toFIGS. 8 through 10 . - At 1325 the
data center 120 ordatabase server 210 may evaluate the plurality of candidate machine learning models based at least in part on a predictive accuracy of each of the plurality of candidate machine learning models. The operations of 1325 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1325 may be performed by a model selection component as described with reference toFIGS. 8 through 10 . - At 1330 the
data center 120 ordatabase server 210 may select the predictive machine learning model based at least in part on the evaluating. The operations of 1330 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1330 may be performed by a model selection component as described with reference toFIGS. 8 through 10 . - At 1335 the
data center 120 ordatabase server 210 may generate, by the database server, the predictive machine learning model based at least in part on the plurality of features. The operations of 1335 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1335 may be performed by a model builder as described with reference toFIGS. 8 through 10 . - At 1340 the
data center 120 ordatabase server 210 may transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model. The operations of 1340 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1340 may be performed by a scoring component as described with reference toFIGS. 8 through 10 . -
FIG. 14 shows a flowchart illustrating amethod 1400 for automatic machine learning model generation in accordance with aspects of the present disclosure. The operations ofmethod 1400 may be implemented by adata center 120 ordatabase server 210 or its components as described herein. For example, the operations ofmethod 1400 may be performed by a predictive model builder as described with reference toFIGS. 8 through 10 . In some examples, adata center 120 ordatabase server 210 may execute a set of codes to control the functional elements of the device to perform the functions described below. Additionally or alternatively, thedata center 120 ordatabase server 210 may perform aspects of the functions described below using special-purpose hardware. - At 1405 the
data center 120 ordatabase server 210 may receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields. The operations of 1405 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1405 may be performed by a data selector as described with reference toFIGS. 8 through 10 . - At 1410 the
data center 120 ordatabase server 210 may receive, at the database server, a selection of a prediction field from the plurality of fields. The operations of 1410 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1410 may be performed by a field selector as described with reference toFIGS. 8 through 10 . - At 1415 the
data center 120 ordatabase server 210 may generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set. The operations of 1415 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1415 may be performed by a feature generator as described with reference toFIGS. 8 through 10 . - At 1420 the
data center 120 ordatabase server 210 may adjust a sampling rate of the data set based at least in part on a representation rate of the plurality of predicted values. The operations of 1420 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1420 may be performed by a data balancing component as described with reference toFIGS. 8 through 10 . - At 1425 the
data center 120 ordatabase server 210 may generate, by the database server, the predictive machine learning model based at least in part on the plurality of features. The operations of 1425 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1425 may be performed by a model builder as described with reference toFIGS. 8 through 10 . - At 1430 the
data center 120 ordatabase server 210 may transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model. The operations of 1430 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1430 may be performed by a scoring component as described with reference toFIGS. 8 through 10 . -
FIG. 15 shows a flowchart illustrating amethod 1500 for automatic machine learning model generation in accordance with aspects of the present disclosure. The operations ofmethod 1500 may be implemented by adata center 120 ordatabase server 210 or its components as described herein. For example, the operations ofmethod 1500 may be performed by a predictive model builder as described with reference toFIGS. 8 through 10 . In some examples, adata center 120 ordatabase server 210 may execute a set of codes to control the functional elements of the device to perform the functions described below. Additionally or alternatively, thedata center 120 ordatabase server 210 may perform aspects of the functions described below using special-purpose hardware. - At 1505 the
data center 120 ordatabase server 210 may receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields. The operations of 1505 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1505 may be performed by a data selector as described with reference toFIGS. 8 through 10 . - At 1510 the
data center 120 ordatabase server 210 may receive, at the database server, a selection of a prediction field from the plurality of fields. The operations of 1510 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1510 may be performed by a field selector as described with reference toFIGS. 8 through 10 . - At 1515 the
data center 120 ordatabase server 210 may generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set. The operations of 1515 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1515 may be performed by a feature generator as described with reference toFIGS. 8 through 10 . - At 1520 the
data center 120 ordatabase server 210 may determine a score for the prediction field based at least in part on generating the predictive machine learning model, wherein transmitting the indication of the plurality of predicted values is based at least in part on the determined score. The operations of 1520 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1520 may be performed by a scoring component as described with reference toFIGS. 8 through 10 . - At 1525 the
data center 120 ordatabase server 210 may generate, by the database server, the predictive machine learning model based at least in part on the plurality of features. The operations of 1525 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1525 may be performed by a model builder as described with reference toFIGS. 8 through 10 . - At 1530 the
data center 120 ordatabase server 210 may transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model. The operations of 1530 may be performed according to the methods described herein. In certain examples, aspects of the operations of 1530 may be performed by a scoring component as described with reference toFIGS. 8 through 10 . - A method of generating a predictive machine learning model is described. The method may include receiving, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields, receiving, at the database server, a selection of a prediction field from the plurality of fields, generating, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set, generating, by the database server, the predictive machine learning model based at least in part on the plurality of features, and transmitting an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- An apparatus for generating a predictive machine learning model is described. The apparatus may include means for receiving, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields, means for receiving, at the database server, a selection of a prediction field from the plurality of fields, means for generating, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set, means for generating, by the database server, the predictive machine learning model based at least in part on the plurality of features, and means for transmitting an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- Another apparatus for generating a predictive machine learning model is described. The apparatus may include a processor, memory in electronic communication with the processor, and instructions stored in the memory. The instructions may be operable to cause the processor to receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields, receive, at the database server, a selection of a prediction field from the plurality of fields, generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set, generate, by the database server, the predictive machine learning model based at least in part on the plurality of features, and transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- A non-transitory computer-readable medium for generating a predictive machine learning model is described. The non-transitory computer-readable medium may include instructions operable to cause a processor to receive, at a database server, a selection of a data set, wherein the data set comprises a plurality of fields, receive, at the database server, a selection of a prediction field from the plurality of fields, generate, by the database server, a plurality of features from the data set, wherein the plurality of features are generated automatically based at least in part on metadata associated with the data set, generate, by the database server, the predictive machine learning model based at least in part on the plurality of features, and transmit an indication of a plurality of predicted values for the prediction field based at least in part on the predictive machine learning model.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for evaluating a statistical relationship between the plurality of predicted values for the prediction field and a subset of the plurality of features. Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for removing at least one feature of the subset of the plurality of features from the predictive machine learning model if the statistical relationship exceeds a threshold.
- In some examples of the method, apparatus, and non-transitory computer-readable medium described above, the statistical relationship comprises a correlation calculation, a Cramer's V calculation, or a combination thereof.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for training a plurality of candidate machine learning models. Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for evaluating the plurality of candidate machine learning models based at least in part on a predictive accuracy of each of the plurality of candidate machine learning models. Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for selecting the predictive machine learning model based at least in part on the evaluating.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for adjusting a sampling rate of the data set based at least in part on a representation rate of the plurality of predicted values.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for determining a score for the prediction field based at least in part on generating the predictive machine learning model, wherein transmitting the indication of the plurality of predicted values may be based at least in part on the determined score.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for receiving a new record associated with the data set. Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for updating the score for the prediction field based at least in part on the new record.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for storing the score in the database server.
- In some examples of the method, apparatus, and non-transitory computer-readable medium described above, the receiving the selection of the data set comprises receiving an indication of an object, wherein the indication of the plurality of predicted values may be based at least in part on receiving the indication of the object.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for receiving an indication of one or more additional objects, wherein the indication of the plurality of predicted values may be based at least in part on receiving the indication of the one or more additional objects.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for receiving a selection of prediction fields from the plurality of fields, wherein the plurality of features from the data set may be generated based at least in part on the selection of prediction fields.
- Some examples of the method, apparatus, and non-transitory computer-readable medium described above may further include processes, features, means, or instructions for saving the plurality of predicted values to the prediction field based at least in part on transmitting the indication of the plurality of predicted values.
- In some examples of the method, apparatus, and non-transitory computer-readable medium described above, the database server comprises a multi-tenant database server.
- In some examples of the method, apparatus, and non-transitory computer-readable medium described above, the metadata associated with the data set comprises a classification of a field and a data type of the field.
- It should be noted that the methods described above describe possible implementations, and that the operations and the steps may be rearranged or otherwise modified and that other implementations are possible. Furthermore, aspects from two or more of the methods may be combined.
- The description set forth herein, in connection with the appended drawings, describes example configurations and does not represent all the examples that may be implemented or that are within the scope of the claims. The term “exemplary” used herein means “serving as an example, instance, or illustration,” and not “preferred” or “advantageous over other examples.” The detailed description includes specific details for the purpose of providing an understanding of the described techniques. These techniques, however, may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the described examples.
- In the appended figures, similar components or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. If just the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.
- Information and signals described herein may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
- The various illustrative blocks and modules described in connection with the disclosure herein may be implemented or performed with a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e.g., a combination of a digital signal processor (DSP) and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration).
- The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Other examples and implementations are within the scope of the disclosure and appended claims. For example, due to the nature of software, functions described above can be implemented using software executed by a processor, hardware, firmware, hardwiring, or combinations of any of these. Features implementing functions may also be physically located at various positions, including being distributed such that portions of functions are implemented at different physical locations. Also, as used herein, including in the claims, “or” as used in a list of items (for example, a list of items prefaced by a phrase such as “at least one of” or “one or more of”) indicates an inclusive list such that, for example, a list of at least one of A, B, or C means A or B or C or AB or AC or BC or ABC (i.e., A and B and C). Also, as used herein, the phrase “based on” shall not be construed as a reference to a closed set of conditions. For example, an exemplary step that is described as “based on condition A” may be based on both a condition A and a condition B without departing from the scope of the present disclosure. In other words, as used herein, the phrase “based on” shall be construed in the same manner as the phrase “based at least in part on.”
- Computer-readable media includes both non-transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A non-transitory storage medium may be any available medium that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, non-transitory computer-readable media can comprise RAM, ROM, electrically erasable programmable read only memory (EEPROM), compact disk (CD) ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include CD, laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of computer-readable media.
- The description herein is provided to enable a person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the scope of the disclosure. Thus, the disclosure is not limited to the examples and designs described herein, but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.
Claims (20)
1. A method for data processing, comprising:
receiving a selection of a dataset, a prediction field from the dataset, and a set of parameters associated with the prediction field;
processing the dataset based at least in part on sampling the dataset according to a sampling rate, removing a portion of the dataset, filtering out a set of outliers from the dataset, using metadata associated with the dataset to generate features for the dataset, and segmenting the dataset into a first subset of training data and a second subset of evaluation data;
training a plurality of candidate predictive machine learning models using the first subset of training data from the processed dataset and the set of parameters associated with the prediction field;
evaluating a predictive accuracy of the plurality of candidate predictive machine learning models using the second subset of evaluation data from the processed dataset;
transmitting an indication of the plurality of candidate predictive machine learning models and the predictive accuracy of the plurality of candidate predictive machine learning models;
receiving a selection of a first predictive machine learning model from the plurality of candidate predictive machine learning models; and
predicting one or more values for the prediction field using the first predictive machine learning model.
2. The method of claim 1 , further comprising:
determining respective scores for the plurality of candidate predictive machine learning models based at least in part on performing a statistical analysis of the predictive accuracy of the plurality of candidate predictive machine learning models.
3. The method of claim 2 , wherein the respective scores for the plurality of candidate predictive machine learning models are displayed on the user interface of the client device.
4. The method of claim 1 , further comprising:
determining a correlation between the one or more values generated by the first predictive machine learning model and actual values from the dataset.
5. The method of claim 4 , further comprising:
displaying the correlation.
6. The method of claim 1 , wherein the metadata indicates respective data types and characteristics of fields in the dataset.
7. The method of claim 1 , wherein the plurality of candidate predictive machine learning models are trained using parameters defined by a user of a client device.
8. An apparatus for data processing, comprising:
a processor;
memory coupled with the processor; and
instructions stored in the memory and executable by the processor to cause the apparatus to:
receive a selection of a dataset, a prediction field from the dataset, and a set of parameters associated with the prediction field;
process the dataset based at least in part on sampling the dataset according to a sampling rate, removing a portion of the dataset, filtering out a set of outliers from the dataset, using metadata associated with the dataset to generate features for the dataset, and segmenting the dataset into a first subset of training data and a second subset of evaluation data;
train a plurality of candidate predictive machine learning models using the first subset of training data from the processed dataset and the set of parameters associated with the prediction field;
evaluate a predictive accuracy of the plurality of candidate predictive machine learning models using the second subset of evaluation data from the processed dataset;
transmit an indication of the plurality of candidate predictive machine learning models and the predictive accuracy of the plurality of candidate predictive machine learning models;
receive a selection of a first predictive machine learning model from the plurality of candidate predictive machine learning models; and
predict one or more values for the prediction field using the first predictive machine learning model.
9. The apparatus of claim 8 , wherein the instructions are further executable by the processor to cause the apparatus to:
determine respective scores for the plurality of candidate predictive machine learning models based at least in part on performing a statistical analysis of the predictive accuracy of the plurality of candidate predictive machine learning models.
10. The apparatus of claim 9 , wherein the respective scores for the plurality of candidate predictive machine learning models are displayed on the user interface of the client device.
11. The apparatus of claim 8 , wherein the instructions are further executable by the processor to cause the apparatus to:
determine a correlation between the one or more values generated by the first predictive machine learning model and actual values from the dataset.
12. The apparatus of claim 11 , wherein the instructions are further executable by the processor to cause the apparatus to:
display the correlation.
13. The apparatus of claim 8 , wherein the metadata indicates respective data types and characteristics of fields in the dataset.
14. The apparatus of claim 8 , wherein the plurality of candidate predictive machine learning models are trained using parameters defined by a user of a client device.
15. A non-transitory computer-readable medium storing code for data processing, the code comprising instructions executable by a processor to:
receive a selection of a dataset, a prediction field from the dataset, and a set of parameters associated with the prediction field;
process the dataset based at least in part on sampling the dataset according to a sampling rate, removing a portion of the dataset, filtering out a set of outliers from the dataset, using metadata associated with the dataset to generate features for the dataset, and segmenting the dataset into a first subset of training data and a second subset of evaluation data;
train a plurality of candidate predictive machine learning models using the first subset of training data from the processed dataset and the set of parameters associated with the prediction field;
evaluate a predictive accuracy of the plurality of candidate predictive machine learning models using the second subset of evaluation data from the processed dataset;
transmit an indication of the plurality of candidate predictive machine learning models and the predictive accuracy of the plurality of candidate predictive machine learning models;
receive a selection of a first predictive machine learning model from the plurality of candidate predictive machine learning models; and
predict one or more values for the prediction field using the first predictive machine learning model.
16. The non-transitory computer-readable medium of claim 15 , wherein the instructions are further executable by the processor to:
determine respective scores for the plurality of candidate predictive machine learning models based at least in part on performing a statistical analysis of the predictive accuracy of the plurality of candidate predictive machine learning models.
17. The non-transitory computer-readable medium of claim 16 , wherein the respective scores for the plurality of candidate predictive machine learning models are displayed on the user interface of the client device.
18. The non-transitory computer-readable medium of claim 15 , wherein the instructions are further executable by the processor to:
determine a correlation between the one or more values generated by the first predictive machine learning model and actual values from the dataset.
19. The non-transitory computer-readable medium of claim 18 , wherein the instructions are further executable by the processor to:
display the correlation.
20. The non-transitory computer-readable medium of claim 15 , wherein the metadata indicates respective data types and characteristics of fields in the dataset.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/196,963 US20230334367A1 (en) | 2017-11-03 | 2023-05-12 | Automatic machine learning model generation |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201762581624P | 2017-11-03 | 2017-11-03 | |
US15/884,878 US11663517B2 (en) | 2017-11-03 | 2018-01-31 | Automatic machine learning model generation |
US18/196,963 US20230334367A1 (en) | 2017-11-03 | 2023-05-12 | Automatic machine learning model generation |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/884,878 Continuation US11663517B2 (en) | 2017-11-03 | 2018-01-31 | Automatic machine learning model generation |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230334367A1 true US20230334367A1 (en) | 2023-10-19 |
Family
ID=66327370
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/884,878 Active 2042-01-08 US11663517B2 (en) | 2017-11-03 | 2018-01-31 | Automatic machine learning model generation |
US18/196,963 Pending US20230334367A1 (en) | 2017-11-03 | 2023-05-12 | Automatic machine learning model generation |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/884,878 Active 2042-01-08 US11663517B2 (en) | 2017-11-03 | 2018-01-31 | Automatic machine learning model generation |
Country Status (1)
Country | Link |
---|---|
US (2) | US11663517B2 (en) |
Families Citing this family (27)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11062792B2 (en) | 2017-07-18 | 2021-07-13 | Analytics For Life Inc. | Discovering genomes to use in machine learning techniques |
US11139048B2 (en) | 2017-07-18 | 2021-10-05 | Analytics For Life Inc. | Discovering novel features to use in machine learning techniques, such as machine learning techniques for diagnosing medical conditions |
US20190197011A1 (en) * | 2017-12-22 | 2019-06-27 | At&T Intellectual Property I, L.P. | Network-based machine learning model discovery and benchmarking |
US11568302B2 (en) * | 2018-04-09 | 2023-01-31 | Veda Data Solutions, Llc | Training machine learning algorithms with temporally variant personal data, and applications thereof |
US11669724B2 (en) * | 2018-05-17 | 2023-06-06 | Raytheon Company | Machine learning using informed pseudolabels |
US11651314B1 (en) * | 2018-06-26 | 2023-05-16 | Gbt Travel Services Uk Limited | Determining customer attrition risk |
US11288699B2 (en) * | 2018-07-13 | 2022-03-29 | Pubwise, LLLP | Digital advertising platform with demand path optimization |
US10970486B2 (en) | 2018-09-18 | 2021-04-06 | Salesforce.Com, Inc. | Using unstructured input to update heterogeneous data stores |
US11436481B2 (en) | 2018-09-18 | 2022-09-06 | Salesforce.Com, Inc. | Systems and methods for named entity recognition |
US11836612B2 (en) * | 2019-06-18 | 2023-12-05 | Sap Se | Maintaining master data using hierarchical classification |
CN110263938B (en) * | 2019-06-19 | 2021-07-23 | 北京百度网讯科技有限公司 | Method and apparatus for generating information |
US11656903B2 (en) * | 2019-06-25 | 2023-05-23 | Intel Corporation | Methods and apparatus to optimize workflows |
EP3786855A1 (en) * | 2019-08-30 | 2021-03-03 | Accenture Global Solutions Limited | Automated data processing and machine learning model generation |
US10963231B1 (en) | 2019-10-15 | 2021-03-30 | UiPath, Inc. | Using artificial intelligence to select and chain models for robotic process automation |
US20210150412A1 (en) * | 2019-11-20 | 2021-05-20 | The Regents Of The University Of California | Systems and methods for automated machine learning |
US11475239B2 (en) * | 2019-11-21 | 2022-10-18 | Paypal, Inc. | Solution to end-to-end feature engineering automation |
EP3828725A1 (en) * | 2019-11-27 | 2021-06-02 | Ubimax GmbH | Method of performing a process using artificial intelligence |
US11354268B2 (en) * | 2020-01-08 | 2022-06-07 | EMC IP Holding Company LLC | Optimizing snapshot creation |
WO2021167997A1 (en) * | 2020-02-17 | 2021-08-26 | BigID Inc. | Machine learning systems and methods for predicting personal information using file metadata |
US11461786B2 (en) * | 2020-03-02 | 2022-10-04 | Salesforce.Com, Inc. | Systems, methods, and apparatuses for implementing cross cloud engagement activity visualization without requiring database merge or data replication |
CN111522797B (en) * | 2020-04-27 | 2023-06-02 | 支付宝(杭州)信息技术有限公司 | Method and device for constructing business model based on business database |
US11763084B2 (en) * | 2020-08-10 | 2023-09-19 | International Business Machines Corporation | Automatic formulation of data science problem statements |
US20220164700A1 (en) * | 2020-11-25 | 2022-05-26 | UiPath, Inc. | Robotic process automation architectures and processes for hosting, monitoring, and retraining machine learning models |
CN112632350B (en) * | 2020-12-07 | 2023-12-05 | 肇庆学院 | Deep learning sample labeling method and system based on online education big data |
US20230094635A1 (en) * | 2021-09-28 | 2023-03-30 | Intuit Inc. | Subscriber retention and future action prediction |
US11983184B2 (en) * | 2021-10-07 | 2024-05-14 | Salesforce, Inc. | Multi-tenant, metadata-driven recommendation system |
CN117877737B (en) * | 2024-03-12 | 2024-07-05 | 北方健康医疗大数据科技有限公司 | Method, system and device for constructing primary lung cancer risk prediction model |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3475888A1 (en) * | 2016-08-22 | 2019-05-01 | Oracle International Corporation | System and method for ontology induction through statistical profiling and reference schema matching |
-
2018
- 2018-01-31 US US15/884,878 patent/US11663517B2/en active Active
-
2023
- 2023-05-12 US US18/196,963 patent/US20230334367A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
US20190138946A1 (en) | 2019-05-09 |
US11663517B2 (en) | 2023-05-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20230334367A1 (en) | Automatic machine learning model generation | |
US11238228B2 (en) | Training systems for pseudo labeling natural language | |
US10853577B2 (en) | Response recommendation system | |
US11645321B2 (en) | Calculating relationship strength using an activity-based distributed graph | |
US11061954B2 (en) | Intent classification system | |
US10824608B2 (en) | Feature generation and storage in a multi-tenant environment | |
US11061955B2 (en) | Intent classification system | |
EP3511887A1 (en) | Automated chat assistant for providing interactive data using npl - natural language processing - system and method | |
US10496657B2 (en) | Displaying an interactive communication time series | |
US10897520B2 (en) | Connected contact identification | |
US20190130013A1 (en) | User clustering based on query history | |
US11030545B2 (en) | Probabilistic framework for determining device associations | |
US20210158174A1 (en) | Equipment maintenance assistant training based on digital twin resources | |
US11385871B2 (en) | Adjusting a user interface layout based on user usage patterns | |
US10708388B2 (en) | Branched nodes in a workflow | |
US20230029697A1 (en) | Dynamic action identification for communication platform | |
US11900320B2 (en) | Utilizing machine learning models for identifying a subject of a query, a context for the subject, and a workflow | |
CN113570269A (en) | Operation and maintenance project management method, device, equipment, medium and program product | |
US20190138958A1 (en) | Category identifier prediction | |
US20210241162A1 (en) | Message objection identification and handling | |
US11184450B2 (en) | Variable content generation and engagement tracking | |
US11928584B2 (en) | Distributed hyperparameter tuning and load balancing for mathematical models | |
US11399068B2 (en) | Distributing data management setup between multiple users | |
US20180253654A1 (en) | Generating rules based on patterns in a communication time series | |
US11983650B2 (en) | Intelligent routing of data objects between paths using machine learning |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |