EP4100894A1 - Predicting outcomes via marketing asset analytics - Google Patents
Predicting outcomes via marketing asset analyticsInfo
- Publication number
- EP4100894A1 EP4100894A1 EP21751184.9A EP21751184A EP4100894A1 EP 4100894 A1 EP4100894 A1 EP 4100894A1 EP 21751184 A EP21751184 A EP 21751184A EP 4100894 A1 EP4100894 A1 EP 4100894A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- marketing
- asset
- discrete
- text
- outcome
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 claims abstract description 111
- 238000012549 training Methods 0.000 claims description 83
- 230000015654 memory Effects 0.000 claims description 21
- 230000004069 differentiation Effects 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 7
- 230000002123 temporal effect Effects 0.000 claims description 7
- 238000013518 transcription Methods 0.000 claims description 7
- 230000035897 transcription Effects 0.000 claims description 7
- 238000004519 manufacturing process Methods 0.000 claims description 6
- 230000009471 action Effects 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 230000001419 dependent effect Effects 0.000 claims description 5
- 230000009193 crawling Effects 0.000 claims description 3
- 230000004438 eyesight Effects 0.000 claims description 2
- 238000004891 communication Methods 0.000 description 49
- 239000013598 vector Substances 0.000 description 46
- 238000010801 machine learning Methods 0.000 description 44
- 238000013527 convolutional neural network Methods 0.000 description 42
- 239000000047 product Substances 0.000 description 36
- 238000010586 diagram Methods 0.000 description 34
- 230000004913 activation Effects 0.000 description 24
- 238000001994 activation Methods 0.000 description 24
- 230000006870 function Effects 0.000 description 18
- 238000004458 analytical method Methods 0.000 description 14
- 238000013528 artificial neural network Methods 0.000 description 14
- 230000008569 process Effects 0.000 description 10
- 235000013361 beverage Nutrition 0.000 description 8
- 230000000875 corresponding effect Effects 0.000 description 7
- 238000007726 management method Methods 0.000 description 7
- 238000012805 post-processing Methods 0.000 description 7
- 238000007781 pre-processing Methods 0.000 description 7
- 230000002860 competitive effect Effects 0.000 description 6
- 230000002596 correlated effect Effects 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 6
- 238000011176 pooling Methods 0.000 description 5
- 238000013473 artificial intelligence Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 4
- 230000008451 emotion Effects 0.000 description 4
- 238000007639 printing Methods 0.000 description 4
- 238000012706 support-vector machine Methods 0.000 description 4
- 238000012546 transfer Methods 0.000 description 4
- 238000003339 best practice Methods 0.000 description 3
- 238000013145 classification model Methods 0.000 description 3
- 239000003086 colorant Substances 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 241000282412 Homo Species 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 238000010411 cooking Methods 0.000 description 2
- 238000013499 data model Methods 0.000 description 2
- 230000002068 genetic effect Effects 0.000 description 2
- 238000007477 logistic regression Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003058 natural language processing Methods 0.000 description 2
- 238000000513 principal component analysis Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 235000002020 sage Nutrition 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 235000014171 carbonated beverage Nutrition 0.000 description 1
- 235000021443 coca cola Nutrition 0.000 description 1
- 238000010835 comparative analysis Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000035622 drinking Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 150000004677 hydrates Chemical class 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 238000007790 scraping Methods 0.000 description 1
- 235000014214 soft drink Nutrition 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
- 230000002087 whitening effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0241—Advertisements
- G06Q30/0242—Determining effectiveness of advertisements
- G06Q30/0244—Optimization
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/131—Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/137—Hierarchical processing, e.g. outlines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0201—Market modelling; Market analysis; Collecting market data
- G06Q30/0204—Market segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/01—Social networking
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/26—Speech to text systems
Definitions
- Predicting an outcome of a marketing or sales campaign has typically involved at least some measure of human guesswork.
- Marketing and sales departments often rely on trial-and- error in their campaigns, in hopes of generating favorable outcomes for at least some fraction of the implemented campaigns.
- Such a haphazard approach is highly subjective and inefficient, frequently resulting in inefficiencies and higher costs.
- implementing a campaign without a robust prediction of likely outcomes can result in the campaign being less effective than it might otherwise be.
- a campaign might even do more harm than good, such as if certain demographics are offended by a campaign’s content, for example.
- a method for predicting an outcome for a marketing asset may include a computing device performing tasks. These tasks may include determining at least one asset type of a marketing asset received by the computing device.
- the marketing asset is parsed, by the computing device, into a plurality of segmented components, based on the determined at least one asset type.
- the method includes determining at least one discrete marketing message conveyed by the marketing asset, by applying at least one of a predictive model or rules to the at least one segmented component, where the at least one predictive model or rules are stored in a memory associated with the computing device.
- the method further includes determining, for each of the at least one discrete marketing messages, at least one associated score. The at least one associated score is provided as input to a trained predictive model to obtain a predicted marketing outcome for the marketing asset.
- a system for predicting an outcome for a marketing asset may include at least one computing device configured to perform tasks.
- the tasks may include determining at least one asset type of a marketing asset received by the computing device.
- the marketing asset is parsed, by the computing device, into a plurality of segmented components, based on the determined at least one asset type.
- the method determines at least one discrete marketing message conveyed by the marketing asset, by applying at least one of a predictive model or rules to the at least one segmented component, where the at least one predictive model or rules are stored in a memory associated with the computing device.
- the tasks further include determining, for each of the at least one discrete marketing messages, at least one associated score. Then, the at least one score is input to a trained predictive model to obtain a predicted marketing outcome.
- an article of manufacture for use in predicting an outcome for a marketing asset includes a non-transitory computer- readable medium having stored program instructions that, upon execution by one or more processors, cause the one or more processors to perform tasks.
- the tasks may include determining at least one asset type of a marketing asset received by the computing device.
- the marketing asset is parsed, by the computing device, into a plurality of segmented components, based on the determined at least one asset type.
- the method determines at least one discrete marketing message conveyed by the marketing asset, by applying at least one of a predictive model or rules to the at least one segmented component, where the at least one predictive model or rules are stored in a memory associated with the computing device.
- the tasks further include determining, for each of the at least one discrete marketing message, at least one associated score. Then, the at least one score is provided as an input to a trained predictive model to obtain a predicted marketing outcome.
- Figure 1 is a schematic block diagram illustrating a marketing outcome prediction network, according to an example embodiment.
- Figure 2 is a schematic block diagram illustrating a computing device, according to an example embodiment.
- Figure 3 is a schematic block diagram illustrating a supervised learning pipeline, according to an example embodiment.
- Figure 4 is a schematic block diagram illustrating a machine learning pipeline, according to an example embodiment.
- Figure 5 is a schematic block/flow diagram illustrating a system and method of predicting outcomes via marketing asset analytics, according to an example embodiment.
- Figure 6A is a pictorial screenshot diagram illustrating a graphical display of brand attributes, according to an example embodiment.
- Figure 6B is a pictorial screenshot diagram illustrating a tabular display of keyword density, according to an example embodiment.
- Figure 6C is a pictorial screenshot diagram illustrating a tabular display of image objects and contents, according to an example embodiment.
- Figure 6D is a pictorial screenshot diagram illustrating a graphical display of product features, according to an example embodiment.
- Figure 6E is a pictorial screenshot diagram illustrating a graphical display of product experiences, according to an example embodiment.
- Figure 6F is a pictorial screenshot diagram illustrating a graphical display of differentiations, according to an example embodiment.
- Figure 6G is a pictorial screenshot diagram illustrating a graphical display of value propositions, according to an example embodiment.
- Figure 6H is a pictorial screenshot diagram illustrating a graphical display of archetypes, according to an example embodiment.
- Figure 61 is a pictorial screenshot diagram illustrating a graphical display of reputations, according to an example embodiment.
- Figure 6J is a pictorial screenshot diagram illustrating a graphical display of habitats, according to an example embodiment.
- Figure 6K is a pictorial screenshot diagram illustrating a graphical display of pastimes, according to an example embodiment.
- Figure 6L is a pictorial screenshot diagram illustrating a graphical display of personalities, according to an example embodiment.
- Figure 6M is a pictorial screenshot diagram illustrating a competitive graphical display of professions, according to an example embodiment.
- Figure 6N is a pictorial screenshot diagram illustrating a competitive graphical display of brand attributes, according to an example embodiment.
- Figure 60 is a pictorial screenshot diagram illustrating a competitive graphical display of differentiations, according to an example embodiment.
- Figure 6P is a pictorial screenshot diagram illustrating a competitive graphical display of product features, according to an example embodiment.
- Figure 6Q is a pictorial screenshot diagram illustrating a competitive graphical display of personalities, according to an example embodiment.
- Figure 6R is a pictorial screenshot diagram illustrating a competitive graphical display of pastimes, according to an example embodiment.
- Figure 7 is a pictorial screenshot diagram illustrating output of a marketing outcome prediction, according to an example embodiment.
- Figure 8 is a pictorial screenshot diagram illustrating an example user interface in the form of a dashboard report, according to an example embodiment.
- This description sets forth example techniques that relate to systems and methods for predicting outcomes of marketing assets (also referred to as “communications” herein) using analytics.
- Such analytics may include, but are not limited to, artificial-intelligence-based analytics. These techniques are designed to improve on the typical practice of using human guesswork or trial-and-error to assist in designing marketing or sales campaigns, resulting in potentially higher efficiency and effectiveness in obtaining desired marketing outcomes.
- the techniques set forth herein are data-driven and include parsing and categorizing, such as by using artificial intelligence, communication data items/components (e.g., text blocks, images, audio, and/or video) into higher-level marketing-based constructs.
- the marketing-based constructs are then presented to one or more outcome prediction data models implemented as part of a machine learning system.
- a machine learning system can be hosted on a computing device or system configured to implement a variety of machine learning models in addition to the aforementioned outcome- prediction data model(s).
- the machine learning model may, but need not, include one or more neural networks or other models (e.g., support vector machines (SVMs), Bayesian networks, genetic algorithms, linear classifiers, non-linear classifiers, algorithms based on kernel methods, logistic regression algorithms, linear discriminant analysis algorithms, and/or principal components analysis algorithm) described in further detail below, to make predictions about marketing outcomes.
- SVMs support vector machines
- Bayesian networks genetic algorithms, linear classifiers, non-linear classifiers, algorithms based on kernel methods, logistic regression algorithms, linear discriminant analysis algorithms, and/or principal components analysis algorithm
- Each model can be trained on a training dataset containing source marketing-based constructs (i.e., discrete marketing messages) having known marketing outcomes. These source marketing-based constructs each can be labeled according to a number of different types and/or levels of marketing outcomes.
- Newly received data items e.g., text blocks, images, audio, and/or video
- Newly received data items can be parsed into individual data items and placed or categorized into the marketing-based constructs, from which marketing outcome predictions can be made using one or more trained machine learning models.
- Various example systems and methods disclosed herein advantageously utilize data analysis and manipulation (e.g., parsing) and/or artificial intelligence to analyze marketing content in order to improve efficiency and reduce subjectivity.
- Some example embodiments include analyzing discrete marketing messages, as well as contents, demographics, emotions, and/or colors of every image and video snippet, for example. This analysis can, in turn, improve marketing-outcome prediction accuracy.
- the methods and systems may link visual characteristics to marketing results, to allow for subsequent alterations to improve marketing performance (i.e., better realization of desired marketing outcomes).
- various example embodiments of the methods and systems may additionally include providing insights and/or recommendations for improving likelihoods of achieving desired marketing outcomes.
- the methods or systems could suggest alterations to visual characteristics, such as introducing additional or alternative colors, or modifying an image or video to display one or more persons matching a desired target persona, for example.
- the methods and systems may provide analysis and specific recommendations, so that future marketing communications can be drafted based on the recommendations. Additionally or alternatively, the methods and systems may further provide examples corresponding to the recommendations.
- the methods and systems couple text and visual analytics data with performance data to identify patterns that predict outcomes.
- the output may include predictions along with recommendations on how to improve performance.
- Some example embodiments may also assist in improving consistency during the copy drafting process. This is accomplished by the system analyzing the messaging and keyword density, for example, which helps to ensure that generated copy is predicted to be effective in producing desired marketing results.
- the system analyzes keywords that connect and guide customers to a particular product. By measuring which brand attributes both a company and/or its competitors are emphasizing and which ones a company is neglecting, the methods and systems can assist in promoting consistent usage of effective keywords.
- the methods and systems monitor (e.g., through “following” and recording social media posts) marketing communications and document target markets, messaging, and visual identity. This allows the system to help ensure that new marketing assets adhere to the identified brand standards.
- a particular system could store the following brand standard: “CompanyA has the best customer service. We are customer-obsessed. It’s in our DNA to do whatever is needed to be successful.”
- the system could compile, store, and present to a user, via a user interface, keywords and images associated with the identified brand standard.
- Example keywords include the following: “Customer success,” “dedicated,” and “customer experience.”
- Example images might include images of happy employees and happy customers.
- some implementations could also provide a list of words and/or image contents to avoid.
- the methods and systems compare marketing performance key metrics with those of industry peers and competitors.
- a server can provide as output a listing of text brand scores showing company name, rank, and predicted engagement score for analyzed marketing content. The methods and systems can provide this output to assist the user in identifying ways to improve future marketing efforts.
- the system can also be used for direct analysis of a competitor’s marketing and/or sales communications.
- the system’s analytics components analyze every text block and image, and determine target markets, positioning, messaging, tone, and voice.
- the system analyzes the contents, emotions, colors, and demographics within every image and illustration, for example.
- the various example method and system implementations described above and as further detailed below utilize computing devices and analytics (e.g., artificial intelligence) to help solve technical problems relating to predicting marketing outcomes for proposed or actual sales or marketing communication assets.
- Some of the technical advantages afforded by these methods and systems include (1) improved prediction accuracy, (2) the ability to consider a larger pool of data (i.e., machine-learning training data) than is realistically feasible for human analysis of marketing communications, and (3) improved efficiency over past trial-and-error techniques, resulting in faster processing of proposed communications.
- Other technical advantages will become clear from the following description of the accompanying drawings.
- Figure 1 is a schematic block diagram illustrating a prediction network 100, according to one example implementation.
- Prediction network 100 includes computing devices 120, 122
- prediction network 100 can have more, fewer, and/or different types of computing devices and/or servers than indicated in Figure 1.
- computing devices 120, 122 and server(s) 130 can be connected to network 140 via one or more, possibly different, network protocols. Data can be transmitted between computing devices 120, 122 and server(s) 130 over wired and/or wireless links between computing devices, servers, and network 140.
- the format of each respective data transmission between devices in prediction network 100 can include one or more of a variety of different formats including: text formats, image formats, extensible mark up language (XML), Simple Network Maintenance Protocol (SNMP) formats, database tables, a flat file format, or another format.
- Computing devices 120, 122 can create, obtain, update, display, and/or delete data (and perhaps related software) for configurations of prediction network 100.
- Example data for configurations of prediction network 100 includes, but is not limited to: data for configuring devices in prediction network 100; data for configuring network protocols (e.g., File Transfer Protocol (FTP), HyperText Transfer Protocol (HTTP), Java Message Service (JMS), Adobe® Page Description Format (PDF), Simple Object Access Protocol (SOAP), Short Message Service (SMS), Simple Message Transfer Protocol (SMTP), SNMP, Transfer Control Protocol / Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Lightweight Directory Access Protocol (LDAP), Message Queue (MQ), and/or other protocols), access-management related data for clients and/or servers; (e.g passwords, signatures, credentials, certificates, subscriptions, licenses, and/or tokens related to accessing part or all of the functionality of network 140 and/or cloud-based services, software and/or solutions), and data for customizing,
- One or more servers 130 associated with one or more entities may store, update, delete, retrieve, and/or provide functionality for parsing data and/or learning patterns, trends, and/or features about data related to prediction network 100. Based on the learned patterns, trends, and/or features, server(s) 130 can generate outputs, such as predictions about marketing outcomes.
- the data stored on server(s) 130 may include data relating to analytics models (e.g., machine learning models), such as training data, for example.
- the data stored on server(s) 130 may also include device information and/or other information related to devices associated with the network 100.
- the stored data can be retrieved from server(s) 130 in response to a received query (or queries) requesting information about proposed or implemented marketing or sales campaigns, including predictions about marketing outcomes for such campaigns.
- server(s) 130 can provide additional or alternative services as well, such as services related to tracking and storing brand standards.
- FIG. 2 is a schematic block diagram illustrating a computing device 200, according to one example implementation.
- Computing device 200 may include one or more input devices 202, one or more output devices 204, one or more processors 206, and memory 208.
- computing device 200 may be configured to perform one or more herein- described functions of and/or functions related to, for example, some or all of at least the functionality described in the context of an artificial neural network, a convolutional neural network, other analytics model, a computer, a personal computer, a smart phone, a smart watch, a wearable computer, a server device, a printing device, a display device, prediction network 100, pipelines 300, 400, and methods described herein.
- Input devices 202 may include user input devices, network input devices, sensors, and/or other types of input devices.
- input devices 202 may include user input devices such as a touch screen, a keyboard, a keypad, a computer mouse, a trackball, a joystick, a camera, a voice recognition module, and/or other similar devices.
- Network input devices may include wired network receivers and/or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or other similar transceiver configurable to communicate via a twisted pair wire, a coaxial cable, a fiber-optic link, or a similar physical connection to a wireline network, such as wired portions of network 140, and/or wireless network receivers and/or transceivers, such as a BluetoothTM transceiver, a Zigbee® transceiver, a Wi-FiTM transceiver, a WiMAXTM transceiver, a wireless wide-area network (WWAN) transceiver, and/or other similar types of wireless transceivers configurable to communicate via a wireless network, such as wireless portions of network 140.
- Other input devices 202 are possible as well.
- Output devices 204 may include user display devices, audible output devices, network output devices, and/or other types of output devices.
- User display devices may include one or more printing components, liquid crystal displays (LCD), light emitting diodes (LEDs), lasers, displays using digital light processing (DLP) technology, cathode ray tubes (CRT), light bulbs, and/or other similar devices.
- Audible output devices may include a speaker, speaker jack, audio output port, audio output device, headphones, earphones, and/or other similar devices.
- Network output devices may include wired network transmitters and/or transceivers, such as an Ethernet transceiver, a USB transceiver, or other similar transceiver configurable to communicate via a twisted pair wire, a coaxial cable, a fiber-optic link, or a similar physical connection to a wireline network, such as wired portions of network 140, and/or wireless network transmitters and/or transceivers, such as a BluetoothTM transceiver, a Zigbee® transceiver, a Wi-FiTM transceiver, a WiMAXTM transceiver, a WWAN transceiver and/or other similar types of wireless transceivers configurable to communicate via a wireless network, such as wireless portions of network 140.
- wired network transmitters and/or transceivers such as an Ethernet transceiver, a USB transceiver, or other similar transceiver configurable to communicate via a twisted pair wire, a coaxial cable, a fiber-optic link, or
- Processor(s) 206 can include one or more general purpose processors, central processing units (CPUs), CPU cores, and/or one or more special purpose processors (e.g graphics processing units (GPUs), digital signal processors (DSPs), field programmable gated arrays (FPGAs), application specific integrated circuits (ASICs), additional graphics-related circuitry / processors, etc.).
- processors 206 may be configured to execute computer-readable instructions 210 stored in memory 208 and/or other instructions as described herein.
- Memory 208 may include one or more computer-readable storage media configured to store data and/or instructions that can be read and/or accessed by at least one of processor(s) 206.
- the one or more computer-readable storage media may include one or more volatile and/or non-volatile storage components, such as optical, magnetic, organic, or other memory or disc storage, which can be integrated in whole or in part with at least one of processor(s) 206.
- the computer-readable storage media may include one or more components that store data for short periods of time like register memories, processor caches, and/or random-access memories (RAM).
- the computer-readable storage media may include non-transitory computer readable media that stores program code and/or data for longer periods of time, such as secondary or persistent long-term storage; for example, read-only memory (ROM), optical or magnetic disks, or compact-disc read-only memory (CD-ROM).
- ROM read-only memory
- CD-ROM compact-disc read-only memory
- memory 208 may be implemented using a single physical device (e.g one optical, magnetic, organic, or other memory or disk storage unit), while in other embodiments, memory 208 may be implemented using two or more physical devices.
- memory 208 may store computer-readable instructions 210 that, when executed by processor(s) 206, may cause a computing device to perform functions, such as but not limited to, some or all of at least the herein-described functionality of devices, networks, methods, diagrams, images, equations, and/or scenarios.
- computer-readable instructions 210 may include at least analytics software 212.
- Analytics software 212 can include software and/or firmware for providing analytics functionality, making use of one or more supervised learning pipelines, such as one or more machine learning algorithms (ML A) and/or predictive models (trained MLA). This may include, for example, some or all of at least the functionality described in the context of an artificial neural network, a convolutional neural network, a computer, a personal computer, a smart phone, a smart watch, a wearable computer, a server device, a printing device, a display device, prediction network 100, pipelines 300, 400, and methods described herein.
- analytics software 212 can be updated or otherwise reprogrammed, such as via connections with network 140, to provide different (e.g., improved or otherwise modified) functionality.
- Computer-readable instructions 210 may also include a parser 214 operable to parse a received marketing communication/asset into one or more components. Parsing is described in additional detail with respect to Figure 5, for example.
- FIG. 3 is a schematic block diagram illustrating a supervised learning pipeline 300, according to one example implementation.
- Supervised learning pipeline 300 includes training data generator 310, training input 320, one or more input feature vectors 322, one or more training data items 330, machine learning algorithm 340, actual input 350, one or more actual input feature vectors 352, predictive model 360, and one or more predictive model outputs 370.
- Part or all of supervised learning pipeline 300 may be implemented by executing software for part or all of supervised learning pipeline 300 on one or more processors and/or by using other circuitry, such as specialized hardware (e.g., an embedded AI processor) for carrying out part or all of supervised learning pipeline 300.
- the aforementioned processor(s), other circuitry, specialized hardware, and/or embedded AI processor may be situated in a computing device 200, such as a computing device 200 implementing some or all of the server(s) 130 and/or computers 120 and 122 shown in Figure 1.
- supervised learning pipeline 300 may involve two phases: a training phase and a prediction phase.
- the training phase may involve machine learning algorithm 340 learning one or more tasks.
- the prediction phase may involve predictive model 360, which may be a trained version of machine learning algorithm 340, making predictions to accomplish the one or more learned tasks.
- machine learning algorithm 340 and/or predictive model 360 may include, but is not limited to, one or more artificial neural networks (ANNs), deep neural networks, convolutional neural networks (CNNs), recurrent neural networks, support vector machines (SVMs), Bayesian networks, genetic algorithms, linear classifiers, non-linear classifiers, algorithms based on kernel methods, logistic regression algorithms, linear discriminant analysis algorithms, and/or principal components analysis algorithms. Other algorithms and/or models may additionally or alternatively be used.
- ANNs artificial neural networks
- CNNs convolutional neural networks
- SVMs support vector machines
- Bayesian networks genetic algorithms, linear classifiers, non-linear classifiers, algorithms based on kernel methods, logistic regression algorithms, linear discrimin
- training data generator 310 may generate training input 320 and training data item(s) 330.
- Training input 320 can be processed to determine input feature vector(s) 322.
- training input 320 may be preprocessed, such as for image recognition tasks.
- training input 320 may include one or more images.
- training input 320 may include images that exhibit known customer responses (e.g., positive, negative, or neutral) associated with the image content.
- the training sets used in various embodiments set forth herein may include data, such as labeled training data, pertaining to hundreds of thousands of brands and millions of marketing assets, for example. As a result, even small businesses, or those with limited resources, may benefit from the technical improvements afforded by the technologies of the methods and systems set forth.
- Input feature vector(s) 322 may be provided to machine learning algorithm 340 to learn one or more tasks. Input feature vector(s) 322 may represent features associated with training input 320.
- Machine learning algorithm 340 may generate one or more outputs based on input feature vector(s) 322 and perhaps training input 320 related to the one or more tasks being learned.
- training data item(s) 330 may be used to assess the output(s) of machine learning algorithm 340 for accuracy, and machine learning algorithm 340 may be updated based on this assessment. More particularly, the assessment may include use of a loss function that measures inconsistencies between predicted values and ground truth values (e.g., data labels from a training data item).
- a loss function LF1 for training machine learning algorithm 340 may be based on a mean-square-error (MSE) value representing a “Euclidean distance” between the predicted value (LFl(O x )) and a data label of a training data item (Ox).
- MSE mean-square-error
- the output of the loss function may be used during the training of machine learning algorithm 340 to decide how much to adjust parameters to reduce an error value (e.g., an error value calculated based on a difference between Ox and LFl(O x ), such as
- the loss function may be based on cross-entropy loss and/or other values.
- training input 320 may be an image in a marketing communication
- training data item(s) 330 may include one or more labels that indicate aspects (e.g., typical specific customer reaction or feeling for a particular demographic) associated with the image.
- machine learning algorithm 340 may make predictions about aspects associated with training input 320. Predictions may be compared to the label(s) of training data item(s) 330 to see if they match. For example, if machine learning algorithm 340 detects the same aspects as mentioned in a label for training data item(s) 330, then they match; otherwise, an error can be determined.
- a vector of errors may be calculated, corresponding to a magnitude of difference between the predicted aspects and the labels of training data items(s) 330. Then, a loss function may compare the vector of errors predicted by machine learning algorithm 340 with a corresponding vector constructed from the labels of aspects in training data item(s) 330. The comparison may be used to adjust parameters of machine learning algorithm 340 to reduce the overall error value.
- an ANN may have a number of nodes, each node having an associated numerical weight. These nodes may be arranged in the ANN into one or more layers (e.g., such as one or more CNN layers for each CNN model(s) 420 as described below). Training an ANN to perform a task may involve changing the weights of the nodes as training inputs related to the task are provided to and processed by the ANN. Once trained, the ANN (or model) may or may not modify the weights of the nodes upon being deployed for inference/detection/prediction tasks.
- Training of machine learning algorithm 340 may continue until machine learning algorithm 340 is considered to be trained to perform the one or more tasks. For example, training of machine learning algorithm 340 may continue until the occurrence of one or more training termination criteria.
- machine learning algorithm 340 may be considered to be a predictive model, such as predictive model 360.
- actual input 350 can be processed to generate actual input feature vector(s) 352.
- Predictive model 360 can perform predictions, such as image recognition tasks, by taking actual input feature vector(s) 352 as inputs.
- predictive model 360 may generate one or more predictions about actual input feature vector(s) 352 as part or all of predictive model output(s) 370. For instance, predictive model 360 may receive a request to make a prediction about actual feature input vector(s) 352, responsively generate one or more predictions about actual input feature vector(s) 352, and provide the prediction(s) as predictive model output(s) 370. The prediction phase/use of predictive model 360 is discussed in more detail below.
- machine learning algorithm 340 may be trained on one or more training computing devices and predictive model 360 may be executed on the same training computing device(s). In other examples, machine learning algorithm 340 may be trained on the training computing device(s). Then, after training, now-trained machine learning algorithm 340 may be communicated as predictive model 360 from the training computing device(s) to one or more other computing devices that can execute predictive model 360 to generate predictive model output(s) 370.
- Figure 4 is a schematic block diagram illustrating a machine learning pipeline 400, according to one example implementation. In some examples, ML pipeline 400 may be related to and/or implement the prediction phase of supervised learning pipeline 300.
- ML pipeline 400 includes actual input 350, actual feature input vector(s) 352, preprocessing 410, predictive model 360, post-processing 440, and predictive model output(s) 370. Part or all of ML pipeline 400 may be implemented by executing software on one or more processing devices and/or by using other circuitry ( e.g specialized hardware for carrying out part or all of ML pipeline 400). [0069] In operation, ML pipeline 400 may begin by receiving actual input 350. In particular, actual input 350 to ML pipeline 400 may include one or more actual electronic images. After receiving actual input 350, ML pipeline 400 may generate actual input feature vector(s) 352 based on actual input 350. Actual feature vector(s) 352 may represent features of actual input 350. The feature(s) may include, but are not limited to color values, size, and others.
- ML pipeline 400 may proceed to pre-processing 410.
- actual input feature vector(s) 352 may be pre- processed to increase the accuracy of predictive model 360.
- pre-processing 410 might include normalization and mean-subtraction for rescaling numeric attributes into a more compact range principal component analysis (PCA) to reduce the dimensionality.
- PCA principal component analysis
- pre-processing 410 might include whitening transformations to reduce underlying noise in input data.
- ML pipeline 400 may proceed to use predictive model 360 to make predictions regarding pre-processed actual input feature vector(s) 352.
- Figure 4 shows an example in which predictive model 360 of ML pipeline 400 includes CNN model(s) 420.
- predictive model 360 may utilize CNN model(s) 420 to classify (or otherwise characterize) pre-processed actual input feature vector(s) 352.
- the input (i.e., pre-processed actual input feature vector(s) 352) into CNN model(s) 420 may be referred to herein as pre- processed input 412.
- CNN model(s) 420 may include, for example, one or more trained convolutional neural networks that perform classification on pre-processed input 412.
- each CNN model in CNN model(s) 420 may contain one or more CNN layers.
- a CNN layer may perform convolution, activation, pooling, or inference tasks using respective one or more convolution layers, one or more activation layers, one or more pooling layers, and/or one or more fully- connected layers.
- Convolution layers may include one or more filters to filter respective inputs. Each filter may work over a subset of an input.
- the output of the convolution layer i.e., activation map
- Activation layers determine whether the activation map is to be provided to a subsequent layer. For example, suppose the activation map has values in the range [0, 1] and that only activation map values above 0.4 are to be provided to a subsequent layer. Then, the activation layer could map activation map values in the range [0, 0.4] to 0 (representing no activation), and map activation map values in the range (0.4, 1] to 1 (representing activation).
- an activation layer can model an activation function, such as a sigmoid / logistic activation function, a hyperbolic tangent activation function, a rectified linear unit (ReLU) activation function, or another activation function, to determine whether the output of the convolution layer (e.g., the activation map) is to be provided to a subsequent layer.
- an activation function such as a sigmoid / logistic activation function, a hyperbolic tangent activation function, a rectified linear unit (ReLU) activation function, or another activation function, to determine whether the output of the convolution layer (e.g., the activation map) is to be provided to a subsequent layer.
- Pooling layers down-sample input provided by convolution and activation layers.
- Fully-connected layers can have full connections to all activations in a previous layer and may act as a soft-max classifier. That is, the fully -connected layer can receive an input from a previous layer and can output a resulting N dimensional vector, where N is the number of possible predictions that could be made about actual input 350. For example, if actual input 350 was an image of a digit and if predictive model 360 was trained to classify digits within images, N would be 10.
- Each output number in the resulting N dimensional vector may represent the probability of a certain class (i.e., possible prediction) occurring in actual input 350.
- a resulting vector could appear as [0 .1 .1 .75 00 000 .05], where the probability that actual input 350 contains the digit 1 is 10%, the probability that actual input 350 contains the 2 is 10%, and so on.
- This resulting vector may be outputted by CNN model(s) 420, for example, as output vector 430.
- CNN model(s) 420 can include CNNs with different combinations (also known as CNN architectures) of convolutional layers, activation layers, pooling layers, and fully-connected layers.
- CNN layers 422, 424, 426, and 428 are provided in Figure 4.
- CNN layers 422, 424, 426 can selectively filter and down-sample input using convolution, activation, and pooling layers to extract features, while CNN layer 428 can provide a final output prediction about actual input 350.
- preprocessed input 412 may be provided to CNN layer 422.
- CNN layer 422 may process preprocessed input 412 (e.g., convolve, activate, and/or pool preprocessed input 412 into a first feature vector) and provide input (e.g., the first feature vector) to CNN layer 424.
- CNN layer 424 may process the input provided by CNN layer 422 (e.g., convolve, activate, and/or pool input provided by CNN layer 424 into a second feature vector) and provide input (e.g., the second feature vector) to CNN layer 426.
- CNN layer 426 may process the input provided by CNN layer 424 (e.g., convolve, activate, and/or pool input provided by CNN layer 424 into a third feature vector) and provide input (e.g., the third feature vector) to CNN layer 428.
- CNN layer 428 may process the input provided by CNN layer 426 (e.g., make a soft-max prediction on input provided by CNN layer 426 into a fourth feature vector) and provide output vector 430 (e.g., the fourth feature vector) as the output prediction of predictive model 360.
- output vector 430 e.g., the fourth feature vector
- ML pipeline 400 may proceed to perform post-processing 440 of output vector 430.
- CNN layers 422, 424, 426, and 428 are solely a convenient conceptual representation of a CNN architecture and not intended to be limiting with respect to example embodiments or techniques described herein.
- CNN model (s) 420 may have more, fewer, and/or different layers than CNN layers 422, 424, 426, and 428.
- predictive model 360 may utilize other models besides CNN models.
- post-processing 440 may include weighting methods that combine multiple predictions into a single, aggregate prediction.
- output vector 430 may represent one out of a myriad of predictions made by predictive model 360 with regards to actual input 350.
- multiple sections of the image e.g., 4 sections
- output vector 430 may represent one out of a total of four predictions made by predictive model 360 with regard to actual input 350.
- weighting methods may be used to combine each of the four predictions into a single, aggregate prediction. Weighting methods may be determined as a sum, or other numerical combination (e.g., an average, a weighted average, a product) of the individual output vectors.
- post-processing 440 may include ranking methods that curtail output vector 430 to only “relevant” classes.
- output vector 430 from predictive model 360 is [0 .1 .1 .75 00 000 .05]
- post-processing 440 may retain any class with a prediction value exceeding a predefined threshold (e.g., 60% or 80%). Many other examples of ranking methods are also possible.
- ranking methods may also be applied to single, aggregate vectors determined by weighting methods.
- ML pipeline 400 may proceed to provide predictive model output(s) 370.
- ML pipeline 400 may provide predictive model output(s) 370 by storing some or all of predictive model output(s) 370, communicating some or all of predictive model output(s) 370 to one or more other computing devices, displaying some or all of predictive model output(s) 370, and/or otherwise furnishing some or all of predictive model output(s) 370 as outputs.
- specific hardware and/or software may be designed to embody part or all of predictive model 360 and/or ML pipeline 400.
- the specific hardware representing predictive model 360 and/or ML pipeline 400 may be embodied as analytics software 212 executed by processor(s) 206.
- Figure 5 is a schematic block/flow diagram illustrating a method 500 of predicting marketing outcomes for a marketing and/or sales communication, according to an example implementation.
- Method 500 may represent a specific sequence or series of actions that, when performed, allows a computing device to predict outcomes for marketing communications.
- Method 500 can be carried out by a computing device, such as computing device 200.
- additional components, steps, or blocks, may be added to method 500 without departing from the scope of the method.
- the method 500 includes receiving a communication (block 502), such as through the input device 202.
- Examples of communications could be proposed or implemented marketing and/or sales communications (also referred to as “assets” herein), in the form of data or other information representing or embodying such communications.
- the received communication may be associated with the user (e.g., an individual or enterprise) of the method 500, or instead may be associated with a competitor of the user.
- the communication may be targeted toward an individual (e.g., a “form” email customized for a particular individual), a group of individuals (e.g., a customer newsletter or blog post), or the general public (e.g., a scanned print advertisement or web ad).
- the method 500 further includes determining a type of asset (block 504) encompassed by the received communication.
- the communication could be determined to be an asset of one of the following types: a web page 508, a social media post 510, a pdf 512, an electronic document (e.g., *** doc 514, MS-Word doc, etc.), a plain-text email 516, an html email 518, a video 520, a sales call (e.g., a call script or transcript of a recorded call), a digitization of a traditional print communication, or some other marketing or sales asset.
- a type of asset encompassed by the received communication.
- the communication could be determined to be an asset of one of the following types: a web page 508, a social media post 510, a pdf 512, an electronic document (e.g., *** doc 514, MS-Word doc, etc.), a plain-text email 516, an html email 518, a video 520, a
- Determining the type of asset could include examining at least a portion (e.g., an initial portion, such as a header or remarks section) of a data file associated with the particular received communication. Alternatively, determining the type of asset could include examining the entire communication.
- blocks 502 and 504 may be swapped, such that a type of marketing asset/communication is first determined or decided, followed by actually receiving marketing assets/communications of that type.
- web communications may be received from a collection service that “crawls” websites, scraping data for use in the outcome prediction methods set forth herein. In such a case, it is first decided that web communications are going to be examined, and then, the communications (e.g., web pages) are received via a robotic web crawler.
- the method 500 could additionally or alternatively include classifying the communication by one or more attributes (block 505), such as marketing objective or intended outcome (e.g., to initiate a sale, to notify and/or inform, or to build a relationship).
- attributes such as marketing objective or intended outcome (e.g., to initiate a sale, to notify and/or inform, or to build a relationship).
- classification attributes e.g., the following are specific examples of classification attributes that could be assigned to a particular marketing or sales communication: business vertical (e.g., home services, electronic retail, soft drinks), brand archetype (e.g., explorer, lover, jester), customer type (e.g., business, consumer), supply chain role (e.g., maker, distributor, retailer), etc.
- Such a classification could be assigned by a creator of the communication, for example.
- the communication creator could supply information via a web form to create a digital “brand book” that indicates potential classification attributes, such as industry or other attributes listed immediately above.
- the system 500 could apply a classification model to the received communication to determine a suitable classification attribute.
- a user associated with the system 500 (where the user is unaffiliated with the communication creator) could apply a classification. While block 505 is illustrated as an optional function to be performed after the function after block 504, the function of block 505 could alternatively be moved to other locations in the method (e.g., immediately after block 502 or elsewhere) or omitted entirely.
- the function of classifying the communication by one or more attributes shown in block 505 may assist with predicting the marketing outcome, such as by informing as to suitable or preferred machine learning models (or sequences of models) and/or by defining a particular industry, vertical, or other group to which the received communication should be evaluated and/or compared against.
- the method 500 yet further includes parsing the communication (block 506) into one or more blocks (e.g., blocks 508-520, as described above), based on the determined type of asset. For example, based on the determined type of asset, the communication could be parsed into one or more of the following: image(s) 522, video(s) 524, audio 526, and/or text 528. This parsing could include utilizing one or more of the following to parse the communication into blocks: html tags, Application Program Interfaces (APIs), image recognition, and/or speech transcription, for example.
- APIs Application Program Interfaces
- the method 500 could include robotically crawling the web page, identifying and storing html code, and parsing the stored html code by identifying html tags.
- each html tag could identify at least one image, headline text, sub-headline text, paragraph text, and/or video.
- the method 500 could include using an API of the respective social media app to identify segmented components, such as image, text, and/or video components.
- the method could include crawling the web page, identifying and storing html code, and parsing the stored html code by identifying html tags, where the html tags identify an image, headline text, sub-headline text, paragraph text, and/or video, for example.
- the method 500 could include using a pdf parser to identify segmented components, such as image, text, or video components.
- the method 500 could include using an API associated with the web-based application (e.g., a *** doc app) to identify segmented components, such as image, text, or video.
- an API associated with the web-based application e.g., a *** doc app
- the method 500 could include parsing the text block of the email.
- the method 500 could include parsing the html email by identifying html tags associated with at least one image, headline text, sub-headline text, paragraph text, and/or video.
- the method 500 could include segmenting the video into temporal blocks, parsing the audio and images from each block, and transcribing the audio into text, such as by using an online or local automated audio transcription service.
- One suitable audio transcription service is Amazon’s audio transcription service, available at https://aws.amazon.com/transcribe/.
- the parsed video could be further parsed into a series of images (e.g., all images in the parsed video or a subset of images in the parsed video), to which image recognition software (e.g., using predictive classification models similar to those described herein) could be applied to identify one or more objects, scenes, or other items present in the parsed video.
- image recognition software e.g., using predictive classification models similar to those described herein
- the method 500 could include segmenting the call audio into temporal blocks, parsing the segmented audio from each block, and transcribing the parsed audio into text.
- the method 500 could identify segmented components using an API for an application associated with that asset. Other parsing methods could additionally or alternatively be used, depending on the type of asset. [0096] After parsing the communication into one or more blocks, based on type of asset, the method 500 includes determining one or more discrete marketing messages 538 being communicated, by analyzing each segmented component from the parsed asset. As shown in Figure 5, this analysis could include providing the parsed components to systems 530, 532, 534, and/or 536. Systems 530, 532, 534, and/or 536, described in further detail below, could each include one or a plurality of machine learning models, applied either in series or parallel (or both).
- the systems 530, 532, 534, and/or 536 could include one or a plurality of natural language models, applied either in series or parallel (or both).
- the systems 530, 532, 534, and/or 536 could include at least one machine learning model and at least one natural language model.
- the systems 530, 532, 534, and/or 536 could subject the asset components to one or more rules, as described in further detail below.
- Video ML/NL system 532 and Audio ML/NL system 534 are illustrated with hashed lines to indicate that video and audio may be first converted to image and/or text, respectively (such as through transcription or video segmenting).
- the systems 530, 532, 534, and/or 536 need not include a model for each of the image, video, audio, and text types.
- systems 530, 532, 534, and/or 536 might not identify any discrete messages in the asset being analyzed.
- a machine-learning system such as one utilizing the pipelines 300 and/or 400 illustrated in Figures 3 and 4 could be used to identify, e.g., at a sentence level, when certain discrete marketing messages are being conveyed.
- Training such a machine- learning system may include providing a large set of training data in which particular sentences, fragments, phrases, and/or words are associated with particular discrete marketing messages.
- a training dataset large enough to achieve an acceptable accuracy in the predictive model is recommended.
- Such a training dataset (e.g., the “Corpa” training dataset) will typically, but need not, include thousands of text blocks, e.g., from many companies and/or many industries.
- a crowdsourcing website such as the Amazon Mechanical Turk crowdsourcing site (available at https://www.mturk.com/), could be used to compile training data.
- the method 500 may include transmitting data representing a multitude of sentences (or other word groupings) to the crowdsourcing website, which, in turn, presents the data to actual humans (e.g., via their networked computing devices) for feedback on what marketing message each sentence communicates, and that feedback is provided back to the system 500.
- Such marketing message feedback could be received and stored to constitute training data for the predictive models set forth herein.
- a rules-based analysis could be performed.
- the system could maintain (i.e., store and/or access, such as via a networked client-server relationship) a dictionary of words identifying when particular discrete marketing messages, such as product features or product types, are being described. Particular words or phrases might identify when a product feature (e.g., relating to a carbonated beverage) is being described in the beverage industry.
- a “sparkling beverage” product feature might be identified by comparing a parsed text block with a specialized beverage industry dictionary.
- An example rules-based version of systems 530, 532, 534, and/or 536 could include identifying word roots from the parsed text asset component 528.
- Another example rules-based analysis of parsed communications could include syntax pattern-matching.
- the method 500 could include one or more of systems 530, 532, 534, and/or 536 (a) identifying that a sentence begins with a verb (e.g., “contact us to leam more”) in a second-person voice, and (b) matching that particular syntax with a “call to action” discrete marketing message. Other syntax pattern matches could also be used.
- Yet another example rules-based analysis of parsed communications could include regular-expression pattern-matching. The method 500 could include identifying regular expressions that are specific to a particular industry.
- the method 500 may include recognizing synonyms for “writing,” “faster,” and “software” in marketing messages as relating to coding software more quickly. This recognition of synonyms common in the software industry could take place regardless of the order in which the synonyms appear in the sentence, phrase, paragraph, or other parsed text block.
- the above-mentioned rules-based analysis techniques may also help to produce training datasets.
- a natural language processing rule could tag a parsed component as relating to a particular discrete marketing message. Tagging of parsed components could be performed by one or more humans or machine entities.
- the publicly available BART-MNLI model could be trained to tag parsed components as relating to particular discrete marketing messages.
- the BART-MNLI model and other “zero-shot” models may advantageously combine (a) context-specific understanding of the meaning of words provided through word embedding with (b) the simplicity and speed of rules-based systems.
- segmented components of types other than text (e.g., images, video, and/or audio) may similarly be analyzed to extract discrete marketing messages.
- one method of analyzing images or video includes first determining, such as via machine learning, the contents of the images or videos (e.g., objects, concepts, emotions, people, demographics of the people, facial expressions, etc.).
- rules-based methods similar to those described above with respect to text-based assets, may be used to map the contents to discrete marketing messages.
- Other techniques may additionally or alternatively be used for content-determining and mapping.
- discrete marketing messages fall within categories such as value propositions, statements of differentiation, product experiences, product features, brand archetypes, brand promise, brand vision, brand mission, brand positioning, brand values, brand reputation, brand archetypes, calls to action, audience demographics, audience pastimes, audience habitats, audience values, audience personality traits, audience lifestyles, and audience professions, among others.
- An example value proposition is for a beverage maker to claim that its water physically and mentally hydrates.
- An example statement of differentiation is that a particular brand of cola has a better cola taste than any other cola.
- An example of a product experience is for a beverage maker to claim an experience of having fun while enjoying a particular beverage, while hanging out with friends (more about what people are doing when drinking that particular beverage, than about the beverage itself.).
- Product features can simply refer to or describe characteristics of a particular product.
- Other discrete marketing messages can communicate the personality (e.g., creative or rebellious) of the brand and the customers the brand is intended to serve, as well as the archetypes (e.g. “Rebel,” “Sage,” or “Lover”) of the brand and the customers the brand is intended to serve.
- Figure 5 illustrates the one or more identified discrete marketing messages being passed from ML/NL/Rules systems 530, 532, 534, and/or 536 to block 540, in which at least one score is determined for each of the one or more discrete marketing messages determined in ML/NL/Rules systems 530, 532, 534 and/or 536. Once a collection of discrete marketing messages is identified, scores are calculated (or otherwise determined) to understand the relative emphasis of each marketing message.
- block 540 is illustrated as a single block following the multiple system blocks 530, 532, 534, and/or 536, in some examples, the functionality of block 540 (calculating scores associated with discrete marketing messages) may be partially or entirely incorporated into one or more of the systems 530, 532, 534, and/or 536.
- Example scores include but are not limited to: a) counts of how many times a discrete marketing message is communicated in text, images, videos, and within the entire marketing asset, b) counts of how many times a discrete marketing message category (e.g., value proposition, brand archetype, audience habitat) was communicated in text, images, videos, and within the entire marketing asset, c) the percentage of all discrete marketing messages that an individual discrete marketing messages or category represents (count divided by the sum of all counts).
- the determined scores are then used as the independent variables within a trained predictive model for predicting a marketing outcome, as described below with respect to block 542.
- scores may be used as training scores during the training of the trained predictive model, as described in further detail below.
- the method 500 then includes inputting the determined scores into a trained predictive model to obtain a predicted marketing outcome, as shown in block 542.
- the scores (and possibly other data) associated with the discrete marketing messages are the independent variables and the marketing outcome data are the dependent variables. Training of the predictive model is described in further detail below, and generally with reference to Figures 3 and 4 and accompanying description.
- the predicted marketing outcome output by the trained predictive model relates to a result invoked by a particular marketing or sales communication.
- the marketing outcome could be a predicted engagement score, a web page bounce rate, a clickthrough rate, an occurrence of a sale, an occurrence of an engagement, or a specified conversion rate, for example.
- a predicted marketing outcome could be users selecting a “click and buy” icon for a particular product.
- a predicted marketing outcome could be to “like” or “share” a particular social media post.
- a predicted marketing outcome could be to click the banner ad to result in a site visit.
- a predicted marketing outcome could be any consumer or business action desired to be prompted based on marketing materials.
- products e.g., Coca-Cola
- brand affection might still be predicted and gauged, such as via tracking “likes,” “follows,” or “shares” on social media.
- the trained predictive model described with reference to block 542 is trained in advance of receiving the marketing asset (block 502), as part of a training process (see Figures 3 and 4 and accompanying description.).
- Training of the predictive model may involve, for example, utilizing a training set having a multitude (perhaps millions or more) of scores (and possibly other data) associated with discrete marketing messages, along with corresponding known marketing outcome data.
- training of the predictive model may involve using a method having one or more blocks similar to blocks 502-540, which parse a multitude of marketing assets (having known marketing outcomes) into segmented components, for which a corresponding multitude of discrete marketing messages and associated scores are determined for use in training a predictive model.
- the scores (and possibly other data) associated with the discrete marketing messages are the independent variables and the marketing outcome data are the dependent variables.
- the training marketing outcome could be an engagement score, a web page bounce rate, a clickthrough rate, an occurrence of a sale, an occurrence of an engagement, or a specified conversion rate associated with each of the discrete marketing messages (or assets) used for training, for example.
- a training marketing outcome could be users selecting a “click and buy” icon for a particular product.
- a training marketing outcome could be to “like” or “share” a particular social media post.
- training a predictive model could optionally include determining a subset of the discrete marketing messages and/or associated scores that each have more than a threshold correlation to an associated marketing outcome.
- the predictive model training process involves determining which of the identified discrete messages from systems 530, 532, 534, and/or 536 are correlated with a marketing outcome.
- determining correlation might involve combining one or more discrete marketing messages identified by systems 530, 532, 534, and/or 536. In other examples, all discrete marketing messages identified by systems 530, 532, 534, and/or 536 will be part of a determined subset, so that the subset does not constitute a reduced number of discrete marketing messages. [00110]
- the correlation between discrete marketing messages and marketing outcomes can be determined based on data output by one or more predictive models (e.g., machine learning models) being used. Many such models provide correlation data, based on minimizing errors or other techniques.
- Other correlation measures e.g., least squares and others
- other threshold correlations may be used; the above are merely examples.
- the method 500 uses discrete marketing messages, rather than raw data (e.g., word frequency, text vectorization (word embeddings), demographics, emotions on faces).
- raw data e.g., word frequency, text vectorization (word embeddings), demographics, emotions on faces.
- Raw data has demonstrated to be too fragmented and cumbersome to be predictive. This is because raw data by itself has been found to produce too many predictive variables, require too much training data, and reduce the ability to infer causality from a predictive model.
- raw data is instead translated into discrete marketing messages, which might, for example, turn 100 independent variables into a single “discrete marketing message” independent variable, better prediction results have been realized. This corresponds to the feature-engineering layer of machine learning (see, e.g., feature vector(s) 322 in Figure 3).
- Such features can be used in standard ML models, like Ridge regression and Lasso regression, for example.
- Regression models tend to work best in marketing and sales contexts because the predicted marketing outcome can take the form of a continuous number (e.g., from 0 to X) that the method 500 is trying to predict. This is in contrast to classification models, which simply attempt to provide yes/no predictions, such as whether an image portrays a cat.
- raw data can be used to help supplement a prediction supplied by the machine learning model that utilizes the predictive variable relying on the discrete marketing message.
- the discrete marketing messages that are correlated with marketing outcomes are highly contextual.
- the components of the context could include, but are not limited to, the industry, the channel (email, web, social, etc.), the company characteristics, the marketing asset’s objective (to sell, to inform, etc.), and the audience attributes. Therefore, in some embodiments, building an accurate predictive model includes training the model using marketing assets having a similar context. Similarly, trained models could be categorized and selected based on context, in order to improve prediction accuracy.
- the predicted marketing outcome can be output 544, such as by displaying in a user interface (see, e.g., the screen shots of Figures 6A-6R, 7, and 8, illustrating output of a marketing outcome prediction) and/or printing in a printed output.
- the predicted marketing outcome can be stored 546 in a storage location (e.g., a snapshot of results published to a storage box location, such as those offered by box, *** drive, or dropbox), provided as an input to a marketing automation platform, such as one offered by Hubspot, Marketo, Mailchimp, or Salesforce, providing as an input to a digital asset management (DAM) system, such as one offered by bynder or Brandfolder, providing as a user experience (UX) input to a creative process system, or providing in a report, for example.
- DAM digital asset management
- UX user experience
- the output could be presented in a web- based Software-as-a-Service (SaaS) user interface, in one example.
- Such a SaaS user interface could serve as a web-based dashboard into which the predicted marketing outcome is pushed.
- the output might include a report of predicted social media outcomes, a predicted number of consumer comments, a predicted number of consumer evaluations (e.g., “likes” or “loves”), and/or a predicted number of consumer “shares.”
- the output may include predictions along with recommendations that tell how to improve performance.
- Figure 6A-6R are pictorial diagrams illustrating example screenshot displays showing example correlated discrete marketing messages (predictive/independent variables) and associated scores for each of those correlated discrete marketing messages.
- Such scores may be used to train a predictive model, as was described above. These same scores may also be used by the trained predictive models to predict a marketing outcome, as was described above with respect to block 542 in Figure 5.
- Each of Figures 6A-6R is briefly described below for the purpose of providing illustrative examples only; other industries, customers, products, etc., may have different types of discrete marketing messages and predicted marketing outcomes, according to example implementations.
- Figs. 6A-6M show correlated discrete marketing messages for the asset being analyzed (or for a user having a suite of assets), while Figs. 6N-6R show a comparative analysis between correlated discrete marketing messages of the user versus those of a competitor.
- Figure 6A illustrates a graphical display 602 of brand attributes 604 listed along ay-axis and relative emphasis 606 (in percentage) listed along an x-axis.
- the brand attributes 604 include value proposition, differentiation, reputation, brand promise, values, product features, product experience, and personas. Emphasis for each of these brand attributes ranges from 0% to 17%, with product features and value proposition being highest.
- a legend 608 provides information relating to brand attribute emphasis for a particular marketing asset being analyzed.
- Figure 6B illustrates a tabular display 610 of keyword density 612, with each keyword tab also including indicator of density.
- the keywords used most frequently in the asset being analyzed include “your” (1076), you (739), mailchimp (385), marketing (229), we (217), audience (182), and so on.
- a legend 615 provides information relating to keyword density, and explains a feature of highlighting one or more tabs as red or green (not shown in Fig. 6B) to signify keywords being brand matches or off-brand, respectively.
- Figure 6C illustrates a tabular display 616 of image objects and contents 618, with each image tab also including indicator of density.
- the images used most frequently in the asset being analyzed include no person (939), dark (663), nature (611), moon (587), bright (529), monochrome (475), eclipse (465), and so on.
- a legend 620 provides information relating to image contents, and explains a feature of highlighting one or more tabs as red or green (not shown in Fig. 6B) to signify images being brand matches or off-brand, respectively.
- Figure 6D illustrates a graphical display 622 of product features 624 listed along a y-axis and relative emphasis (in percentage) listed along an x-axis.
- the product features 624 include helpful content, email marketing, analytics, market segmentation, marketing automation platform, templates, landing page builder, advertising management, postcard marketing, and campaign management. Emphasis for each of these product features ranges from 6% to 17%, with helpful content and email marketing being highest.
- a legend 626 provides information relating to product feature emphasis for a particular marketing asset being analyzed.
- Figure 6E illustrates a graphical display 628 of product experience 630 listed along a y-axis and relative emphasis (in percentage) listed along an x-axis.
- the product experiences 630 include easy-to-use, flexible pricing, and affordable pricing. Emphasis for each of these product experiences ranges from 29% to 41%, with easy-to-use being highest.
- a legend 632 provides information relating to product experience emphasis for a particular marketing asset being analyzed.
- Figure 6F illustrates a graphical display 634 of differentiations 636 listed along a y-axis and relative emphasis (in percentage) listed along an x-axis.
- the differentiations 636 include free, top ranked, award-winning, and better support. Emphasis for each of these differentiations ranges from 3% to 79%, with free being highest.
- a legend 638 provides information relating to differentiation emphasis for a particular marketing asset being analyzed.
- Figure 6G illustrates a graphical display 640 of value propositions 642 listed along ay-axis and relative emphasis (in percentage) listed along an x-axis.
- the value propositions 642 include increase revenue, customer acquisition, build relationships, increase engagement, increase traffic, ROI, hit your goals, build a brand, increase loyalty, and increase conversion. Emphasis for each of these value propositions ranges from 2% to 41%, with increase revenue and customer acquisition being highest.
- a legend 644 provides information relating to value proposition emphasis for a particular marketing asset being analyzed.
- Figure 6H illustrates a graphical display 646 of archetypes 648 listed along ay- axis and relative emphasis (in percentage) listed along an x-axis.
- the archetypes 648 include creator, sage, lover, innocent, explorer, jester, everyman, ruler, hero, and caregiver. Emphasis for each of these archetypes ranges from 0% to 58%, with creator being highest.
- a legend 650 provides information relating to archetype emphasis for a particular marketing asset being analyzed.
- Figure 61 illustrates a graphical display 652 of reputations 654 listed along a y- axis and relative emphasis (in percentage) listed along an x-axis.
- the reputations 654 include simple, innovative, and trusted, ranging from 2% to 52%, with simple and innovative being highest.
- a legend 656 provides information relating to reputation emphasis for a particular marketing asset being analyzed.
- Figure 6J illustrates a graphical display 658 of habitats 658 listed along a y-axis and relative emphasis (in percentage) listed along an x-axis. As shown, the habitats 658 include a single “urban” habitat having an emphasis of 100%. A legend 662 provides information relating to habitat emphasis for a particular marketing asset being analyzed.
- Figure 6K illustrates a graphical display 664 of pastimes 666 listed along a y- axis and relative emphasis (in percentage) listed along an x-axis.
- the pastimes 666 include creating art, enjoying nature, travel, photography, social media, sports, shopping, cooking, writing, and running/jogging. Emphasis for each of these pastimes ranges from 1% to 55%, with creating art and enjoying nature being highest.
- a legend 668 provides information relating to pastimes emphasis for a particular marketing asset being analyzed.
- Figure 6L illustrates a graphical display 670 of personalities 672 listed along a y-axis and relative emphasis (in percentage) listed along an x-axis.
- the personalities 672 include creative, fun-loving, health-conscious, fashionable, refined, cool, hardworking, energetic, increasingly, and unique. Emphasis for each of these personalities ranges from 1% to 58%, with creative being highest.
- a legend 674 provides information relating to personalities emphasis for a particular marketing asset being analyzed.
- Figure 6M illustrates a graphical display 676 of professions 678 listed along a y-axis and relative emphasis (in percentage) listed along an x-axis.
- the professions 678 include sales and marketing, education, information worker, construction/craftsman, information technology, entrepreneur, writer, arts, agriculture, and journalism. Emphasis for each of these professions ranges from 1% to 58%, with creative being highest.
- a legend 680 provides information relating to professions emphasis for a particular marketing asset being analyzed.
- Figure 6N illustrates a graphical display 682 of brand attributes 684 listed along ay-axis and relative emphasis (in percentage) listed along an x-axis.
- brand attributes are listed for both a particular asset or user and for a competitor, such as a competitor pre specified by the user for comparison.
- the brand attributes 684 shown include value proposition, differentiation, reputation, brand promise, values, product features, product experience, and personas. The highest emphasis for both the user and competitor is on product features, with the competitor placing a slightly higher emphasis (23%) compared to the user (17%).
- a legend 686 provides information relating to brand attribute emphasis compared to competitors.
- Figure 60 illustrates a graphical display 688 of differentiations 689 listed along ay-axis and relative emphasis (in percentage) listed along an x-axis.
- differentiations 689 are listed for both a particular asset or user and for a competitor, such as a competitor pre specified by the user for comparison.
- the differentiations 689 shown include free, top ranked, award-winning, and better support.
- the highest emphasis for both the user and competitor is free, with the competitor placing a higher emphasis (90%) compared to the user (79%), while the user places a higher emphasis (13%) on top ranked compared to the competitor (10%).
- a legend 690 provides information relating to differentiations emphasis compared to competitors.
- Figure 6P illustrates a graphical display 688 of product features 692 listed along a y-axis and relative emphasis (in percentage) listed along an x-axis.
- product features 692 are listed for both a particular asset or user and for a competitor, such as a competitor pre-specified by the user for comparison.
- the product features 692 shown include helpful content, email marketing, analytics, market segmentation, marketing automation platform, templates, landing page builder, advertising management, postcard marketing, campaign management, CRM, automation integrations, meeting scheduler, social media marketing, online chat, and user management.
- the highest emphasis for the competitor is placed on CRM, templates, and email marketing, while the highest emphasis for the user is placed on helpful content and email marketing.
- a legend 693 provides information relating to product feature emphasis compared to competitors.
- Figure 6Q illustrates a graphical display 694 of personalities 695 listed along a y-axis and relative emphasis (in percentage) listed along an x-axis.
- personalities 695 are listed for both a particular asset or user and for a competitor, such as a competitor pre specified by the user for comparison.
- the personalities 695 shown include creative, fun-loving, health-conscious, fashionable, refined, cool, hardworking, energetic, increasingly, unique, trendy, and progressive. The highest emphasis for both the user and the competitor is placed on creative (58% and 36%).
- a legend 696 provides information relating to personality emphasis compared to competitors.
- Figure 6R illustrates a graphical display 697 of pastimes 698 listed along a y- axis and relative emphasis (in percentage) listed along an x-axis.
- pastimes 698 are listed for both a particular asset or user and for a competitor, such as a competitor pre-specified by the user for comparison.
- the pastimes 698 shown include creating art, enjoying nature, travel, photography, social media, sports, shopping, cooking, writing, running/jogging, driving, and arts and crafts. The highest emphasis for both the user and the competitor is placed on creating art (55% and 50%), followed by enjoying nature (31% and 21%).
- a legend 690 provides information relating to pastimes emphasis compared to competitors.
- FIG. 7 is a pictorial screenshot diagram illustrating example output 700 of an example marketing outcome prediction.
- the output 700 includes the asset/communication being analyzed — in this case, an image 702 of a mountain bike and corresponding text 704 relating to the image 702.
- the output also includes a predicted marketing outcome 706 — in this case a predicted engagement score of 46%.
- the illustrated example includes tips 708 and 710 on how to increase visual and textual engagement.
- Figure 8 is a pictorial screenshot diagram illustrating an example user interface 800 in the form of a dashboard report providing example marketing-outcome prediction information.
- the user interface 800 can provide outputs including a brand identification 802, channel 804, a graphical plot 806 of messaging emphasis for the specific brand and channel, a graphical plot 808 of competitor messaging emphasis, a chart 810 graphically illustrating a best practices and standards score (81%) in this example, and a plot 812 showing a best practices and standards adherence trend over time.
- the best practices and standards adherence score was 81%.
- the score was 82%, a difference of 1%.
- That difference is illustrated in chart 810 as segment 814, signifying a portion decreased (if in red) or increased (if in green) from a previous time period’s score.
- the metrics and analytics illustrated in Figure 8 are merely examples, and many other or additional metrics, analytics, and other dashboards may be included or substituted.
- the information illustrated in Figures 6A-6R may be included in the user interface 800.
- the user interface 800 may also receive user inputs, such as to select or modify which information is displayed.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Strategic Management (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Accounting & Taxation (AREA)
- Entrepreneurship & Innovation (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Economics (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Molecular Biology (AREA)
- Game Theory and Decision Science (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Human Resources & Organizations (AREA)
- Primary Health Care (AREA)
- Tourism & Hospitality (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202062970036P | 2020-02-04 | 2020-02-04 | |
US202062970051P | 2020-02-04 | 2020-02-04 | |
PCT/US2021/016539 WO2021158737A1 (en) | 2020-02-04 | 2021-02-04 | Predicting outcomes via marketing asset analytics |
Publications (2)
Publication Number | Publication Date |
---|---|
EP4100894A1 true EP4100894A1 (en) | 2022-12-14 |
EP4100894A4 EP4100894A4 (en) | 2024-01-10 |
Family
ID=83978914
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP21751184.9A Pending EP4100894A4 (en) | 2020-02-04 | 2021-02-04 | Predicting outcomes via marketing asset analytics |
Country Status (1)
Country | Link |
---|---|
EP (1) | EP4100894A4 (en) |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9633399B2 (en) * | 2009-08-19 | 2017-04-25 | Oracle International Corporation | Method and system for implementing a cloud-based social media marketing method and system |
US10572908B2 (en) * | 2017-01-03 | 2020-02-25 | Facebook, Inc. | Preview of content items for dynamic creative optimization |
US10789610B2 (en) * | 2017-09-08 | 2020-09-29 | Adobe Inc. | Utilizing a machine learning model to predict performance and generate improved digital design assets |
US10991012B2 (en) * | 2017-11-01 | 2021-04-27 | Adobe Inc. | Creative brief-based content creation |
-
2021
- 2021-02-04 EP EP21751184.9A patent/EP4100894A4/en active Pending
Also Published As
Publication number | Publication date |
---|---|
EP4100894A4 (en) | 2024-01-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11195203B2 (en) | Predicting outcomes via marketing asset analytics | |
US12010073B2 (en) | Systems and processes for operating and training a text-based chatbot | |
US11568260B2 (en) | Exponential modeling with deep learning features | |
US11748555B2 (en) | Systems and methods for machine content generation | |
US11710136B2 (en) | Multi-client service system platform | |
US11775494B2 (en) | Multi-service business platform system having entity resolution systems and methods | |
US20220206993A1 (en) | Multi-service business platform system having custom object systems and methods | |
US11544177B2 (en) | Mapping of test cases to test data for computer software testing | |
US11710070B2 (en) | Machine learned model framework for screening question generation | |
US10726025B2 (en) | Standardized entity representation learning for smart suggestions | |
US11308505B1 (en) | Semantic processing of customer communications | |
US11163845B2 (en) | Position debiasing using inverse propensity weight in machine-learned model | |
US11204973B2 (en) | Two-stage training with non-randomized and randomized data | |
Pröllochs et al. | Negation scope detection for sentiment analysis: A reinforcement learning framework for replicating human interpretations | |
JP2018128942A (en) | Analyzing apparatus, analyzing method, and program | |
CN114662696A (en) | Time series exception ranking | |
US20230418793A1 (en) | Multi-service business platform system having entity resolution systems and methods | |
Khan et al. | Comparative analysis on Facebook post interaction using DNN, ELM and LSTM | |
Fan et al. | A text analytics framework for automated communication pattern analysis | |
CN115659995A (en) | Text emotion analysis method and device | |
Gurcan | What issues are data scientists talking about? Identification of current data science issues using semantic content analysis of Q&A communities | |
US20230316186A1 (en) | Multi-service business platform system having entity resolution systems and methods | |
EP4100894A1 (en) | Predicting outcomes via marketing asset analytics | |
KR102669575B1 (en) | Method and apparatus for optimizing customized marketing funnel based on artificial intelligence model | |
US11977843B2 (en) | Dynamic ontology for intelligent data discovery |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20220902 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: INTUIT INC. |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230522 |
|
A4 | Supplementary search report drawn up and despatched |
Effective date: 20231213 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G10L 15/26 20060101ALN20231207BHEP Ipc: G06F 40/216 20200101ALI20231207BHEP Ipc: G06N 3/08 20060101ALI20231207BHEP Ipc: G06N 3/045 20230101ALI20231207BHEP Ipc: G06F 40/30 20200101ALI20231207BHEP Ipc: G06F 40/284 20200101ALI20231207BHEP Ipc: G06F 40/137 20200101ALI20231207BHEP Ipc: G06F 40/131 20200101ALI20231207BHEP Ipc: G06Q 30/0242 20230101ALI20231207BHEP Ipc: G06Q 50/00 20120101ALI20231207BHEP Ipc: G06Q 30/0204 20230101ALI20231207BHEP Ipc: G06Q 10/00 20120101ALI20231207BHEP Ipc: G06N 99/00 20190101ALI20231207BHEP Ipc: G06N 7/00 20060101ALI20231207BHEP Ipc: G06F 16/00 20190101ALI20231207BHEP Ipc: G06N 20/00 20190101AFI20231207BHEP |