WO2016139666A1 - Predictive strategic outcomes by combining human crowdsourcing - Google Patents

Predictive strategic outcomes by combining human crowdsourcing Download PDF

Info

Publication number
WO2016139666A1
WO2016139666A1 PCT/IL2016/050237 IL2016050237W WO2016139666A1 WO 2016139666 A1 WO2016139666 A1 WO 2016139666A1 IL 2016050237 W IL2016050237 W IL 2016050237W WO 2016139666 A1 WO2016139666 A1 WO 2016139666A1
Authority
WO
WIPO (PCT)
Prior art keywords
computer
human
data
inputs
indicators
Prior art date
Application number
PCT/IL2016/050237
Other languages
French (fr)
Inventor
Elad SCHAFFER
Daniel Green
Joel Zamel
Shay HERSHKOVITZ
Joel MARKS-BLUTH ALEXANDER
Gilad BAUM
Original Assignee
Wikistrat Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wikistrat Ltd. filed Critical Wikistrat Ltd.
Publication of WO2016139666A1 publication Critical patent/WO2016139666A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/067—Enterprise or organisation modelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G06Q10/101—Collaborative creation, e.g. joint development of products or services

Definitions

  • Crowdsourcing generally refers to methods for soliciting solutions to tasks via open calls to a large scale community. Crowd sourcing tasks are commonly broadcasted through a central website. The website may describe a task, offer a reward for completion of the task, and set a time limit in which to complete the task. A reward can be provided for merely participating in the task. The reward can also be provided as a prize for submitting the best solution or one of the best solutions. Thus, the reward can provide an incentive for members of the community to complete the task as well as to ensure the quality of the submissions.
  • a crowd sourcing community generally includes a network of members. For a given task, the expertise of members who are available, capable, and willing to participate in the task is finite. Further, only a subset of those members may provide the best solutions. As the number of crowd sourcing tasks increases, the number of desirable members who can contribute to the tasks may diminish. As a result, the ability to efficiently receive the most qualitative and quantitative inputs from the crowd sourcing community can be crucial with the increasing application of crowd sourcing as a means for completing tasks.
  • the present invention provides a computer-implemented method for creating predictive strategic developments and outcomes, comprising the steps of: generating a data model having a plurality of levels, namely: at least one "top level”, at least one "domain level”, at least one "category level” and at least one "indicator level” (100); receiving first human-based inputs for characterizing at least one of said data model's levels (30, 301, 302, 303); receiving computer-based inputs for populating said data model (40, 50, 401, 402); receiving second human-based inputs for populating said data model (303); generating a predictive strategic developments and outcomes (101, 60); and provide alerts and warnings to a user when an indicator (100) either crosses a predefined threshold (302) or is predicted to cross that threshold in the immediate future; where said step of generating a predictive strategic development and outcome further comprises steps of combining said computer-based inputs and said first and second human-based inputs (102, 103, 104, 302) to generate a processing module comprising
  • It is another object of the present invention to provide a computer-implemented method where said computer-based inputs for populating said data model comprises: searching a variety of digital sources for relevant information based on said indicators; parsing said information to extract the relevant key item; parsing said information by counting specific parameters in order to get a numerical value; storing said information in the memory of the system; integrating said information in said model; connecting to previously found digital sources to find updated information according to a schedule of how often said data is expected to change.
  • It is another object of the present invention to provide a computer-implemented method where said second human-based inputs for populating said data model comprises: analyzing said computer-based inputs; proposing scenarios for the development of a situation related to said indicators in the future, and derive implications to said indicators future weighting, scores and importance as well as to how said indicators will change in response to said scenarios' unfolding.
  • the present invention also provides a computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a computer, cause the computer to: generate a data model having a variety of levels, namely: at least one "top level”, at least one "domain level”, at least one "category level” and at least one "indicator level” (100); generate a processing module for combining computer-based inputs and human-based inputs (101, 102, 103, 104, 302); and generate a predictive strategic development and outcome (60) by combining said computer-based inputs and said human-based inputs according to said processing module.
  • the present invention also provides a computer system for creating predictive strategic developments and outcomes in a crowd sourcing application and big data information retrieval, comprising: a processor; a memory communicatively coupled to the processor; and a task distribution module which executes in the processor from the memory and which, when executed by the processor, causes the computer system to create predictive strategic developments and outcomes based on inputs from the crowd sourcing by: generating a model based on indicators (100, 104); receiving human-based inputs for defining said indicators (30, 302, 303); receiving computer-based inputs for populating said data model (40, 10, 402); receiving a second human-based inputs for populating said data model (30, 302, 303); generating a processing module for combining said computer- based inputs and said second human-based inputs (102, 103, 104, 302); and generating a predictive strategic development and outcome (60) by combining said computer-based inputs and said second human-based inputs according to said processing module.
  • Fig. 1 is a schematic example of the present invention crowdsourcing and machine interactions, in accordance with some embodiments.
  • Fig. 2a is a schematic example of how the model data tree-based can be constructed, in accordance with some embodiments.
  • Fig. 2b is a GUI example of how the model data tree-based is applied as described in Figure 2a, in accordance with some embodiments.
  • Fig. 3 is an example of how the crowd can provide scale scores/weights to each different indicator, in accordance with some embodiments.
  • Fig. 4a is a schematic table of the Iterative process that refines the model, in accordance with some embodiments.
  • Fig. 4b is a GUI example for managing domains and weights, in accordance with some embodiments.
  • Fig. 4c is a GUI example for managing the indicators, in accordance with some embodiments. Detail description of the Preferred Embodiment
  • the present invention provides an automated system combined with gamification, collaborative expert crowd to build data models (through identifying, rating, scaling and weighting indicators and their sources), in order to monitor entities and trends, identify early warnings and provide recommendations.
  • the invention provides an advanced monitoring, analysis and early warning system which leverages innovative research and analysis techniques and technologies. Based on a variety of sources (e.g. experts, open source, big data analysis)
  • the invention enables decision makers in real time to: identify trends, events, shocks and changes in a given ecosystem (i.e. geopolitical stability); monitor specific entities (i.e. commodities, individuals, competitors, financial assets); receive data-model driven predictions, early warnings and risk analysis, accompanied with policy recommendations.
  • a given ecosystem i.e. geopolitical stability
  • monitor specific entities i.e. commodities, individuals, competitors, financial assets
  • the present invention comprises data models which integrates and synthesizes insights, information and analysis from numerous sources, in real time.
  • the expert crowd is both source of information, a “tool” for refining the model, a “tool” for assessing the relevancy and weight and scale of each indicator, and also a source for analysis and predictions.
  • the present invention can be use in a wide spectrum of fields: political, security, economic and social stability; evaluating risks emanating from civil disorder, terror and cyber activities; evaluating censorship regimes and other policies and their level of implementations; commodity tracking and price/supply/demand/ prediction, and financial market applications;
  • the invention includes a unique methodology that incorporates various indicators of different data types to a single or several, holistic analytic outputs: statistical indicators (i.e. GDP, financial indicators); big-data indicators (social media sentiment analysis, media coverage); crowd based quantitative indicators (i.e.: voting); crowd based qualitative indicators (i.e.: expert opinion); automated weighting and scaling system; automated data gathering from open sources;
  • statistical indicators i.e. GDP, financial indicators
  • big-data indicators social media sentiment analysis, media coverage
  • crowd based quantitative indicators i.e.: voting
  • crowd based qualitative indicators i.e.: expert opinion
  • automated weighting and scaling system automated data gathering from open sources
  • the present invention is standalone system comprised of five major components:
  • Fig. 1 a schematic example of the interactions between the crowdsource (human-based inputs) and the system (machine -based inputs and process activities).
  • the complete system 1000 is comprised by the interactions between the main controlling application (ATA application) 10; the overall continuous quality control by the company 20; the crowd which provides human-based inputs 30; the machine -based inputs generally found in the Internet 40, through public and private data crawlers 50, and directed to the main controlling application (ATA application) 10; and, the client user 60, which can see in real-time the results of the system.
  • ATA application main controlling application
  • the first step of the present invention is to create at least one data model construct.
  • a model tree is a hierarchical analysis tree of how indicators together add up to an estimate of an issue (stability, prices, status of policy/event/phenomena).
  • the tree divides the subject at hand (example: regime stability) into high level domain branches, (such as economic, social, security) and each branch is further divided into categories. These categories are representative of the specific type(s) of instability or country (ies) that is (are) being analyzed. The categories can be further subdivided, until a level of a specific data indicator is reached.
  • the indicators are pulled from the database pool, and are used at the lowest level of the tree to start the model algorithm.
  • each leaf/connection/branch can be the weighted sum of the sub branches, and itself contains a score and a weighting. Other weighting metrics can be used as well.
  • indicators that are connected to multiple branches can be added several times, with different scales and weightings each time. (e.g. lower oil price in an oil exporting country will be scaled good for a branch looking at cost of living, but bad for a branch looking at oil export revenues of a country)
  • each tree takes the data and provides a results set
  • trigger - fast changing indicators representative of events that have potential to "trigger” events of a larger scale, e.g. snowballing protest events
  • stability metrics can be defined as distance in relation to previous instability events, by adding historical data
  • the “Top Level” (101) provides the main topic that is being analyzed.
  • the “Domain Level” (102) is the first branching of the tree, which divides the main topic into principal groups to be evaluated.
  • the “Category Level” (103) provides different types of sub topics in that can influence each "Domain”. This level can be further subdivided as needed until the final level is reached, which is the "Indicator Level” (104). Each indicator receives a score/weight that will influence the whole tree.
  • Fig. 2b is a GUI example of how the model data tree-based is applied as described in Figure 2a.
  • “Top Level” (101) in this case is named “Total”.
  • the "Domain Level” (102) in this example is “Security”.
  • the "Category Level” (103) in this example shows several categories, like “Successful Exercise of Statehood Functions", “effectiveness of Rule of Law", etc. Under the last one, it can be found “Civil Rights Enforcement / criminality”. This level is further subdivided until the final level is reached, which is the "Indicator Level” (104), in this example the “indicator” is "Rule of Law Index”.
  • the indicator have a score/weight (302) of 17.50.
  • the invention is capable of populating, in an automated matter, the feeds (Quantitative) - this data is directly retrieved and saved into a database on an ongoing basis.
  • Types of sources from which the present invention can populate the feeds either manually or automatically by the machine:
  • Websites relevant to the project financial information from relevant stock exchanges, financial databases; weather reports and data; supermarket prices in areas, actual vs published black market exchange data; following persons and topics relevant to the project on news websites; add several control entities by also looking at other similar topics with no direct connection, to provide a comparison for news spikes and to act as a normalizer in numerical analysis; news websites analysis with NLP (natural language processing), quantitative analysis; NGO and think tank social and economic data (e.g. World Bank, IMF, Transparency International); research data from journals and universities; government published statistics data; open source survey data (e.g. Gallup); open source data and databases; media collation databases (e.g.
  • GDELT internet usage and activity databases
  • Google Facebook, other open API monitoring services
  • social media monitoring Twitter and Facebook scraping and collation, analysis of qualitative and quantitative data
  • 3rd party data sources data collection databases
  • media monitoring companies for social media and traditional media
  • sentiment and semantic analysis services and, additionally to add several control entities by also looking at other similar topics with no direct connection, to provide a comparison for news spikes and to act as a normalizer in numerical analysis.
  • the present invention also describes the data input that Experts or Analysts provide (Qualitative and quantitative) - These data are manually fed into a database, on an ad-hoc or regular basis. It is generated leveraging a crowd of dozens or hundreds of vetted experts who work collaboratively or potentially not collaboratively, if done in a survey.
  • some examples for Expert inputs type are:
  • Weighting and scaling of indicators is based on comparisons with historical data, reference groups, current trends and expert assessments. All these can be blended together with different importance given, and changed based on time and results.
  • the experts use two interrelated crowdsourcing methodologies:
  • Collaborative competition a group of subject-matter experts are formulating, simultaneously, an in-depth analysis regarding a specific topic or question. Competing over a prevailing opinions and hypothesis, generating sets of scenarios and policy options.
  • Fig. 3 an example of how the crowd can provide scale scores/weights to each different indicator. Once the indicator reaches certain number in the "red zone", the machine alerts about it in the visualization panel (see below).
  • the refinement of the model is done by a bidirectional feedback mechanism that is comprised of an iterative process:
  • the calculated scores produced by the model algorithm are then presented to the analysts, who are required to assess them, score them, identify analytic gaps and suggest means to measure them and finally make predictions and suggest policy options on the future developments and/or outcome of various scenarios.
  • the entire methodology is implemented and executed using proprietary online platforms.
  • the crowd feedback is then re-implemented in the model to fine tune the relevant scores, scales and weightings - to produce more accurate results through collaboration and exchange between masses of analysts, through updating the model (automatically and manually) and through Machine-Learning (see below).
  • the model is benefitting from experts input, and vice versa (analysts fed by model).
  • the key iterative value is that as the amount of historical data grows, so does the possibility that changes in the indicators (thus, changes in the whole data model) can be detected algorithmically as it takes time for a "statistically significant" amount of data to be available.
  • Fig. 4a a schematic table of the Iterative process that refines the model. It is divided mainly in 3 zones: 1) the crowdsourced Analytic insights and Inputs (A) - which provide the human-based inputs; 2) the data model (B) - in which all the human and machine -based inputs are collected; and 3) the outputs and feedbacks (C), which receives further human and machine -based inputs.
  • A the crowdsourced Analytic insights and Inputs
  • B data model
  • C the outputs and feedbacks
  • the first step is to create the "Data Model” (100) which will include all the necessary parameters in order to provide the predictions.
  • Said Data Model (100) can be organized in a typical hierarchic tree (as shown also in Fig.
  • the expert crowd provide inputs in the form of scenarios, insights, risks, weights, scales, etc. (301, 302, 303). After that, information (inputs) commence to arrive from different sources.
  • the inputs are human-based inputs from an expert crowd (30, 301, 303) and machine-based inputs from a variety of on-line and off-line sources (40, 401, 50, 10, 402).
  • the inputs can be evaluated by the expert crowd (30) and/or the machine (10).
  • Fig. 4b is a GUI example for managing domains and weights.
  • “Top Level” (101) in this case is named “Total”.
  • the "Domain Level” (102) in this example is “Security”.
  • the "Category Level” (103) in this example shows a category called “Successful Exercise of Statehood Functions” having a score/weight (301) of 0.7.
  • Fig. 4C is a GUI example for managing the indicators.
  • the "Indicator” name is "Youth Dependency Ratio” (104).
  • the score is 57.93 in the scale as presented in the Fig.
  • the present invention provides an algorithm that can learn correlations in order to discover patterns and make predictions. For example, if an uptake in negative tweets, followed by a dip in the stock markets have previously shown to lead to riots, the algorithm can make that identification, look for this pattern in the future, and make a determination and suggest which values will most likely change next.
  • the same algorithm can learn to classify scraped data to make determinations as to how to represent that data numerically both from the crowd and from data which is scraped. This includes both data being provided by the crowd, and that collected from sources such as social media, blogs, news sites and other Internet-based sources.
  • this is by learning to look for important words and phrases and the order in which they appear as well as the semantic meaning of words. This means includes detecting if words represent entities (such as countries, political entities, persons of interest). Afterwards the algorithm can determine which part of the data model is potentially affected by the new information.
  • the model cannot make an automatic determination, it can queue the data to be shown to a human and request manual classification in order to learn the correct answer for the next time it is seen.
  • this request can be pushed to the crowd with a reward for the fastest solve in order to incentivize quick responses to not slow down the processing of information.
  • the algorithm should evolve to be able to automatically search for sources of information, and process crowd responses into a mathematical model.
  • the present invention provides a system capable of stress testing itself against: 1) Historical Data: the model can determine the date/time of information in order to construct a timeline. Therefore past information (in which the development and/or outcome is known) can be assessed by the model to determine if the correct result is predicted (or conversely alter the model/algorithm until the correct result is predicted); and, 2) Simulations/ Scenarios: fictional inputs can be provided to the model that would represent hypothetical situations to see how the model adjusts.
  • users are incentivized to contribute and are rewarded for correct predictions and active participation. This involves both human and machine based intervention. Rewards are both material (such as cash) and non- material (such as "badges” which are visual enhancements to how a user appears to other users on the site, and can be discussed by the user in real life). Incentives can be based on needs of the model, including paying higher rewards for more valuable information or bonuses for speed encouraging quick results.
  • the data model is presented via a live, internet based dashboard that shows the live state of the model.
  • Presentable data include:
  • Historical model data searchable by date, compare model developments and/or outputs on different dates, searchable by levels of indicator value - e.g. a search for all indicators scaled lower than 33 to highlight all "at risk” indicators).
  • users are able to input criteria which they would want to trigger an alert or warning via email or other messaging service when a part of the data model crosses one a threshold as defined at (302).
  • a threshold as defined at (302)
  • the alert can be triggered when a threshold is crossed, but also when the pace of change is reaching a certain level (for example, the threshold can be "20", but if there's a drop from 55 to 30, the system will alert, because of a change in pace).

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Strategic Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Economics (AREA)
  • Tourism & Hospitality (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Game Theory and Decision Science (AREA)
  • Educational Administration (AREA)
  • Development Economics (AREA)
  • Data Mining & Analysis (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The present invention provides a computer-implemented method for creating predictive strategic developments and outcomes, comprising the steps of: generating a data model having a plurality of levels, namely: at least one "top level", at least one "domain level", at least one "category level" and at least one "indicator level"; receiving first human- based inputs for characterizing at least one of said data model's levels; receiving computer-based inputs for populating said data model; receiving second human -based inputs for populating said data model; generating a predictive strategic developments and outcomes; and provide alerts and warnings to a user when an indicator either crosses a predefined threshold or is predicted to cross that threshold in the immediate future; where said step of generating a predictive strategic development and outcome further comprises steps of combining said computer-based inputs and said first and second human-based inputs to generate a processing module comprising self-iterating system-dynamics analysis based on said human and machine inputs.

Description

PREDICTIVE STRATEGIC OUTCOMES BY COMBINING HUMAN
CROWDSOURCING
Background
Crowdsourcing generally refers to methods for soliciting solutions to tasks via open calls to a large scale community. Crowd sourcing tasks are commonly broadcasted through a central website. The website may describe a task, offer a reward for completion of the task, and set a time limit in which to complete the task. A reward can be provided for merely participating in the task. The reward can also be provided as a prize for submitting the best solution or one of the best solutions. Thus, the reward can provide an incentive for members of the community to complete the task as well as to ensure the quality of the submissions.
A crowd sourcing community generally includes a network of members. For a given task, the expertise of members who are available, capable, and willing to participate in the task is finite. Further, only a subset of those members may provide the best solutions. As the number of crowd sourcing tasks increases, the number of desirable members who can contribute to the tasks may diminish. As a result, the ability to efficiently receive the most qualitative and quantitative inputs from the crowd sourcing community can be crucial with the increasing application of crowd sourcing as a means for completing tasks.
Thus, there is a long felt need for a solution that can increase and facilitate the crowdsourcing system with a vast amount of inputs from an intelligent machine-based source and the tools to analyze the overall of inputs in an efficient and productive way. It is with respect to these and other considerations that the disclosure made herein is presented.
Summary
The present invention provides a computer-implemented method for creating predictive strategic developments and outcomes, comprising the steps of: generating a data model having a plurality of levels, namely: at least one "top level", at least one "domain level", at least one "category level" and at least one "indicator level" (100); receiving first human-based inputs for characterizing at least one of said data model's levels (30, 301, 302, 303); receiving computer-based inputs for populating said data model (40, 50, 401, 402); receiving second human-based inputs for populating said data model (303); generating a predictive strategic developments and outcomes (101, 60); and provide alerts and warnings to a user when an indicator (100) either crosses a predefined threshold (302) or is predicted to cross that threshold in the immediate future; where said step of generating a predictive strategic development and outcome further comprises steps of combining said computer-based inputs and said first and second human-based inputs (102, 103, 104, 302) to generate a processing module comprising self-iterating system- dynamics analysis based on said human and machine inputs.
It is an object of the present invention to provide a computer-implemented method where said creating a data model comprises: creating at least one interactive tree format comprising branches; assigning high level domains to branches; dividing said branches into categories; further dividing said categories into sub-categories as necessary; arriving at a final level of branch and assigning a specific indicator to said level.
It is another object of the present invention to provide a computer-implemented method where said first human-based inputs for creating said data model comprises: identifying new indicators; proposing scales for said new indicators; suggesting ratings and scores per said new indicators; suggesting relative weighting per said new indicators; suggesting thresholds for said new indicators; providing measures of importance for said new indicators; providing input on half-life definitions for said inputs.
It is another object of the present invention to provide a computer-implemented method where said computer-based inputs for populating said data model comprises: searching a variety of digital sources for relevant information based on said indicators; parsing said information to extract the relevant key item; parsing said information by counting specific parameters in order to get a numerical value; storing said information in the memory of the system; integrating said information in said model; connecting to previously found digital sources to find updated information according to a schedule of how often said data is expected to change. It is another object of the present invention to provide a computer-implemented method where said variety of digital sources are selected from a group consisting of websites, open-source data and databases, social-media monitoring, data collection databases, media monitoring companies for social media and traditional media, sentiment and semantic analysis services, and any combination thereof.
It is another object of the present invention to provide a computer-implemented method where said second human-based inputs for populating said data model comprises: analyzing said computer-based inputs; proposing scenarios for the development of a situation related to said indicators in the future, and derive implications to said indicators future weighting, scores and importance as well as to how said indicators will change in response to said scenarios' unfolding.
It is another object of the present invention to provide a computer-implemented method where said first and second human-based inputs are ranked in order to conduct internal rankings of analysts based on predetermined expertise, accuracy in surveys compared to the data trends and allows weighting of experts in further surveys and participations.
It is another object of the present invention to provide a computer-implemented method where said first and second human-based inputs are performed in a collaborative competitive matter.
It is another object of the present invention to provide a computer-implemented method where said predictive strategic developments and outcomes are presented in a score-scale format.
It is another object of the present invention to provide a computer-implemented method where said predictive strategic developments and outcomes are updated in an iterative bidirectional feedback process.
It is another object of the present invention to provide a computer-implemented method where said process comprises: asking human inputs to quantify the relative importance of said model's indicators and categories, thus producing the weightings for said score-scale format; presenting said predictive strategic developments and outcomes in said score- scale format produced by said processing module to said human inputs, then requiring said humans to assess, score, identify analytic gaps and suggest means to measure said predictive strategic developments and outcomes and finally make predictions and suggest policy options on the future development and outcome of various scenarios; re- implementing said bidirectional feedback process in said model to fine tune the relevant scores, scales and weightings in order to produce more accurate results; exchanging information and collaboration between masses of humans; updating automatically and manually the model.
It is another object of the present invention to provide a computer-implemented method where said processing module further comprises computer-implemented operations for machine learning.
It is another object of the present invention to provide a computer-implemented method where said machine learning comprises: learning correlations in order to discover patterns and make predictions; learning to classify scraped data to make determinations as to how to represent said data numerically both from the human-based inputs and from said data which is scraped; learning to look for important words and phrases and the order in which they appear as well as the semantic meaning of words; learning to detect if words represent entities; learning to determine which part of said data model is potentially affected by the new input; learning from human-based determinations which part of said data model is potentially affected by the new input; evolving to be able to automatically search for sources of information, and process human-based responses into a mathematical model.
It is another object of the present invention to provide a computer-implemented method where said processing module comprises operations for stress testing itself using known development and outcome-historical data to assess said processing module responses.
It is another object of the present invention to provide a computer-implemented method where said processing module comprises operations for stress testing itself using simulations and fictional scenarios to assess said processing module responses. It is another object of the present invention to provide a computer-implemented method where operations of gamification are used to incentivize said first and second human- based inputs.
It is another object of the present invention to provide a computer-implemented method where said operations of gamification include rewards that can be both material and non- material.
It is another object of the present invention to provide a computer-implemented method where incentives can be based on needs of said processing module, including paying higher rewards for more valuable information or bonuses for speed encouraging quick results. It is another object of the present invention to provide a computer-implemented method where said predictive strategic developments and outcomes are presented via a live, internet based, dashboard showing the live state of said processing module; said presented outcomes include: visualization of tree model; total scores for lead indicators, for said domains and for overall rating of the topic; demonstration via numbers, gauges or other matrix; presentation of thresholds and proximity to such, downstream scenarios in case of crossing thresholds; historical model data; trends analysis to show: correlations between indicators; trending indicators; graphs and other visual representations of data model and experts input; graphs of trigger vs baseline, with a time axis represented by color, allowing graphing of proximity of current events to past events in a 2D way; historical data used to create zones of "danger".
The present invention also provides a computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a computer, cause the computer to: generate a data model having a variety of levels, namely: at least one "top level", at least one "domain level", at least one "category level" and at least one "indicator level" (100); generate a processing module for combining computer-based inputs and human-based inputs (101, 102, 103, 104, 302); and generate a predictive strategic development and outcome (60) by combining said computer-based inputs and said human-based inputs according to said processing module.
It is an object of the present invention to provide a computer-readable storage medium where to generate said predictive strategic development and outcome according to said processing module, the computer-readable storage medium having computer-executable instructions stored thereon which, when executed by the computer, cause the computer to run said processing module with prior performances of human inputs with respect to historical inputs.
It is another object of the present invention to provide a computer-readable storage medium where to generate said predictive strategic development and outcome according to said processing module, the computer-readable storage medium having computer- executable instructions stored thereon which, when executed by the computer, cause the computer to learn from past runnings of said processing module.
It is another object of the present invention to provide a computer-readable storage medium where said learning comprises: learning which data matters; learning to collate the relevant data; learning to correlate the relevant data; learning from human-based inputs; learning how the importance of all relevant data changes and develops over time.
It is another object of the present invention to provide a computer-readable storage medium where to generate said predictive strategic development and outcome according to said processing module, the computer-readable storage medium having computer- executable instructions stored thereon which, when executed by the computer, cause the computer to run stress tests of said processing module against historical data and fictional simulations and scenarios.
It is another object of the present invention to provide a computer-readable storage medium, having computer-executable instructions stored thereon which, when executed by the computer, further cause the computer to: search a variety of digital sources for relevant information based on said indicators; parse said information to extract the relevant key item, so if it knows that the price of petrol is always in the format; parse said information by counting specific parameters in order to get a numerical value; store said information in said model; connect to previously found digital sources to find updated information according to a schedule of how often that data is expected to change.
It is another object of the present invention to provide a computer-readable storage medium where said variety of digital sources are selected from a group consisting of websites, open-source data and databases, social-media monitoring, data collection databases, media monitoring companies for social media and traditional media, sentiment and semantic analysis services, and any combination thereof.
It is another object of the present invention to provide a computer-readable storage medium, having computer-executable instructions stored thereon which, when executed by the computer, further cause the computer to update said predictive strategic developments and outcomes in an iterative bidirectional feedback process, which causes said computer to: ask human inputs to quantify the relative importance of said model's indicators and categories, thus producing the weightings for said score-scale format; present said predictive strategic developments and outcomes in said score-scale format produced by said processing module to said humans, then requiring said humans to assess, score, identify analytic gaps and suggest means to measure said predictive strategic developments and outcomes and finally make predictions and suggest policy options on the future development and outcome of various scenarios; re-implement said bidirectional feedback process in said model to fine tune the relevant scores, scales and weightings in order to produce more accurate results; ask for exchange information and collaboration between masses of humans; update automatically the model.
It is another object of the present invention to provide a computer-readable storage medium, having computer-executable instructions stored thereon which, when executed by the computer, further cause the computer to begin operations of gamification to incentivize said human-based inputs.
It is another object of the present invention to provide a computer-readable storage medium where said operations of gamification include rewards that can be both material and non-material.
It is another object of the present invention to provide a computer-readable storage medium where incentives can be based on needs of said processing module, including paying higher rewards for more valuable information or bonuses for speed encouraging quick results. It is another object of the present invention to provide a computer-readable storage medium, having computer-executable instructions stored thereon which, when executed by the computer, further cause the computer to present said predictive strategic developments and outcomes via a live, internet based, dashboard showing the live state of said processing module; said presented developments and outcomes include: visualization of tree model; total scores for lead indicators, for said domains and for overall rating of the topic; demonstration via numbers, gauges or any other matrix; presentation of thresholds and proximity to such, downstream scenarios in case of crossing thresholds; historical model data; trends analysis to show: correlations between indicators; trending indicators; graphs and other visual representations of data model and experts input; graphs of trigger vs baseline, with a time axis represented by color, allowing graphing of proximity of current events to past events in a 2D way; historical data used to map zones of "danger", and measures, degrees or measurements of proximity to said zones of "danger".
The present invention also provides a computer system for creating predictive strategic developments and outcomes in a crowd sourcing application and big data information retrieval, comprising: a processor; a memory communicatively coupled to the processor; and a task distribution module which executes in the processor from the memory and which, when executed by the processor, causes the computer system to create predictive strategic developments and outcomes based on inputs from the crowd sourcing by: generating a model based on indicators (100, 104); receiving human-based inputs for defining said indicators (30, 302, 303); receiving computer-based inputs for populating said data model (40, 10, 402); receiving a second human-based inputs for populating said data model (30, 302, 303); generating a processing module for combining said computer- based inputs and said second human-based inputs (102, 103, 104, 302); and generating a predictive strategic development and outcome (60) by combining said computer-based inputs and said second human-based inputs according to said processing module.
It is an object of the present invention to provide a computer system where said processing module comprises classifying inputs with a selected indicator from a plurality of indicators. It is another object of the present invention to provide a computer system where receiving computer-based inputs comprises receiving predictive outputs from said processing module based on features of said inputs identified through said computer analysis techniques; where receiving human-based inputs for creating predictive strategic developments and outcomes comprises receiving human votes, each of the human votes classifying the contribution with one of the plurality of indicators; and further where creating predictive strategic developments and outcomes by combining the computer- based inputs and the human-based inputs according to said processing module comprises assigning a weighting score to each of the plurality of indicators in order to correctly classify the input.
Brief description of the Drawings
Fig. 1 is a schematic example of the present invention crowdsourcing and machine interactions, in accordance with some embodiments.
Fig. 2a is a schematic example of how the model data tree-based can be constructed, in accordance with some embodiments.
Fig. 2b is a GUI example of how the model data tree-based is applied as described in Figure 2a, in accordance with some embodiments.
Fig. 3 is an example of how the crowd can provide scale scores/weights to each different indicator, in accordance with some embodiments.
Fig. 4a is a schematic table of the Iterative process that refines the model, in accordance with some embodiments.
Fig. 4b is a GUI example for managing domains and weights, in accordance with some embodiments.
Fig. 4c is a GUI example for managing the indicators, in accordance with some embodiments. Detail description of the Preferred Embodiment
In some embodiments, the present invention provides an automated system combined with gamification, collaborative expert crowd to build data models (through identifying, rating, scaling and weighting indicators and their sources), in order to monitor entities and trends, identify early warnings and provide recommendations.
In some embodiments, the invention provides an advanced monitoring, analysis and early warning system which leverages innovative research and analysis techniques and technologies. Based on a variety of sources (e.g. experts, open source, big data analysis)
In some embodiments, the invention enables decision makers in real time to: identify trends, events, shocks and changes in a given ecosystem (i.e. geopolitical stability); monitor specific entities (i.e. commodities, individuals, competitors, financial assets); receive data-model driven predictions, early warnings and risk analysis, accompanied with policy recommendations.
The present invention comprises data models which integrates and synthesizes insights, information and analysis from numerous sources, in real time.
In some embodiments of the present invention the expert crowd is both source of information, a "tool" for refining the model, a "tool" for assessing the relevancy and weight and scale of each indicator, and also a source for analysis and predictions.
In some embodiments, the present invention can be use in a wide spectrum of fields: political, security, economic and social stability; evaluating risks emanating from civil disorder, terror and cyber activities; evaluating censorship regimes and other policies and their level of implementations; commodity tracking and price/supply/demand/ prediction, and financial market applications;
In some embodiments, the invention includes a unique methodology that incorporates various indicators of different data types to a single or several, holistic analytic outputs: statistical indicators (i.e. GDP, financial indicators); big-data indicators (social media sentiment analysis, media coverage); crowd based quantitative indicators (i.e.: voting); crowd based qualitative indicators (i.e.: expert opinion); automated weighting and scaling system; automated data gathering from open sources;
In some embodiments, technically, the present invention, is standalone system comprised of five major components:
• Automated collection tools - with access to various and different data sources.
• Proprietary Algorithms - flexible, tailor made data mining capabilities and Big Data analytics, using statistical modelling and machine learning methods.
• Collaborative Expert crowd - Crowd source insights and analysis, community management and gamification.
• Dashboard and mobile app - Integrative and innovative presentation and delivery methods
• Calculated predictions and recommendations model - a unique methodology which synthesizes between the quantitative algorithms and the qualitative crowd- sourced analysis, using a bi-directional feedback mechanism.
Referring now to Fig. 1, a schematic example of the interactions between the crowdsource (human-based inputs) and the system (machine -based inputs and process activities). The complete system 1000 is comprised by the interactions between the main controlling application (ATA application) 10; the overall continuous quality control by the company 20; the crowd which provides human-based inputs 30; the machine -based inputs generally found in the Internet 40, through public and private data crawlers 50, and directed to the main controlling application (ATA application) 10; and, the client user 60, which can see in real-time the results of the system.
Data model constructs
In some embodiments, the first step of the present invention is to create at least one data model construct.
Multiple ways exist to build these data models. One of which is a tree format.
A model tree is a hierarchical analysis tree of how indicators together add up to an estimate of an issue (stability, prices, status of policy/event/phenomena). In some embodiments, the tree divides the subject at hand (example: regime stability) into high level domain branches, (such as economic, social, security) and each branch is further divided into categories. These categories are representative of the specific type(s) of instability or country (ies) that is (are) being analyzed. The categories can be further subdivided, until a level of a specific data indicator is reached. The indicators are pulled from the database pool, and are used at the lowest level of the tree to start the model algorithm.
In some embodiments, each leaf/connection/branch can be the weighted sum of the sub branches, and itself contains a score and a weighting. Other weighting metrics can be used as well.
In some embodiments, indicators that are connected to multiple branches can be added several times, with different scales and weightings each time. (e.g. lower oil price in an oil exporting country will be scaled good for a branch looking at cost of living, but bad for a branch looking at oil export revenues of a country)
Tree options and discussion:
In some embodiments, where multiple trees are created, each tree takes the data and provides a results set
• trees can be suggested by experts, previous analysis, trends analysis of comparing indicators to indexes
• the main tree presented to a client consists of an initial predetermined seed model constantly being updated
In some embodiments, for certain types of models, it is possible to have two separate trees for baseline and trigger indicators, or other types of indicators, to provide several dimensions (as well as a time axis)
• baseline - slow moving indicators
• trigger - fast changing indicators, representative of events that have potential to "trigger" events of a larger scale, e.g. snowballing protest events • stability metrics can be defined as distance in relation to previous instability events, by adding historical data
Referring now to Fig. 2a a schematic example of how the model data tree-based can be constructed. The "Top Level" (101) provides the main topic that is being analyzed. The "Domain Level" (102) is the first branching of the tree, which divides the main topic into principal groups to be evaluated. The "Category Level" (103) provides different types of sub topics in that can influence each "Domain". This level can be further subdivided as needed until the final level is reached, which is the "Indicator Level" (104). Each indicator receives a score/weight that will influence the whole tree.
Referring now to Fig. 2b is a GUI example of how the model data tree-based is applied as described in Figure 2a. "Top Level" (101) in this case is named "Total". The "Domain Level" (102) in this example is "Security". The "Category Level" (103) in this example shows several categories, like "Successful Exercise of Statehood Functions", "effectiveness of Rule of Law", etc. Under the last one, it can be found "Civil Rights Enforcement / Criminality". This level is further subdivided until the final level is reached, which is the "Indicator Level" (104), in this example the "indicator" is "Rule of Law Index". The indicator have a score/weight (302) of 17.50.
Inputs to the Data Model
In some embodiments, the invention is capable of populating, in an automated matter, the feeds (Quantitative) - this data is directly retrieved and saved into a database on an ongoing basis.
An example for a technique for web-scraping is shown. It is clear that might be other types of data or web scraping that might differ from the following example:
1. Connect to a website at a previously known location to find that information according to a schedule of how often that data is expected to change (i.e. minutely, hourly, daily, weekly, monthly, quarterly, etc.);
2. Parse that information to extract the relevant key items, as an example, if the program knows that the price of petrol is always in the format, "Today the price of petrol is $3 per liter" is will look for "petrol is" and "per liter" and then extract the middle component of that text;
3. Parse the information by counting references (such as word counting) or counting numbers of links, (etc.), to get a numerical value;
4. Parsing the information and examining where on a page the information is presented; e.g. measuring the importance of a news item based on where it appears on the front page of a news website;
5. Store that information in the model (including retaining the previous values)
Types of sources from which the present invention can populate the feeds, either manually or automatically by the machine:
Websites relevant to the project; financial information from relevant stock exchanges, financial databases; weather reports and data; supermarket prices in areas, actual vs published black market exchange data; following persons and topics relevant to the project on news websites; add several control entities by also looking at other similar topics with no direct connection, to provide a comparison for news spikes and to act as a normalizer in numerical analysis; news websites analysis with NLP (natural language processing), quantitative analysis; NGO and think tank social and economic data (e.g. World Bank, IMF, Transparency International); research data from journals and universities; government published statistics data; open source survey data (e.g. Gallup); open source data and databases; media collation databases (e.g. GDELT); internet usage and activity databases; Google, Facebook, other open API monitoring services; social media monitoring; Twitter and Facebook scraping and collation, analysis of qualitative and quantitative data; 3rd party data sources; data collection databases; media monitoring companies, for social media and traditional media; sentiment and semantic analysis services; and, additionally to add several control entities by also looking at other similar topics with no direct connection, to provide a comparison for news spikes and to act as a normalizer in numerical analysis.
In some embodiments, the present invention also describes the data input that Experts or Analysts provide (Qualitative and quantitative) - These data are manually fed into a database, on an ad-hoc or regular basis. It is generated leveraging a crowd of dozens or hundreds of vetted experts who work collaboratively or potentially not collaboratively, if done in a survey.
In some embodiments, some examples for Expert inputs type are:
• Identify new indicators, categories, ways of bundling indicators, connected indicators, theoretical connections between indicators through surveys, simulations, open discussions.
• Propose scales for indicators via surveys
• Propose options of types of scales and scaling
• Propose graphical presentation of sliding scale for easy interaction. E.g.: suggest rating/score per indicator after scaling, suggest relating weighting per indicator and category.
• Weighting and scaling of indicators is based on comparisons with historical data, reference groups, current trends and expert assessments. All these can be blended together with different importance given, and changed based on time and results.
• Suggest thresholds for when an indicator becomes more sensitive/dangerous/worth prompting, important levels for indicators through surveys, simulations and/or open discussions.
• Provide measures of importance for indicators, categories and quantitatively though surveys
• Executing data half-life surveys: data half-life is the concept that older data is less relevant as it ages. This is put into the model for specific data that is only published very sporadically, so allows it to be incorporated into the model with an aging process, {give example} for example, internet usage statistics are only released yearly, however since each year the data quickly becomes irrelevant (as there is a sharp increase), it is possible to add a "half-life" to decrease the relevance of the data over time. On very specific topics, surveys are performed of a subgroup within the expert community that are experts in the topic, to get scales for how the relevance of data degrades over time.
• Collaboratively propose scenarios for the development of a situation related to the indicators in the future, and derive implications to the indicators future weighting, scores and importance as well as to how indicators will change in response to scenarios' unfolding.
The above allows to conduct rankings of analysts based on predetermined expertise, accuracy in surveys compared to the data trends, allows weighting of experts in further surveys and participations. These rankings may not be shown to the experts, to avoid "group thinking".
In some embodiments, the experts use two interrelated crowdsourcing methodologies:
• Collaborative competition - a group of subject-matter experts are formulating, simultaneously, an in-depth analysis regarding a specific topic or question. Competing over a prevailing opinions and hypothesis, generating sets of scenarios and policy options.
• Voting - Ranging from simple multiple choice questionnaires to sophisticated 2 and 3 dimensional matrices on various issues and dilemmas
Referring now to Fig. 3 an example of how the crowd can provide scale scores/weights to each different indicator. Once the indicator reaches certain number in the "red zone", the machine alerts about it in the visualization panel (see below).
Iterative process of model refinement
In some embodiments, the refinement of the model is done by a bidirectional feedback mechanism that is comprised of an iterative process:
• Analysts are asked to quantify the relative importance of the model's numerous indicators and categories, thus producing the weightings for the algorithm.
• The calculated scores produced by the model algorithm are then presented to the analysts, who are required to assess them, score them, identify analytic gaps and suggest means to measure them and finally make predictions and suggest policy options on the future developments and/or outcome of various scenarios. The entire methodology is implemented and executed using proprietary online platforms. In some embodiments, the crowd feedback is then re-implemented in the model to fine tune the relevant scores, scales and weightings - to produce more accurate results through collaboration and exchange between masses of analysts, through updating the model (automatically and manually) and through Machine-Learning (see below).
In some embodiments, the model is benefitting from experts input, and vice versa (analysts fed by model).
In some embodiments, the key iterative value is that as the amount of historical data grows, so does the possibility that changes in the indicators (thus, changes in the whole data model) can be detected algorithmically as it takes time for a "statistically significant" amount of data to be available.
Referring now to Fig. 4a a schematic table of the Iterative process that refines the model. It is divided mainly in 3 zones: 1) the crowdsourced Analytic insights and Inputs (A) - which provide the human-based inputs; 2) the data model (B) - in which all the human and machine -based inputs are collected; and 3) the outputs and feedbacks (C), which receives further human and machine -based inputs.
In a "story-telling" way, the process will be now explained. The first step is to create the "Data Model" (100) which will include all the necessary parameters in order to provide the predictions. Said Data Model (100) can be organized in a typical hierarchic tree (as shown also in Fig. 2) that comprises: at least one "Top Level" (101) - which describes the main topic and the overall score of the prediction; at least one "Domain Level" (102) - which divides the "Top Level" (101) into main branches (in case there is more than one); at least one "Category Level" (103) - or also called "Sub-Domain", that further divides said "Domain Level" (102) into several categories that will be evaluated; there might be also Sub-sub-domains as necessary; and, "Indicator Level" (104) - which define the final descriptions that need to be evaluated.
Once the Data Model (100) is done, the expert crowd provide inputs in the form of scenarios, insights, risks, weights, scales, etc. (301, 302, 303). After that, information (inputs) commence to arrive from different sources. The inputs are human-based inputs from an expert crowd (30, 301, 303) and machine-based inputs from a variety of on-line and off-line sources (40, 401, 50, 10, 402).
At any point, the inputs can be evaluated by the expert crowd (30) and/or the machine (10). In fact, as time passes and more information/inputs arrive, the more accurate the predictions can be, and the more scenarios the machine learns.
Referring now to Fig. 4b is a GUI example for managing domains and weights. "Top Level" (101) in this case is named "Total". The "Domain Level" (102) in this example is "Security". The "Category Level" (103) in this example shows a category called "Successful Exercise of Statehood Functions" having a score/weight (301) of 0.7.
Referring now to Fig. 4C is a GUI example for managing the indicators. The "Indicator" name is "Youth Dependency Ratio" (104). The score is 57.93 in the scale as presented in the Fig.
Machine Learning
In some embodiments, the present invention provides an algorithm that can learn correlations in order to discover patterns and make predictions. For example, if an uptake in negative tweets, followed by a dip in the stock markets have previously shown to lead to riots, the algorithm can make that identification, look for this pattern in the future, and make a determination and suggest which values will most likely change next.
In some embodiments, the same algorithm can learn to classify scraped data to make determinations as to how to represent that data numerically both from the crowd and from data which is scraped. This includes both data being provided by the crowd, and that collected from sources such as social media, blogs, news sites and other Internet-based sources.
In some embodiments, this is by learning to look for important words and phrases and the order in which they appear as well as the semantic meaning of words. This means includes detecting if words represent entities (such as countries, political entities, persons of interest). Afterwards the algorithm can determine which part of the data model is potentially affected by the new information.
If the model cannot make an automatic determination, it can queue the data to be shown to a human and request manual classification in order to learn the correct answer for the next time it is seen.
In some embodiments, this request can be pushed to the crowd with a reward for the fastest solve in order to incentivize quick responses to not slow down the processing of information.
In some embodiments, therefore, given enough time, the algorithm should evolve to be able to automatically search for sources of information, and process crowd responses into a mathematical model.
Stress testing
In some embodiments, the present invention provides a system capable of stress testing itself against: 1) Historical Data: the model can determine the date/time of information in order to construct a timeline. Therefore past information (in which the development and/or outcome is known) can be assessed by the model to determine if the correct result is predicted (or conversely alter the model/algorithm until the correct result is predicted); and, 2) Simulations/ Scenarios: fictional inputs can be provided to the model that would represent hypothetical situations to see how the model adjusts.
Gamified environment
In some embodiments of the present invention, users are incentivized to contribute and are rewarded for correct predictions and active participation. This involves both human and machine based intervention. Rewards are both material (such as cash) and non- material (such as "badges" which are visual enhancements to how a user appears to other users on the site, and can be discussed by the user in real life). Incentives can be based on needs of the model, including paying higher rewards for more valuable information or bonuses for speed encouraging quick results.
Presentation of output:
In some embodiments, the data model is presented via a live, internet based dashboard that shows the live state of the model. Presentable data include:
• Visualization of tree model.
• Total scores for lead indicators, for domains and for overall rating of the topic ("stability in country X").
• Demonstrated via numbers, gauges, etc.
• Presentation of thresholds and proximity to such, downstream scenarios in case of crossing thresholds
• Historical model data (searchable by date, compare model developments and/or outputs on different dates, searchable by levels of indicator value - e.g. a search for all indicators scaled lower than 33 to highlight all "at risk" indicators).
• trends analysis to show
o correlations between indicators
o trending indicators
• Graphs and other visual representations of data model and experts inputs.
• Graphs of trigger vs baseline, with a time axis represented by color - this allows graphing of proximity of current events to past events in a 2D way. Historical data can be used to create zones of "danger", in which past historical levels of baseline and trigger instabilities has caused an extreme event.
Alerts and Warnings:
In some embodiments of the present invention, users are able to input criteria which they would want to trigger an alert or warning via email or other messaging service when a part of the data model crosses one a threshold as defined at (302). As the model evolves so does the ability to predict a future point at which an indicator will likely cross this threshold and provide early warning. The alert can be triggered when a threshold is crossed, but also when the pace of change is reaching a certain level (for example, the threshold can be "20", but if there's a drop from 55 to 30, the system will alert, because of a change in pace).

Claims

Claims
1. A computer-implemented method for creating predictive strategic developments and outcomes, comprising the steps of:
a. generating a data model having a plurality of levels, namely: at least one "top level", at least one "domain level", at least one "category level" and at least one "indicator level" (100);
b. receiving first human-based inputs for characterizing at least one of said data model's levels (30, 301, 302, 303);
c. receiving computer-based inputs for populating said data model (40, 50, 401, 402);
d. receiving second human-based inputs for populating said data model (303);
e. generating a predictive strategic developments and outcomes (101, 60);
and
f. provide alerts and warnings to a user when an indicator (100) either crosses a predefined threshold (302) or is predicted to cross that threshold in the immediate future or when the pace of change is reaching a certain level; wherein said step of generating a predictive strategic development and outcome further comprises steps of combining said computer-based inputs and said first and second human-based inputs (102, 103, 104, 302) to generate a processing module comprising self-iterating system-dynamics analysis based on said human and machine inputs.
2. The computer-implemented method of claim 1, wherein said creating a data model comprises:
a. creating at least one interactive tree format comprising branches; b. assigning high level domains to branches;
c. dividing said branches into categories;
d. further dividing said categories into sub-categories as necessary; e. arriving at a final level of branch and assigning a specific indicator to said level.
3. The computer-implemented method of claim 1, wherein said first human -based inputs for creating said data model comprises:
a. identifying new indicators;
b. proposing scales for said new indicators;
c. suggesting ratings and scores per said new indicators;
d. suggesting relative weighting per said new indicators;
e. suggesting thresholds for said new indicators;
f. providing measures of importance for said new indicators;
g. providing input on half-life definitions for said inputs;
4. The computer-implemented method of claim 1, wherein said computer-based inputs for populating said data model comprises:
a. searching a variety of digital sources for relevant information based on said indicators;
b. parsing said information to extract the relevant key item;
c. parsing said information by counting specific parameters in order to get a numerical value;
d. storing said information in the memory of the system;
e. integrating said information in said model;
f. connecting to previously found digital sources to find updated information according to a schedule of how often said data is expected to change.
5. The computer-implemented method of claim 4, wherein said variety of digital sources are selected from a group consisting of websites, open-source data and databases, social-media monitoring, data collection databases, media monitoring companies for social media and traditional media, sentiment and semantic analysis services, and any combination thereof.
6. The computer-implemented method of claim 1 , wherein said second human-based inputs for populating said data model comprises:
a. analyzing said computer-based inputs; b. proposing scenarios for the development of a situation related to said indicators in the future, and derive implications to said indicators future weighting, scores and importance as well as to how said indicators will change in response to said scenarios' unfolding.
7. The computer-implemented method of claim 1, wherein said first and second human-based inputs are ranked in order to conduct internal rankings of analysts based on predetermined expertise, accuracy in surveys compared to the data trends and allows weighting of experts in further surveys and participations.
8. The computer-implemented method of claim 1, wherein said first and second human-based inputs are performed in a collaborative competitive matter.
9. The computer-implemented method of claim 1, wherein said predictive strategic developments and outcomes are presented in a score-scale format.
10. The computer-implemented method of claim 1, wherein said predictive strategic developments and outcomes are updated in an iterative bidirectional feedback process.
11. The computer-implemented method of claim 10, wherein said process comprises: a. asking human inputs to quantify the relative importance of said model's indicators and categories, thus producing the weightings for said score- scale format;
b. presenting said predictive strategic developments and outcomes in said score-scale format produced by said processing module to said human inputs, then requiring said humans to assess, score, identify analytic gaps and suggest means to measure said predictive strategic developments and outcomes and finally make predictions and suggest policy options on the future development and outcome of various scenarios;
c. re-implementing said bidirectional feedback process in said model to fine tune the relevant scores, scales and weightings in order to produce more accurate results;
d. exchanging information and collaboration between masses of humans; and e. updating automatically and manually the model.
12. The computer-implemented method of claim 1, wherein said processing module further comprises computer-implemented operations for machine learning.
13. The computer-implemented method of claim 12, wherein said machine learning comprises:
a. learning correlations in order to discover patterns and make predictions; b. learning to classify scraped data to make determinations as to how to represent said data numerically both from the human-based inputs and from said data which is scraped;
c. learning to look for important words and phrases and the order in which they appear as well as the semantic meaning of words;
d. learning to detect if words represent entities;
e. learning to determine which part of said data model is potentially affected by the new input;
f. learning from human-based determinations which part of said data model is potentially affected by the new input; and
g. evolving to be able to automatically search for sources of information, and process human-based responses into a mathematical model;
14. The computer-implemented method of claim 1, wherein said processing module comprises operations for stress testing itself using known development and outcome -historical data to assess said processing module responses.
15. The computer-implemented method of claim 1, wherein said processing module comprises operations for stress testing itself using simulations and fictional scenarios to assess said processing module responses.
16. The computer-implemented method of claim 1, wherein operations of gamification are used to incentivize said first and second human-based inputs.
17. The computer-implemented method of claim 16, wherein said operations of gamification include rewards that can be both material and non-material.
18. The computer-implemented method of claim 16, wherein incentives can be based on needs of said processing module, including paying higher rewards for more valuable information or bonuses for speed encouraging quick results.
19. The computer-implemented method of claim 1, wherein said predictive strategic developments and outcomes are presented via a live, internet based, dashboard showing the live state of said processing module; said presented outcomes include:
a. visualization of tree model;
b. total scores for lead indicators, for said domains and for overall rating of the topic;
c. demonstration via numbers, gauges or other matrix;
d. presentation of thresholds and proximity to such, downstream scenarios in case of crossing thresholds;
e. historical model data;
f. trends analysis to show:
i. correlations between indicators;
ii. trending indicators;
g. graphs and other visual representations of data model and experts input; h. graphs of trigger vs baseline, with a time axis represented by color, allowing graphing of proximity of current events to past events in a 2D way; and
i. historical data used to create zones of "danger".
20. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a computer, cause the computer to:
a. generate a data model having a variety of levels, namely: at least one "top level", at least one "domain level", at least one "category level" and at least one "indicator level" (100);
b. generate a processing module for combining computer-based inputs and human-based inputs (101, 102, 103, 104, 302); and
c. generate a predictive strategic development and outcome (60) by combining said computer-based inputs and said human-based inputs according to said processing module.
21. The computer-readable storage medium of claim 20, wherein to generate said predictive strategic development and outcome according to said processing module, the computer-readable storage medium having computer-executable instructions stored thereon which, when executed by the computer, cause the computer to run said processing module with prior performances of human inputs with respect to historical inputs.
22. The computer-readable storage medium of claim 20, wherein to generate said predictive strategic development and outcome according to said processing module, the computer-readable storage medium having computer-executable instructions stored thereon which, when executed by the computer, cause the computer to learn from past runnings of said processing module.
23. The computer-readable storage medium of claim 22, wherein said learning comprises:
a. learning which data matters;
b. learning to collate the relevant data;
c. learning to correlate the relevant data;
d. learning from human -based inputs; and
e. learning how the importance of all relevant data changes and develops over time.
24. The computer-readable storage medium of claim 20, wherein to generate said predictive strategic development and outcome according to said processing module, the computer-readable storage medium having computer-executable instructions stored thereon which, when executed by the computer, cause the computer to run stress tests of said processing module against historical data and fictional simulations and scenarios.
25. The computer-readable storage medium of claim 20, having computer-executable instructions stored thereon which, when executed by the computer, further cause the computer to:
a. search a variety of digital sources for relevant information based on said indicators; b. parse said information to extract the relevant key item, so if it knows that the price of petrol is always in the format;
c. parse said information by counting specific parameters in order to get a numerical value;
d. store said information in said model; and
e. connect to previously found digital sources to find updated information according to a schedule of how often that data is expected to change.
26. The computer-readable storage medium of claim 25, wherein said variety of digital sources are selected from a group consisting of websites, open-source data and databases, social-media monitoring, data collection databases, media monitoring companies for social media and traditional media, sentiment and semantic analysis services, and any combination thereof.
27. The computer-readable storage medium of claim 20, having computer-executable instructions stored thereon which, when executed by the computer, further cause the computer to update said predictive strategic developments and outcomes in an iterative bidirectional feedback process, which causes said computer to:
a. ask human inputs to quantify the relative importance of said model's indicators and categories, thus producing the weightings for said score- scale format;
b. present said predictive strategic developments and outcomes in said score- scale format produced by said processing module to said humans, then requiring said humans to assess, score, identify analytic gaps and suggest means to measure said predictive strategic developments and outcomes and finally make predictions and suggest policy options on the future development and outcome of various scenarios;
c. re-implement said bidirectional feedback process in said model to fine tune the relevant scores, scales and weightings in order to produce more accurate results;
d. Ask for exchange information and collaboration between masses of humans; and
e. update automatically the model.
28. The computer-readable storage medium of claim 20, having computer-executable instructions stored thereon which, when executed by the computer, further cause the computer to begin operations of gamification to incentivize said human-based inputs.
29. The computer-readable storage medium of claim 28, wherein said operations of gamification include rewards that can be both material and non-material.
30. The computer-readable storage medium of claim 28, wherein incentives can be based on needs of said processing module, including paying higher rewards for more valuable information or bonuses for speed encouraging quick results.
31. The computer-readable storage medium of claim 20, having computer-executable instructions stored thereon which, when executed by the computer, further cause the computer to present said predictive strategic developments and outcomes via a live, internet based, dashboard showing the live state of said processing module; said presented developments and outcomes include:
a. visualization of tree model;
b. total scores for lead indicators, for said domains and for overall rating of the topic;
c. demonstration via numbers, gauges or any other matrix;
d. presentation of thresholds and proximity to such, downstream scenarios in case of crossing thresholds;
e. historical model data;
f. trends analysis to show:
i. correlations between indicators;
ii. trending indicators;
g. graphs and other visual representations of data model and experts input; h. graphs of trigger vs baseline, with a time axis represented by color, allowing graphing of proximity of current events to past events in a 2D way; and
i. historical data used to map zones of "danger", and measures, degrees or measurements of proximity to said zones of "danger".
32. A computer system for creating predictive strategic developments and outcomes in a crowd sourcing application and big data information retrieval, comprising: a. a processor;
b. a memory communicatively coupled to the processor; and
c. an task distribution module which executes in the processor from the memory and which, when executed by the processor, causes the computer system to create predictive strategic developments and outcomes based on inputs from the crowd sourcing by:
i. generating a model based on indicators (100, 104);
ii. receiving human-based inputs for defining said indicators (30, 302, 303);
iii. receiving computer-based inputs for populating said data model (40, 10, 402);
iv. receiving a second human-based inputs for populating said data model (30, 302, 303);
v. generating a processing module for combining said computer- based inputs and said second human-based inputs (102, 103, 104, 302); and
vi. generating a predictive strategic development and outcome (60) by combining said computer-based inputs and said second human- based inputs according to said processing module.
33. The computer system of claim 32, wherein said processing module comprises classifying inputs with a selected indicator from a plurality of indicators.
34. The computer system of claim 32, wherein receiving computer-based inputs comprises receiving predictive outputs from said processing module based on features of said inputs identified through said computer analysis techniques; wherein receiving human-based inputs for creating predictive strategic developments and outcomes comprises receiving human votes, each of the human votes classifying the contribution with one of the plurality of indicators; and further wherein creating predictive strategic developments and outcomes by combining the computer-based inputs and the human-based inputs according to said processing module comprises assigning a weighting score to each of the plurality of indicators in order to correctly classify the input.
PCT/IL2016/050237 2015-03-05 2016-03-02 Predictive strategic outcomes by combining human crowdsourcing WO2016139666A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201562128544P 2015-03-05 2015-03-05
US62/128,544 2015-03-05

Publications (1)

Publication Number Publication Date
WO2016139666A1 true WO2016139666A1 (en) 2016-09-09

Family

ID=56849344

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IL2016/050237 WO2016139666A1 (en) 2015-03-05 2016-03-02 Predictive strategic outcomes by combining human crowdsourcing

Country Status (1)

Country Link
WO (1) WO2016139666A1 (en)

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106815305A (en) * 2016-12-15 2017-06-09 安徽扬能电子科技有限公司 A kind of vehicle-mounted wisdom terminal system based on data analysis
CN107301519A (en) * 2017-06-16 2017-10-27 佛山科学技术学院 A kind of task weight pricing method in mass-rent express system
WO2019051136A1 (en) * 2017-09-11 2019-03-14 N3, Llc Dynamic script for tele-agents
CN109767131A (en) * 2019-01-15 2019-05-17 国家电网有限公司客户服务中心 Traffic emergency regulation method and system
US10623572B1 (en) 2018-11-21 2020-04-14 N3, Llc Semantic CRM transcripts from mobile communications sessions
CN111105103A (en) * 2020-02-04 2020-05-05 江苏星月测绘科技股份有限公司 Method for constructing space-time big data acquisition model based on crowdsourcing mode
US10742813B2 (en) 2018-11-08 2020-08-11 N3, Llc Semantic artificial intelligence agent
US10923114B2 (en) 2018-10-10 2021-02-16 N3, Llc Semantic jargon
US10972608B2 (en) 2018-11-08 2021-04-06 N3, Llc Asynchronous multi-dimensional platform for customer and tele-agent communications
CN113190673A (en) * 2021-04-01 2021-07-30 华南师范大学 Artificial intelligence report generation method and innovation-driven development strategy audit analysis system
US11132695B2 (en) 2018-11-07 2021-09-28 N3, Llc Semantic CRM mobile communications sessions
US11392960B2 (en) 2020-04-24 2022-07-19 Accenture Global Solutions Limited Agnostic customer relationship management with agent hub and browser overlay
US11443264B2 (en) 2020-01-29 2022-09-13 Accenture Global Solutions Limited Agnostic augmentation of a customer relationship management application
US11468882B2 (en) 2018-10-09 2022-10-11 Accenture Global Solutions Limited Semantic call notes
US11481785B2 (en) 2020-04-24 2022-10-25 Accenture Global Solutions Limited Agnostic customer relationship management with browser overlay and campaign management portal
US11507903B2 (en) 2020-10-01 2022-11-22 Accenture Global Solutions Limited Dynamic formation of inside sales team or expert support team
US11797586B2 (en) 2021-01-19 2023-10-24 Accenture Global Solutions Limited Product presentation for customer relationship management
US11816677B2 (en) 2021-05-03 2023-11-14 Accenture Global Solutions Limited Call preparation engine for customer relationship management
US11853930B2 (en) 2017-12-15 2023-12-26 Accenture Global Solutions Limited Dynamic lead generation
US12001972B2 (en) 2018-10-31 2024-06-04 Accenture Global Solutions Limited Semantic inferencing in customer relationship management

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130018685A1 (en) * 2011-07-14 2013-01-17 Parnaby Tracey J System and Method for Tasking Based Upon Social Influence
US20130282630A1 (en) * 2012-04-18 2013-10-24 Tagasauris, Inc. Task-agnostic Integration of Human and Machine Intelligence
US20140039985A1 (en) * 2011-06-29 2014-02-06 CrowdFlower, Inc. Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks
US20140172767A1 (en) * 2012-12-14 2014-06-19 Microsoft Corporation Budget optimal crowdsourcing
US20150046435A1 (en) * 2011-11-17 2015-02-12 Sri International Method and System Utilizing a Personalized User Model to Develop a Search Request

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140039985A1 (en) * 2011-06-29 2014-02-06 CrowdFlower, Inc. Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks
US20130018685A1 (en) * 2011-07-14 2013-01-17 Parnaby Tracey J System and Method for Tasking Based Upon Social Influence
US20150046435A1 (en) * 2011-11-17 2015-02-12 Sri International Method and System Utilizing a Personalized User Model to Develop a Search Request
US20130282630A1 (en) * 2012-04-18 2013-10-24 Tagasauris, Inc. Task-agnostic Integration of Human and Machine Intelligence
US20140172767A1 (en) * 2012-12-14 2014-06-19 Microsoft Corporation Budget optimal crowdsourcing

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106815305A (en) * 2016-12-15 2017-06-09 安徽扬能电子科技有限公司 A kind of vehicle-mounted wisdom terminal system based on data analysis
CN107301519A (en) * 2017-06-16 2017-10-27 佛山科学技术学院 A kind of task weight pricing method in mass-rent express system
WO2019051136A1 (en) * 2017-09-11 2019-03-14 N3, Llc Dynamic script for tele-agents
CN111542852B (en) * 2017-09-11 2023-08-22 埃森哲环球解决方案有限公司 Dynamic scenarios for telecommunications agents
US11475488B2 (en) 2017-09-11 2022-10-18 Accenture Global Solutions Limited Dynamic scripts for tele-agents
CN111542852A (en) * 2017-09-11 2020-08-14 N3有限责任公司 Dynamic scenarios for telecommunications agents
US11853930B2 (en) 2017-12-15 2023-12-26 Accenture Global Solutions Limited Dynamic lead generation
US11468882B2 (en) 2018-10-09 2022-10-11 Accenture Global Solutions Limited Semantic call notes
US10923114B2 (en) 2018-10-10 2021-02-16 N3, Llc Semantic jargon
US12001972B2 (en) 2018-10-31 2024-06-04 Accenture Global Solutions Limited Semantic inferencing in customer relationship management
US11132695B2 (en) 2018-11-07 2021-09-28 N3, Llc Semantic CRM mobile communications sessions
US10742813B2 (en) 2018-11-08 2020-08-11 N3, Llc Semantic artificial intelligence agent
US10951763B2 (en) 2018-11-08 2021-03-16 N3, Llc Semantic artificial intelligence agent
US10972608B2 (en) 2018-11-08 2021-04-06 N3, Llc Asynchronous multi-dimensional platform for customer and tele-agent communications
US10623572B1 (en) 2018-11-21 2020-04-14 N3, Llc Semantic CRM transcripts from mobile communications sessions
CN109767131A (en) * 2019-01-15 2019-05-17 国家电网有限公司客户服务中心 Traffic emergency regulation method and system
US11443264B2 (en) 2020-01-29 2022-09-13 Accenture Global Solutions Limited Agnostic augmentation of a customer relationship management application
CN111105103B (en) * 2020-02-04 2020-10-30 江苏星月测绘科技股份有限公司 Method for constructing space-time big data acquisition model based on crowdsourcing mode
CN111105103A (en) * 2020-02-04 2020-05-05 江苏星月测绘科技股份有限公司 Method for constructing space-time big data acquisition model based on crowdsourcing mode
US11481785B2 (en) 2020-04-24 2022-10-25 Accenture Global Solutions Limited Agnostic customer relationship management with browser overlay and campaign management portal
US11392960B2 (en) 2020-04-24 2022-07-19 Accenture Global Solutions Limited Agnostic customer relationship management with agent hub and browser overlay
US11507903B2 (en) 2020-10-01 2022-11-22 Accenture Global Solutions Limited Dynamic formation of inside sales team or expert support team
US11797586B2 (en) 2021-01-19 2023-10-24 Accenture Global Solutions Limited Product presentation for customer relationship management
CN113190673B (en) * 2021-04-01 2023-07-11 华南师范大学 Artificial intelligence report generation method and innovation driving development strategy audit analysis system
CN113190673A (en) * 2021-04-01 2021-07-30 华南师范大学 Artificial intelligence report generation method and innovation-driven development strategy audit analysis system
US11816677B2 (en) 2021-05-03 2023-11-14 Accenture Global Solutions Limited Call preparation engine for customer relationship management

Similar Documents

Publication Publication Date Title
WO2016139666A1 (en) Predictive strategic outcomes by combining human crowdsourcing
Bücker et al. Transparency, auditability, and explainability of machine learning models in credit scoring
Haleblian et al. High‐reputation firms and their differential acquisition behaviors
Abbasi Towards socially sustainable supply chains–themes and challenges
Rudolf et al. Key risks in the supply chain of large scale engineering and construction projects
CN113614757A (en) System and method for human-machine hybrid prediction of events
Kermanshachi et al. Development of the project complexity assessment and management framework for heavy industrial projects
Lane Democratizing our data: A manifesto
Hoepner et al. Significance, relevance and explainability in the machine learning age: an econometrics and financial data science perspective
Baizyldayeva et al. Multi-criteria decision support systems. Comparative analysis
Chisala Quantitative bid or no-bid decision-support model for contractors
Ziaja What do fragility indices measure? Assessing measurement procedures and statistical proximity
Lam et al. MBNQA‐oriented self‐assessment quality management system for contractors: fuzzy AHP approach
Shokouhyar et al. Implementing a fuzzy expert system for ensuring information technology supply chain
Miller Statistics for data science: Leverage the power of statistics for data analysis, classification, regression, machine learning, and neural networks
Ong Business intelligence and big data analytics for higher education: Cases from UK higher education institutions
Parthiban et al. An integrated multi-objective decision making process for the performance evaluation of the vendors
Crum et al. The use of cluster analysis in entrepreneurship research: Review of past research and future directions
Cuervo‐Cazurra et al. Host country politics and internationalization: a meta‐analytic review
Küpper et al. Features for social CRM technology–An organizational perspective
Ailenei Process mining tools: A comparative analysis
Ng et al. Classifying revenue management: A taxonomy to assess business practice
CN106575418A (en) Suggested keywords
Li et al. Risk assessment for supply chain based on Cloud model
Wu A comprehensive approach for the evaluation of the impact of blockchain on photovoltaic supply chain using hybrid data analytic method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16758560

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16758560

Country of ref document: EP

Kind code of ref document: A1