CN111813640A - Method, apparatus, electronic device and storage medium for predicting user behavior - Google Patents

Method, apparatus, electronic device and storage medium for predicting user behavior Download PDF

Info

Publication number
CN111813640A
CN111813640A CN202010219740.XA CN202010219740A CN111813640A CN 111813640 A CN111813640 A CN 111813640A CN 202010219740 A CN202010219740 A CN 202010219740A CN 111813640 A CN111813640 A CN 111813640A
Authority
CN
China
Prior art keywords
action
user
graph
actions
departure
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010219740.XA
Other languages
Chinese (zh)
Inventor
罗文娟
张含
杨晓庆
李奘
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Didi Infinity Technology and Development Co Ltd
Original Assignee
Beijing Didi Infinity Technology and Development Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Didi Infinity Technology and Development Co Ltd filed Critical Beijing Didi Infinity Technology and Development Co Ltd
Priority to CN202010219740.XA priority Critical patent/CN111813640A/en
Publication of CN111813640A publication Critical patent/CN111813640A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3438Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment monitoring of user actions
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Navigation (AREA)

Abstract

The present disclosure relates to a method, apparatus, electronic device, and storage medium for predicting user behavior. In one method, a user action graph is generated based on a plurality of historical actions of a plurality of types performed by a user in response to a first action performed by the user at a first point in time, the user action graph describing an order in which the plurality of historical actions were performed by the user. Based on the user action graph, temporal dependencies between the plurality of types are obtained. Based on the first action and the temporal dependency, a second action to be performed by the user at a second point in time after the first point in time is determined. Further, a corresponding apparatus, electronic device and storage medium are provided. With the implementation described above, the temporal dependency between multiple types can be obtained based on multiple historical actions including multiple types, thereby predicting the behavior of the user in a more efficient manner.

Description

Method, apparatus, electronic device and storage medium for predicting user behavior
Technical Field
Implementations of the present disclosure relate to user management, and more particularly, to methods, apparatuses, and computer storage media for predicting user behavior.
Background
With the development of computer technology and network technology, a large number of applications have been developed. The user may perform actions in the application in order to achieve the intended purpose. When the user performs different actions, the application will schedule different resources. Thus, it is desirable to be able to predict the user's behavior in order to manage applications in a more efficient manner.
Disclosure of Invention
It is desirable to develop and implement a solution that predicts user behavior in a more efficient manner. It is desirable that the solution be compatible with existing applications so that the applications can be managed in a more efficient manner.
According to a first aspect of the present disclosure, a method for predicting user behavior is provided. In the method, a user action graph is generated based on a plurality of historical actions of a plurality of types performed by a user in response to a first action performed by the user at a first point in time, the user action graph describing an order in which the plurality of historical actions were performed by the user. Based on the user action graph, temporal dependencies between the plurality of types are obtained. Based on the first action and the temporal dependency, a second action to be performed by the user at a second point in time after the first point in time is determined.
According to a second aspect of the present disclosure, an apparatus for predicting user behavior is provided. The device includes: a generation module configured to generate a user action graph based on a plurality of historical actions of a plurality of types performed by a user in response to a first action performed by the user at a first point in time, the user action graph describing an order in which the plurality of historical actions were performed by the user; an acquisition module configured to acquire a time dependency relationship between a plurality of types based on a user action graph; and a determination module configured to determine, based on the first action and the temporal dependency, a second action to be performed by the user at a second point in time after the first point in time.
According to a third aspect of the present disclosure, there is provided an electronic device comprising: a memory and a processor; wherein the memory is for storing one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to the first aspect of the disclosure.
According to a fourth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement a method according to the first aspect of the present disclosure.
Drawings
The features, advantages and other aspects of various implementations of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, which illustrate, by way of example and not by way of limitation, several implementations of the present disclosure. In the drawings:
FIG. 1 schematically illustrates a block diagram of an interface in which an application according to an exemplary implementation of the present disclosure may be used;
FIG. 2 schematically shows a block diagram of a process for determining user behavior based on multiple types of historical actions, according to an exemplary implementation of the present disclosure;
FIG. 3 schematically illustrates a flow chart of a method for determining user behavior according to an exemplary implementation of the present disclosure;
FIG. 4 schematically shows a block diagram of a process for processing a plurality of historical actions, according to an exemplary implementation of the present disclosure;
FIG. 5 schematically illustrates a block diagram of an operational procedure for generating temporal dependencies, according to an exemplary implementation of the present disclosure;
FIG. 6 schematically illustrates a block diagram of a process for collecting data from other users located around a user-specified origin, according to an exemplary implementation of the present disclosure;
FIG. 7 schematically illustrates a block diagram of a process for collecting data from other users destined for a user-specified origin, according to an exemplary implementation of the present disclosure;
FIG. 8 schematically shows a block diagram of a process for training temporal dependencies, according to an exemplary implementation of the present disclosure;
FIGS. 9A and 9B respectively schematically illustrate block diagrams of a preview interface displayed to a user according to an exemplary implementation of the present disclosure; and
fig. 10 schematically illustrates a block diagram of an apparatus for determining user behavior according to an exemplary implementation of the present disclosure.
Detailed Description
Preferred implementations of the present disclosure will be described in more detail below with reference to the accompanying drawings. While a preferred implementation of the present disclosure is shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited by the implementations set forth herein. Rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
The term "include" and variations thereof as used herein is meant to be inclusive in an open-ended manner, i.e., "including but not limited to". Unless specifically stated otherwise, the term "or" means "and/or". The term "based on" means "based at least in part on". The terms "one example implementation" and "one implementation" mean "at least one example implementation". The term "another implementation" means "at least one additional implementation". The terms "first," "second," and the like may refer to different or the same object. Other explicit and implicit definitions are also possible below.
An application environment of an exemplary implementation of the present disclosure is first described with reference to fig. 1. Fig. 1 schematically illustrates a block diagram 100 of a user interface in which an application according to an exemplary implementation of the present disclosure may be used. Applications such as online vehicle services have been developed. As shown in the user interface 110 of FIG. 1, block 120 shows the current time, the user may specify a departure location for the vehicle at block 130 and a destination for the vehicle at block 132. The user may press a button 140 to preview information related to a trip from a departure point to a destination. The user may then decide whether to continue or cancel the call. It will be appreciated that since scheduling vehicle resources requires some time, in order to increase response speed and reduce waiting time for a user, it is desirable that the user's behavior can be predicted in order to more efficiently schedule vehicles.
Technical solutions for predicting user behavior based on a cancel action of a user in the past have been proposed. The technical scheme judges the probability of canceling the current trip of the user only based on the past canceling action of the user, but cannot accurately predict the next action of the user.
To address, at least in part, one or more of the above issues and other potential issues, an exemplary implementation of the present disclosure proposes a technical solution to predict user behavior. In particular, it is proposed to build a user action graph for determining actions of a user in the future based on a plurality of historical actions of a plurality of types performed by the user. Compared with the technical scheme of performing prediction based on a single type of action in the prior art, multiple historical actions of multiple types can reflect multiple factors when the user makes the next action, thereby being helpful for performing prediction in a more accurate manner.
An overview of an exemplary implementation of the present disclosure is first described with reference to fig. 2, which schematically illustrates a block diagram of a process for determining user behavior based on multiple types of historical actions, according to an exemplary implementation of the present disclosure. The actions performed by the user may be detected, for example, upon determining that the user performed the first action 210, a plurality of historical actions 220 of a plurality of types performed by the user may be collected to generate a user action graph 230. Here, the user action graph 230 may describe the order in which a plurality of historical actions were performed by the user. Based on the user action graph 230, temporal dependencies 240 between the plurality of types may be obtained to determine a second action 250 to be performed by the user based on the first action 210 and the temporal dependencies 240.
In the following, further details of an exemplary implementation of the present disclosure will be described with reference to fig. 3. Fig. 3 schematically illustrates a flow chart of a method 300 for determining user behavior according to an exemplary implementation of the present disclosure. At block 310, a user action graph 230 is generated based on a plurality of historical actions 220 of a plurality of types performed by a user in response to a first action 210 performed by the user at a first point in time. More details of exemplary implementations of the present disclosure will be described below in the context of an online vehicle service application.
The user actions may include a plurality of types, including, for example, at least one of: preview type, commit type, cancel type, and complete type. An act of the preview type representing a user specifying a start and destination for the trip, in order to cause the online vehicle service application to display information related to the trip from the start to the destination; the submitting type represents the action of submitting a taxi calling request after the user views the travel related information; the cancellation type represents an action of canceling the vehicle request; and an act of completing the action indicating that the user's request to call the vehicle has been responded to and that the vehicle has arrived at the user-specified destination on demand. It will be appreciated that the present disclosure is directed to predicting a user's actions after a preview action, and thus the first action 210 herein represents a preview type action.
According to an exemplary implementation of the present disclosure, if a user performs a preview action at 13:00:00 (abbreviated as a first time point) on day 1, 20 of 2020, multiple types of historical actions 220 performed by the user before the first time point may be obtained. The number of the plurality of historical actions may be specified in advance. If the user's number of historical actions is insufficient, all of the existing historical actions may be selected. Alternatively and/or additionally, obtaining a plurality of historical actions within a predetermined time range may be specified in advance.
Where multiple historical actions 220 have been acquired, a user action graph 230 may be generated based on the multiple historical actions 220. Referring to FIG. 4, a block diagram 400 of a process for processing a plurality of historical actions 220 is schematically illustrated, according to an exemplary implementation of the present disclosure. As shown, in the plurality of historical actions 220, a first column indicates a sequence number of the action, a second column indicates a time stamp of the action, and a third column indicates an action type. It will be appreciated that the taxi calling process is triggered by a preview action and thus the sequence of actions triggered by a preview type preview action may be determined in a plurality of historical actions.
As shown in FIG. 4, action 1 in plurality of actions 220 is a preview action, and a commit action and a complete action are triggered after the action, respectively. At this time, the action sequence (preview-submit-complete) including the 1 st to 3 rd actions indicates that the user submits a request for a vehicle call after the preview, and the vehicle arrives at the destination from the departure place according to the user's request. As another example, the 4 th action is a preview action, and a cancel action is triggered after the action. The action sequence (preview-cancel) including the 4 th to 5 th actions at this time indicates that the user cancels the request for calling the car after the preview. Similarly, one or more other sequences of actions may be obtained from the plurality of actions 220.
According to an exemplary implementation of the present disclosure, the user action graph 230 describes the order in which the plurality of historical actions 220 are performed by the user, at which point each sequence of actions may be processed one by one to gradually generate a user action graph. In particular, two nodes in the user action graph 230 may be generated based on the type of two actions in the action sequence, respectively. The 1 st through 3 rd actions of the plurality of historical actions 220 may be processed first, at which point a preview node 420 (represented by an ellipse) may be generated based on the 1 st action, a commit node 422 (represented by a rectangle) may be generated based on the 2 nd action, and a complete node 424 may be generated based on the 3 rd action. Further, an edge between two nodes may be generated based on the order in which the two actions were performed by the user. Since the actions of 1 st through 3 rd are performed in order, the direction of the edges between nodes 420, 422, and 424 at this time will be as shown in fig. 4.
Similarly, other sequences of actions may be processed one by one. For a sequence of actions including the 4 th to 5 th actions, a preview node 430 (represented by an ellipse) may be generated based on the 4 th action, a cancel node 432 (represented by a pentagon) may be generated based on the 5 th action, and the chronological order in which the two nodes 430 and 432 were performed by the user may be obtained to generate an edge between the nodes 430 and 432. Other sequences of actions may be processed in a similar manner to form a complete user action graph 230.
Where the user action graph 230 has been generated, the temporal dependencies 240 may be further obtained. Returning to FIG. 3, at block 320, based on the user action graph 230, temporal dependencies 240 between the plurality of types are obtained. According to example implementations of the present disclosure, the temporal dependencies 240 may be determined based on machine learning techniques or other techniques. With continued reference to fig. 4, the embedding (embedding) of the user action Graph 230 may be determined as a timing feature 450, for example, based on a Graph Neural Network (Graph Neural Network)440, and the temporal dependencies 240 may be determined based on the timing feature 450.
The Graph neural Network is a neural Network learning technology that has emerged in recent years, and mainly includes a Graph attention Network (GAT), a Graph Convolution Network (GCN), and Graph sampling and aggregation (Graph Sample and aggregation, Graph sage), and the like. The graph neural network technology learns and spreads node information and graph structure information in a graph through a plurality of layers of graph operators, and can effectively capture rich information of graph nodes and dependency relations among the nodes. According to an example implementation of the present disclosure, the temporal order between the various nodes may be captured from the user action graph 230, for example, based on GAT techniques, to form the timing feature 450.
According to an example implementation of the present disclosure, the timing characteristics 450 may be determined based on a dynamic graph embedding method. An action sequence in the user action graph 230 that includes multiple actions of different types may be considered a heterogeneous session. According to example implementations of the present disclosure, GAT operations within a heterogeneous session may be performed to obtain corresponding embeddings. For example, in some implementations, this may be performed based on equations 1 and 2 below.
Figure BDA0002425649510000071
Figure BDA0002425649510000072
Wherein
Figure BDA0002425649510000073
Represents a series connection, OVRepresenting different types of events in a user action graph (corresponding toActions performed by the user), K represents the number of heads involved in the multi-head attention technique,
Figure BDA0002425649510000074
representing a hidden state of the heterogeneous session after performing the GAT operation,
Figure BDA0002425649510000075
indicating the initial state of the heterogeneous session,
Figure BDA0002425649510000076
representing the attention weight of the kth head calculated by the attention mechanism,
Figure BDA0002425649510000077
and
Figure BDA0002425649510000078
the correlation transfer matrices for heterogeneous sessions and actions i, respectively.
It will be appreciated that the above only schematically illustrates an example of the formula used in the GAT operation. The specific formulation of the GAT operation is not limited in the context of this disclosure, but may be implemented based on a variety of techniques that have been currently developed and/or that will be developed in the future.
Further, the temporal dependencies 240 may be trained based on the timing characteristics 450, such that the trained temporal dependencies 240 may describe a temporal dependency between a preview type of action performed by the user and another type of action subsequently performed. Hereinafter, description will be made with reference to fig. 5. FIG. 5 schematically illustrates a block diagram 500 of an operational procedure for generating temporal dependencies 240 according to an exemplary implementation of the present disclosure. A model of temporal dependencies 240 may be constructed based on machine learning techniques, which may include, for example, Gated-loop units (GRUs) 512, full connections 514, and Sigmoid functions 516. The specific structure of the machine learning model is not limited in the context of this disclosure, and the model may be constructed based on a variety of techniques that have been developed currently and/or will be developed in the future.
The time dependency 240 may be trained using the timing characteristics 450 obtained as described above, i.e., here the timing characteristics 450 will be used as training samples. Since the training sample includes the execution order between the types of actions, the temporal dependency 240 obtained based on the training sample can accurately reflect the temporal order between the types. At this time, after a certain action is input to the time dependency 240, another action that may be performed after the action may be obtained.
Returning to FIG. 3, at block 330, based on the first action 210 and the temporal dependency 240, a second action 250 is determined that the user will perform at a second point in time after the first point in time. It will be appreciated that because the temporal dependencies 240 are obtained based on the historical behavior of the user, this allows the temporal dependencies 240 to accurately understand the behavior habits of the user over a period of time in the past. In this way, the predicted second action 250 can be made to conform to the past behavior habits of the user as much as possible, thereby improving the accuracy of the prediction. According to an example implementation of the present disclosure, a probability that the second action 250 is performed after the first action 210 may also be output. For example, one may output: the probability of an action of the raise type being performed is 80%, the probability of an action of the cancel type being performed is 20%, and so on.
The specific steps of determining the temporal dependencies 240 and making predictions based on historical actions from the user have been described above with reference to fig. 2-5. According to example implementations of the present disclosure, the temporal dependencies 240 may also be determined based on actions from other users. It will be appreciated that the preview action may specify a start and destination for the trip, and that the user calling for himself is currently in the vicinity of the start, at which point information may also be considered for other users who are calling for cars in the vicinity of the start and/or other users who have called for cars and have arrived at (or are about to arrive at) the start. With the exemplary implementation of the present disclosure, by using various types of user's actions, it is possible to more accurately reflect changes in vehicle resources near the departure place, thereby predicting the potential behavior of the user in a more accurate manner.
According to an exemplary implementation of the present disclosure, relevant actions of other users near the origin may be obtained. A time range (e.g., 10 minutes) and a location range (e.g., 100 meters) may be specified in advance, and a user may be selected based on the time range and the location range. Continuing with the above example, the user performs the preview action at the departure location "train station" at 13:00:00 on day 1, 20 of 2020, at which point other users (referred to as a group of departure users) may be selected that depart from within 100 meters of the vicinity of the train station, between 12:50:00 and 13:00: 00.
Further, a set of departure action sequences of action sequences performed by a set of departure users may be obtained, as will be described in more detail below with reference to fig. 6. FIG. 6 schematically shows a block diagram 600 of a process for collecting data from other users located around a user-specified origin, according to an exemplary implementation of the present disclosure. As shown in FIG. 6, assuming a group of departing users includes user X1, user X2, etc., then the action sequence 610 for user X1 and the action sequence 620 for user X2, respectively, may be obtained, and so on. Further, the departure user action graph 630 may be generated based on the method described above with reference to fig. 4. The number of users desired to be selected may be predefined, for example, 10 users or other numbers of users may be selected. If the number of originating users near the origin is less than the predetermined number, only data related to existing users near the origin may be selected.
It will be appreciated that the more car users at the origin, the more stressful the vehicles near the origin will be. By taking into account the behavior of the departure user in the vicinity of the departure place, the time dependency can be generated in a more realistic manner based on the vehicle scheduling conditions in the vicinity of the departure place, thereby providing a more accurate prediction.
In accordance with an example implementation of the present disclosure, an aggregated action graph may be generated based on the user action graph 230 and the departure user action graph 630. For example, a connected action graph may be generated in the following manner.
Figure BDA0002425649510000091
Wherein
Figure BDA0002425649510000092
Shows a linked kinetic diagram, HetGp,TRepresenting user actions 230, and HetGo,TA departure user action diagram 630 is shown. The sign "+" indicates a join operation. Connection operation G ═ G1+G2Showing a connection diagram G1(shown as G)1=(V1,E1) And graph G)2(shown as G)2=(V2,E2) Where V represents a node and E represents an edge in the respective graph.
In the generated graph G, the nodes include a set V1∪V2And the edge comprises a set E1∪E2Of (2). The connected action graph may be generated based on the definition of the connection operation described above. Further, the corresponding embedding may be determined by applying the GAT technique to the action graph of the connection in a manner as described in fig. 4, thereby obtaining the temporal dependency 240.
According to an example implementation of the present disclosure, information of other users who have called a car and whose vehicles have arrived (or are about to arrive) at a departure place may be acquired. A time range (e.g., 10 minutes) and a location range (e.g., 100 meters) may be specified in advance, and a user may be selected based on the time range and the location range. Continuing with the example above, the user performs the preview action at 13:00:00 on day 1/20 of 2020 and at the origin "train station," at which point it may be determined that the user (referred to as a group of arriving users) has arrived or is about to arrive within 100 meters of the vicinity of the train station between 12:50:00 and 13:00: 00.
It will be appreciated that the time ranges for the departing user and the arriving user may have the same or different values. According to an example implementation of the present disclosure, the time range for an arriving vehicle may be less than the time range for a departing vehicle, as the arriving vehicle may receive the next order and leave immediately. Further, the location ranges for the departing user and the arriving user may have the same or different values. According to an exemplary implementation of the present disclosure, the destination and the origin of two orders of a vehicle in tandem are typically at a distance, and the driver of the vehicle does not mind to travel a distance to reach the origin of the next order. At this time, the position range for the arriving vehicle may be larger than the position range for the departing vehicle.
Having determined a set of arriving users, a set of arriving action sequences of action sequences performed by the set of arriving users may be obtained, as will be described in more detail below with reference to FIG. 7. Fig. 7 schematically illustrates a block diagram of a process of collecting data from other users destined to a user-specified origin according to an exemplary implementation of the present disclosure. As shown in fig. 7, assuming that a group of arriving users includes user Y1, user Y2, and the like, the action sequence 710 of user Y1 and the action sequence 720 of user Y2 may be acquired, respectively. Further, the nodes 712, 714, 716, 722, and 724 and edges in the reach user action graph 730 may be generated based on the method described above with reference to FIG. 4. The number of users desired to be selected may be predefined, for example, 10 users or other numbers of users may be selected. If the number of arriving users near the origin is less than the predetermined number, only data related to existing arriving users may be selected.
It will be appreciated that the more vehicles that arrive at or are about to arrive at the origin, the more abundant will be the vehicle resources near the origin. By taking into account the behavior of the arriving user near the departure location, the time dependency can be generated in a more realistic manner based on the vehicle scheduling conditions near the departure location, thereby providing a more accurate prediction.
According to an example implementation of the present disclosure, an aggregated action graph may be generated based on the user action graph 230 and the arrival user action graph 730. For example, the connected action graph may be generated in the following manner.
Figure BDA0002425649510000101
Wherein,
Figure BDA0002425649510000111
shows a linked kinetic diagram, HetGp,TRepresenting user actions 230, and HetGd,TThe representation arrives at the user action graph 730. The corresponding embedding may be determined based on the connected action graph in the manner described above.
According to example implementations of the present disclosure, the temporal dependencies 240 may also be determined based on a user, a set of departing users, and a set of actions of arriving users. FIG. 8 schematically shows a block diagram of a process for training temporal dependencies 240 according to an exemplary implementation of the present disclosure. As shown in fig. 8, a connected action graph 810 may be generated based on the user action graph 230, the departure user action graph 630, and the arrival user action graph 730. For example, in some implementations, a connected action graph may be generated by:
Figure BDA0002425649510000112
wherein,
Figure BDA0002425649510000113
shows a linked kinetic diagram, HetGp,TRepresenting user actions graph 230, HetGo,TPresentation departure user action graph 630 and HetGd,TThe representation arrives at the user action graph 730. According to an example implementation of the present disclosure, GAT operations across heterogeneous sessions may be performed, for example, by:
Figure BDA0002425649510000114
wherein N isp、NoAnd NdRepresenting the number of timestamps in user action graph 230, departure user action graph 630 and arrival user action graph 730 respectively,
Figure BDA0002425649510000115
and
Figure BDA0002425649510000116
respectively, represent the states after GAT operations within the heterogeneous session at time T-T in the three action diagrams described above. The mechanism of interest for each action graph can be determined based on the GAT technique
Figure BDA0002425649510000117
And
Figure BDA0002425649510000118
the specific formula of (c).
Further, GAT operations may be performed across multiple action graphs. For example, the GAT operation across the user action graph 230, the departure user action graph 630, and the arrival user action graph 730 may be performed based on equation 7 below.
Figure BDA0002425649510000121
Wherein,
Figure BDA0002425649510000122
representing final event embedding for a heterogeneous session at time T, and associated with a preview action performed at a first point in time
Figure BDA0002425649510000123
Is updated to
Figure BDA0002425649510000124
Determining a GAT for computing global attention based on GAT techniquesgA specific formula of the attention mechanism of (1).
With the exemplary implementations of the present disclosure, the user's own behavior habits, the number of vehicles about to depart from the vicinity of the departure location, the vehicles arriving at or about to arrive at the destination, and the behavior habits of the relevant users may be fully considered to determine the time dependencies 240 in a more comprehensive and accurate manner. Further, the prediction performed based on the temporal dependency 240 can be made more accurate.
In the above it has been described how to determine the actions to be performed by the user. Further, at a second point in time after the first point in time, one or more actions actually performed by the user may be collected. The actions, as well as one or more actions subsequently performed, may be previewed to update a set of historical sequences of actions. With an exemplary implementation of the present disclosure, content in a set of historical action sequences may be scrolled and older action sequences are continually updated with the most recent action sequence.
Specifically, the plurality of action sequences may be ordered in chronological order, and a maximum number of action sequences may be specified (e.g., a maximum number of 30). When the number of historical action sequences is less than 30, the newly acquired action sequence can be added to the existing plurality of historical action sequences continuously. When the number of historical action sequences equals 30, a newly acquired action sequence may be added and the oldest action sequence removed. In this way, the temporal dependencies may be generated based on recent data so that the temporal dependencies may better reflect the user's most recent behavior habits.
According to an example implementation of the present disclosure, a preview interface may be provided to the user based on the determined second action. Hereinafter, more details will be described with reference to fig. 9A and 9B. Fig. 9A schematically illustrates a block diagram 900A of a preview interface displayed to a user according to an exemplary implementation of the present disclosure. At the point in time shown at block 910A, preview information 920A may be displayed to the user based on the predicted result. If the user is about to perform a cancellation action based on the prediction obtained from the temporal dependencies 240, the user may be given a higher discount (e.g., 85 discounts) to prompt the user to submit an order. Alternatively and/or additionally, a higher priority may also be set for the user to provide vehicle resources to the user. The user can press the submit button 930A or the cancel button 932A based on the preview information 920A and the judgment of himself/herself.
Fig. 9B schematically illustrates a block diagram 900B of a preview interface displayed to a user according to an exemplary implementation of the present disclosure. At the point in time shown at block 910B, preview information 920B may be displayed to the user based on the predicted result. If the user is about to perform a commit action based on the prediction results obtained from the temporal dependencies 240, the user may be given a general discount (e.g., a 95-fold discount). Alternatively and/or additionally, a general priority may also be set for the user. The user can press the submit button 930A or the cancel button 932A based on the preview information 920A and the judgment of himself/herself.
By means of the method and the device for providing the preview information, different preview information can be provided for the user based on different prediction results, and different requirements of different users can be met. Further, vehicle resources associated with the predicted action may also be scheduled in advance. Assuming that the probability of the user submitting an order is predicted to be 90%, the vehicle can be scheduled in advance to the vicinity of the departure place. In this way, the allocation speed of vehicle resources can be further increased, and the user waiting time can be reduced.
Examples of the method according to the present disclosure have been described in detail above with reference to fig. 2 to 9B, and implementations of the respective apparatuses will be described below. According to an example implementation of the present disclosure, there is provided an apparatus for predicting user behavior, comprising: a generation module configured to generate a user action graph based on a plurality of historical actions of a plurality of types performed by a user in response to a first action performed by the user at a first point in time, the user action graph describing an order in which the plurality of historical actions were performed by the user; an acquisition module configured to acquire a time dependency relationship between a plurality of types based on a user action graph; and a determination module configured to determine, based on the first action and the temporal dependency, a second action to be performed by the user at a second point in time after the first point in time.
According to an example implementation of the present disclosure, the type of the plurality of types includes at least one of: a preview type, a submit type, a cancel type, and a finish type, and the first action includes a preview type action for displaying information of a trip of the vehicle from the departure point to the destination.
According to an exemplary implementation of the present disclosure, the generating module includes: a sequence determination module configured to determine, among a plurality of historical actions, a sequence of actions triggered by a preview action of a preview type; and a graph generation module configured to generate a portion of the user action graph based on the sequence of actions.
According to an exemplary implementation of the present disclosure, a graph generation module includes: the node generation module is configured to generate two nodes in the user action graph based on the types of the two actions in the action sequence respectively; and an edge generation module configured to generate an edge between the two nodes based on an order in which the two actions are performed by the user.
According to an exemplary implementation of the present disclosure, the obtaining module includes: the extraction module is configured for extracting the time sequence characteristics of the user action diagram; and a training module configured to train the temporal dependency based on the timing characteristics such that the temporal dependency describes a temporal dependency between a preview type of action performed by the user and another type of action subsequently performed.
According to an exemplary implementation of the present disclosure, the apparatus further comprises: a starting user determination module configured to determine a set of starting users starting from around a starting place within a predetermined time range around a first time point; a departure action determination module configured to obtain a set of departure action sequences performed by a set of departure users; the generation module is further configured to generate a departure user action graph based on the set of departure action sequences; and the obtaining module is further configured to obtain the time dependency relationship based on the user action graph and the departure user action graph.
According to an exemplary implementation of the present disclosure, the obtaining module includes: a connection module configured to connect the user action graph and the departure user action graph to generate a connected action graph; and the training module is further configured to train the temporal dependencies based on the connected action graph.
According to an exemplary implementation of the present disclosure, the apparatus further comprises: an arrival user determination module configured to determine a group of arrival users who arrive around the departure place within a predetermined time range around a first time point; an arrival action determination module configured to obtain a set of arrival action sequences performed by a set of arrival users; the generation module is further configured to generate an arrival user action graph based on the set of arrival action sequences; and the obtaining module is further configured to obtain the time dependency relationship based on the user action graph and the arrival user action graph.
According to an exemplary implementation of the present disclosure, the apparatus further comprises: and the providing module is configured to provide a preview interface for the user based on the second action.
According to an exemplary implementation of the present disclosure, the apparatus further comprises: the action acquisition module is configured to acquire at least one action executed by the user at a second time point and after the second time point; and an update module configured to update a set of historical motion sequences based on the first motion and the at least one motion.
According to an exemplary implementation of the present disclosure, there is provided an electronic device including: a memory and a processor; wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described above.
Fig. 10 illustrates a block diagram of a computing device/server 1000 in which one or more embodiments of the disclosure may be implemented. It should be understood that the computing device/server 1000 illustrated in fig. 10 is merely exemplary and should not constitute any limitation as to the functionality and scope of the embodiments described herein.
As shown in fig. 10, computing device/server 1000 is in the form of a general purpose computing device. The components of computing device/server 1000 may include, but are not limited to, one or more processors or processing units 1010, memory 1020, storage 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. The processing unit 1010 may be a real or virtual processor and can perform various processes according to programs stored in the memory 1020. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device/server 1000.
Computing device/server 1000 typically includes a number of computer storage media. Such media may be any available media that is accessible by computing device/server 1000 and includes, but is not limited to, volatile and non-volatile media, removable and non-removable media. Memory 1020 may be volatile memory (e.g., registers, cache, Random Access Memory (RAM)), non-volatile memory (e.g., Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory), or some combination thereof. Storage 1030 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a magnetic disk, or any other medium, which may be capable of being used to store information and/or data (e.g., training data for training) and which may be accessed within computing device/server 1000.
Computing device/server 1000 may further include additional removable/non-removable, volatile/nonvolatile storage media. Although not shown in FIG. 10, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 1020 may include a computer program product 1025 having one or more program modules configured to perform the various methods or acts of the various embodiments of the present disclosure.
The communication unit 1040 enables communication with other computing devices over a communication medium. Additionally, the functionality of the components of computing device/server 1000 may be implemented in a single computing cluster or multiple computing machines capable of communicating over a communications connection. Thus, computing device/server 1000 may operate in a networked environment using logical connections to one or more other servers, network Personal Computers (PCs), or another network node.
Input device 1050 may be one or more input devices such as a mouse, keyboard, trackball, or the like. Output device 1060 may be one or more output devices such as a display, speakers, printer, or the like. Computing device/server 1000 may also communicate with one or more external devices (not shown), such as storage devices, display devices, etc., as desired through communication unit 1040, with one or more devices that enable a user to interact with computing device/server 1000, or with any device (e.g., network card, modem, etc.) that enables computing device/server 1000 to communicate with one or more other computing devices. Such communication may be performed via input/output (I/O) interfaces (not shown).
According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which one or more computer instructions are stored, wherein the one or more computer instructions are executed by a processor to implement the above-described method.
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products implemented in accordance with the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processing unit of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various implementations of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The foregoing has described implementations of the present disclosure, and the above description is illustrative, not exhaustive, and not limited to the implementations disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The terminology used herein was chosen in order to best explain the principles of implementations, the practical application, or improvements to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the implementations disclosed herein.

Claims (22)

1. A method for predicting user behavior, comprising:
generating a user action graph based on a plurality of historical actions of a plurality of types performed by the user in response to a first action performed by the user at a first point in time, the user action graph describing an order in which the plurality of historical actions were performed by the user;
acquiring time dependency relationships among the multiple types based on the user action graph; and
determining, based on the first action and the temporal dependency, a second action to be performed by the user at a second point in time after the first point in time.
2. The method of claim 1, wherein the plurality of types comprise at least one of: a preview type, a submit type, a cancel type, and a finish type, the first action including an action of the preview type, and the action of the preview type being for displaying information of a trip of a vehicle from a departure place to a destination.
3. The method of claim 2, wherein generating the user action graph based on the plurality of historical actions of the plurality of types performed by the user comprises:
determining, among the plurality of historical actions, a sequence of actions triggered by the preview-type preview action; and
based on the sequence of actions, a portion of the user action graph is generated.
4. The method of claim 3, wherein generating the portion of the user action graph based on the sequence of actions comprises:
generating two nodes in the user action graph based on the types of the two actions in the action sequence respectively; and
generating an edge between the two nodes based on an order in which the two actions are performed by the user.
5. The method of claim 3, wherein obtaining the temporal dependencies between the plurality of types based on the user action graph comprises:
extracting time sequence characteristics of the user action diagram; and
training the temporal dependency based on the timing features such that the temporal dependency describes a temporal dependency between the preview type of action performed by the user and another type of action subsequently performed.
6. The method of claim 5, further comprising:
determining a set of departure users departing from around the departure location within a predetermined time range around the first time point;
obtaining a set of departure action sequences performed by the set of departure users;
generating a starting user action graph based on the set of starting action sequences; and
and acquiring the time dependence relationship based on the user action diagram and the starting user action diagram.
7. The method of claim 6, wherein obtaining the temporal dependency based on the user action graph and the departure user action graph comprises:
connecting the user action graph and the departure user action graph to generate a connected action graph; and
training the temporal dependencies based on the connected action graph.
8. The method of claim 5, further comprising:
determining a set of arriving users that arrive around the origin within a predetermined time range around the first time point;
obtaining a set of arrival action sequences performed by the set of arrival users;
generating an arrival user action graph based on the set of arrival action sequences; and
and acquiring the time dependency relationship based on the user action diagram and the arrival user action diagram.
9. The method of claim 1, further comprising: providing a preview interface to the user based on the second action.
10. The method of claim 1, further comprising:
acquiring at least one action executed by the user at the second time point and after the second time point; and
updating the set of historical sequences of actions based on the first action and the at least one action.
11. An apparatus for predicting user behavior, comprising:
a generation module configured to generate a user action graph based on a plurality of historical actions of a plurality of types performed by the user in response to a first action performed by the user at a first point in time, the user action graph describing an order in which the plurality of historical actions were performed by the user;
an obtaining module configured to obtain a time dependency relationship between the plurality of types based on the user action graph; and
a determination module configured to determine, based on the first action and the temporal dependency, a second action to be performed by the user at a second point in time after the first point in time.
12. The apparatus of claim 11, wherein the plurality of types includes at least one of: a preview type, a submit type, a cancel type, and a finish type, and the first action includes an action of the preview type for displaying information of a trip of a vehicle from a departure place to a destination.
13. The apparatus of claim 12, wherein the generating means comprises:
a sequence determination module configured to determine, among the plurality of historical actions, a sequence of actions triggered by the preview action of the preview type; and
a graph generation module configured to generate a portion of the user action graph based on the sequence of actions.
14. The apparatus of claim 13, wherein the graph generation module comprises:
a node generation module configured to generate two nodes in the user action graph based on types of two actions in the action sequence, respectively; and
an edge generation module configured to generate an edge between the two nodes based on an order in which the two actions are performed by the user.
15. The apparatus of claim 13, wherein the acquisition module comprises:
the extraction module is configured to extract the time sequence characteristics of the user action diagram; and
a training module configured to train the temporal dependency based on the timing characteristics such that the temporal dependency describes a temporal dependency between the preview type of action performed by the user and another type of action subsequently performed.
16. The apparatus of claim 15, further comprising:
a departure user determination module configured to determine a set of departure users departing from the vicinity of the departure location within a predetermined time range around the first time point;
a set of departure actions determination module configured to obtain a set of departure action sequences performed by the set of departure users;
the generation module is further configured to generate a departure user action graph based on the set of departure action sequences; and
the obtaining module is further configured to obtain the temporal dependency based on the user action graph and the departure user action graph.
17. The apparatus of claim 16, wherein the acquisition module comprises:
a connection module configured to connect the user action graph and the departure user action graph to generate a connected action graph; and
the training module is further configured to train the temporal dependencies based on the connected action graph.
18. The apparatus of claim 15, further comprising:
an arrival user determination module configured to determine a group of arrival users who arrive around the departure place within a predetermined time range around the first time point;
an arrival action determination module configured to obtain a set of arrival action sequences performed by the set of arrival users;
the generation module is further configured to generate an arrival user action graph based on the set of arrival action sequences; and
the obtaining module is further configured to obtain the temporal dependency based on the user action graph and the arrival user action graph.
19. The apparatus of claim 11, further comprising: a providing module configured to provide a preview interface to the user based on the second action.
20. The apparatus of claim 11, further comprising:
an action obtaining module configured to obtain at least one action performed by the user at the second time point and after the second time point; and
an update module configured to update the set of historical sequence of actions based on the first action and the at least one action.
21. An electronic device, comprising:
a memory and a processor;
wherein the memory is to store one or more computer instructions, wherein the one or more computer instructions are to be executed by the processor to implement the method of any one of claims 1 to 10.
22. A computer readable storage medium having one or more computer instructions stored thereon, wherein the one or more computer instructions are executed by a processor to implement the method of any one of claims 1 to 10.
CN202010219740.XA 2020-03-25 2020-03-25 Method, apparatus, electronic device and storage medium for predicting user behavior Pending CN111813640A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010219740.XA CN111813640A (en) 2020-03-25 2020-03-25 Method, apparatus, electronic device and storage medium for predicting user behavior

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010219740.XA CN111813640A (en) 2020-03-25 2020-03-25 Method, apparatus, electronic device and storage medium for predicting user behavior

Publications (1)

Publication Number Publication Date
CN111813640A true CN111813640A (en) 2020-10-23

Family

ID=72848145

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010219740.XA Pending CN111813640A (en) 2020-03-25 2020-03-25 Method, apparatus, electronic device and storage medium for predicting user behavior

Country Status (1)

Country Link
CN (1) CN111813640A (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105893532A (en) * 2016-03-30 2016-08-24 北京金山安全软件有限公司 Consultation information pushing method and device and terminal equipment
US20180209808A1 (en) * 2017-01-10 2018-07-26 Beijing Didi Infinity Technology And Development Co., Ltd. Method and system for estimating time of arrival
CN108446374A (en) * 2018-03-16 2018-08-24 北京三快在线科技有限公司 User view prediction technique, device, electronic equipment, storage medium
CN108875983A (en) * 2017-05-15 2018-11-23 北京嘀嘀无限科技发展有限公司 Intelligence about vehicle method and apparatus
CN109818839A (en) * 2019-02-03 2019-05-28 三星电子(中国)研发中心 Personalized behavior prediction methods, devices and systems applied to smart home
CN110119467A (en) * 2019-05-14 2019-08-13 苏州大学 A kind of dialogue-based item recommendation method, device, equipment and storage medium
CN110647929A (en) * 2019-09-19 2020-01-03 京东城市(北京)数字科技有限公司 Method for predicting travel destination and method for training classifier

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105893532A (en) * 2016-03-30 2016-08-24 北京金山安全软件有限公司 Consultation information pushing method and device and terminal equipment
US20180209808A1 (en) * 2017-01-10 2018-07-26 Beijing Didi Infinity Technology And Development Co., Ltd. Method and system for estimating time of arrival
CN108875983A (en) * 2017-05-15 2018-11-23 北京嘀嘀无限科技发展有限公司 Intelligence about vehicle method and apparatus
CN108446374A (en) * 2018-03-16 2018-08-24 北京三快在线科技有限公司 User view prediction technique, device, electronic equipment, storage medium
CN109818839A (en) * 2019-02-03 2019-05-28 三星电子(中国)研发中心 Personalized behavior prediction methods, devices and systems applied to smart home
CN110119467A (en) * 2019-05-14 2019-08-13 苏州大学 A kind of dialogue-based item recommendation method, device, equipment and storage medium
CN110647929A (en) * 2019-09-19 2020-01-03 京东城市(北京)数字科技有限公司 Method for predicting travel destination and method for training classifier

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
GREY, M.X. ET AL.: "Traversing Environments using Possibility Graphs with Multiple Action Types", ARXIV, 1 October 2016 (2016-10-01) *
黎景壮;温惠英;林龙;漆巍巍;: "基于QPSO__RBF神经网络的网约车需求量预测模型", 广西大学学报(自然科学版), no. 02, 25 April 2018 (2018-04-25) *

Similar Documents

Publication Publication Date Title
US10599449B1 (en) Predictive action modeling to streamline user interface
JP6821447B2 (en) Smoothing dynamic modeling of user travel preferences in public transport systems
JP6911603B2 (en) How to generate predictive models for the categories of facilities visited by users, programs, server equipment, and processing equipment
JP2011086295A (en) Estimating service resource consumption based on response time
WO2019047595A1 (en) Evaluation method and device for comfort level of end-to-end-based automatic driving system
CN106803799B (en) Performance test method and device
CN109146280A (en) A kind of method, apparatus and system of pushed information
WO2020077871A1 (en) Event prediction method and apparatus based on big data, computer device, and storage medium
CN114663198A (en) Product recommendation method, device and equipment based on user portrait and storage medium
CN110209467A (en) A kind of flexible resource extended method and system based on machine learning
CN112949973A (en) AI-combined robot process automation RPA process generation method
Rahaman et al. Predicting Imbalanced Taxi and Passenger Queue Contexts in Airport.
CN111210315B (en) Travel order processing method and device, electronic equipment and readable storage medium
CN110471945A (en) Processing method, system, computer equipment and the storage medium of alive data
CN111160638B (en) Conversion estimation method and device
US20220327650A1 (en) Transportation bubbling at a ride-hailing platform and machine learning
CN112561371B (en) Scheduling determination method and device, electronic equipment and storage medium
CN112199715B (en) Object generation method based on block chain and cloud computing and digital financial service center
CN111813640A (en) Method, apparatus, electronic device and storage medium for predicting user behavior
CN112749899A (en) Order dispatching method, device and storage medium
CN112288115B (en) Method, device, equipment and storage medium for generating reservation service information
CN114692014A (en) Method, apparatus, device, medium and program product for determining candidate get-off location
CN113129102B (en) Delayed order dispatching method and device, electronic equipment and storage medium
CN113128597B (en) Method and device for extracting user behavior characteristics and classifying and predicting user behavior characteristics
CN114896482A (en) Model training and energy supplementing intention recognition method, device, equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination