WO2016111068A1 - 情報処理装置、情報処理方法、およびプログラム - Google Patents
情報処理装置、情報処理方法、およびプログラム Download PDFInfo
- Publication number
- WO2016111068A1 WO2016111068A1 PCT/JP2015/079176 JP2015079176W WO2016111068A1 WO 2016111068 A1 WO2016111068 A1 WO 2016111068A1 JP 2015079176 W JP2015079176 W JP 2015079176W WO 2016111068 A1 WO2016111068 A1 WO 2016111068A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- information
- action
- time
- user
- processing apparatus
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/34—Route searching; Route guidance
- G01C21/3453—Special cost functions, i.e. other than distance or default speed limit of road segments
- G01C21/3484—Personalized, e.g. from learned user behaviour or user-defined profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C1/00—Registering, indicating or recording the time of events or elapsed time, e.g. time-recorders for work people
- G07C1/10—Registering, indicating or recording the time of events or elapsed time, e.g. time-recorders for work people together with the recording, indicating or registering of other data, e.g. of signs of identity
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B29/00—Maps; Plans; Charts; Diagrams, e.g. route diagram
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B29/00—Maps; Plans; Charts; Diagrams, e.g. route diagram
- G09B29/10—Map spot or coordinate position indicators; Map reading aids
Definitions
- the present disclosure relates to an information processing apparatus, an information processing method, and a program.
- Patent Literature 1 has a plurality of behavior determination units specialized for specific behavior among user behaviors recognized by threshold processing of sensor data, and the behavior based on the determination result in each behavior determination unit. An information processing apparatus that generates information is described.
- a user's action (action) detected by the technique described in Patent Document 1 is not limited to the generation of information, and can be used in various ways. However, it cannot be said that such a method of utilization has been sufficiently proposed so far.
- the present disclosure proposes a new and improved information processing apparatus, information processing method, and program capable of providing various benefits to the user based on the detection result of the user's action.
- the time information in executing the map representation of the action information based on the spatial information and the information acquisition unit that acquires spatial information and time information associated with the action information of the user, and the time information, the time information
- An information processing apparatus includes a map expression processing unit that takes into account.
- the time information in executing the map expression of the action information based on the spatial information and the function of obtaining the action information of the user and the spatial information and time information associated with the action information, the time information There is provided a program for causing a computer to realize a function that considers the above.
- FIG. 2 is a block diagram illustrating a schematic functional configuration of an information processing apparatus according to an embodiment of the present disclosure.
- FIG. 9 is a flowchart illustrating a first example of a process for detecting a jump included in a user action according to an embodiment of the present disclosure. It is a flowchart which shows the example of the high impact detection process shown in FIG.
- FIG. 3 is a flowchart showing a first example of free fall detection processing shown in FIG. 2.
- FIG. 6 is a flowchart illustrating a process including some examples of a map representation of action information according to an embodiment of the present disclosure. It is a figure showing the 1st example of map representation of action information in one embodiment of this indication. In the example shown in FIG. 16, it is a figure which shows the example in which map expression changed in real time. It is a figure showing the 2nd example of map representation of action information in one embodiment of this indication. It is a figure showing an example of safety map generation in one embodiment of this indication. 4 is a diagram for describing an example of navigation in consideration of a skill level and an action history according to an embodiment of the present disclosure.
- FIG. 5 is a diagram for describing an example of changing an action detection setting according to an embodiment of the present disclosure.
- FIG. 5 is a diagram for describing an example of changing an action detection setting according to an embodiment of the present disclosure.
- FIG. FIG. 3 is a block diagram illustrating a hardware configuration example of an information processing apparatus according to an embodiment of the present disclosure.
- 6 is a flowchart illustrating processing in a first example of device control included in an embodiment of the present disclosure.
- FIG. 6 is a diagram for further describing an image or video captured in a first example of device control included in an embodiment of the present disclosure. It is a figure for demonstrating the 2nd example of the device control included in one Embodiment of this indication. It is a figure for demonstrating the 2nd example of the device control included in one Embodiment of this indication.
- FIG. 9 is a diagram for describing a third example of device control included in an embodiment of the present disclosure.
- FIG. 10 is a diagram for describing a fourth example of device control included in an embodiment of the present disclosure.
- FIG. 9 is a diagram for describing a fifth example of device control included in an embodiment of the present disclosure.
- FIG. 9 is a diagram for describing a fifth example of device control included in an embodiment of the present disclosure.
- FIG. 9 is a diagram for conceptually explaining a sixth example of device control included in an embodiment of the present disclosure.
- 14 is a flowchart illustrating processing in a sixth example of information output included in an embodiment of the present disclosure. It is a figure for demonstrating the user interface for browsing the free viewpoint image
- FIG. 13 is a block diagram illustrating a hardware configuration example of an information processing apparatus according to an embodiment of the disclosure.
- FIG. 1 is a block diagram illustrating a schematic functional configuration of an information processing apparatus according to an embodiment of the present disclosure.
- an information processing apparatus 100 includes a transmission unit 101, a reception unit 102, a sensor device control unit 103, a sensor data analysis unit 104, an analysis result processing unit 107, and a detection section information holding unit 110.
- the additional information holding unit 111 and the service control unit 112 are included.
- the information processing apparatus 100 may be a single device or a collection of devices that configure a server on a network, as shown in some specific examples described later, for example.
- the information processing apparatus 100 may be a terminal device that communicates with a server via a network or a terminal device that operates alone.
- the functions of the information processing apparatus 100 may be realized by being distributed to a server and a terminal device that communicate with each other via a network.
- the hardware configuration of each of the information processing apparatus 100 or a plurality of apparatuses that implement the functions of the information processing apparatus 100 will be described later.
- the transmission unit 101 and the reception unit 102 are realized by a communication device that communicates with the sensor device by various wired or wireless communication methods, for example.
- the sensor device includes at least one sensor mounted on a user or an instrument used by the user.
- the transmission unit 101 transmits a control signal output from the sensor device control unit 103 to the sensor device.
- the receiving unit 102 receives sensor data and time information (time stamp) from the sensor device, and inputs them to the sensor device control unit 103.
- the receiving unit 102 realizes a sensor data receiving unit that receives sensor data provided by a user or a sensor attached to an instrument used by the user.
- the information processing apparatus 100 is a terminal device including at least one sensor, more specifically, a mobile device or a wearable device, a CPU that executes a driver program that receives sensor data from the sensor (
- the sensor data receiving unit may be realized by a processor such as Central Processing Unit.
- the information processing apparatus according to the present embodiment may include an acquisition unit that acquires sensor data from an external apparatus including a sensor, for example.
- the acquisition unit is, for example, a processor such as a CPU that executes “a driver program that receives sensor data from an external device including a sensor via the communication device that implements the transmission unit 101 and the reception unit 102”. It is realized by.
- the information processing apparatus which concerns on this embodiment can also take the structure which is not provided with a sensor data receiving part.
- the sensor device control unit 103 is realized by a processor such as a CPU operating according to a program stored in a memory.
- the sensor device control unit 103 acquires sensor data and time information from the reception unit 102.
- the sensor device control unit 103 provides these data to the sensor data analysis unit 104 and the analysis result processing unit 107.
- the sensor device control unit 103 may perform preprocessing of data as necessary.
- the sensor device control unit 103 outputs a control signal for the sensor device to the transmission unit 101.
- the sensor device control unit 103 may output a control signal based on feedback of a processing result in the sensor data analysis unit 104 or the analysis result processing unit 107.
- the sensor data analysis unit 104 is realized by a processor such as a CPU operating according to a program stored in a memory.
- the sensor data analysis unit 104 performs various analyzes using the sensor data provided from the sensor device control unit 103.
- the sensor data analysis unit 104 includes a feature amount extraction unit 105 and an action detection unit 106.
- the feature amount extraction unit 105 extracts various feature amounts from the sensor data.
- the action detection unit 106 detects a user action based on the feature amount extracted from the sensor data by the feature amount extraction unit 105.
- the user action detected by the action detection unit 106 includes a user turn and / or jump.
- the action detection unit 106 may detect other user actions such as walking, running, standing still, and moving by a vehicle.
- the user's action can be detected in association with time information (time stamp) indicating a section (action section) in which the user action occurred.
- the sensor data analysis unit 104 stores the analysis result, more specifically, for example, information including a user action section detected by the action detection unit 106 in the detection section information holding unit 110. Further, the sensor data analysis unit 104 provides the analysis result to the analysis result processing unit 107.
- the analysis result processing unit 107 is realized by a processor such as a CPU operating according to a program stored in a memory. Based on the analysis result of the sensor data analysis unit 104, more specifically, the information of the user action detected by the action detection unit 106, the analysis result processing unit 107 performs various types used by the service control unit 112 in the subsequent stage. Generate additional information.
- the analysis result processing unit 107 includes a clustering processing unit 108 and a scoring processing unit 109. For example, when the detected user action includes a plurality of actions of the same type, the clustering processing unit 108 may identify these actions as feature amounts (feature amounts extracted by the feature amount extraction unit 105). Or an intermediate feature amount calculated by the action detection unit 106).
- the scoring processing unit 109 may calculate a score indicating action evaluation based on the feature amount. Further, the clustering processing unit 108 and / or the scoring processing unit 109 may newly calculate a feature amount based on the sensor data provided from the sensor device control unit 103.
- the analysis result processing unit 107 holds additional information on the processing result, more specifically, the result of clustering by the clustering processing unit 108 and the score information calculated by the scoring processing unit 109 together with time information (time stamp). Stored in the unit 111.
- the detection section information holding unit 110 and the additional information holding unit 111 are realized by various memories or storage devices, for example.
- the detection section information holding unit 110 and the additional information holding unit 111 temporarily or permanently store the information provided from the sensor data analysis unit 104 and the analysis result processing unit 107 as described above.
- the information stored in the detection section information holding unit 110 and the information stored in the additional information holding unit 111 can be associated with each other by, for example, time information (time stamp). Further, the detection section information holding unit 110 and the additional information holding unit 111 may store information regarding each of a plurality of users.
- the service control unit 112 is realized by a processor such as a CPU operating according to a program stored in a memory.
- the service control unit 112 controls the service 113 using information stored in the detection section information holding unit 110 and / or the additional information holding unit 111. More specifically, for example, the service control unit 112 generates information provided to the user in the service 113 based on the information read from the detection section information holding unit 110 and / or the additional information holding unit 111.
- the information processing apparatus 100 is a server
- the information output by the service control unit 112 can be transmitted to the terminal apparatus via the communication apparatus.
- the information output by the service control unit 112 can be provided to an output device such as a display, a speaker, or a vibrator included in the terminal device.
- a sensor device including an acceleration sensor, an angular velocity sensor, and the like may be directly attached to a user by being embedded in wear, or incorporated in a wearable terminal device or a mobile terminal device.
- the sensor device may be mounted on a snowboard tool, such as a board.
- the action detection process executed in the present embodiment is not limited to jumps and turns that occur on snowboards.
- the action detection process may be executed on jumps and turns that occur in sports other than snowboarding. Since jumps and turns are actions that can occur in common in various sports, it may be possible to detect jumps and turns regardless of the type of sports by, for example, detection processing described below.
- an action other than a jump or turn may be detected.
- various techniques used in the action recognition technique described in, for example, Japanese Patent Application Laid-Open No. 2010-198595 can be applied.
- FIG. 2 is a flowchart illustrating a first example of a process for detecting a jump included in a user action according to an embodiment of the present disclosure. The illustrated process is executed by, for example, the sensor data analysis unit 104 included in the information processing apparatus 100 described above.
- the sensor data analysis unit 104 performs a high impact detection process (S110) and a free fall detection process (S120) for each predetermined time frame. Details of these processes will be described later. Based on the results of these processes, the action detection unit 106 included in the sensor data analysis unit 104 determines whether or not a section between two high impact sections (estimated as crossing and landing) has occurred. (S101). When such a section occurs, the action detection unit 106 determines whether or not the duration of the section is between two threshold values (TH1, TH2) (S102). These threshold values are set, for example, for the purpose of excluding sections that are too long or too short for a jump.
- TH1, TH2 threshold values
- the action detection unit 106 further determines whether or not the ratio of the free fall section in the section exceeds the threshold (TH) (S103). When the ratio of the free fall section exceeds the threshold, it is detected that the section (section sandwiched between two high impact sections) is a jump section (S104).
- FIG. 3 is a flowchart showing an example of the high impact detection process (S110) shown in FIG.
- acceleration D111 included in the sensor data is used.
- the feature quantity extraction unit 105 included in the sensor data analysis unit 104 calculates a norm of acceleration (S112), and further smoothes the norm with an LPF (Low Pass Filter) (S113).
- the feature amount extraction unit 105 calculates the power of the amplitude in a predetermined time frame for the smoothed norm of acceleration (S114).
- the action detection unit 106 determines whether or not the power exceeds the threshold value (TH) (S115), and when the power exceeds the threshold value, detects that the time frame is a high impact section (S116).
- TH threshold value
- S116 detects that the time frame is a high impact section
- FIG. 4 is a flowchart showing a first example of the free fall detection process (S120) shown in FIG.
- acceleration D121
- angular velocity D125
- the feature quantity extraction unit 105 calculates the norm of acceleration (S122), and the action detection unit 106 determines whether or not the norm in each section is below a threshold value (TH) (S123).
- the action detection unit 106 detects that the section is a free fall section with respect to a section in which the norm of acceleration is below the threshold (S124).
- the feature quantity extraction unit 105 calculates the norm for the angular velocity (S126), and further calculates the norm variance in a predetermined time frame (S127).
- the action detection unit 106 determines whether or not the variance of the norm of the angular velocity is lower than the threshold (TH) (S128), and when the variance is lower than the threshold, masks the free fall section detected in S124 (that is, free The determination as a fall section is canceled) (S129).
- TH threshold
- S129 masks the free fall section detected in S124 (that is, free The determination as a fall section is canceled)
- Such a mask process based on angular velocity causes a change in angular velocity when the user jumps, so that the free fall section where the change (dispersion) in angular velocity is small is caused by a cause other than the jump. Based.
- the mask processing in S126 to S129 does not necessarily have to be executed after the free fall section determination processing in S121 to S124.
- the action detection unit 106 may perform the mask process in advance, and may not execute the free fall section determination process for the section specified as the section to be masked.
- the mask process may be executed after the jump section detection process (S104) shown in FIG. 2, and a section once detected as a jump section may be masked.
- the free fall process (S120) shown in FIG. 4 or the like does not necessarily need to be executed before the section occurrence determination (S101) shown in FIG. Before the determination regarding the ratio of the section (S103), the free fall detection process may be executed for the section.
- FIG. 5 is a flowchart showing a second example of the free fall detection process (S120) shown in FIG.
- acceleration D121 included in the sensor data provided by the acceleration sensor mounted on the user or an instrument used by the user is used.
- the feature amount extraction unit 105 and the action detection unit 106 execute the same process as in the first example, and detect a free fall section.
- the feature quantity extraction unit 105 extracts the X-axis component and the Y-axis component of acceleration (S132), and further calculates the covariance between the X-axis component and the Y-axis component of acceleration (S132). S133). More specifically, for example, when the user is walking or running on a reference plane (which is not limited to a horizontal plane but may be an inclined plane), the feature amount extraction unit 105 performs coordinate axes of the acceleration sensor.
- the X axis is the axis closest to the user's direction of travel
- the Y axis is the axis closest to the normal direction of the reference plane
- the covariance of acceleration components (X axis component, Y axis component) in these axis directions is calculated.
- the action detection unit 106 determines whether or not the covariance is lower than the threshold (TH) (S134), and when the covariance is lower than the threshold, masks the free fall section detected in S124 (S129).
- Such mask processing based on the covariance of acceleration is performed when, for example, the jump to be detected is not a so-called vertical jump with a displacement in the normal direction of the reference plane but a jump with a displacement in the user's traveling direction. It is valid.
- FIG. 6 is a flowchart illustrating a second example of a process for detecting a jump included in a user action according to an embodiment of the present disclosure. The illustrated process is executed in the sensor data analysis unit 104 included in the information processing apparatus 100, for example, as in the first example.
- the sensor data analysis unit 104 executes candidate section detection processing (S140). Details of this process will be described later.
- the action detection unit 106 included in the sensor data analysis unit 104 determines whether a candidate section has occurred (S105). When a candidate section occurs, the action detection unit 106 determines whether or not the duration (duration) of the section is between two threshold values (TH1, TH2), as in the first example ( S102). When the duration is between two threshold values, the action detection unit 106 further determines whether or not the average value (mean) of the acceleration in the vertical direction and the horizontal direction in the section exceeds the respective threshold values (THs) ( S106). When the average value of acceleration exceeds each threshold value, it is detected that the candidate section is a jump section (S104).
- FIG. 7 is a flowchart showing an example of the candidate section detection process (S140) shown in FIG.
- the candidate section detection process first, the high impact detection process (S110) described above with reference to FIG. 3, the vertical acceleration calculation process (S141), and the horizontal acceleration calculation process (S142). ) And are executed. Further, the feature amount extraction unit 105 included in the sensor data analysis unit 104 calculates the difference between the vertical acceleration and the horizontal acceleration calculated in S141 and S142 for each section (S143). After that, the action detection unit 106 determines whether or not a section between two high impact sections (estimated as crossing and landing) has occurred (S144).
- the action detection unit 106 determines whether or not the difference between the vertical acceleration and the horizontal acceleration calculated in S143 exceeds a threshold (TH) in the section (S145). . When the difference exceeds the threshold, it is detected that the section (a section sandwiched between two high impact sections) is a jump section candidate section (S146).
- FIG. 8 is a flowchart showing an example of the vertical acceleration calculation process (S141) shown in FIG.
- the acceleration (D151) included in the sensor data is used.
- the feature amount extraction unit 105 included in the sensor data analysis unit 104 calculates an average value (mean) of acceleration (S152).
- the average value calculated here can be, for example, a moving average.
- the feature amount extraction unit 105 executes gravity component acceleration calculation processing (S153). Further, the feature amount extraction unit 105 calculates the norm of the calculated gravity component acceleration (S154).
- the gravity component acceleration may be calculated based on an average value such as a moving average, or may be calculated using a filter such as an LPF.
- the feature quantity extraction unit 105 processes the acceleration (D151) by BPF (Band Pass Filter) separately from the processing of S152 to S154 described above (S155).
- BPF Band Pass Filter
- the BPF is used for the purpose of removing a DC component (that is, gravity component) included in acceleration by a filter in a low frequency region and further smoothing acceleration by a filter in a high frequency region.
- the BPF in S155 may be replaced by a combination of other types of filters such as LPF and HPF (High Pass Filter).
- the feature amount extraction unit 105 calculates the inner product of the acceleration processed by the BPF and the gravity component acceleration calculated in S153 (S156).
- the feature amount extraction unit 105 divides the inner product calculated in S156 by the norm of the gravity component acceleration calculated in S154 (S157). Thereby, the vertical acceleration (V158) is obtained.
- the vertical acceleration is calculated by projecting the acceleration from which the gravity component is removed by the BPF (S155) in the direction of the gravity component acceleration.
- FIG. 9 is a flowchart showing an example of the horizontal acceleration calculation process (S142) shown in FIG.
- the acceleration (D151) included in the sensor data is also used in the horizontal acceleration calculation process.
- the vertical acceleration calculated in the vertical acceleration calculation process (S141) described above with reference to FIG. 8 is used.
- the feature quantity extraction unit 105 included in the sensor data analysis unit 104 squares and uses the vertical acceleration (S161).
- the feature amount extraction unit acceleration (D151) is processed by the BPF (S162), and the DC component included in the acceleration is removed and the acceleration is smoothed.
- the BPF in S162 may also be replaced by a combination of other types of filters such as LPF and HPF.
- the feature amount extraction unit 105 calculates the norm of the acceleration processed by the BPF (S163), and squares it (S164). Further, the feature amount extraction unit 105 calculates a difference between the square of the vertical acceleration calculated in S161 and the square of the horizontal acceleration calculated in S164 (S165), and the square root of the difference (S166). Obtain horizontal acceleration (V167).
- the jump detection according to the embodiment of the present disclosure is the same as the case where the first example (FIG. 4) is adopted for the free fall detection process in the first example of jump detection (FIG. 2).
- the second example (FIG. 5) is adopted for the free fall detection process
- a total of three types Jump detection processing is possible.
- the sensor data analysis unit 104 including the action detection unit 106 may detect the final jump section based on the results after executing these three types of jump detection processing. More specifically, for example, when a jump section is detected by at least one of the three types of jump detection processing, the action detection unit 106 may detect the section as a final jump section. . Alternatively, the action detection unit 106 may detect the section as a final jump section when a jump section is detected by two or more of the three types of jump detection processing or all three types.
- FIG. 10 is a flowchart illustrating an example of a process for detecting a turn section included in a user action according to an embodiment of the present disclosure.
- the illustrated process is executed by, for example, the sensor data analysis unit 104 included in the information processing apparatus 100 described above.
- the sensor data analysis unit 104 detects rotation included in the user's action (S210), and further detects non-turning rotation included in the rotation (S230).
- a turn is detected from those other than the rotation (S250).
- the non-turning rotation includes, for example, rotation generated by swinging the user when the sensor includes a sensor attached to the user's head or an instrument attached to the user's head.
- the non-turning rotation includes other rotations generated by the user's body movement, more specifically, a sensor that is mounted on the user's arm or a device that is mounted on the user's arm. In some cases, it may include rotation generated by a user's arm swing or arm rotation.
- the sensor data analysis unit 104 can detect a turn section with higher accuracy by detecting a turn section after removing such non-turning rotation. In this sense, it can be said that the non-turning rotation is noise with respect to the detection target turn.
- the sensor data analysis unit 104 detects the rotation included in the user action, and further detects the rotation. It can be said that a turn is detected from a noise obtained by detecting noise included in the rotation and removing the noise from the rotation.
- the sensor data analysis unit 104 executes a rotation section detection process (S210).
- the rotation section is defined as a section where the angular velocity in the horizontal plane direction exceeds a threshold value.
- the sensor data analysis unit 104 determines whether a rotation section has occurred (S201). When a rotation section occurs, first, the sensor data analysis unit 104 performs a head shake detection process (S230). Further, the sensor data analysis unit 104 determines whether or not the swing is detected (S203), and when the swing is not detected, further performs a turn detection process (S250).
- the section generated by the user swinging (for example, when the sensor is mounted on a head-mounted wearable terminal device) is excluded from the rotation section, and the rotation radius or angular velocity is further removed.
- the turn section in which the duration or the like satisfies a desired condition can be extracted.
- FIG. 11 is a flowchart showing an example of the rotation section detection process (S210) shown in FIG.
- acceleration D211
- angular velocity D214
- the feature amount extraction unit 105 included in the sensor data analysis unit 104 calculates an average value (mean) of acceleration (S212).
- the average value calculated here can be, for example, a moving average.
- the feature amount extraction unit 105 executes gravity component acceleration calculation processing (S213).
- the feature amount extraction unit 105 calculates the inner product of the gravity component acceleration calculated in S213 and the angular velocity (D214) (S215). Thereby, the projection of the angular velocity in the direction of the gravitational component acceleration, that is, the angular velocity (V216) in the horizontal plane direction (around the vertical axis) is obtained.
- the feature amount extraction unit 105 temporarily integrates the calculated angular velocity (S217), and calculates the angular displacement (V218) in the horizontal plane direction.
- the feature amount extraction unit 105 processes the angular displacement with the LPF (S219). Further, the feature amount extraction unit 105 differentiates the angular displacement (S220) to obtain the angular velocity (V221) in the horizontal plane direction.
- the angular velocity of V221 is once integrated in S217 as compared with the angular velocity of V218, and the angular displacement after integration is smoothed by being processed by the LPF in S219, and noise is removed from the waveform.
- the action detection unit 106 included in the sensor data analysis unit 104 determines whether or not the angular velocity (V221) in the horizontal plane direction exceeds a threshold (S222), and detects a section where the angular speed exceeds the threshold as a rotation section (S223). .
- FIG. 12 is a flowchart showing an example of the swing detection process (S230) shown in FIG.
- the angular velocity (V221) in the horizontal direction after smoothing calculated in the rotation section detection process shown in FIG. 11 is used.
- the feature amount extraction unit 105 acquires the sign of the angular velocity (S231). Any sign may be defined for the direction of rotation. In the illustrated example, clockwise rotation (V232) and counterclockwise rotation (V233) are defined as signs of angular velocity (V221). To do. Further, the feature amount extraction unit 105 calculates a time interval at which reverse rotation has occurred (S234).
- the feature amount extraction unit 105 determines the time interval from the occurrence of the clockwise rotation (V232) to the occurrence of the counterclockwise rotation (V233), and the counterclockwise rotation ( The time interval from the occurrence of V233) to the occurrence of clockwise rotation (V232) is calculated.
- the action detection unit 106 determines whether or not the time interval calculated in S234 is below a threshold value (TH) (S235), and detects that a swing has occurred when the time interval is below the threshold value. (S236).
- FIG. 13 is a chart showing an example of the turn detection process (S250) shown in FIG.
- the turn detection process a plurality of feature amounts are calculated by the feature amount extraction unit 105, and the action detection unit 106 performs determination based on each threshold based on each feature amount.
- FIG. 13 shows a process for the feature quantity extraction unit 105 to calculate each feature quantity.
- the calculation processing of each feature amount will be described in order, but the processing by the feature amount extraction unit 105 does not necessarily have to be executed in the description order, and the presumed amount is acquired or calculated. If so, the processing can be executed in an arbitrary order.
- the feature quantity extraction unit 105 calculates a norm of acceleration (D251) included in the sensor data (S252), and further calculates an average value of norms in a predetermined time frame (S253).
- the acceleration norm average (V254) calculated in this way is used as one of the feature amounts for detecting the turn.
- the feature amount extraction unit 105 processes the acceleration (D251) with the first LPF (S273), and calculates the gravity component acceleration (V274). Further, the feature amount extraction unit 105 calculates the inner product of the angular velocity (D255) and the gravity component acceleration included in the sensor data (S256). Thereby, the projection of the angular velocity in the direction of the gravitational component acceleration, that is, the angular velocity (V257) in the horizontal plane direction (around the vertical axis) is obtained. The feature quantity extraction unit 105 integrates the calculated angular velocity (S258), and calculates the angular displacement (V259) in the horizontal plane direction. Angular displacement (V259) is also used as one of feature quantities for turn detection.
- the feature amount extraction unit 105 calculates the angular velocity (V261) based on the angular displacement (V259) and the duration (V260) of the rotation section to be processed.
- the angular velocity of V261 can be a longer time frame (for example, smoothed over the entire rotation section, for example, compared to the angular speed of D255.
- the duration of the rotation section (V260) and the angular change rate (V261) are also detected by the turn detection. Is used as one of the feature quantities for
- the feature quantity extraction unit 105 calculates several feature quantities by analyzing the angular displacement (V259) for a predetermined time frame (S262). More specifically, the feature amount extraction unit 105 determines the maximum value (S263, V268), average value (S264, V269), variance (S265, V270), and kurtosis (S266, V271) in the time frame. , And skewness (S267, V272). These feature amounts are also used as feature amounts for turn detection.
- the feature quantity extraction unit 105 processes the acceleration (D251) with the second LPF (S275).
- the first LPF (S273) is used to extract the gravitational component acceleration (V274), which is a DC component included in the acceleration, whereas the second LPF (S275) Used to smooth acceleration by filtering the high frequency region. Therefore, the passband settings of these LPFs can be different.
- the feature amount extraction unit 105 calculates the inner product of the acceleration smoothed by the second LPF (S275) and the gravity component acceleration (V274) extracted by the first LPF (S273) (S276). Thereby, vertical acceleration (V277) is obtained. Further, the feature amount extraction unit 105 calculates the difference between the acceleration vector obtained by combining the gravity component acceleration (V274) and the vertical acceleration (V277) and the acceleration smoothed by the second LPF (S275) (S278). ). Thereby, horizontal acceleration (V279) is obtained. The feature amount extraction unit 105 calculates the average value of the horizontal acceleration (S280). The average value (V281) of the horizontal acceleration calculated in this way is also used as a feature value for turn detection.
- the action detection unit 106 determines whether or not a turn has occurred based on the feature amount extracted from the sensor data as described above, for example.
- the action detection unit 106 includes the duration of the rotation section (V260), the angular displacement in the horizontal plane (V259), the smoothed angular velocity (V261), the acceleration norm average (V254), and the horizontal acceleration average. The determination is performed based on the value (V281), the maximum value (V268) of the angular velocity in the time frame, the average value (V269), the variance (V270), the kurtosis (V271), and the skewness (V272).
- the feature amount used for the determination is not limited to the above example.
- a feature amount other than the above example may be used, or a part of the feature amount of the above example may not be used.
- the type of feature quantity used for turn detection may be determined by principal component analysis based on sensor data when a turn actually occurs.
- the feature-value used for determination may be determined based on the tendency of the sensor data that appears when a turn actually occurs.
- the acceleration norm average (V254) and the horizontal acceleration average value (V281) are feature quantities related to the turning radius of the turn.
- the threshold value of each feature amount applied in the determination by the action detection unit 106 is determined according to the result of machine learning based on sensor data when a turn actually occurs, for example. At this time, whether or not a turn has actually occurred may be determined manually with reference to, for example, an action video acquired simultaneously with the sensor data. Further, not only whether or not a turn has occurred, but a label indicating what kind of turn may be given. More specifically, for example, as a result of referring to the video, a label indicating each attribute of an action determined on the service provider side that the service provider wants to detect as a turn, does not want to detect as a turn, or either may be detected May be given.
- action detection processing executed in an embodiment of the present disclosure have been described.
- the action detection processing executed in the present embodiment is not limited to jumps and turns that occur on snowboards.
- action detection processing is performed on jumps and turns that occur in sports other than snowboards or scenes other than sports. May be executed.
- an action other than a jump or turn may be detected.
- the action detection unit 106 may detect a fall that occurs on a snowboard or the like.
- the feature amount extraction unit 105 calculates the norm of the acceleration in the same manner as the above-described jump and turn detection, and the action detection unit 106 determines that the acceleration norm is a threshold (for example, large enough not to occur in normal sliding). The occurrence of a fall may be detected when the value exceeds (which may be a value).
- the scoring processing unit 109 included in the analysis result processing unit 107 may perform the action section including the jump section and / or the turn section detected by the processing described with reference to FIGS. 2 to 13 above.
- a score (action score) for evaluating the generated action is calculated.
- the action score can be calculated, for example, by extracting physical quantities (feature quantities) representing good or bad actions and features from sensor data in the action section and weighting and adding them.
- the service control unit 112 generates information on the action (for example, jump or turn) based on the score calculated in this way.
- the duration of the section (the angular displacement around the X axis / Y axis / Z axis in the section), the ratio of the free fall section, the magnitude of the impact at the time of crossing / landing, etc. It can be extracted as a feature amount for calculating a score.
- the duration of the section, the displacement angle, the average value of each speed, the maximum value, and the standard deviation, the maximum value of the angular acceleration and the standard deviation, etc. are the feature quantities for calculating the score. Can be extracted.
- the weighted addition coefficient can be set according to the nature of the action emphasized in the service 113 provided by the information processing apparatus 100, for example.
- the method for calculating the action score from the feature amount is not limited to the weighted addition, and other calculation methods may be used.
- the action score may be calculated by applying a machine learning algorithm such as a linear regression model.
- the clustering processing unit 108 included in the analysis result processing unit 107 performs an action section including a jump section and / or a turn section detected by the processing described above with reference to FIGS. Then, a clustering algorithm such as the k-means method is applied using the feature amount extracted for scoring, and the detected actions are classified into clusters.
- a clustering algorithm such as the k-means method is applied using the feature amount extracted for scoring, and the detected actions are classified into clusters.
- actions may be classified into clusters according to the length of the duration of the section or the magnitude of rotation.
- the result of clustering is used, for example, to extract action sections so that various types of actions such as jumps and turns are included in the moving image when a digest moving image is provided as a service. Also, by classifying good actions and bad actions into separate clusters, the user may look back on the actions or use them for coaching to improve the actions.
- the analysis result processing unit 107 may calculate the similarity between the action sections based on the correlation coefficient of the feature amount as the same process as the clustering (action sections with high similarity are classified into the same cluster). It can be treated in the same way as the action section that was made). In addition, for example, the analysis result processing unit 107 prepares a characteristic amount pattern of a typical type of action in advance, and determines which type the newly generated action corresponds to by a k-NN method or the like. May be.
- FIG. 14 is a block diagram illustrating an example of processing for estimating a sensor mounting state according to an embodiment of the present disclosure. More specifically, the illustrated configuration determines whether a sensor that provides sensor data is mounted directly on the user's body or on an instrument used by the user. The illustrated process is executed by, for example, the sensor data analysis unit 104 included in the information processing apparatus 100 described above. In the illustrated example, the cut-off frequency (Fc) of the filter and the length of the time frame are specifically described. However, these numerical values are examples, and may be appropriately changed according to actual sensor characteristics. Can be done.
- Fc cut-off frequency
- the receiving unit 102 of the information processing apparatus 100 receives sensor data provided by a three-axis (u, v, w) acceleration sensor 121.
- the sensor data analysis unit 104 acquires this sensor data via the sensor device control unit 103.
- the above-described determination processing is based on the fact that when the sensor is directly attached to the user's body, the high-frequency component of acceleration is attenuated by the body functioning as an LPF.
- A amplitude of the low frequency component that has passed through the LPF 124)
- B amplitude of the high frequency component that has passed through the HPF
- the threshold determination 130 if the value obtained by processing A / B with the HPF 129 is larger than the threshold, it is determined that the sensor is directly attached to the user's body, and if not, the sensor is attached to the instrument. It can be determined that it is attached.
- the estimation result as described above may be used inside the sensor data analysis unit 104, for example.
- the sensor data analysis unit 104 changes the threshold value, the filter setting value, and the like depending on whether the sensor is attached to the body or the appliance in the process of detecting the user action as described above. Also good.
- the estimation result as described above is fed back to the sensor device control unit 103 and used for setting parameters relating to measurement of the sensor device, determining the sensor data preprocessing method by the sensor device control unit 103, and the like. May be.
- adaptive control related to sensor data processing may be performed based on estimation related to the state of the sensor data providing side, such as estimation of the sensor mounting state described above.
- the sensor data analysis unit 104 estimates the type of sport in which an action has occurred using an algorithm such as machine learning from the impact strength or movement pattern detected by an acceleration sensor or the like. Also good.
- the sports may be estimated for each generally recognized event, or may be estimated for each system such as board sports, water sports, bicycle competitions, and motor sports.
- the sensor data analysis unit 104 estimates the type of device (for example, in the case of skiing, whether it is mounted on a ski or mounted on a stock). May be.
- the estimation result may be used, for example, for controlling a threshold value or a filter setting value in action detection or the like, or fed back to the sensor device control unit 103 and similar to the sensor mounting state estimation result described above. It may be used for device control and determination of sensor data pre-processing method.
- the receiving unit 102 receives the position information of the user from the sensor device in addition to the sensor data and time information (time stamp).
- the action detection unit 106 included in the sensor data analysis unit 104 stores, in the detection section information holding unit 110, information that associates the time stamp and the position information with the user action detected based on the sensor data.
- the service control unit 112 can acquire user action information, and spatial information (position information) and time information (time stamp) associated with the action information from the detection section information holding unit 110.
- the action detecting unit 106 displays time information (time stamp) at the time when the action is detected, It may be used instead of the time stamp received together with the sensor data.
- time information time stamp
- the action detection unit 106 receives the position information acquired by the terminal device at the time when the action is detected, together with the sensor data. It may be used instead of the position information.
- the service control unit 112 can acquire user action information, spatial information (position information) and time information (time stamp) associated with the action information from the detection section information holding unit 110.
- the analysis result processing unit 107 associates the time stamp and the position information provided together with the action detection result from the action detection unit 106 with the additional information generated based on the action detection result, and the additional information holding unit 111, the service control unit 112 acquires the generated additional information, spatial information (position information) and time information (time stamp) associated with the action information from the additional information holding unit 111. be able to.
- the action information is not limited to information directly indicating the user action detected by the action detection unit 106 but may include various information related to the detected user action. Therefore, in the above example, not only the action detection result provided by the action detection unit 106 but also additional information generated by the analysis result processing unit 107 is included in the action information. Moreover, action information is not restricted to what was detected by the action detection part 106 based on sensor data, For example, what was detected when the user input own condition may be contained.
- the service control unit 112 that has acquired the user action information, the spatial information (position information) and the time information (time stamp) associated with the action information, expresses the action information as a map based on the spatial information.
- the map representation includes, for example, arranging display elements such as text, graphics, icons, or textures corresponding to the action information on the map based on the spatial information.
- the display element may correspond to a single action information, or a plurality of action information, for example, action information generated based on sensor data respectively provided by a plurality of users, or a single user. May correspond to action information generated based on sensor data sequentially provided in time series.
- the service control unit 112 considers time information associated with the action information when expressing the action information as a map. More specifically, for example, the service control unit 112 may determine display elements corresponding to two pieces of action information associated with different time information in consideration of the time relationship indicated by the time information. Good. For example, the service control unit 112 may determine a display element corresponding to action information associated with certain time information in consideration of the relationship between the time indicated by the time information and the current time.
- FIG. 15 is a flowchart illustrating processing including some examples of map representation of action information according to an embodiment of the present disclosure.
- the action detection unit 106 included in the sensor data analysis unit 104 detects an action section (S301).
- the action section may include, for example, a jump section and / or a turn section detected by the processing described above with reference to FIGS.
- the action section may include a section in which another user action detected based on sensor data such as walking, running, stationary, moving with a vehicle, and the like has occurred.
- the scoring processing unit 109 included in the analysis result processing unit 107 calculates an action score for the action section detected in S301 (S302). Further, action information related to the action section and action score, and data including user ID, position information, separately acquired action video data, and the like are uploaded (S303).
- the upload in S303 may be, for example, an upload from a server that implements the functions of the sensor data analysis unit 104 or the analysis result processing unit 107 to a server that implements the service control unit 112. Alternatively, the upload in S303 may be upload from a terminal device that implements the functions of the sensor data analysis unit 104 and the analysis result processing unit 107 to a server that implements the service control unit 112. When these servers or terminal devices are the same, upload is read as registration in an internal database, for example.
- the service control unit 112 that has received an upload of action sections and action scores detected for individual users calculates the skill level of the user as an additional process (S304).
- the skill level is calculated based on a history of action scores calculated for each user.
- a server that implements the service control unit 112 can use a database that holds a history of user action scores.
- a database holding the skill level of the user may be available, and the service control unit 112 that has calculated the skill level in S304 may update the skill level database.
- the service control unit 112 updates the location DB based on the result of the processing up to S304 (S305).
- the location DB is realized in, for example, a memory or a storage (which may be an external storage connected via a network) that can be used by the information processing apparatus including the service control unit 112, and is a target for providing information by the service control unit 112 Stores location information within a geographic area to become.
- action information is associated with spatial information defining the place.
- the place here may be, for example, an arbitrary place defined on a map, a grid of a predetermined size, or a discrete spot defined in advance.
- the definition of a place differs depending on, for example, the use form of place information described later.
- an example using location information will be further described.
- the service control unit 112 uses the action information associated with the spatial information in the place DB to create expanded map information (S306). Furthermore, the service control unit 112 draws the created map in response to, for example, a user request (S307). In the map, based on the action information for a single user or a plurality of users, the action that occurs at the highest frequency in each place is displayed as a characteristic of the place. Here, the service control unit 112 considers time information associated with the action information when creating the extended map information.
- FIG. 16 is a diagram illustrating a first example of a map representation of action information according to an embodiment of the present disclosure.
- a pattern 1101 indicating an action occurring frequently in each place (a pattern 1101a indicating walking, a pattern 1101b indicating riding on a train, a pattern 1101c indicating staying)
- a pattern 1101d indicating shopping labels 1103 of the respective actions (labels 1103a to 1103d corresponding to the above-described patterns 1101a to 1101d), and an attribute selection 1105 are displayed.
- the service control unit 112 considers the relationship between the time indicated by the time information and the reference time when creating the map information for drawing the map screen 1100. For example, the service control unit 112 performs control such that the influence of the action information in the map expression becomes smaller as the time indicated by the time information is farther from the reference time.
- the action information may include a score of each category when the user's action is classified into any of a plurality of categories (more specifically, The score may be 1 for the action category specified by the action detection unit 106 and 0 for the other categories, or the occurrence probability of each category calculated by the action detection unit 106 may be treated as a score. Good).
- the service control unit 112 determines that the time indicated by the time information is the current time (the time when the map information is generated) in the score included in the action information associated with the common spatial information (position information).
- the category of the representative action associated with the common spatial information in the map representation on the map screen 1100 may be determined by adding weights that become smaller as they leave.
- the service control unit 112 scores for each category (for example, walking, train, stay, shopping, etc.) indicated by the action information associated with the spatial information in the grid. Are added with a weight based on the time information as described above. As a result, the category for which the highest score is calculated becomes the category of the representative action associated with the grid.
- the pattern 1101 and the label 1103 displayed on the map screen 1100 correspond to the representative action category determined as described above.
- the service control unit 112 may use the pattern 1101 and the label 1103 in the map screen 1100 based on action information associated with time information indicating a time at which the difference from the current time is less than a threshold among available action information. May be displayed.
- the action information extracted based on the difference between the current time and the time indicated by the time information may be handled uniformly, or the time indicated by the current time and the time information as in the above example. Further processing such as weighting of the score may be executed in accordance with
- the current time may be automatically determined as the time when the map information is generated, or any past time may be determined by a user operation or the like as a time that replaces the current time. Therefore, the current time described in this specification may be read as a reference time for generating map information.
- the service control unit 112 when the service control unit 112 displays the pattern 1101 and the label 1103 in the map screen 1100, the action information associated with time information having an attribute common to the current time (time when map information is generated). More specifically, for example, control may be performed such that the action information indicating the action that occurred in the same time zone or season as the current time has a greater influence on the map representation. Similar to the above example, the service control unit 112 may assign a weight to the score included in the action information that increases as the commonality of the attribute with the current time increases. Alternatively, the service control unit 112 may use the pattern 1101 and the label 1103 in the map screen 1100 based on the action information associated with the time information whose commonality with the current time exceeds the threshold among the available action information. May be displayed.
- the service control unit 112 may generate map information by selectively using action information associated with the same time zone (for example, morning, noon, evening, night, etc.) as the current time. . Further, the service control unit 112 may generate map information by selectively using action information associated with the same season (for example, spring, summer, vacation season, etc.) as the current time. Note that the time zone, season, and the like may be arbitrarily selected by attribute selection 1105 described below. That is, the attribute selection 1105 is not limited to selecting a user attribute associated with action information, and may be used to select an attribute of time information associated with action information.
- the attribute selection 1105 can select the attribute of the user whose action is to be displayed by the pattern 1101 or the label 1103. For example, as shown in the figure, it may be possible to select the user's age by the attribute selection 1105 (the twenties are selected). In this case, the service control unit 112 generates map information based on user action information having the selected attribute. In another example, other attributes of the user, more specifically gender, occupation, etc. may be selectable by the attribute selection 1105. Alternatively, the service control unit 112 may automatically select an attribute of a user to display an action in the map information according to an attribute of the user who provides the map information.
- FIG. 17 is a diagram showing an example in which the map expression changes in real time in the example shown in FIG.
- the map screen 1100a similar to the example shown in FIG. 16, as a result of the operation of the railway (RWY) ahead of the station (STATION) displayed in the center due to the accident being stopped, the map screen As shown in 1100b, the action displayed by the pattern 1101 and the label 1103 changes. More specifically, the pattern 1101 indicating that the train is displayed along the railway (RWY) is not displayed, and instead, the user is on the bus along another place (bus route). A pattern 1101 and a label 1103 indicating this are newly displayed.
- a route navigation 1109 based on a condition 1107 set in advance or input by the user may be displayed.
- the route navigation 1109 is displayed by extracting a combination of actions that meet the condition 1107 from the action detection result of the most recent user as shown on the map screen 1100b, for example. More specifically, the route navigation 1109 in the illustrated example is displayed as a combination of walking from the station, staying, and subsequent bus rides.
- the service control unit 112 is included in, for example, the user's planned travel route (which can be determined based on the result of route search performed in advance, a schedule, a route that customarily travels, etc.). In such a case, the route navigation as shown in the map screen 1100c is automatically provided based on the fact that the action of another user on the train is no longer detected as shown in the map screen 1100b. May be.
- FIG. 18 is a diagram illustrating a second example of a map representation of action information according to an embodiment of the present disclosure.
- icons 1201, 1203, and 1205 are displayed on the map screen 1200, which indicate actions that occur frequently at each location. More specifically, the map screen 1200 may show an action at a venue for a music festival held outdoors, for example. Icons 1201 and 1203 indicate the excitement of users gathered at the venue, and the icon 1205 indicates the movement of users between the venues. Also on the map screen 1200, the icons 1201, 1203, and 1205 can intuitively grasp the actions occurring in the respective places. Also in this example, the service control unit 112 considers time information associated with each action information when creating map information for drawing the map screen 1200.
- the service control unit 112 assigns a weight so that an action occurring at a time closer to the current time (the time when the map is generated) is more strongly reflected. It may be attached. Or the service control part 112 may limit the action information used when specifying the action in each place to the latest thing linked
- the service control unit 112 when the service control unit 112 specifies an action at each place, the service control unit 112 has a time having an attribute common to the current time (time when map information is generated), more specifically, the previous time or the previous time.
- the action information may be weighted so that actions that occur during the music festival are more strongly reflected. Similar to the first example described above, the service control unit 112 may further weight the action information according to a time zone, a season, or the like.
- elements for attribute selection are not shown in the map screen 1200, the same elements as the attribute selection 1105 in the first example are displayed, and the attributes of time information and user attributes associated with the action information to be displayed are displayed. It may be possible to select.
- the time context indicated by the time information associated with the action information is considered.
- the movement of the user indicated by the icon 1205 may be determined by a plurality of pieces of action information respectively associated with a series of time stamps being associated with position information that changes along the movement path of the user. .
- the direction of movement can be specified based on the time relationship indicated by the time information associated with the action information.
- the first action information associated with the position P1 position indicated by the first spatial information shown in FIG. 18 and the first action information associated with the position P2 (position indicated by the second spatial information).
- the time information associated with the second action information indicates a time later than the time information associated with the first action information, the user has moved from the position P1 to the position P2. Is judged
- the action information itself may be a condition that indicates an action that occurs with movement such as walking or running, and the action information indicates other actions such as stay or jump. Also good. Since the movement of the user is often slow at festival venues, etc., the user is based on the position indicated by the spatial information associated with the action information and the context of the time indicated by the time information as in the above example. It may be appropriate to detect the movement of.
- the action score calculated by the scoring processing unit 109 included in the analysis result processing unit 107 based on the action information is used to specify the action at each place.
- the icons 1201 and 1203 indicate the excitement of users gathered at the venue. A user's excitement is detected by, for example, a certain number of users gathering at the venue.
- the scoring processing unit 109 may calculate a jump action score, and the service control unit 112 may display an icon 1201 indicating a large excitement when there are many users jumping more actively.
- the service control unit 112 may display an icon 1203 indicating a quiet excitement. Good.
- the action information is associated with the time information in this embodiment, the degree of instantaneous excitement is based on whether the number of users who are gathering or jumping is increasing or decreasing. Even if they are the same, it is also possible to distinguish and display the venue that will be excited from now on and the venue that has already passed the peak of the excitement.
- the action is not limited to the illustrated pattern, text, and icon, and may be displayed on the map by a heat map, an animation using an avatar, or the like.
- the action score may be reflected not only in the icon display as in the example shown in FIG. 18 but also in the pattern density and text font in the examples shown in FIGS.
- the actions displayed on the map may be classified into categories given in advance, or may be classified according to the result of clustering by the clustering processing unit 108 included in the analysis result processing unit 107.
- the above example is not limited to the examples of urban areas and festival venues as shown in FIGS. 16 to 18, and can be applied to maps of various places.
- a map showing how to get to the venue (walking, cycling, etc.) and how to spend at the venue (standing, sitting, eating, etc.) .
- the slope is congested based on the fact that there are many users sliding, or the lift waiting time is long (before the action of getting on the lift, (For example, if the action of stay is detected for a long time) or the unevenness made of snow based on the fact that there are many jumping users (since it changes from day to day, the map display based on the most recent action)
- the presence of (active) can be displayed.
- a jump with a long flight time may be detected separately from a normal jump, and a ski resort where many such jumps are detected may be specified as a facility for advanced users. In this case, the skill level of the user may be considered when specifying the facility.
- the service control unit 112 uses the action information associated with the spatial information in the location DB to calculate the safety level and / or the risk level for each location (S308). Furthermore, the service control unit 112 updates the safety map DB based on the calculated safety level and / or risk level (S309), and further, for example, in response to a user request, a safety map based on the content of the safety map DB is updated. Drawing is performed (S310).
- the safety level and / or the risk level may be calculated for a place on a course where a specific sport is performed, such as a skiing or snowboarding course or a running course. More specifically, for example, it may be determined that the risk level is high for a place where many falling actions are detected in skiing or snowboarding. At this time, the skill level of the user associated with the falling action may be considered. Similarly, it may be determined that the safety level is high for a place where many successful actions are detected in skiing or snowboarding. Again, the skill level of the user associated with the successful action may be taken into account.
- a place where a user with a high skill level has fallen is estimated to have a high degree of danger.
- a place where a user with a high skill level has not fallen is estimated to be less dangerous for a user with a high skill level.
- the skill level can be calculated based on the action score by the service control unit 112 as an additional process for the analysis result of the sensor data.
- the action score can be calculated by the scoring processing unit 109 included in the analysis result processing unit 107. Therefore, in the present embodiment, when considering the skill level of the user who executed the action in creating the safety map, it can be said that the service control unit 112 further considers information regarding the action score included in the action information. .
- the action detection unit 106 may detect a certain tempo run as an action.
- the service control unit 112 determines that the action that the user runs at a constant tempo is abruptly terminated without speed reduction or shift to walking (running is suddenly interrupted), and at the same time, such as pulse or sweating.
- Estimating that a dangerous event has occurred on the running course eg, likely to come in contact with a car, bicycle, other runner, etc.
- the service control unit 112 can estimate that the risk level is high for a place where the dangerous events detected in this manner frequently occur.
- the degree of safety and / or risk on a driving route or a walking route may be calculated without being limited to sports.
- the risk level or safety level of the course changes every moment depending on weather conditions such as temperature and weather. For example, even in the same place on the course, what was safe the previous day may be dangerous, and what was dangerous in the morning may become unsafe because the temperature rose in the daytime. Similarly, in the running course and the driving course, the degree of danger or the degree of safety may change from moment to moment depending on weather conditions and traffic conditions. For this reason, even in this example of generating a safety map, actions that occurred at a time closer to the current time are more strongly reflected in the map information, as in the example described above with reference to FIGS. Is valid.
- FIG. 19 is a diagram illustrating an example of safety map generation according to an embodiment of the present disclosure.
- FIG. 19 for example, when the risk or safety level is continuously distributed as in the course of skiing or snowboarding, the risk level or safety level of the entire course is estimated from actions detected in a limited place.
- An example of the method is shown.
- R1 and R2 indicate positions where a fall has occurred
- S1 indicates a position where an action has been successful
- X indicates an arbitrary position that is a target for calculating the degree of risk or safety.
- the process for generating the safety map in the illustrated example will be further described.
- the service control unit 112 calculates the risk scores P R1 and P R2 at the positions R1 and R2 and the safety score P S1 at the position S1 by, for example, the following formulas 1 to 3.
- W LV is a weighting coefficient corresponding to the skill level of the user associated with the falling or successful action.
- risk scores P R1 and P R2 negative evaluation of action
- P S1 positive evaluation of action
- the score is corrected downward as the user skill level increases.
- Power R1 and Power R2 are the impact strengths detected at the time of falling. In this example, the strength of impact is treated as corresponding to the action score of falling.
- Score S1 is an action score calculated for a successful action (for example, jump or turn).
- W R and W S are the action corresponding to the risk score (e.g. falling), the action corresponding to the safe score (e.g. successful jumps and turns), a weighting coefficient set respectively.
- t PASS_R1 , t PASS_R2 , and t PASS_S1 are elapsed times from the occurrence of each action to the time when the safety map is generated (for example, the current time).
- P R1, P R2 as can be seen from the fact that the reciprocal of the time these are combined over the respective P S1, in the present embodiment, as the action time elapses from the occurrence, the action safe score and The impact on the risk score is small.
- the safety score and the risk score are inversely proportional to the elapsed time, but the score may be inversely proportional to an arbitrary function including the elapsed time.
- the service control unit 112 averages the calculated risk score and safety score according to the distance from each of the positions R1, R2, and S1 to the position X to be calculated as shown in Equation 4. sum in, to calculate the score P X of the position X.
- D R1 , D R2 , and D S1 are distances from the positions R1, R2, and S1 to the position X as shown in FIG.
- the safety score is positive
- the risk score is summed as negative.
- the magnitude of the safety score and the risk score added together decreases in inverse proportion to the distance, but the score may decrease in inverse proportion to any function including the distance.
- the position X when the score P X is a positive value is relatively safe
- position X when the score P X is a negative value so as to allow determination that a relatively dangerous it may adjust the value of the weighting coefficient of the W R and W S.
- the position X is indicated.
- the action information associated with the spatial information is between the position Sk or Rj (the position indicated by the spatial information associated with the action information) where the action for which a safety score or risk score is given and the position X. It can be said that it is estimated based on the distance.
- the map when generating a safety map as described above at a ski resort, the map can be generated based on the latest action information to reflect the environmental change on the slope on the safety map. Further, the risk score and the safety score may be calculated according to the skill level of the user who refers to the safety map.
- the weighting coefficient W LV corresponding to the skill level of the user described above may be changed according to the skill level of the user who refers to the safety map (for example, the coefficient W LV is the user who performed the action). May be set to be 1 when the skill level is the same as the skill level of the user who refers to the safety map).
- a safety map that matches the environmental conditions such as time zone and climate and the skill level for each user, and to present a course that allows each user to descend safely. For example, a place where an ice burn occurred in the morning and a user with a high skill level (advanced user) had fallen, and even snow with a lower level of snow falls in the daytime so that even a user with a low skill level (beginner user) can slide safely
- a course including such a place is not presented as a safe course to at least beginner users in the morning, but can be presented as a safe course to beginner users from noon. It is.
- the safety map as described in the above example is not necessarily generated on a sports course such as skiing, and may be generated in daily life, for example. More specifically, for example, when there is a road whose road surface is bad due to rain or snow, it is determined that the road is dangerous due to such a road (for example, the fall of another user actually occurs). It may be possible to search for a route for safely moving to the destination while avoiding (a) and a route that can recognize the danger and can move to the destination in the shortest distance.
- the safety and danger criteria in the above example may be replaced with other positive / negative criteria, specifically, for example, comfort and discomfort criteria.
- comfort and discomfort criteria for example, in the case of train movement, if congestion in the vehicle is detected, this is regarded as an uncomfortable action, and a route for comfortably moving to the destination avoiding such a crowded route The route that can move to the destination in the shortest time even when there is congestion may be searchable.
- the service control unit 112 may provide navigation based on the past action history of other users using action information associated with the spatial information in the place DB. More specifically, the service control unit 112 performs matching based on spatial information of the action history of the first user detected up to a certain time point with the past action history of another user (second user).
- the service control unit 112 performs matching based on spatial information of the action history of the first user detected up to a certain time point with the past action history of another user (second user).
- S311 For example, in this matching, the action history of the second user detected together with the movement route similar to the movement route of the first user up to a certain point in time can be extracted. At this time, the action history may be extracted in consideration of the skill level of the associated user. The action history may be extracted in consideration of the action history of the first user himself detected in the past.
- the service control unit 112 may match the action history of the first user and the action history of the second user in consideration of the co-occurrence probability and the transition probability.
- the service control unit 112 calculates a route based on the extracted action history (S312), and presents navigation to the first user (S313).
- FIG. 20 is a diagram for describing an example of navigation considering a skill level and an action history according to an embodiment of the present disclosure.
- FIG. 20 shows three types of navigation examples (A) to (C) at the ski resort.
- A the first user arrives at a ski resort and gets on the A lift.
- the service control unit 112 presents the route (A) shown in the figure.
- the route (A) is a route that is extracted based on the past action history of the second user, who is also a beginner user, for example, and includes the downhill of the B course.
- the service control unit 112 repeats the next and repeats the route of (A) including the downhill of the B course as the first route. Present to users. If the score of the action such as turn or jump executed when the first user actually descended B course is high and no failure (falling, etc.) is detected, the service control unit 112 gets on the A lift again.
- the route (B) or (C) extracted based on the past action of the second user who is an intermediate user with a high skill level is presented to the first user. These routes include D course downhill.
- the service control unit 112 when the action score when the first user slides down the B course is particularly high for the turn, the service control unit 112 is in the course passing through the place where many turns occur in the D course.
- the route of (B) including navigation is presented.
- the service control unit 112 presents the route (C) including navigation in the course that passes through the place where many jumps occurred in the D course.
- the navigation in the courses provided in (B) and (C) is, for example, among users who have run down the D course in the past, especially those who have performed many turns and those who have performed many jumps. Provided based on the route.
- the service control unit 112 may present the route (A) to the first user again.
- the service control unit 112 may present the route (B) or (C) to the first user from the beginning. Good.
- whether to present the route (B) suitable for performing many turns or the route (C) suitable for performing many jumps depends on the past action of the first user. The determination can be made based on the ratio of the turn and the jump in the action executed in the history (regardless of the position information) or the action score of each of the turn and the action in the action history.
- the action information acquired by the service control unit 112 is the first associated with the first series of spatial information (corresponding to the route from the arrival at the ski resort to the A lift).
- Second action information associated with one action information series (corresponding to the action of the first user) and a second series of spatial information (root of (A), (B), or (C)) Series (corresponding to the action of the second user).
- the second action information series is associated with time information indicating a time before the first action information series.
- the service control unit 112 determines that the first movement route indicated by the first series of spatial information is partially (arrived at the ski resort) on the second movement route indicated by the second series of spatial information.
- the second user walking down the mountain in the past Provides navigation based on action history, and provides navigation based on the action history of a second user who has run down the mountain in the past if the first user is running down (trailing) can do.
- the conditions for searching the action history of the second user based on the attributes and preferences including the skill level of the first user are not limited to the action history actually executed by the first user so far. May be set.
- navigation suitable for the current situation of the first user is set by setting weights and the like so that the action history of the second user that matches the conditions such as the time zone, season, and weather is easily detected. Is likely to be able to provide.
- the service control unit 112 may change the setting for action detection using action information associated with the spatial information in the place DB. More specifically, the service control unit 112 extracts an action detection tendency for each position indicated by the spatial information from the information stored in the place DB (S314). In accordance with the extracted action detection tendency, the service control unit 112 adjusts settings for detecting an action. More specifically, for example, the service control unit 112 updates the parameter table referred to by the action detection unit 106 included in the sensor data analysis unit 104 (S315).
- FIGS. 21 and 22 are diagrams for describing an example of changing the action detection setting according to an embodiment of the present disclosure.
- (A) is an example of the change of the action detection setting in the area
- the “jump” detected from time to time is estimated to be a false detection.
- the “jump” probability is lowered and the “drive” probability is raised.
- (B) is an example of changing the action detection setting in the illustrated area 1303.
- the area 1303 there is a night club, and there is a tendency that a jump having a short duration in the vertical direction is continuously detected by a user who enjoys the provided music.
- the jump detection probability for a short duration may be increased with respect to the jump detection probability for each duration.
- jumps can be reliably detected, and the excitement of the nightclub can be expressed on the map depending on, for example, the frequency of jumps and the number of users jumping (this example is shown in FIG. 18 for example). The same applies to the music festival example shown in the above).
- (C) is an example of changing the action detection setting at the place of the kicker 1305 provided in the course of the ski resort.
- Such kickers tend to detect jumps that include rotation as an action.
- snowboard jumps are often accompanied by large rotations (eg, over 360 degrees) that do not normally occur. Therefore, when the location of the kicker 1305 is given in advance, or the presence of the kicker 1305 is estimated based on the action detection result (a lot of jumping down on the snowboard is detected), the action detection is performed.
- the upper limit of 360 degrees may be expanded to a range including 720 degrees and 1080 degrees in normal times), and then the jump detection probability is increased. Also good.
- the service control unit 112 searches the ID of the facility where the action is detected from the position information included in the data uploaded in S303 (S316), and is detected in the facility.
- the facility level is calculated based on the distribution of the action score of the action and the skill level of the user who executed each action (S317), and the facility data including the facility level information is provided (S318).
- a low level (indicating that the difficulty level is low) may be given to a facility where an action with a high score is detected for many users regardless of the skill level.
- a facility in which an action with a high score is detected only for a user with a high skill level may be given a high level (indicating that the difficulty level is high).
- the fact that an action such as a fall (indicating that the attempted action has failed) may also be reflected on the facility level.
- a level indicating the popularity level can be calculated according to the number of users whose actions are detected.
- the facility can take various forms depending on the type of sport in which the action occurs, such as a course, a court, or a field.
- facilities can be defined in various units such as a park including a plurality of courses, a specific jumping table in the course, and the like.
- facilities can be defined in various units.
- the facility ID is searched based on the location information.
- the service control unit 112 refers to the location DB and determines the facility ID from the spatial information associated with the action information. You may specify.
- FIG. 23 is a block diagram illustrating a hardware configuration example of the information processing apparatus according to the embodiment of the present disclosure.
- the information processing apparatus 900 includes a CPU (Central Processing unit) 901, a ROM (Read Only Memory) 903, and a RAM (Random Access Memory) 905.
- the information processing apparatus 900 may include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925.
- the information processing apparatus 900 may include an imaging device 933 and a sensor 935 as necessary.
- the information processing apparatus 900 may include a processing circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array) instead of or in addition to the CPU 901.
- DSP Digital Signal Processor
- ASIC Application Specific Integrated Circuit
- FPGA Field-Programmable Gate Array
- the CPU 901 functions as an arithmetic processing device and a control device, and controls all or a part of the operation in the information processing device 900 according to various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927.
- the ROM 903 stores programs and calculation parameters used by the CPU 901.
- the RAM 905 primarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like.
- the CPU 901, the ROM 903, and the RAM 905 are connected to each other by a host bus 907 configured by an internal bus such as a CPU bus. Further, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.
- PCI Peripheral Component Interconnect / Interface
- the input device 915 is a device operated by the user, such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever.
- the input device 915 may be, for example, a remote control device that uses infrared rays or other radio waves, or may be an external connection device 929 such as a mobile phone that supports the operation of the information processing device 900.
- the input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the input signal to the CPU 901. The user operates the input device 915 to input various data and instruct processing operations to the information processing device 900.
- the output device 917 is configured by a device capable of notifying the acquired information to the user using a sense such as vision, hearing, or touch.
- the output device 917 can be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, an audio output device such as a speaker or headphones, or a vibrator.
- the output device 917 outputs the result obtained by the processing of the information processing device 900 as video such as text or image, sound such as sound or sound, or vibration.
- the storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900.
- the storage device 919 includes, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device.
- the storage device 919 stores, for example, programs executed by the CPU 901 and various data, and various data acquired from the outside.
- the drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built in or externally attached to the information processing apparatus 900.
- the drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905.
- the drive 921 writes a record in the attached removable recording medium 927.
- the connection port 923 is a port for connecting a device to the information processing apparatus 900.
- the connection port 923 can be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, or the like.
- the connection port 923 may be an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, or the like.
- the communication device 925 is a communication interface configured with, for example, a communication device for connecting to the communication network 931.
- the communication device 925 can be, for example, a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), NFC (Near Field Communication), or a communication card for WUSB (Wireless USB).
- the communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communication.
- the communication device 925 transmits and receives signals and the like using a predetermined protocol such as TCP / IP with the Internet and other communication devices, for example.
- the communication network 931 connected to the communication device 925 is a network connected by wire or wireless, and may include, for example, the Internet, a home LAN, infrared communication, radio wave communication, satellite communication, or the like.
- the imaging device 933 uses various members such as an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device), and a lens for controlling the formation of a subject image on the imaging element. It is an apparatus that images a real space and generates a captured image.
- the imaging device 933 may capture a still image or may capture a moving image.
- the sensor 935 is various sensors such as an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, an illuminance sensor, a temperature sensor, an atmospheric pressure sensor, a pressure sensor, a distance sensor, or a sound sensor (microphone).
- the sensor 935 acquires information about the state of the information processing apparatus 900 itself, such as the posture of the information processing apparatus 900, and information about the surrounding environment of the information processing apparatus 900, such as brightness and noise around the information processing apparatus 900, for example. To do.
- the sensor 935 may also include a GNSS receiver that receives a GNSS (Global Navigation Satellite System) signal and measures the latitude, longitude, and altitude of the device.
- GNSS Global Navigation Satellite System
- Each component described above may be configured using a general-purpose member, or may be configured by hardware specialized for the function of each component. Such a configuration can be appropriately changed according to the technical level at the time of implementation.
- an information processing apparatus for example, an information processing apparatus, a system, an information processing method executed by the information processing apparatus or system, a program for causing the information processing apparatus to function, and a program are recorded. It may include tangible media that is not temporary.
- FIG. 24 is a flowchart illustrating processing in the first example of device control included in an embodiment of the present disclosure.
- an action more specifically, a jump
- an image or video capturing the action is executed.
- the action detected in this example is an action that occurs in a series of user states (for example, snowboarding)
- the service control unit 112 includes a first section corresponding to the section in which the action is detected, The control of the device that is different from the second section that is not so is performed. More specifically, the service control unit controls the device so as to record the captured video or image in the first section and not record the video or image in the second section. For example, the service control unit controls the device so as to specify a captured video or image in the first section, and controls the device so as to record the captured video or image in the first section.
- the action detection unit 106 included in the sensor data analysis unit 104 detects a jump crossing (S301).
- the service control unit 112 performs continuous shooting using an imaging device such as a digital video camera paired with the sensor device (S303).
- the continuous shooting may be, for example, continuous still image shooting or video shooting. Shooting continues until the action detection unit 106 detects the landing of the jump (S305).
- the action detection unit 106 executes a jump detection process as shown in FIGS. 2 and 6 in real time, for example.
- the service control unit 112 continuously performs shooting of an action (jump), and controls the imaging device to buffer an image or video shot for a predetermined time (longer than the duration of the jump). .
- the service control unit 112 controls the imaging device so as to record an image or video shot from the start point (crossing) to the end point (landing) of the jump section. .
- the first section corresponding to the section in which the action is detected by the action detection unit 106 does not necessarily exactly match the action section, for example, before and after the action section as in the above example. Or may correspond to the action section, but may be a section shifted on the time axis.
- FIG. 25 is a diagram for further describing an image or video captured in the first example of device control included in an embodiment of the present disclosure.
- still images P1 to P8 are taken between the jump crossing detected by the action detection unit 106 and landing.
- a continuous photograph as shown in the figure can be obtained by synthesizing images taken in this way.
- the service control unit 112 may control the imaging device so that a still image is captured at a user's altitude peak (jump top). More specifically, the service control unit 112 determines the image captured at the top of the jump (still image P5) from the frame images continuously captured by the imaging device at predetermined intervals. A still image may be extracted.
- the altitude information can be extracted from, for example, the atmospheric pressure detected by the atmospheric pressure sensor or information acquired by RTK-GPS (Real Time Kinematic-Global Positioning System). When the atmospheric pressure is used, the top of the jump may be detected as the time when the altitude becomes maximum as described above, or may be detected as the time when the atmospheric pressure starts to increase.
- FIG. 26 is a diagram for describing a second example of device control included in an embodiment of the present disclosure.
- the compression rate of the video capturing the action is controlled based on the detection result of the action (more specifically, jump).
- the action detected in this example is an action that occurs in a series of user states (for example, snowboarding), and the service control unit 112 includes a first section corresponding to the section in which the action is detected, The control of the device that is different from the second section that is not so is performed.
- the service control unit 112 when the action detection unit 106 detects a jump section, the service control unit 112 records video in this section (jump operation) at a different compression rate from the preceding and subsequent sections (normal state). As described above, an imaging apparatus such as a digital video camera paired with the sensor device is controlled. As a result, during the jump operation, the video is recorded at a lower compression rate than in the normal state. By reducing the compression rate only in the jump section, it is possible to improve the image quality of the jump section that can be the highlight of the image while suppressing the data size of the entire image. Furthermore, the service control unit 112 may set different compression ratios for the section of the crossing and landing operations included in the jump section and the section in the stagnant state. In this case, as illustrated, the service control unit 112 may temporarily lower the compression rate in the section between the crossing operation and the landing operation.
- the action detection unit 106 executes a jump detection process as shown in FIGS. 2 and 6 in real time, for example.
- the service control unit 112 continuously performs action (jump) shooting, and buffers a video shot during a predetermined time (longer than the jump duration) in an uncompressed or low-compressed state.
- the imaging device is controlled.
- the service control unit 112 records the video from the start point (crossing) to the end point (landing) of the jump section at a low compression rate, and other sections (normal state)
- the image pickup apparatus is controlled so as to record the video at a high compression rate.
- the service control unit 112 may record a video with a lower compression rate in a section having a predetermined length immediately before the start and immediately before the end of the jump section as a section of the crossing operation and the landing operation.
- the action detection unit 106 may execute a high impact detection process as shown in FIG. 3, for example, and the detected high impact section may be treated as a sectioning operation and landing operation section included in the jump section. .
- FIG. 27 is also a diagram for describing a second example of device control included in an embodiment of the present disclosure.
- the frame rate or size of the video capturing the action is controlled based on the detection result of the action (specifically, jump).
- the service control unit 112 when the action detection unit 106 detects a jump section, the service control unit 112 has a different frame rate (fps) and / or size in this section (jump operation) from the previous and subsequent sections (normal state).
- the image pickup apparatus such as a digital video camera paired with the sensor device is controlled so that the video is recorded at the same time.
- the video is recorded at a frame rate higher than the normal state or a size larger than the normal state.
- the image size of the jump section that can be the highlight of the image is suppressed while suppressing the data size of the entire image. Can be improved.
- the service control unit 112 may set different frame rates or image sizes for the section of the crossing and landing operations included in the jump section and the section in the stagnant state. In this case, as illustrated, the service control unit 112 may temporarily further increase the frame rate or size in the section between the crossing operation and the landing operation.
- the action detection unit 106 executes a jump detection process as shown in FIGS. 2 and 6 in real time, for example.
- the service control unit 112 continuously performs shooting of an action (jump), and takes a video shot for a predetermined time (longer than the duration of the jump) as a maximum frame rate or a maximum (original) size.
- the image pickup apparatus is controlled so as to be buffered.
- the service control unit 112 records a video from the start point (crossing) to the end point (landing) of the jump section at a high frame rate and / or a large size.
- the imaging apparatus is controlled to record the video in the section (normal state) at a low frame rate and / or a small size. Furthermore, the service control unit 112 records a video with a higher frame rate and / or a larger size in a section having a predetermined length immediately before the start of the jump section and immediately before the end as a section of a crossing operation and a landing operation. You may let them.
- the action detection unit 106 may execute a high impact detection process as shown in FIG. 3, and the high impact section detected there may be handled as a section of the crossing action and landing action included in the jump section.
- the service control unit 112 may control the sound collection level of the audio included in the video in the same manner as the compression rate, frame rate, and size control as described above.
- the crossing and landing sections included in the jump section and the service control unit 112 is not limited to the jump section but also the turn section detected by the turn detection process as shown in FIG. Similar control may be performed. Table 1 below shows an example of the detected action section and the control corresponding to the detected action section.
- the service control unit 112 may control the video quality based on the action score calculated by the scoring processing unit 109 included in the analysis result processing unit 107. More specifically, for example, the service control unit 112 may set a lower compression rate, a higher frame rate, a larger image size, and / or a larger sound collection level for an action with a higher action score. At this time, for example, when the action score changes sequentially within the action section, the quality of the video may be changed according to the action score even within the action section. Alternatively, even in such a case, the video quality may be constant within the same action section.
- FIG. 28 is a diagram for describing a third example of device control included in an embodiment of the present disclosure.
- recording of sensor data and the sensor device itself are controlled based on the detection result of the action (specifically, jump).
- the action detected in this example is an action that occurs in a series of states of the user (for example, snowboarding), and the sensor device control unit 103 detects the sensor device when the action detection unit 106 detects a jump section.
- control in this section jump operation
- the sensor device control unit 103 records the data such as acceleration and angular velocity included in the sensor data together with the analysis result after the analysis by the sensor data analysis unit 104 during the jump operation.
- control is performed so that data is discarded after analysis.
- the sensor device control unit 103 performs jump operation when recording data such as acceleration and angular velocity included in the sensor data together with the analysis result after analysis by the sensor data analysis unit 104. It may be controlled to record data at a high sampling rate while recording data at a low sampling rate in a normal state.
- the sensor device control unit 103 records data at different sampling rates in the sectioning operation and landing operation section included in the jump section and the section in the stagnant state. Also good. More specifically, the sensor device control unit 103 may record data at a higher sampling rate in the section between the crossing operation and the landing operation.
- acceleration data and angular velocity data included in the sensor data are used for analysis in the sensor data analysis unit 104.
- the waveform can be displayed based on the sensor data when the action image is reproduced, or the analysis can be performed again afterwards.
- the size of the sensor data increases. Therefore, in this example, it is limited to sensor data acquired in an action section (not limited to the jump section described above but may be a turn section or other sections) that is likely to be used for information display or reanalysis.
- the size of the sensor data is suppressed by reducing the sampling data of the sensor data recorded in the action section and other sections.
- the sensor device control unit 103 sets the sleep mode in the normal state for the GPS receiver or the like mounted on the sensor device, and activates it during the jump operation.
- the sensor device may be controlled.
- the position information acquired during activation of the GPS receiver is provided to the analysis result processing unit 107 or the service control unit 112 via the sensor device control unit 103.
- the position information may be provided to the sensor data analysis unit 104 and stored in the detection section information holding unit 110 together with the detection result of the action section.
- the sensor device control unit 103 transmits a control signal to the sensor device so as to start a sensor such as a GPS receiver when the jump section is detected, and then, more specifically, for a predetermined time, more specifically, For example, the control signal may be transmitted to the sensor device so as to set the sensor to the sleep mode again after starting the sensor for a time sufficient to obtain accurate position information.
- the section in which the sensor is activated after the detection of the jump section is a section corresponding to the section in which the action is detected (the jump section and the section in which the sensor is activated do not necessarily overlap).
- the analysis result processing unit 107 when the analysis result processing unit 107 generates additional information regarding an action (such as action score and cluster information), the position where the action has occurred is detailed (for example, the user's trajectory in the action section can be drawn). ) It is not always necessary to specify, for example, it is often sufficient that the location can be specified to such an extent that the facility where the action has occurred can be specified. In such a case, as described above, sufficient information for generating information can be obtained even if the sensor is activated after detecting the action.
- additional information regarding an action such as action score and cluster information
- the position information is not necessarily used for analysis in the sensor data analysis unit 104, but is used for generation of additional information in the analysis result processing unit 107 and information generation in the service control unit 112.
- the analysis result processing unit 107 and the service control unit 112 generate information about an action section such as a jump section in the illustrated example. Therefore, for sensor data that is not directly involved in action detection, such as position information, the sensor itself that provides such sensor data is activated only in the action section, for example, consumption of the sensor device. You may save power.
- the sensor data provided continuously is recorded only in the action section, or the sensor data is continuously recorded, as in the above example (a) or (b). In doing so, control for increasing the sampling rate only in the action section may be performed.
- the type of sensor data to be controlled as in (a), (b), or (c) is not limited to the above example, and the type of sensor data provided from the sensor device or the detection of action. May differ depending on the type of sensor data used for the sensor, the type of sensor data used for generating additional information, and the like.
- the acceleration or angular velocity may be controlled as described in (c) above.
- the control target can include all other types of sensor data such as atmospheric pressure and geomagnetism.
- FIG. 29 is a diagram for describing a fourth example of device control included in an embodiment of the present disclosure.
- appropriate device control is realized by comparing the latest history of action detection with the past history.
- the action detection unit 106 included in the sensor data analysis unit 104 detects the latest action history R including stay, drive, walking, and the like.
- video recording is started by an imaging device such as a digital camera paired with the information processing device 100.
- the service control unit 112 searches the detected section information holding unit 110 for past action histories similar to the latest action history R.
- the search is executed using conditions such as, for example, an action history before the start of video recording, where walking is detected after repeated stay and drive.
- the similarity between action histories may be calculated using an algorithm such as an HMM (Hidden Markov Model) or a k-NN (k-Nearest Neighbor) method.
- HMM Hidden Markov Model
- k-NN k-Nearest Neighbor
- the service control unit 112 reads the setting of the imaging device when the video is recorded with the past action history P1 (the setting information is stored in the detection section information holding unit 110 together with the information of the detected action section, for example). The same setting is applied to the imaging apparatus that has started recording video at the current time (Now). More specifically, the service control unit 112 controls the imaging device according to the read setting information.
- the latest action history R and the past action history P1 can both be action histories when hiking outdoors.
- the appropriate setting of the imaging device may be different from the setting that was appropriate when the user spent indoors, for example, when the illustrated past action history P2 (mainly staying) occurred.
- the service control unit 112 detects such a difference in the situation in advance based on the action history and automatically changes the setting of the imaging device, so that the user records video with appropriate settings from the beginning. Make it possible to do.
- (7-5. Fifth example) 30 and 31 are diagrams for describing a fifth example of device control included in an embodiment of the present disclosure.
- display of an image on an HMD (Head Mounted Display) worn by a user wearing a snowboard is controlled based on the action detection result.
- HMD Head Mounted Display
- a communication window 1101 and facility navigation 1103 are displayed. These images are transparently superimposed on the real space image visually recognized by the user on the screen 1100a.
- a guide 1105 of equipment in the course (kicker in the illustrated example) and course navigation 1107 are displayed.
- the service control unit 112 selectively displays, for example, the screens 1100a and 1100b as described above on the HMD based on the detection result of the user action by the action detection unit 106 included in the sensor data analysis unit 104. . More specifically, for example, when the action detection unit 106 continuously detects a jump section or a turn section as described above, the service control unit 112 is in a state where the user is sliding on a snowboard (attention is required) The information related to the situation of the user who is running, such as the screen 1100b, is presented.
- the service control unit 112 determines that the user is not running (moving or resting), and the image Information such as 1100a is presented.
- the service control unit 112 may provide information suitable for sliding during a run as in the above example, or may not display an image on the HMD during the run and may substantially turn off the display. .
- the service control unit 112 may notify the user of information that is not provided by switching the display of the HMD during the slide, by voice or vibration.
- the service control unit 112 may control the presentation of information by various devices, not limited to the HMD, based on the action detection result. For example, when it is determined that the user is running based on the detection result of the action, the service control unit 112 may suppress the incoming call notification by the voice or vibration of the user's mobile phone. Note that the same control may be possible when it is estimated that the user is concentrating on the action based on the detection result of the action, not only during the sliding.
- FIG. 32 is a diagram for conceptually explaining a sixth example of device control included in an embodiment of the present disclosure.
- a camera 161 installed in the facility acquires an image of a user 143 sliding on the snowboard.
- the camera 161 includes a plurality of cameras 161a to 161e.
- the cameras 161a to 161e may include a camera installed in a facility, or may include a smartphone or a digital camera owned by a user or his friend. By combining the images respectively acquired by these cameras, a free viewpoint image of the user who is executing the action can be obtained.
- FIG. 33 is a flowchart illustrating processing in a sixth example of information output included in an embodiment of the present disclosure.
- the sensor device control unit 103 of the information processing apparatus 100 uses a device (such as a digital camera or a smartphone; hereinafter also referred to as a camera device) used as the camera 161 illustrated in FIG. Is paired (S361).
- the pairing is performed by communication means such as Bluetooth (registered trademark) or wireless LAN.
- the camera device may be arranged toward the planned point where the action is executed at the time of pairing, or may be arranged toward the planned point after the pairing until the action is executed. Good.
- the sensor device control unit 103 After the pairing is completed, for example, at the timing when the sensor device is activated, the sensor device control unit 103 transmits a synchronization signal to each camera device (S362), and the time between the sensor device and each camera device (sensor data or Synchronize the time stamp given to the video data. After the synchronization is completed, the sensor device control unit 103 transmits a shooting start signal to each camera device (S363). The camera device that has received the shooting start signal starts acquiring video data by shooting.
- the action detection unit 106 detects an action section based on sensor data provided by the sensor device during shooting (S364), for example, action information is recorded in the detection section information holding unit 110 (S365).
- the end of shooting may be determined by, for example, a user operation, or the sensor device control unit 103 automatically ends shooting when the action interval is not detected for a predetermined time or when the sensor device is turned off. You may decide.
- the sensor device control unit 103 transmits a shooting end signal to each camera device (S366). After the end of shooting, each camera device uploads the acquired video data to the information processing apparatus 100 (S367).
- the service control unit 112 of the information processing device 100 After uploading in S367, the service control unit 112 of the information processing device 100, according to the information of the action section detected by the action detection unit 106, the first section corresponding to the action section (for example, may include before and after the action section). Is cut out (S368). Further, the service control unit 112 generates a free viewpoint video for the first section using video data provided from each camera device (S369). More specifically, for example, the service control unit 112 generates point cloud data from video data provided from each camera device. There are many methods for providing a free viewpoint video other than the method using point cloud data, and the service control unit 112 may generate data for providing a free viewpoint video by such a method. A technique for providing a free viewpoint video is described in, for example, Japanese Patent Application Laid-Open No. 2007-133660.
- FIG. 34 is a diagram for describing a user interface for browsing the free viewpoint video generated by the processing shown in FIG.
- a free viewpoint video 1630 is displayed on the display 149 of the terminal device 148 (a mobile device such as a tablet or a smartphone in the illustrated example).
- the user changes the viewpoint of the free viewpoint video by an operation input via a touch panel or the like provided on the display 149 and causes the display 149 to display videos such as the video 1630a to video 1630e shown in the figure. Is possible.
- the three-dimensional object data of the action section may be provided instead of or together with the free viewpoint video.
- the process until the service control unit 112 of the information processing apparatus 100 cuts out the video of the action section from the video provided from each camera device is the same as S361 to S368 of the process shown in FIG. is there.
- the generation of three-dimensional object data is similar to the generation of a free viewpoint video in that a three-dimensional space including objects is modeled. Therefore, the three-dimensional object data is based on the video acquired by the plurality of camera devices (camera 161 shown in FIG. 32) as described above, the depth information in the video additionally measured by the camera device, and the like. Can be generated.
- 3D object data at a time specified by the user in the action section may be transmitted to the user's terminal device as it is.
- three-dimensional object data at a specified time may be output using a three-dimensional printer, and the output object may be provided to the user (distributed on the spot or sent).
- a user profile can be created according to the types of actions detected in the past. For example, a profile of “advanced jump expert” is given to a user whose high action score is stably calculated in the jump action. For a user who has a low jump action score and often detects a fall during the jump, a profile of “beginner of jump” is given.
- the device control process in the present embodiment may be made different using the profile as described above.
- the user profile may be reflected in determining whether or not to record sensor data and the recording method. More specifically, when the user is an advanced jumper, the sensor data of a jump in which a rare fall is detected may be recorded in detail at a higher sampling rate. This is because it is natural for advanced players to succeed in jumping, and detailed analysis of rare failures will help them to improve further. Conversely, if the user is a beginner in jumping, the sensor data of a successful jump in which no fall is detected may be recorded in detail at a higher sampling rate. This is because beginners may find hints that enable them to succeed in a stable jump by looking back at a successful jump.
- FIG. 35 is a block diagram illustrating a hardware configuration example of the information processing apparatus according to the embodiment of the present disclosure.
- the information processing apparatus 900 includes a CPU (Central Processing unit) 901, a ROM (Read Only Memory) 903, and a RAM (Random Access Memory) 905.
- the information processing apparatus 900 may include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925.
- the information processing apparatus 900 may include an imaging device 933 and a sensor 935 as necessary.
- the information processing apparatus 900 may include a processing circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array) instead of or in addition to the CPU 901.
- DSP Digital Signal Processor
- ASIC Application Specific Integrated Circuit
- FPGA Field-Programmable Gate Array
- the CPU 901 functions as an arithmetic processing device and a control device, and controls all or a part of the operation in the information processing device 900 according to various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927.
- the ROM 903 stores programs and calculation parameters used by the CPU 901.
- the RAM 905 primarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like.
- the CPU 901, the ROM 903, and the RAM 905 are connected to each other by a host bus 907 configured by an internal bus such as a CPU bus. Further, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.
- PCI Peripheral Component Interconnect / Interface
- the input device 915 is a device operated by the user, such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever.
- the input device 915 may be, for example, a remote control device that uses infrared rays or other radio waves, or may be an external connection device 929 such as a mobile phone that supports the operation of the information processing device 900.
- the input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the input signal to the CPU 901. The user operates the input device 915 to input various data and instruct processing operations to the information processing device 900.
- the output device 917 is configured by a device capable of notifying the acquired information to the user using a sense such as vision, hearing, or touch.
- the output device 917 can be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, an audio output device such as a speaker or headphones, or a vibrator.
- the output device 917 outputs the result obtained by the processing of the information processing device 900 as video such as text or image, sound such as sound or sound, or vibration.
- the storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900.
- the storage device 919 includes, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device.
- the storage device 919 stores, for example, programs executed by the CPU 901 and various data, and various data acquired from the outside.
- the drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built in or externally attached to the information processing apparatus 900.
- the drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905.
- the drive 921 writes a record in the attached removable recording medium 927.
- the connection port 923 is a port for connecting a device to the information processing apparatus 900.
- the connection port 923 can be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, or the like.
- the connection port 923 may be an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, or the like.
- the communication device 925 is a communication interface configured with, for example, a communication device for connecting to the communication network 931.
- the communication device 925 can be, for example, a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), NFC (Near Field Communication), or a communication card for WUSB (Wireless USB).
- the communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communication.
- the communication device 925 transmits and receives signals and the like using a predetermined protocol such as TCP / IP with the Internet and other communication devices, for example.
- the communication network 931 connected to the communication device 925 is a network connected by wire or wireless, and may include, for example, the Internet, a home LAN, infrared communication, radio wave communication, satellite communication, or the like.
- the imaging device 933 uses various members such as an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device), and a lens for controlling the formation of a subject image on the imaging element. It is an apparatus that images a real space and generates a captured image.
- the imaging device 933 may capture a still image or may capture a moving image.
- the sensor 935 is various sensors such as an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, an illuminance sensor, a temperature sensor, an atmospheric pressure sensor, a pressure sensor, a distance sensor, or a sound sensor (microphone).
- the sensor 935 acquires information about the state of the information processing apparatus 900 itself, such as the posture of the information processing apparatus 900, and information about the surrounding environment of the information processing apparatus 900, such as brightness and noise around the information processing apparatus 900, for example. To do.
- the sensor 935 may also include a GNSS receiver that receives a GNSS (Global Navigation Satellite System) signal and measures the latitude, longitude, and altitude of the device.
- GNSS Global Navigation Satellite System
- Each component described above may be configured using a general-purpose member, or may be configured by hardware specialized for the function of each component. Such a configuration can be appropriately changed according to the technical level at the time of implementation.
- an information processing apparatus for example, an information processing apparatus, a system, an information processing method executed by the information processing apparatus or system, a program for causing the information processing apparatus to function, and a program are recorded. It may include tangible media that is not temporary.
- An information acquisition unit that acquires user action information, spatial information associated with the action information, and time information;
- An information processing apparatus comprising: a map expression processing unit that considers the time information when executing map expression of the action information based on the spatial information.
- the map expression processing unit considers a relationship between a time indicated by the time information and a reference time.
- the map expression processing unit reduces the influence of the action information in the map expression as the time indicated by the time information is farther from the reference time.
- the action information includes a score of each category when the user action is classified into any of a plurality of categories
- the map expression processing unit adds the scores included in the action information associated with the common spatial information with a weight that becomes smaller as the time indicated by the time information becomes farther from the reference time.
- the information processing apparatus according to (3), wherein a category of a representative action associated with the common spatial information in the map representation is determined.
- the map expression processing unit executes the map expression based on the action information associated with the time information indicating a time at which a difference from the reference time falls below a threshold value.
- the information processing apparatus increases an influence of the action information in the map expression when the time information has an attribute common to the reference time.
- the action information includes first action information associated with the first time information and second action information associated with the second time information,
- the map expression processing unit when executing the map expression of at least one of the first action information and the second action information, the time indicated by the first time information and the second time information.
- the first action information is associated with the first spatial information
- the second action information is associated with second spatial information different from the first spatial information
- the map expression processing unit starts the second space from a position indicated by the first space information in the map expression.
- the information processing apparatus according to (7), wherein movement to a position indicated by information is expressed.
- the action information includes information on an action score for evaluating the action,
- the information processing apparatus according to any one of (1) to (8), wherein the map expression processing unit further considers information related to the score of the action when executing the map expression.
- the action information includes information related to the skill level of the user estimated based on the score, The information processing apparatus according to (9), wherein the map expression processing unit further considers information on the skill level when executing the map expression.
- the action information includes information regarding a score indicating a negative evaluation of the user's action, The information processing apparatus according to (10), wherein the map expression processing unit corrects the score upward as the skill level is higher for the negative evaluation.
- the action information includes information on a score indicating a positive evaluation of the user's action, The information processing apparatus according to (10) or (11), wherein the map expression processing unit corrects the score downward as the skill level is higher for the positive evaluation.
- the map expression processing unit identifies a facility where an action indicated by the action information is executed based on the spatial information, and performs evaluation for each facility according to a skill level of the user.
- the information processing apparatus according to any one of (10) to (12), wherein information including the information is generated.
- the map expression processing unit is configured such that the distance between the position indicated by the first spatial information associated with the action information and the position indicated by the second spatial information not associated with the action information.
- the information processing apparatus according to any one of (1) to (13), wherein action information associated with the second spatial information is estimated based on the information.
- the action information includes a first action information sequence associated with the first sequence of the spatial information, and a second action information sequence associated with the second sequence of the spatial information,
- the second action information series is associated with time information indicating a time before the first action information series
- the map representation processing unit when a first movement path indicated by the first series of spatial information partially matches a second movement path indicated by the second series of spatial information, Navigation to a user associated with the first action information sequence based on the second action information sequence associated with the second sequence of spatial information indicating the remaining portion of the second travel route.
- the action information includes information indicating the action detected by the user,
- the map expression processing unit adjusts a setting for detecting the user's action based on a tendency of the detected action for each position indicated by the spatial information indicated by the map expression result;
- the information processing apparatus according to any one of (15).
- (17) acquiring user action information, and spatial information and time information associated with the action information;
- An information processing method comprising: a processor considering the time information when executing a map representation of the action information based on the spatial information.
- a sensor data receiving unit that receives sensor data provided by a user or a sensor mounted on an instrument used by the user;
- An action detection unit that detects an action of the user based on the sensor data;
- An information processing apparatus comprising: a device control unit that controls a device related to the user or the action according to the detected action.
- the action detection unit detects the action that occurs in a series of states of the user.
- the device control unit controls the device different between a first section corresponding to a section in which the action is detected and a second section in which the action is not detected.
- the information processing apparatus according to (2) which is executed.
- the device has a function of recording a video or an image of the action, The information processing apparatus according to (3), wherein the device control unit controls the device to specify the video or the image in the first section.
- the sensor data includes data indicating the altitude of the user,
- the action includes a jump,
- the device has a function of recording a video or an image of the action, The device control unit records the video or the image with a first quality in the first section, and the second quality lower than the first quality in the second section with the video or the image.
- the information processing apparatus according to (3), wherein the device is controlled so as to perform recording.
- the first quality includes a first compression rate, a first frame rate, a first frame size, or a first sound collection level
- the second quality includes a second compression rate higher than the first compression rate, a second frame rate lower than the first frame rate, and a second frame smaller than the first frame size.
- the information processing apparatus according to (6), including a size or a second sound collection level lower than the first sound collection level.
- the device has a function of recording the sensor data, The information processing apparatus according to any one of (3) to (7), wherein the device control unit controls the device to record the sensor data in the first section.
- the device has a function of recording the sensor data, The device control unit records the sensor data at a first sampling rate in the first section, and records the sensor data at a second sampling rate lower than the first sampling rate in the second section.
- the information processing apparatus according to any one of (3) to (7), wherein the device is controlled to record.
- the device has a function of providing additional sensor data used for generating information on the action, The information processing apparatus according to any one of (3) to (9), wherein the device control unit controls the device so that the additional sensor data is provided in the first section.
- the device control unit compares the latest history of the action detected until the device is newly activated with the past history of the action detected until the device is activated in the past.
- the device has a function of presenting information to the user, The actions include specific types of actions that occur in situations where attention is required of the user, The device control unit controls the device so that the content or the presentation method of the information changes when the specific type of action is detected, any one of (1) to (12) The information processing apparatus described in 1.
- the information processing apparatus controls the device so that the information presented by the image is presented or not presented by voice or vibration.
- the device has a function of processing a video or an image of the action, The device control unit according to any one of (1) to (15), wherein the device control unit controls the device to process the video or image in a section corresponding to a section in which the action is detected.
- Information processing device 17.
- the processing of the video or the image by the device includes generating data for providing a free viewpoint video or a free viewpoint image or three-dimensional object data based on the video or the image.
- the information processing apparatus according to (16).
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Remote Sensing (AREA)
- Business, Economics & Management (AREA)
- Radar, Positioning & Navigation (AREA)
- Educational Administration (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Educational Technology (AREA)
- Data Mining & Analysis (AREA)
- Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- Mathematical Physics (AREA)
- Human Resources & Organizations (AREA)
- Strategic Management (AREA)
- Automation & Control Theory (AREA)
- Game Theory and Decision Science (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Development Economics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- User Interface Of Digital Computer (AREA)
- Navigation (AREA)
Abstract
Description
1.情報処理装置の機能構成
2.アクション検出処理の例
2-1.ジャンプの検出-1
2-2.ジャンプの検出-2
2-3.ターンの検出
3.付加的な処理の例
3-1.アクションスコアの算出
3-2.クラスタリング処理
3-3.センサ装着状態の推定
4.マップ表現の例
4-1.マップ表現の例
4-2.安全マップの生成
4-3.ナビゲーションの提供
4-4.アクション検出設定の変更
4-5.施設レベルの算出
5.ハードウェア構成
6.補足
7.機器制御の例
7-1.第1の例
7-2.第2の例
7-3.第3の例
7-4.第4の例
7-5.第5の例
7-6.第6の例
7-7.ユーザのプロファイルについて
8.ハードウェア構成
9.補足
図1は、本開示の一実施形態に係る情報処理装置の概略的な機能構成を示すブロック図である。図1を参照すると、情報処理装置100は、送信部101と、受信部102と、センサデバイス制御部103と、センサデータ解析部104と、解析結果処理部107と、検出区間情報保持部110と、付加的情報保持部111と、サービス制御部112とを含む。
以下では、本開示の一実施形態において実行されるアクション検出処理の例について説明する。これらの例では、ユーザがスノーボードをしている場合に発生するジャンプおよびターンが検出される。例えば、スノーボードの場合、加速度センサおよび角速度センサなどを含むセンサデバイスは、ウェアに埋め込まれたり、ウェアラブル端末装置やモバイル端末装置に組み込まれたりすることによって、ユーザに直接的に装着されてもよい。あるいは、センサデバイスは、スノーボードの用具、例えばボードに装着されてもよい。
図2は、本開示の一実施形態において、ユーザのアクションに含まれるジャンプを検出するための処理の第1の例を示すフローチャートである。図示された処理は、例えば上記の情報処理装置100に含まれるセンサデータ解析部104において実行される。
図6は、本開示の一実施形態において、ユーザのアクションに含まれるジャンプを検出するための処理の第2の例を示すフローチャートである。図示された処理は、上記の第1の例と同様に、例えば情報処理装置100に含まれるセンサデータ解析部104において実行される。
図10は、本開示の一実施形態において、ユーザのアクションに含まれるターン区間を検出するための処理の例を示すフローチャートである。図示された処理は、例えば上記の情報処理装置100に含まれるセンサデータ解析部104において実行される。以下の処理では、センサデータ解析部104が、ユーザのアクションに含まれる回転を検出し(S210)、さらにその回転に含まれる非ターン性の回転を検出し(S230)、回転から非ターン性の回転を除いたものの中からターンを検出する(S250)。
ここで、非ターン性の回転は、例えば、センサがユーザの頭部、またはユーザの頭部に装着される器具に装着されるセンサを含む場合に、ユーザの首振りによって発生する回転を含む。非ターン性の回転は、他にも、ユーザの体動によって発生する回転、より具体的には、センサがユーザの腕部、またはユーザの腕部に装着される器具に装着されるセンサを含む場合に、ユーザの腕振りや腕回しによって発生する回転などを含みうる。
本実施形態では、センサデータ解析部104が、このような非ターン性の回転を除いた上でターン区間を検出することで、より精度の高いターン区間の検出が可能になる。そのような意味で、非ターン性の回転は、検出対象であるターンに対するノイズであるともいえ、本実施形態において、センサデータ解析部104は、ユーザのアクションに含まれる回転を検出し、さらにその回転に含まれるノイズを検出し、回転からノイズを除いたものの中からターンを検出するともいえる。
以上、本開示の一実施形態において実行されるアクション検出処理のいくつかの例について説明した。既に説明した通り、本実施形態で実行されるアクション検出処理はスノーボードで発生するジャンプやターンには限られず、例えばスノーボード以外のスポーツ、またはスポーツ以外のシーンで発生するジャンプやターンについてアクション検出処理が実行されてもよい。また、本実施形態で実行されるアクション検出処理では、ジャンプやターン以外のアクションが検出されてもよい。一例として、アクション検出部106は、スノーボードなどで発生する転倒を検出してもよい。この場合、特徴量抽出部105が上述したジャンプやターンの検出と同様にして加速度のノルムを算出し、アクション検出部106が、加速度のノルムが閾値(例えば、通常の滑走では発生しない程度の大きな値でありうる)を上回った場合に、転倒の発生を検出してもよい。
(3-1.アクションスコアの算出)
例えば、解析結果処理部107に含まれるスコアリング処理部109は、上記で図2~図13を参照して説明したような処理によって検出されるジャンプ区間および/またはターン区間を含むアクション区間について、発生したアクションを評価するスコア(アクションスコア)を算出する。アクションスコアは、例えばアクション区間におけるセンサデータから、アクションの良し悪しや特徴を表す物理量(特徴量)を抽出し、それらを重みづけ加算することによって算出されうる。サービス制御部112は、このようにして算出されたスコアに基づいて、アクション(例えばジャンプまたはターン)に関する情報を生成する。
また、例えば、解析結果処理部107に含まれるクラスタリング処理部108は、上記で図2~図13を参照して説明したような処理によって検出されるジャンプ区間および/またはターン区間を含むアクション区間について、スコアリングのために抽出した特徴量などを利用してk-means法などのクラスタリングアルゴリズムを適用し、検出されたアクションをクラスタに分類する。ジャンプ区間やターン区間の場合、例えば、区間の持続時間の長短や、回転の大小によってアクションがクラスタに分類されてもよい。クラスタリングの結果は、例えば、サービスとしてダイジェスト動画を提供するような場合に、様々な種類のジャンプやターンなどのアクションが動画に含まれるようにアクション区間を抽出するために利用される。また、良かったアクションとそうでなかったアクションとを別々のクラスタに分類することによって、ユーザ自身がアクションを振り返ったり、アクションの改善のためのコーチングに役立てたりしてもよい。
図14は、本開示の一実施形態において、センサ装着状態を推定するための処理の例を示すブロック図である。より具体的には、図示された構成によって、センサデータを提供するセンサがユーザの身体に直接的に装着されているか、ユーザによって使用される器具に装着されているかが判定される。図示された処理は、例えば上記の情報処理装置100に含まれるセンサデータ解析部104において実行される。なお、図示された例では、フィルタの遮断周波数(Fc)やタイムフレームの長さが具体的に説明されているが、これらの数値は一例であり、実際のセンサの特性などに応じて適宜変更されうる。
ここで、再び図1を参照して、本実施形態に係る情報処理装置のマップ表現に関する構成について説明する。本実施形態では、情報処理装置100において、ユーザのアクション情報、およびアクション情報に関連付けられる空間情報ならびに時間情報を取得する情報取得部と、アクション情報を空間情報に基づいてマップ表現するにあたり時間情報を考慮するマップ表現処理部とが実現される。
例えば、サービス制御部112は、場所DBにおいて空間情報に関連付けられたアクション情報を利用して、拡張されたマップ情報を作成する(S306)。さらに、サービス制御部112は、例えばユーザのリクエストに応じて、作成されたマップを描画する(S307)。マップでは、単一または複数のユーザについてのアクション情報に基づいて、それぞれの場所において最も高い頻度で発生しているアクションが場所の特性として表示される。ここで、サービス制御部112は、拡張されたマップ情報の作成にあたり、アクション情報に関連付けられた時間情報を考慮する。
再び図15を参照して、例えば、サービス制御部112は、場所DBにおいて空間情報に関連付けられたアクション情報を利用して、場所ごとの安全度および/または危険度を算出する(S308)。さらに、サービス制御部112は、算出された安全度および/または危険度に基づいて安全マップDBを更新し(S309)、さらに例えばユーザのリクエストに応じて、安全マップDBの内容に基づく安全マップを描画する(S310)。
再び図15を参照して、例えば、サービス制御部112は、場所DBにおいて空間情報に関連付けられたアクション情報を利用して、他のユーザの過去のアクション履歴に基づくナビゲーションを提供してもよい。より具体的には、サービス制御部112は、ある時点までに検出された第1のユーザのアクション履歴について、他のユーザ(第2のユーザ)の過去のアクション履歴との空間情報に基づくマッチングを実施する(S311)。例えば、このマッチングでは、第1のユーザのある時点までの移動ルートと同様の移動ルートと共に検出された第2のユーザのアクション履歴が抽出されうる。このとき、アクション履歴は、関連付けられたユーザのスキルレベルを考慮して抽出されてもよい。また、アクション履歴は、過去に検出された第1のユーザ自身のアクション履歴を考慮して抽出されてもよい。サービス制御部112は、第1のユーザのアクション履歴と第2のユーザのアクション履歴とを、共起確率と遷移確率とを考慮してマッチングしてもよい。サービス制御部112は、抽出されたアクション履歴に基づいてルートを計算し(S312)、第1のユーザにナビゲーションを提示する(S313)。
再び図15を参照して、例えば、サービス制御部112は、場所DBにおいて空間情報に関連付けられたアクション情報を利用して、アクション検出のための設定を変更してもよい。より具体的には、サービス制御部112は、場所DBに格納された情報から、空間情報によって示される位置ごとのアクション検出傾向を抽出する(S314)。抽出されたアクション検出傾向に応じて、サービス制御部112は、アクションを検出するための設定を調整する。より具体的には、サービス制御部112は、例えばセンサデータ解析部104に含まれるアクション検出部106が参照するパラメータテーブルを更新する(S315)。
再び図15を参照して、例えば、サービス制御部112は、S303でアップロードされたデータに含まれる位置情報から、アクションが検出された施設のIDを検索し(S316)、当該施設において検出されたアクションのアクションスコアと、それぞれのアクションを実行したユーザのスキルレベルとの分布に基づいて施設レベルを算出し(S317)、施設レベルの情報を含む施設データを提供する(S318)。
次に、図23を参照して、本開示の実施形態に係る情報処理装置のハードウェア構成について説明する。図23は、本開示の実施形態に係る情報処理装置のハードウェア構成例を示すブロック図である。
本開示の実施形態は、例えば、上記で説明したような情報処理装置、システム、情報処理装置またはシステムで実行される情報処理方法、情報処理装置を機能させるためのプログラム、およびプログラムが記録された一時的でない有形の媒体を含みうる。
以下、本開示の一実施形態に含まれる機器制御のいくつかの例について説明する。
図24は、本開示の一実施形態に含まれる機器制御の第1の例における処理を示すフローチャートである。本例では、アクション(より具体的にはジャンプ)の検出結果に基づいて、アクションを捉えた画像または映像の撮影が実行される。本例で検出されるアクションは、ユーザの一連の状態(例えばスノーボードの滑走)の中で発生するアクションであり、サービス制御部112は、アクションが検出されている区間に対応する第1の区間と、そうでない第2の区間との間で異なる機器の制御を実施する。より具体的には、サービス制御部は、第1の区間では撮影された映像または画像を記録し、第2の区間では映像または画像を記録しないように機器を制御する。サービス制御部は、例えば、第1の区間では撮影された映像または画像を特定するように機器を制御して、第1の区間では撮影された映像または画像を記録するように機器を制御する。
図26は、本開示の一実施形態に含まれる機器制御の第2の例について説明するための図である。図示された例では、アクション(より具体的にはジャンプ)の検出結果に基づいて、アクションを捉えた映像の圧縮率が制御される。本例で検出されるアクションは、ユーザの一連の状態(例えばスノーボードの滑走)の中で発生するアクションであり、サービス制御部112は、アクションが検出されている区間に対応する第1の区間と、そうでない第2の区間との間で異なる機器の制御を実施する。
図28は、本開示の一実施形態に含まれる機器制御の第3の例について説明するための図である。図示された例では、アクション(具体的にはジャンプ)の検出結果に基づいて、センサデータの記録や、センサデバイスそのものが制御される。本例で検出されるアクションは、ユーザの一連の状態(例えばスノーボードの滑走)の中で発生するアクションであり、センサデバイス制御部103は、アクション検出部106がジャンプ区間を検出した場合、センサデバイスによって提供されるセンサデータ、またはセンサデバイスの制御について、この区間(ジャンプ動作)ではその前後の区間(通常状態)とは異なる制御を実施する。
図29は、本開示の一実施形態に含まれる機器制御の第4の例について説明するための図である。図示された例では、アクション検出の直近の履歴を過去の履歴と比較することによって、適切な機器制御が実現される。
図30および図31は、本開示の一実施形態に含まれる機器制御の第5の例について説明するための図である。図示された例では、アクションの検出結果に基づいて、スノーボードをしているユーザが装着しているHMD(Head Mounted Display)における画像の表示が制御される。
図32は、本開示の一実施形態に含まれる機器制御の第6の例を概念的に説明するための図である。図32を参照すると、スノーボードのコース142(施設の例)において、施設に設置されたカメラ161が、スノーボードで滑走しているユーザ143の映像を取得している。本例では、カメラ161が、複数のカメラ161a~161eを含む。カメラ161a~161eは、施設に設置されたカメラを含んでもよいし、ユーザやその友人などが所持しているスマートフォンやデジタルカメラなどを含んでもよい。これらのカメラによってそれぞれ取得された映像を組み合わせれば、アクションを実行中のユーザの自由視点映像が得られる。
上述したような本開示の一実施形態では、過去に検出されたアクションの種類によって、ユーザのプロファイルを作成することができる。例えば、ジャンプのアクションで、高いアクションスコアが安定的に算出されているユーザについては、「ジャンプの上級者」というプロファイルが与えられる。また、ジャンプのアクションスコアが低く、またジャンプの途中で転倒が検出されることが多いユーザについては、「ジャンプの初級者」というプロファイルが与えられる。
次に、図35を参照して、本開示の実施形態に係る情報処理装置のハードウェア構成について説明する。図35は、本開示の実施形態に係る情報処理装置のハードウェア構成例を示すブロック図である。
本開示の実施形態は、例えば、上記で説明したような情報処理装置、システム、情報処理装置またはシステムで実行される情報処理方法、情報処理装置を機能させるためのプログラム、およびプログラムが記録された一時的でない有形の媒体を含みうる。
(1)ユーザのアクション情報、および前記アクション情報に関連付けられる空間情報ならびに時間情報を取得する情報取得部と、
前記空間情報に基づく前記アクション情報のマップ表現を実行するにあたり、前記時間情報を考慮するマップ表現処理部と
を備える情報処理装置。
(2)前記マップ表現処理部は、前記時間情報によって示される時刻と基準時刻との関係を考慮する、前記(1)に記載の情報処理装置。
(3)前記マップ表現処理部は、前記時間情報によって示される時刻が前記基準時刻から離れるほど、前記マップ表現における前記アクション情報の影響を小さくする、前記(2)に記載の情報処理装置。
(4)前記アクション情報は、前記ユーザのアクションを複数のカテゴリのいずれかに分類するときの前記各カテゴリのスコアを含み、
前記マップ表現処理部は、共通の前記空間情報に関連付けられた前記アクション情報に含まれる前記スコアに前記時間情報によって示される時刻が前記基準時刻から離れるほど小さくなる重みをつけて足し合わせることによって、前記マップ表現において前記共通の空間情報に関連付けられる代表アクションのカテゴリを決定する、前記(3)に記載の情報処理装置。
(5)前記マップ表現処理部は、前記基準時刻との差が閾値を下回る時刻を示す前記時間情報に関連付けられた前記アクション情報に基づいて前記マップ表現を実行する、前記(3)に記載の情報処理装置。
(6)前記マップ表現処理部は、前記時間情報が前記基準時刻と共通する属性を有する場合に、前記マップ表現における前記アクション情報の影響を大きくする、前記(2)に記載の情報処理装置。
(7)前記アクション情報は、第1の時間情報に関連付けられた第1のアクション情報と第2の時間情報に関連付けられた第2のアクション情報とを含み、
前記マップ表現処理部は、前記第1のアクション情報または前記第2のアクション情報の少なくともいずれかの前記マップ表現を実行するにあたり、前記第1の時間情報によって示される時刻と前記第2の時間情報によって示される時刻との前後関係を考慮する、前記(1)~(6)のいずれか1項に記載の情報処理装置。
(8)前記第1のアクション情報は、第1の空間情報に関連付けられ、
前記第2のアクション情報は、前記第1の空間情報とは異なる第2の空間情報に関連付けられ、
前記マップ表現処理部は、前記第2の時間情報が前記第1の時間情報よりも後の時刻を示す場合に、前記マップ表現において前記第1の空間情報によって示される位置から前記第2の空間情報によって示される位置への移動を表現する、前記(7)に記載の情報処理装置。
(9)前記アクション情報は、アクションを評価するアクションのスコアに関する情報を含み、
前記マップ表現処理部は、前記マップ表現を実行するにあたり、前記アクションのスコアに関する情報をさらに考慮する、前記(1)~(8)のいずれか1項に記載の情報処理装置。
(10)前記アクション情報は、前記スコアに基づいて推定される前記ユーザのスキルレベルに関する情報を含み、
前記マップ表現処理部は、前記マップ表現を実行するにあたり、前記スキルレベルに関する情報をさらに考慮する、前記(9)に記載の情報処理装置。
(11)前記アクション情報は、前記ユーザのアクションのネガティブな評価を示すスコアに関する情報を含み、
前記マップ表現処理部は、前記ネガティブな評価については前記スキルレベルが高いほどスコアを上方修正する、前記(10)に記載の情報処理装置。
(12)前記アクション情報は、前記ユーザのアクションのポジティブな評価を示すスコアに関する情報を含み、
前記マップ表現処理部は、前記ポジティブな評価については前記スキルレベルが高いほどスコアを下方修正する、前記(10)または(11)に記載の情報処理装置。
(13)前記マップ表現処理部は、前記空間情報に基づいて、前記アクション情報によって示されるアクションが実行された施設を特定し、前記ユーザのスキルレベルの高さに応じて前記施設ごとの評価を含む情報を生成する、前記(10)~(12)のいずれか1項に記載の情報処理装置。
(14)前記マップ表現処理部は、前記アクション情報に関連付けられた第1の空間情報によって示される位置と、前記アクション情報に関連付けられていない第2の空間情報によって示される位置との間の距離に基づいて前記第2の空間情報に関連付けられるアクション情報を推定する、前記(1)~(13)のいずれか1項に記載の情報処理装置。
(15)前記アクション情報は、前記空間情報の第1の系列に関連付けられた第1のアクション情報系列と、前記空間情報の第2の系列に関連付けられた第2のアクション情報系列とを含み、前記第2のアクション情報系列は、前記第1のアクション情報系列よりも前の時刻を示す時間情報に関連付けられ、
前記マップ表現処理部は、前記空間情報の第1の系列によって示される第1の移動経路が、前記空間情報の第2の系列によって示される第2の移動経路に部分的に一致する場合に、前記第2の移動経路の残りの部分を示す前記空間情報の第2の系列に関連付けられた前記第2のアクション情報系列に基づいて前記第1のアクション情報系列に関連付けられたユーザへのナビゲーションを提供する、前記(1)~(14)のいずれか1項に記載の情報処理装置。
(16)前記アクション情報は、前記ユーザの検出されたアクションを示す情報を含み、
前記マップ表現処理部は、前記マップ表現の結果によって示される前記空間情報によって示される位置ごとの前記検出されたアクションの傾向に基づいて、前記ユーザのアクションを検出するための設定を調整する、前記(1)~(15)のいずれか1項に記載の情報処理装置。
(17)ユーザのアクション情報、および前記アクション情報に関連付けられる空間情報ならびに時間情報を取得することと、
プロセッサが、前記空間情報に基づく前記アクション情報のマップ表現を実行するにあたり、前記時間情報を考慮することと
を含む情報処理方法。
(18)ユーザのアクション情報、および前記アクション情報に関連付けられる空間情報ならびに時間情報を取得する機能と、
前記空間情報に基づく前記アクション情報のマップ表現を実行するにあたり、前記時間情報を考慮する機能と
をコンピュータに実現させるためのプログラム。
(1)ユーザまたは前記ユーザによって使用される器具に装着されたセンサによって提供されるセンサデータを受信するセンサデータ受信部と、
前記センサデータに基づいて前記ユーザのアクションを検出するアクション検出部と、
前記検出されたアクションに応じて、前記ユーザまたは前記アクションに関連する機器を制御する機器制御部と
を備える情報処理装置。
(2)前記アクション検出部は、前記ユーザの一連の状態の中で発生する前記アクションを検出する、前記(1)に記載の情報処理装置。
(3)前記機器制御部は、前記一連の状態の中で、前記アクションが検出されている区間に対応する第1の区間と、そうでない第2の区間との間で異なる前記機器の制御を実行する、前記(2)に記載の情報処理装置。
(4)前記機器は、前記アクションの映像または画像を記録する機能を有し、
前記機器制御部は、前記第1の区間で前記映像または前記画像を特定するように前記機器を制御する、前記(3)に記載の情報処理装置。
(5)前記センサデータは、前記ユーザの高度を示すデータを含み、
前記アクションは、ジャンプを含み、
前記機器制御部は、前記ジャンプが検出されている前記第1の区間で、少なくとも前記ユーザの高度のピークで前記画像を記録するように前記機器を制御する、前記(4)に記載の情報処理装置。
(6)前記機器は、前記アクションの映像または画像を記録する機能を有し、
前記機器制御部は、前記第1の区間では前記映像または前記画像を第1の品質で記録し、前記第2の区間では前記映像または前記画像を前記第1の品質よりも低い第2の品質で記録するように前記機器を制御する、前記(3)に記載の情報処理装置。
(7)前記第1の品質は、第1の圧縮率、第1のフレームレート、第1のフレームサイズ、または第1の集音レベルを含み、
前記第2の品質は、前記第1の圧縮率よりも高い第2の圧縮率、前記第1のフレームレートよりも低い第2のフレームレート、前記第1のフレームサイズよりも小さい第2のフレームサイズ、または前記第1の集音レベルよりも低い第2の集音レベルを含む、前記(6)に記載の情報処理装置。
(8)前記機器は、前記センサデータを記録する機能を有し、
前記機器制御部は、前記第1の区間で前記センサデータを記録するように前記機器を制御する、前記(3)~(7)のいずれか1項に記載の情報処理装置。
(9)前記機器は、前記センサデータを記録する機能を有し、
前記機器制御部は、前記第1の区間では前記センサデータを第1のサンプリングレートで記録し、前記第2の区間では前記センサデータを前記第1のサンプリングレートよりも低い第2のサンプリングレートで記録するように前記機器を制御する、前記(3)~(7)のいずれか1項に記載の情報処理装置。
(10)前記機器は、前記アクションに関する情報を生成するために用いられる付加的なセンサデータを提供する機能を有し、
前記機器制御部は、前記第1の区間で前記付加的なセンサデータが提供されるように前記機器を制御する、前記(3)~(9)のいずれか1項に記載の情報処理装置。
(11)前記機器制御部は、前記検出されたアクションの履歴に応じて前記機器を制御する、前記(1)~(10)のいずれか1項に記載の情報処理装置。
(12)前記機器制御部は、前記機器が新たに起動されるまでに検出された前記アクションの直近履歴を、前記機器が過去に起動されるまでに検出された前記アクションの過去履歴と比較し、前記直近履歴に類似する前記過去履歴に対応する前記機器の設定を、新たに起動される前記機器にも適用する、前記(11)に記載の情報処理装置。
(13)前記機器は、前記ユーザに情報を提示する機能を有し、
前記アクションは、ユーザに注意力が要求される状況で発生する特定の種類のアクションを含み、
前記機器制御部は、前記特定の種類のアクションが検出された場合に、前記情報の内容または提示方法が変化するように前記機器を制御する、前記(1)~(12)のいずれか1項に記載の情報処理装置。
(14)前記機器制御部は、前記情報の内容が前記状況に関連するものに変更されるように前記機器を制御する、前記(13)に記載の情報処理装置。
(15)前記機器制御部は、画像によって提示されていた前記情報を、音声またはバイブレーションによって提示するか、または提示しないように前記機器を制御する、前記(13)に記載の情報処理装置。
(16)前記機器は、前記アクションの映像または画像を加工する機能を有し、
前記機器制御部は、前記アクションが検出されている区間に対応する区間で前記映像または画像を加工するように前記機器を制御する、前記(1)~(15)のいずれか1項に記載の情報処理装置。
(17)前記機器による前記映像または前記画像の加工は、前記映像または前記画像に基づいて、自由視点映像もしくは自由視点画像を提供するためのデータ、または3次元オブジェクトデータを生成することを含む、前記(16)に記載の情報処理装置。
(18)ユーザまたは前記ユーザによって使用される器具に装着されたセンサによって提供されるセンサデータを受信することと、
前記センサデータに基づいて前記ユーザのアクションを検出することと、
プロセッサが、前記検出されたアクションに応じて、前記ユーザまたは前記アクションに関連する機器を制御することと
を含む情報処理方法。
(19)ユーザまたは前記ユーザによって使用される器具に装着されたセンサによって提供されるセンサデータを受信する機能と、
前記センサデータに基づいて前記ユーザのアクションを検出する機能と、
前記検出されたアクションに応じて、前記ユーザまたは前記アクションに関連する機器を制御する機能と
をコンピュータに実現させるためのプログラム。
101 送信部
102 受信部
103 センサデバイス制御部
104 センサデータ解析部
105 特徴量抽出部
106 アクション検出部
107 解析結果処理部
108 クラスタリング処理部
109 スコアリング処理部
112 サービス制御部
Claims (18)
- ユーザのアクション情報、および前記アクション情報に関連付けられる空間情報ならびに時間情報を取得する情報取得部と、
前記空間情報に基づく前記アクション情報のマップ表現を実行するにあたり、前記時間情報を考慮するマップ表現処理部と
を備える情報処理装置。 - 前記マップ表現処理部は、前記時間情報によって示される時刻と基準時刻との関係を考慮する、請求項1に記載の情報処理装置。
- 前記マップ表現処理部は、前記時間情報によって示される時刻が前記基準時刻から離れるほど、前記マップ表現における前記アクション情報の影響を小さくする、請求項2に記載の情報処理装置。
- 前記アクション情報は、前記ユーザのアクションを複数のカテゴリのいずれかに分類するときの前記各カテゴリのスコアを含み、
前記マップ表現処理部は、共通の前記空間情報に関連付けられた前記アクション情報に含まれる前記スコアに前記時間情報によって示される時刻が前記基準時刻から離れるほど小さくなる重みをつけて足し合わせることによって、前記マップ表現において前記共通の空間情報に関連付けられる代表アクションのカテゴリを決定する、請求項3に記載の情報処理装置。 - 前記マップ表現処理部は、前記基準時刻との差が閾値を下回る時刻を示す前記時間情報に関連付けられた前記アクション情報に基づいて前記マップ表現を実行する、請求項3に記載の情報処理装置。
- 前記マップ表現処理部は、前記時間情報が前記基準時刻と共通する属性を有する場合に、前記マップ表現における前記アクション情報の影響を大きくする、請求項2に記載の情報処理装置。
- 前記アクション情報は、第1の時間情報に関連付けられた第1のアクション情報と第2の時間情報に関連付けられた第2のアクション情報とを含み、
前記マップ表現処理部は、前記第1のアクション情報または前記第2のアクション情報の少なくともいずれかの前記マップ表現を実行するにあたり、前記第1の時間情報によって示される時刻と前記第2の時間情報によって示される時刻との前後関係を考慮する、請求項1に記載の情報処理装置。 - 前記第1のアクション情報は、第1の空間情報に関連付けられ、
前記第2のアクション情報は、前記第1の空間情報とは異なる第2の空間情報に関連付けられ、
前記マップ表現処理部は、前記第2の時間情報が前記第1の時間情報よりも後の時刻を示す場合に、前記マップ表現において前記第1の空間情報によって示される位置から前記第2の空間情報によって示される位置への移動を表現する、請求項7に記載の情報処理装置。 - 前記アクション情報は、アクションを評価するアクションのスコアに関する情報を含み、
前記マップ表現処理部は、前記マップ表現を実行するにあたり、前記アクションのスコアに関する情報をさらに考慮する、請求項1に記載の情報処理装置。 - 前記アクション情報は、前記スコアに基づいて推定される前記ユーザのスキルレベルに関する情報を含み、
前記マップ表現処理部は、前記マップ表現を実行するにあたり、前記スキルレベルに関する情報をさらに考慮する、請求項9に記載の情報処理装置。 - 前記アクション情報は、前記ユーザのアクションのネガティブな評価を示すスコアに関する情報を含み、
前記マップ表現処理部は、前記ネガティブな評価については前記スキルレベルが高いほどスコアを上方修正する、請求項10に記載の情報処理装置。 - 前記アクション情報は、前記ユーザのアクションのポジティブな評価を示すスコアに関する情報を含み、
前記マップ表現処理部は、前記ポジティブな評価については前記スキルレベルが高いほどスコアを下方修正する、請求項10に記載の情報処理装置。 - 前記マップ表現処理部は、前記空間情報に基づいて、前記アクション情報によって示されるアクションが実行された施設を特定し、前記ユーザのスキルレベルの高さに応じて前記施設ごとの評価を含む情報を生成する、請求項10に記載の情報処理装置。
- 前記マップ表現処理部は、前記アクション情報に関連付けられた第1の空間情報によって示される位置と、前記アクション情報に関連付けられていない第2の空間情報によって示される位置との間の距離に基づいて前記第2の空間情報に関連付けられるアクション情報を推定する、請求項1に記載の情報処理装置。
- 前記アクション情報は、前記空間情報の第1の系列に関連付けられた第1のアクション情報系列と、前記空間情報の第2の系列に関連付けられた第2のアクション情報系列とを含み、前記第2のアクション情報系列は、前記第1のアクション情報系列よりも前の時刻を示す時間情報に関連付けられ、
前記マップ表現処理部は、前記空間情報の第1の系列によって示される第1の移動経路が、前記空間情報の第2の系列によって示される第2の移動経路に部分的に一致する場合に、前記第2の移動経路の残りの部分を示す前記空間情報の第2の系列に関連付けられた前記第2のアクション情報系列に基づいて前記第1のアクション情報系列に関連付けられたユーザへのナビゲーションを提供する、請求項1に記載の情報処理装置。 - 前記アクション情報は、前記ユーザの検出されたアクションを示す情報を含み、
前記マップ表現処理部は、前記マップ表現の結果によって示される前記空間情報によって示される位置ごとの前記検出されたアクションの傾向に基づいて、前記ユーザのアクションを検出するための設定を調整する、請求項1に記載の情報処理装置。 - ユーザのアクション情報、および前記アクション情報に関連付けられる空間情報ならびに時間情報を取得することと、
プロセッサが、前記空間情報に基づく前記アクション情報のマップ表現を実行するにあたり、前記時間情報を考慮することと
を含む情報処理方法。 - ユーザのアクション情報、および前記アクション情報に関連付けられる空間情報ならびに時間情報を取得する機能と、
前記空間情報に基づく前記アクション情報のマップ表現を実行するにあたり、前記時間情報を考慮する機能と
をコンピュータに実現させるためのプログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP15876941.4A EP3244321A4 (en) | 2015-01-05 | 2015-10-15 | Information processing device, information processing method, and program |
US15/527,396 US10571289B2 (en) | 2015-01-05 | 2015-10-15 | Information processing device, information processing method, and program |
JP2016568285A JP6683134B2 (ja) | 2015-01-05 | 2015-10-15 | 情報処理装置、情報処理方法、およびプログラム |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2015-000411 | 2015-01-05 | ||
JP2015000411 | 2015-01-05 | ||
JP2015000413 | 2015-01-05 | ||
JP2015-000413 | 2015-01-05 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2016111068A1 true WO2016111068A1 (ja) | 2016-07-14 |
Family
ID=56342239
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2015/079176 WO2016111068A1 (ja) | 2015-01-05 | 2015-10-15 | 情報処理装置、情報処理方法、およびプログラム |
Country Status (5)
Country | Link |
---|---|
US (1) | US10571289B2 (ja) |
EP (1) | EP3244321A4 (ja) |
JP (1) | JP6683134B2 (ja) |
CN (1) | CN105759954A (ja) |
WO (1) | WO2016111068A1 (ja) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2018116476A1 (ja) * | 2016-12-22 | 2018-06-28 | 富士通株式会社 | 情報処理装置、情報処理方法および情報処理プログラム |
EP3361449A1 (en) * | 2017-02-14 | 2018-08-15 | Kabushiki Kaisha Toshiba | Information processing device, information processing method, computer-readable medium, and moving object |
IT201700025374A1 (it) * | 2017-03-07 | 2018-09-07 | Zehus S P A | Sistema per la stima del comportamento di un ciclista su una bicicletta e/o della qualità di una strada lungo un percorso seguito da detta bicicletta |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6653459B2 (ja) | 2015-10-29 | 2020-02-26 | パナソニックIpマネジメント株式会社 | 画像処理装置及びこれを備えた脈拍推定システムならびに画像処理方法 |
US11050991B2 (en) * | 2016-09-29 | 2021-06-29 | Koninklijke Philips N.V. | Image processing using a plurality of images for a three dimension scene, having a different viewing positions and/or directions |
CN106486002B (zh) * | 2016-09-30 | 2019-09-20 | 深圳市华傲数据技术有限公司 | 危险品存放地波及程度的数据处理及展示方法、*** |
CN108702442B (zh) * | 2016-10-14 | 2021-04-16 | 深圳市大疆创新科技有限公司 | 用于时刻捕获的***和方法 |
US11460306B2 (en) * | 2017-08-25 | 2022-10-04 | Stroly Inc. | Information processing apparatus, information processing method, and program |
JP7051503B2 (ja) * | 2018-03-13 | 2022-04-11 | アズビル株式会社 | 多変量時系列データの同期方法および多変量時系列データ処理装置 |
US10664489B1 (en) | 2018-11-16 | 2020-05-26 | Fuvi Cognitive Network Corp. | Apparatus, method, and system of cognitive data blocks and links for personalization, comprehension, retention, and recall of cognitive contents of a user |
JP7223629B2 (ja) * | 2019-05-13 | 2023-02-16 | 日立Astemo株式会社 | 車載システム、外界認識センサ、電子制御装置 |
CN110860088B (zh) * | 2019-11-14 | 2023-04-07 | 网易(杭州)网络有限公司 | 竞速游戏中小地图的渲染方法、装置以及游戏终端 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2012230496A (ja) * | 2011-04-25 | 2012-11-22 | Toshiba Corp | 情報処理装置、及び情報処理方法 |
JP2013003649A (ja) * | 2011-06-13 | 2013-01-07 | Sony Corp | 情報処理装置、情報処理方法およびコンピュータプログラム |
JP2013239115A (ja) * | 2012-05-17 | 2013-11-28 | Hitachi Ltd | 運動支援システム及び運動支援方法 |
JP2014002036A (ja) * | 2012-06-18 | 2014-01-09 | Nippon Telegr & Teleph Corp <Ntt> | 歩行経路推薦装置、方法及びプログラム |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5649808B2 (ja) * | 2009-01-28 | 2015-01-07 | ソニー株式会社 | 情報処理装置、情報処理方法、プログラム |
JP5440080B2 (ja) * | 2009-10-02 | 2014-03-12 | ソニー株式会社 | 行動パターン解析システム、携帯端末、行動パターン解析方法、及びプログラム |
JP2011107977A (ja) * | 2009-11-17 | 2011-06-02 | Fujitsu Ten Ltd | 情報処理装置、車載装置、情報処理システム、情報処理方法、および、プログラム |
WO2012050898A1 (en) * | 2010-09-29 | 2012-04-19 | The Board Of Regents Of The University Of Nebraska | Lifespace data collection from discrete areas |
CN103905978B (zh) | 2012-12-27 | 2018-03-23 | 中国电信股份有限公司 | 基于位置的信息推送方法、推送装置和网络 |
JP5805128B2 (ja) * | 2013-03-28 | 2015-11-04 | 本田技研工業株式会社 | 地図提供サーバ及び地図提供方法 |
-
2015
- 2015-10-15 WO PCT/JP2015/079176 patent/WO2016111068A1/ja active Application Filing
- 2015-10-15 JP JP2016568285A patent/JP6683134B2/ja active Active
- 2015-10-15 EP EP15876941.4A patent/EP3244321A4/en not_active Ceased
- 2015-10-15 US US15/527,396 patent/US10571289B2/en active Active
- 2015-12-29 CN CN201511009375.5A patent/CN105759954A/zh active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2012230496A (ja) * | 2011-04-25 | 2012-11-22 | Toshiba Corp | 情報処理装置、及び情報処理方法 |
JP2013003649A (ja) * | 2011-06-13 | 2013-01-07 | Sony Corp | 情報処理装置、情報処理方法およびコンピュータプログラム |
JP2013239115A (ja) * | 2012-05-17 | 2013-11-28 | Hitachi Ltd | 運動支援システム及び運動支援方法 |
JP2014002036A (ja) * | 2012-06-18 | 2014-01-09 | Nippon Telegr & Teleph Corp <Ntt> | 歩行経路推薦装置、方法及びプログラム |
Non-Patent Citations (1)
Title |
---|
See also references of EP3244321A4 * |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2018116476A1 (ja) * | 2016-12-22 | 2018-06-28 | 富士通株式会社 | 情報処理装置、情報処理方法および情報処理プログラム |
JPWO2018116476A1 (ja) * | 2016-12-22 | 2019-07-18 | 富士通株式会社 | 情報処理装置、情報処理方法および情報処理プログラム |
EP3361449A1 (en) * | 2017-02-14 | 2018-08-15 | Kabushiki Kaisha Toshiba | Information processing device, information processing method, computer-readable medium, and moving object |
US10803683B2 (en) | 2017-02-14 | 2020-10-13 | Kabushiki Kaisha Toshiba | Information processing device, information processing method, computer program product, and moving object |
IT201700025374A1 (it) * | 2017-03-07 | 2018-09-07 | Zehus S P A | Sistema per la stima del comportamento di un ciclista su una bicicletta e/o della qualità di una strada lungo un percorso seguito da detta bicicletta |
WO2018163027A1 (en) * | 2017-03-07 | 2018-09-13 | Zehus S.P.A. | System and method for estimating the behavior of a cyclist on a bicycle and/or the quality of a road along a path followed by said bicycle |
US11772730B2 (en) | 2017-03-07 | 2023-10-03 | Zehus S.P.A. | System and method for estimating a behavior of a cyclist on a bicycle and a quality of a road along a path followed by the bicycle |
Also Published As
Publication number | Publication date |
---|---|
CN105759954A (zh) | 2016-07-13 |
US10571289B2 (en) | 2020-02-25 |
US20170328726A1 (en) | 2017-11-16 |
JPWO2016111068A1 (ja) | 2017-10-12 |
EP3244321A1 (en) | 2017-11-15 |
JP6683134B2 (ja) | 2020-04-15 |
EP3244321A4 (en) | 2018-07-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2016111068A1 (ja) | 情報処理装置、情報処理方法、およびプログラム | |
CN105749548B (zh) | 信息处理装置、信息处理方法以及程序 | |
US11895068B2 (en) | Automated content curation and communication | |
JP6658545B2 (ja) | 情報処理装置、情報処理方法、およびプログラム | |
US20190383620A1 (en) | Information processing apparatus, information processing method, and program | |
JP6311478B2 (ja) | 情報処理装置、情報処理方法およびプログラム | |
CN104115180B (zh) | 信息处理设备、信息处理方法和程序 | |
WO2016002285A1 (ja) | 情報処理装置、情報処理方法およびプログラム | |
US11710422B2 (en) | Driving analysis and instruction device | |
US20110190008A1 (en) | Systems, methods, and apparatuses for providing context-based navigation services | |
CN106767818B (zh) | 可穿戴电子设备及其健康出行*** | |
JPWO2018198281A1 (ja) | 情報処理装置、空撮経路生成方法、空撮経路生成システム、プログラム、及び記録媒体 | |
WO2016002284A1 (ja) | 情報処理装置、情報処理方法およびプログラム | |
JPWO2018163560A1 (ja) | 情報処理装置、情報処理方法およびプログラム | |
WO2015194270A1 (ja) | 情報処理装置、情報処理方法およびプログラム | |
JP6855825B2 (ja) | 情報処理装置、情報処理方法およびプログラム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 15876941 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2016568285 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 15527396 Country of ref document: US |
|
REEP | Request for entry into the european phase |
Ref document number: 2015876941 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |