GB2518623A

GB2518623A - Audio analysis apparatus

Info

Publication number: GB2518623A
Application number: GB1317030.3A
Authority: GB
Inventors: Antti Eronen; Jussi Artturi Leppanen; Igor Danilo Diego Curcio
Original assignee: Nokia Technologies Oy
Current assignee: Nokia Technologies Oy
Priority date: 2013-09-25
Filing date: 2013-09-25
Publication date: 2015-04-01
Also published as: WO2015044521A1; GB201317030D0

Abstract

An apparatus 109 comprises at least one periodicity estimator 16 configured to determine estimates of periodicity based on positional data associated with at least two apparatus 19 experiencing the same audio event 103, such as a live music performance or radio broadcast. A joint tempo determiner determines a joint tempo (BPM) estimate 163 based on the estimates of periodicity associated with the at least two apparatus. The apparatus may also determine a periodicity distribution and a dominant periodicity estimate, which may be displayed to an event audience and/or an event creator. The at least two apparatus may be mobile devices, containing accelerometers or other position sensors, which transmit position data relating to the dancing or other rhythmic motion of people in an audience. The joint tempo estimate may be used, along with signals recorded by microphones, to estimate the tempo of the audio produced during the event. Claims to an apparatus for determining and transmitting positional data relating to at least one apparatus experiencing an audio event is also disclosed.

Description

Intellectual Property Office Applicacion Nc,. (lB 1317030.3 RTM Dace:4 Dircinbcr 2013 The following terms are registered trade marks and should he rcad as such wherever they occur in this document: Bluetooth Wi-Fi Inlelleclual Property Office is an operaling name of the Pateni Office www.ipo.gov.uk

I

AUDIO ANALYSIS APPARATUS

Field

The present application relates to apparatus for analysing audio. The invention further relates to, but is not limited to, apparatus for analysing audio from mobile devices.

Background

Analysing audio content such as live recorded audio content is well known.

The estimation of the tempo (or beats per minute, bpm) of songs, especially from live performances, can be difficult. BPM estimation Is traditionally done by analIng an audio recording of the performance. The quality of the estimation depends on the quality of the recorded audio. In scenarios, such as those encountered in concerts, the audio quality recorded might not be of very high quality due to the noises made by the audience, audio recording technology sometimes present in mobile devices, or due to a non optimal recording position.

Also the acoustic characteristics of various concert venues can have an effect on the recorded audio and thus will have an effect on the BPM estimation. Moreover, in many situations the BPM is difficult to estimate with algorithms from audio only.

Summary

Aspects of this application thus provide suitable audio capture and audio playback for multichannel audio signals using single channel storage or recording to permit a better audio listening experience.

There is provided according to a first aspect an apparatus comprising at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to at least: determine estimates of periodicity based on positional data associated with at least two apparatus experiencing the same audio event; determine a joInt tempo estimate based on the estimates of periodicity associated with the at least two apparatus.

The apparatus may be further caused to: determine at east one audio based tempo estimate, wherein the at least one audio based tempo estimate is further based on the joint tempo estimate.

Determining at least one audio based tempo estimate may cause the apparatus to: synchronise at least two audio signals associated with the at least two apparatus experiencing the same audio event; determine the at least one audio based tempo estimate based on the synchronised at least two audio signals further based on the joint tempo estimate.

The apparatus may be further caused to receive the at least two audio signals from the at least two apparatus experiencing the event.

The at least two audio signals may comprise the positional data associated with the at least two apparatus experiencing the same audio event.

The apparatus may be further caused to receive the positional data associated with the at least two apparatus experiencing the same audio event.

The apparatus may be further caused to determine a periodicity distribution from 26 the estimates of periodicity associated with the at least two apparatus.

The apparatus may be further caused to determine a dominant periodicity estimate based on the periodicity distribution.

The apparatus may be further caused to display the dominant periodicity estimate based on the periodicity distribution to at least one of: an event audience; an event creator.

The apparatus may be further caused to transmit the dominant perlodicity estimate to at least one of the at least two apparatus experiencing the same audio event.

The apparatus may be at least one of the at least two apparatus experiencing the same audio event.

Determining estimates of peilodicity based on positional data associated with at least two apparatus experiencing the same audio event may cause the apparatus to receive the estimates of perlodicity based on positional data from the at least two apparatus experiencing the same audio event.

According to a second aspect there is provided an apparatus comprising at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to: determine positional data; transmit the positional data to a further apparatus, wherein the apparatus is one of at least two apparatus experiencing the same audio event.

The apparatus may be further caused to: capture at least one audio signal based on the audio event; transmit the audio signal to the further apparatus.

Transmitting the audio signal to the further apparatus may cause the apparatus to: multiplex the positional data into the at least one audio signal; transmit the at least one audio signal comprising the positional data to the further apparatus.

The apparatus may be further caused to receive from the further apparatus a dominant perlodicity estimate based on a determination of the perlodlclty distribution of the positional data to at least one of the at least two apparatus experIencing the same audio event.

The apparatus may be further caused to determine a difference between the dominant perlodicity estimate and a perlodicity estimate of the apparatus.

The apparatus may further be caused to output the difference between the dominant peulodicity estimate and the periodicity estimate of the apparatus.

According to a third aspect them is provided a method comprising: determining estimates of periodicity based on positional data associated with at least two apparatus experiencing the same audio event; determining a joint tempo estimate based on the estimates of periodlcity associated with the at least two apparatus.

The method may further comprise determining at least one audio based tempo estImate, wherein the at least one audio based tempo estimate is further based on the joint tempo estimate.

Determining at least one audio based tempo estimate may further comprise: synchronising at least two audio signals associated with the at least two apparatus experiencing the same audio event; determining the at least one audio based tempo estimate based on the synchronised at least two audio signals further based on the joint tempo estimate.

The method may further comprise receMng the at least two audio signals from the at least two apparatus experiencing the event.

The at least two audio signais may comprise the positional data associated with the at least two apparatus experiencing the same audio event.

The method may further comprise receMng the positional data associated with the at least two apparatus experiencing the same audio event.

The method may further comprise determining a peuiodicity distribution from the estimates of perlodicity associated with the at least two apparatus.

The method may further comprise determining a dominant perlodicity estimate based on the periodicity distribution.

The method may further comprise displaying the dominant periodicity estimate based on the perlodiclty distribution to at least one of: an event audience; an event creator.

The method may further comprise transmitting the dominant pertodicity estimate to at least one of the at least two apparatus experiencing the same audio event.

The method may be implemented on an apparatus, wherein the apparatus is least one of the at least two apparatus experiencing the same audio event.

Determining estimates of perlodicity based on positional data associated with at least two apparatus experiencing the same audio event may comprise receMng the estimates of periodicity based on positional data from the at least two apparatus experiencing the same audio event.

According to a fourth aspect there is provided a method comprising: determining positional data; transmitting the positional data to a further apparatus, wherein the method is performed on one of at least two apparatus experiencing the same audio event, and wherein the further apparatus performs a method comprising determining estimates of pedodicity based on positional data associated with at least two apparatus experiencing the same audio event; and determining a Joint tempo estimate based on the estimates of periodicity associated with the at least two apparatus.

The method may comprise capturing at least one audio signal based on the audio event; transmitting the audio signal to the further apparatus.

Transmitting the audio signal to the further apparatus may comprise: multiplexing the positional data Into the at least one audio signal; transmitting the at least one audio signal comprising the positional data to the further apparatus.

The method may further comprise receiving from the further apparatus a dominant periodicity estimate based on a determination of the periodicity distribution of the positional data to at least one of the at least two apparatus experiencing the same audio event.

The method may further comprise determining a difference between the dominant perlodlcity estimate and a periodicity estimate of the apparatus.

The method may further comprise outputting the difference between the dominant perlodiclty estimate and the perlodicity estimate of the apparatus.

According to a fifth aspect there is pnvided an apparatus comprising: means for determining estimates of peclodicity based on positional data associated with at least two apparatus experiencing the same audio event; means for determining a joint tempo estimate based on the estimates of periodicity associated with the at least two apparatus.

The apparatus may further comprise means for determining at least one audio based tempo estimate, wherein the at least one audio based tempo estimate is further based on the Joint tempo estimate.

The means for determining at least one audio based tempo estimate may further comprise: means for synchronising at least two audio signals associated with the at least two apparatus experiencing the same audio event; means for determining the at least one audio based tempo estimate based on the synchronised at least two audio signals further based on the Joint tempo estimate.

The apparatus may further comprise means for receiving the at least two audio signals from the at least two apparatus experiencing the event.

The apparatus may further comprise means for receMng the positional data associated with the at least two apparatus experiencing the same audio event The apparatus may further comprise means for determining a perlodicity distribution from the estimates of perlodicity associated with the at least two apparatus.

The apparatus may further comprise means for determining a dominant periodicity estimate based on the peilodicity distribution.

The apparatus may further comprise means for displaying the dominant periodicity estimate based on the periodicity distribution to at least one of: an event audience; an event creator.

The apparatus may further comprise means for transmitting the dominant perlodicity estimate to at least one of the at least two apparatus experiencing the same audio event.

The means for determining estimates of perlodicity based on positional data associated with at least two apparatus experiencing the same audio event may comprise means for receMng the estimates of perlodicity based on positional data from the at least two apparatus experiencing the same audio event.

According to a sixth aspect there is provided an apparatus comprising: means for determining positional data; means for transmitting the positional data to a further apparatus, wherein the apparatus is at least one of the two apparatus experiencing the same audio event, and wherein the further apparatus comprises means for determining estimates of peilodiclty based on positional data assocIated with at least two apparatus experiencing the same audio event; and means for determining a joint tempo estimate based on the estimates of periodicity associated with the at least two apparatus.

The apparatus may comprise means for capturing at least one audio signal based on the audio event; transmitting the audio signal to the further apparatus.

The means for transmitting the audio signal to the further apparatus may comprIse: means for multiplexing the positional data into the at least one audio signal; means for transmitting the at least one audio signal comprising the positional data to the further apparatus.

The apparatus may further comprise means for receMng from the further apparatus a dominant perlodicity estimate based on a determination of the perlodicity distribution of the positional data to at least one of the at least two apparatus experiencing the same audio event.

The apparatus may further comprise means for determining a difference between the dominant periodicity estimate and a periodicity estimate of the apparatus.

The apparatus may further comprise means for outputting the difference between the dominant petiodicity estimate and the periodicity estimate of the apparatus.

According to a seventh aspect there is provided an apparatus comprising: at least one perlodicity estimator configured to determine estimates of peilodicity based on positional data associated with at least two apparatus experiencing the same audio event a joint tempo determiner configure to determine a joint tempo estimate based on the estimates of perlodiclty associated with the at least two apparatus.

The apparatus may further comprise an audio tempo determiner configured to determine at least one audio based tempo estimate, wherein the at least one audio based tempo estimate Is further based on the joint tempo estimate.

The audio tempo determiner may further comprise: an audio synchronlser configured to synchronlse at least two audio signals associated with the at least two apparatus experiencing the same audio event; wherein the audio tempo determiner Is configured to determine the at least one audio based tempo estimate based on the synchronised at least two audio signals further based on the joint tempo estimate.

The apparatus may further comprise an input configured to receive the at least two audio signals from the at least two apparatus experiencing the event The at least two audio signals may comprise the positional data associated with the at least two apparatus experiencing the same audio event.

The apparatus may further comprise a positional data input configured to receive the positional data associated with the at least two apparatus experiencing the same audio event The apparatus may further comprise a perlodicity distribution determiner configured to determine a periodicity distribution from the estimates of perlodicity associated with the at least two apparatus.

The periodicity distribution determiner may further be configured to determine a dominant perlodicity estimate based on the peilodicity distribution.

The apparatus may further comprise an output configured to display the domInant periodicity estimate based on the petiodicity distribution to at least one of: an event audience; an event creator.

The apparatus may further comprise an output configured to transmit the dominant perlodicity estimate to at least one of the at least two apparatus experiencing the same audio event The apparatus may be at least one of the at least two apparatus experiencing the same audio event.

The Joint tempo determiner may comprise an input configured to receive at least two estimates of perlodicity based on positional data from the at least two apparatus experiencing the same audio event.

According to an eighth aspect there is provided an apparatus comprising: a sensor configured to determine positional data; an output configured to transmit the positional data to a further apparatus, wherein the apparatus is at least one of the two apparatus experiencing the same audio event, and wherein the further apparatus comprises at least one perlodiclty estimator configured to determine estimates of periodicity based on positional data associated with at least two apparatus experiencing the same audio event; a joint tempo determiner configure to determine a joint tempo estimate based on the estimates of perlodicity associated with the at least two apparatus.

The sensor may comprise at least one ot a compass; a gyroscope; a positional determiner a directional determiner; a level detector; a velocity determiner; a gps; and an accelerometer.

The apparatus may comprise at least one microphone configured to capture at least one audio signal based on the audio event; and the output may be configured to transmit the audio signal to the further apparatus.

The output may comprise: a multiplexer configured to multiplex the positional data into the at least one audio signal; the output being further configured to transmit the at feast one audio signal comprising the positional data to the further apparatus.

The apparatus may further comprise an Input configured to receive from the further apparatus a dominant peilodiclty estimate based on a determination of the periodicity distribution of the positional data to at least one of the at least two apparatus experiencing the same audio event, ii The apparatus may further comprise a tempo difference determiner configured to determine a difference between the dominant periodicity estimate and a periodicity estimate of the apparatus.

The apparatus may further comprise an output for displaying the difference between the dominant periodicity estimate and a periodicity estimate of the apparatus.

A computer program product stored on a medium may cause an apparatus to perform the method as described herein.

An electronic device may comprise apparatus as described herein.

A chipset may comprise apparatus as described herein.

Embodiments of the present appUcation aim to address problems associated with the state of the art.

Summary of the Figures

For better understanding of the present apphcation, reference wifi now be made by way of example to the accompanying drawings in which: Figure 1 shows schematicafly example audio analysis system apparatus according to some embodiments; Figure 2 shows schematicafly an apparatus suitable for being empked in some embodiments; Figure 3 shows a flow diagram of the operation of the example recording apparatus suitable for operating within the system as shown in Figure 1; Figure 4 shows a flow diagram of the operation of the example tempo analysis apparatus according to sonic embodiments; Figure 5 shows a flow diagram of further operations of the example tempo analysis according to some embodiments; and Figure 6 shows an example visualisation of the hotness' scale output from the example tempo analysis apparatus.

Embodiments of the Aoolication The following describes in further detail suitable apparatus and possible mechanism for the provision of effective audio signal analysis and In particular tempo analysis. In the following examples, audio signals and audio capture sIgnals are described. However It would be appreciated that in some embodiments the audio signal/audio capture is a part of an audio-video system.

The concept of this application Is related to assisting the estimation of the tempo of audio signals. In the application described herein the audio signals are those captured or recorded by microphones from a live event. For example the live event could be an orchestral performance, a popular music concert, a DJ set, or any event where audio signals can be captured from the environment by more than one apparatus. It would be understood however that In some embodiments the teachings of the application can be furthermore applied to non-live' or pie-recorded events. For example In some embodiments the beat or tempo determination can be based on a broadcast audio signal such as a radio or television event being replayed by the apparatus. In such embodiments the apparatus in the group receives the audio signal, for example via a data network communication or from a conventional FM or AM radio signal, rather than from a mIcrophone or microphone array. It Is noted that In some embodiments the participants do not need to be present In the same physical venue, but they can locate In different places but just experience the same audio signal.

The concept as described by the embodiments herein is to utilize crowd motion to either perform tempo or 8PM estimation or to provide extra information to assist other tempo or 8PM estimators. In the embodiments described herein the crowd motion Is determined by using data collected by the accelerometers, gyroscopes or other sensors In the apparatus such as mobile devices experiencing the event (for example a concert). These sensors generate measuring data describing the rate of movement of the people. The concept is an apphcation of the idea that most people will be dancing and shaking to the heat of the music during the event. Furthermore an estimate of the current tempo or BPM can then be S obtained by analyzing the dominant periodicities in the accelerometer (or other sensor) data. These dominant periodicities from the sensor data or several apparatus can then he analysed to jointly determine the BPM.

With respect to Figure 1 an example system for determining enhanced audio analysis according to some embodiments is shown. In the example as shown herein the enhanced audio analysis system is shown configured to output two BPM or tempo determinations, an enhanced audio based BPM determination and a motion based BPM determinaUon and further configured to output a hotness' determination.

Figure 1 shows schematicafly the event 103 configured to generate the audio signals. As described herein the event 103 can be suitable event for example a concert or other live or prerecorded event which is captured (recorded) or received (or input) on at least one mobile device or user equipment. In the example shown in Figure 1 the event 103 is experienced by five user equipment l9, 19?, 193, 194, and 195 (or mobile devices or clients) however it would be understood that there could be more than or fewer than five user equipment (or mobile devices or clients) capable of experiencing, receiving or capturing the event 103 audio signals.

In this regard reference is first made to Figure 2 which shows a schematic block diagram of an exemplary apparatus or electronic device 10, which may operate as the user equipment 19.

The electronic device 10 may for example be a mobHe terminal or user equipment of a wireless communication system. In some embodiments the apparatus can be an audio player or audio recorder, such as an MP3 player, a media recorder/player (also known as an MP4 player), or any suitable portable device suitable for recording audio or audio/video camcorder/memory audio or video recorder.

The apparatus 10 can in some embodiments comprise an audio subsystem. The S audio subsystem for example can comprise in some embodiments a microphone or array of microphones 11 for audio signal capture. In some embodiments the microphone or array of microphones can be a solid state microphone, in other words capable of capturing audio signals and outputting a suitable digital format signaL In some other embodiments the microphone or array of microphones 11 can comprise any suitable microphone or audio capture means, for example a condenser microphone, capacitor microphone, electrostatic microphone, electret condenser microphone, dynamic microphone, ribbon microphone, carbon microphone, piezoelectric microphone, or microelectricaLmechanical system (MEMS) microphone. The microphone 11 or array of microphones can in some embodiments output the audio captured signal to an analogu&4o-digital converter (ADC) 14.

In some embodiments the apparatus can further comprise an analogue4o.digital converter (AOC) 14 configured to receive the analogue captured audio signal from the microphones and outputting the audio captured signal in a suitable digital form. The analogue4o-digital converter 14 can be any suitable analogue4odigital conversion or processing means.

In some embodiments the apparatus 10 audio subsystem further comprises a digital-toanalogue converter 32 for converting digital audio signals from a processor 21 to a suitable analogue format. The digital-to-analogue converter (DAC) or signal processing means 32 can in some embodiments be any suitable DAC technology.

Furthermore the audio subsystem can comprise in some embodiments a speaker 33. The speaker 33 can in some embodiments receive the output from the digital-to-analogue converter 32 and present the analogue audio signal to the user. In some embodiments the speaker 33 can be representaflve of a headset, for example a set of headphones, or cordless headphones.

Afthough the apparatus 10 is shown having both audio capture and audio playback components, it would be understood that in some embodiments the apparatus 10 can comprise one or the other of the audio capture and audio playback parts of the audio subsystem such that in some embodiments of the apparatus the microphone (for audio capture) or the speaker (for audio playback) are present.

In some embodiments the apparatus 10 comprises a processor 21. The processor 21 is coupd to the audio subsystem and specil9caUy in some examples the analogue-to-digital converter 14 for receiving digital signals representing audio signals from the microphone 11, and the digftako-analogue converter (DAC) 12 configured to output processed digital audio signals. The processor 21 can be configured to execute various program codes. The implemented program codes can comprise for example audio signal processing routines.

In some embodiments the apparatus further comprises a memory 22. In some embodiments the processor is coupled to memory 22. The memory can be any suitable storage means. In some embodiments the memory 22 comprises a program code section 23 for storing program codes implementable upon the processor 21. Furthermore in some embodiments the memory 22 can further comprise a stored data section 24 fDr storing data, for example data that has been processed in accordance with the appUcation or data to he processed via the application embodiments as described later. The implemented program code stored within the program code section 23. and the data stored within the stored data section 24 can be retrieved by the processor 21 whenever needed via the memory-processor coupling.

In some further embodiments the apparatus 10 can comprise a user interlace 15.

The user interface 15 can be coupled in some embodiments to the processor 21.

In some embodiments the processor can control the operatbn of the user interface and receive inputs from the user interface 15. In some embodiments the user interface 15 can enable a user to input commands to the electronic device or apparatus 10, for example via a keypad, and/or to obtain information from the apparatus 10, for example via a display which is part of the user interface 15. The user interface 15 can in some embodiments comprise a touch screen or touch interface capable of both enabling information to be entered to the apparatus 10 and further displaying information to the user of the apparatus 10.

In some embodiments the apparatus further comprises a transceiver 13, the transceiver in such embodiments can be coupled to the processor and configured to enable a communication with other apparatus or electronic devices, for example via a wireless communications network, The transceiver 13 or any suitable transceiver or transmitter and/or receiver means can in some embodiments be configured to communicate with other electronic devices or apparatus via a wire or wired coupUng.

The transceiver 13 can communicate with further devices by any suitable known communications protocol, for example in some embodiments the transceiver 13 or transceiver means can use a suitable universal mobile telecommunications sysLem (UMTS) protocol, a wireless local area network (WLAN) protocol such as for example IEEE $02.X, a suitable shortrange radio frequency communication protocol such as Bluetooth, or infrared data communication pathway (IRDA), In some embodiments the apparatus comprises a position sensor 16 contigured to estimate the position of the apparatus 10. The position sensor 16 can in some embodiments be a satellite positioning sensor such as a GPS (Global Positioning System), GLONASS or GalUeo receiver.

In some embodiments the positioning sensor can be a cellular ID system or an assisted GPS system.

in some embodiments the position sensor 16 comprises a direction or orientation sensor. The orientation/direction sensor can in some embodiments be an electronic compass, accelerometer, a gyroscope or be determined by the motion of the apparatus using the positioning estimate.

It is to be understood again that the structure of the electronic device 10 could be supplemented and varied in many ways.

With respect to FigureS an example flow diagram is shown showing the operation of the user equipment 19 with respect to some embodiments.

The user equipment for example l9 is configured to experience (in other words capture/record) the audio event. it would be understood that in some embodiments the experiencing by capturing or recording the audio event can be achIeved by the user equipment microphone or microphones recording the event 103. However It would be understood that in some embodiments, as described herein, the event 103 is a broadcast event and in such circumstances the user equipment experiences the audio event by receMng the audio signal. For example the user equipment may receive the audio event over a conventional FM or AM radio signal or as a streamed data event (which could for example be received over a Wi-Fl or cellular radio communications data connection).

The operation of capturing/recording the audio event is shown in Figure 3 by step 201.

Furthermore the user equIpment 19 can be configured to capture or record sensor data concerning the motion of the apparatus or user equipment For example in the following examples the apparatus gyroscope or accelerometer data is determined or recorded. However It would be understood that in some embodiments the motion of the apparatus could be determined by determining positional information using any suitable means for determining the posItIon/motIon/locatIon of the apparatus.

The user equipment eftectiv&y records positional data using the positional sensor (for example accelerometer or gyroscope data using the accelerometer or gyroscope) of the apparatus or device. Where the user is carrying their user equipment the accelerometer and/or gyroscope data can be captured or recorded using a suitable app or program running in the background. As the user is dancing or waving their hands in the air while listening to the event, the acc&erometer signal (or suitable positional sensor) can reveal movements which are synced to the music tempo.

The operation of capturing or recording the positional data such as from the gyroscope or accelerometer or other suitable sensor is shown in Figure 3 by step 203.

In some embodiments ihe user equipment or apparatus 19 can be configured to transmit the captumd or recorded audio signals to an audio server 109.

Furthermore in some embodiments the user equipment or apparatus 19 can similarly transmit or send the positional data, shown in Figure 1 as accelerometer data, to the audio server 109, In the following examples the processing of the audio signals and the processing of the positional data is performed by the audio server 109 however it would be understood that in some embodiments at least part of the processing or the positional and/or audio data could be perFormed locally with respect to the user equipment. Furthermore it would be understood that in sonic embodiments that the audio server 109 could be implemented at least partially within one of the user equipment experiencing the audio event, In other words one of the user equipment assuming a master role while the other user equipment supplying either sensor and audio event data or parbafly processed sensor and audio event data. Furthermore in some embodiments the audio server 109 operations can be implemented by more than one entity, for example an audio data processing entity or server and a positional or motion data processing entity or server, In some embodiments the audio data and the posftional data is transmitted or sent separately to the audio server 109. However It would be understood that in some embodiments the audio data can be encoded with the positional data. For example in some embodiments the positional data is encoded as metadata within the audio signal in order to permit audio/sensor signal synchronisation. Any suitable encoding of the audio data with the positional data can be performed.

For example in some embodiments the audio data can be band limited with the positional data quantised and frequency shifted into the frequency range removed by the filter. The audio and positional data can then in some embodiments be combined to generate a suitable combined audio signal with positional metadata signal.

The operation of encoding (combining) the audio data with the positional data is shown in Figure 3 by step 205.

The user equipment can then transmit the audio and positional data (for example the combined audio and positional data) to the audio server 109.

The operation of transmitting the audio and positional data to the audio server is shown in Figure 3 by step 207.

In some embodiments the audio analysis system, tempo or 8PM determiner, comprises an audio server 109. The audio server 109 can in some embodiments be configured to receive the audio and positional data from the user equipment or mobile devices. For example in some embodiments the audio server 109 can be configured to receive separate audio and positional data streams and pass these to suitable processing entities. In some embodiments the audio server 109 is configured to receive a combined signal or stream from the mobile device or apparatus.

The operation of receiving the audio and positional data from the mobile devices is shown in Agure 4 by step 301.

In some embodiments the combined signal or data stream from the user equipment or mobile devices is separated into separate audio data and positional data components. For example in some embodiments where the positional data is encoded in the audio data the positional data is extracted from the audio data (using the earner encoded audio signal example, this can be performed by band pass ffltering the audio signal and decoding the filtered signals to extract the positional data signal).

In some embodiments the audio data can then be passed to an audio track synchroniser/creator 153 and the posiUonal data such as the acc&erometer data can be passed to a crowd motion-based BPM estimator 151.

The operation of separating the audio data from positional data is shown in Figure 4 by step 303.

In some embodiments the audio server 109 comprises an audio track synchroniser/creator 153. The audio track synchroniser/creator 153 can in some embodiments be configured to receive the audio signals from the user equipment (or other mobile devices/apparatus). The audio track synchroniser/creator 153 can then be configured to synchronise the received audio data streams such that the audio signals are synchronised with each other (in other words aligned by suitable means in order to enable an audio based tempo or BPM estimator to estimate the BPM or tempo of the audio signals from the event). Any suitable means for synchronising the audio signals such as for example segment correlation synchronisation where segments of audio signals are cross-correlated to determine a time difference between the audio signals.

The synchronised audio signals can then be passed to an audio-based BPM estimator 155.

The operation of synchronising the audio data is shown in Figure 4 by step 305.

In some embodiments the audio server comprises a crowd motion based BPM estimator 151. The crowd motion based BPM estimator 151 is configured to receive the positional or accelerometer data from the user equipment and generate motion or petlodicity based tempo or 8PM esUmate. This motion based 6PM or tempo estimate can in some embodiments be output and used as a 6PM or tempo estimate or in some embodiments the motion based tempo or 6PM estimate can be passed in some embodiments to an audiobased 8PM estimator 155 to assist in the estimation of the tempo or 8PM using audiobased estimation techniques.

In some embodiments the crowd motion based 6PM estimator 151 comprises a periodicity estimator 161 configured to receive the location data from the user equipment and determined to generate individual motion based tempo or 6PM estimates. In the example shown in Figure 1 each apparatus accelerometer or positional data stream is passed to a separate periodicity estimator (in other words however it would be understood that in some embodiments the periodicity estimator is a single device suitable for receiving the accelerometer or positional data from each user equipment or apparatus and performing an individual periodicity estimate for each apparatus. In the example shown in Figure 1 there are shown live periodicity estimators, a first periodicity estimator 161 configured to receive the positional data from the first apparatus 191 and generate an individual motion periodicity estimate for the first apparatus 19, a second periodicity estimator 1812 configured to receive the accelerometer data from the second apparatus 192 and generate an individual motion petiodictty estimate for the second apparatus 192, a third periodicity estimator 1613 configured to receive the positional data from the third apparatus 19:3 and generate an individual motion periodicity estimate for the third apparatus 18, a fourth periodicity estimator 1614 configured to receive the accelerometer data from the fourth apparatus 194 and generate an individual motion periodicity estimate for the fourth apparatus 194 and a fifth periodicity estimator 161 configured to receive the positional data from the firth apparatus 195 and generate an individual motion periodicity estimate for the fifth apparatus 19. These individual 6PM or tempo estimates can then in some embodiments be output to a joint BPM estimator 163. In some embodiments the periodicities estimator 161 is configured to receive the accelerometer input and determine a dominant motion periodicity.

The periodicity estimator can in some embodiments thus receive the sensor data of each user equipment and independently analyse the data to obtain periodidty information of the data. The concept as expressed herein is that as the user of the apparatus is dancing or shaking to the beat of the music, the sensor signal from that user's apparatus has a periodic component that matches the tempo, beat (or BPM) of the music. Furthermore the periodic component matching the beat of the music is Ukely to be the most dominant periodicity found in the signaL In some embodiments therefore the sensor data can be processed by applying a time to frequency domain transform such as fOr example by a Fast Fourier Transform (FFT). The output of the time4o-domain transform (the FFT) of the signal can then be analysed and the dominant frequency is determined or selected. In other words the frequency corresponding to the largest value in the magnitude of the FFT output can be output. The periodicity estimator can further comprise a frequency to RPM converter which receives the determined or selected frequency and converts the frequency to a 8PM value which is then output from the periodicity estimator. It would be understood that in sonic embodiments rather than applying a time to frequency domain converter such as a FFT any suitable methods for periodicity analysis such as ones based on the autocorrelation or generated autocorrelaUon can he used.

In some embodiments, the tempo might be ana&sed by subjecting a measured periodicity vector to knearest neighbour regression tempo estimation. A periodicity vector might be obtained by performing accent signal analysis on the magnitude of the accelerometer data. Accent signal analysis typicafly involves measuring the degree of spectral change in the signal. The accent signal might further be subjected to periodicity estimation, using, for example, the generalized autocorrelation fundilon (GACF). The periodicity vector obtained using the GACF might then be compared to a set of tempo annotated GACF periodicity vectors, and the tempo be determined using knearest neighbour regression.

The operation of estimating individu& periodicity is shown in Figure 4 by step 304.

In some embodiments the crowd moIionbased BPM estimator 151 comprises a joint 6PM esflmator 183. The joint 8PM estimator 163 can be configured to receive the individual 6PM or tempo estimates from the periodicity estimators 161 and generate a joint 6PM estimaUon, The joint 8PM estimation can be performed according to any suitable method.

For examp'e in some embodiments the joint 6PM estimator 163 can be configured to determine the joint 6PM estimate using a statistical analysis of the input individual 6PM or tempo estimates. For example in some embodiments the joint 8PM estimator 163 can be configured to output the mode of the received estimates in other words finding the most common 8PM in the set of indMdual 6PM estimates. In other embodiments other averaging approached can be employed by the joint 8PM estimator 163 such as finding the median (or mean) BPM in the set of individual BPM estimates. In some embodiments the joint 8PM estimator 163 can be configured to determine a joint BPM estimate based on a time averaged 8PM input. For example in some embodiments the joint 8PM estimator 163 can be configured to determine a periodicity vector from each of the periodicity estimation blocks and then calculate a median (or other average) based on the periodicity vectors, and perform a 8PM estimation on the median periodicity vector. In some embodiments rather than computing a median over all periodicity vectors the joint 6PM estimator 163 can be configured to apply clustering methods such as Kmeans clustering to obtain N representative periodicity vectors, and the 8PM estimation could be performed on the clusters.

Thus for example a final joint 6PM could be obtained as an average (mean or median) 6PM over the cluster 6PM estimates, or as the most common (mode) 6PM based on the cluster 8PM estimates.

The joint 8PM estimate can then in some embodiments be output as a motion based 8PM estimate or be passed to an audiobased 8PM estimator 155 to assist the audiobased 6PM or tempo estimate, The operation of generating a joint BPM estimation from individual periodicity estimates is shown in Figure 4 by step 306.

It would be understood that In some embodiments the joint 6PM estimator in some embodiments can be configured to receive the IndMdual sensor signals to determine how correlated the individual signals are to each other. in such embodiments the joint 8PM estimation can be configured to first find the lamest group of users that is moving In synchronisation with each other (in other words have accelerometer signals that are correlated). In some embodiments this can be performed where the received signals have timing information for the accelerometer signals. It would be understood that in some embodiments the timing does not have to be exactly precise as in some embodiments the correlation can be performed over a relatively long period of time (for example one minute). In some embodiments the correlation can be calculated by calculating the cross-correlation between the signals at a time difference determined by the timing Information.

In some embodiments the audio server 109 comprIses an audio-based 8PM estimator 155. The audio-based 6PM estimator 155 can receive the audio signals from the audio tracks synchroniser/creator 153 and perform any suitable audIo-based estimation to determine the tempo or 8PM estImation. Furthermore in some embodiments the BPM or tempo estimate from the motion estimator can be used to assist in the generation of the audio-based 8PM estimation. Note that in some embodiments there is a single audio track created by the audio track synchronizatIon/creation module 153. In some embodiments, the audio tracks obtained from each user device are analysed Independently.

For example in some embodiments the obtained crowd motion based 8PM or tempo estimate can be used as a prior for the audio-based periodicity estimate. In other words that when the audio-based perlodicity estimator Is looking for the periodicity corresponding to the 8PM, It can be configured to select a perlodicity with a larger probability which corresponds to the 8PM estimated from the crowd motion.

Furthermore in some embodiments the audiobased 8PM estimator 155 is configured to perform 8PM estimation implementing knearest neighbour regression using training data. In these embodiments the 8PM estimate can be calculated as a weighted average of, for example, five closest periodicity vectors on the training data, In such embodiments the crowdsourced periodicity 8PM estimates can be used to increase the weight of corresponding 8PM estimates.

The operation of generating the audiobased 8PM estimate based on the motion joint BPM estimate is shown in Figure 4 by step 307.

With respect to Figures 5 and 6 further application of the motion based estimates are described. In some embodiments the crowd based 8PM estimator 151 further comprises a periodicity distribution determiner 165, such as shown in Figure 1.

The periodicity distribution determiner 165 can be configured to receive the individual periodicity estimates from the individual periodicity estimators.

The operation of receiving individual periodicity estimates is shown in Figure 5 by step 401.

The periodicity distribution determiner 165 can be configured to then determine a periodicity distribution (or 8PM distribution or tempo distribution). This distribution can then be used to determine how synchronised or coherent the motion of the crowd is.

The operation of determining periodicity distribution is shown in Figure 5 by step 403.

For example in some embodiments the periodicity distribution determiner can determine from the periodicity distribution determiner 165 a dominant periodicity and furthermore the frequency (such as expressed as the percentage) of the dominant periodicity value. This frequency (percentage) dominant periodicity value can for example be used to indicate a hotness' Index to be relayed to the artist or visualised and shown to the audience.

The operation of outputting a percentage dominant periodicities hotness rating to be relayed to the artistMsualised and shown to the audience is shown in Figure 5 by step 405.

With respect to Figure 6 two example visualisatlons can be shown. In the first vlsualisatlon 501 the perlodicity distribution determiner 165 determines that the dominant periodiclty is one which has a relatively low percentage of the number of apparatus in other words only a small number of the users are moving in time with the music (or in other words the users experiencing the event are not moving as one'). In such an example the visualisation can show that the hotness' is weak or mild. A second vlsualisatlon 503 shows where the percentage of the dominant perlodiclty is high (a large number for example over 90% of the people experiencing the event are dancing/moving to the same beat or as one) in which case the visualisatlon indicates that the hotness is sizzling hot or very hot'.

In some embodiments, the system further provides feedback to the clients, where the feedback provided for a particular client depends on whether it has been determined by the audio server 109 to be in synchrony of the most common tempo of the event, That is, the audio server 109 may signal a client apparatus that it is in synchrony of the most common tempo of the event. Determination whether a client is in synchrony with the most common tempo can be detentiined by analysing whether the tempo analysed from the positional data captured from the client matches with the most common tempo analysed from the audio and positional sIgnals from several clIents. When a client receives signalling from the audio server 109 that it Is in synchrony with the most common event tempo, It can further create some visualization in its screen. In some embodiments, the visualizatIon may be related to showing a certain colour on the display. In some embodiments, the visualization may be related to providing a visualization which changes In synchrony to the most common tempo of the event. To create a visualization which Is in synchrony to the most common tempo, the audio analysis server 109 either sends a synchronization signal indicating the desired times of the visualization changes (such as blinking the screen) or the client apparatus continues to analyse the tempo and beat at the client using the posiflonal data or audio data or both (which should match the global tempo as the client was determined to be in synchrony with the global tempo). As a result, the partidpants in the event wili see a visualization where the displays of aU users in the crowd who are in the sync of the music wili show some visualizations, such as blinking at the beat of the music.

In some embodiments, signaRing may be sent to devices which are not in sync with the most common tempo. Such signaling may include, for example, time syncing signals indicating the most common tempo, which may further cause a vibra or other tactile pattern in the signal receiving client to be created. Such feedback may help the user of the client which is out of sync of the global tempo to be able to start moving to the global tempo and beat, and thus to get synchronized to the crowd movement. As the user continues to move with a different pace, eventually synchronizing his movements to the gbbal tempo and beat, the audio analysis server 109 may notice this and send a signal indicating the synchrony to the client. As a result, the client may start to show the visualization which changes according to the most common tempo and beat of the event.

The foregoing description has provided by way of exemplary and non-limiting examples a full and informative description of the exemplary embodiment of this invention. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings.

Although the above has been described with regards to audio signals, or audio-visual signals it would be appreciated that embodiments may also be applied to audio-video signals where the audio signal components of the recorded data are processed in terms of the determining of the base signS and the determination of the time alignment factors for the remaining signals and the video signal components may he synchtnised using the above embodiments of the invention, In other words the video parts may be synchroriised using the audb synchronisation information.

It shaH be appreciated that the term user equipment is intended to cover any suitable type of wireless user equipment, such as mobile telephones, portable data processing devices or portable web browsers.

Furthermore &ements of a pubilc and mobUe network (FLMN) may also comprise apparatus as described above.

In general, the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof.

For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controfler, microprocessor or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is weil understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

The embodiments of this invention may be implemented by computer software executable by a data processor of the mobUe device, such as in the processor entity, or by hardware, or by a combination of software and hardware. Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and opUcal media such as for example DVD and the data variants thereof, CD.

The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitab'e to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific iniegrated circuits (ASIC), gate level circuits and processors based on multi-core processor architecture, as non-limiting examples.

Embodiments of the inventions may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

Programs, such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, CaHfornia automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre-stored design modules. Once the design for a semiconductor circuit has been completed, the resultant design, in a standardized electronic format (e.g., Opus, GDSH, or the like) may be transmitted to a semiconductor fabrication facUlty or "fab" for fabrication, The foregoing description has provided by way of exemplary and nonlimiting examples a full and informative description of the exemplary embodiment of this invention. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims.

However, all such and similar modifications of the teachings of this invention will still fall within the scope of this invention as defined In the appended claims.

Claims

CLAIMS: I, Apparatus comprising at east one processor and at east one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at east one processor cause the apparatus to: determine estimates of periodicity based on positional data associated with at least two apparatus experiencing the same audio event; determine a joint tempo estimate based on the estimates of periodidty associated with the at least two apparatus.
2. The apparatus as claimed in claim 1, further caused to: determine at least one audio based tempo estimate, wherein the at least one audio based tempo estimate is further based on the joint tempo estimate.
3. The apparatus as claimed in daim 2, wherein determining at east one audio based tempo estimate causes the apparatus to: synchronise at least two audb signals associated with the at least two apparatus experiencing the same aucho event; determine the at least one audio based tempo estimate based on the synchronised at least two audio signals further based on the joint tempo estimate.
4. The apparatus as claimed in claim 3, further caused to receive the at least two audio signals from the at east two apparatus experiencing the event.
5. The apparatus as claimed in claim 4, wherein the at least two audio signals comprise the positional data associated with the at east two apparatus experiendng the same audio event.
6. The apparatus as claimed in any of claims Ito 5, further caused to receive the positional data associated with the at least two apparatus experiencing the same audio event.
7. The apparatus as claimed in any of claims I to 6, further caused to determine a periodicity distribution from the estimates of periodicity associated
8. The apparatus as claimed in claim 7, further caused to determine a dominant periodicity estimate based on the periodicity distribution.
9. The apparatus as claimed in claim 8, further caused to display the dominant periodicity estimate based on the perlodicity distribution to at least one ot an event audience; an event creator.
10. The apparatus as claimed in any of claims 8 and 9, further caused to transmit the dominant periodicity estimate to at least one of the at least two apparatus experiencing the same audio event.
11. The apparatus as claimed in any of claims I to 10, is least one of the at least two apparatus experiencing the same audio event.
12. The apparatus as claimed in any of claims I to 11, wherein determining estimates of perlodicity based on positional data associated with at least two apparatus experiencing the same audio event causes the apparatus to receive the estimates of perlodicity based on positional data from the at least two apparatus experiencing the same audio event.
13. An apparatus comprising at least one processor and at least one memory Including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to: determine positional data; transmit the positional data to a further apparatus, wherein the apparatus is one of at least two apparatus experiencing the same audio event.
14. The apparatus as claImed in claIm 13, further caused to: capture at least one audio signal based on the audio event; transmit the audio signal to the further apparatus.
15. The apparatus as claimed in claIm 14, wherein transmitting the audio signal to the further apparatus cause the apparatus to: multiplex the positional data into the at least one audio signal; transmit the at least one audio signal comprising the positional data to the further apparatus.
16. The apparatus as claimed in any of claims 13 to 15, further caused to receIve from the further apparatus a dominant perlodicity estimate based on a determination of the perlodicity distribution of the positional data to at least one of the at least two apparatus experiencing the same audio event.
17. The apparatus as claimed in claim 16, further caused to determine a difference between the dominant peilodicity estimate and a periodlclty estimate of the apparatus.
18. A method comprising: determining estimates of pedodicity based on positional data associated wIth at least two apparatus experiencing the same audio event; determinIng a Joint tempo estimate based on the estimates of periodicity associated with the at least two apparatus.
19. A method comprising: determIning positional data; transmitting the positional data to a further apparatus, wherein the method Is performed on one of at least two apparatus experiencing the same audio event.and wherein the further apparatus performs a method comprising determining estimates of periodicity based on positional data associated with at least two apparatus experiencing the same audio event; and determining a joint tempo estimate based on the estimates of perlodicity associated with the at least two apparatus.
20. An apparatus comprising: means for determining estimates of periodicity based on positional data associated with at least two apparatus experiencing the same audio event; means hr determining a Joint tempo estimate based on the estimates of periodiclty associated with the at least two apparatus.
21. An apparatus comprising; means for determining positional data; means hr transmitting the positional data to a further apparatus, wherein the apparatus is at least one of the two apparatus experiencing the same audio event, and wherein the further apparatus comprises means for determining estimates of peiiodlclty based on positional data associated with at least two apparatus experiencing the same audio event; and means for determining a joint tempo estimate based on the estimates of perlodicity associated with the at least two apparatus.
22. An apparatus comprising: at least one pesiodicity estimator configured to determine estimates of perlodicity based on positional data associated with at least two apparatus experiencing the same audio event a Joint tempo determiner configure to determine a Joint tempo estimate based on the estimates of peilodicity associated with the at least two apparatus.
23. An apparatus comprising; a sensor configured to determine positional data; an output configured to transmit the positional data to a further apparatus, wherein the apparatus is at least one of the two apparatus experiencing the same audio event, and wherein the further apparatus comprises at least one perlodicity estimator configured to determine estimates of pertodicity based on positional data associated with at least two apparatus experiencing the same audio event; a Joint tempo determiner configure to determine a joint tempo estimate based on the estimates of periodicity associated with the at least two apparatus.