GB2465841A - Identifying audio output by comparing samples with the contents of a database - Google Patents

Abstract

A computer has a sampler connected to its audio output. The sampler transmits the samples to a database server. The server compares the samples to those in the database in order to identify the audio output. The audio samples may be taken from an audio output of the computer, such as a loudspeaker or headphone socket. They may be taken from the sound card. Alternatively, the samples may be extracted by the BIOS or operating system of the computer. The samples may be sent via the Internet. The samples may be stored on the computer before being transmitted.

Description

INTELLECTUAL

. .... PROPERTY OFFICE Application No. GB0822262.2 RTM Date:1 March 2010 The following terms are registered trademarks and should be read as such wherever they occur in this document: Blackberry Windows Real Player SecurlD Intellectual Property Office is an operating name of the Patent Office www.ipo.gov.uk Monitoring Computer Usage The present invention relates to methods and apparatus for monitoring usage of computing devices.

In the context of the present application the term computing devices comprehends, but is not restricted to, PC's, laptops, notebooks, netbooks, FDA's, other mobile computing devices such as mobile telephony devices, BlackBerrylsTM, games devices, and media players.

There are many known legitimate ways (and non-legitimate ways) in which usage of computing devices can be monitored and logged. For example PC usage can be monitored by techniques such as logging web page links clicked on or by using cookies. These techniques could provide information on sites visited, downloads, etc..

Very simple examples of legitimate monitoring might also be the logging or caching of information on (recently) used files or programs, e.g. by computer operating systems such as WindowsTM, or logging or caching of information on (recently) visited websites, e.g. by web browsers, or logging or caching of information on (recently) played audio (e.g. mp3's) or video, e.g. in player software such as Real PlayerTM etc. In this way it could be considered that a quite complete record of the user's use of a computing device could readily be made available.

The inventor has had the insight, however, that it is unlikely that a user would grant even legitimate third parties access to such records, which could affect the integrity of the records and hence the stability of the computer. At least parts of such records might be considered to be private.

The inventor has also had the insight that a further issue may be that such records, if transmitted across a network such as the Internet, could be captured by non-authorised parties and used to monitor the behaviour of the user of the computing device and to identify the user, at least to some extent.

Thus, the inventor has had the insight that computing device usage information should be derived and, if applicable, delivered across a network such as the Internet, in an "anonymous" manner, such that preferably the identity of the user, and preferably also the significance of the transmitted data, is obscured from a non-authorised person capturing the delivered information.

The inventor has further had the insight that another issue could be the volume of information to be transmitted across a network such as the Internet, which the inventor has realised should be as small as possible, inter alia to minimise risk of unauthorised information capture.

The present invention provides that only a specific class of information, i.e. audio related, concerning the user's computing device usage is intercepted in the computing device itself and is then transmitted across a network, e.g. the Internet, only in a compressed form, e.g. as audio signatures, the significance of which is obscure to an unauthorised information capturer, and which is of little volume.

Although only audio related information is intercepted and only this information may be transmitted across a network, e.g. the Internet, this information can provide invaluable usage information, especially when collected from a plurality of PC's or other computing devices.

Embodiments of the present invention can provide this invaluable usage information without access to log or cache information normally held on the computer and without interacting in its monitoring activity with other software on the computer, such as web browsers or media players. Integrity of normal programs on the computer is not affected.

In an embodiment of the present invention there is monitored only an output audio signal, for example delivered to the headphone output/sound card input of the computer. The embodiment of the invention is unaware of and does not log the source (e.g. a web page or a media file) from which the audio originates. The embodiment monitors only the "anonymous" output audio signal.

The embodiment of the present invention does not record the "anonymous" audio signal as such. It records, and thus monitors, only "cryptic" samples derived from the audio. These "cryptic" samples (which may be called audio signatures) contain sufficient information for the audio to be identified by an authorised third party to whom the samples are sent, but only if the third party has a reference library of similarly "cryptic" samples, or "audio signatures", derived in the same way, from a catalogue of possible original audio sources.

Thus the monitored information does not rely on access to "personal" log or cache information on the computer, and even if non-legitimately captured when downloaded (e.g. via the Internet) reveals essentially nothing other than cryptic, meaningless, sample information.

In this way a user is able to share monitoring information with a third party (e.g. via the Internet) in an agreed manner, with a high degree of security, with low bandwidth requirement etc. Reference is made to the accompanying drawings which show:-Figure 1 is a schematic illustration of a computing device adapted in accordance with the present invention for providing usage information, and Figure 2 is a schematic illustration of an authorised collector/reference site of a system in accordance with the present invention.

As illustrated in Figure 1, a computing device 100, for example a PC, laptop, notebook, netbook, FDA, other mobile computing device such as a mobile telephony device, BlackBerryTM, games device, or media player, typically has a main board 110 and a sound card 120 providing an audio output 125. The sound card may, of course, be integrated in the main board.

In a computing device adapted in accordance with the present invention, an audio interceptor 130 taps the audio delivered to the audio output 125, or the audio signal from the main board 110 to the sound card 120.

The audio interceptor 130 may be software embedded in the computing device BIOS, the computing device operating system or may be audio interceptor application software run on the computing device operating system. The audio interceptor may also be embodied in an ASIC or other dedicated hardware component of the computing device.

The use of an ASIC or other dedicated hardware component, or BIOS embedded software, may be preferable since then the functioning of the operating system may be essentially unaffected and the burden of running audio interceptor application software on the computing device operating system may be avoided.

Although the use of an ASIC or other dedicated hardware component, or BIOS embedded software, may involve customisation of the PC or other computing device, this may be of advantage in ensuring that usage information is reliably provided without being affected for example by other software running on the operating system, and without interference by the user of the device.

Provision of a suitably customised PC or other device to the user at no charge or at a reduced charge compared to a normal PC etc. may encourage users to allow collection of usage information.

The output of the audio interceptor 130 may be delivered to local storage 140 and thence, e.g. at intervals, to a download means 150 via which it can be downloaded to a WAN, e.g. the Internet, 155.

Alternatively, the output of the audio interceptor 130 may be delivered directly to download 155.

The audio interceptor 130 samples the intercepted audio, generating audio signatures. These samples or signatures may represent only a very small fraction of the audio sampled, such as to be essentially meaningless until compared to a database of similarly derived samples or signatures at the download destination.

The precise method of derivation of the samples or signatures is not relevant, so long as a sequence of samples or signatures can be identified by comparison with a sequence of "reference" samples or signatures in the database. Conventional sampling or signature deriving techniques may be used.

Transmission of the samples or signatures requires little bandwidth, and the samples or signatures are per se essentially meaningless (without the comparison "reference" samples or signatures in the database) and therefore do not expose the provider of the samples to the risk of unauthorised monitoring, or reduce such risk significantly.

The samples or signatures represent "anonymous" audio, i.e. provide no indication of the source (e.g. an mp3, file, a DVD, the internet etc.) and therefore further protect the privacy of the user, whilst provithng the authorised collector of the information (the database owner) with useful information.

Tapping the audio delivered to the audio output 125 -for example a PC headphone output or an output for use by an external amplifier -means that if the user has turned off sound from the PC or other computing device only silence is tapped. Sampling can thus be suspended, for example after a predetermined period of silence.

Sampling can be restarted when sound resumes. In some cases this may be of advantage, for example to reduce data volumes and/or provide a record more closely reflecting what is heard by the user of the PC or other computing device.

On the other hand, tapping the audio signal at a point on the path from the main board 110 to the sound card 120, or at some other point prior to de:Livery to audio reproduction -such that the user turning off sound from the PC or other computing device does not affect the audio signal -may in other cases be of advantage.

Tapping at such a point can provide a record of audio accessed or accessible to the user, even if the user chooses not to listen to the audio at some times.

The audio sampling may manifest a degree of "intelligence", for example excluding from the record what could be called "system" sounds. For example, standard sounds associated with the operating system or the starting, ending of software applications, may be recognised at the sampling stage and excluded from recording or download. Thus, for example, samples corresponding to WindowsTM jingles can be excluded from recording or download to reduce data volumes and/or provide a record of more relevance to user audio selections and thus usage.

In case where the computing device is a telephony device, telephone calls may be excluded from sampling, to further protect user privacy and reduce data volumes. Of course, even if telephone calls are sampled the resulting sequence of samples or signatures could not normally be identified by comparison with a sequence of "reference" samples or signatures in a database, because such a database would not exist for what is essentially random spoken-word telephone call content. It would be possible, however, to maintain a database of samples or signatures of some spoken phrases which may thus possibly be identifiable.

The samples or signatures may be time stamped if needed, e.g. if stored for later transmission via the Internet. The samples or signatures may be "streamed" essentially in real time or at frequent intervals to the Internet or other network via which the samples or signatures are delivered the authorised collector of the information.

The samples or signatures may also include an identification code indicating the PC or other computing device from which the samples or signatures are provided. When such an ID code is included it may be a variable code changing at intervals in an apparently random manner which can be tracked only by the authorised collector of the samples or signatures.

Techniques for providing this are known in the art, an example being the SecurlD'TM system provided by RSA.

However, in some embodiments of the invention, inclusion of an ID code may be omitted. In such a case the samples or signatures may be sent to the Internet, for example, at a specific time. They may also include a time code indicative of the sending time. In such a case, the specific time effectively indicates to the authorised collector of the information the specific PC or other computing device from which the samples or signatures are provided. The authorised collector knows when to "listen" for samples or signatures from a specific PC for example, with different PC's or computing devices being assigned different time slots for sending samples or signatures. This is also advantageous in equalling out the load on the authorised collector.

In other embodiments, the samples or signatures may be sent to a specific network or Internet address, or for example to a specific port at that address, the address and/or port being used only for the samples or signatures from a specific PC or other computing device. Other PC's or computing devices supplying samples or signatures to the authorised collector use different addresses/ports.

The samples or signatures, and any associated information, sent to the network/Internet for delivery to the authorised collector may be encrypted.

In some embodiments of the present invention the PC or other computing device may build, with the authorised collector, a VPN for the transmission of the samples or signatures, and any associated information, to the authorised collector.

However, given the nature of the samples or signatures transmitted, additional security measures such as encryption or VPN building may be considered superfluous.

Figure 2 schematically illustrates an authorised collector/reference site 200 of a system in accordance with the present invention. The authorised collector receives via a NAN, e.g. the Internet, 155 samples or signatures from a plurality of computing devices 100 such as PCi, P02, P03.

The system operator may have available demographic information concerning the users of the computing devices 100, or the households in which those devices are used. This demographic data can be used for analysing collected usage information (samples/signatures) to provide audience information related to the collected audio usage information.

Samples/signatures from the computing devices 100 are delivered from the device first to an identifier 210, which identifies from which device, e.g. PCi, P02 or P03, the samples/signatures originate. This identification can be effected as explained above, for example based on ID information associated with the received samples/signatures, or time of receipt of the samples/signatures etc., or a combination of such techniques.

As necessary, the received samples/signatures are extracted from any superfluous associated information in a sample extractor 220, from which they are passed to a comparator 240 which effects or attempts to match samples from the extractor 220 with samples from a sample reference library or database 230. Received samples may of course be stored before being subjected to comparison with reference samples in the database 230. Unmatched or unidentified samples may also be stored for later examination and analysis. For example, time-stamps may aid in the matching attempts.

Information derived from matched samples, for example the audio source from which the samples are considered to originate (e.g. a particular song or Internet radio program or Internet video played by the computing device 100 from which the samples have been received) can then be stored in an audio usage record 250 in association with the identity of the computing device 100, e.g. (PC) device 1, 2 or 3.

This information, relating to a plurality or panel of PC's or other computing devices, possibly together with related demographic information, can be used to establish audience analysis information.

Thus, the present invention provides a system affording PC (or other computing device) based audio matching. To monitor what a PC user is listening to (or watching), software or hardware in the PC samples the PC's audio for audio matching with a sample database. The collected data is sent by the PC (typically via internet) back to base where the actual matching process is done (comparing to known references, typically TV/radio broadcasts captured at a reference site but could include libraries of known material such as movie DVDs)

Claims (11)

1. Claims 1. A system including a plurality of computing devices each comprising an audio output sampler and means operable to store for transmission and/or to transmit the audio samples provided by the sampler, and a reference site operable to receive audio samples from the plurality of computing devices and to match the samples with a reference library of stored samples.
2. 2. A method including operating a plurality of computing devices, each comprising an audio output sampler and means operable to store for transmission and/or to transmit the audio samples provided by the sampler, and operating a reference site to receive audio samples from the plurality of computing devices and to match the samples with a reference library of stored samples.
3. 3. A system as claimed in claim 1, or a method as claimed in claim 2, wherein the audio output samplers comprised in the computing devices are operable to derive the audio samples from audio signals at a hardware audio output of the computing device, for example the hardware audio output being an audio output or loudspeaker or headphone output of the computing device, or the hardware audio output to a sound card of the computing device.
4. 4. A system or a method, as the case may be, as claimed in claim 1, 2 or 3, wherein the computing devices are operable to transmit the audio samples to the Internet, and the reference site is operable to receive the audio samples from the Internet.
5. 5. A system or a method, as the case may be, as claimed in claim 1, 2, 3 or 4, wherein the audio samples lack identification of the computing device from which they originate.
6. 6. A computing device for use in a system or method, as the case may be, as claimed in any of claims 1 to 5, comprising an audio output sampler and means operable to store for transmission and/or to transmit audio samples provided by the sampler.
7. 7. A computing device as claimed in claim 6, being a PC, laptop, notebook, netbook, PDA, other mobile computing device such as a mobile telephony device, BlackBerryTM, games device, or media player.
8. 8. A computing device as claimed in claim 6 or 7, wherein the audio sampler is embedded in the computing device BIOS, the computing device operating system or in audio interceptor application software run on the audio interceptor, or is embodied in an ASIC or other hardware component of the computing device.
9. 9. A computing device substantially as described herein with reference to the accompanying drawings.
10. 10. A system substantially as described herein with reference to the accompanying drawings.
11. 11. A method substantially as described herein with reference to the accompanying drawings.