[go: up one dir, main page]

WO2009053869A2 - Method and mobile device for detecting highlights - Google Patents

Method and mobile device for detecting highlights Download PDF

Info

Publication number
WO2009053869A2
WO2009053869A2 PCT/IB2008/054188 IB2008054188W WO2009053869A2 WO 2009053869 A2 WO2009053869 A2 WO 2009053869A2 IB 2008054188 W IB2008054188 W IB 2008054188W WO 2009053869 A2 WO2009053869 A2 WO 2009053869A2
Authority
WO
WIPO (PCT)
Prior art keywords
highlight
mobile device
module
information
determining
Prior art date
Application number
PCT/IB2008/054188
Other languages
French (fr)
Other versions
WO2009053869A3 (en
Inventor
Gang Wang
Sheng Jin
Mo Li
Original Assignee
Koninklijke Philips Electronics N.V.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics N.V. filed Critical Koninklijke Philips Electronics N.V.
Publication of WO2009053869A2 publication Critical patent/WO2009053869A2/en
Publication of WO2009053869A3 publication Critical patent/WO2009053869A3/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/16—Analogue secrecy systems; Analogue subscription systems
    • H04N7/173—Analogue secrecy systems; Analogue subscription systems with two-way working, e.g. subscriber sending a programme selection signal
    • H04N7/17309—Transmission or handling of upstream communications
    • H04N7/17318—Direct or substantially direct transmission and handling of requests
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
    • H04N21/2387—Stream processing in response to a playback request from an end-user, e.g. for trick-play
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/27—Server based end-user applications
    • H04N21/274—Storing end-user multimedia data in response to end-user request, e.g. network recorder
    • H04N21/2747—Remote storage of video programs received via the downstream path, e.g. from the server
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/414—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
    • H04N21/41407—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42201—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42203—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] sound input device, e.g. microphone
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
    • H04N21/4331—Caching operations, e.g. of an advertisement for later insertion during playback
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/44213—Monitoring of end-user related data
    • H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
    • H04N21/4622—Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/488—Data services, e.g. news ticker
    • H04N21/4882—Data services, e.g. news ticker for displaying messages, e.g. warnings, reminders
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/65—Transmission of management data between client and server
    • H04N21/658—Transmission by the client directed to the server
    • H04N21/6587—Control parameters, e.g. trick play commands, viewpoint selection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04H—BROADCAST COMMUNICATION
    • H04H60/00—Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
    • H04H60/29—Arrangements for monitoring broadcast services or broadcast-related services
    • H04H60/33—Arrangements for monitoring the users' behaviour or opinions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04H—BROADCAST COMMUNICATION
    • H04H60/00—Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
    • H04H60/61—Arrangements for services using the result of monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
    • H04H60/66—Arrangements for services using the result of monitoring, identification or recognition covered by groups H04H60/29-H04H60/54 for using the result on distributors' side

Definitions

  • the present invention relates to a method and mobile device for detecting highlights.
  • big screens are provided in different places on the sports field to provide a live broadcast of the match and play the exciting moments, or highlights.
  • big screens still have some shortcomings, since not everybody can watch them easily. People positioned at a bad viewing angle with respect to a big screen (e.g., seat just beside the big screen) have difficulty watching it.
  • TV-on-mobile like DVB-H and H264(digital video encoding standards developed by the joint video team of the VCEG of ITU-T and MPEG of ISO/IEC), people sitting at a stadium can watch the same TV programs on their mobile phones as on a conventional TV at home . For example, while watching a football match in a football stadium, people can also watch, on their mobile phone, the live TV broadcast of the football match, which in the past was only broadcast for home TV reception.
  • the proposed apparatus is based on such a TV-on-mobile enabled mobile phone.
  • the object of the invention is to overcome the above mentioned deficiencies by means of the following method and apparatus.
  • the present invention provides a method of detecting a highlight by a mobile device, and the method comprises the steps of: playing digital program signals on the mobile device, collecting the reaction information of the user through a sensor embedded in the mobile device, and determining the emergence of a highlight according to the detected reaction information.
  • the invention also provides a method of playing the highlight determined by the method as mentioned above.
  • the invention further provides a mobile device for detecting a highlight, and the mobile device comprises: a digital program receiver module, a sensor module for collecting reaction information of a user, and a processing module for determining the emergence of a highlight according to the detected reaction information.
  • the present invention provides a system for playing a highlight, the system comprising: a mobile device used to send a request for acquiring the highlight according to any one of claims 9 to 11, and a server for receiving the request and sending the highlight to the mobile device according to the request.
  • the advantage of the invention is to provide more intelligent personalized program to audience.
  • Figure 1 depicts a schematic diagram of a mobile device according to one embodiment of the invention
  • Figure 2 depicts a schematic diagram of a mobile device according to another embodiment of the invention.
  • Figure 3 is a flow chart of a method for detecting highlight according to one embodiment of the invention.
  • the mobile device comprises three modules, the mobile device can be a mobile phone, a PDA (Personal Digital Assistant), a laptop etc.
  • the three modules comprise a digital program receiver module 13 having a mobile TV function, a sensor module 11 and an information receiver module 12.
  • the digital program receiver module 13 can receive a TV program, for example, either a broadcast network (e.g., DVB-H) or a mobile network (e.g., 3G). Taking DVB-H as an example, the input of this module is a DVB-H transport stream. After processing, the output of the digital program receiver module 13 is RTP packages with H264 AV content as the payload. These RTP packages can be sent to a video player 19 and the RTP packages are unpacked and the H264 AV is decoded. The output of the digital program receiver module 13 may also be put into a ring buffer enabling a short period (e.g., three minutes) of a TV program to be temporarily stored for further processing. When the ring buffer is used to store a highlight, the time period should be longer than the predetermined time period of the highlight.
  • a broadcast network e.g., DVB-H
  • a mobile network e.g., 3G
  • the input of this module is a DVB-H transport stream.
  • the sensor module 11 comprises sensors of different types, for example, an audio sensor for collecting the audio information of the surrounding environment, e.g. the sound of applause or cheers, and /or an acceleration sensor for collecting sound information and action information of the user of the mobile device. All of the collected information will be sent to a processor module 14 for further analysis.
  • an audio sensor for collecting the audio information of the surrounding environment, e.g. the sound of applause or cheers
  • an acceleration sensor for collecting sound information and action information of the user of the mobile device. All of the collected information will be sent to a processor module 14 for further analysis.
  • Message receiver module 12 is a complementary module which receives some realtime messages released by service providers, for example, a message saying that Raul scored the equalizer for Spain in the 71 st minute. These real time messages can be associated with highlights (e.g. instantaneously during a match) and act as notes for the highlights by the processor module.
  • highlights e.g. instantaneously during a match
  • the generation of a highlight has been described in more detail in Chinese application No.200310122527.3 applied by the name of the current applicant, the whole content of which will be included in this application for reference.
  • the processor module 14 of the apparatus in Figure 1 is used to process the information collected by the sensor module 11 and information receiver module 12.
  • the processor module 14 can use some analysis algorithms to determine the highlights that may have emerged according to the information collected by the sensor module 11.
  • the processor module 14 can comprises a calculation module, a comparison module, and a determining module.
  • the processor module 14 can comprise a highlight detector module and a highlight recorder module.
  • the highlight detector module will keep on checking whether a possible highlight has emerged. Detection takes place mainly on the basis of the information collected by the sensor module 11. For example, the sensor module 11 collects sound information of the surrounding environment (e.g., the stadium) and sends it to the detector module. Sound effects like a cheer or applause may indicate an exciting moment.
  • the method of detecting such sound effects can be based on features of the collected sound signals, such as short time energy, which provides a convenient representation of the amplitude variation over time; or zero crossing rate, which gives a rough estimate of frequency content; sub-band energies that model the characteristics of spectral distribution more accurately, or a combination thereof.
  • the sound effect of applause/cheers has its own distinguishable features, which can be calculated by the calculation module and distinguished from other sound effects such as "shouting nearby" by the comparison module, so that the determining module does make an incorrect determination of a highlight. Based on an appropriate threshold, the algorithms can also distinguish the applause/cheer around the user from that of the user himself.
  • the sensor module 11 can also collect the motion information of the user taking the mobile device. If an accelerated upward motion is detected, it may indicate the user was suddenly standing up or raising his hands, which further indicates the user saw something exciting. For both cases, the past period of time (e.g., 2 minutes) of a
  • the detector module may also use the messages collected by the message receiver module 12 for highlight detection. If the message receiving system is in real-time, it is quite possible that the message receiver module can get a highlight message within the time capacity (e.g., 2 minutes) of the ring buffer.
  • the determining module determines that a highlight has emerged, then, in the situation that highlight recording is needed, the highlight detector module will instruct the highlight recorder module to start recording.
  • the recorded highlight, as well as some notes in respect of the highlight, will be stored in a highlight database. In this situation, storage capacity for highlights is provided on the mobile device.
  • the controller will cause the playing of the digital program to stop and sends the highlight to the player 19 to start playing when it is required to show the highlight.
  • the highlight recorder module When requested by the highlight detector module, the highlight recorder module first processes (part of) the data in the ring buffer 15, for example, unpacking the RTP packages to get H264 AV packages, and puts them into the highlight database as one highlight. Then, simple information (in the case that highlight is detected on the basis of the information collected by the sensor module) like time and match name, or some extended information (in the case the highlight is detected based on the messages received by the message receiver module) like "Raul scored the equalizer for Spain in the 71 st minute", can be added to the recorded highlight in the database as annotations by the highlight recorder module.
  • simple information in the case that highlight is detected on the basis of the information collected by the sensor module
  • some extended information in the case the highlight is detected based on the messages received by the message receiver module
  • the processor module 14 can also comprise a highlight post-processor module. Based on analysis of the messages collected by the message receiver module, the highlight postprocessor module updates the highlight database. The updating can be periodical (e.g., every 5 minutes) or triggered by newly received messages. The main updating operation is performed to update the annotations to the recorded highlights. Since most highlights are determined on the basis of the information collected by the sensor module 11, only some simple annotations are added to the recorded highlight when it is put into the database.
  • the processor module 14 may also not comprise the highlight recorder module. It means that the highlight recording is not necessarily performed on the mobile device, but may be done on a server (shown as "S" in Figure 2) instead. In this situation, the highlight database can be used just to store some records, for example, the emerging time of a highlight, etc. It is even not necessary to provide a highlight database and highlight storage on a mobile device.
  • highlight recording When highlight recording is performed on the server, it will be unnecessary to provide a ring buffer on a mobile device, as shown in Figure 2.
  • the mobile device determines, by means of the processor module 14, that a highlight has emerged, highlight information will be sent to the server by an information sending module 20.
  • the server will send to the mobile device a message asking whether or not a highlight will be watched.
  • the server Upon receiving a "yes" answer from the mobile device, the server interrupts the transmission of the digital program and starts to transmit the highlight. Transmission of the digital program will continue after the transmission of the highlight has ended.
  • the processor module can read a segment of a predetermined prior time period as the highlight directly from the ring buffer. Then the highlight is directly sent to the controller 18 so as to be played by the player 19. Or the controller 18 could provide a hint of a highlight to the user first, and instruct the player 19 to play the highlight upon receiving a "yes" reply.
  • the moment or time label of the highlight should be recorded so as to enable play of the highlight.
  • the mobile device can send a request to the server. Then the server searches for the moment or time label to locate the highlight in the recorded video of the match and transmits the highlight.
  • Figure 3 is flow chart of the method according to one embodiment of the invention.
  • the mobile device receives a digital program in step 201.
  • the digital program can be transmitted by a broadcast network (e.g. DVB-H) or a mobile network (e.g.
  • the reaction information of the user will be collected by the sensor(s) integrated in the mobile device. As mentioned above, it can be audio information or action information that is detected. It would be possible to detect other information, as corresponding sensors are provided in the mobile device, if needed.
  • the collected or detected information is calculated and quantified, e.g. the feature(s) of the audio or acceleration information are calculated. Then the calculated feature is compared with a predetermined threshold. It is determined whether the user has just had an exciting reaction, i.e. when the calculated feature is larger or equal to the threshold, e.g.
  • step 204 the highlight is played on the mobile device.
  • step 205 the highlight is played on the mobile device.
  • a flag can be added to the program stream. Then the highlight would be a segment of a predetermined time period before the flag.
  • the basic concept of the method comprises: 1 ) Using the information collected by the integrated sensor as main hints to detect the emergence of a possible highlight.
  • the recording of a highlight is triggered automatically.
  • the recorded highlights can be stored into a database, which can be updated by a postprocessor using messages sent by service providers.
  • the recorded highlights can be annotated using the received notifications, which helps users know easily what kinds of highlights have been recorded.
  • recording of the highlight is not a must. It is possible to have the server stores only the time/moments when the highlights emerge and the video of the whole match/event. When the server receives a request from a user to view a certain highlight, the server can read the highlight from the recorded video of the whole match according to the time/moment of the highlight and transmits the highlight.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • General Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Social Psychology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Analytical Chemistry (AREA)
  • Neurosurgery (AREA)
  • Chemical & Material Sciences (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • General Engineering & Computer Science (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Telephonic Communication Services (AREA)
  • Telephone Function (AREA)
  • Mobile Radio Communication Systems (AREA)
  • Position Fixing By Use Of Radio Waves (AREA)

Abstract

A method of detecting a highlight by a mobile device is provided. The method comprises the steps of: receiving (201) digital program signals on the mobile device, collecting (202) reaction information of a user through the mobile device, and determining (203) an emergence of a highlight according to the reaction information.

Description

METHOD AND MOBILE DEVICE FOR DETECTING HIGHLIGHTS
TECHNICAL FIELDS
The present invention relates to a method and mobile device for detecting highlights.
BACKGROUND
Generally, the live atmosphere cannot be achieved by watching TV or recorded videos. This is exactly why so many people choose to go to the actual live events. However, these too have their shortcomings.
Take watching a match, for example; since many people sit far away from the sports field (close seat ticket is expensive), it's difficult for them to get a clear view of the game, especially of exciting moments. For example, when there is a goal in a football match, many spectators can't see very clearly who scored it and how it was scored. Or, in a swimming race, when the swimmers are approaching the finish line, it is hard for some spectators to see clearly who the final winner is. It will be worse for the spectators having a bad seat.
To overcome such problems, big screens are provided in different places on the sports field to provide a live broadcast of the match and play the exciting moments, or highlights. However, big screens still have some shortcomings, since not everybody can watch them easily. People positioned at a bad viewing angle with respect to a big screen (e.g., seat just beside the big screen) have difficulty watching it.
The most important point is that even the big screens as provided on sports fields would play highlights which are not chosen by the spectators themselves. That is to say, the spectators may not be interested in the highlights being played. While the highlights in which the spectators are interested are not being played. This would be annoying to the spectators watching the event live.
On the other hand, with the rapid increase of the mobile phone market, more and more users are using their mobile phones not only for communication purposes but also for entertainment and information services. For example, TV-on-mobile, i.e., watching TV on your mobile phone, is expected to become a key mobile application in the near future. Mobile phones are also becoming the platforms of mobile context-awareness applications by integrating sensors in them. For example, through embedded accelerometer sensors, a mobile phone can detect the motions of its owner; through embedded sound sensors, it can detect the acoustic information of the surrounding environment.
Based on enabling technologies of TV-on-mobile, like DVB-H and H264(digital video encoding standards developed by the joint video team of the VCEG of ITU-T and MPEG of ISO/IEC), people sitting at a stadium can watch the same TV programs on their mobile phones as on a conventional TV at home . For example, while watching a football match in a football stadium, people can also watch, on their mobile phone, the live TV broadcast of the football match, which in the past was only broadcast for home TV reception. The proposed apparatus is based on such a TV-on-mobile enabled mobile phone.
CONTENTS OF THE INVENTION
The object of the invention is to overcome the above mentioned deficiencies by means of the following method and apparatus.
The present invention provides a method of detecting a highlight by a mobile device, and the method comprises the steps of: playing digital program signals on the mobile device, collecting the reaction information of the user through a sensor embedded in the mobile device, and determining the emergence of a highlight according to the detected reaction information.
The invention also provides a method of playing the highlight determined by the method as mentioned above.
The invention further provides a mobile device for detecting a highlight, and the mobile device comprises: a digital program receiver module, a sensor module for collecting reaction information of a user, and a processing module for determining the emergence of a highlight according to the detected reaction information.
The present invention provides a system for playing a highlight, the system comprising: a mobile device used to send a request for acquiring the highlight according to any one of claims 9 to 11, and a server for receiving the request and sending the highlight to the mobile device according to the request. The advantage of the invention is to provide more intelligent personalized program to audience.
DESCRIPTION OF FIGURES
Figure 1 depicts a schematic diagram of a mobile device according to one embodiment of the invention,
Figure 2 depicts a schematic diagram of a mobile device according to another embodiment of the invention,
Figure 3 is a flow chart of a method for detecting highlight according to one embodiment of the invention. DETAIL DESCRIPTION OF THE INVENTION
As shown in figure 1, the mobile device comprises three modules, the mobile device can be a mobile phone, a PDA (Personal Digital Assistant), a laptop etc. The three modules comprise a digital program receiver module 13 having a mobile TV function, a sensor module 11 and an information receiver module 12.
The digital program receiver module 13 can receive a TV program, for example, either a broadcast network (e.g., DVB-H) or a mobile network (e.g., 3G). Taking DVB-H as an example, the input of this module is a DVB-H transport stream. After processing, the output of the digital program receiver module 13 is RTP packages with H264 AV content as the payload. These RTP packages can be sent to a video player 19 and the RTP packages are unpacked and the H264 AV is decoded. The output of the digital program receiver module 13 may also be put into a ring buffer enabling a short period (e.g., three minutes) of a TV program to be temporarily stored for further processing. When the ring buffer is used to store a highlight, the time period should be longer than the predetermined time period of the highlight.
The sensor module 11 comprises sensors of different types, for example, an audio sensor for collecting the audio information of the surrounding environment, e.g. the sound of applause or cheers, and /or an acceleration sensor for collecting sound information and action information of the user of the mobile device. All of the collected information will be sent to a processor module 14 for further analysis.
Message receiver module 12 is a complementary module which receives some realtime messages released by service providers, for example, a message saying that Raul scored the equalizer for Spain in the 71st minute. These real time messages can be associated with highlights (e.g. instantaneously during a match) and act as notes for the highlights by the processor module. The generation of a highlight has been described in more detail in Chinese application No.200310122527.3 applied by the name of the current applicant, the whole content of which will be included in this application for reference.
The processor module 14 of the apparatus in Figure 1 is used to process the information collected by the sensor module 11 and information receiver module 12. The processor module 14 can use some analysis algorithms to determine the highlights that may have emerged according to the information collected by the sensor module 11. The processor module 14 can comprises a calculation module, a comparison module, and a determining module.
The processor module 14 can comprise a highlight detector module and a highlight recorder module. During the event (e.g., a football match), the highlight detector module will keep on checking whether a possible highlight has emerged. Detection takes place mainly on the basis of the information collected by the sensor module 11. For example, the sensor module 11 collects sound information of the surrounding environment (e.g., the stadium) and sends it to the detector module. Sound effects like a cheer or applause may indicate an exciting moment. The method of detecting such sound effects can be based on features of the collected sound signals, such as short time energy, which provides a convenient representation of the amplitude variation over time; or zero crossing rate, which gives a rough estimate of frequency content; sub-band energies that model the characteristics of spectral distribution more accurately, or a combination thereof. The sound effect of applause/cheers has its own distinguishable features, which can be calculated by the calculation module and distinguished from other sound effects such as "shouting nearby" by the comparison module, so that the determining module does make an incorrect determination of a highlight. Based on an appropriate threshold, the algorithms can also distinguish the applause/cheer around the user from that of the user himself. The sensor module 11 can also collect the motion information of the user taking the mobile device. If an accelerated upward motion is detected, it may indicate the user was suddenly standing up or raising his hands, which further indicates the user saw something exciting. For both cases, the past period of time (e.g., 2 minutes) of a
TV program containing a possible highlight will be temporarily stored in the ring buffer. As a complementary method, the detector module may also use the messages collected by the message receiver module 12 for highlight detection. If the message receiving system is in real-time, it is quite possible that the message receiver module can get a highlight message within the time capacity (e.g., 2 minutes) of the ring buffer.
The determining module determines that a highlight has emerged, then, in the situation that highlight recording is needed, the highlight detector module will instruct the highlight recorder module to start recording. The recorded highlight, as well as some notes in respect of the highlight, will be stored in a highlight database. In this situation, storage capacity for highlights is provided on the mobile device.
The controller will cause the playing of the digital program to stop and sends the highlight to the player 19 to start playing when it is required to show the highlight.
When requested by the highlight detector module, the highlight recorder module first processes (part of) the data in the ring buffer 15, for example, unpacking the RTP packages to get H264 AV packages, and puts them into the highlight database as one highlight. Then, simple information (in the case that highlight is detected on the basis of the information collected by the sensor module) like time and match name, or some extended information (in the case the highlight is detected based on the messages received by the message receiver module) like "Raul scored the equalizer for Spain in the 71st minute", can be added to the recorded highlight in the database as annotations by the highlight recorder module.
The processor module 14 can also comprise a highlight post-processor module. Based on analysis of the messages collected by the message receiver module, the highlight postprocessor module updates the highlight database. The updating can be periodical (e.g., every 5 minutes) or triggered by newly received messages. The main updating operation is performed to update the annotations to the recorded highlights. Since most highlights are determined on the basis of the information collected by the sensor module 11, only some simple annotations are added to the recorded highlight when it is put into the database. The processor module 14 may also not comprise the highlight recorder module. It means that the highlight recording is not necessarily performed on the mobile device, but may be done on a server (shown as "S" in Figure 2) instead. In this situation, the highlight database can be used just to store some records, for example, the emerging time of a highlight, etc. It is even not necessary to provide a highlight database and highlight storage on a mobile device.
When highlight recording is performed on the server, it will be unnecessary to provide a ring buffer on a mobile device, as shown in Figure 2. When the mobile device determines, by means of the processor module 14, that a highlight has emerged, highlight information will be sent to the server by an information sending module 20. Preferably, the server will send to the mobile device a message asking whether or not a highlight will be watched. Upon receiving a "yes" answer from the mobile device, the server interrupts the transmission of the digital program and starts to transmit the highlight. Transmission of the digital program will continue after the transmission of the highlight has ended.
In other circumstances, there is no need to record the highlight. When the highlight detector module determines that a highlight has emerged, the processor module can read a segment of a predetermined prior time period as the highlight directly from the ring buffer. Then the highlight is directly sent to the controller 18 so as to be played by the player 19. Or the controller 18 could provide a hint of a highlight to the user first, and instruct the player 19 to play the highlight upon receiving a "yes" reply. Of course, the moment or time label of the highlight should be recorded so as to enable play of the highlight. When it needs to play the highlight, the mobile device can send a request to the server. Then the server searches for the moment or time label to locate the highlight in the recorded video of the match and transmits the highlight.
Figure 3 is flow chart of the method according to one embodiment of the invention.
First, the mobile device receives a digital program in step 201. As described above, the digital program can be transmitted by a broadcast network (e.g. DVB-H) or a mobile network (e.g.
3G). Obviously other types of digital TV transmission would be suitable. Then, in step 202, the reaction information of the user will be collected by the sensor(s) integrated in the mobile device. As mentioned above, it can be audio information or action information that is detected. It would be possible to detect other information, as corresponding sensors are provided in the mobile device, if needed. In step 203, the collected or detected information is calculated and quantified, e.g. the feature(s) of the audio or acceleration information are calculated. Then the calculated feature is compared with a predetermined threshold. It is determined whether the user has just had an exciting reaction, i.e. when the calculated feature is larger or equal to the threshold, e.g. cheering and/or getting up abruptly, whereby the emergence of a highlight is determined, i.e. step 204. In step 205, the highlight is played on the mobile device. When the playing of the highlight is finished, the display of the digital program will continue, i.e. step
206. The above procedure will be repeated until the program is over or being terminated by the user.
With respect to processing a highlight, various kinds of known methods would be also suitable. For example, as described in Chinese Patent application 200310122527.3, a flag can be added to the program stream. Then the highlight would be a segment of a predetermined time period before the flag.
In accordance with the above description, the basic concept of the method comprises: 1 ) Using the information collected by the integrated sensor as main hints to detect the emergence of a possible highlight.
2) Using the messages sent by service providers as complementary hints to detect the emergence of a possible highlight.
3) Based on these hints, the recording of a highlight is triggered automatically. 4) The recorded highlights can be stored into a database, which can be updated by a postprocessor using messages sent by service providers.
The advantages of the invention are:
1 ) Intelligent. The whole process of highlight detection and recording is completed automatically without the user's intervention. 2) Personalized. The segments that the user is interested in are recorded according to his/her reaction. Users can choose to watch the recorded highlight anytime they want.
3) The recorded highlights can be annotated using the received notifications, which helps users know easily what kinds of highlights have been recorded.
As previously mentioned, recording of the highlight is not a must. It is possible to have the server stores only the time/moments when the highlights emerge and the video of the whole match/event. When the server receives a request from a user to view a certain highlight, the server can read the highlight from the recorded video of the whole match according to the time/moment of the highlight and transmits the highlight.
It should be noted that the above description of the invention and the associated drawings illustrate rather than limit the scope of the invention. It will be obvious that many amendments and alternatives are possible based on the above teaching, without departing from the scope of the invention as defined in the following claims.

Claims

WHAT IS CLAIMED IS:
1. A method of detecting a highlight by a mobile device, comprising the steps of: receiving (201) digital program signals on the mobile device, collecting (202) reaction information of a user through the mobile device, and determining (203) an emergence of a highlight according to the detected reaction information.
2. The method according to claim 1, wherein the reaction information comprises audio information.
3. The method according to claim 2, wherein the step of determining (203) the emergence of a highlight comprises: determining a feature of the audio information, and determining the highlight which has emerged by comparing the feature is not lower than the predetermined threshold.
4. The method according to claim 3, wherein the feature of the audio information comprises short-time energy, zero crossing rate, sub-band energies, or a combination of the short the short time energy and the zero crossing rate.
5. The method according to claim 1, wherein the reaction information comprises action information.
6. The method according to claim 5, wherein the action information comprises acceleration.
7. A method of playing a highlight detected by the method as claimed in any one of claims 1- 6.
8. The method according to claim 7, further comprising the steps of: sending a request a server for acquiring the highlight, receiving the highlight from the server according to the request, and playing the highlight.
9. A mobile device for detecting a highlight, comprising: a digital program receiver module (12); a sensor module (11) for collecting reaction information of a user; and a processing module (14) for determining an emergence of a highlight according to the reaction information.
10. The mobile device according to claim 9, wherein the sensor module (11) comprises an audio sensor and/or an acceleration sensor.
11. The mobile device according to claim 9, wherein the processing module (14) comprises a calculation module for calculating the value of a feature from the reaction information, a comparison module for comparing the value with a predetermined threshold, and a determining muddle for determining a highlight to emerge if the value is lower than the predetermined threshold.
12. A system for playing a highlight, comprising: a mobile device used to send a request for acquiring the highlight according to any one of claims 9 to 11, and a server for receiving the request and sending the highlight to the mobile device according to the request.
PCT/IB2008/054188 2007-10-22 2008-10-13 Method and mobile device for detecting highlights WO2009053869A2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN200710167014.2 2007-10-22
CNA2007101670142A CN101420579A (en) 2007-10-22 2007-10-22 Method, apparatus and system for detecting exciting part

Publications (2)

Publication Number Publication Date
WO2009053869A2 true WO2009053869A2 (en) 2009-04-30
WO2009053869A3 WO2009053869A3 (en) 2009-07-02

Family

ID=40580169

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2008/054188 WO2009053869A2 (en) 2007-10-22 2008-10-13 Method and mobile device for detecting highlights

Country Status (2)

Country Link
CN (1) CN101420579A (en)
WO (1) WO2009053869A2 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011071460A1 (en) * 2009-12-10 2011-06-16 Echostar Ukraine, L.L.C. System and method for adjusting presentation characteristics of audio/video content in response to detection of user sleeping patterns
WO2012162038A1 (en) * 2011-05-25 2012-11-29 Echostar Technologies L.L.C. Apparatus, systems and methods for presentation management of erotica-related media content
US20160292510A1 (en) * 2015-03-31 2016-10-06 Zepp Labs, Inc. Detect sports video highlights for mobile computing devices
EP3163868A4 (en) * 2014-06-30 2017-07-05 ZTE Corporation Method and device for detecting video conference hotspot scenario

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102693739A (en) * 2011-03-24 2012-09-26 腾讯科技（深圳）有限公司 Method and system for video clip generation
CN103207662A (en) * 2012-01-11 2013-07-17 联想(北京)有限公司 Method and device for obtaining physiological characteristic information
CN102842327A (en) * 2012-09-03 2012-12-26 深圳市迪威视讯股份有限公司 Method and system for editing multimedia data streams
CN103873492B (en) * 2012-12-07 2019-01-15 联想(北京)有限公司 A kind of electronic equipment and data transmission method
CN104349099B (en) * 2013-07-25 2018-04-27 联想(北京)有限公司 The method and apparatus for storing image
CN104951479A (en) * 2014-03-31 2015-09-30 小米科技有限责任公司 Video content detecting method and device
CN104185064B (en) * 2014-05-30 2018-04-27 华为技术有限公司 media file identification method and device
CN106339655A (en) * 2015-07-06 2017-01-18 无锡天脉聚源传媒科技有限公司 Video shot marking method and device
CN106851424B (en) * 2017-04-07 2019-08-30 腾讯科技（深圳）有限公司 Video broadcasting method and device
CN108093297A (en) * 2017-12-29 2018-05-29 厦门大学 A kind of method and system of filmstrip automatic collection
CN108509931B (en) * 2018-04-11 2021-06-01 河南工学院 Football wonderful video capturing method and system

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1582965A1 (en) * 2004-04-01 2005-10-05 Sony Deutschland Gmbh Emotion controlled system for processing multimedia data
KR100690819B1 (en) * 2005-07-21 2007-03-09 엘지전자 주식회사 Mobile terminal having bookmark function of content service and its operation method
US20070150916A1 (en) * 2005-12-28 2007-06-28 James Begole Using sensors to provide feedback on the access of digital content

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011071460A1 (en) * 2009-12-10 2011-06-16 Echostar Ukraine, L.L.C. System and method for adjusting presentation characteristics of audio/video content in response to detection of user sleeping patterns
WO2012162038A1 (en) * 2011-05-25 2012-11-29 Echostar Technologies L.L.C. Apparatus, systems and methods for presentation management of erotica-related media content
US10097875B2 (en) 2011-05-25 2018-10-09 Echostar Technologies L.L.C. Apparatus, systems and methods for presentation management of erotica-related media content
US10555029B2 (en) 2011-05-25 2020-02-04 DISH Technologies L.L.C. Apparatus, systems and methods for presentation management of media content
US11323762B2 (en) 2011-05-25 2022-05-03 DISH Technologies L.L.C. Apparatus, systems and methods for presentation management of media content
US12114033B2 (en) 2011-05-25 2024-10-08 DISH Technologies L.L.C. Apparatus, systems and methods for presentation management of media content
EP3163868A4 (en) * 2014-06-30 2017-07-05 ZTE Corporation Method and device for detecting video conference hotspot scenario
US20160292510A1 (en) * 2015-03-31 2016-10-06 Zepp Labs, Inc. Detect sports video highlights for mobile computing devices
US10572735B2 (en) * 2015-03-31 2020-02-25 Beijing Shunyuan Kaihua Technology Limited Detect sports video highlights for mobile computing devices

Also Published As

Publication number Publication date
WO2009053869A3 (en) 2009-07-02
CN101420579A (en) 2009-04-29

Similar Documents

Publication Publication Date Title
WO2009053869A2 (en) Method and mobile device for detecting highlights
US11115721B2 (en) Methods and apparatus for monitoring video games
US8726304B2 (en) Time varying evaluation of multimedia content
US20200395033A1 (en) Removal of Audio Noise
US10021457B2 (en) System and method for engaging a person in the presence of ambient audio
TWI411301B (en) Methods and apparatus to determine audience viewing of recorded programs
US20060294565A1 (en) Television-based security monitoring system
MX2007015263A (en) Methods and apparatus to detect a time-shift event associated with the presentation of media content.
CN102193794A (en) Linking real time media context to related applications and services
CN107785037A (en) Use the method, system and medium of audio time code synchronized multimedia content
CN103179459A (en) Information playback device and control method thereof
US20050001903A1 (en) Methods and apparatuses for displaying and rating content
US10200722B2 (en) Method and apparatus for classifying and transmitting data from an event
CN103546725A (en) Mobile phone client and remote video monitor and control system and method
JP4828785B2 (en) Information processing device and portable terminal device
JP3638591B2 (en) Content provision system
JP2010206440A (en) Digital broadcast receiver and commercial effect evaluation method
JP2010266880A (en) Mobile terminal device, information processing method, and program
KR101025065B1 (en) Set-top box and its image processing method
JP2005332404A (en) Content providing system
JP2004023375A (en) Viewing situation data acquisition method, center device and program
US20230291954A1 (en) Stadium videograph
WO2003054855A1 (en) Voice recognition method and multimedia terminal therefor
TW200920137A (en) Method and mobile device for detecting highlights
US9443133B2 (en) Electronic device having message-recording and message-playback function and related method

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 08807962

Country of ref document: EP

Kind code of ref document: A2