WO2011138637A1 - System, method, and apparatus for facilitating group video communication - Google Patents
System, method, and apparatus for facilitating group video communication Download PDFInfo
- Publication number
- WO2011138637A1 WO2011138637A1 PCT/IB2010/051929 IB2010051929W WO2011138637A1 WO 2011138637 A1 WO2011138637 A1 WO 2011138637A1 IB 2010051929 W IB2010051929 W IB 2010051929W WO 2011138637 A1 WO2011138637 A1 WO 2011138637A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- video
- determining
- sent
- adjustment
- amount
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
- H04N7/148—Interfacing a video terminal to a particular transmission medium, e.g. ISDN
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/1066—Session management
- H04L65/1083—In-session procedures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/40—Support for services or applications
- H04L65/403—Arrangements for multi-party communication, e.g. for conferences
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/752—Media network packet handling adapting media to network capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/18—Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals
Definitions
- Embodiments of the present invention relate generally to communication technology and, more particularly, relate to a system, method, and apparatus for facilitating group video communication.
- Systems, methods, apparatuses, and computer program products described herein facilitate group video communication.
- the systems, methods, apparatuses, and computer program products provided in accordance with example embodiments of the invention may provide several advantages to computing devices, computing device users, and network operators.
- example embodiments may enable performance limited mobile devices to engage in group video communication, such as group video calls, group conference calls, and/or the like.
- Some example embodiments provide for signaling devices transmitting video streams in a group video communication with indications of requested adjustments to the amount of data transmitted in the video streams. In some example embodiments, this signaling may be responsive to detection of a modification of display of a video stream.
- some example embodiments determine an adjustment to an amount of media data sent in one or more video streams so that unnecessary media data is not received. For example, if only a subset of video streams from a group video communication is displayed on a device, some example embodiments provide for determination of an adjustment so that unnecessary media data from video streams that are not displayed is not received or is received with a lower bandwidth. As another example, if multiple video streams from a group video communication are concurrently displayed, some example embodiments provide for determination of an adjustment such that the bandwidth of the plurality of displayed video streams may be reduced to a point enabling the receiving apparatus to process and display each of the displayed video streams without exceeding available media decoding resources.
- a method comprises determining a modification of display of a video stream comprising one of a plurality of video streams transmitted for a group video communication.
- the method of this example embodiment further comprises determining, based on the determined modification, an adjustment of an amount of media data being sent in and/or in association with at least one of the plurality of video streams.
- the method of this example embodiment additionally comprises causing, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of data being sent in and/or in association with the at least one of the plurality of video streams.
- an apparatus comprising at least one processor and at least one memory storing computer program code, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to at least determine a modification of display of a video stream comprising one of a plurality of video streams transmitted for a group video communication.
- the at least one memory and stored computer program code are configured, with the at least one processor, to further cause the apparatus of this example embodiment to determine, based on the determined modification, an adjustment of an amount of media data being sent in and/or in association with at least one of the plurality of video streams.
- the at least one memory and stored computer program code are configured, with the at least one processor, to additionally cause the apparatus of this example embodiment to cause, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of data being sent in and/or in association with the at least one of the plurality of video streams.
- a computer program product includes at least one computer- readable storage medium having computer-readable program instructions stored therein.
- the program instructions of this example embodiment comprise program instructions configured to determine a modification of display of a video stream comprising one of a plurality of video streams transmitted for a group video communication.
- the program instructions of this example embodiment further comprise program instructions configured to determine, based on the determined modification, an adjustment of an amount of media data being sent in and/or in association with at least one of the plurality of video streams.
- the program instructions of this example embodiment also comprise program instructions configured to cause, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of data being sent in and/or in association with the at least one of the plurality of video streams.
- an apparatus comprises means for determining a modification of display of a video stream comprising one of a plurality of video streams transmitted for a group video communication.
- the apparatus of this example embodiment further comprises means for determining, based on the determined modification, an adjustment of an amount of media data being sent in and/or in association with at least one of the plurality of video streams.
- the apparatus of this example embodiment additionally comprises means for causing, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of data being sent in and/or in association with the at least one of the plurality of video streams.
- FIG. 1 illustrates a block diagram of a system for facilitating group video communication according to example embodiments of the invention
- FIG. 2 is a schematic block diagram of a mobile terminal according to example embodiments of the invention.
- FIG. 3 illustrates a block diagram of a video communication apparatus for facilitating group video communication according to example embodiments of the invention
- FIGs. 4a and 4b illustrate example embodiments of group video communication
- FIG. 5 illustrates processing of received media streams in a group video communication according to example embodiments of the invention
- FIG. 6 illustrates a signaling diagram according to an example flow of signals exchanged between terminals participating in a group video communication according to example embodiments of the invention
- FIG. 7 illustrates processing of received media streams in a group video communication according to example embodiments of the invention
- FIG. 8 illustrates a signaling diagram according to an example flow of signals exchanged between terminals participating in a group video communication according to example embodiments of the invention
- FIG. 9 illustrates a flowchart according to an example methods for facilitating group video communication according to example embodiments of the invention.
- FIG. 10 illustrates a flowchart according to an example methods for facilitating group video communication according to example embodiments of the invention.
- a computing device is described herein to receive data from another computing device, it will be appreciated that the data may be received directly from the another computing device or may be received indirectly via one or more intermediary computing devices, such as, for example, one or more servers, relays, routers, network access points, base stations, and/or the like.
- intermediary computing devices such as, for example, one or more servers, relays, routers, network access points, base stations, and/or the like.
- circuitry refers to (a) hardware-only circuit implementations (for example, implementations in analog circuitry and/or digital circuitry); (b) combinations of circuits and computer program product(s) comprising software and/or firmware instructions stored on one or more computer readable memories that work together to cause an apparatus to perform one or more functions described herein; and (c) circuits, such as, for example, a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation even if the software or firmware is not physically present.
- This definition of 'circuitry' applies to all uses of this term herein, including in any claims.
- the term 'circuitry' also includes an implementation comprising one or more processors and/or portion(s) thereof and accompanying software and/or firmware.
- the term 'circuitry' as used herein also includes, for example, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, other network device, and/or other computing device.
- FIG. 1 illustrates a block diagram of a system 100 for facilitating group video communication according to example embodiments of the present invention.
- the system 100 as well as the illustrations in other figures are each provided as example embodiments of the invention and should not be construed to narrow the scope or spirit of the invention in any way.
- the scope of the disclosure encompasses many potential embodiments in addition to those illustrated and described herein.
- FIG. 1 illustrates one example of a configuration of a system for facilitating remote data processing, numerous other configurations may also be used to implement embodiments of the present invention.
- the system 100 includes a video communication apparatus 102 and a plurality of terminal apparatuses 102. Although three such terminal apparatuses 104 are illustrated in FIG. 1 for purposes of example, it will be appreciated that the system 100 may comprise additional or fewer terminal apparatuses 104. Further, although the video communication apparatus 102 and terminal apparatuses 104 are separately labeled, the video communication apparatus 102 may be configured to implement functionality attributed to the terminal apparatuses 104 and a terminal apparatus 104 may be configured to implement functionality attributed to the video communication apparatus 102. In this regard, a video communication apparatus 102 may be used for participation in a group video communication with users of a plurality of terminal apparatuses 104. Accordingly, a video communication apparatus 102 and terminal apparatus 104 may each be configured to receive and transmit media streams supporting a group video communication.
- the video communication apparatus 102 may be in communication with a plurality of terminal apparatuses 104 over the network 106.
- the network 106 may comprise a wireless network (for example, a cellular network, wireless local area network, wireless personal area network, wireless metropolitan area network, and/or the like), a wireline network, or some combination thereof, and in some example embodiments comprises a portion of the internet.
- the network 106 may comprise an ad hoc network formed by the video communication apparatus 102 and a plurality of terminal apparatuses 104, such as through various radio frequency communication means that may be used for device-to-device communication.
- the network 106 may comprise any network enabling the communication of media streams supporting group video communication between a user of the video communication apparatus 102 and users of a plurality of terminal apparatuses 104.
- the video communication apparatus 102 may be embodied as any computing device, such as, for example, a desktop computer, laptop computer, mobile terminal, mobile computer, mobile phone, mobile communication device, game device, digital camera/camcorder, audio/video player, television device, radio receiver, digital video recorder, positioning device, wrist watch, portable digital assistant (PDA), any combination thereof, and/or the like.
- the video communication apparatus 102 may be embodied as any computing device configured to engage in group video communication with a plurality of terminal apparatuses 104 over the network 106, as will be described further herein below.
- a terminal apparatus 104 may be embodied as any computing device, such as, for example, a desktop computer, laptop computer, mobile terminal, mobile computer, mobile phone, mobile communication device, game device, digital camera/camcorder, audio/video player, television device, radio receiver, digital video recorder, positioning device, wrist watch, portable digital assistant (PDA), any combination thereof, and/or the like.
- a terminal apparatus 104 may be embodied as any computing device configured to ⁇ engage in group video communication with a video communication apparatus 102 over the network 106, as will be described further herein below.
- one or more of the video communication apparatus 102 or terminal apparatus 104 may be embodied as a mobile terminal, such as that illustrated in FIG. 2.
- FIG. 2 illustrates a block diagram of a mobile terminal 10 representative of one embodiment of a video communication apparatus 102 and/or terminal apparatus 104 in accordance with example embodiments of the present invention. It should be understood, however, that the mobile terminal 10 illustrated and hereinafter described is merely illustrative of one type of video communication apparatus 102 and/or terminal apparatus 104 that may implement and/or benefit from embodiments of the present invention and, therefore, should not be taken to limit the scope of the present invention.
- the mobile terminal 10 may include an antenna 12 (or multiple antennas 12) in communication with a transmitter 14 and a receiver 16.
- the mobile terminal 10 may also include a processor 20 configured to provide signals to and receive signals from the transmitter and receiver, respectively.
- the processor 20 may, for example, be embodied as various means including circuitry, one or more microprocessors with accompanying digital signal processor(s), one or more processor(s) without an accompanying digital signal processor, one or more coprocessors, one or more multi-core processors, one or more controllers, processing circuitry, one or more computers, various other processing elements including integrated circuits such as, for example, an ASIC (application specific integrated circuit) or FPGA (field
- the processor 20 may comprise a plurality of processors.
- These signals sent and received by the processor 20 may include signaling information in accordance with an air interface standard of an applicable cellular system, and/or any number of different wireline or wireless networking techniques, comprising but not limited to Wireless-Fidelity (Wi-Fi), wireless local access network (WLAN) techniques such as Institute of Electrical and Electronics Engineers (IEEE) 802.11, 802.16, and/or the like.
- these signals may include speech data, user generated data, user requested data, and/or the like.
- the mobile terminal may be capable of operating with one or more air interface standards,
- the mobile terminal may be capable of operating in accordance with various first generation (1G), second generation (2G), 2.5G, third-generation (3G) communication protocols, fourth-generation (4G) communication protocols, Internet Protocol Multimedia Subsystem (IMS) communication protocols (for example, session initiation protocol (SIP)), H.323 and/or the like.
- IMS Internet Protocol Multimedia Subsystem
- the mobile terminal may be capable of operating in accordance with 2G wireless communication protocols IS- 136 (Time Division Multiple Access (TDMA)), Global System for Mobile communications (GSM), IS-95 (Code Division Multiple Access (CDMA)), and/or the like.
- TDMA Time Division Multiple Access
- GSM Global System for Mobile communications
- CDMA Code Division Multiple Access
- the mobile terminal may be capable of operating in accordance with 2.5G wireless communication protocols General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), and/or the like. Further, for example, the mobile terminal may be capable of operating in accordance with 3G wireless communication protocols such as Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access 2000 (CDMA2000), Wideband Code Division Multiple Access (WCDMA), Time Division- Synchronous Code Division Multiple Access (TD-SCDMA), and/or the like. The mobile terminal may be additionally capable of operating in accordance with 3.9G wireless communication protocols such as Long Term Evolution (LTE) or Evolved Universal Terrestrial Radio Access Network (E-UTRAN) and/or the like. Additionally, for example, the mobile terminal may be capable of operating in accordance with fourth-generation (4G) wireless communication protocols and/or the like as well as similar wireless communication protocols that may be developed in the future.
- GPRS General Packet Radio Service
- EDGE Enhanced Data GSM Environment
- 3G wireless communication protocols such as Universal Mobile T
- a Narrow-band Advanced Mobile Phone System In some example embodiments, a Narrow-band Advanced Mobile Phone System
- NAMPS Network Access Communication System
- TACS Total Access Communication System
- mobile terminals may also benefit from embodiments of this invention, as may dual or higher mode phones (for example, digital/analog or TDMA/CDMA/analog phones). Additionally, the mobile terminal 10 may be capable of operating according to Wireless Fidelity (Wi-Fi) or
- WiMAX Worldwide Interoperability for Microwave Access
- the processor 20 may comprise circuitry for implementing audio/video and logic functions of the mobile terminal 10.
- the processor 20 may comprise a digital signal processor device, a microprocessor device, an analog-to- digital converter, a digital-to-analog converter, and/or the like. Control and signal processing functions of the mobile terminal may be allocated between these devices according to their respective capabilities.
- the processor may additionally comprise an internal voice coder (VC) 20a, an internal data modem (DM) 20b, and/or the like.
- the processor may comprise functionality to operate one or more software programs, which may be stored in memory.
- the processor 20 may be capable of operating a connectivity program, such as a web browser.
- the connectivity program may allow the mobile terminal 10 to transmit and receive web content, such as location-based content, according to a protocol, such as Wireless Application Protocol (WAP), hypertext transfer protocol (HTTP), and/or the like.
- WAP Wireless Application Protocol
- HTTP hypertext transfer protocol
- the mobile terminal 10 may be capable of using a Transmission Control Protocol/Internet Protocol (TCP/IP) to transmit and receive web content across the internet or other networks.
- TCP/IP Transmission Control Protocol/Internet Protocol
- the mobile terminal 10 may also comprise a user interface including, for example, an earphone or speaker 24, a ringer 22, a microphone 26, a display 28, a user input interface, and/or the like, which may be operationally coupled to the processor 20.
- the processor 20 may comprise user interface circuitry configured to control some functions of one or more elements of the user interface, such as, for example, the speaker 24, the ringer 22, the microphone 26, the display 28, and/or the like.
- the processor 20 and/or user interface circuitry comprising the processor 20 may be configured to control one or more functions of one or more elements of the user interface through computer program instructions (for example, software and/or firmware) stored on a memory accessible to the processor 20 (for example, volatile memory 40, non-volatile memory 42, and/or the like).
- the mobile terminal may comprise a battery for powering various circuits related to the mobile terminal, for example, a circuit to provide mechanical vibration as a detectable output.
- the user input interface may comprise devices allowing the mobile terminal to receive data, such as a keypad 30, a touch display, a joystick, and/or other input device. In embodiments including a keypad, the keypad may comprise numeric (0-9) and related keys (#, *), and/or other keys for operating the mobile terminal.
- the mobile terminal 10 may also include one or more means for sharing and/or obtaining data.
- the mobile terminal may comprise a short-range radio frequency (RF) transceiver and/or interrogator 64 so data may be shared with and/or obtained from electronic devices in accordance with RF techniques.
- the mobile terminal may comprise other short-range transceivers, such as, for example, an infrared (IR) transceiver 66, a BluetoothTM (BT) transceiver 68 operating using BluetoothTM brand wireless technology developed by the BluetoothTM Special Interest Group, a wireless universal serial bus (USB) transceiver 70 and/or the like.
- IR infrared
- BT BluetoothTM
- USB wireless universal serial bus
- the BluetoothTM transceiver 68 may be capable of operating according to ultra-low power BluetoothTM technology (for example, WibreeTM) radio standards.
- the mobile terminal 10 and, in particular, the short-range transceiver may be capable of transmitting data to and/or receiving data from electronic devices within a proximity of the mobile terminal, such as within 10 meters, for example.
- the mobile terminal may be capable of transmitting and/or receiving data from electronic devices according to various wireless networking techniques, including Wireless Fidelity (Wi-Fi), WLAN techniques such as IEEE 802.11 techniques, IEEE 802.15 techniques, IEEE 802.16 techniques, and/or the like.
- Wi-Fi Wireless Fidelity
- WLAN techniques such as IEEE 802.11 techniques, IEEE 802.15 techniques, IEEE 802.16 techniques, and/or the like.
- the mobile terminal 10 may include a media capturing element, such as a camera, video and/or audio module, in communication with the processor 20.
- the media capturing element may be any means for capturing an image, video and/or audio for storage, display or transmission.
- the camera module 36 may include a digital camera capable of forming a digital image file from a captured image.
- the digital camera of the camera module 36 may be capable of capturing a video clip.
- the camera module 36 may include all hardware, such as a lens or other optical component(s), and software necessary for creating a digital image file from a captured image as well as a digital video file from a captured video clip.
- the camera module 36 may include only the hardware needed to view an image, while a memory device of the mobile terminal 10 stores instructions for execution by the processor 20 in the form of software necessary to create a digital image file from a captured image.
- an object or objects within a field of view of the camera module 36 may be displayed on the display 28 of the mobile terminal 10 to illustrate a view of an image currently displayed which may be captured if desired by the user.
- the camera module 36 may further include a processing element such as a co-processor which assists the processor 20 in processing image data and an encoder and/or decoder for compressing and/or decompressing image data.
- the encoder and/or decoder may encode and/or decode according to, for example, a joint photographic experts group (JPEG) standard, a moving picture experts group (MPEG) standard, or other format.
- JPEG joint photographic experts group
- MPEG moving picture experts group
- the mobile terminal 10 may comprise memory, such as a subscriber identity module (SIM) 38, a universal subscriber identity module (USIM), a removable user identity module (R-UIM), and/or the like, which may store information elements related to a mobile subscriber.
- SIM subscriber identity module
- USIM universal subscriber identity module
- R-UIM removable user identity module
- the mobile terminal 10 may include volatile memory 40 and/or non- volatile memory 42.
- volatile memory 40 may include Random Access Memory (RAM) including dynamic and/or static RAM, on-chip or off-chip cache memory, and/or the like.
- RAM Random Access Memory
- Nonvolatile memory 42 which may be embedded and/or removable, may include, for example, read-only memory, flash memory, magnetic storage devices (for example, hard disks, floppy disk drives, magnetic tape, etc.), optical disc drives and/or media, nonvolatile random access memory (NVRAM), and/or the like. Like volatile memory 40 non- volatile memory 42 may include a cache area for temporary storage of data.
- the memories may store one or more software programs, instructions, pieces of information, data, and/or the like which may be used by the mobile terminal for performing functions of the mobile terminal.
- the memories may comprise an identifier, such as an international mobile equipment identification (IMEI) code, capable of uniquely identifying the mobile terminal 10.
- IMEI international mobile equipment identification
- FIG. 3 illustrates a block diagram of a video
- the video communication apparatus 102 may include various means, such as one or more of a processor 110, memory 112, communication interface 114, user interface 1 16, and media stream management circuitry 118 for performing the various functions herein described.
- These means of the video communication apparatus 102 as described herein may be embodied as, for example, circuitry, hardware elements (for example, a suitably programmed processor, combinational logic circuit, and/or the like), a computer program product comprising computer-readable program instructions (for example, software or firmware) stored on a computer-readable medium (for example, memory 1 12) that is executable by a suitably configured processing device (for example, the processor 110), or some combination thereof.
- a terminal apparatus 104 may be configured to perform some of the functionality of a video communication apparatus 102. Accordingly, it will be appreciated that a terminal apparatus 104 may comprise one or more of the means of the video communication apparatus 102 illustrated in and described with respect to FIG. 3.
- the processor 110 may, for example, be embodied as various means including one or more microprocessors with accompanying digital signal processor(s), one or more processor(s) without an accompanying digital signal processor, one or more coprocessors, one or more multi-core processors, one or more controllers, processing circuitry, one or more computers, various other processing elements including integrated circuits such as, for example, an ASIC (application specific integrated circuit) or FPGA (field
- the processor 110 may comprise a plurality of processors.
- the plurality of processors may be in operative communication with each other and may be collectively configured to perform one or more functionalities of the video communication apparatus 102 as described herein.
- the processor 110 may be embodied as or comprise the processor 20.
- the processor 110 may be configured to execute instructions stored in the memory 112 or otherwise accessible to the processor 110.
- the processor 110 may comprise an entity capable of performing operations according to example embodiments of the present invention while configured accordingly.
- the processor 110 may comprise specifically configured hardware for conducting one or more operations described herein.
- the processor 110 when the processor 110 is embodied as an executor of instructions, such as may be stored in the memory 112, the instructions may specifically configure the processor 110 to perform one or more algorithms and operations described herein.
- the memory 112 may comprise, for example, volatile memory, non- volatile memory, or some combination thereof. Although illustrated in FIG. 3 as a single memory, the memory 112 may comprise a plurality of memories. In example embodiments, the memory 112 may comprise, for example, a hard disk, random access memory, cache memory, flash memory, a compact disc read only memory (CD-ROM), digital versatile disc read only memory (DVD-ROM), an optical disc, circuitry configured to store information, or some combination thereof. In example embodiments wherein the video communication apparatus 102 is embodied as a mobile terminal 10, the memory 112 may comprise the volatile memory 40 and/or the non-volatile memory 42.
- the memory 112 may be configured to store information, data, applications, instructions, or the like for enabling the video communication apparatus 102 to carry out various functions in accordance with example embodiments of the present invention.
- the memory 112 may be configured to buffer input data for processing by the processor 110.
- the memory 112 may be configured to store program instructions for execution by the processor 110.
- the memory 112 may store information in the form of static and/or dynamic information. This stored information may be stored and/or used by the media stream management circuitry 118 during the course of performing its functionalities.
- the communication interface 114 may be embodied as any device or means embodied in circuitry, hardware, a computer program product comprising computer readable program instructions stored on a computer readable medium (for example, the memory 112) and executed by a processing device (for example, the processor 110), or a combination thereof that is configured to receive and/or transmit data from/to an entity of the system 100, such as, for example, a terminal apparatus 104.
- the communication interface 114 may be at least partially embodied as or otherwise controlled by the processor 110.
- the communication interface 114 may, for example, be in communication with the processor 110, such as via a bus.
- the communication interface 114 may include, for example, an antenna, a transmitter, a receiver, a transceiver and/or supporting hardware or software for enabling communications with one or more entities of the system 100.
- the communication interface 114 may be configured to receive and/or transmit data using any protocol that may be used for communications between entities of the system 100.
- the communication interface 114 may additionally be in communication with the memory 112, user interface 116, and/or media stream management circuitry 1 18, such as via a bus.
- the user interface 116 may be in communication with the processor 110 to receive an indication of a user input and/or to provide an audible, visual, mechanical, or other output to a user.
- the user interface 116 may include, for example, a keyboard, a mouse, a joystick, a display, a touch screen display, a microphone, a speaker, and/or other input/output mechanisms.
- the user interface 116 may additionally comprise, control, or otherwise communicate with a camera configured to capture video of a user of the video communication apparatus 102 to enable participation by the user in a group video communication.
- a camera configured to capture video of a user of the video communication apparatus 102 to enable participation by the user in a group video communication.
- the user interface 116 may comprise, control, or otherwise communicate with the camera module 36, which may be configured to capture video of a user of the video communication apparatus 102 to enable participation by the user in a group video communication.
- the user interface 116 additionally may be in communication with the memory 112, communication interface 114, and/or media stream management circuitry 118, such as via a bus.
- the media stream management circuitry 118 may be embodied as various means, such as circuitry, hardware, a computer program product comprising computer readable program instructions stored on a computer readable medium (for example, the memory 112) and executed by a processing device (for example, the processor 1 10), or some combination thereof and, in some example embodiments, may be embodied as or otherwise controlled by the processor 110.
- the media stream management circuitry 118 may be in communication with the processor 110.
- the media stream management circuitry 118 may be configured to receive video and/or still images captured by a camera (for example, the camera module 36 or other camera of the user interface 116) that is intended for communication to a terminal apparatus 104 in support of a group video communication.
- the media stream management circuitry 118 may further be configured to receive audio captured by a microphone or other audio input (for example, a microphone of the user interface 116) that is intended for communication to a terminal apparatus 04 in support of a group video communication.
- the media stream management circuitry 118 may further be in communication with one or more of the memory 112, communication interface 114, or user interface 116, such as via a bus.
- the media stream management circuitry 118 may be configured to manage media streams received and transmitted by the video
- the group video communication may be, for example, a group video call, a group conference call, a video conference having more than two video streams, or the like.
- FIG. 4a illustrates an example embodiments of such a group video communication.
- the terminal 402, terminal 404, terminal 406, and terminal 408 are engaged in group video communication.
- the terminals 402-408 may comprise embodiments of a video communication apparatus 102 or terminal apparatus 104.
- Each of the terminals 402-408 may capture media content (for example, video and audio or just video), such as of a user of the respective terminal and/or of the surrounding environment of the respective terminal, and transmit that captured media content to the other terminals engaged in the group video communication.
- each of the terminals 402-408 may also receive media content captured and transmitted by the other terminals.
- the media content exchanged in a group video communication may, for example, be transmitted and received using a real-time data transport protocol.
- a real-time data transport protocol that may be used in accordance with some example embodiments is real-time transport protocol (RTP).
- RTP real-time transport protocol
- RTP is merely an example of one protocol that may be used for transmission of media streams in support of a group video communication. Accordingly, where use of RTP is described herein, such description is for purposes of example and is not limiting. In this regard, other appropriate protocols, whether standardized or proprietary, may be substituted for RTP in other embodiments.
- the audio content received by a terminal from the other terminals may be multiplexed together into a single audio stream, which may be played back to a user of the terminal.
- Each of the plurality of video streams received by a terminal may be displayed concurrently on a display of the terminal.
- a subset of the video streams received by a terminal may be displayed.
- a single video stream 410 may be displayed on the terminal 406. This display of a single video stream is referred to herein as "full screen" display or full screen mode. It will be appreciated, however, that full screen display of a single video stream may, but need not cover the entirety of a display.
- FIG. 4a also illustrates a plurality of video streams - video streams 412-416 - displayed on the terminal 408. Display of a plurality of video streams is referred to herein as "split screen" display or split screen mode.
- the video streams 412- 416 are displayed in non-overlapping, equally sized portions of the display.
- split screen display of a plurality of video streams may also encompass other presentations of the displayed video streams, such as, tiled, mosaic, partially overlapping, display in non-equally sized portions, or other display arrangements.
- each or the terminals 402-408 illustrated in FIG. 4a may be sending and receiving video streams with each of the other terminals, it will be appreciated that not every terminal that is party to a group video communication has to send a video stream to and/or receive a video stream from every other terminal that is party to a group video communication.
- one or more terminals e.g., terminal apparatuses 104
- a terminal or user thereof may selectively elect to only establish video stream
- FIG. 4b illustrates an example embodiment of group video communication where not every terminal engaged in a group video communication is group video capable.
- FIG. 4b illustrates terminals 420-426, which may be engaged in a group video communication. However, only the terminal 426 is group video capable and sends a video stream to and receives a video stream from each of the other terminals.
- the media stream management circuitry 118 may be configured to determine a modification of display of a video stream in a group video communication.
- the modification may comprise modification of a display size of a video stream, such as, increasing or decreasing a display size of the video stream.
- the modification may comprise cessation of display of a video stream or display of a video stream that was not previously displayed.
- the modification may comprise modification of a display size of a video stream, such as, increasing or decreasing a display size of the video stream.
- the modification may comprise cessation of display of a video stream or display of a video stream that was not previously displayed.
- modification may comprise a modification to a display mode for the group video communication.
- the modification may comprise switching from a full screen mode to a split screen mode or vice versa.
- the modification of display of a video stream may alter an amount of video data needed to support display of one or more video streams in the group video communication.
- the media stream management circuitry 118 may be configured in some example embodiments to determine an adjustment of an amount of media data being sent in at least one of the video streams that comprise the group video communication based on a determined modification of display of a video stream.
- the media stream management circuitry 118 may base determination of an adjustment at least in part upon available media decoding resources.
- the processor 110, media stream management circuitry 118, and/or other element of the video communication apparatus 102 may have a finite amount of available resources for decoding media data received in a group video communication.
- the media stream management circuitry 118 may determine an adjustment of media data being sent in one or more video streams such that the aggregate of media data received and processed in the plurality of video streams that comprise the group video communication does not exceed available decoding resources.
- the determined adjustment for a video stream may comprise an adjustment of a bandwidth of video data, a bandwidth of audio data associated with the video stream, a frame rate of video data, a resolution of video data, a bit rate of video data, a bit rate of audio data associated with the video stream, a codec used for encoding video data, a codec used for encoding audio data associated with the video stream, a number of video frames packed to one RTP packet, a number of audio frames associated with the video stream packed to one RTP packet, some combination thereof, or the like.
- the media stream management circuitry 118 may determine a reduction of the amount of data being sent in the video stream. In this regard, when display size of a video stream has been reduced, less data may be needed to support play out of the video stream with sufficient video quality to satisfy a user of the video communication apparatus. Accordingly, the media stream management circuitry 118 may be configured to determine a reduced amount of media data that is sufficient to allow play out of the video stream to satisfy a predefined video quality threshold without providing more media data than is required for satisfying the predefined video quality threshold at the reduced display size.
- the predefined video quality threshold may be defined by a user of the video communication apparatus 102.
- the media stream management circuitry 118 may determine an adjustment so that only a minimum amount of media data needed to maintain a communication channel over which the video stream is transmitted by a terminal apparatus 104.
- the media stream management circuitry 118 may determine an increase of the amount of data being sent in the video stream.
- the media stream management circuitry 118 may be configured to determine an increased amount of media data that may be sufficient to allow play out of the video stream to satisfy a predefined video quality threshold.
- the predefined video quality threshold may be defined by a user of the video communication apparatus 102.
- the video quality threshold may comprise the video quality at which the video stream was played out prior to the increase in the display size such that a user does not notice a drop in video quality after increasing the display size of the video stream.
- additional media data may be needed to support play out of the video stream.
- the media stream management circuitry 118 may determine a reduction in the amount of media data being sent in at least one of the video streams that comprise the group video communication. In this regard, when multiple video streams are played out in the split screen mode, each of the displayed video streams must be decoded. Accordingly, the media stream management circuitry 118 may determine an adjustment to one or more video streams such that the decoding resources of the video communication apparatus 102 are not exceeded.
- the media stream management circuitry 118 may determine an increase in the amount of media data being sent in the video stream displayed in full screen mode.
- the increased amount of media data may comprise a sufficient amount of media data to support play out of the video stream in full screen mode to satisfy a predefined video quality threshold.
- the media stream management circuitry 118 may additionally or alternatively determine a reduction in an amount of media data being sent in one or more video streams that are not displayed. In this regard, only a single video stream has to be decoded and played out in full screen mode. Accordingly, it may be undesirable to burden available computing and network resources with transmission of a large amount of media data in video streams that are not displayed.
- the media stream management circuitry 118 may determine that only intraframes should be sent. In this regard, interframes may not be needed for those video streams that are not displayed. The media stream management circuitry 118 may further determine a period with which intraframes should be sent. In this regard, the media stream management circuitry 118 may determine that intraframes should be sent at less frequent intervals than intraframes would otherwise be sent to support play out of the video stream. In this regard, the media stream management circuitry 118 may determine an interval between intraframes that may be sufficient to maintain the communication channel over which a video stream is transmitted without transmitting an excessive amount of media data.
- the media stream management circuitry 118 may also determine an adjustment of an amount of audio data. According to some example embodiments, the media stream management circuitry 118 may determine an adjustment to a bit rate, sampling rate, codec used to encode audio, frames per RTP packet and/or the like. Adjusting an amount of audio data associated with a video stream may provide additional means for the media stream management circuitry 118 to ensure that processing and decoding resources of the video communication apparatus 102 are not exceeded by processing media streams received in a group video communication. In addition to or in lieu of adjustment to audio/video data, the media stream management circuitry 118 may determine an adjustment to other media data transmitted in and/or in association with a video stream. Such other media data may include, for example, real-time text.
- the media stream management circuitry 118 may be further configured to cause an indication of a determined adjustment of an amount of media data being sent in a video stream to be sent to a terminal apparatus 104 that is transmitting the video stream.
- sending the indication of the determined adjustment may trigger the terminal apparatus 104 to adjust the amount of media data being sent in the video stream in accordance with the determined adjustment.
- a terminal apparatus 104 may be configured to receive an indication of an adjustment and adjust the amount of media data being sent in a video stream in accordance with the adjustment.
- the video communication apparatus 102 may be configured to implement the functionality of a terminal apparatus 104
- a terminal apparatus 104 may send an indication of a determined adjustment to a video stream being transmitted by the video communication apparatus 102 to the video communication apparatus 102.
- the media stream management circuitry 118 may be configured to receive an indication of an adjustment determined by a terminal apparatus 104 and adjust the amount of media data being sent in the video stream transmitted by the video communication apparatus 102 to the respective terminal apparatus 104.
- an indication of a determined adjustment may be sent as a parameter to any communication signaling message such as session initiation protocol (SIP) or H.323 sent by the video communication apparatus 102 to a terminal apparatus 104.
- an indication of a determined adjustment may be sent in a dedicated media stream control message.
- a feedback mechanism of the real-time data transport protocol may be used for signaling an indication of a determined adjustment to a terminal apparatus 104.
- real-time transport protocol control protocol RTCP
- an indication of a determined adjustment may be communicated to a terminal apparatus 104 by way of a temporary maximum media stream bit rate request, temporary maximum media stream bit rate notification, and/or the like.
- an RTCP packet that may be used for signaling an indication of a determined adjustment to a terminal apparatus 104 may be described as follows: 0 1 2 3
- the first section, the header, may be 8 octets long.
- the fields may have the following meaning:
- the second section, the video media adjust may be X octets long.
- the fields may have the following meaning:
- bitrate 32 bits
- Value zero may be valid and may be interpreted as indicating that there is no bitrate limit.
- Pre-defined value mapping may be as follows,
- Value zero may valid and may be interpreted as indicating
- Value zero may be valid and may be interpreted as indicating that there is no framerate limit.
- RTCP packet is provided merely by way of example and not by way of limitation. Accordingly, other packet formats and other communication protocols may be used for signaling a determined adjustment in accordance with various example embodiments.
- FIG. 5 illustrates processing of received media streams in a group video communication according to an example embodiment of the invention wherein a plurality of video streams are displayed in split screen mode.
- FIG. 5 illustrates an example structure that may be used by example embodiments of the video communication apparatus 102 wherein RTP may be used to facilitate transmission of media streams in a group video communication involving four parties.
- the media stream management circuitry 118 and/or communication interface 114 may implement an RTP layer 502, Internet Protocol (IP)/User Datagram Protocol (UDP) layer 504, and protocol layer 506 to enable the transmission and receipt of media streams in support of the group video communication.
- the protocol layer 506 may be configured to implement any protocol used for communication over the network 106.
- the protocol layer 506 may comprise a wireless local area network (WLAN) protocol layer, Third Generation cellular communication (3G) protocol layer, Long Term Evolution (LTE) protocol layer, high speed packet access (HSPA) protocol layer, and/or the like.
- WLAN wireless local area network
- 3G Third Generation cellular communication
- LTE Long Term Evolution
- HSPA high speed packet access
- the media stream management circuitry 118 may additionally implement or control an audio encoder 510, video encoder 514, audio decoder 518, and video decoder 522.
- audio may be captured locally by a microphone 508 and encoded by the audio encoder 510.
- the encoded captured audio may be replicated at the RTP layer 502 into individual audio streams for transmission to the individual terminal apparatuses 104 that are participating in the group video
- Video may be captured locally by the camera 512 and encoded by the video encoder 514.
- the encoded captured video may be replicated at the RTP layer 502 into individual video steams for transmission to the individual terminal apparatuses 104 that are participating in the group video communication by way of the IP/UDP layer 504 and Protocol layer 506.
- the individual audio streams received from the terminal apparatuses participating in the group video communication may come through the protocol layer 506 and passed to the IP/UDP layer 504, which may provide the audio streams to the RTP layer 502.
- the received audio streams may, for example, be multiplexed together at the RTP layer 502. However, it will be appreciated that multiplexing of the received audio streams may be performed earlier or later in the audio processing chain, such as at another protocol layer.
- the combined audio stream may be decoded by the audio decoder 518 and output by the speaker 516, or other audio output, such as, an earpiece.
- the individual video streams received from the terminal apparatuses participating in the group video communication may come through the protocol layer 506 and may be passed to the IP/UDP layer 504, which may provide the video streams to the RTP layer 502.
- the individual video streams may be decoded by the video decoder 522 and displayed in split screen mode on the display 520.
- the frame rate of a received video stream may, for example, be reduced artificially at the RTP layer 502 such as by dropping one or more frames, (for example, dropping non-intraframes) of the received video stream. It will be appreciated, however, that any artificial reduction of the frame rate may be performed earlier or later in the video processing chain, such as at another protocol layer.
- This artificial reduction may be performed for video streams received from terminal apparatuses that do not understand the adjustment signaling described herein or if the terminal apparatuses have not yet reacted to the adjustment signaling. Additionally or alternatively, artificial reduction may be performed if additional media data reduction is needed to avoid overburdening the video decoder 522.
- FIG. 6 illustrates a signaling diagram according to an example flow of signals exchanged between terminals participating in a group video communication wherein a video communication apparatus 102 may be displaying received video streams in split screen mode according to an example embodiment of the invention.
- the video communication apparatus 102 may be engaged in a group video communication with three terminal apparatuses - the terminal B 602, terminal C 604, and terminal D 606.
- the user of the video communication apparatus 102 may switch to split screen display mode.
- the media stream management circuitry 118 may detect the activation of split screen mode and may, in response, determine that the media data being transmitted in the video streams received from the terminals 602-606 should be reduced. Accordingly, the media stream
- management circuitry 118 may determine an adjustment lowering the bandwidth, frame rate, resolution, and/or the like of the video streams received from terminal B 602, terminal C 604, and terminal D 606.
- operation 615 may comprise the video communication apparatus 102 transmitting an indication of the determined adjustment to the terminal B 602 to trigger the terminal B 602 to reduce the amount of media data transmitted to the video communication apparatus 102 in accordance with the determined adjustment.
- Operation 620 may comprise the video communication apparatus 102 transmitting an indication of the determined adjustment to the terminal C 604 to trigger the terminal C 604 to reduce the amount of media data transmitted to the video
- Operation 625 may comprise the video communication apparatus 102 transmitting an indication of the determined adjustment to the terminal D 606 to trigger the terminal D 606 to reduce the amount of media data transmitted to the video communication apparatus 102 in accordance with the determined adjustment.
- the terminals 602-606 may receive the messages transmitted in operations 615-625, respectively, and adjust the amount of media data transmitted in their respective media streams in accordance with the adjustments indicated in the respective messages.
- the terminals 602-606 may further transmit an acknowledgement to the video communication apparatus in operations 630-640, respectively.
- the video communication apparatus 102 may receive video streams having a lower bandwidth, frame rate, resolution, and/or the like from the terminals 602-606 in response to the indications transmitted in operations 615- 625.
- FIG. 7 illustrates processing of received media streams in a group video communication according to example embodiments of the invention wherein one of a plurality of video streams may be displayed in full screen mode.
- FIG. 7 illustrates an example structure that may be used by example embodiments of the video communication apparatus 102 wherein RTP may be used to facilitate transmission of media streams in a group video communication involving four parties.
- the media stream management circuitry 118 and/or communication interface 114 may implement an RTP layer 702, Internet Protocol (IP)/User Datagram Protocol (UDP) layer 704, and protocol layer 706 to enable the transmission and receipt of media streams in support of the group video communication.
- the protocol layer 706 may be configured to implement any protocol used for communication over the network 106.
- the protocol layer 706 may comprise a wireless local area network (WLAN) protocol layer, Third Generation cellular communication (3G) protocol layer, Long Term Evolution (LTE) protocol layer, high speed packet access (HSPA) protocol layer, and/or the like.
- WLAN wireless local area network
- 3G Third Generation cellular communication
- LTE Long Term Evolution
- HSPA high speed packet access
- the media stream management circuitry 118 may additionally implement or control an audio encoder 710, video encoder 714, audio decoder 718, and video decoder 722.
- audio may be captured locally by a microphone 708 and encoded by the audio encoder 710.
- the encoded captured audio may be replicated at the RTP layer 702 into individual audio streams for transmission to the individual terminal apparatuses 104 that are participating in the group video
- Video may be captured locally by the camera 712 and encoded by the video encoder 714.
- the encoded captured video may be replicated at the RTP layer 702 into individual video steams for transmission to the individual terminal apparatuses 104 that are participating in the group video communication by way of the IP/UDP layer 704 and Protocol layer 706.
- the individual audio streams received from the terminal apparatuses participating in the group video communication may come through the protocol layer 706 and passed to the IP/UDP layer 704, which may provide the audio streams to the RTP layer 702.
- the received audio streams may, for example, be multiplexed together at the RTP layer 702. However, it will be appreciated that multiplexing of the received audio streams may be performed earlier or later in the audio processing chain, such as at another protocol layer.
- the combined audio stream may be decoded by the audio decoder 718 and output by the speaker 716, or other audio output device.
- the video streams received from the terminal apparatuses participating in the group video communication may come through the protocol layer 706 and passed to the IP/UDP layer 704, which may provide the video streams to the RTP layer 702.
- the RTP layer 702 or a previous layer may drop media data received for the individual video streams that are not displayed in the full screen mode.
- the video stream that is displayed in the full screen mode may be passed to the video decoder 722 by the RTP layer 702.
- the video decoder 722 may decode the video stream, which may be displayed in full screen mode on the display 720. If needed, the frame rate of a received video stream may be reduced artificially at the RTP layer 702, such as by dropping one or more non intra frames of the received video stream.
- This artificial reduction may be performed for video streams received from terminal apparatuses that do not understand the adjustment signaling described herein or have not yet reacted to the signaling. Additionally or alternatively, artificial reduction may be performed if additional media data reduction is needed to avoid overburdening the video decoder 722.
- FIG. 8 illustrates a signaling diagram according to an example flow of signals exchanged between terminals participating in a group video communication wherein a video communication apparatus 102 may be displaying a single video stream in full screen mode according to example embodiments of the invention.
- the video communication apparatus 102 may be engaged in a group video communication with three terminal apparatuses - the terminal B 802, terminal C 804, and terminal D 806.
- the user of the video communication apparatus 102 may switch to full screen display mode with the video stream received from the terminal B 802 being the video stream selected for display in the full screen mode.
- the media stream management circuitry 118 may detect the activation of full screen mode and may, in response, determine that the amount of media data being transmitted in the video stream received from terminal B 802 should be increased to support the full screen display. In this regard, the media stream management circuitry 118 may, for example, determine that the amount of media data being transmitted in the video stream received from terminal B 802 should be increased to a maximum bandwidth, maximum frame rate, maximum resolution, and/or the like. The media stream management circuitry 118 may further determine that the amount of media data being transmitted in the video streams received from the terminals 604-606 should be reduced such that only periodic intraframes are sent in the video streams.
- operation 815 may comprise the video communication apparatus 102 transmitting a message to the terminal B 802 indicating the determined adjustment to the video stream received from the terminal B 802 to trigger the terminal B 802 to increase the amount of media data transmitted to the video
- Operation 820 may comprise the video communication apparatus 102 transmitting a message to the terminal C 804 indicating the determined adjustment to the video stream received from the terminal C 804 to trigger the terminal C 804 to reduce the amount of media data transmitted to the video communication apparatus 102 in accordance with the determined adjustment.
- Operation 825 may comprise the video communication apparatus 102 transmitting a message to the terminal D 806 indicating the determined adjustment to the video stream received from the terminal D 806 to trigger the terminal D 804 to reduce the amount of media data transmitted to the video communication apparatus 102 in accordance with the determined adjustment.
- the terminals 802-806 may receive the messages transmitted in operations 815-825, respectively, and adjust the amount of media data transmitted in their respective media streams in accordance with the adjustments indicated in the respective messages.
- the terminals 802-806 may further transmit an acknowledgement to the video communication apparatus in operations 830, 840, and 845, respectively.
- the video communication apparatus 102 may receive video having a maximum bandwidth, frame rate, resolution, and/or the like from the terminal B 802 in response to the indication transmitted in operation 815. Further, as illustrated in operation 850, the video communication apparatus 102 may receive video streams comprising only periodic intraframes from the terminal C 804 and terminal D 806 in response to the indications transmitted in operations 820 and 825.
- FIG. 9 illustrates a flowchart according to example embodiments of methods for facilitating group video communication according to an example embodiment of the invention.
- FIG. 9 illustrates operations that may, for example, be performed at the video communication apparatus 102 and/or at a terminal apparatus 104.
- the operations illustrated in and described with respect to FIG. 9 may, for example, be performed by, under control of, and/or with the assistance of one or more of the processor 110, memory 112, communication interface 114, user interface 116, or media stream management circuitry 118.
- Operation 900 may comprise determining a modification of display of a video stream.
- the video stream may comprise one of a plurality of video streams transmitted by remote terminal apparatuses and received by the video
- Operation 910 may comprise determining, based on the determined modification, an adjustment of an amount of media data being sent in and/or in association with at least one of the plurality of video streams.
- Operation 920 may comprise causing, in response to the determined
- an indication of the determined adjustment to be sent to trigger adjustment of the amount of media data being sent in and/or in association with the at least one of the plurality of video streams.
- FIG. 10 illustrates a flowchart according to example embodiments of methods for facilitating group video communication according to an example embodiment of the invention.
- FIG. 10 illustrates operations that may, for example, be performed at a terminal apparatus 104 and/or video communication apparatus 102.
- the operations illustrated in and described with respect to FIG. 10 may, for example, be performed by, under control of, and/or with the assistance of one or more of the processor 110, memory 112, communication interface 114, user interface 116, or media stream management circuitry 118.
- Operation 1000 may comprise receiving an indication of a requested adjustment of an amount of media data being sent in and/or in association with a video stream.
- Operation 1010 may comprise adjusting the amount of media data being sent in and/or in association with the video stream in response to the received indication.
- FIGs. 9-10 are flowcharts of a system, method, and computer program product according to example embodiments of the invention. It will be understood that each block of the flowcharts, and combinations of blocks in the flowcharts, may be implemented by various means, such as hardware and/or a computer program product comprising one or more computer-readable mediums having computer readable program instructions stored thereon. For example, one or more of the procedures described herein may be embodied by computer program instructions of a computer program product. In this regard, the computer program product(s) which embody the procedures described herein may be stored by one or more memory devices of a mobile terminal, server, or other computing device and executed by a processor in the computing device.
- the computer program instructions comprising the computer program product(s) which embody the procedures described above may be stored by memory devices of a plurality of computing devices.
- any such computer program product may be loaded onto a computer or other programmable apparatus to produce a machine, such that the computer program product including the instructions which execute on the computer or other programmable apparatus creates means for implementing the functions specified in the flowchart block(s).
- the computer program product may comprise one or more computer-readable memories (for example, memory 112) on which the computer program instructions may be stored such that the one or more computer-readable memories can direct a computer or other programmable apparatus to function in a particular manner, such that the computer program product comprises an article of manufacture which implements the function specified in the flowchart block(s).
- the computer program instructions of one or more computer program products may also be loaded onto a computer or other programmable apparatus (for example, the video communication apparatus 102 and/or a terminal apparatus 104) to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus implement the functions specified in the flowchart block(s).
- blocks of the flowcharts support combinations of means for performing the specified functions. It will also be understood that one or more blocks of the flowcharts, and combinations of blocks in the flowcharts, may be implemented by special purpose hardware -based computer systems which perform the specified functions, or combinations of special purpose hardware and computer program product(s).
- a suitably configured processor for example, the processor 110
- all or a portion of the elements may be configured by and operate under control of a computer program product.
- the computer program product for performing the methods of embodiments of the invention includes a computer-readable storage medium, such as the non-volatile storage medium, and computer-readable program code portions, such as a series of computer instructions, embodied in the computer- readable storage medium.
- example embodiments may enable performance limited mobile devices to engage in group video communication, such as group video calls, group conference calls, and/or the like.
- Some example embodiments provide for signaling devices transmitting video streams in a group video communication with indications of requested adjustments to the amount of data transmitted in the video streams. In some example embodiments, this signaling may be responsive to detection of a modification of display of a video stream. In this regard, in order to reduce the amount of processing required of a device receiving video streams, some example embodiments determine an adjustment to an amount of media data sent in one or more video streams so that unnecessary media data is not received.
- some example embodiments provide for determination of an adjustment so that unnecessary media data from video streams that are not displayed is not received.
- some example embodiments provide for determination of an adjustment such that the bandwidth of the plurality of displayed video streams may be reduced to a point enabling the receiving apparatus to process and display each of the displayed video streams without exceeding available media decoding resources.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
Methods and apparatuses are provided for facilitating group video communication. A method may include determining a modification of display of a video stream. The video stream may be one of multiple video streams transmitted for a group video communication. The method may additionally include determining, based on the determined modification, an adjustment of an amount of media data being sent in at least one of the video streams. The method may further include causing, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of media data being sent in the at least one of the video streams. Corresponding apparatuses are also provided.
Description
SYSTEM, METHOD, AND APPARATUS
FOR FACILITATING GROUP VIDEO COMMUNICATION
TECHNOLOGICAL FIELD
Embodiments of the present invention relate generally to communication technology and, more particularly, relate to a system, method, and apparatus for facilitating group video communication.
BACKGROUND
The modern communications era has brought about a tremendous expansion of wireline and wireless networks. Wireless and mobile networking technologies have addressed related consumer demands, while providing more flexibility and immediacy of information transfer. Concurrent with the expansion of networking technologies, an expansion in computing power has resulted in development of affordable computing devices capable of taking advantage of services made possible by modern networking technologies. This expansion in computing power has led to a reduction in the size of computing devices and given rise to a new generation of mobile devices that are capable of performing functionality that only a few years ago required processing power that could be provided only by the most advanced desktop computers. Consequently, mobile computing devices having a small form factor have become ubiquitous and are used to access network applications and services by consumers of all socioeconomic backgrounds. Nevertheless, many computing devices, and especially mobile computing devices, continue to encounter performance limitations that prevent them from taking full advantage of communication services made possible by modern networking technologies.
BRIEF SUMMARY
Systems, methods, apparatuses, and computer program products described herein facilitate group video communication. The systems, methods, apparatuses, and computer program products provided in accordance with example embodiments of the invention may provide several advantages to computing devices, computing device users, and
network operators. In this regard, example embodiments may enable performance limited mobile devices to engage in group video communication, such as group video calls, group conference calls, and/or the like. Some example embodiments provide for signaling devices transmitting video streams in a group video communication with indications of requested adjustments to the amount of data transmitted in the video streams. In some example embodiments, this signaling may be responsive to detection of a modification of display of a video stream. In this regard, in order to reduce the amount of processing required of a device receiving video streams, some example embodiments determine an adjustment to an amount of media data sent in one or more video streams so that unnecessary media data is not received. For example, if only a subset of video streams from a group video communication is displayed on a device, some example embodiments provide for determination of an adjustment so that unnecessary media data from video streams that are not displayed is not received or is received with a lower bandwidth. As another example, if multiple video streams from a group video communication are concurrently displayed, some example embodiments provide for determination of an adjustment such that the bandwidth of the plurality of displayed video streams may be reduced to a point enabling the receiving apparatus to process and display each of the displayed video streams without exceeding available media decoding resources.
In some example embodiments, a method is provided, which comprises determining a modification of display of a video stream comprising one of a plurality of video streams transmitted for a group video communication. The method of this example embodiment further comprises determining, based on the determined modification, an adjustment of an amount of media data being sent in and/or in association with at least one of the plurality of video streams. The method of this example embodiment additionally comprises causing, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of data being sent in and/or in association with the at least one of the plurality of video streams.
In other example embodiments, an apparatus is provided. The apparatus of this example embodiment comprises at least one processor and at least one memory storing computer program code, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to at least determine a modification of display of a video stream comprising one of a plurality of video streams transmitted for a group video communication. The at least one memory and stored computer program code are configured, with the at least one processor, to further
cause the apparatus of this example embodiment to determine, based on the determined modification, an adjustment of an amount of media data being sent in and/or in association with at least one of the plurality of video streams. The at least one memory and stored computer program code are configured, with the at least one processor, to additionally cause the apparatus of this example embodiment to cause, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of data being sent in and/or in association with the at least one of the plurality of video streams.
In other example embodiments, a computer program product is provided. The computer program product of this example embodiment includes at least one computer- readable storage medium having computer-readable program instructions stored therein. The program instructions of this example embodiment comprise program instructions configured to determine a modification of display of a video stream comprising one of a plurality of video streams transmitted for a group video communication. The program instructions of this example embodiment further comprise program instructions configured to determine, based on the determined modification, an adjustment of an amount of media data being sent in and/or in association with at least one of the plurality of video streams. The program instructions of this example embodiment also comprise program instructions configured to cause, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of data being sent in and/or in association with the at least one of the plurality of video streams.
In other example embodiments, an apparatus is provided that comprises means for determining a modification of display of a video stream comprising one of a plurality of video streams transmitted for a group video communication. The apparatus of this example embodiment further comprises means for determining, based on the determined modification, an adjustment of an amount of media data being sent in and/or in association with at least one of the plurality of video streams. The apparatus of this example embodiment additionally comprises means for causing, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of data being sent in and/or in association with the at least one of the plurality of video streams.
The above summary is provided merely for purposes of summarizing some example embodiments of the invention so as to provide a basic understanding of some aspects of the invention. Accordingly, it will be appreciated that the above described
example embodiments are merely examples and should not be construed to narrow the scope or spirit of the invention in any way. It will be appreciated that the scope of the invention encompasses many potential embodiments, some examples of which will be further described below, in addition to those here summarized.
BRIEF DESCRIPTION OF THE DRAWING(S)
Having thus described example embodiments of the invention in general terms, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and wherein:
FIG. 1 illustrates a block diagram of a system for facilitating group video communication according to example embodiments of the invention;
FIG. 2 is a schematic block diagram of a mobile terminal according to example embodiments of the invention;
FIG. 3 illustrates a block diagram of a video communication apparatus for facilitating group video communication according to example embodiments of the invention;
FIGs. 4a and 4b illustrate example embodiments of group video communication;
FIG. 5 illustrates processing of received media streams in a group video communication according to example embodiments of the invention;
FIG. 6 illustrates a signaling diagram according to an example flow of signals exchanged between terminals participating in a group video communication according to example embodiments of the invention;
FIG. 7 illustrates processing of received media streams in a group video communication according to example embodiments of the invention;
FIG. 8 illustrates a signaling diagram according to an example flow of signals exchanged between terminals participating in a group video communication according to example embodiments of the invention;
FIG. 9 illustrates a flowchart according to an example methods for facilitating group video communication according to example embodiments of the invention; and FIG. 10 illustrates a flowchart according to an example methods for facilitating group video communication according to example embodiments of the invention.
DETAILED DESCRIPTION
Some example embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the invention are shown. Indeed, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these example embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals refer to like elements throughout. As used herein, the terms "data," "content," "information" and similar terms may be used interchangeably to refer to data capable of being transmitted, received and/or stored in accordance with embodiments of the present invention. Thus, use of any such terms should not be taken to limit the spirit and scope of embodiments of the present invention. Further, where a computing device is described herein to receive data from another computing device, it will be appreciated that the data may be received directly from the another computing device or may be received indirectly via one or more intermediary computing devices, such as, for example, one or more servers, relays, routers, network access points, base stations, and/or the like.
Additionally, as used herein, the term 'circuitry' refers to (a) hardware-only circuit implementations (for example, implementations in analog circuitry and/or digital circuitry); (b) combinations of circuits and computer program product(s) comprising software and/or firmware instructions stored on one or more computer readable memories that work together to cause an apparatus to perform one or more functions described herein; and (c) circuits, such as, for example, a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation even if the software or firmware is not physically present. This definition of 'circuitry' applies to all uses of this term herein, including in any claims. As a further example, as used herein, the term 'circuitry' also includes an implementation comprising one or more processors and/or portion(s) thereof and accompanying software and/or firmware. As another example, the term 'circuitry' as used herein also includes, for example, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, other network device, and/or other computing device.
FIG. 1 illustrates a block diagram of a system 100 for facilitating group video communication according to example embodiments of the present invention. It will be
appreciated that the system 100 as well as the illustrations in other figures are each provided as example embodiments of the invention and should not be construed to narrow the scope or spirit of the invention in any way. In this regard, the scope of the disclosure encompasses many potential embodiments in addition to those illustrated and described herein. As such, while FIG. 1 illustrates one example of a configuration of a system for facilitating remote data processing, numerous other configurations may also be used to implement embodiments of the present invention.
In some example embodiments, the system 100 includes a video communication apparatus 102 and a plurality of terminal apparatuses 102. Although three such terminal apparatuses 104 are illustrated in FIG. 1 for purposes of example, it will be appreciated that the system 100 may comprise additional or fewer terminal apparatuses 104. Further, although the video communication apparatus 102 and terminal apparatuses 104 are separately labeled, the video communication apparatus 102 may be configured to implement functionality attributed to the terminal apparatuses 104 and a terminal apparatus 104 may be configured to implement functionality attributed to the video communication apparatus 102. In this regard, a video communication apparatus 102 may be used for participation in a group video communication with users of a plurality of terminal apparatuses 104. Accordingly, a video communication apparatus 102 and terminal apparatus 104 may each be configured to receive and transmit media streams supporting a group video communication.
According to example embodiments, the video communication apparatus 102 may be in communication with a plurality of terminal apparatuses 104 over the network 106. The network 106 may comprise a wireless network (for example, a cellular network, wireless local area network, wireless personal area network, wireless metropolitan area network, and/or the like), a wireline network, or some combination thereof, and in some example embodiments comprises a portion of the internet. As another example, the network 106 may comprise an ad hoc network formed by the video communication apparatus 102 and a plurality of terminal apparatuses 104, such as through various radio frequency communication means that may be used for device-to-device communication. In this regard, the network 106 may comprise any network enabling the communication of media streams supporting group video communication between a user of the video communication apparatus 102 and users of a plurality of terminal apparatuses 104.
According to example embodiments, the video communication apparatus 102 may be embodied as any computing device, such as, for example, a desktop computer, laptop
computer, mobile terminal, mobile computer, mobile phone, mobile communication device, game device, digital camera/camcorder, audio/video player, television device, radio receiver, digital video recorder, positioning device, wrist watch, portable digital assistant (PDA), any combination thereof, and/or the like. In this regard, the video communication apparatus 102 may be embodied as any computing device configured to engage in group video communication with a plurality of terminal apparatuses 104 over the network 106, as will be described further herein below.
According to example embodiments, a terminal apparatus 104 may be embodied as any computing device, such as, for example, a desktop computer, laptop computer, mobile terminal, mobile computer, mobile phone, mobile communication device, game device, digital camera/camcorder, audio/video player, television device, radio receiver, digital video recorder, positioning device, wrist watch, portable digital assistant (PDA), any combination thereof, and/or the like. In this regard, a terminal apparatus 104 may be embodied as any computing device configured to \ engage in group video communication with a video communication apparatus 102 over the network 106, as will be described further herein below.
In some example embodiments, one or more of the video communication apparatus 102 or terminal apparatus 104 may be embodied as a mobile terminal, such as that illustrated in FIG. 2. In this regard, FIG. 2 illustrates a block diagram of a mobile terminal 10 representative of one embodiment of a video communication apparatus 102 and/or terminal apparatus 104 in accordance with example embodiments of the present invention. It should be understood, however, that the mobile terminal 10 illustrated and hereinafter described is merely illustrative of one type of video communication apparatus 102 and/or terminal apparatus 104 that may implement and/or benefit from embodiments of the present invention and, therefore, should not be taken to limit the scope of the present invention. While several example embodiments of the electronic device are illustrated and will be hereinafter described for purposes of example, other types of electronic devices, such as mobile telephones, mobile computers, portable digital assistants (PDAs), pagers, laptop computers, desktop computers, gaming devices, televisions, and other types of electronic systems, may employ embodiments of the present invention.
According to example embodiments, the mobile terminal 10 may include an antenna 12 (or multiple antennas 12) in communication with a transmitter 14 and a receiver 16. The mobile terminal 10 may also include a processor 20 configured to provide signals to and receive signals from the transmitter and receiver, respectively. The
processor 20 may, for example, be embodied as various means including circuitry, one or more microprocessors with accompanying digital signal processor(s), one or more processor(s) without an accompanying digital signal processor, one or more coprocessors, one or more multi-core processors, one or more controllers, processing circuitry, one or more computers, various other processing elements including integrated circuits such as, for example, an ASIC (application specific integrated circuit) or FPGA (field
programmable gate array), or some combination thereof. Accordingly, although illustrated in FIG. 2 as a single processor, in some example embodiments the processor 20 may comprise a plurality of processors. These signals sent and received by the processor 20 may include signaling information in accordance with an air interface standard of an applicable cellular system, and/or any number of different wireline or wireless networking techniques, comprising but not limited to Wireless-Fidelity (Wi-Fi), wireless local access network (WLAN) techniques such as Institute of Electrical and Electronics Engineers (IEEE) 802.11, 802.16, and/or the like. In addition, these signals may include speech data, user generated data, user requested data, and/or the like. In this regard, the mobile terminal may be capable of operating with one or more air interface standards,
communication protocols, modulation types, access types, and/or the like. More particularly, the mobile terminal may be capable of operating in accordance with various first generation (1G), second generation (2G), 2.5G, third-generation (3G) communication protocols, fourth-generation (4G) communication protocols, Internet Protocol Multimedia Subsystem (IMS) communication protocols (for example, session initiation protocol (SIP)), H.323 and/or the like. For example, the mobile terminal may be capable of operating in accordance with 2G wireless communication protocols IS- 136 (Time Division Multiple Access (TDMA)), Global System for Mobile communications (GSM), IS-95 (Code Division Multiple Access (CDMA)), and/or the like. Also, for example, the mobile terminal may be capable of operating in accordance with 2.5G wireless communication protocols General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), and/or the like. Further, for example, the mobile terminal may be capable of operating in accordance with 3G wireless communication protocols such as Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access 2000 (CDMA2000), Wideband Code Division Multiple Access (WCDMA), Time Division- Synchronous Code Division Multiple Access (TD-SCDMA), and/or the like. The mobile terminal may be additionally capable of operating in accordance with 3.9G wireless communication protocols such as Long Term Evolution (LTE) or Evolved Universal
Terrestrial Radio Access Network (E-UTRAN) and/or the like. Additionally, for example, the mobile terminal may be capable of operating in accordance with fourth-generation (4G) wireless communication protocols and/or the like as well as similar wireless communication protocols that may be developed in the future.
In some example embodiments, a Narrow-band Advanced Mobile Phone System
(NAMPS), as well as Total Access Communication System (TACS), mobile terminals may also benefit from embodiments of this invention, as may dual or higher mode phones (for example, digital/analog or TDMA/CDMA/analog phones). Additionally, the mobile terminal 10 may be capable of operating according to Wireless Fidelity (Wi-Fi) or
Worldwide Interoperability for Microwave Access (WiMAX) protocols.
It is understood that the processor 20 may comprise circuitry for implementing audio/video and logic functions of the mobile terminal 10. For example, the processor 20 may comprise a digital signal processor device, a microprocessor device, an analog-to- digital converter, a digital-to-analog converter, and/or the like. Control and signal processing functions of the mobile terminal may be allocated between these devices according to their respective capabilities. The processor may additionally comprise an internal voice coder (VC) 20a, an internal data modem (DM) 20b, and/or the like. Further, the processor may comprise functionality to operate one or more software programs, which may be stored in memory. For example, the processor 20 may be capable of operating a connectivity program, such as a web browser. The connectivity program may allow the mobile terminal 10 to transmit and receive web content, such as location-based content, according to a protocol, such as Wireless Application Protocol (WAP), hypertext transfer protocol (HTTP), and/or the like. The mobile terminal 10 may be capable of using a Transmission Control Protocol/Internet Protocol (TCP/IP) to transmit and receive web content across the internet or other networks.
The mobile terminal 10 may also comprise a user interface including, for example, an earphone or speaker 24, a ringer 22, a microphone 26, a display 28, a user input interface, and/or the like, which may be operationally coupled to the processor 20. In this regard, the processor 20 may comprise user interface circuitry configured to control some functions of one or more elements of the user interface, such as, for example, the speaker 24, the ringer 22, the microphone 26, the display 28, and/or the like. The processor 20 and/or user interface circuitry comprising the processor 20 may be configured to control one or more functions of one or more elements of the user interface through computer program instructions (for example, software and/or firmware) stored on a memory
accessible to the processor 20 (for example, volatile memory 40, non-volatile memory 42, and/or the like). The mobile terminal may comprise a battery for powering various circuits related to the mobile terminal, for example, a circuit to provide mechanical vibration as a detectable output. The user input interface may comprise devices allowing the mobile terminal to receive data, such as a keypad 30, a touch display, a joystick, and/or other input device. In embodiments including a keypad, the keypad may comprise numeric (0-9) and related keys (#, *), and/or other keys for operating the mobile terminal.
According to some example embodiments, as shown in FIG. 2, the mobile terminal 10 may also include one or more means for sharing and/or obtaining data. For example, the mobile terminal may comprise a short-range radio frequency (RF) transceiver and/or interrogator 64 so data may be shared with and/or obtained from electronic devices in accordance with RF techniques. The mobile terminal may comprise other short-range transceivers, such as, for example, an infrared (IR) transceiver 66, a Bluetooth™ (BT) transceiver 68 operating using Bluetooth™ brand wireless technology developed by the Bluetooth™ Special Interest Group, a wireless universal serial bus (USB) transceiver 70 and/or the like. The Bluetooth™ transceiver 68 may be capable of operating according to ultra-low power Bluetooth™ technology (for example, Wibree™) radio standards. In this regard, the mobile terminal 10 and, in particular, the short-range transceiver may be capable of transmitting data to and/or receiving data from electronic devices within a proximity of the mobile terminal, such as within 10 meters, for example. The mobile terminal may be capable of transmitting and/or receiving data from electronic devices according to various wireless networking techniques, including Wireless Fidelity (Wi-Fi), WLAN techniques such as IEEE 802.11 techniques, IEEE 802.15 techniques, IEEE 802.16 techniques, and/or the like.
In example embodiments, the mobile terminal 10 may include a media capturing element, such as a camera, video and/or audio module, in communication with the processor 20. The media capturing element may be any means for capturing an image, video and/or audio for storage, display or transmission. For example, in an example embodiment in which the media capturing element is a camera module 36, the camera module 36 may include a digital camera capable of forming a digital image file from a captured image. In example embodiments, the digital camera of the camera module 36 may be capable of capturing a video clip. In example embodiments, the camera module 36 may include all hardware, such as a lens or other optical component(s), and software necessary for creating a digital image file from a captured image as well as a digital video
file from a captured video clip. In some example embodiments, the camera module 36 may include only the hardware needed to view an image, while a memory device of the mobile terminal 10 stores instructions for execution by the processor 20 in the form of software necessary to create a digital image file from a captured image. In some example embodiments, an object or objects within a field of view of the camera module 36 may be displayed on the display 28 of the mobile terminal 10 to illustrate a view of an image currently displayed which may be captured if desired by the user. In some example embodiments, the camera module 36 may further include a processing element such as a co-processor which assists the processor 20 in processing image data and an encoder and/or decoder for compressing and/or decompressing image data. The encoder and/or decoder may encode and/or decode according to, for example, a joint photographic experts group (JPEG) standard, a moving picture experts group (MPEG) standard, or other format.
According to some example embodiments, the mobile terminal 10 may comprise memory, such as a subscriber identity module (SIM) 38, a universal subscriber identity module (USIM), a removable user identity module (R-UIM), and/or the like, which may store information elements related to a mobile subscriber. In addition to the SIM, the mobile terminal may comprise other removable and/or fixed memory. The mobile terminal 10 may include volatile memory 40 and/or non- volatile memory 42. For example, volatile memory 40 may include Random Access Memory (RAM) including dynamic and/or static RAM, on-chip or off-chip cache memory, and/or the like. Nonvolatile memory 42, which may be embedded and/or removable, may include, for example, read-only memory, flash memory, magnetic storage devices (for example, hard disks, floppy disk drives, magnetic tape, etc.), optical disc drives and/or media, nonvolatile random access memory (NVRAM), and/or the like. Like volatile memory 40 non- volatile memory 42 may include a cache area for temporary storage of data. The memories may store one or more software programs, instructions, pieces of information, data, and/or the like which may be used by the mobile terminal for performing functions of the mobile terminal. For example, the memories may comprise an identifier, such as an international mobile equipment identification (IMEI) code, capable of uniquely identifying the mobile terminal 10.
Referring now to FIG. 3, FIG. 3 illustrates a block diagram of a video
communication apparatus 102 for facilitating group video communication according to example embodiments of the invention. According to example embodiments, the video communication apparatus 102 may include various means, such as one or more of a
processor 110, memory 112, communication interface 114, user interface 1 16, and media stream management circuitry 118 for performing the various functions herein described. These means of the video communication apparatus 102 as described herein may be embodied as, for example, circuitry, hardware elements (for example, a suitably programmed processor, combinational logic circuit, and/or the like), a computer program product comprising computer-readable program instructions (for example, software or firmware) stored on a computer-readable medium (for example, memory 1 12) that is executable by a suitably configured processing device (for example, the processor 110), or some combination thereof. As previously described, a terminal apparatus 104 may be configured to perform some of the functionality of a video communication apparatus 102. Accordingly, it will be appreciated that a terminal apparatus 104 may comprise one or more of the means of the video communication apparatus 102 illustrated in and described with respect to FIG. 3.
The processor 110 may, for example, be embodied as various means including one or more microprocessors with accompanying digital signal processor(s), one or more processor(s) without an accompanying digital signal processor, one or more coprocessors, one or more multi-core processors, one or more controllers, processing circuitry, one or more computers, various other processing elements including integrated circuits such as, for example, an ASIC (application specific integrated circuit) or FPGA (field
programmable gate array), or some combination thereof. Accordingly, although illustrated in FIG. 3 as a single processor, in some example embodiments the processor 110 may comprise a plurality of processors. The plurality of processors may be in operative communication with each other and may be collectively configured to perform one or more functionalities of the video communication apparatus 102 as described herein. In example embodiments wherein the video communication apparatus 102 is embodied as a mobile terminal 10, the processor 110 may be embodied as or comprise the processor 20. In an example embodiment, the processor 110 may be configured to execute instructions stored in the memory 112 or otherwise accessible to the processor 110. These
instructions, when executed by the processor 110, may cause the video communication apparatus 102 to perform one or more of the functionalities of the video communication apparatus 102 as described herein. As such, whether configured by hardware or software methods, or by a combination thereof, the processor 110 may comprise an entity capable of performing operations according to example embodiments of the present invention while configured accordingly. Thus, for example, when the processor 110 is embodied as
an ASIC, FPGA or the like, the processor 110 may comprise specifically configured hardware for conducting one or more operations described herein. Alternatively, as another example, when the processor 110 is embodied as an executor of instructions, such as may be stored in the memory 112, the instructions may specifically configure the processor 110 to perform one or more algorithms and operations described herein.
The memory 112 may comprise, for example, volatile memory, non- volatile memory, or some combination thereof. Although illustrated in FIG. 3 as a single memory, the memory 112 may comprise a plurality of memories. In example embodiments, the memory 112 may comprise, for example, a hard disk, random access memory, cache memory, flash memory, a compact disc read only memory (CD-ROM), digital versatile disc read only memory (DVD-ROM), an optical disc, circuitry configured to store information, or some combination thereof. In example embodiments wherein the video communication apparatus 102 is embodied as a mobile terminal 10, the memory 112 may comprise the volatile memory 40 and/or the non-volatile memory 42. The memory 112 may be configured to store information, data, applications, instructions, or the like for enabling the video communication apparatus 102 to carry out various functions in accordance with example embodiments of the present invention. For example, in some example embodiments, the memory 112 may be configured to buffer input data for processing by the processor 110. Additionally or alternatively, in some example embodiments, the memory 112 may be configured to store program instructions for execution by the processor 110. The memory 112 may store information in the form of static and/or dynamic information. This stored information may be stored and/or used by the media stream management circuitry 118 during the course of performing its functionalities.
The communication interface 114 may be embodied as any device or means embodied in circuitry, hardware, a computer program product comprising computer readable program instructions stored on a computer readable medium (for example, the memory 112) and executed by a processing device (for example, the processor 110), or a combination thereof that is configured to receive and/or transmit data from/to an entity of the system 100, such as, for example, a terminal apparatus 104. In some example embodiments, the communication interface 114 may be at least partially embodied as or otherwise controlled by the processor 110. The communication interface 114 may, for example, be in communication with the processor 110, such as via a bus. The
communication interface 114 may include, for example, an antenna, a transmitter, a
receiver, a transceiver and/or supporting hardware or software for enabling communications with one or more entities of the system 100. The communication interface 114 may be configured to receive and/or transmit data using any protocol that may be used for communications between entities of the system 100. The communication interface 114 may additionally be in communication with the memory 112, user interface 116, and/or media stream management circuitry 1 18, such as via a bus.
According to some example embodiments, the user interface 116 may be in communication with the processor 110 to receive an indication of a user input and/or to provide an audible, visual, mechanical, or other output to a user. As such, the user interface 116 may include, for example, a keyboard, a mouse, a joystick, a display, a touch screen display, a microphone, a speaker, and/or other input/output mechanisms. The user interface 116 may additionally comprise, control, or otherwise communicate with a camera configured to capture video of a user of the video communication apparatus 102 to enable participation by the user in a group video communication. In example
embodiments wherein the video communication apparatus 102 is embodied as a mobile terminal 10, the user interface 116 may comprise, control, or otherwise communicate with the camera module 36, which may be configured to capture video of a user of the video communication apparatus 102 to enable participation by the user in a group video communication. The user interface 116 additionally may be in communication with the memory 112, communication interface 114, and/or media stream management circuitry 118, such as via a bus.
According to example embodiments, the media stream management circuitry 118 may be embodied as various means, such as circuitry, hardware, a computer program product comprising computer readable program instructions stored on a computer readable medium (for example, the memory 112) and executed by a processing device (for example, the processor 1 10), or some combination thereof and, in some example embodiments, may be embodied as or otherwise controlled by the processor 110. In example embodiments wherein the media stream management circuitry 118 is embodied separately from the processor 110, the media stream management circuitry 118 may be in communication with the processor 110. The media stream management circuitry 118 may be configured to receive video and/or still images captured by a camera (for example, the camera module 36 or other camera of the user interface 116) that is intended for communication to a terminal apparatus 104 in support of a group video communication. The media stream management circuitry 118 may further be configured to receive audio
captured by a microphone or other audio input (for example, a microphone of the user interface 116) that is intended for communication to a terminal apparatus 04 in support of a group video communication. The media stream management circuitry 118 may further be in communication with one or more of the memory 112, communication interface 114, or user interface 116, such as via a bus.
In some example embodiments, the media stream management circuitry 118 may be configured to manage media streams received and transmitted by the video
communication apparatus 103 in support of a group video communication. The group video communication may be, for example, a group video call, a group conference call, a video conference having more than two video streams, or the like.
FIG. 4a illustrates an example embodiments of such a group video communication. As illustrated in FIG. 4a, the terminal 402, terminal 404, terminal 406, and terminal 408 are engaged in group video communication. The terminals 402-408 may comprise embodiments of a video communication apparatus 102 or terminal apparatus 104. Each of the terminals 402-408 may capture media content (for example, video and audio or just video), such as of a user of the respective terminal and/or of the surrounding environment of the respective terminal, and transmit that captured media content to the other terminals engaged in the group video communication. According to example embodiments, each of the terminals 402-408 may also receive media content captured and transmitted by the other terminals. The media content exchanged in a group video communication may, for example, be transmitted and received using a real-time data transport protocol. One example of a real-time data transport protocol that may be used in accordance with some example embodiments is real-time transport protocol (RTP). It will be appreciated, however, that RTP is merely an example of one protocol that may be used for transmission of media streams in support of a group video communication. Accordingly, where use of RTP is described herein, such description is for purposes of example and is not limiting. In this regard, other appropriate protocols, whether standardized or proprietary, may be substituted for RTP in other embodiments.
According to example embodiments, the audio content received by a terminal from the other terminals may be multiplexed together into a single audio stream, which may be played back to a user of the terminal. Each of the plurality of video streams received by a terminal may be displayed concurrently on a display of the terminal. Alternatively, a subset of the video streams received by a terminal may be displayed. As illustrated in FIG. 4a, a single video stream 410 may be displayed on the terminal 406. This display of
a single video stream is referred to herein as "full screen" display or full screen mode. It will be appreciated, however, that full screen display of a single video stream may, but need not cover the entirety of a display. Rather full screen display is used to describe that a portion of a display used for display of one or more video streams received in a group video communication is used only for display of a single video stream rather than a plurality of video streams. FIG. 4a also illustrates a plurality of video streams - video streams 412-416 - displayed on the terminal 408. Display of a plurality of video streams is referred to herein as "split screen" display or split screen mode. The video streams 412- 416 are displayed in non-overlapping, equally sized portions of the display. However, it will be appreciated that split screen display of a plurality of video streams may also encompass other presentations of the displayed video streams, such as, tiled, mosaic, partially overlapping, display in non-equally sized portions, or other display arrangements.
According to example embodiments, although each or the terminals 402-408 illustrated in FIG. 4a may be sending and receiving video streams with each of the other terminals, it will be appreciated that not every terminal that is party to a group video communication has to send a video stream to and/or receive a video stream from every other terminal that is party to a group video communication. In this regard, one or more terminals (e.g., terminal apparatuses 104) that are party to a group video communication may not be configured to send/receive multiple video streams. As another example, a terminal or user thereof may selectively elect to only establish video stream
communication with a subset of parties participating in a group video communication. FIG. 4b illustrates an example embodiment of group video communication where not every terminal engaged in a group video communication is group video capable. In this regard, FIG. 4b illustrates terminals 420-426, which may be engaged in a group video communication. However, only the terminal 426 is group video capable and sends a video stream to and receives a video stream from each of the other terminals.
In some example embodiments, the media stream management circuitry 118 may be configured to determine a modification of display of a video stream in a group video communication. The modification may comprise modification of a display size of a video stream, such as, increasing or decreasing a display size of the video stream. As another example, the modification may comprise cessation of display of a video stream or display of a video stream that was not previously displayed. As a further example, the
modification may comprise a modification to a display mode for the group video
communication. For example, the modification may comprise switching from a full screen mode to a split screen mode or vice versa.
According to example embodiments, the modification of display of a video stream may alter an amount of video data needed to support display of one or more video streams in the group video communication. Accordingly, the media stream management circuitry 118 may be configured in some example embodiments to determine an adjustment of an amount of media data being sent in at least one of the video streams that comprise the group video communication based on a determined modification of display of a video stream. The media stream management circuitry 118 may base determination of an adjustment at least in part upon available media decoding resources. In this regard, the processor 110, media stream management circuitry 118, and/or other element of the video communication apparatus 102 may have a finite amount of available resources for decoding media data received in a group video communication. Accordingly, the media stream management circuitry 118 may determine an adjustment of media data being sent in one or more video streams such that the aggregate of media data received and processed in the plurality of video streams that comprise the group video communication does not exceed available decoding resources. According to example embodiments, the determined adjustment for a video stream may comprise an adjustment of a bandwidth of video data, a bandwidth of audio data associated with the video stream, a frame rate of video data, a resolution of video data, a bit rate of video data, a bit rate of audio data associated with the video stream, a codec used for encoding video data, a codec used for encoding audio data associated with the video stream, a number of video frames packed to one RTP packet, a number of audio frames associated with the video stream packed to one RTP packet, some combination thereof, or the like.
According to example embodiments, when the media stream management circuitry
118 determines that a display size of a video stream has been reduced or a cessation of display of a video stream, the media stream management circuitry 118 may determine a reduction of the amount of data being sent in the video stream. In this regard, when display size of a video stream has been reduced, less data may be needed to support play out of the video stream with sufficient video quality to satisfy a user of the video communication apparatus. Accordingly, the media stream management circuitry 118 may be configured to determine a reduced amount of media data that is sufficient to allow play out of the video stream to satisfy a predefined video quality threshold without providing more media data than is required for satisfying the predefined video quality threshold at
the reduced display size. The predefined video quality threshold may be defined by a user of the video communication apparatus 102. Similarly, if a video stream is no longer displayed, then significantly less media data may be needed in the video stream because the video stream does not need to be decoded and displayed. In this regard, the media stream management circuitry 118 may determine an adjustment so that only a minimum amount of media data needed to maintain a communication channel over which the video stream is transmitted by a terminal apparatus 104.
According to example embodiments, when the media stream management circuitry 118 determines that a display size of a video stream has been increased or a video stream that was not previously displayed is to be displayed, the media stream management circuitry 118 may determine an increase of the amount of data being sent in the video stream. In this regard, when display size of a video stream has been increased, more data may be needed to support play out of the video stream without a noticeable reduction in video quality as compared to the previous smaller display size. Accordingly, the media stream management circuitry 118 may be configured to determine an increased amount of media data that may be sufficient to allow play out of the video stream to satisfy a predefined video quality threshold. The predefined video quality threshold may be defined by a user of the video communication apparatus 102. As another example, the video quality threshold may comprise the video quality at which the video stream was played out prior to the increase in the display size such that a user does not notice a drop in video quality after increasing the display size of the video stream. Similarly, if a video stream that was not previously displayed is to be displayed, additional media data may be needed to support play out of the video stream.
According to example embodiments, if the media stream management circuitry 118 determines that split screen mode has been activated, the media stream management circuitry 118 may determine a reduction in the amount of media data being sent in at least one of the video streams that comprise the group video communication. In this regard, when multiple video streams are played out in the split screen mode, each of the displayed video streams must be decoded. Accordingly, the media stream management circuitry 118 may determine an adjustment to one or more video streams such that the decoding resources of the video communication apparatus 102 are not exceeded.
According to example embodiments, when full screen mode is activated, the media stream management circuitry 118 may determine an increase in the amount of media data being sent in the video stream displayed in full screen mode. The increased amount of
media data may comprise a sufficient amount of media data to support play out of the video stream in full screen mode to satisfy a predefined video quality threshold. The media stream management circuitry 118 may additionally or alternatively determine a reduction in an amount of media data being sent in one or more video streams that are not displayed. In this regard, only a single video stream has to be decoded and played out in full screen mode. Accordingly, it may be undesirable to burden available computing and network resources with transmission of a large amount of media data in video streams that are not displayed. For those video streams that are not displayed, the media stream management circuitry 118 may determine that only intraframes should be sent. In this regard, interframes may not be needed for those video streams that are not displayed. The media stream management circuitry 118 may further determine a period with which intraframes should be sent. In this regard, the media stream management circuitry 118 may determine that intraframes should be sent at less frequent intervals than intraframes would otherwise be sent to support play out of the video stream. In this regard, the media stream management circuitry 118 may determine an interval between intraframes that may be sufficient to maintain the communication channel over which a video stream is transmitted without transmitting an excessive amount of media data.
Although many of the foregoing examples focus on an adjustment of an amount of video data, it will be appreciated that the media stream management circuitry 118 may also determine an adjustment of an amount of audio data. According to some example embodiments, the media stream management circuitry 118 may determine an adjustment to a bit rate, sampling rate, codec used to encode audio, frames per RTP packet and/or the like. Adjusting an amount of audio data associated with a video stream may provide additional means for the media stream management circuitry 118 to ensure that processing and decoding resources of the video communication apparatus 102 are not exceeded by processing media streams received in a group video communication. In addition to or in lieu of adjustment to audio/video data, the media stream management circuitry 118 may determine an adjustment to other media data transmitted in and/or in association with a video stream. Such other media data may include, for example, real-time text.
In some example embodiments, the media stream management circuitry 118 may be further configured to cause an indication of a determined adjustment of an amount of media data being sent in a video stream to be sent to a terminal apparatus 104 that is transmitting the video stream. In this regard, sending the indication of the determined adjustment may trigger the terminal apparatus 104 to adjust the amount of media data
being sent in the video stream in accordance with the determined adjustment.
Accordingly, a terminal apparatus 104 may be configured to receive an indication of an adjustment and adjust the amount of media data being sent in a video stream in accordance with the adjustment.
As it has been previously described that the video communication apparatus 102 may be configured to implement the functionality of a terminal apparatus 104, it will be appreciated that in some example embodiments a terminal apparatus 104 may send an indication of a determined adjustment to a video stream being transmitted by the video communication apparatus 102 to the video communication apparatus 102. For example, the media stream management circuitry 118 may be configured to receive an indication of an adjustment determined by a terminal apparatus 104 and adjust the amount of media data being sent in the video stream transmitted by the video communication apparatus 102 to the respective terminal apparatus 104.
In some example embodiments, an indication of a determined adjustment may be sent as a parameter to any communication signaling message such as session initiation protocol (SIP) or H.323 sent by the video communication apparatus 102 to a terminal apparatus 104. According to other example embodiments, an indication of a determined adjustment may be sent in a dedicated media stream control message. In example embodiments wherein a real-time data transport protocol is used for transmission of a video stream, a feedback mechanism of the real-time data transport protocol may be used for signaling an indication of a determined adjustment to a terminal apparatus 104. In this regard, real-time transport protocol control protocol (RTCP) may be used for signaling an indication of a determined adjustment in embodiments wherein RTP is used for transmission of a video stream. For example, an indication of a determined adjustment may be communicated to a terminal apparatus 104 by way of a temporary maximum media stream bit rate request, temporary maximum media stream bit rate notification, and/or the like.
In some example embodiments, an RTCP packet that may be used for signaling an indication of a determined adjustment to a terminal apparatus 104 may be described as follows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ header |V=2|P| Res | PT=VMA=222 | length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ I SSRC of sender I
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ video I bitrate I media +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ adjust I resolution | framerate |I| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ I intra frames I
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ video media adjusting packet may consist of two sections
The first section, the header, may be 8 octets long. The fields may have the following meaning:
version (V) : 2 bits
As described in RFC3550
padding (P) : 1 bit
As described in RFC3550
reserved (Res) : 5 bits
Reserved for future usage.
packet type (PT) : 8 bits
May contain the constant 222 to identify this as an RTCP Video Media Adjust packet.
length: 16 bits
As described in RFC3550
SSRC: 32 bits
As described in RFC3550
The second section, the video media adjust, may be X octets long. The fields may have the following meaning:
bitrate: 32 bits
Maximum bitrate of the video media in bytes per second value.
Value zero may be valid and may be interpreted as indicating that there is no bitrate limit.
resolution: 8 bits
Maximum resolution of the video, Pre-defined value mapping may be as follows,
9 - 16CIF (1408 x 1152 pixels)
8 - SXGA (1280 x 1024 pixels)
7 - SVGA (800 x 600 pixels)
6 - 4CIF (704 x 576 pixels)
5 - VGA (640 x 480 pixels)
4 - CIF (352 x 288 pixels)
3 - QVGA (320 x 240 pixels)
2 - QCIF (176 x 144 pixels)
1 - Sub-QCIF (128 x 96 pixels)
Values above nine may be reserved for future usage.
Value zero is valid and is interpreted that there is no resolution limit .
framerate: 8 bits
Maximum framerate of the video media as frames per second,
Value zero may valid and may be interpreted as indicating
there is no framerate limit.
intra frame only (I) : 1 bits
May define if only intraframes should be sent or not. Possible values ,
0 - All kind of frames including intraframes can be sent
1 - Only intraframes shall be sent.
Value zero may be valid and may be interpreted as indicating that there is no framerate limit.
reserved: 15 bits
Reserved for future usage
intraframes: 32 bits
Maximum frequency in which intrafames are generated as intra frame per milliseconds.
Value zero is valid and is interpretted that there is no intra frame limit.
It will be appreciated that the above example configuration of an RTCP packet is provided merely by way of example and not by way of limitation. Accordingly, other packet formats and other communication protocols may be used for signaling a determined adjustment in accordance with various example embodiments.
According to some example embodiments, FIG. 5 illustrates processing of received media streams in a group video communication according to an example embodiment of the invention wherein a plurality of video streams are displayed in split screen mode. In this regard, FIG. 5 illustrates an example structure that may be used by example embodiments of the video communication apparatus 102 wherein RTP may be used to facilitate transmission of media streams in a group video communication involving four parties. The media stream management circuitry 118 and/or communication interface 114 may implement an RTP layer 502, Internet Protocol (IP)/User Datagram Protocol (UDP) layer 504, and protocol layer 506 to enable the transmission and receipt of media streams in support of the group video communication. The protocol layer 506 may be configured to implement any protocol used for communication over the network 106. In this regard, the protocol layer 506 may comprise a wireless local area network (WLAN) protocol layer, Third Generation cellular communication (3G) protocol layer, Long Term Evolution
(LTE) protocol layer, high speed packet access (HSPA) protocol layer, and/or the like. In the embodiment illustrated in FIG. 5, the media stream management circuitry 118 may additionally implement or control an audio encoder 510, video encoder 514, audio decoder 518, and video decoder 522.
According to example embodiments, audio may be captured locally by a microphone 508 and encoded by the audio encoder 510. The encoded captured audio may be replicated at the RTP layer 502 into individual audio streams for transmission to the individual terminal apparatuses 104 that are participating in the group video
communication by way of the IP/UDP layer 504 and Protocol layer 506. Video may be captured locally by the camera 512 and encoded by the video encoder 514. The encoded captured video may be replicated at the RTP layer 502 into individual video steams for transmission to the individual terminal apparatuses 104 that are participating in the group video communication by way of the IP/UDP layer 504 and Protocol layer 506.
According to example embodiments, the individual audio streams received from the terminal apparatuses participating in the group video communication may come through the protocol layer 506 and passed to the IP/UDP layer 504, which may provide the audio streams to the RTP layer 502. The received audio streams may, for example, be multiplexed together at the RTP layer 502. However, it will be appreciated that multiplexing of the received audio streams may be performed earlier or later in the audio processing chain, such as at another protocol layer. The combined audio stream may be decoded by the audio decoder 518 and output by the speaker 516, or other audio output, such as, an earpiece.
According to example embodiments, the individual video streams received from the terminal apparatuses participating in the group video communication may come through the protocol layer 506 and may be passed to the IP/UDP layer 504, which may provide the video streams to the RTP layer 502. The individual video streams may be decoded by the video decoder 522 and displayed in split screen mode on the display 520. If needed, the frame rate of a received video stream may, for example, be reduced artificially at the RTP layer 502 such as by dropping one or more frames, (for example, dropping non-intraframes) of the received video stream. It will be appreciated, however, that any artificial reduction of the frame rate may be performed earlier or later in the video processing chain, such as at another protocol layer. This artificial reduction may be performed for video streams received from terminal apparatuses that do not understand the adjustment signaling described herein or if the terminal apparatuses have not yet reacted to
the adjustment signaling. Additionally or alternatively, artificial reduction may be performed if additional media data reduction is needed to avoid overburdening the video decoder 522.
According to some example embodiments, FIG. 6 illustrates a signaling diagram according to an example flow of signals exchanged between terminals participating in a group video communication wherein a video communication apparatus 102 may be displaying received video streams in split screen mode according to an example embodiment of the invention. As illustrated in FIG. 6, the video communication apparatus 102 may be engaged in a group video communication with three terminal apparatuses - the terminal B 602, terminal C 604, and terminal D 606. At operation 610, the user of the video communication apparatus 102 may switch to split screen display mode. The media stream management circuitry 118 may detect the activation of split screen mode and may, in response, determine that the media data being transmitted in the video streams received from the terminals 602-606 should be reduced. Accordingly, the media stream
management circuitry 118 may determine an adjustment lowering the bandwidth, frame rate, resolution, and/or the like of the video streams received from terminal B 602, terminal C 604, and terminal D 606.
According to some example embodiments, operation 615 may comprise the video communication apparatus 102 transmitting an indication of the determined adjustment to the terminal B 602 to trigger the terminal B 602 to reduce the amount of media data transmitted to the video communication apparatus 102 in accordance with the determined adjustment. Operation 620 may comprise the video communication apparatus 102 transmitting an indication of the determined adjustment to the terminal C 604 to trigger the terminal C 604 to reduce the amount of media data transmitted to the video
communication apparatus 102 in accordance with the determined adjustment. Operation 625 may comprise the video communication apparatus 102 transmitting an indication of the determined adjustment to the terminal D 606 to trigger the terminal D 606 to reduce the amount of media data transmitted to the video communication apparatus 102 in accordance with the determined adjustment. The terminals 602-606 may receive the messages transmitted in operations 615-625, respectively, and adjust the amount of media data transmitted in their respective media streams in accordance with the adjustments indicated in the respective messages. The terminals 602-606 may further transmit an acknowledgement to the video communication apparatus in operations 630-640, respectively. As illustrated in operation 645, the video communication apparatus 102 may
receive video streams having a lower bandwidth, frame rate, resolution, and/or the like from the terminals 602-606 in response to the indications transmitted in operations 615- 625.
Referring now to FIG. 7, FIG. 7 illustrates processing of received media streams in a group video communication according to example embodiments of the invention wherein one of a plurality of video streams may be displayed in full screen mode. In this regard, FIG. 7 illustrates an example structure that may be used by example embodiments of the video communication apparatus 102 wherein RTP may be used to facilitate transmission of media streams in a group video communication involving four parties. The media stream management circuitry 118 and/or communication interface 114 may implement an RTP layer 702, Internet Protocol (IP)/User Datagram Protocol (UDP) layer 704, and protocol layer 706 to enable the transmission and receipt of media streams in support of the group video communication. The protocol layer 706 may be configured to implement any protocol used for communication over the network 106. In this regard, the protocol layer 706 may comprise a wireless local area network (WLAN) protocol layer, Third Generation cellular communication (3G) protocol layer, Long Term Evolution (LTE) protocol layer, high speed packet access (HSPA) protocol layer, and/or the like. In the embodiment illustrated in FIG. 7, the media stream management circuitry 118 may additionally implement or control an audio encoder 710, video encoder 714, audio decoder 718, and video decoder 722.
According to some example embodiments, audio may be captured locally by a microphone 708 and encoded by the audio encoder 710. The encoded captured audio may be replicated at the RTP layer 702 into individual audio streams for transmission to the individual terminal apparatuses 104 that are participating in the group video
communication by way of the IP/UDP layer 704 and Protocol layer 706. Video may be captured locally by the camera 712 and encoded by the video encoder 714. The encoded captured video may be replicated at the RTP layer 702 into individual video steams for transmission to the individual terminal apparatuses 104 that are participating in the group video communication by way of the IP/UDP layer 704 and Protocol layer 706.
According to some example embodiments, the individual audio streams received from the terminal apparatuses participating in the group video communication may come through the protocol layer 706 and passed to the IP/UDP layer 704, which may provide the audio streams to the RTP layer 702. The received audio streams may, for example, be multiplexed together at the RTP layer 702. However, it will be appreciated that
multiplexing of the received audio streams may be performed earlier or later in the audio processing chain, such as at another protocol layer. The combined audio stream may be decoded by the audio decoder 718 and output by the speaker 716, or other audio output device.
According to some example embodiments, the video streams received from the terminal apparatuses participating in the group video communication may come through the protocol layer 706 and passed to the IP/UDP layer 704, which may provide the video streams to the RTP layer 702. The RTP layer 702 or a previous layer may drop media data received for the individual video streams that are not displayed in the full screen mode. The video stream that is displayed in the full screen mode may be passed to the video decoder 722 by the RTP layer 702. The video decoder 722 may decode the video stream, which may be displayed in full screen mode on the display 720. If needed, the frame rate of a received video stream may be reduced artificially at the RTP layer 702, such as by dropping one or more non intra frames of the received video stream. This artificial reduction may be performed for video streams received from terminal apparatuses that do not understand the adjustment signaling described herein or have not yet reacted to the signaling. Additionally or alternatively, artificial reduction may be performed if additional media data reduction is needed to avoid overburdening the video decoder 722.
FIG. 8 illustrates a signaling diagram according to an example flow of signals exchanged between terminals participating in a group video communication wherein a video communication apparatus 102 may be displaying a single video stream in full screen mode according to example embodiments of the invention. As illustrated in FIG. 8, the video communication apparatus 102 may be engaged in a group video communication with three terminal apparatuses - the terminal B 802, terminal C 804, and terminal D 806. At operation 810, the user of the video communication apparatus 102 may switch to full screen display mode with the video stream received from the terminal B 802 being the video stream selected for display in the full screen mode. The media stream management circuitry 118 may detect the activation of full screen mode and may, in response, determine that the amount of media data being transmitted in the video stream received from terminal B 802 should be increased to support the full screen display. In this regard, the media stream management circuitry 118 may, for example, determine that the amount of media data being transmitted in the video stream received from terminal B 802 should be increased to a maximum bandwidth, maximum frame rate, maximum resolution, and/or the like. The media stream management circuitry 118 may further determine that the
amount of media data being transmitted in the video streams received from the terminals 604-606 should be reduced such that only periodic intraframes are sent in the video streams.
According to example embodiments, operation 815 may comprise the video communication apparatus 102 transmitting a message to the terminal B 802 indicating the determined adjustment to the video stream received from the terminal B 802 to trigger the terminal B 802 to increase the amount of media data transmitted to the video
communication apparatus 102 in accordance with the determined adjustment. Operation 820 may comprise the video communication apparatus 102 transmitting a message to the terminal C 804 indicating the determined adjustment to the video stream received from the terminal C 804 to trigger the terminal C 804 to reduce the amount of media data transmitted to the video communication apparatus 102 in accordance with the determined adjustment. Operation 825 may comprise the video communication apparatus 102 transmitting a message to the terminal D 806 indicating the determined adjustment to the video stream received from the terminal D 806 to trigger the terminal D 804 to reduce the amount of media data transmitted to the video communication apparatus 102 in accordance with the determined adjustment. The terminals 802-806 may receive the messages transmitted in operations 815-825, respectively, and adjust the amount of media data transmitted in their respective media streams in accordance with the adjustments indicated in the respective messages. The terminals 802-806 may further transmit an acknowledgement to the video communication apparatus in operations 830, 840, and 845, respectively.
According to example embodiments, at operation 835, the video communication apparatus 102 may receive video having a maximum bandwidth, frame rate, resolution, and/or the like from the terminal B 802 in response to the indication transmitted in operation 815. Further, as illustrated in operation 850, the video communication apparatus 102 may receive video streams comprising only periodic intraframes from the terminal C 804 and terminal D 806 in response to the indications transmitted in operations 820 and 825.
FIG. 9 illustrates a flowchart according to example embodiments of methods for facilitating group video communication according to an example embodiment of the invention. In this regard, FIG. 9 illustrates operations that may, for example, be performed at the video communication apparatus 102 and/or at a terminal apparatus 104. The operations illustrated in and described with respect to FIG. 9 may, for example, be
performed by, under control of, and/or with the assistance of one or more of the processor 110, memory 112, communication interface 114, user interface 116, or media stream management circuitry 118. Operation 900 may comprise determining a modification of display of a video stream. The video stream may comprise one of a plurality of video streams transmitted by remote terminal apparatuses and received by the video
communication apparatus 102 for a group video communication. Operation 910 may comprise determining, based on the determined modification, an adjustment of an amount of media data being sent in and/or in association with at least one of the plurality of video streams. Operation 920 may comprise causing, in response to the determined
modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of media data being sent in and/or in association with the at least one of the plurality of video streams.
FIG. 10 illustrates a flowchart according to example embodiments of methods for facilitating group video communication according to an example embodiment of the invention. In this regard, FIG. 10 illustrates operations that may, for example, be performed at a terminal apparatus 104 and/or video communication apparatus 102. The operations illustrated in and described with respect to FIG. 10 may, for example, be performed by, under control of, and/or with the assistance of one or more of the processor 110, memory 112, communication interface 114, user interface 116, or media stream management circuitry 118. Operation 1000 may comprise receiving an indication of a requested adjustment of an amount of media data being sent in and/or in association with a video stream. Operation 1010 may comprise adjusting the amount of media data being sent in and/or in association with the video stream in response to the received indication.
FIGs. 9-10 are flowcharts of a system, method, and computer program product according to example embodiments of the invention. It will be understood that each block of the flowcharts, and combinations of blocks in the flowcharts, may be implemented by various means, such as hardware and/or a computer program product comprising one or more computer-readable mediums having computer readable program instructions stored thereon. For example, one or more of the procedures described herein may be embodied by computer program instructions of a computer program product. In this regard, the computer program product(s) which embody the procedures described herein may be stored by one or more memory devices of a mobile terminal, server, or other computing device and executed by a processor in the computing device. In some embodiments, the computer program instructions comprising the computer program product(s) which
embody the procedures described above may be stored by memory devices of a plurality of computing devices. As will be appreciated, any such computer program product may be loaded onto a computer or other programmable apparatus to produce a machine, such that the computer program product including the instructions which execute on the computer or other programmable apparatus creates means for implementing the functions specified in the flowchart block(s). Further, the computer program product may comprise one or more computer-readable memories (for example, memory 112) on which the computer program instructions may be stored such that the one or more computer-readable memories can direct a computer or other programmable apparatus to function in a particular manner, such that the computer program product comprises an article of manufacture which implements the function specified in the flowchart block(s). The computer program instructions of one or more computer program products may also be loaded onto a computer or other programmable apparatus (for example, the video communication apparatus 102 and/or a terminal apparatus 104) to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus implement the functions specified in the flowchart block(s).
Accordingly, blocks of the flowcharts support combinations of means for performing the specified functions. It will also be understood that one or more blocks of the flowcharts, and combinations of blocks in the flowcharts, may be implemented by special purpose hardware -based computer systems which perform the specified functions, or combinations of special purpose hardware and computer program product(s).
The above described functions may be carried out in many ways. For example, any suitable means for carrying out each of the functions described above may be employed to carry out embodiments of the invention. In one embodiment, a suitably configured processor (for example, the processor 110) may provide all or a portion of the elements. In another embodiment, all or a portion of the elements may be configured by and operate under control of a computer program product. The computer program product for performing the methods of embodiments of the invention includes a computer-readable storage medium, such as the non-volatile storage medium, and computer-readable program code portions, such as a series of computer instructions, embodied in the computer- readable storage medium.
As such, then, some embodiments of the invention provide several advantages to computing devices, computing device users, and network operators. In this regard, example embodiments may enable performance limited mobile devices to engage in group video communication, such as group video calls, group conference calls, and/or the like. Some example embodiments provide for signaling devices transmitting video streams in a group video communication with indications of requested adjustments to the amount of data transmitted in the video streams. In some example embodiments, this signaling may be responsive to detection of a modification of display of a video stream. In this regard, in order to reduce the amount of processing required of a device receiving video streams, some example embodiments determine an adjustment to an amount of media data sent in one or more video streams so that unnecessary media data is not received. For example, if only a subset of video streams from a group video communication is displayed on a device, some example embodiments provide for determination of an adjustment so that unnecessary media data from video streams that are not displayed is not received. As another example, if multiple video streams from a group video communication are concurrently displayed, some example embodiments provide for determination of an adjustment such that the bandwidth of the plurality of displayed video streams may be reduced to a point enabling the receiving apparatus to process and display each of the displayed video streams without exceeding available media decoding resources.
Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings.
Therefore, it is to be understood that the embodiments of the invention are not to be limited to the specific embodiments disclosed and that modifications and other
embodiments are intended to be included within the scope of the invention. Moreover, although the foregoing descriptions and the associated drawings describe example embodiments in the context of certain example combinations of elements and/or functions, it should be appreciated that different combinations of elements and/or functions may be provided by alternative embodiments without departing from the scope of the invention. In this regard, for example, different combinations of elements and/or functions than those explicitly described above are also contemplated within the scope of the invention.
Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
Claims
1. A method comprising:
determining a modification of display of a video stream, the video stream comprising one of a plurality of video streams transmitted for a group video
communication;
determining, based on the determined modification, an adjustment of an amount of media data being sent in or in association with at least one of the plurality of video streams; and
causing, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of media data being sent in or in association with the at least one of the plurality of video streams.
2. The method of Claim 1, wherein:
determining a modification of display of a video stream comprises determining a reduction in display size of a video stream or determining a cessation of display of a stream; and
determining an adjustment of an amount of media data comprises determining a reduction in an amount of data being sent in or in association with the video stream for which a modification of display is determined.
3. The method of Claim 1, wherein:
determining a modification of display of a video stream comprises determining an increase in display size of a video stream or determining a display of a video stream that was not previously displayed; and
determining an adjustment of an amount of media data being comprises determining an increase in an amount of data being sent in or in association with the video stream for which a modification of display is determined.
4. The method of any of Claims 1-3, wherein determining a modification of display of a video stream comprises determining a switch between a full screen mode in which one of the plurality of video streams is displayed and a split screen mode in which two or more of the plurality of video streams are displayed.
5. The method of any of Claims 1-4, wherein:
determining a modification of display of a video stream comprises determining activation of a split screen mode in which two or more of the plurality of video streams are concurrently displayed; and
determining an adjustment of an amount of media data being sent comprises determining a reduction in the amount of media data being sent in or or in association with at least one of the plurality of video streams.
6. The method of any of Claims 1-4, wherein:
determining a modification of display of a video stream comprises determining activation of a full screen mode in which only one of the plurality of video streams is displayed; and
determining an adjustment of an amount of media data being sent comprises determining an increase in an amount of media data being sent in or in association with the one of the plurality of video streams that is displayed.
7. The method of Claim 6, wherein determining an adjustment of an amount of media data being sent further comprises determining a reduction in an amount of media data being sent in or in association with at least one video stream that is not displayed.
8. The method of any of Claims 6-7, wherein determining an adjustment of an amount of media data being sent further comprises determining that only periodic video intraframes should be sent in at least one video stream that is not displayed.
9. The method of any of Claims 1-8, wherein determining an adjustment of an amount of media data being sent comprises determining, for a video stream that is not displayed, a minimum amount of data sufficient to maintain a communication channel over which the video stream that is not displayed is transmitted.
10. The method of any of Claims 1-9, wherein determining an adjustment comprises determining an adjustment of an amount of media data being sent in or in association with at least one of the plurality of video streams so that available media decoding resources are not exceeded.
11. The method of any of Claims 1-10, wherein determining an adjustment comprises determining an adjustment of one or more of a bandwidth of video data, a bandwidth of audio data, a frame rate of video data, a resolution of video data, a bit rate of video data, a codec used for encoding video data, or a codec used for encoding audio data.
12. The method of any of Claims 1-11, wherein the plurality of video streams are transmitted using a real-time data transport protocol, and wherein causing an indication of the determined adjustment to be sent comprises causing the indication of the determined adjustment to be sent using a feedback mechanism of the real-time data transport protocol.
13. The method of any of Claims 1-12, wherein the adjustment is determined by media stream management circuitry.
14. The method of any of Claims 1-12, wherein the adjustment is determined by a processor.
15. The method of any of Claims 1-14, wherein the group video
communication comprises a group video call, a group conference call, or a video conference having more than two video streams.
16. The method of any of Claims 1-15, wherein determining a modification of display of a video stream comprises determining a modification of display of a video stream on a display of a mobile terminal used for participation in the group video communication.
17. An apparatus comprising at least one processor and at least one memory storing computer program code, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to at least:
determine a modification of display of a video stream, the video stream comprising one of a plurality of video streams transmitted for a group video communication; determine, based on the determined modification, an adjustment of an amount of media data being sent in or in association with at least one of the plurality of video streams; and
cause, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of media data being sent in or in association with the at least one of the plurality of video streams.
18. The apparatus of Claim 17, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to:
determine a modification of display of a video stream by determining a reduction in display size of a video stream or determining a cessation of display of a stream; and determine an adjustment of an amount of media data being sent by determining a reduction in an amount of data being sent in or in association with the video stream for which a modification of display is determined.
19. The apparatus of Claim 17, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to:
determine a modification of display of a video stream by determining an increase in display size of a video stream or determining a display of a video stream that was not previously displayed; and
determine an adjustment of an amount of media data being sent by determining an increase in an amount of data being sent in or in association with the video stream for which a modification of display is determined.
20. The apparatus of any of Claims 17-19, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to determine a modification of display of a video stream by determining a switch between a full screen mode in which one of the plurality of video streams is displayed and a split screen mode in which two or more of the plurality of video streams are displayed.
21. The apparatus of any of Claims 17-20, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to:
determine a modification of display of a video stream by determining activation of a split screen mode in which two or more of the plurality of video streams are concurrently displayed; and
determine an adjustment of an amount of media data being sent by determining a reduction in the amount of media data being sent in or in association with at least one of the plurality of video streams.
22. The apparatus of any of Claims 17-20, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to:
determine a modification of display of a video stream by determining activation of a full screen mode in which only one of the plurality of video streams is displayed; and determine an adjustment of an amount of media data being sent by determining an increase in an amount of media data being sent in or in association with the one of the plurality of video streams that is displayed.
23. The apparatus of Claim 22, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to determine an adjustment of an amount of media data being sent at least in part by determining a reduction in an amount of media data being sent in or in association with at least one video stream that is not displayed.
24. The apparatus of any of Claims 22-23, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to determine an adjustment of an amount of media data being sent at least in part by determining that only periodic video intraframes should be sent in at least one video stream that is not displayed.
25. The apparatus of any of Claims 17-24, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to determine an adjustment of an amount of media data being sent at least in part by determining, for a video stream that is not displayed, a minimum amount of data sufficient to maintain a communication channel over which the video stream that is not displayed is transmitted.
26. The apparatus of any of Claims 17-25, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to determine an adjustment at least in part by determining an adjustment of an amount of media data being sent in or in association with at least one of the plurality of video streams so that available media decoding resources are not exceeded.
27. The apparatus of any of Claims 17-26, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to determine an adjustment by determining an adjustment of one or more of a bandwidth of video data, a bandwidth of audio data, a frame rate of video data, a resolution of video data, a bit rate of video data, a codec used for encoding video data, or a codec used for encoding audio data.
28. The apparatus of any of Claims 17-27, wherein the plurality of video streams are transmitted using a real-time data transport protocol, and wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to cause an indication of the determined adjustment to be sent by causing the indication of the determined adjustment to be sent using a feedback mechanism of the real-time data transport protocol.
29. The apparatus of any of Claims 17-28, wherein the group video
communication comprises a group video call, a group conference call, or a video conference having more than two video streams.
30. The apparatus of any of Claims 17-29, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to determine a modification of display of a video stream by determining a modification of display of a video stream on a display of a mobile terminal used for participation in the group video communication.
31. The apparatus of any of Claims 17-30, wherein the apparatus comprises or is embodied on a mobile phone, the mobile phone comprising user interface circuitry and user interface software stored on one or more of the at least one memory; wherein the user interface circuitry and user interface software are configured to:
facilitate user control of at least some functions of the mobile phone through use of a display; and
cause at least a portion of a user interface of the mobile phone to be displayed on the display to facilitate user control of at least some functions of the mobile phone.
32. A computer program product comprising at least one computer-readable storage medium having computer-readable program instructions stored therein, the computer-readable program instructions comprising:
program instructions configured to determine a modification of display of a video stream, the video stream comprising one of a plurality of video streams transmitted for a group video communication;
program instructions configured to determine, based on the determined
modification, an adjustment of an amount of media data being sent in or in association with at least one of the plurality of video streams; and
program instructions configured to cause, in response to the determined
modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of media data being sent in or in association with the at least one of the plurality of video streams.
33. The computer program product of Claim 32, wherein:
the program instructions configured to determine a modification of display of a video stream comprise program instructions configured to determine a reduction in display size of a video stream or determining a cessation of display of a stream; and
the program instructions configured to determine an adjustment of an amount of media data being sent comprise program instructions configured to determine a reduction in an amount of data being sent in or in association with the video stream for which a modification of display is determined.
The computer program product of Claim 32, wherein: the program instructions configured to determine a modification of display of a video stream comprise program instructions configured to determine an increase in display size of a video stream or determining a display of a video stream that was not previously displayed; and
the program instructions configured to determine an adjustment of an amount of media data being sent comprise program instructions configured to determine an increase in an amount of data being sent in or in association with the video stream for which a modification of display is determined.
35. The computer program product of any of Claims 32-34, wherein the program instructions configured to determine a modification of display of a video stream comprise program instructions configured to determine a switch between a full screen mode in which one of the plurality of video streams is displayed and a split screen mode in which two or more of the plurality of video streams are displayed.
36. The computer program product of any of Claims 32-35, wherein:
the program instructions configured to determine a modification of display of a video stream comprise program instructions configured to determine activation of a split screen mode in which two or more of the plurality of video streams are concurrently displayed; and
the program instructions configured to determine an adjustment of an amount of media data being sent comprise program instructions configured to determine a reduction in the amount of media data being sent in or in association with at least one of the plurality of video streams.
37. The computer program product of any of Claims 32-35, wherein:
the program instructions configured to determine a modification of display of a video stream comprise program instructions configured to determine activation of a full screen mode in which only one of the plurality of video streams is displayed; and
the program instructions configured to determine an adjustment of an amount of media data being sent comprise program instructions configured to determine an increase in an amount of media data being sent in or in association with the one of the plurality of video streams that is displayed.
38. The computer program product of Claim 37, wherein the program instructions configured to determine an adjustment of an amount of media data being sent comprise program instructions configured to determine a reduction in an amount of media data being sent in or in association with at least one video stream that is not displayed.
39. The computer program product of any of Claims 37-38, wherein the program instructions configured to determine an adjustment of an amount of media data being sent comprise program instructions configured to determine that only periodic video intraframes should be sent in at least one video stream that is not displayed.
40. The computer program product of any of Claims 32-39, wherein the program instructions configured to determine an adjustment of an amount of media data being sent comprise program instructions configured to determine, for a video stream that is not displayed, a minimum amount of data sufficient to maintain a communication channel over which the video stream that is not displayed is transmitted.
41. The computer program product of any of Claims 32-40, wherein the program instructions configured to determine an adjustment comprise program
instructions configured to determine an adjustment of an amount of media data being sent in or in association with at least one of the plurality of video streams so that available media decoding resources are not exceeded.
42. The computer program product of any of Claims 32-41 , wherein the program instructions configured to determine an adjustment comprise program
instructions configured to determine an adjustment of one or more of a bandwidth of video data, a bandwidth of audio data, a frame rate of video data, a resolution of video data, a bit rate of video data, a codec used for encoding video data, or a codec used for encoding audio data.
43. The computer program product of any of Claims 32-42, wherein the plurality of video streams are transmitted using a real-time data transport protocol, and wherein the program instructions configured to cause an indication of the determined adjustment to be sent comprise program instructions configured to cause the indication of the determined adjustment to be sent using a feedback mechanism of the real-time data transport protocol.
44. The computer program product of any of Claims 32-43, wherein the group video communication comprises a group video call, a group conference call, or a video conference having more than two video streams.
45. The computer program product of any of Claims 32-49, wherein the program instructions configured to determine a modification of display of a video stream comprise program instructions configured to determine a modification of display of a video stream on a display of a mobile terminal used for participation in the group video communication.
46. An apparatus comprising:
means for determining a modification of display of a video stream, the video stream comprising one of a plurality of video streams transmitted for a group video communication;
means for determining, based on the determined modification, an adjustment of an amount of media data being sent in or in association with at least one of the plurality of video streams; and
means for causing, in response to the determined modification, an indication of the determined adjustment to be sent to trigger adjustment of the amount of media data being sent in or in association with the at least one of the plurality of video streams.
47. The apparatus of Claim 46, wherein:
the means for determining a modification of display of a video stream comprises means for determining a reduction in display size of a video stream or determining a cessation of display of a stream; and
the means for determining an adjustment of an amount of media data being sent comprises means for determining a reduction in an amount of data being sent in or in association with the video stream for which a modification of display is determined.
48. The apparatus of Claim 46, w herein: the means for determining a modification of display of a video stream comprises means for determining an increase in display size of a video stream or determining a display of a video stream that was not previously displayed; and
the means for determining an adjustment of an amount of media data being sent comprises means for determining an increase in an amount of data being sent in or in association with the video stream for which a modification of display is determined.
49. The apparatus of any of Claims 46-48, wherein the means for determining a modification of display of a video stream comprises means for determining a switch between a full screen mode in which one of the plurality of video streams is displayed and a split screen mode in which two or more of the plurality of video streams are displayed.
50. The apparatus of any of Claims 46-49, wherein:
the means for determining a modification of display of a video stream comprises means for determining activation of a split screen mode in which two or more of the plurality of video streams are concurrently displayed; and
the means for determining an adjustment of an amount of media data being sent comprises means for determining a reduction in the amount of media data being sent in or in association with at least one of the plurality of video streams.
51. The apparatus of any of Claims 46-49, wherein:
the means for determining a modification of display of a video stream comprises means for determining activation of a full screen mode in which only one of the plurality of video streams is displayed; and
the means for determining an adjustment of an amount of media data being sent comprises means for determining an increase in an amount of media data being sent in or in association with the one of the plurality of video streams that is displayed.
52. The apparatus of claim 51 , wherein the means for determining an adjustment of an amount of media data being sent further comprises means for
determining a reduction in an amount of media data being sent in or in association with at least one video stream that is not displayed.
53. The apparatus of any of Claims 51-52, wherein the means for determining an adjustment of an amount of media data being sent further comprises means for determining that only periodic video intraframes should be sent in at least one video stream that is not displayed.
54. The apparatus of any of Claims 46-53, wherein the means for determining an adjustment of an amount of media data being sent comprises means for determining, for a video stream that is not displayed, a minimum amount of data sufficient to maintain a communication channel over which the video stream that is not displayed is transmitted.
55. The apparatus of any of Claims 46-54, wherein the means for determining an adjustment comprises means for determining an adjustment of an amount of media data being sent in or in association with at least one of the plurality of video streams so that available media decoding resources are not exceeded.
56. The apparatus of any of Claims 46-55, wherein the means for determining an adjustment comprises means for determining an adjustment of one or more of a bandwidth of video data, a bandwidth of audio data, a frame rate of video data, a resolution of video data, a bit rate of video data, a codec used for encoding video data, or a codec used for encoding audio data.
57. The apparatus of any of Claims 46-56, wherein the plurality of video streams are transmitted using a real-time data transport protocol, and wherein the means for causing an indication of the determined adjustment of be sent comprises means for causing the indication of the determined adjustment to be sent using a feedback
mechanism of the real-time data transport protocol.
58. The apparatus of any of Claims 46-57, wherein the group video
communication comprises a group video call, a group conference call, or a video conference having more than two video streams.
59. The apparatus of any of Claims 46-58, wherein the means for determining a modification of display of a video stream comprises means for determining a modification of display of a video stream on a display of a mobile terminal used for participation in the group video communication.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/IB2010/051929 WO2011138637A1 (en) | 2010-05-03 | 2010-05-03 | System, method, and apparatus for facilitating group video communication |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/IB2010/051929 WO2011138637A1 (en) | 2010-05-03 | 2010-05-03 | System, method, and apparatus for facilitating group video communication |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2011138637A1 true WO2011138637A1 (en) | 2011-11-10 |
Family
ID=44903663
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/IB2010/051929 Ceased WO2011138637A1 (en) | 2010-05-03 | 2010-05-03 | System, method, and apparatus for facilitating group video communication |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2011138637A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015089132A1 (en) * | 2013-12-10 | 2015-06-18 | Qualcomm Incorporated | System and method for balanced user experience in a multimedia conferencing community |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030174202A1 (en) * | 2000-01-13 | 2003-09-18 | Noam Eshkoli | Method and system for controlling multimedia video communication |
| EP1381237A2 (en) * | 2002-07-10 | 2004-01-14 | Seiko Epson Corporation | Multi-participant conference system with controllable content and delivery via back-channel video interface |
| US20050122392A1 (en) * | 2003-11-14 | 2005-06-09 | Tandberg Telecom As | Distributed real-time media composer |
| US7007098B1 (en) * | 2000-08-17 | 2006-02-28 | Nortel Networks Limited | Methods of controlling video signals in a video conference |
| US20080068446A1 (en) * | 2006-08-29 | 2008-03-20 | Microsoft Corporation | Techniques for managing visual compositions for a multimedia conference call |
| US20090231415A1 (en) * | 2008-03-14 | 2009-09-17 | Microsoft Corporation | Multiple Video Stream Capability Negotiation |
| US20100002069A1 (en) * | 2008-06-09 | 2010-01-07 | Alexandros Eleftheriadis | System And Method For Improved View Layout Management In Scalable Video And Audio Communication Systems |
-
2010
- 2010-05-03 WO PCT/IB2010/051929 patent/WO2011138637A1/en not_active Ceased
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030174202A1 (en) * | 2000-01-13 | 2003-09-18 | Noam Eshkoli | Method and system for controlling multimedia video communication |
| US7007098B1 (en) * | 2000-08-17 | 2006-02-28 | Nortel Networks Limited | Methods of controlling video signals in a video conference |
| EP1381237A2 (en) * | 2002-07-10 | 2004-01-14 | Seiko Epson Corporation | Multi-participant conference system with controllable content and delivery via back-channel video interface |
| US20050122392A1 (en) * | 2003-11-14 | 2005-06-09 | Tandberg Telecom As | Distributed real-time media composer |
| US20080068446A1 (en) * | 2006-08-29 | 2008-03-20 | Microsoft Corporation | Techniques for managing visual compositions for a multimedia conference call |
| US20090231415A1 (en) * | 2008-03-14 | 2009-09-17 | Microsoft Corporation | Multiple Video Stream Capability Negotiation |
| US20100002069A1 (en) * | 2008-06-09 | 2010-01-07 | Alexandros Eleftheriadis | System And Method For Improved View Layout Management In Scalable Video And Audio Communication Systems |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015089132A1 (en) * | 2013-12-10 | 2015-06-18 | Qualcomm Incorporated | System and method for balanced user experience in a multimedia conferencing community |
| CN105814514A (en) * | 2013-12-10 | 2016-07-27 | 高通股份有限公司 | Systems and methods for a balanced user experience in a multimedia conferencing community |
| US9423853B2 (en) | 2013-12-10 | 2016-08-23 | Qualcomm Incorporated | System and method for balanced user experience in a multimedia conferencing community |
| CN105814514B (en) * | 2013-12-10 | 2019-10-25 | 高通股份有限公司 | Systems and methods for a balanced user experience in a multimedia conferencing community |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101946518B (en) | Methods, apparatuses for adaptive synchronized decoding of digital video | |
| US8364024B2 (en) | Constructing video frames and synchronizing audio data in a media player from data received via a plurality of diverse protocol stack paths | |
| EP3092793B1 (en) | Interactive video conferencing | |
| US9819717B2 (en) | Video adaptation for content-aware wireless streaming | |
| US8681629B2 (en) | System and method for adjusting transmission data rates to a device in a communication network | |
| US20130042013A1 (en) | Methods, apparatuses and computer program products for enabling live sharing of data | |
| US10334223B2 (en) | System and method for multi-view video in wireless devices | |
| US10085029B2 (en) | Switching display devices in video telephony | |
| CN107079132B (en) | Feeding intra-coded video frames following port reconfiguration in video telephony | |
| KR101701742B1 (en) | Apparatus and method for live streaming between mobile communication terminals | |
| TW201424299A (en) | Sound information transmission method and packet communication system | |
| CN105979284B (en) | Mobile terminal video sharing method | |
| US20130227053A1 (en) | Method and Apparatus for Changing the Configuration of an Ongoing Streaming Session | |
| Shen et al. | Mobius: enable together-viewing video experience across two mobile devices | |
| US10686859B2 (en) | Content scenario and network condition based multimedia communication | |
| WO2011138637A1 (en) | System, method, and apparatus for facilitating group video communication | |
| EP2882164A1 (en) | Communication system, server apparatus, server apparatus controlling method and program | |
| KR20150035140A (en) | Method And Computer Device For Adaptive Video Streaming | |
| KR102109607B1 (en) | System for reducing delay of transmission and reception in communication network, and apparatus thereof | |
| GB2640153A (en) | Apparatus, method and computer program for optimisation of a gaze aware rendering and encoding process | |
| HK1149404B (en) | Communication method and communication system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 10851036 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 10851036 Country of ref document: EP Kind code of ref document: A1 |