TW200941233A - System and method for intelligently allocating client requests to server centers - Google Patents

Abstract

A system and method are described for intelligently allocating client requests to server centers provide real-time streaming interactive video. For example, one embodiment of a computer-implemented method comprises: strategically positioning a plurality of application server centers at different geographical locations; receiving a request from a client to execute an online application; determining the latency requirements based on the type of application requested by the client; and forwarding the client request to a particular application server center within the plurality based at least on the latency requirements of the requested application.

Description

200941233 IX. INSTRUCTIONS: TECHNICAL FIELD OF THE INVENTION The present disclosure is generally in the field of data processing systems that improve the user's ability to manipulate and access audio and video media. This application is entitled "Apparatus and Method fW wheless for the application of wireless video game" on December 10, 2002.

Video Gaming) No. 10/315, 460 part of the continuation (CIp) application, which has been given to the assignee of this CIP application. Q [Prior Art] Since the days of Thomas Edison, recorded audio and film media have become one aspect of society. In the early 2nd century, there were widespread distributions of recorded audio media (magnetic columns and records) and movie media (coin-operated automatic p-machines and movies), but the two technologies are still in their infancy. In the late 1920s, film and audio were combined on the basis of a bestseller, followed by a combination of color film and audio. Radio broadcasts have evolved into broadcast-selling audio media that largely support the form of advertising. When the television (τν) broadcast standard was established in the mid-1940s, television and radio were joined in the form of broadcast-selling media to bring previously recorded or live-lived movies to the home. By the middle of the 20th century, most American households had a phonograph record player for playing recorded audio media, a radio for receiving live broadcasts, and for live broadcasts. A television that broadcasts audio/video (A/V) media. Often the three "media players" (recorders, radios and TVs) group 136720.doc 200941233 are combined with a common public speaker cabinet, which becomes the "media in the media". Media choices are limited, but the media “ecosystem” is very stable. Most consumers know how to use the Media Player and enjoy the full range of capabilities. At the same time, publishers of the media (Daysday for film and television studios, and music companies) are able to distribute their media to both cinemas and families without extensive piracy or "secondary lock sales" (ie 'has Use media resale). Often, publishers do not receive revenue from secondary sales, and as a result, secondary sales reduce publishers; G's new sales can be derived from the purchasers who use the media themselves. Although there are indeed used records sold during the mid-20th century, such sales will not have a major impact on record publishers because they are different from movies or video shows (which are usually viewed once or only several times by adults). Music tracks can be listened to hundreds of people or even thousands of times. Therefore, music media is far more durable than film/video media (also, for adult consumers, it has lasting value). Once the record is purchased, if the consumer likes the music, the consumer may keep it for a long time. 0 A series of fundamental changes have been experienced since the 2G century towel period to today's (4) ecological line pair (4) fee and publisher's interests and losses. In the context of the widespread introduction of audio recorders (especially cassettes with high quality stereos), there is indeed a high degree of consumer convenience. But it also marks the beginning of a wide range of consumer media practices and piracy. Indeed, many of the payers use cassette tapes to record their own albums for convenience, but increasingly consumers (for example, students in the dorms who are ready to access each other's album collection) will pirate. Also, consumers will record 136720.doc 200941233 via radio, rather than buying a record or tape from a publisher. The emergence of consumer vcr has led to more consumer convenience, because now VCR can be set to record TV programs 'which can be viewed at a later time, and it seems to lead to the establishment of video rental industry, where the movie is designed to be Requests are based on access. The rapid development of popular home media devices since the mid-1980s has led to an unprecedented choice and convenience for consumers, and has also led to a rapid expansion of the media publishing market.

❹ Today's consumers are faced with multimedia choices and multimedia devices, many of which are tied to specific forms of media or to specific publishers. A keen media consumer may connect the stack of devices to the TVs and computers in each room of the house to create a "snake" cable to one or more televisions and/or personal computers (PCs) and a group of remote controls. (In the context of this application, the term "personal computer" or "PC" refers to any type of computer suitable for use in a home or office), including desktop computers, Macintosh® or other non-Windows computers, and Windows compatible devices, 1; 1^ variant laptops, etc.). Such devices may include video game consoles, VCRs, DVD players, audio surround sound processors/amplifiers, satellite tops, cable TV set-top boxes, and the like. In addition, for enthusiastic consumers, there may be multiple devices with similar functions due to compatibility issues. For example, you can have both HD-DVD and Blu-ray DVD players, or Microsoft Xbox® and Sony Playstation® video game systems, both of which are actually 'because some games cross the game console. Several versions of incompatibilities 'consumers may own both XBox and later versions (such as 'Xbox 3 60®'). Frequently, consumers are confused about which video 136720.doc 200941233 input and which remote end to use. Even after placing the disc in the correct player (eg DVD, HD-DVD, Blu-ray, xb〇hlPlaystati〇n), selecting the video and audio inputs for the device and finding the correct remote control, the consumer Still facing technical challenges. For example, in the case of a wide-screen DVD, the user may need to first determine the correct aspect ratio (eg, 4:3, full, zoom in, wide zoom, cinema wide, etc.) and then on their TV or monitor screen. Set the correct aspect ratio on the top. Similarly, the user may need to first determine the correct audio surround sound system format (eg, eg, ACJ, Dolby Digital, DTS, etc.) and then set the correct audio surround sound system format. Often, consumers are unaware that they may not have access to the media content of their television or audio system (for example, watching a movie that is squeezed in the wrong aspect ratio, or listening to stereo audio instead of surround sound) . Increasingly, Internet-based media devices have been added to the stack of devices. Audio devices like the Sonos® digital music system stream audio directly from the Internet. Similarly, devices like the SlingboxTM entertainment player record video and stream it over a home network or via an internetwork network, where the video can be viewed remotely on a PC. And Internet Protocol Television (IPTV) services provide cable TV-like services via digital subscriber line (DSL) or other home Internet connections. There have also been recent efforts to integrate multiple media functions into a single device, such as the Mox^ Media Center and the PC running the Windows XP Media Center version. While each of these devices provides a little convenience for the functions they perform, each lacks universal and simple access to most media. In addition, these devices are often costly manufactured for hundreds of dollars due to the need for processing and/or local storage of the 136720.doc 200941233. In addition, such modern consumer electronic devices typically consume: a large amount of power, even when idle, which consumes a large amount of power, which means that it is expensive and wastes energy over time. For example, if the consumer forgets to cut the device or switch it to a different video input, the device may continue to operate. In addition, because none of these devices is a complete solution, it must be integrated with other devices in the home, leaving the user with a mouse-line and many remote controls.

In addition, when many newer Internet-based devices work properly, they typically provide media in a more general form (compared to the form that would otherwise be available). For example, devices that stream video over the Internet often stream only video material, rather than being accompanied by dvd interactive "additional billing items", such as video "production", games, or director reviews. . This is due to the fact that interactive materials are often produced in a specific format intended for the specific device that handles interactivity at the local end. For example, each of DVD, HD-DVD, and Blu-ray Disc has its own specific interactive format. Any home media device or local computer (which may be developed to support all popular formats) will require some degree of sophistication and flexibility that would be too expensive and complicated for consumer operations. This problem is exacerbated. If a new format is introduced in the future, the local device may not have the hardware capability to support the new format, which means that the consumer will have to upgrade the local media device. For example, if a video or stereo video with a higher resolution is introduced in the later period (for example, each eye 136720.doc •10·200941233 stream), the local device may not have decoded the video. Computing power, or it may not have hardware for outputting the video in a new format (eg, assuming stereo vision is achieved by l2〇fps video synchronized with shuttered gIassess, where 60 impressions are delivered To each eye, if the consumer's video hardware can only support 60 fps video, this option will not be available in the absence of an upgraded hardware purchase). The problem of media device obsolescence and complexity is a serious problem when Tian Yan and cutting-edge interactive media (especially video games). 〇 Modern video game applications are basically divided into four main non-portable hardware platforms: Sony PlayStation® 1, 2 and 3 (PS1, PS2, and PS3);

Microsoft Xbo?^&Xb〇x 36〇8; and Sichuan prints such as Gamecube8 and PC-based games. Each of these platforms is different from the others, so that games written to be executed on one platform are typically not executed on another platform. There may also be compatibility issues between a generation of devices and next-generation devices. Even though most software game developers build software games designed to be independent of the special platform, in order to execute a specific game on a specific platform, a proprietary software layer (which is often referred to as the "game development engine") is needed to adapt. The game is used on a specific platform. Each platform is sold to the consumer or itself as a "console" (i.e., a separate box attached to the TV or monitor/speaker). Typically, video games are sold on optical media such as Blu-ray, DVD_R0M or CD_R〇M, which contain video games embodied in sophisticated real-time software applications. As home broadband speeds increase, video games are increasingly becoming available for download 0, 136720.doc -11 - 200941233 Due to the immediacy and high computing requirements of high-end video games, platform compatibility with video game software is achieved. Particular requirements (4) demanding. For example, 'we may expect a full game from a video game to a next video game (for example, from XBox to XBox 360, or from Playstati〇n 2 ("ps2") to PlayStation 3 (rps3)) Capacity is as universal as the productivity application (eg, Microsoft Word) from -pc to another pc with a faster processing unit or core. However, this is not the case for video games. Because when a generation of video games is released, video game makers often seek the highest possible performance for a given price point, 2 to frequently make dynamic architectural changes to the system so that many games written for use in previous generation systems are It won't work on a later generation system. For example, the XBox is based on the χ86 series processor, while the χΒ〇χ 36〇 is based on the PowerPC series. Techniques can be used to simulate the previous architecture, but assuming that the video game is an instant application, it is often impractical to achieve the exact same behavior in the simulation. Jt is aware of the loss of fees, video game console manufacturers and video game software publishers. For the consumer, it means keeping the old-to-one video game console and the new-generation video game console connected to τν, which is enough to play all games. For the console manufacturer, it means the cost associated with the simulation of the stomach and the new console and the slower adoption. And for publishers, it means that multiple versions of the new game may have to be released: cover all potential consumers - not only for each version of the video game, such as 'XBox, Playstati〇n'. It is also often used to give versions of each of the trademarks (for example, PS2 and PS3). For example, 136720.doc -12. 200941233 For the development of the separate version of the electronic art "Crazy 撖 〇 8" for XBox XBox 360, PS2, PS3, Gamecube, Wii and PC platforms and other platforms. Portable devices such as cellular phones and portable media players also challenge game developers. With increased benefits, these devices connect to the wireless data network and are able to download video games. However, there are a variety of cellular phones and media devices in the market that have a variety of different resolutions and computing powers. Also, because such devices typically have power consumption, cost, and weight G constraints, they typically lack high-level graphics acceleration hardware like a graphics processing unit ("GPU") (such as those manufactured by Nvidia of CAiSanta Ciara). Therefore, game software developers typically develop a given game title that is used for many different types of portable devices. The user can find: The given game title is not available #its specific cellular phone or portable media player. In the case of a home game console, the hardware platform S-seller typically asks the software game developer for the version of the game for publishing the game on its platform. Honeycomb telephony wireless communication companies often also ask game publishers for royalties to download games to cellular phones. In the case of 1> (: game, there is no royalties paid for publishing the game, but the game developers usually face due to the high consumer service burden for supporting multiple PC configurations and possible installation problems. High cost. In addition, pcs often have fewer obstacles to piracy of game software because they can be easily re-styled by technically savvy users and games can be more easily pirated and more easily distributed (eg, via Internet.) Therefore, for software game developers 136720.doc 200941233, there are costs and disadvantages found in game consoles, cellular phones and PCs. For console and Pc software game publishers In order to distribute the game through the retail channel, the publisher asks the retailer for a wholesale price below the sale price to give the retailer a profit margin. The publisher usually has to pay for the cost of manufacturing and distributing the physical media that holds the game. Retailers often ask publishers for "price protection fees" to cover possible accidental expenses (such as game sales). Out, or the price of the game is reduced, or the retailer must refund some or all of the wholesale price and/or withdraw the game from the buyer.) In addition, the retailer usually asks the publisher for the cost of selling the game in the flyer. In addition, retailers are increasingly buying back games from users who have completed the game, and then selling the games in the games they have used, usually not sharing the revenue of the used games with the game publisher. The following facts are added to the game. Publisher's cost burden: Games are often pirated and distributed via the Internet for users to download and copy for free. W As the Internet broadband speed increases and broadband connectivity becomes more widespread in the US and worldwide (Specifically, to the home and to the "Internet cafe" that rents a PC connected to the Internet), games are increasingly being distributed to PCs or consoles via downloads. Moreover, broadband connections are increasingly used to play multiple people. And large multiplayer online games (both of which are referred to in this disclosure by the acronym "MMOG") ^ such changes Lightening some of the costs and issues associated with retail distribution "Downloading online games solves some of the disadvantages of game publishers because distribution costs are usually small and there are fewer or no 136720.doc •14- 200941233 There are unsold media The cost of the game, but the downloaded game is still subject to piracy, and because of its size (often a large number of billions of bytes), it may take a very long time to download. In addition, multiple games can fill small magnetic A disk drive, such as a disk drive sold with a portable computer or with a video game console. However, the piracy problem is mitigated insofar as the game or MM〇G requires an online connection to make the game playable. Because it usually requires the user to have a valid user account. It is different from the video that can be used to capture the screen by the camera or to record the audio from the speaker by the microphone 0 | Copy the linear media (example #, video and music) 'each The video game experience is unique and cannot be copied using simple video/audio recordings. As a result, MMOGs can be protected from piracy and thus support commerce even in areas where copyright law is not enforced and pirated. For example, Vivendi SA's World of Warcraft MM〇G has been successfully deployed and has not been pirated throughout the world. And many online or MM〇G games (such as Linden Lab's "Second Life" MM〇G) generate revenue from game operators through the economic model built into the game, where assets can be brought using online tools. , sold and even established. Therefore, mechanisms other than the purchase or subscription of conventional game software can be used to pay for the use of online games. Although often due to the nature of online or MMOG, piracy can often be mitigated, online game operators still face the remaining challenges. Many games require a large number of local (i.e., in-home) processing resources to work properly on the line. If the user has a low-performance local computer (for example, a computer without a Gpu, such as a low-end laptop), it may not be able to play the game. In addition, as the game console ages, it lags far behind the current state of the art and may not be able to handle higher-order games. Even if it is assumed that the user's local PC can handle the calculation requirements of the game, there is often installation complexity. I can have drive incompatibility (for example, if a new game is downloaded, a new version of the graphics driver may be installed, which renders the previously installed game that is dependent on the old version of the graphics driver inoperable). As more games are downloaded, the console may run out of local disk space. When a defect is found and repaired, or if the game has been modified (eg 'If the game developer finds the level of the game too difficult to play or too easy to play), the complex game usually receives the download from the player during the time. Patch. These patches require a new download. But sometimes not all users complete all patch downloads. At other times, the tau fix introduces other compatibility or disk space consumption issues. Also, during game play, large data downloads may be required to provide graphics or behavioral sfl to the local PC or console. For example, if a user enters a room in a MMOG and encounters a scene or character that is composed of graphical material or has behavior that is not available on the user's home machine, then the material of the scene or character must be downloaded. If the internet connection is not fast enough, this can result in substantial delays during game play. In addition, if the scene or character encountered needs more than the storage space or computing power of the storage space or computing power of the local PC or console, it may generate a situation in which the user cannot continue in the game or must be of quality The reduced graphics continue. Therefore, online or MMOG games often limit their storage and/or computational complexity requirements. In addition, it often limits the amount of data transfer during the game. Online or MMOG games can also narrow the market for users of playable games. 136720.doc -16 · 200941233 In addition, 'technically erudite users are increasingly adding a reverse-process design to the native replica and modifying the game to make it cheating. Cheating may be as simple as pressing a flash with a repeating button that is faster than the human hand (for example, to shoot very quickly). In a game that supports asset trading in games, cheating can lead to a degree of deception that leads to fraudulent transactions involving assets of real economic value. When online or MMOG economic models are based on such asset transactions, this can lead to substantial harmful consequences for game operators. The cost of developing new games grows as consoles can produce increasingly sophisticated 0 games (such as 'having more realistic graphics (such as instant ray tracing), and more realistic behaviors (such as real-time physics simulation)) . In the early days of the video game industry, video game development was very similar to the development of application software; that is, most of the development costs were in the development of software (as opposed to the development of graphics, audio and behavioral elements or "assets"). ), such as their software development that can be developed for use in movies with a wide range of special effects. Today, many cutting-edge video game development efforts are more class-oriented than software development. (4) Film development with special effects. For example, many video games provide simulations of the 3-D world, and produce characters, props, and % i brothers that are increasingly realistic (i.e., computer graphics that appear to be realistic with live action images taken by photography). One of the most challenging aspects of photo-realistic game development is to create a human face that can't be distinguished from a living person's face. Facial capture techniques (such as 'C〇ntourm authenticity capture system developed by Mova of San Francisco, CA) capture the precise geometry of the performer's face and track the performer's face with high resolution while the performer is in motion Precise geometry. This technology allows the reproduction of the 136720.doc 200941233 3D face on a PC or game console. The 3D face can't actually be distinguished from the captured action.

Face. Accurately capturing and reproducing "photo-realistic" faces is useful in several ways. First, 'highly recognizable celebrities or athletes are often used in video games (often hired at a cost)' and imperfections may be obvious to the user', making the viewing experience distracting or unpleasant. Often, high levels of detail are required to achieve a high degree of photorealistic sensibility - potentially the need to reproduce a large number of polygons and high resolution textures (in the case where polygons and/or textures change with the movement of the face on the basis of the frame) . When a high polygon count scene with detailed texture changes quickly, the PC or game console supporting the game may not have enough RAM to store enough polygons and texture data for the required number of animated frames generated in the game segment. . In addition, a single or single-disk drive that is typically used on a 15 (or game console) is typically much slower than RAM and typically cannot keep up with the maximum data rate that the GPU can accept in rendering polygons and textures. Games usually carry most polygons and textures to the fortune, which means that the complexity and duration of a given scene is largely limited by the capacity of the ram. In the case of, for example, facial animation, this might % or the game console is limited to low-resolution faces that are not realistic, or limited to a limited number of figures before the game is paused and the polygons and textures (and the potted data) are used for more frames. The realistic face of the frame is made into an animation. When the PC: console displays a message similar to "Loading...", the viewer = the bar is moving slowly and slowly by the users of today's complex video games. The next scene is from the disk (unless there is another 136720.doc • 18 · 200941233 pieces) otherwise the word "disk" in this document refers to non-volatile optical media or magnetic media alpha and such as semiconductors. Flashing "non-disk media of memory" can take a few seconds or even a few minutes to load. This wastes time and can be quite frustrating for gamers. As discussed earlier, a large or all delay may be due to magnetic The loading time of the polygon, texture or other data of the disc, but it may also be the case that when the processor and/or GPU in the PC or console prepares the data for the scene, it takes a part of the loading time. For example, Soccer video games allow players to choose among a large number of players, groups, groups, stadiums, and weather conditions. Therefore, depending on the choice of Hetery, 'and knowing*, different polygons, textures, and other features are needed for the scene. Information (collectively referred to as "objects") (for example, different groups have different colors and patterns on their uniforms). It is possible to enumerate many or all of the various arrangements and pre-calculate many or all of the objects in advance and store the objects in Used to store the game's disk. However, if the number of arrays is large, the storage required for all objects may be too large. Can be installed on a disk (or too unrealistic to download). Therefore, existing PC and console systems are usually constrained in both the complexity and playback duration of a given scene and are prone to complex scenes. Loading time. Another significant limitation of prior art video game systems and application software systems is that they increasingly use large databases such as polygons and textures of 3D objects, which need to be Loaded into a pc or game console for processing. As mentioned above, when the libraries are stored on the disk at the local end, the databases can take a long time to load. However, if the database is It is stored in a remote location and stored via the Internet 136720.doc •19- 200941233 Take 'the time of manned time is usually much more serious. In this case, it may take a few minutes, hours, or even days to download the large library. In addition, these two stocks often generate a large amount of expense (e.g., a detailed high-altitude sailing 3D model for use in games, movies, or historical recordings) and are intended for sale to local end users. However, once the database is downloaded to the local (four), it is at risk of being pirated. In many cases, the user only wants to estimate the database to see if it is appropriate for the user (for example, when making 〇

It is desirable to download the database for the purpose of whether the user's clothing for the game character has a satisfactory appearance or appearance when the user moves. Long-term man time can be a hindrance when estimating (4) the treasury before deciding to buy. - Similar issues arise in MMOGs (specifically, games that allow users to take advantage of increasingly customized characters). For showing people? (:: or game console, which needs to be able to access a database with 31) geometry (polygons, textures, etc.) and the behavior of the character (for example, if the character has a shield, then the shield is strong enough to deflect the spear) . Typically, when MM〇G is played for the first time by a user, a large database for the character is available under the initial copy of the game, and the initial copy of the game is available on the game disc or downloaded to the disk at the local end. However, as the game progresses, if the user encounters a person or object that the database is not available at the local end (for example, if another user has created a custom character), the user or the object must be downloaded before the person or object can be displayed. database. This can result in a substantial delay in the game. Given the sophistication and complexity of video games, another challenge for video game developers and publishers in the case of prior art video game consoles is 136720.doc •20· 200941233. From three to three years, the cost is tens of millions of dollars. Assume that the new video game console platform is introduced at a rate almost every five years. 'The game developers need to start the development of their games in the years before the new game console is released, so that the video will be available when the new platform is released. The game is also available. Several consoles from competing manufacturers are sometimes released at about the same time (for example, within a year or two of each other), but it remains to be seen that each console is popular (for example, which console will produce the maximum video) Game software sales). For example, in recent console loops, the Microsoft XBox 360, Sony Playstation 3, and Nintendo Wii are scheduled to be introduced in approximately the same general time frame. But in the years leading up to these introductions, game developers must essentially “press” which console platforms will be more successful than others and invest their development resources accordingly. Film production companies must also share their limited production resources for a long time before the film is released based on the film that it estimates may be successful. Given the degree of growth in investment required for video games, game production is becoming more and more similar to film production, and game production companies routinely invest in their production resources based on their estimates of the future success of a particular video game. However, unlike film companies, this bet is not based solely on the success of the production itself; the truth is that it is based on the success of the game console on which the game is intended to execute. Simultaneously launching games on multiple consoles mitigates the risk, but this extra effort adds cost and often delays the actual release of the game. The application software and user environment on the PC are becoming more computationally intensive, dynamic and interactive, making it more visually appealing to the user and making it more useful and intuitive. For example, the new Windows VistaTM operating system 136720.doc 21 200941233 and the successive versions of the Macintosh® operating system have visual effects. Good-looking graphical tools, such as MayaTM from Autodesk, offer very cutting-edge 3D rendering and dynamic production capabilities that push the limits of current state of the art CPUs and GPUs. However, the calculation requirements for these new tools create many practical problems for users and software developers of such products. Because the visual display of the operating system (OS) must work on a variety of computers (including those that are no longer sold but can still be upgraded with the new 〇S), the 要求 Ο 要求 requirement is largely intended by os The minimum commonality limit for computers (which typically include computers that do not include GPUs). This severely limits the graphics capabilities. In addition, battery powered portable computers (e.g., laptops) limit visual display capabilities because high computing activity in the CPU or GPU typically results in higher power consumption and shorter battery life. Portable computers typically include software that automatically reduces processor activity to reduce power consumption when the processor is not being utilized. In some computer models, the user can manually reduce processor activity. For example, 'The Sony VGN-SZ280P laptop has a side labeled "Stamina" on one side (for low performance, longer battery life) and the other side is labeled "Speed" (for high performance, Switch for shorter battery life). The S must be executed on a portable computer to function even when the computer is executed at one of its peak performance capabilities. As a result, OS graphics performance often remains available at much lower computing power than current state of the art. High-end, computationally intensive applications such as Maya are often sold, and it is expected that these applications will be used for high-performance PCi. This usually yields much higher efficiency. 136720.doc •22- 200941233 can, and is more expensive and less portable, with the least commonality. As a result, such applications have a much lower target audience than generic os (or generic productivity applications, similar to Microsoft Office) and are typically sold in much lower quantities than general purpose OS software or general purpose application software. Potential audiences are subject to restrictions' because it is often difficult for prospective users to try out these computationally intensive applications in advance. For example, suppose students want to know how to use M — or already know the potential buyers of such applications Try to rely on aya before buying a purchase (this may involve buying a high-end computer that can execute Maya). When a student or prospective purchaser can download a demo version of Maya or a physical media copy of the Maya demo version, it will not be able to perform Maya to its full potential (for example, a computer that handles complex 3 scenes) A comprehensive assessment of the product. This substantially limits the audience of these high-end applications. It also makes the sale price higher because the development cost is usually paid off through the purchases that are much smaller than the number of purchases of the generic application. It is more irritating to individuals and businesses that use pirated copies of the application software. As a result, high-end application software suffers from piracy, although the publisher of the software has made great efforts to mitigate the piracy through various technologies. But even When using pirated high-end applications, it is impossible for users to rule out the need to invest in expensive current state of the art: to perform pirated copies. Therefore, although users can get a fraction of the actual retail price of the software application Use of software applications, but users of pirated software are still Buy or get an expensive PC to fully utilize the app. 136720.doc •23- 200941233 This is also true for users of high-performance pirated video games. Although pirates can get games at a fraction of the actual price of the game, But it still needs to purchase the expensive computing hardware needed to play the game properly (for example, GPU_Enhanced PC' or a high-end video game console like ΧΒοχ 360.) Assuming that video games are usually entertainment for consumers, then The extra cost of a high-end video game system can be too expensive. This situation is worse in countries where the average annual income of current workers is relatively low (relative to the current average annual income of workers in the US) (eg 'China'). A large percentage of the population has high-end video game systems or high-end PCs. In these countries, it is quite common for users to pay for the use of computers connected to the Internet. "Often." Coffee has older features that don't have high-performance features, such as GPUs that would otherwise allow players to play computationally intensive video games. Number or low-end PC. This is a key factor in the success of games executed on low-end PCs (such as Vivendi's "World of Warcraft", which is highly successful in China and is usually played in Chinese Internet cafes). Under the circumstance, the computationally intensive game Q (such as "Second Life") is less likely to play on PCs installed in Chinese Internet cafes. These games actually do not reach the low-performance PCs that only have access to Internet cafes. Users who are considering the purchase of video games and are first willing to try out the demo version of the game by downloading them to their home via the Internet. Video game demos are often the all-around version of the game, some of them. Feature deactivation, or restrictions on the amount of game play. This may involve a long process of loading billions of bytes of data before the game can be installed on a PC or console and executed on a PC or console ( It may be a few hours) ^ In the case of pc 136720.doc * 24 - 200941233, it may also involve calculating which special drivers (such as DirectX or 〇penGL drivers) the game needs, and downloading the correct ones. Version, install the correct version, and then determine if the PC can play the game. The latter step may involve determining if the PC has sufficient processing (cpiJ and GPU) capabilities, sufficient RAM, and a compatible OS (for example, some games are executed on Wind〇ws XP and not on Vista). Therefore, after attempting to perform the video game presentation process, the user may find that the video game demo is not playable (given the user's PC configuration). Worse, once the user has downloaded a new drive to try the demo, these drive versions may be incompatible with other games or applications that the user is accustomed to using on the PC, so the installation of the demo may result in the previous The game or application being operated cannot be operated. These obstacles not only frustrated users, but they also created obstacles for video game software publishers and video game developers to sell their games. Another problem that leads to non-economic benefits is related to the fact that a given pc or game console is typically designed to accommodate a particular level of performance requirements for the application and/or game. For example, some PCs have more or less RAM, a slower or faster CPU, and a slower or faster Gpu (if they have a GPU). - Some games or applications take advantage of a given pc or console. Some games or applications do not take advantage of the full computing power of a given pC or console. If the user's game or application selection does not meet the peak performance capabilities of the local PC or console, the user may be wasting money on the PC or console due to unused features. In the case of the console, the manufacturer of the platform may pay more than the cost of the console. 136720.doc -25- 200941233

Another problem that exists in the sale and enjoyment of video games involves allowing the user to watch others play the game before the user implements the purchase of the game. There are several prior art methods for recording video games for playback at a later time. For example, U.S. Patent No. 5,558,339 teaches that game state information (including game controller actions) is recorded during a "game play" on a video game client computer (owned by the same or different users). This status information can be used at a later time to replay 4 or all games (4) on a video game client computer (eg, a pc or console). &method< significant disadvantage is that for the user who views the recorded game, the user must have a video game client computer capable of playing the game and must have a video game application executed on the computer so that when In addition to the playback of the recorded game state, the game playback is equivalent. In addition, the video game application must be programmed in such a way that there is no possible difference in execution between the recorded game and the played back game. For example, the game graphics system is calculated based on the frame of the frame. For many games, depending on whether the scene is particularly complex or if there are other delays that slow down execution (eg 'on the PC' another process may be executing 'so that the CPU loop is taken from the game application), the game logic may sometimes cost - The frame time is short or longer than a frame time to calculate the graphic displayed as the next frame. In such a game, a "threshold value" frame calculated with less time than a frame time (for example, a few CPU clock cycles) may eventually appear. When using the same game state information to calculate the same - scene again, it may be easier to spend more than a few CPU clock cycles than the frame time (for example, if the CPU bus is slightly external to 136720.doc • 26- 200941233 DRAM buss are out of phase' and even if there is no other slave from the game processing taking a few milliseconds of CPU time, β

The large delay of the program 'which also introduces a delay of several CPU cycles). Therefore, when playing back the game, the frame becomes calculated in two frame times instead of a single frame time. Some behaviors are based on the frequency at which the game calculates new frames (for example, when the game samples input from the game controller). This deviation in the time reference for different behaviors does not affect the game play when the game is played, but it can cause the playback of the game to produce different results. For example, a basketball track is measured at a steady 60 fps rate, but the game controller input is sampled based on the calculated frame rate. When the game is recorded, the calculated frame rate may be 53 fps, and when replaying the game, the calculated frame rate may be 52 fpS, which can cause differences in whether basketball is prevented from entering the basket, resulting in different results. Therefore, the use of game state recording video games requires a very cautious game software design to ensure that the same game state information playback produces exactly the same results. Another prior art method for recording video games is to record only the video output of a PC or video game system (e.g., to a VCR, a DVD recorder, or to a video capture board on a PC). The video can then be replayed and replayed, or alternatively, the recorded video can be uploaded to the Internet (usually after video compression). The disadvantage of this method is that when playing back a 3 game sequence, the user is limited to viewing only the self-test viewpoint (the sequence is recorded from it). In other words, the user cannot change the view point of the scene. In addition, when the compressed video of the recorded game sequence played on the home PC or the game console via the Internet is available to other users 136720.doc -27- 200941233, even if the video system is compressed immediately, It is not possible to upload compressed video to the Internet in real time. The reason for this is because many homes connected to the Internet in the world have highly asymmetric broadband connections (for example, DSL and data modems typically have a much lower downlink bandwidth than the upload bandwidth). Compressed high resolution video sequences often have a higher bandwidth than the network's upload bandwidth, making it impossible to upload on the fly. Therefore, after playing the game sequence (perhaps for a few minutes or even hours), there will be a significant delay before another user on the Internet can view the game. Although this delay 0 is tolerable in certain situations (eg, viewing the outcome of a game player that appeared at a previous time), it eliminates the ability to watch a live game of the game (eg, a basketball tournament played by a winning player) or live broadcast. "Replay Replay J capability when playing a game. Another prior art method allows viewers with TV receivers to watch video game live broadcasts" but only under the control of TV producers. Some TV channels in the US and other countries A video game video channel is provided in which a Q TV viewer can view a particular video game user (eg, a top player participating in a tournament) on a video game channel. This is by video of the video game system (PC and/or console). The output is fed into a video distribution and processing device for the television channel. This is the case when a television channel broadcasts a live basketball game, several of which provide live feeds from different angles around the basketball court. Can then be manipulated with its video/audio processing and effects devices Output from various video game systems. For example, a TV channel can overlay text indicating the status of different players on top of video from a video game (as it can be live broadcast in the basket 136720.doc -28- 200941233 ball game During the period, the text is overwritten, and the TV channel can add the audio from the commentator (the action that τ 'discussed during the game). In addition, the video game can be output to the camera that records the video of the actual player of the game (for example, Demonstrate the emotional response of the player to the game. One of the problems with this method is that the live video feed must be instantly available for the video distribution and processing equipment of the TV channel so that it has the stimulating nature of the live broadcast. However, as previously discussed, when the video game system is implemented from the home (especially when broadcasting a package, including live video from a camera that captures the real world video of the game player), this is often Impossible. In addition, in the case of tournaments, the focus is on the players in the family who can modify the game and make Disadvantages, as previously described. For these reasons, such video game broadcasts on television channels are often configured with players and video game systems that are gathered at a common location (eg, at a television studio or in an arena). The television production equipment can accept video feeds from multiple video game systems and potential live broadcast cameras. Although these prior art video game television channels can provide very exciting performances for television viewers (which is a live broadcast campaign) The experience of similar events (for example, the same as video game players presented by "athletes") is based not only on their actions in the video game world but also on their actions in the real world. 'But such video game systems are often limited The situation where the players are very close to each other. In addition, because the television channels are broadcast, each broadcast channel can only display a video stream selected by the producer of the television channel. Due to these limitations and the cost of broadcast time, production equipment and production staff, these TV channels usually only show top players participating in top tournaments. In addition, a given television channel that broadcasts a full screen image of a video game to all television viewers displays only one video game at a time. This severely limits the choice of TV viewers. For example, a TV viewer may not be interested in a game displayed at a given time. Another viewer may only be interested in watching a particular player's game play that is not shown by the video channel at a given time. In other situations, the viewer may only be interested in watching the insider in the game. #他看者 may wish to control the view point (from which the video game is viewed), which is different from the view point selected by the production team or the like. In short, TV viewers may have countless preferences in watching video games (even if several different TV channels are available, special broadcasts on TV networks should not be preferred). For all of the above reasons, prior art video game television channels have significant limitations in presenting video games to television viewers. Another disadvantage of prior art video game systems and application software systems is that they are complex and often suffer from errors, crashes, and/or undesired and undesirable behaviors (collectively "defects"). Although games and applications typically go through debugging and tuning processes (often referred to as "software quality assurance" or SQA) before release, they are almost unchanged: once the game or application is fought to a wide audience in the field. The defect will suddenly appear. Unfortunately, it is difficult for software developers to identify and trace many defects after the release. Software developers may have difficulty recognizing defects. Even when it understands a defect, there may be only a limited amount of information that can be used to identify what caused the defect. 136720.doc -30· 200941233 For example, the user can call the game developer's consumer service hotline and leave a message stating that when the game is played, the screen begins to flash 'and then changes to solid blue and the PC freezes. It provides the SQA team with very little information that is useful in tracking down defects. One of the online connections Some games or applications sometimes provide more information under certain conditions. For example, you can sometimes use the "watchdog" process to monitor whether a game or application "crashes". The watchdog process collects information about the state of the game or application process when the game or application crashes (for example, memory). The statistics of the status, game or application progress to the stack, etc., and then upload the information to the SQA team via the Internet. But in a complex game or application, the information can take a very long time to decrypt, in order to accurately determine what the user is doing during the crash. Despite this, it is impossible to determine what sequence of events caused the crash. Another problem associated with PCs and game consoles is that they suffer from service problems that are extremely inconvenient for consumers. Service issues also affect the manufacturer of the pc or game console 'because it usually needs to send a special box to safely ship a broken PC or console, and thus incur the cost of repair (if the pc or console is under warranty) Or application software publishers may also be affected by sales losses (or online service usage) caused by PCs and/or consoles in a repaired state. Figure 1 illustrates a prior art video game system such as Sony Playstation® 3, Microsoft Xb® 360®, Nintendo WiiTM, Windows-based PC or Apple Macintosh. Each of these systems includes a central processing unit for executing code, typically I36720.doc -31 · 200941233, a graphics processing unit (GPU) for performing high-level graphics operations, and for use with external devices and Multiple forms of input/output (1/〇) for communication. For the sake of simplicity, these components are shown as being combined into a single unit. The prior art video game system of FIG. 1 is also shown to include an optical media drive 104 (eg, a DVD_R〇M drive); for storing video game programs. Code and data hard drive! 03; a network connection for playing a multiplayer game, for downloading games, patches, demos, or other media; a random access memory for storing code currently being executed by the CPU/GPU 100; Body (RAM) i〇i; game controller 1〇6 for receiving input commands from the user during game play; and display device 1〇2 (for example, SDTV/HDTV or computer monitor). The prior art system shown in Figure 1 suffers from several limitations. First, compared to the access speed of the RAM 101, the optical disk drive 1〇4 and the hard disk drive 1〇3 tend to have a much slower access speed. When operating directly via the RAM 1〇1, since the RAM 101 is usually high The bandwidth is much wider and does not suffer from the (4) long search delay of the optical disc mechanism. The CPU/Gpu i(10) can be used in practice to read the code and f material directly from the hard disk drive 103 or the optical disk drive (10). The possible number of polygons per second is much more than the number of polygons per second. However, only a limited amount of RAM is provided in such prior art systems (e.g., mu bits). Therefore, the sequence of "Loading.." is often required, where rami〇i is periodically filled with data for a scene under the video game. - Some systems attempt to load and play the game at the same time, but this can only be done when there is an e-order event (for example, if driving along the road, you can load the road while driving the car) The geometry of the building 136720.doc •32- 200941233). For complex and/or fast scene changes, this type of overlay usually does not work. For example, 'in the case where the user is in the middle of the campaign and the RAM 101 is completely filled with the information indicating the object in the view of the moment', if the user quickly moves the view to the left to view that the view is not currently loaded. The object in the RAM 101 will cause a discontinuity in motion because there is not enough time to load the new object from the hard disk drive 1 〇 3 or the optical disk drive 1 〇 4 into the RAM 101. Another problem with the system of Figure 1 is due to the limitations of the storage capacity of the hard disk drive 1〇3 and the optical media 1〇40. Although the disk storage device can be manufactured to have a relatively large storage capacity (for example, 500 million bytes or more than 500 million bytes), it is still not provided for use in current video games. Sufficient storage capacity for a particular situation. For example, as previously described, a British football video game may allow a user to select among many groups, players, and sports fields around the world. For each group, each player, and each stadium, a large amount of texture mapping and environment mapping is required to characterize the 3D surface of the world (for example, each group has a unique jersey, each requiring a unique W texture map) . One of the techniques used to solve this latter problem is that for the game, the texture and environment map are pre-computed once the user has selected the texture and environment map. This can involve a number of computationally intensive processes, including decompressing images, 3D mapping plus shadowing, organizing data structures, and the like. Therefore, when the video game performs such calculations, there may be a delay for the user. One way to reduce this delay is, in principle, to perform all such calculations when the game is first developed—including the group, the player roster, and each permutation of the playing field. The release version of the game thus 136720.doc •33- 200941233 will include all such pre-processed material stored on optical media 1〇4 or on one or more servers on the Internet, when the user makes a selection' The selected pre-processed data for a given group, player roster, and sports field selection is downloaded to the hard disk drive 1〇3 only via the Internet. However, as a practical matter, the pre-loaded material for each permutation possible in game play may easily be a terabyte of data that far exceeds the capacity of today's optical media devices. In addition, the information used for a given group, playlist, and sports field selection may easily be hundreds of millions of bytes of information or hundreds of millions of bytes. In the case of a home network connection (e.g., '10 Mbps), downloading this data via the Internet connection 1〇5 takes longer than calculating the data at the local end. Thus, the prior art gaming architecture shown in Figure 1 allows the user to experience significant delays between large scene transitions of complex games. Another problem with prior art methods such as the prior art method shown in Figure 1 is that video games tend to become higher order and require more CPU/GPU processing power over the years. Therefore, even with an unlimited amount of RAM, video game hardware requirements exceed the peak levels of processing power available in such systems. Therefore, users are required to upgrade their game hardware every few years to stay in sync (or play newer games at a lower quality level). One consequence of the trend of higher-order video games over the past is that machines that play video games for home use are often not economical because their cost is usually determined by the requirements of the highest-performance games they can support. For example, you might use XBox 360 to play a game like "War Machine (& curry f琛(4)", which requires a high-performance CPU, GPU, and hundreds of millions of bytes of ram, or 136720.doc •34 - 200941233 * You can use XBox 360 to play "pac Man", a game from the 1970s that requires only a few thousand bytes of RAM and a very low-performance CPU. In fact, XBox 360 has enough computing power to host many simultaneous "Pacman" games at the same time. In most of the week, 'the video game console is usually cut off. According to July 2006, Nielsen Entertainment is 13 years old and older. Active gamer's research 'on average' active gamers spend 14 hours a week or only 12 baht per week for all console games. This means that Pingyi video game console is at 88. Idle idle for % of time, this is an inefficient use of expensive resources. "Assume that video game consoles are often funded by manufacturers to lower the purchase price. It is expected that the funding will be based on royalties from future video game software purchases. Back), this makes a lot of sense. Video game consoles also incur the costs associated with almost any consumer electronics. For example, the electronics and mechanisms of the system need to be housed in the enclosure. Manufacturers need to provide service guarantees. Retailers selling the system are required to receive a profit on the sale of the system and/or sales of video game software. All of these factors add to the cost of the video game console, which must be financed by the manufacturer and delivered to the consumer. Or it can be funded by both manufacturers and consumers. In addition, piracy is a big problem in the video game industry. In fact, the security agencies used in each of the larger video game systems have been "broken" over the years, resulting in video games. Unauthorized copying. For example, the xb〇x 360 security system ruptured in July of the next 2 years and users can now download illegal copies online. Downloadable games (for example, for games) I36720.doc - 35- 200941233 Don't be vulnerable to piracy in a specific area of the world (where piracy is not strong) The viable market for independent video game software, because users and legal copies are generally easy to purchase pirated copies at a fraction of the cost. Also, in many parts of the world, the cost of the game console is a high percentage of revenue, so that even Piracy is under control and few people can afford the current state of the art gaming system. In addition, the market for used games has reduced the revenue of the video game industry. When users become tired of the game, they can sell the game to Selling games to other users' stores. This unauthorized but common practice significantly reduces the revenue of game publishers. Similarly, when there is a platform shift every few years, there is usually a 50% reduction in sales. . This is because when the user knows that a newer version of the platform is about to be released, the user stops buying the game for the older platform (for example, when the playstati〇n 3 is about to be released, the user stops purchasing the PlayStation 2 game). In combination, the loss of sales and the increased development costs associated with the new platform can have a significant adverse effect on the profitability of game developers. The beta new game console is also very expensive. Xb〇x 36〇, Nintend〇 wu&

Sony Playstation 3 is retailed for hundreds of dollars. High-capacity PC gaming systems can cost up to $8. This represents a significant investment by the user, in particular, considering the fact that the hardware became obsolete in a few years and many systems were purchased for the child. One of the above problems is an online game in which the game code and data host are hosted on a server and delivered to the client machine as required, and the compressed video and audio are streamed via a digital broadband network. Some of the companies I36720.doc • 36 - 200941233 (such as 'G-Cluster of Finland, which is now a subsidiary of SOFTBANK Broadmedia in Japan) currently offer these services online.

Business. Similar gaming services become available in local networks such as hotels and their networks provided by DSL and cable TV providers. The major drawback of these systems is the problem of latency, i.e., the time it takes for the signal to travel to the game server and travel from the game server, which is typically located at the "head end" of the operator. Fast motion video game (also known as "speed

A (twitch) video game requires very low latency between when the user performs an action by the game controller and when the display screen is updated to show the result of the user action. A low latency is required to make the user feel the game respond "immediately". The user is satisfied with different delay intervals depending on the type of visual game and the proficiency of the user. For example, for slow informal games (like backgammon) or slow action role-playing games, a 100 millisecond delay may be tolerable, but in fast action games, over 70 milliseconds or 80 milliseconds. The delay can cause the user to perform worse in the game, 1 and therefore is unacceptable. For example, in a game that requires fast response time, there is a sharp drop in accuracy when the delay is increased from 5 〇 milliseconds to 丨〇〇 milliseconds. Although the game or application is installed in a nearby controlled network environment or to the user's network path can be predicted and / or tolerable, peak bandwidth in the network environment ^ 'in the largest material and material In terms of the nature, it is easy to control the delay (for example, 'the user can observe the steady motion from the digital video stream through the network. The degree of control can be achieved in the following: in the head of the network. Between TV users' families, or from the DSL Central Office 136720.doc •37· 200941233 to the DSL user's home, or in the commercial office area network (LAN) environment from the server or user. Again, it is possible Obtain a specific hierarchical point-to-point private connection between businesses with guaranteed bandwidth and latency, but host the game console in a server center connected to the universal Internet and then connect via broadband And the resulting video stream to the user's game or application system, many factors cause delays, leading to severe limitations in the deployment of prior art systems. In a typical connected broadband family Users can have DSL or electrical data modems for broadband piggyback services. These broadband services typically result in up to 25 milliseconds of travel delay between the user's home and the universal Internet (and sometimes In addition, there is a round trip delay caused by routing data to the server center via the Internet. The delay via the Internet changes based on the route given the data and the delay caused by the data being routed. In addition to routing delays, it also causes round trip delays due to the speed of light passing through the fibers interconnecting most of the Internet. For example, for every 1 mile, due to the speed of light passing through the fiber and Other consumption results in a round trip delay of approximately 22 milliseconds. The extra delay can be incurred due to the data rate of the data streamed over the Internet. For example, if the user has a DSL sold under "6 Mbps DSL Service" Service, in practice, the user will most likely get at most less than 5 Mbps under the line throughput, and will likely see periodically due to various factors (such as Connection degradation caused by congestion at the Digital Subscriber Line Access Multiplexer (DSLAM) during peak load time. If there is congestion in the local shared coaxial cable via the adjacent loop, or 136720 in the network of the cable modem system .doc •38- 200941233 There is congestion elsewhere, and similar problems can arise, reducing the data rate of the cable modem used for the sale of the “6 Mbps Cable Data Service” to much less than the data rate◊ If the data packet at a stable rate of 4 Mbps is streamed unidirectionally from the server center via the connection in the User Datagram Protocol (UDp) format, if everything works properly, then the -before packet will pass Without incurring additional delays, but if there is congestion (or other obstruction) and only 3.5 Mbps can be used to stream data to the user, in a typical situation, the packet will be discarded, resulting in lost data, or the packet will be in the frame. The queue is queued until it can be sent, thereby introducing additional delays. Different congestion points have different capacity for storing delayed packets, so in some cases, the packets that cannot be successfully resolved are discarded immediately. In other cases, millions of bits of data are queued and eventually sent. However, in almost all cases, the queue at the congestion point has a capacity limit, and once it exceeds these limits, the queue will overflow and the packet will be discarded. Therefore, in order to avoid incurring additional delays (or worse, loss of packets), data rate capacity beyond the game or application server to the U user must be avoided. Delays are also incurred due to the time required to compress the video in the feeder and decompress the video in the client device. When the video game is executed on the feeder, it is calculating the frame to be displayed. The video compression algorithm currently used by I suffers from high (four) rates or high delays. For example, the 3' motion JPEG is a frame-only lossy algorithm, which is characterized in that each frame is compressed independently of each other frame of the video. When the client device receives a frame of collapsed (10) video, its 136720.doc 39-200941233 can immediately decompress the frame and display the frame, resulting in very low latency. However, because each frame is compressed separately, the algorithm cannot take advantage of the similarity between successive frames' and therefore only the in-frame video compression algorithm suffers from non-South data rates. For example, '6〇 fpS (frames per second) 640x480 motion JPEG video may require more than 40 Mbps (megabits per second) or 40 Mbps (megabits per second). These high data rates for such low resolution video windows will be too expensive in many broadband applications (and indeed for most consumer Internet-based applications). In addition, because each frame is independently compressed, artifacts in the frame that may be due to lossy compression may appear at different locations in successive frames. This can result in a visual artifact that appears to the viewer as moving when the video is decompressed. Other compression algorithms (such as MPEG2, H.264, or VC9 from Microsoft Corporation) can achieve high compression ratios when used in prior art configurations, but at the expense of high latency. These algorithms utilize inter-frame compression and in-frame compression. Periodically, the algorithms perform only in-frame compression of the frame. This type of frame is called a key frame (usually called a "!" frame). Then, these algorithms usually compare the I frame with the previous frame and the successive frames. Instead of compressing the previous frame and the subsequent frame independently, the algorithm determines the image from the I frame to the previous image. The box and subsequent frames are changed, and then their changes are stored as: "B" frame (for changes before I frame) and "P" frame (for changes after I frame). This results in a much lower data rate than just in-frame compression. However, it usually comes at the expense of higher latency. The I frame is usually much larger (usually 10 times larger) than the B or P frame, and therefore, 136720.doc -40· 200941233 takes a prolonged time proportionally at a given data rate. Consider, for example, a situation where the I frame is 10 times the size of the B frame and the P frame and for each single frame there are 29 3 frames + 3 frames? Frame = 59 frames, or a total of 6 frames for each "frame group" (G〇p). Therefore, at 60 fps~p, there is one 6-inch frame G〇p per second. Assume that the transmission channel has a maximum data rate of 2 Mbps. In order to achieve the most quality video in the channel, the compression algorithm will generate a 2 Mbps stream, and given the above ratio, this will result in 2 million bits per frame (Mb) / (59 + 1 〇) = 3 〇, 394 © bits and 303, 935 bits per I frame. When the compressed video stream is received by the decompression algorithm, in order to stably play the video, each frame needs to be decompressed and displayed at regular intervals (e.g., 60 fps). To achieve this result, if any frame is subjected to a transmission delay, all frames need to be delayed by at least one delay, so the worst case delay will define the delay for each video frame. because! The frame is the largest, so the j frame introduces the longest transmission delay, and the entire I frame will have to be received before the decompressible and display 1 frame (or between any frames of the I frame). Assume that the channel data rate is 2 Mbps ‘and transmit one! The frame will cost 3〇3,935/2 Mb=145 milliseconds. An inter-frame video compression system (as described above) that uses a large percentage of the bandwidth of the transmission channel will experience long delays due to the large size of the frame relative to the average size of the frame. Or, in other words, when the prior art interframe compression algorithm achieves a lower average data rate per frame than the in-frame compression algorithm (eg, 2 Mbps versus 40 Mbps), it still suffers from high frame size Peak data rate per frame (for example, 3〇3,935*6〇=182 Mbps). But remember that the above analysis assumes that the P and B frames are much smaller than the frame. I36720.doc •41- 200941233 Although this is generally true, this does not hold for a tile with high image complexity that is not related to previous frames, high motion, or scene changes. In these cases, the 'P frame or B frame can become larger than the j frame. (If the p frame or the closed frame becomes larger than the I frame, the central end of the reduction algorithm will usually be "forced". Item box and replace the P frame or B frame with the I frame). Therefore, the peak data rate of the frame size can appear in the digital video stream at any time. Therefore, for compressed video, when the average video data rate is close to the data rate of the transmission channel (often for this situation, given the high data rate for video, the rate is required from the I frame or the large P frame or The high peak data rate of frame B results in a high frame delay. 4 However, the above discussion only characterizes the compression algorithm delay generated by the large B frame, p frame or frame in G〇p. In the case of frame B, the delay will be even worse. This is because all b frames and I frames after the B frame must be received before the B frame can be displayed. Therefore, in the picture group (GOP) such as BBBBBIPPPPPBBBBBIPPPPP In the sequence, where Ο there are 5 B frames before each frame, the first B frame can be displayed by the video decompressor only after receiving the subsequent b frame and I frame. Therefore, if Video streaming at 60 fps (ie, 16.67 ms/frame), before receiving the first B frame, regardless of the fast channel bandwidth, receiving five B frames and I frames will cost i6 67 *6=100 milliseconds, and this is only the case of 5 b frames. There are 30 B The frame's compressed video sequence is quite common. In addition, at low channel bandwidths such as 2 Mbps, the delay due to the size of the I frame is largely added to the delay due to waiting for the B frame to arrive. Impact. Therefore, on the 2 Mbps channel, in the case of a large number of b 136720.doc -42· 200941233 frames, it is quite easy to use the prior art video compression technology for more than 500 milliseconds or more than 500 milliseconds. If you do not use the b frame (For a given quality level, at the expense of a lower compression ratio), the B-frame delay is not incurred, but still causes the delay due to the size of the peak frame described above. The problem is precisely due to the nature of many video games. And the emphasis. The video compression algorithm using the GOP structure described above is largely optimized for use with live video or movie material © which is intended for passive viewing. Usually 'camera (real The camera, or the virtual camera produced by the computer, and the scene are relatively stable, just because the camera or scene moves too far and bumpy Then the video or movie material (a) is usually unpleasant to watch 'and (b) if it is being viewed', when the camera suddenly bumps back and forth, the viewer is usually unable to follow the action closely (for example, if the camera is blowing When the child who wiped out the butterfly candle on the birthday cake was disturbed and suddenly bumped back and forth between the cakes, the viewer usually concentrated on the child and the cake, and ignored the brief interruption of the camera when it suddenly moved. In the video talk or video In the case of a conference call, the camera can be held in a fixed position and not moved at all, resulting in very little data spikes. However, 3D high motion video games are characterized by constant motion (for example, considering 3D competitions, The entire frame is in rapid motion during the duration of the competition, or considers a first-person shooter game in which the virtual camera constantly moves back and forth in a bumpy manner. These video games can produce a sequence of frames with large and frequent peaks, where the user may need to clearly see what happens during their sudden movements. Therefore, in the 3D high motion video game, the collapse of the shadow 136720.doc -43- 200941233 is far intolerable. Therefore, the video output of many video games (due to their nature) produces a compressed video stream with very high and frequent peaks. Assuming that the user of the fast motion video game has a small tolerance for high latency, and given all of the above delays, there have been limitations to video games hosted by server hosts that stream video over the Internet. In addition, users of such applications suffer from similar restrictions if a highly interactive application host is required to host a common Internet and stream video. These services require network configuration 'where the colocation server is set directly at the head end (in the case of cable broadband) or central office (in the case of digital subscriber line (DSL)), or business background In the ^^ (or on a specially graded private connection) to control the route and distance from the client device to the server to minimize latency and adapt to peaks without incurring delays. LANs (typically rated at 1 Mbps to 1 Gbps) and leased lines with sufficient bandwidth typically support peak bandwidth requirements (for example, 18 Mbps peak bandwidth is one of the 100 Mbps LAN capacity fractions). If special adaptation is applied, the peak bandwidth requirement can also be accommodated by the residential broadband base infrastructure. For example, on a cable TV system, a dedicated bandwidth can be given to a digital video message. The dedicated bandwidth can handle values such as a large I frame. In addition, 'on the DSL system' can supply higher speed dsl data machines (considering high peaks) or can be supplied with specially graded connections that can handle higher data rates. However, conventional electronic data processors and DSL infrastructure attached to the universal Internet are far from tolerable for peak bandwidth requirements for compressed video. Therefore, 'online service (hosting a video game or application host in a server center that is long distances from the client device, and 136720.doc 44 200941233 is connected via the Internet via a conventional residential broadband connection) The shrinking video output stream suffers from significant latency and peak bandwidth requirements - especially for games and applications that require very low latency (eg, first-person shooters and other multi-user, interactive action games, &; applications that need a quick response time). The present disclosure will be more fully understood from the following description of the embodiments and the accompanying drawings, which are not to be construed as limited. And understanding. Specific details (such as device type, system group communication method, etc.) are set forth in the following description in order to provide a thorough understanding of the present disclosure. However, it will be appreciated by those skilled in the art that the specific embodiments described in the practice may not require such specific details. 2a to 2b provide a high-level architecture of two embodiments, wherein the video game and the software application are hosted by the colocation service 21 and hosted by the user site 211 under the subscription service (note, "user location The user device 205 at the location of the user, wherever the user is located, if the mobile device is used, is connected via the Internet 2〇6 (or other public or private network) access. The client device 205 can be a general purpose computer having a wired or wireless connection to the Internet, with an internal or external display device 222 (such as a 'Microsoft Windows or Linux based PC or Apple Macintosh computer), or A dedicated client device (such as a wired or wireless connection to the Internet) for outputting video and audio to a monitor or television 222, such as a set top box, or it can be speculated to be 136720.doc -45-200941233 A wirelessly connected mobile device for the Internet. Any of these devices may have its own user input device (eg, a keyboard, button, touch screen, track pad or inertial-sensing wand, video capture camera and / or motion tracking camera, etc., or it can use an external input device 221 that is connected by wire or wirelessly (eg, keyboard, mouse, game controller, inertial sensor, video capture camera, and/or motion tracking) Camera, etc.). As described in more detail below, the colocation service 21 includes servers of various performance levels (including those with high-capacity CPU/GPU processing capabilities). During playback of the game or application using the colocation service 21, the home or office-to-user device 205 receives the keyboard and/or controller input from the user' and then enters the controller via the Internet 2 〇6 is transmitted to the colocation service 210, in response to which the colocation service 21 executes the game code and generates successive frames for the video output (video image sequence) of the game or application software (eg, if used) Pressing the button that will direct the person on the screen to the right, the game program will then produce a sequence of video images showing the person 0 moving to the right. The video image sequence is then compressed using a low latency video compressor and the colocation service 21 then transmits the low latency video stream over the internet 206. The home or office client device then decodes the compressed video stream and reproduces the decompressed video image on a monitor or TV. Thus, the computational and graphical hardware requirements of the client device 205 are significantly reduced. The client 205 only needs to have the processing capability for forwarding the keyboard/controller input to the Internet 206 and decoding and decompressing the compressed video stream received from the Internet 2〇6, actually 136720 .doc -46· 200941233 Any personal computer can do this on its (:1>1; software (for example, Intei dual-core cpu executed at about 2 GHz can decompress using VCs such as H.264 and Windows Media The compressor encodes 72〇p hdtv). In addition, in the case of any client device, the dedicated chip can also be much lower cost than the general-purpose CPU and much less power consumption than the general-purpose CPU (such as modern PC Required) to perform video decompression for these standards on the fly. Notably, in order to perform the functions of the transfer controller input and decompress video, the home client device 2〇5 does not require any special graphics〇 Processing unit (gpu), optical drive or hard drive (such as the prior art video game system shown in Figure i). As games and application software become more complex and more photo-realistic, Will require higher performance CPU, GPU, more RAM, and larger and faster drives, and the computing power of the colocation service 210 will be continuously upgraded, but the end user will not need to make the family Or the office client platform 205 is upgraded because the processing requirements of the court or office client platform 2〇5 will remain constant for display resolution and frame rate by a given video decompression algorithm. There are no hardware limitations and compatibility issues seen today in the system illustrated in Figure 2b.

This display reduces the game or application, including, for example, the game or application in the classroom, the form of the media, or the downloaded software. Or the possibility of illegal copying (piracy) of the application software 136720.doc •47- 200941233 Sexuality and the possibility of mitigating the piracy of valuable libraries that can be used by games or application software. In fact, if you need a specialized server (for example, a very expensive, large or noisy device) to play games or application software that is not feasible for home or office use, even if you get a game or application software. A copy of the pirated copy will not be operated in the home or office. In one embodiment, the colocation service 2 is directed to a game or application software developer who designs video games (which generally refers to a software development company, a game play or movie studio, or a game or application software publisher). 220 provides a software development tool to enable it to design games that can be executed on the colocation service 21〇. The s-specific tool allows developers to take advantage of the features of colocation services (the features are usually not available in a single iPC or game console) (for example, a very large database of fast access to complex geometries (unless otherwise conditional) Otherwise, "geometry" will be used herein to refer to the polygons, textures, rigging, lighting, behavior, and other components and parameters that define the 3D dataset). Under this architecture, different business models are possible. Under a model, the host escrow service 210 charges a subscription fee from the end user and pays royalties to the developer, as shown in Figure 2a. In an alternate implementation (shown in the figure), the developer 220 charges the subscription fee directly from the user and pays the colocation service 210 for the cost of hosting the game or application content. These basic principles are not limited to any particular business model used to host an online game or application host. Slow compression video features 136720.doc •48- 200941233 As mentioned earlier, the obvious problem with video game services or application software services is delay. The delay of 7 〇 milliseconds - 80 milliseconds (the time from when the input device is actuated by the user to when the response is displayed on the display device) is the upper limit for games and applications that require fast response times. * 'This is very difficult to achieve due to the large number of actual and physical constraints that make this architecture not shown in Figures 2a and 2b. As indicated in Figure 3, when the user subscribes to the Internet service, the connection constant S is the nominal maximum data rate to the user's home or office. 301 #Depends on the provider's policy and routing device capabilities. , | The maximum data rate can be more or less strictly enforced 'but usually for a number of different reasons, the actual available data rate is lower. For example, there may be too many network traffic at the DSL central office or at the local end of the data logger loop, or there may be noise on the electrical winding, causing the discarded seal & or the provider may Establish the maximum number of bits per month for each use. Currently, the maximum downlink data rate for cable and DSL services is typically in the range of hundreds of kilobits per second (Kbps) to 30 Mbps. Honeycomb type The data is usually limited to (four) hundred Kbps. However, (iv) the speed of services and the number of users subscribed to broadband services will increase dramatically over time. Currently, some analysts estimate that 33% of US broadband users have data rates below 2 Mbps or above 2 Mbps. For example, some analysts predict that by 2010, more than 85% of US broadband users will have data rates of 2 Mbps or more. As indicated in Figure 3, the actual available maximum data rate of 3 〇 2 can fluctuate over time. Therefore, in the case of low latency, online gaming or application software, 136720.doc -49- 200941233 I ‘it is sometimes difficult to predict the actual available data rate for a particular video stream. 2 The scene complexity and motion at a certain amount are given to the resolution of the number of frames per second ps (for example, the data rate 303 required for the quality level of 64〇x48〇@6 is higher than the real) =; = 'If indicated by the bee value in Figure 3, then it can appear = example and 3, some internet services will only discard the packet, resulting in lost data and distortion on the user's video screen. , lost image. Other services will temporarily buffer (i.e., queue) extra packets and provide them to User# at the available data rate, resulting in an increase in latency that is unacceptable for many video games and applications. Finally, some Internet service providers view the increase in data rate as a malicious attack (such as a denial of service attack (a well-known technique used by computer hackers to disable network connections)) and will be cut in a specific time period. Disconnect the user's internet connection. Thus, the embodiments described herein seek to ensure that the required data rate for a video game does not exceed the maximum available data rate.

Host Managed Service Architecture Figure 4a illustrates the architecture of a hosted service 21〇 according to an embodiment. The colocation service 210 can be located in a single server center or can be spread across multiple server centers (to provide a low latency connection to users with a lower latency to a particular server center than others) to Provides load balancing between users and provides redundancy in the event of one or more server center failures. The colocation service 21〇 can eventually include tens of thousands or even millions of feeders 402, so that the servo is very large. 136720.doc • 50- 200941233 The user base. The colocation service control system 4-1 provides overall control over the host escrow service and directs routers, feeders, video compression systems, § ten fees, and accounting systems. In one embodiment, the hosted service control system 401 is implemented on a Linux-based distributed processing system that is tied to storage for user information, server negative afl, and system hierarchy δ A RAID array of data databases. In the above description, the various actions implemented by the colocation service 2 i 起始 are initiated and controlled by the colocation service control system 401 unless otherwise attributed to other specific systems. The colocation service 210 includes a number of servers 402, such as those currently available from Intel, IBM, and Hewlett Packard, among others. Alternatively, the server 402 can be assembled into a custom component configuration, or the server 402 can ultimately be integrated to implement the entire server as a single wafer. Although this figure shows a small number of servers 402 for purposes of illustration, in actual deployments there may be as few as one server 402 or as many as millions or millions of servers 402. Servers 402 can all be the same

G mode configuration (as an example of some configuration parameters, with the same CPU type and performance; with or without GPU, and with GPU, with the same GPU type and performance; with the same number of CPU and GPU; with the same amount And the same type/speed RAM; and with the same RAM configuration), or various subsets of the server 402 can have the same configuration (for example, 25% of the feeders can be configured in a specific way, 50% of the servers Configured in a different way, and 25% of the servers are configured in a further way), or each server 4〇2 can be different. I36720.doc 51 200941233 In an embodiment, the server 402 has no disk, That is, it does not have its own local mass storage (which is an optical or magnetic storage (four), or a semiconductor-based storage, such as flash memory or other large-capacity storage components of the feeding function) Each server accesses the shared mass storage via a fast backplane or network connection. In one embodiment, the fast connection is connected to an independent redundant disk array (11 〇 1 〇) 4 〇 5 series storage area network (SAN) 4 〇 3 ' implemented in the use of ultra-high speed Ethernet There is a connection between the devices. As is known to those skilled in the art, SAN 403 can be used to combine many RAID arrays 405, resulting in extremely high bandwidth - close to or possibly exceeding the RAM available for use in current gaming consoles and PCs. The resulting bandwidth. Moreover, while RAID arrays based on rotating media such as magnetic media often have significant seek time access latency, RAID arrays based on semiconductor memories can be implemented with much lower access latency. In another configuration, some or all of the servers 4〇2 provide some or all of their own mass storage at the local end. For example, the server 402 can store frequently accessed information (such as a copy of its operating system and video game or application) on a low-latency local flash memory-based storage. The SAN can be used to access a RAID array 405 based on rotating media with two more search delays to access the large database of geometry or game state information less frequently. In addition, in an embodiment, the colocation service 21 uses the low latency video compression logic 4〇4 described in detail below. Video compression logic 404 can be implemented in software, hardware, or any combination thereof (described in its specific embodiments below). Video compression logic 404 includes logic for compressing audio and visual material 136720.doc • 52-200941233. In operation, when a video game is played via a keyboard, mouse, game controller or other input device 421 or an application at the user location 211 is used, the control signal logic 413 on the client 415 will indicate The control signal 4〇6a_b (usually in the form of a UDP packet) of the push button press (and other types of user input) is transmitted to the colocation service 21〇. In the future, the control signal from the user is routed to the appropriate server (or to multiple servers if multiple servos respond to the user's input device) 〇 402. Control signal 406a can be routed to server 402 via the SAN as illustrated in Figure 4a. Alternatively or additionally, control signals 4〇6b may be routed directly to server 402 via a hosted service network (e.g., an Ethernet-based local area network). Regardless of how the control signals 4〇6a_b are transmitted, the or the servers execute the game or application software in response to the control signals 4〇6a_b. Although not illustrated in FIG. 4a, various network connectivity components, such as firewalls and/or gateways, may be handled at the edge of the colocation service 21 (eg, between the colocation service 210 and the Internet 410) And/or incoming and outgoing traffic at the edge of the user premises 211 (between the Internet 41〇 and the home or office client 415). The graphics and audio output of the executed game or application software (ie, the new video image sequence) is provided to the low-latency video compression logic 4〇4, and the low-latency video compression logic 4〇4 is based on the low-latency video compression technique (such as The video image sequence is compressed and compressed via the Internet (or, as described below, via the best high speed network service bypassing the universal Internet). The video stream (usually with compressed or uncompressed audio) is transmitted back to 136720, doc-53·200941233 to the client 415. The low latency video decompression logic 412 on the client 415 then decompresses the video and audio streams and reproduces the decompressed video stream, and typically plays the decompressed audio stream on the display device 422. Alternatively, the audio may be played on a separate speaker from the display device 422 or not at all. Note that although input device 421 and display device 422 are shown as stand-alone devices in Figures 2a and 2b, they may be integrated into a client device such as a destructible computer or mobile device. The home or office client 415 (previously described as home or office client 205 in Figures 2a and 2b) can be a very low cost and low power device with very limited computational or graphical performance and possibly very limited The local large-capacity storage or the local large-capacity storage. In contrast, each server 4〇2 coupled to the SAN 403 and the plurality of RAIDs 405 can be an exceptionally efficient computing system, and in fact, if multiple servers are cooperatively used in a side-by-side configuration configuration, There is almost no limit to the amount of computation and graphics processing that can be tolerated. In addition, the computational power of the server 402 is provided to the user due to the low latency video compression 4〇4 and the low B video decompression 412 (perceived by the user). When the user presses a button on the input device, the image on display 422 is updated in response to the button press (with no meaningful delay in perception) as if the game or application software was executing at the local end. Thus, for a very low-performance computer or a home or office client 415 that is only a low-cost chip that implements low-latency video decompression and control signal logic 413, it is effectively available to the user from a remote location that appears to be available at the local end. Provide any computing power. This gives the user the ability to play up to 1% of processor-intensive (usually new) video games and applications with maximum performance of 136720.doc •54· 200941233. Figure 4c shows a very basic and inexpensive home or office client device 465. This device is an embodiment of the home or office client 415 from Figures 4a and 4b. It is about 2 inches long. It has an Ethernet jack 462 that interfaces with an Ethernet cable (PoE), which receives its power from the Ethernet jack 462 and its connection to the Internet. Sex. The device is capable of performing NAT within a network that supports Network Address Translation (NAT). In an office environment, 'many new Ethernet switches have ©PoE and bring PoE directly to the Ethernet jack in the office.' In this case, 'the required wall is the self-walled jack to the user. End 465 Ethernet cable. If the available Ethernet connection does not carry power (for example, in a home with a DSL or cable modem but no PoE) then there is an inexpensive wall-type "brick" (ie, power supply) available. 'It will accept a powerless Ethernet mirror and output an Ethernet with PoE. The client 465 contains control signal logic 41 3 (Fig. 4a) that is consuming the Bluetooth wireless interface. The Bluetooth wireless interface and the blue-through input device 4 such as a keyboard, mouse, game control and/or microphone and/or earphone 4 7 9 interface. Moreover, one embodiment of the client 465 can output video at 120 fps with the display device 468 coupled, and the display device 468 can support 12 〇 fps video and signal a pair of opaque glasses 466 (typically via infrared) for Each successive frame alternately shades one eye and then obscures the other eye. The effect felt by the user is to "jump out" the stereoscopic 3D image of the display screen. One of the operations is supported. The display device 468 is a Samsung HL-T5076S. Since the video stream for each eye is separate, in one embodiment where the two independent video streams are compressed by the host 136720.doc - 55 - 200941233, the frames are interleaved in time, and The frame is decompressed in client 465 in two separate decompression processes. The client 465 also includes low-latency video decompression logic 412, which decompresses incoming video and audio and outputs 'HDMI (High Definition Multimedia Interface), connector 463 via HDMI (High Definition Multimedia Interface), connector 463 It is inserted in SDTV (standard definition TV) or HDTV (high definition TV) 468 to provide video and audio to the TV, or to be inserted into the monitor 468 supporting HDMI. If the user's monitor 468 does not support O HDMI ’, HDMI to DVI (Digital Visual Interface) can be used, but the audio will be lost. The 'display capability (eg, supported resolution 'frame rate) 464 is conveyed from the display device 468 under the HDMI standard, and then this information is passed back to the colocation service 21 via the internet connection 462, thus The colocation service 2 1 0 can cause the compressed video to be streamed in a format suitable for the display device. Figure 4d shows a home or office client device 475, which is identical to the home © or office client device 465 shown in Figure 4c, except that the client device 475 has more external interfaces. Again, the client 475 can accept P〇E to power, or it can occupy an external power adapter (not shown) that is plugged into the wall. The video camera 477 provides the compressed video to the client 475 using the client 475 USB input, and the compressed video is uploaded by the client 475 to the colocation service 210 for use as described below. The low latency compressor will be built into camera 477 using the compression techniques described below. In addition to having an Ethernet connector for its Internet connection, the client 475 also has an 8 〇 211 g wireless interface to the Internet. Both interfaces 136720.doc -56· 200941233 are able to use NAT in a network that supports NAT. In addition to the HDMI connector for outputting video and audio, the client 475 also has a dual-link DVI-I connector, and the dual-link DVI-I connector includes an analog output (and has a standard that will provide VGA output). Adapter cable). It also has an analog output for composite video and S video. For audio, the client 475 has a left/right analog stereo RCA jack and has a TOSLINK output for digital audio output. In addition to the Bluetooth wireless interface to input device 479, it also has a USB jack for interfacing to the input device. Figure 4e shows an embodiment of the internal architecture of the client 465. All or some of the devices shown in this figure can be implemented in a field programmable logic array, in a custom ASIC, or in several discrete devices (custom design or off-the-shelf). Ethernet 497 with PoE is attached to the Ethernet interface 481. Power 499 is derived from Ethernet 497 with PoE and is connected to the remaining devices in client 465. Bus 480 is a common sink for communication between devices. A control CPU 483 that executes a small client control application from flash memory 476 (almost any small (:1>1; suitable, such as MIPS R4000 series CPu at 100 MHz with embedded RAM) is implemented for The network (i.e., the Ethernet interface) is stacked and also communicates with the colocation service 2, and configures all of the devices in the client 465. It also processes the interface with the input device 469 and encapsulates ( If necessary, 'along with the user controller data protected by the forward error correction, send back to the colocation service plus. 136720.doc •57- 200941233 And 'control the CPU 483 to monitor the packet traffic (for example, the packet is lost) It is also the delay, and the timestamp of its arrival.) This information is sent back to the colocation service 210 so that it can constantly monitor the network connection and adjust the content it sends accordingly (initially at the time of manufacture is flash memory) Body 476 loads the control program for controlling CPU 483 and the serial number unique to the particular client 465. This sequence number allows the host hosting service 21 to uniquely identify the client 465 unit. The bud interface 484 wirelessly communicates via its antenna (within the user terminal 465) to the input device 469. The video decompressor 486 is a low latency video decompressor configured to implement the video decompression described herein. The video decompression device exists either as an off-the-shelf product or as an intellectual property (IP) with a design that can be integrated into an FPGA or custom ASIC. A company that provides IP for the H 264 decoder is Ocean Logic of NSW Australia. Of Manly. The advantage of using IP is that the compression techniques used in this article do not match the compression standards. Some standard decompressors are flexible enough to be configured to accommodate the compression techniques in this article, but some standard decompressors may not However, in the case of IP, there is full flexibility in redesigning the decompressor as needed. The output of the video decompressor is coupled to the video output subsystem 487, which couples the video to the HDMI. The video output of interface 490. The audio decompression subsystem 488 is implemented using an available standard audio decompressor, or it can be Implemented as IP, or can be implemented, for example, 136720.doc -58· 200941233

The audio decompression is implemented in the control processor 483 of the Vorbis audio decompressor. The device that implements the audio decompression is coupled to an audio output subsystem 489 that couples the audio to the audio output of the HDMI interface 490. Figure 4f shows an embodiment of the internal architecture of the client 475. As can be seen, in addition to the additional interface and optional external DC power from the power adapter plugged into the wall (and if so, the optional external DC power replacement will be from the Ethernet PoE 497), The architecture is the same as that of the client 465. The functionality common to the client 465 will not be repeated below, but the additional functionality will be described below. The CPU 483 communicates with additional devices and configures additional devices.

The WiFi subsystem 482 provides wireless internet access via its antenna as an alternative to Ethernet 497. The WiFi subsystem is available from a number of manufacturers, including Atheros Communications of Santa Clara, CA. USB subsystem 485 provides an alternative to Bluetooth communication for wired USB input device 479. The USB subsystem is fairly standard and can be easily used in FPGAs 0 and ASICs, and is often built into existing devices that perform other functions such as video decompression. The video output subsystem 487 produces a wider range of video outputs than the video output in the client 465. In addition to providing HDMI 490 video output, it provides DVI-I 491, S-Video 492 and Composite Video 493. Also, when the DVI-I 491 interface is used for digital video, the display capability 464 is passed back from the display device to the control CPU 483 so that it can notify the host escrow service 210 of the capabilities of the display device 478. The 136720.doc -59- 200941233 interface provided by the video output subsystem 487 is a fairly standard interface and is readily available in many forms. The audio output subsystem 489 digitally outputs audio via the digital interface 494 (S/PDIF and/or Toslink) and outputs an analog form of audio via the stereo analog interface 495. Round trip delay analysis Of course, in order to realize the benefits of the previous paragraph, the user's action between the input device 421 and the effect of seeing the action on the display device 42〇 should be no more than 7 milliseconds _8 milliseconds. This delay must account for all factors in the path from the input device 421 to the colocation service 21A in the user premises 211 and back to the user premises 211 to the display device 422 again. Figure 4b illustrates the various components and networks through which signals must travel, and above these components and the network is a timetable that lists exemplary delays that can be expected in actual implementation. Note that Figure 4b is simplified to show only important path routing. Other routes for information on other features of the system are described below. A double-headed arrow (e.g., arrow 453) indicates a round-trip delay and a single, an arrow (e.g., arrow 457) indicates a one-way delay, and a "~" indicates an approximate S-test. It should be noted that there will be real-world situations in which the listed delays are not achievable. In a large number of situations, in the United States, DSL and cable modem connections to the user premises 211 are used, and such delays can be described in the next paragraph. Achieved. Also, 'note that despite the cellular wireless connectivity to the Internet! · Life will indeed work in the system that is not being deployed, most current US cellular data systems (such as EVD〇) incur very high latency and will The delay shown in Figure 4b cannot be achieved. However, these basic principles can be implemented on future cellular technologies that may be able to implement this level of delay. 136720.doc • 60 · 200941233 Beginning with the input device 421 at the user premises 211, once the user causes the device 42ι, the user control signal is sent to the user terminal (which may be a standalone device such as a set top box, or It can be a software or hardware implemented in another device such as a pc or mobile device, and it is packetized (in a UDP format) and the destination address is given to the packet to reach the colocation service. 210. The packet will also contain information indicating which user the control signal is from. The control signal packet is then forwarded to the interface ©® 442 via the Firewall/Router/NAT (Network Address Translation) device 443. ~Yan Jie® 442 is an interface device provided to the user's premises 211 by the user's ISP (Internet Service Provider). The WAN interface 442 can be a cable or DSL modem, a WiMax transceiver, a fiber optic transceiver, a cellular data interface, a power line internet protocol interface (Internet pn>t()e()i_ 〇ver-p0werline interface) or Any other interface in many interfaces of the Internet. Additionally, firewall/router/ΝΑτ device 443 (and (possibly) WAN interface 442) may be integrated into client 415. An example of this would be a mobile phone that includes software for implementing the functionality of a home or office client 415, and for wirelessly routing and connecting to the Internet via a standard (eg, 802.11g) Components. The WAN interface 4 4 2 then routes the control signals to the "presence point" 44 ι for the user's Internet Service Provider (ISP), which is the WAN that provides connectivity to the user premises 211. A facility between the conveyor and the interface between the universal Internet or the private network. The nature of the point of existence will vary depending on the nature of the Internet service provided. For dsl, it will typically be the central office of the telephone company that the DSLAM is targeting. For the 136720.doc -61- 200941233 electric slow data machine, it will usually be the head end of the electric multi-system operator (Ms〇). For a honeycomb system, it will typically be the control room associated with the honeycomb tower. But regardless of the nature of the point, it will then route the control signal packet to the universal internet 410. The control signal packets are then routed to the WAN interface 444 to the host escrow service 210 via the interface that is most likely to be the fiber optic transceiver interface. WAN 444 will then route the control signal packets to routing logic 409 (which can be implemented in many different ways, including Ethernet switches and routing servers), routing logic 4〇9 to estimate the user's address and will control The signal is routed to the correct server 4〇2 for a given user. The server 402 then treats the control signals as inputs to the game or application software executing on the server 4〇2 and uses the control signals to process the next frame of the game or application. Once the next frame is generated, the video and audio are output from the server 402 to the video compressor 404. Video and audio can be output from the server 402 to the compressor 404 via various components. First, the compressor 404 can be built into the server 402 so that compression can be performed at the local end within the server 402. Alternatively, it may be connected via a network (either a private network between the server 402 and the video compressor 404, or a network via a shared network such as the SAN 403) (such as, The Ethernet connection) outputs video and/or audio in a packetized form. Alternatively, the video may be output from the server 402 via a video output connector, such as a DVI or VGA connector, and then captured by the video compressor 404. Alternatively, the audio can be output from the server 402 as digital audio (e.g., via a TOSLINK or S/PDIF connector) or analog audio, and the analog audio is digitized and encoded by the audio compression logic within the video compressor 404. 136720.doc -62- 200941233 Once the video compressor 404 has captured the video frame from the server 402 and the audio generated during the frame time, the video compressor will compress the video and audio using the techniques described below. Once video and audio are pressed

By shrinking, it is packetized by a single address to send it back to the user's client 415 and routed to the WAN interface 444, which then routes the video and audio packets over the universal Internet 410. The universal Internet 410 then routes the video and audio packets to the Isp presence point 441 of the user. The presence point 441 routes the video and audio packets to the WAN interface 442 at the user premises, and the WAN interface 442 routes the video and audio packets. To firewall/router/NAT device 443, firewall/router/NAT device 443 then routes the video and audio packets to client 415. The client 415 demultiplexes the video and audio, and then displays the video on the display device 422 (or the built-in display device of the user terminal) and sends the audio to the display device 422 or to a separate amplifier/speaker or to the user terminal. In the amplifier / speaker. After you don't stagnate, the back and forth delay needs less than 7〇 milliseconds or 8〇 milliseconds. Some of the delay delays described in the return path are controlled by the colocation service 21 and/or the user, while other delay delays are not subject to the colocation service η. And / or user control. Nonetheless, based on the analysis and testing of a large number of real-world situations, the following are approximate measurements. The one-way transmission time 451 of the knife control signal is usually small through the user's place, ΛΑ J· ^ ^ 逋 often uses the easy-to-use firewall/router/NAT switch of the Ethernet network. 】 136720.doc •63- 200941233 completed in seconds. The user ISP extensively changes its round trip delay 453, but in the case of DSL and cable modem providers, it is typically seen between 1 and 25 milliseconds. The back-and-forth delay of the Universal Internet 41 is how the routing is routed and whether there are any faults on the route (and these issues are discussed below), but the general Internet is usually quite optimal. Routing and delay are largely determined by the speed at which light passes through the fiber (distance given to the destination). As discussed further below, 1000 miles have been determined as a desire to host the service 21 away from the user© The approximate farthest distance at which the location 211 is placed. At 1000 miles (2,000 miles back and forth), the actual transmission time for signals via the Internet is approximately 22 milliseconds. The WAN interface 444 to the colocation service 21 is typically a commercial grade fiber optic high speed interface with negligible latency. Therefore, the universal internet latency 454 is typically between 1 millisecond and 10 milliseconds. The one-way routing 455 delay via the colocation service 210 can be achieved in less than one millisecond. Server 4〇2 will typically calculate a new frame for the game or application at less than one frame time (which is 167 milliseconds at 60 fps), so 16 milliseconds is the reasonable maximum one to be used. Delay 456. In the best hardware implementation of the video compression and audio compression algorithms described herein, the dust shrinkage can be completed in less than 1 millisecond. In the next best version, compression can take up to 6 milliseconds (of course, a less desirable version can take longer, but these implementations will affect the total latency of the round trip and will require other delays to be shorter (eg, subtractable) Small via the allowable distance of the universal Internet) to maintain a 7 〇 _ 8 〇 millisecond delay target). The round trip delay of Internet 454, user ISP 453 and user location route 452 has been considered, so the rest is video decompression 458 extension 136720.doc • 64-200941233 'Video decompression 458 delay depends on video decompression 458 is implemented in a dedicated hardware or in a software implemented on a client device 415 (such as a Pc or mobile device), the size of the visual display and the performance of the decompressing CPU. Usually 'decompressing 458 takes a time between 1 millisecond and 8 milliseconds. Thus, the worst-case back-and-forth delay that the user of the system shown in Figure 4a can expect to experience can be determined by adding together all the worst-case delays seen in practice. It is: 1 + 1+25+22+1 + 16+6+8=8() milliseconds. In addition, in practice, in practice (with the explanation of Prevention® misunderstandings discussed below), this is roughly the prototype version of the system shown in Figure 4a (using an off-the-shelf Windows PC as a client device and home DSL in the United States) And the cable data machine connection) see the back and forth travel delay. Of course, situations that are better than the worst can lead to much shorter delays, but you can't rely on them to develop widely used business services. In order to achieve the delays listed in Figure 4b via the universal internet, the video compressor 404 and the video decompressor 412 (from Figure 4a) in the client 415 are required to generate a packet stream having very specific characteristics such that The sequence of packets generated from the entire path of the colocation service 210 to the display device 422 is not subject to delay or excessive packet loss' and, in detail, is consistently connected via the user's Internet connection (via the WAN interface 442 and Firewall / Router / NAT 43 3) can be used within the constraints of the user's bandwidth. In addition, the video compressor must generate a sufficiently robust packet stream so that it can withstand the inevitable packet loss and packet reordering that occurs in normal Internet and network transmissions. Low-latency videoconferencing 136720.doc •65- 200941233 To accomplish the above objectives, an embodiment employs a new video compression method that reduces the latency and peak bandwidth requirements for video transmission. Prior to describing such embodiments, an analysis of current video compression techniques will be provided with respect to Figures 5 and 6a through 6b. Of course, if the user has sufficient bandwidth to handle the data rate required by such techniques, then such techniques can be used in accordance with the underlying principles. Note that this does not address audio compression, but rather states that audio compression is implemented concurrently and synchronously with video compression. Prior art audio compression techniques that meet the requirements for this system exist. Figure 5 illustrates a particular prior art for compressing video in which each individual video frame 501-503 is compressed by compression logic 520 using a particular compression algorithm to produce a series of compressed frames 511-513. One embodiment of this technique is "Motion JPEG" where each frame is compressed based on a discrete cosine transform (DCT) according to the Joint Image Experts Group (jpEG) compression algorithm. Various different types of compression algorithms can be used, however, these basic principles are still adhered to (eg 'wavelet-based compression algorithms such as JpEG_2000). One of the problems with this type of compression is that it reduces each figure The data rate of the box 'but it does not take advantage of the similarity between successive frames to reduce the data rate of the total video stream. For example, as illustrated in Figure 5, assume a frame of 64〇x48〇x24 bits/pixel=64〇*480*24/8/1024=900 kilobytes/frame (KB/frame) Rate, then for a given quality image, Motion JPEG may only compress the stream by 1/10, resulting in a 9 〇 KB/frame data stream. In 6 frames/sec, 'this will require 90 KB*8 bits*60 frames/sec = 42.2 Mbps channel bandwidth, which is available for almost all home Internet connections in the United States today 136720.doc •66· 200941233 will eventually be extremely high bandwidth and too high bandwidth for many office internet connections. In fact, assuming that it requires a constant data stream in such a high bandwidth, and it will only serve one user, it will consume 1 Mbps 6 too wide LAN LAN bandwidth even in an office LAN environment. A large percentage and a heavy Ethernet switch that supports the LAN. Therefore, compression for motion video is inefficient when compared to other compression techniques, such as those described below. In addition, a single frame compression algorithm (such as JpEG& JpEG_2〇〇〇) using a lossy compression algorithm produces © 压缩 a compression artifact that may be unobtrusive in still images (for example, in dense foliage in a scene) False shadows may not appear as artifacts because the eye does not know exactly how dense leaves should be rendered). But 'when the scene is in motion, the artifacts may stand out because the eye is the artifact that changes from the frame to the frame, even though the artifact is in the area of the scene (in this area, the artifact is still The image may not be noticeable). This results in the perception of "background noise" in the sequence of frames, and the appearance of the "background noise" of the stomach is similar to the "snow" noise seen during TV reception. Of course, this type of compression can still be used in the specific embodiments described herein, but in general, in order to avoid background noise in the scene, a high data rate (ie, a low compression ratio) is required for a given perceived quality. ). Other types of compression (such as H.264, or Windows Media VC9, MPEG2, and MPEG4) are more efficient in compressing video streams because they exploit the similarities between successive frames. These techniques all rely on the same general techniques for compressing video. Therefore, although the Η.]" standard will be described, the same general principles apply to various other compression algorithms. A large number of H 264 pressures 136720.doc •67· 200941233 Reducers and decompressors are available, including the χ264 open source software library for compression 264.264 and the FFmpeg open source software library for decompressing H.264. 6a and 6b illustrate an exemplary prior art compression technique in which a series of uncompressed video frames 501-503, 559-561 are compressed by a compression logic 620 into a series of "I frames" 611, 671; Frames 612-613; and "B-frames" 670. The vertical axis in Figure 6a generally represents the resulting size of each of the encoded frames (although the frames are not scaled). As described above, those skilled in the art have a good understanding of the video writing code using the I frame, the B frame, and the P frame. Briefly, block I 611 is a DCT-based compression of frame 501 that is completely uncompressed (similar to a compressed JPEG image as described above). The size of P-frames 612-613 is typically significantly smaller than the size of I-frame 611 'because it utilizes data from previous I-frames or p-frames; that is, 'which contains an indication between the previous I-frame or P-frame Changed information. The B-frame 670 is similar to the p-frame except that the B-frame uses the frame of the subsequent reference frame and (possibly) the frame of the previous reference frame. For the following discussion, it will be assumed that the desired frame rate is 6 frames/sec, ® each I frame is about 160 Kb, the average P frame and B frame are 16 Kb and a new one every second. I frame. Under this set of parameters, the average data rate will be: 160 Kb + 16 Kb * 59 = 1.1 Mbps. This data rate falls appropriately within the maximum data rate for many current broadband Internet connections to homes and offices. This technique also tends to avoid background noise problems from intra-frame coding only. Because the difference between the P-frame and the B-frame tracking frame, the compression artifact tends to appear and disappear from the frame to the frame. To reduce the background noise problem described above. 136720.doc -68 - 200941233 One of the problems with compression of the above type is that although the average data rate is relatively low (for example, 1.1 Mbps), a single I frame may take several frame times to transmit. For example, using prior art, a 2.2 Mbps network connection (eg, a DSL or cable modem with a 2.2 Mbps peak from the maximum available data rate 302 of Figure 3a) would typically be sufficient to stream video at 1.1 Mbps 'per 60 frames a 160 Kbps I frame. This will be done by having the decompressor put a 1 second video into the queue before decompressing the video, and will transmit 1.1 Mb of data in 1 second, which will be easily tuned by the 2.2 Mbps maximum available data rate. Adaptation, even if it is assumed that the available data rate may periodically drop by as much as 50%. Unfortunately, this prior art method would result in a 1 second delay in video due to the 1 second video buffer at the receiver. This delay is sufficient for many prior art applications (e.g., playback of linear video), but it is an extremely long delay for fast motion video games that cannot tolerate delays greater than 70 milliseconds _8 milliseconds. If an attempt is made to eliminate the 1 second video buffer, it will still not result in a sufficient delay reduction for fast motion video games. For example, the use of the 'B frame' as previously described would require receiving all of the b-frames before the I-frame. If it is assumed that approximately 59 non-I frames are split between the P frame and the B frame, there will be at least 29 B frames and the I frame received before any B frames can be displayed. Therefore, regardless of the available bandwidth of the channel, each of the 29 +1-30 frames is required to have a delay of 60 seconds duration or a delay of 5 milliseconds. Obviously, his time is extremely long. Therefore, another method would be to eliminate the B frame and only use the I frame and the p frame. (One of the consequences is that for a given quality level, the data rate will increase by 136720.doc -69- 200941233 plus, but for the consistency of this example ten, 'continue to assume that the size of each j frame is 160 Kb and the average P frame size is 16 Kb, and therefore the data rate is still 1 · 1 Mbps). This method eliminates the unavoidable delay introduced by the B-frame because the decoding of each P-frame depends only on the previously received frame. The problem with this approach is that the frame is much larger than the average p-frame, so that on low-frequency wide channels (as is typical in most homes and many offices), the transmission of I-frames adds substantial latency. This is illustrated in the figure. The video stream data rate 624 is lower than the available maximum data rate 621 (except for the 1 frame), where the peak data rate 623 required for the I frame is well above the maximum available data rate 622 (and even exceeds the nominal maximum data rate 621). The data rate required for the frame is less than the maximum data rate available. Even if the maximum available data rate peak at 2.2 Mbps is steadily maintained at its 2 2 Mbps peak rate', it will cost 16 〇 Kb / 22 Mb = 71 milliseconds to transmit the I frame' and if the available maximum data rate 622 drops by 50% ( Ll Mbps), the lb will be 142 milliseconds to transmit the I frame. Therefore, the delay in the transmission I frame will fall somewhere between 71 milliseconds and 142 milliseconds. This delay is added to the delay identified in the servant (the delays are up to 7 〇 milliseconds in the worst case), so this will result from the moment the user actuates the input device 421 until the image is presented on the display device 422. The total round trip delay of 141-222 milliseconds is extremely high. And if the maximum available data rate drops below 2.2 Mbps, the delay will increase further. It is also noted that 'the peak data rate 623 far exceeds the available data rate 622. The 1SP "blocking J usually has serious consequences. Devices in different ISPs will behave differently, but at a data rate 136720 that is much higher than the available data rate 622. Doc •70- 200941233 And, the following behaviors are quite s in the DSL and cable modem ISP. (a) Delay the packet by introducing the packet into the queue (introduction delay), W discard some or All packets, (4) deactivate the connection duration - time period (most likely because the ISP is concerned about malicious attacks, such as "denial of service" attacks). Therefore, transmitting packet streams at full data rate (having characteristics such as those not shown in Figure 11) is not a viable option. The peak 623 can be queued (4) at the host-hosted service 2H) and sent at a data rate lower than the maximum data rate available, from the unacceptable delay described in the previous paragraph. In addition, the video stream rate sequence shown in Figure 6b is a very "tame" video stream data rate sequence, and will be the data rate sequence expected to be born due to compression of video from the video sequence. The video sequence does not change very much and has very little motion (for example, as in a video conference call will be common in video calls)

In a fixed position and with very little motion, and objects in the scene (eg, a person's conversation) show less movement). The video stream data rate sequence 634 shown in Figure 6c is a typical sequence that is expected to be visible from video with much more motion, such as may be produced in a movie or video game or in an application software. Note that in addition to the peak 633 of the worksite, there are also P-frame peaks (such as 635 and 636) that are quite large and in many cases exceed the maximum available data rate. Although the peaks of these P-frames are not quite as large as the I-frame peaks, they are still too large to be carried by the channel at the full data rate, and as the pivot peaks of Figure 1 'P-frame peaks must be transmitted slowly ( This increases the delay). I36720.doc -71- 200941233 〇❹ On a high-frequency wide channel (such as '100Mbps LAN, or high-bandwidth 连接 connection), the network will be able to tolerate large peaks such as I-frame peak 633 or ρ frame peak 636 However, in principle, low latency can be maintained. But $, these networks are often shared among many users (for example, in an office environment), and the "peak" data will affect the performance of the LAN, especially when the network traffic is routed to a private shared connection. (For example, from the remote financial center to the office) n, remember that this example is an example of a relatively low resolution video stream of 640x480 pixels at 60 fpS. The HDTV stream at 6〇 fps is easily handled by modern computers and monitors, and 2560x1440 resolution displays at 60 fps are increasingly available (for example, a卯&30" With H 264 compression, a 1920x1080 high motion video sequence with 6 〇 may require 4 5 Mbps to achieve a reasonable quality level. If the I-frame peak is assumed to be 1〇 of the nominal data rate, it will produce a 45 Mbps peak and a small but still quite large p-frame peak. If several users are receiving video streams on the same 1 Mbps network (for example, a private network connection between the office and the data center), it is easy to see how the peak of the video stream from several users can happen to The bandwidth of the network is flooded and may overwhelm the bandwidth of the backplane of the switch supporting the user on the network. Even in the case of ultra-high-speed Ethernet, if enough users have enough peaks for simultaneous alignment, they can flood the network or network parent converter. In addition, once 256 〇χ 144 〇 resolution video becomes more common, the average video stream data rate may be 9.5 Mbps, which may result in a 95 Mbps peak data rate. Needless to say, the 100 Mbps connection between the data center and the office (which is now an extraordinarily fast connection) will be completely defeated by a single user lacking peak traffic from I36720.doc -72- 200941233. Therefore, even if LAN and private network connections are more tolerant for peak-stream video, streaming video with high peaks is not required and may require special planning and adaptation by the office department. Of course, for standard and quasi-linear video applications, these issues are not a problem because the data rate is "smoothed" at the transmission point and the data for each frame is below the maximum available data rate 622 and is decompressed. Before the frame, p frame and B frame sequence, the buffer in the client side stores the I frame, the P © frame and the B frame sequence. Therefore, the data rate on the network remains close to the average data rate of the video stream. Unfortunately, this introduction delay, even if the B-frame is not used, is unacceptable for low-latency applications such as video games and applications that require fast response times. A prior art solution for mitigating video streams with high peaks is to use techniques often referred to as "blame locating element rate" (CBR) encoding. Although the term CBR appears to imply that all frames are compressed to have the same bit rate (ie, size)' but it is often referred to as a compression paradigm, in the compression paradigm, a certain number of frames are allowed to be crossed (in my case) In the case of a frame, the maximum bit rate. For example, in the context of the figure, if a CBR constraint is imposed on the code that limits the bit rate to, for example, 7〇% of the nominal maximum data rate 621, the compression algorithm will limit the frames. Each frame is shrunk so that any frame that would normally be reduced by more than 70% of the nominal maximum data rate 621 will be compressed with fewer bits. The result is that it will usually require more bits. Yuan will maintain the "extremely lacking" position of the frame of the given quality level and the image quality of these frames will be more than the required ratio of 136720.doc -73. 200941233 rated maximum data rate 621 of 7 ()% The quality of the other frames of the yuan is bad. This method can produce a result for a particular type of compressed video (where (4) less motion or scene change and (8) the user can accept periodic quality degradation). A good example of a suitable CBR application is that the π phone will be 6 meanings because there are fewer peaks and temporarily degraded in quality: (for example, if the camera is glanced, resulting in significant scene motion and large peaks, during the glance There may not be enough bits for high quality image compression, which will result in degraded image quality), which most users can accept. Unfortunately, CBR is not well suited for many other applications with high complexity scenes or large amounts of motion and/or requiring a reasonably constant quality level. The low latency compression logic 4〇4 used in one embodiment uses several different techniques to solve the problem of streaming low latency compressed video while maintaining high quality. First, the low-latency compression logic 4〇4 only generates the j-frame and the p-frame, thereby alleviating the need to wait for several frame times to decode each B-frame. Additionally, as illustrated in Figures 7 & In one embodiment, low latency compression logic 404 divides each uncompressed frame 7〇1_76 into a series of "tiles" and each image The blocks are individually encoded as a frame or a p-frame. In this paper, the compressed I frame and p frame of the group are referred to as "r frame" 711-770. In the particular example shown in Figure 7a, each uncompressed frame is subdivided into 16 image blocks of a 4M matrix. However, these basic principles are not limited to any particular sub-dividing mechanism. In one embodiment, the 'low latency compression logic 4〇4 divides the video frame into a plurality of scene blocks and encodes one of the image blocks from each frame (ie, 136720.doc • 74· 200941233 compression). Is an i frame (ie, compressing the image block as if it were a 1/16 of a full video frame size, and the compression for this "mini" frame is I frame compression) and Encode the remaining image blocks as? The frame (that is, the compression for each "mini" 1/16 frame is p-frame compression). The image blocks compressed into I frames and the image blocks compressed into p frames will be referred to as "I image blocks" and "P image blocks", respectively. The image block to be encoded as a block of image blocks is changed with each successive video frame. Therefore, in a given frame time, only one of the image blocks in the video frame is a work image block, and the remaining ones of the image blocks are p image blocks. For example, in Figure h, the image block of uncompressed frame 701 is encoded as image block 1 and the remaining 1-15 image blocks are encoded as p image blocks (Ρι to p) 5 An R frame 711 is generated. In the next uncompressed video frame 7〇2, the image block of the uncompressed frame 71 is encoded as an image block and the remaining image blocks 2 and 2 to 15 are encoded as p image blocks. (p〇, 匕 to 卜5) to generate r frame 712. Therefore, the image block and the P image block used for the image block are gradually interleaved in time on successive frames. The process continues until an R image block 770 is generated (the last image block in the matrix is encoded as an I image block (i.e., the process is then restarted, resulting in, for example, frame 711 (i.e., for the image block, Another R frame, such as an I picture block, etc. Although not illustrated, in the embodiment, the first R frame of the video sequence of the R frame contains only the image block (ie, so that The p-frame has reference image data (from which the juice motion is started. Or, in one embodiment, the start-up sequence uses the same I-image block pattern as normal, but does not include those that have not been encoded with the image block yet. The p-image block of the image block. In other words, the specific image block is not encoded together with any data before the arrival of the first frame of image 136720.doc -75- 200941233, thereby avoiding the start peak in the video stream data rate 934 in Figure 9a. Further, as described in more detail below, various sizes and shapes can be used for the image blocks while still complying with these basic principles. The video decompression logic executed on the client terminal 41 5 412 decompresses each image block as if it were a separate video sequence of a small I frame and a P frame, and then reproduces each image block to a frame buffer that drives display device 422. For example, using from R Frames 711 to 770, 10 and P〇, decompress and reproduce the image block of the video image. Similarly, the image block 1 'class is reconstructed using Ιι and Pi from R frames 7丨丨 to 77〇. As described above, the decompression of the I frame and the p frame is well known in the art, and the decompression of the I picture block and the p picture block can be performed by causing multiple execution entities of the video decompressor to be in the user. The implementation is performed on terminal 415. Although the multiplication process appears to increase the computational burden on the client usus, it does not actually increase the computational burden on the client 415 because the image block itself is proportionally smaller (as opposed to The number of additional processing and §) 'There are the same number of pixels displayed, as if there is one processing and using the full-size I-frame and P-frame of the conventional size. This R-frame technique significantly reduces the usual I-picture The peak width associated with the frame (described in Figures 6b and 6c) Ming), because any given frame is mainly composed of P frames that are usually smaller than the I frame. For example, suppose that the typical frame is 160 Kb, then each frame in the frame illustrated in Figure 7a The j image block of one will be approximately 1/16 or 10 Kb for this amount. Similarly, assuming a typical p frame of 16 Kb, then for each of the image blocks illustrated in Figure 7a. The frame can be approximately 1 Kb. The final result is approximately 10 Kb + 15J | 51 Kb = 25 Kb "R Figure 136720.doc .76 · 200941233 box. Therefore, each 60 frame sequence will be 25 Kb * 60 = 1.5 Mbps. Therefore, at 60 frames per second, this would require a channel capable of maintaining a bandwidth of 1.5 Mbps, but with a much lower peak due to the dispersion of the I-picture block throughout the 6-frame interval. Note 'In the previous example, it was used! The same assumptions for the frame and p-frames, the average data rate is 丨]Mbps. This is because in the previous example, only one new 1 frame was introduced every 60 frame time, and in this example, the 16 image blocks of the frame were cycled in 16 frame times, and thus 'Every 16 frames time bow 1 into one! The equalization of the frame results in a slightly higher average data rate. However, in practice, the introduction of more frequent I® boxes does not linearly increase the f-rate rate. This is due to the fact that the p-frame (or P-image block) is mainly encoded from the previous frame to the next. The difference in the box. Therefore, the previous frame is quite similar to the next frame, and the P frame will be very small. If the previous frame is quite different from the next frame, the p frame will be very large. However, because the p-frame is largely derived from the previous frame, rather than being derived from the actual frame, the resulting coded frame can contain more errors than I frames with a sufficient number of bits ( For example, visual artifacts). In addition, when a p-frame follows another P-frame, an error accumulation can occur (when there is a long p-frame sequence, it gets worse). Now, the cutting-edge video compressor will detect the shadow drop (four) declaration after a sequence of P frames = facts and if necessary, it will allocate more bits to the subsequent P frame to mention two qualities, Or if it is the most active process of 'p.', replace the P frame with an I frame. Thus, when a long box is used, as in the previous example above, a particular sequence (eg, a 59-sided feature S scene has a large number of complex 136720.doc -77 - 200941233 noise and/or motion, usually, for In the case of a p-frame, more bits are needed (because it becomes farther away from the I-frame). Or, looking at the P-frame from the relative view point, closely following the p-frame of the frame tends to require distance The frame is farther away from the bit of the p frame. Therefore, in the example shown in Figure 7a, the p-free frame is farther away from the frame by 15 frames (before the I frame)' In the previous example, the ρ frame can be separated from the frame by 59 frames. Therefore, in the case of more frequent frames, the ρ frame is smaller. Of course, the exact relative size will be based on the video stream. The nature changes but © In the example of Fig., if the I image block is 10 Kb, the size of the Bay JP image block can be only 0.75 kb on average, resulting in 10 Kb+15*0.75 Kb=21.25 Kb, or 60 graphs. The data rate in frame/sec will be 21 a KbMOMj Mbps, or a data rate with a frame frame followed by 59 p frames at 1·1 Mbps. The rate is about 16% higher. Again, the relative results between the two methods for video compression will vary depending on the video sequence, but in general, we have empirically found that for a given quality level, the R frame is used. It takes about 20% more bits than using the I/p frame sequence. However, 'of course, the R frame sharply reduces the peak value, which makes the video sequence available at a delay much smaller than the j/ρ frame sequence. The R frame is configured in a number of different ways depending on the nature of the video sequence, the reliability of the channel, and the available data rate. In an alternate embodiment, a number of image blocks other than 16 are used in the 4><4 configuration. For example, two image blocks can be used in a 2χΐ or 1x2 configuration, and 4 image blocks can be used in 2x2, 4><1 or 1><4 configuration commands, available in 3x2, 2x3, 6xl Or use 6 image blocks in the 1χ6 configuration or use 8 image blocks in 4x2 (shown in Figure 7b), 2x4, 8x1 or lx8 configurations. Note that the image block does not need to be square, video map 136720. Doc -78- 200941233 The box does not have to be square or even rectangular. The image block can be decomposed into the best fit. Whatever the shape of the video stream and application used.

In another embodiment, the loop of the I image block and the P image block is not locked to the number of image blocks. For example, in an 8 image block 4χ2 configuration, a 16-cycle sequence can still be used as illustrated in Figure 7b. The sequential uncompressed frames 721, 722, 723 are each divided into 8 image blocks 〇_7, and each image block is individually compressed. From R frame 731, only the image block is compressed into a frame of image, and the remaining block is compressed into a block of P blocks. For subsequent frame 732, all 8 image blocks are compressed into p-image blocks, and then for subsequent R frame 733, image block 1 is compressed into I image blocks and other image blocks are compressed into P-image blocks. . In addition, the 16 frames continue to be sorted 'only every time - frame is generated -; [image block, therefore during the 15th frame time (not shown in Figure 7b) and at the 16th frame time During the period, the last I image block for the image block 7 is generated (use all the p image blocks to compress the r frame 78〇). Then the sequence is again shredded by the image block! The image block and other image blocks are compressed into P image blocks. As in the previous embodiment, the first frame of the entire video sequence will usually be! Image block to provide a reference for P blocks that are forward from the other point. ! The loop of the image block and (4) the block does not even need to be an even multiple of the number of image blocks. For example, in the case of 8 image blocks, before each other image block is used, each frame having the image block can be followed by 2 frames of all the image blocks. In yet another embodiment, 'if, for example, a particular area of the screen is known to have more motion (requiring more frequent I-picture blocks), while other areas are more static (eg, showing game scores) (requires more tedious !block), compared with other image blocks 136720.doc -79· 200941233, the specific image blocks can be sorted together with the image blocks more often. Further, although each frame is illustrated as having a single block in Fig. 7 of Fig. 7, a plurality of blocks of one block can be encoded in a single frame (depending on the bandwidth of the transmission channel). Conversely, a particular frame or frame sequence can be transmitted without a work image block (i.e., only a p-picture block). The reason why the method of the previous paragraph works properly is that although the j-image block that does not have a spread across each single frame appears to cause a large peak, the behavior of the system is not so simple. Because each image block is shrunk separately from other image © blocks, the coding of each image block can become less effective when the image block becomes smaller, because the compressor of a given image block cannot be utilized from other Similar image features and similar motion of image blocks. Therefore, dividing the screen into 16 image blocks will generally result in less efficient coding than dividing the screen into 8 image blocks. However, if the screen is divided into 8 image blocks and it causes the introduction of a complete I frame every 8 frames (rather than every 16 frames), it results in a much higher overall data. rate. Therefore, by introducing a full 1 frame every 16 frames instead of every 8 frames, the total data rate is reduced. Moreover, by using eight larger image blocks (rather than 16 smaller image blocks), the total data rate is reduced, which also reduces the peak of the data caused by the larger image blocks to some extent. In another embodiment, the low-latency video compression logic 4〇4 in FIGS. 7a and 7b is pre-configured or based on image quality in each image block by virtue of known characteristics of the video sequence to be compressed. The ongoing analysis automatically controls the allocation of bits to each of the image blocks in the R frame. For example, in some competitive video games, the player's car (which is the scene 136720.doc 200941233 ❹

= =:) The former party occupied most of the lower half of the screen, and /1 was filled with the roads, buildings and landscapes that were approaching, and X*疋 was in motion. If the grinding logic 4〇4 will give an equal number of bits to each block, then the image block (image block 4_7) on the lower half of the picture frame a! It is compressed by the buckle between the image block (image block 曰3) in the upper half of the (four) 72 (middle) screen of the unrestricted (four) 72l. Township knows that this particular scene of this particular game or game has material characteristics 'Mt丨 Host Generation f Service 21()'s #商可组缘房缩逻辑404 to assign more bits to the image block at the top of the screen (Compared to the bit of the image block at the bottom of the screen assigned to the screen). Alternatively, compression logic 404 may estimate the compression quality of the image block after compressing the frame (using one or more of a number of compression quality metrics, such as peak signal to noise ratio (PSNR)), and if determined to be on a particular time window, When a particular image block consistently produces better quality results, it gradually assigns more bits to the image blocks that produce lower quality results until the various image blocks reach a similar level of quality. In an alternate embodiment, the compressor logic 4〇4 assigns bits to a higher quality in a particular image block or image block group. For example, it can provide a better overall perceived appearance' to have a higher quality at the center of the screen than at the edges. In one embodiment, in order to improve the resolution of a particular region of the video stream, video compression logic 404 uses a smaller image block to encode a region of the video stream that has relatively more scene complexity and/or motion (with video stream). Relatively small compared to the complexity of the scene and/or the area of motion). For example, as illustrated in Figure 8, in an area of R frame 811 (possibly followed by a series of R frames (not shown) having the same image block size 136720.doc -81 - 200941233 J) A smaller image block is used around the moving character 805. Then, when the character 8〇5 moves to a new area of the image, a smaller image block is used around the new area in the other-R frame 812. . As mentioned above, various sizes and shapes can be used as "image blocks" while still complying with these basic principles. Although the cyclic I/P image block described above substantially reduces the peak in the data rate of the video stream, it does not completely eliminate the peak, especially in fast changing or highly complex video images (such as in movies, video). In the case of games and under an application software). For example, during a sudden scene transition, a complex frame may be followed by another complex frame that is completely different. Even if several I image blocks can lead the scene transition by only a few frame times, it does not help in this case because the material of the new frame is independent of the previous I image block. In such a situation (and in other situations where a large number of images change, even if not everything changes), video compressor 404 will determine that many, if not all, p-image blocks are more efficiently coded as image blocks. And the resulting result is a very large peak in the data rate of the frame. As previously discussed, it is only for most consumer-grade Internet connections (and many office connections), which can only be "blocked" beyond the maximum data rate available and the maximum data rate shown in Figure 6c as 622. 621 information. Note that the nominal maximum data rate of 621 (e.g., "6 Mbps DSL") is essentially a sales figure for users considering Internet access, but generally does not guarantee performance levels. For the purposes of this application, it is not the same as the maximum data rate 622 available when streaming video over a connection. Therefore, in Figure 9a and %, when the solution to the peak problem is described in 136720.doc • 82· 200941233, the nominal maximum data rate is omitted from the graph and only the maximum available data rate 922 is shown. The video streaming data rate must not exceed the maximum available data rate of 922. To solve this problem, the first thing the video compressor 404 does is to determine the peak data rate 941, which is the data rate at which the channel can be processed steadily. This rate can be determined by a number of techniques. One technique is to gradually increase the data rate test stream from the colocation service 21 to the user terminal 415 (Figs. 4a and 4b), and to cause the client to lose information about the packet loss and delay. The quasi-feedback is provided to the colocation service. When the packet is lost and/or the delay begins to show a sharp increase, it is indicative of the maximum data rate 922 available. Thereafter, the colocation service 21 can gradually reduce the data rate of the test stream until the client 415 reports that the test stream has been received within a reasonable period of time (the packet loss and the acceptable level of the delay are near minimum). This determines the peak maximum data rate 941, which will then be used as the peak data rate for streaming video. Over time, the peak data rate 941 will fluctuate (eg, the other in the theater - the user begins to seriously use the internet connection), and the client 415 will need to monitor the peak rate 941 to see the packet loss. Or whether the delay is increased (indicating that the available maximum data rate 922T falls below the previously determined value (4), and if so 'the peak data rate 941. Similarly, if the user 415 finds the loss and delay Keeping the money standard can ask the video compressor to slowly increase the data rate to increase the data rate (for example, if the family chooses another serious use of the Internet connection), and wait again until the 136720.doc • 83- 200941233 The missing/or longer delay indication has exceeded the maximum data rate 922 available, and the lower level for the peak data rate 941 can be found again, but the lower level may be higher than the increased test data. The level before the rate. Therefore, the peak data rate 941 can be found by using this technique (and other techniques like it) and periodically as needed. The line data rate 941 determines the maximum data rate that can be used by the video compressor 4〇4 to stream the video to the user. The logic for determining the peak data rate can be at the user location 211 and/or at the host generation. The service is implemented on the server 21. At the location 211, the client device 415 performs a calculation to determine the peak data rate and transmits this information back to the colocation service 21; at the colocation service 210, the host The server 4〇2 of the hosting service performs calculations to determine the peak data rate based on statistics received from the client 415 (e.g., peak loss, delay, maximum data rate, etc.) Figure 9a shows substantial scene complexity And/or an instance of the video stream data rate 934, which is generated using the cyclic I/P block compression technique described previously and illustrated in Figures & Figures, and 8. Video Navigator U 404 The video is configured to be below the peak data rate 941, and note that most of the time, the video stream is held below the peak data rate 941. Data rate 934 and Figure 6c A comparison of the video stream data rate 634 shown in the figure (which is generated using I/p/B or I/p frames) shows that the cyclic/P image block compression produces a much smoother data rate. However, in frame 2 The peak data 952 (which is close to 2 times the peak data rate 942) and the frame * times the peak 954 (which is close to 4 times the peak data rate of 944) are still exceeding the peak data rate 941, which is unacceptable. In practice Even for 136720.doc -84 - 200941233 high-motion video from fast-changing video games, the peak value above the ridge data rate 941 appears in the frame of less than 2%, and the peak value of the peak data rate 942 is less than 2 times. Occurs, and the peak of more than 3 times the peak data rate 943 is rare. However, when it does occur (for example, during a scene transition), the data rate required must produce a good quality video image. One way to solve this problem is to simply configure the video compressor 404 to have its maximum data rate output as the peak data rate 941. Unfortunately, the resulting video output during the peak © value frame is poor because the compression algorithm is extremely lacking. What is caused is that compression artifacts occur when there is a sudden transition or rapid motion, and in time, the user begins to realize that the artifact always suddenly appears when there is a sudden change or rapid motion, and it can become quite annoying. . The visual system of the person is quite sensitive to visual artifacts that occur during sudden changes or rapid movements, but is not very sensitive to the small/salty of the frame rate measured in such cases. In fact, when these sudden: changes occur, it seems that the human visual system is focused on tracking the changes, and if the frame rate temporarily drops from 60 fps to 3 〇 fps and then immediately returns to 6 () 枷, Then the visual system of = will not notice. In addition, in the case of very sharp transitions (such as sudden scene changes), if the frame rate drops to 2 〇 fps or even 15 fps and then immediately returns to 6 〇, the human visual system will not notice. As long as the frame rate reduction occurs only occasionally, it seems to the human observer that the video system is continuously executed at 60 fps. Utilizing the technology described in the figure to take advantage of the human visual system 136720.doc • 85 - 200941233

Sex. Server 402 (from Figures 4a and 4b) produces an uncompressed video output stream at a stable frame rate (in one embodiment, at 60 fps). The timetable shows the output of each frame 961-970 for each 1/60 second. Starting from frame 961, each uncompressed video frame is output to the low latency video compressor 404, and the low latency video compressor 4〇4 compresses the frame for less than one frame time, resulting in The compressed frame of the first frame is 981. The data generated for the compressed frame 1 98 can be larger or smaller depending on many factors as previously described. If the data is small enough to transmit the peak data rate 941 to the client 415 at frame time (1/6 sec) or less than one frame time, then at the transmission time (xmit time) 991 (indicating the transmission time) It is transmitted during the length of the arrow of duration. In the next Figure I: time, the server 402 generates an uncompressed frame 2 962, which is compressed into a compressed frame 2 982 'and at a peak data rate during a transmission time 992 less than the frame time. 941 transmits it to the client 415. Next, in the next frame time, the server 402 generates an uncompressed frame 3 963 ° when compressed by the video compression (4) 4 to compress the uncompressed frame 3 963, the compressed frame 3 983 is comparable The peak data rate 94 is the data that is transmitted in a frame time. Therefore, it is transmitted during the transmission time (2 times peak) 993, which is part of the (four) frame time and the next frame time. Now, during the next frame time, the server 402 generates another un-collapsed frame 4 964 and outputs it to the video compressor 4G4' but the data is ignored and illustrated by 974. This is because video compressor 404 is configured to ignore other uncompressed video frames that arrive when they are still transmitting previously compressed frames. Of course, the video decompressor of the client I36720.doc -86 - 200941233 41 5 will not receive the frame 4, but it simply continues to display the frame 3 on the display device 422 for 2 frame times (ie , temporarily reduce the frame rate from 60 fps to 3〇fpS). For the next frame 5, server 402 outputs uncompressed frame 5 965, compresses it into compressed frame 5 985 and transmits it in a frame during the 5th transmission time. The video decoder of the client 415 solves the frame 5 and displays it on the display device 422. Next, the server 牝 2 outputs the uncompressed frame 6 966, and the video compressor 4 〇 4 compresses it into a frame 6 986 which is compressed and collapsed, but the f material obtained at this time is very large ^ at the transmission time ( 4 times peak) During the 996 period, the compressed frame is transmitted at the peak data rate 941, but it takes almost 4 frame times to transmit the frame. During the next three frame times, the video compressor 404 ignores the three frames from the server 4〇2, and the decompressor of the client 415 stably holds the frame 6 on the display device 422 for a period of time 4 Frame time (ie, temporarily reduce the frame rate from 6〇fps to 15 fps). Finally, the server 4〇2 outputs the frame 1〇97〇, and the video compressor 404 compresses it into the compressed frame 10 987 and transmits it during the transmission time 997, and the solution of the user terminal 4丨5 The compressor decompresses the frame 1 〇 and displays it on the display device 422 and restarts the video again at 6 〇 fps. In the meantime, although the video compressor 404 discards the video frame from the video stream generated by the server 4〇2, it does not discard the audio material (regardless of the form of the audio system)' and when discarding the videogram The frame-time video compressor 4〇4 continues to compress the audio data and transmits it to the client 415. The client 415 continues to decompress the audio data and provides the audio to the user for playback of the sound 136720.doc •87-200941233 No matter what device. Therefore, during the period in which the frame is discarded, the audio continues without being attenuated. Compressed audio consumes a relatively small percentage of the bandwidth compared to compressed video and therefore does not have a significant impact on the overall data rate. Although not illustrated in any of the data rate maps, there is always a data rate capacity reserved for the compressed audio stream within the peak data rate 941. Select the example just described in Figure 9b to illustrate the peak data rate

During the period when the frame rate drops, but the undescribed system # uses the cyclic Ι/P image block technique described previously, the data rate peaks and the consequent discarded frames are few 'even in high scene complexity. This is also true during periods of high/high motion, such as those found in video games, movies, and some application software. The frame rate reduced by A' is rare and temporary, and the human visual system does not detect them. Right, the frame rate reduction mechanism just described is applied to the graph % = visual data rate, and the resulting video stream data rate is illustrated in Figure 9c. In this example, the 2x peak 952 has been reduced to a flat = value (5), and the double peak 955 has been reduced to 55 times the flattened 4 times the peak 9: 9 full 41 data items 934 remain at or below Peak Data Rate Consumer Internet:: Via the Universal Internet and, in addition, (in the case of ~ high-speed video streaming, 100 Mbs Ethernet or 8 〇 2) or private network The 100-to-loop between the roads (for example, the data center and the office a L-line network Mbps connection) can transmit the high 136720.doc •88- 200941233 motion video stream without peaking, so that multiple uses (for example, transmitting 192 〇χ 1080 at 6 Mbps at 45 Mbps) can use LAN or shared private data connections without flooding the network or network switch backplane. Data Rate Adjustment In one embodiment, the colocation service 210 initially evaluates the channel's available maximum data rate 622 and latency to determine the appropriate data rate for the video stream and then dynamically adjusts the data rate in response thereto. To adjust the data rate, the colocation service 210 can, for example, modify the image resolution and/or the number of frames per second of the video stream to be sent to the client 415. In addition, the colocation service can adjust the quality level of the compressed video. When changing the resolution of the video stream (e.g., from 1280X720 resolution to 640x360), the video decompression logic 412 on the client 415 can scale the image to maintain the same image size on the display screen. In one embodiment, the colocation service 210 suspends the game in the event that the channel is completely exited. In the case of a multiplayer game, the colocation service reports to other users that the user has quit the game and/or paused the game for other users. Discarded or delayed packet In one embodiment, if the data is lost due to packet loss between video compressor 4〇4 and client 415 in Figure 4a or Figure 4b, or cannot be decompressed due to late arrival And the packets satisfying the delay requirements of the decompressed frame are lost without being received in order, and the video decompression logic 412 can mitigate visual artifacts. In the streaming/P-frame implementation, if there is a lost/delayed packet, the entire screen is affected, which may cause the screen to freeze completely. 136720.doc -89- 200941233 ❹ One time period or other screen wide visual false Shadow. For example, if a lost/delayed packet is caused! The frame is missing' then the decompressor will lack a reference for all p-frames to follow before receiving a new frame. If you lose the P frame, will it affect the following for the entire screen? Frame ^ Vision The number of A before the frame appears, this will have a longer or shorter visual impact. Using the interleaved I/p blocks shown in Figures 7a and 7b, the lost/delayed packets are unlikely to affect the entire screen' because it only affects the image I contained in the affected packet. (4) Sending 'If the packet is lost' in an individual packet, it only affects one image block. Of course, the duration of the visual artifact will depend on whether the image block is lost and how many frames will be spent before the image block appears in the event of a loss of the image block. However, assuming that the different image blocks on the screen are updated very frequently (perhaps each frame) by the i-frame, even if the image block on the screen is affected, other image blocks may not be affected. In addition, if several packets are lost at the same time (for example, the spike of the temporarily interrupted data stream in the power adjacent to the DSL line), then some of the image blocks will be more affected than other image blocks, but because some image blocks will be used by The new image block is updated quickly so it is only temporarily affected. Also, in the case of • rogue/P® box implementation, 'not only the I frame is the most critical frame, but the frame is extremely large, so if there is an event that causes a drop/delay packet, it is much smaller. [Image block t is more likely to be affected than the 1 frame (that is, if any part of the I frame is lost, it is impossible to decompress the frame) because of all the original and frame cases. Using a Ι/p image block results in a much smaller visual artifact when the packet is discarded/delayed. 136720.doc 200941233 An embodiment attempts to transmit (four) protocol run or __ data by compressing the image block = Install = to reduce the effect of missing packets. For example, in the embodiment - the seal = can 'align the image block with the packet boundary. Figure iGa shows that the image block is packaged in the series without implementing this feature Packet ^ Ο

Deleted. Specifically, in Figure 1 Gat, the image block crosses the packet boundary and is inefficiently packaged such that the loss of a single packet results in multiple losses. For example, if the packet circle or _4* loses m block, it causes visual artifacts. & In contrast, Figure (4) illustrates the image block encapsulation logic used to wrap image blocks intelligently in a packet to reduce the effects of packet loss. First, image block encapsulation logic 1010 aligns the image block with the packet boundary. Therefore, the boundary between the image blocks Ti, T3, T4, T7, and T2 and the packet bribe 1〇〇5 = standard. The image block encapsulation logic also attempts to combine the image blocks into the packet in a more efficient manner of T's without crossing the packet boundary. Based on the size of each of the image blocks, the image blocks are combined in a packet 1〇〇1; the Τ3 and Τ5 are combined in a packet 1002; the image blocks are combined in a package 1 In 〇〇3; the image block D8 is added to the packet 1〇〇4; and the image block τ2 is added to the packet 1005. Therefore, under this scheme, a single packet loss will result in the loss of no more than 2 image blocks (rather than the 3 image blocks as illustrated in Figure 10a). An additional benefit of the embodiment shown in Figure 10b is that the image blocks are transmitted in a different order in which they are displayed within the scene> image. If the adjacent packet is due to the same event that interferes with the transmission (which will affect the area on the screen that is not close to each other 136720. Doc •91 · 200941233 domain) is missing, this method produces less noticeable artifacts on the display. An embodiment uses forward error correction (FEC) techniques to protect a particular portion of the video stream from channel errors. As is known in the art, techniques such as Reed-Solomon and Viterbi2FEC generate error correction information and attach it to the material transmitted via the communication channel. If an error occurs in the basic data (for example, I frame), FEC can be used to correct the error. 〇 The FEC code increases the data rate of the transmission, so ideally it is used only when it is most needed. If the material is being sent and it will not result in a very noticeable visual artifact, then the FEC code may preferably not be used to protect the material. For example, a p-image block immediately before a lost I-picture block will only produce a 1/60 second visual artifact on the screen (ie, the image block on the screen will not be updated). This visual artifact can hardly be detected by the human eye. As the P image block is further backward from the I image block, the loss of the P image block becomes more and more noticeable. For example, if the image block cycle pattern is used again in the I image block, then one image block is followed by 15 P image blocks, then if the P image block immediately after the I image block is lost, then The image that caused the image block to display an incorrect image lasted for 15 frame times (under 6 inches, it will be 25 milliseconds). The human eye will easily detect a 250 millisecond flow interruption. Therefore, the more the p image block is from the new I image block (that is, the closer the P image block follows the image block), the more noticeable the artifact is. As discussed earlier, in spite of this, in general, the closer a P-picture block follows an I-picture block, the smaller the data used for the P-picture block. Therefore, the p-image block following the I image block is not only protected from loss 136720. Doc -92- 200941233 is more critical' and its size is smaller. In addition, in general, the smaller the information that needs to be protected, the smaller the FEC code needed to protect it. Thus, as illustrated in Figure 11a, in one embodiment, only the image block has an FEC code due to the importance of the I-picture block in the video stream. Therefore, the FEC 1101 contains an error correction code for the 1 video block 1100 and the FEC 1104 contains an error correction code for the I picture block 1103. In this embodiment, no FEC is generated for the p-picture block. In one embodiment illustrated in Figure lib, an FEC code is also generated for P-picture blocks that are most likely to cause visual artifacts when lost. In this embodiment, FEC 1105 provides error correction codes for the first 3 p-image blocks but not for the following p-picture blocks. In another embodiment, an FEC code is generated for the p-image block with the smallest data size (which would tend to be the first Ρ image block that appears after the τ image block, which is most critical for protection). In another embodiment, instead of transmitting the FEC code along with the image block, the image block is transmitted twice, each time in a different packet. If one packet is lost/delayed, another packet is used. In one embodiment shown in Figure lie, fec codes 1111 and 1113 are generated for audio packets 111 and 1112, respectively, for simultaneous transmission from video to the colocation service. Maintaining the integrity of the audio in the video stream is especially important because distorted audio (eg, clicks or clicks) will result in a particularly inconvenient user experience. The FEC code helps ensure that the audio content is reproduced without distortion at the user's end 415. The ν %s玳 tribute is sent together, but the audio data is transmitted twice, each time the mother transmits I36720 without asking the packet. Doc •93- 200941233 If one packet is lost/delayed, another packet is used. In addition, in one embodiment illustrated in FIG. 9D, FEC codes 1121 and 1123 are respectively used for user input commands (eg, button presses) 112〇 and 1122 for uplink transmission from the client terminal 415 to the colocation service 210. . This is important because 'missing button presses or mouse movements in video games or applications can lead to an undesirable user experience. In another embodiment, rather than transmitting the FEC code along with the user input command data, the user input command data is transmitted twice, each time in a different packet. If one packet is lost/delayed, another packet is used. In one embodiment, the colocation service 21 evaluates the quality of the communication channel with the client 415 to determine whether to use the FEC, and if so, determines which portion of the video, audio, and user commands should be applied for FEC. The "quality" of the evaluation channel may include functions such as estimating packet loss, delay, etc. as described above. If the channel is particularly unreliable, the colocation service 21 can apply FEc to all I video blocks, p video blocks, audio and user commands. In contrast, if the channel is reliable, the colocation service 21 can apply FEC only for audio and user commands, or FEC may not be applied to audio or video, or may not use FEC at all. Various other arrangements of FEC applications can be used while still adhering to these basic principles. In an embodiment, the colocation service 210 constantly monitors the status of the channel and changes the policy accordingly. In another embodiment, referring to FIG. 4 & and FIG. 4b, when the packet is lost/delayed, resulting in loss of image block data, or if the packet may be lost due to a particularly bad packet, the FEC cannot correct the lost image. Block data, with 136720. Doc -94 - 200941233 The client 415 evaluates how many frames remain before receiving a new i-picture block and compares it to the round-trip delay from the client 41 5 to the colocation service 210. If the round trip delay is less than the number of the new image block that should arrive at the previous frame, the client 415 sends a message to the colocation service 21 to request a new 1 image block. This message is routed to the video compressor 404, and it does not generate a p-picture block for the image block in which the data has been lost, but instead generates a video block. Assuming that the system shown in Figures 4a and 4b is designed to provide a round trip delay of typically less than 80 milliseconds, this results in the image block being corrected within © milliseconds (at 60 fpS, the frame has a duration of j 6 milliseconds) Therefore, in the full frame time, the 80 millisecond delay will result in 83. The corrected image block within 33 milliseconds, 83. 33 milliseconds is 5 frame times, which is an attractive interruption 'but far less (for example) for 15 frames with 25 milliseconds of interruption being noticeable). When the compressor 404 is out of its normal cyclic order to generate such a block of image blocks, if the I block will cause the bandwidth of the frame to exceed the available bandwidth, the compressor 404 will delay the loop of the other blocks so that The other image blocks receive the P image block during the frame time (even if the image block will normally be an I image block during the frame), and then the usual loop will continue from the next frame, and will usually The jersey block that received the image block in the previous frame will receive the I image block. Although this action temporarily delays the phase of the R frame cycle, it will generally be visually unobtrusive. Video and Audio Compressor/Decompressor Implementation Figure 12 illustrates a particular embodiment in which multiple cores and/or multiple processors 1200 are used to compress eight image blocks in parallel. In an embodiment, it is used in . ...Double-core processing performed at a higher or higher level ^ quad-core and (10) (four) computer 136720. Doc -95- 200941233 System 'Each core as an independent process to implement open source x264 Η. 264 compressor. However, a variety of other hardware/software configurations can be used while still adhering to these basic principles. For example, each of the CPU cores can be implemented by an FPGA.  264 compressor to replace. In the example shown in Figure 12, the core 1201-1208 is used to simultaneously process the image block and the P image block as eight independent threads. As is well known in the art, current multi-core and multi-processor computer systems are inherently integrated with Microsoft Windows XP Professional (64-bit or 32-bit versions) and Linux multi-threaded processing systems. The ground can perform multi-line processing. In the embodiment illustrated in Figure 12, since each of the eight cores is only responsible for one image block, it operates largely independently of the other cores, each performing a separate instantiation of x264. Use a PCI Express xl-based DVI capture card (such as the Sendero video imaging ip development board from Microtronix of Oosterhout in the Netherlands) to capture uncompressed video at 640x480, 800x600 or 1280x720 resolution, and on the card The FPGA uses Direct Memory Access (DMA) to transfer the captured video to the system ram via the DVI bus. The image blocks are configured in a 4x2 configuration 1205 (although this is illustrated as a square image block, in this embodiment 'which has a 160x240 resolution). Each instantiation of x264 is configured to compress one of the eight 160x240 image blocks and is synchronized such that each core enters a loop after initial I image block compression, each frame and another The frame is out of phase' to compress an I image block followed by seven P image blocks, as illustrated in FIG. At each frame time, the resulting pressure is 136720 using the techniques previously described. Doc • 96- 200941233 The reduced image blocks are combined into a packet stream, and then the compressed image block is transmitted to the destination client terminal 4 1 5 . Although not illustrated in Figure 12, if the data rate of the combined 8 image blocks exceeds the specified peak data rate 941, all 8 x264 processes will temporarily suspend the necessary frame time until the 8 for the combination has been transmitted. The data of the image block. In one embodiment, client 415 is implemented as a software executing on 8 instantiated PCs of FFmpeg. The receiving process receives eight image blocks and routes each image block to FFmpeg for instantiation, and FFmpeg instantiates the decompressed image block and reproduces it to the appropriate image block location on display device 422. The client 415 receives the keyboard, mouse or game controller input from the input device driver of the PC and transmits it to the server 402. The server 402 then receives the input device data received by the application and applies it to the game or application executed on the server 402. The server 402 uses Intel 2. The 16 GHz dual-core CPU executes the Windows PC. Server 402 then generates a new frame and outputs the new frame via the DVI output to the motherboard-based graphics ® system or via the DVI output of the NVIDIA 8800GTX PCI Express card. At the same time, the server 402 outputs audio generated by the game or application via its digital audio output (eg, S/PDIF), which is coupled to a dual quad-core Xeon-based PC that implements video compression. Digital audio input. The Vorbis open source audio compressor is used to compress audio simultaneously with video, regardless of the core that can be used to process the thread. In one embodiment, the core of the image block is first compressed to first perform audio pressure 136720. Doc -97- 200941233 Contraction. The compressed audio is then transmitted along with the compressed video and the compressed audio is decompressed at the client 415 using a Vorbis audio decompressor. The colocation service server center distributes light traveling through the glass (such as an optical fiber) at a fraction of the speed of the light in the vacuum, and thus the exact propagation speed of the light in the fiber can be determined. However, in practice, considering the routing delay, transmission inefficiency, and other elapsed time, we have observed that the optimal delay on the Internet reflects a transmission speed that is closer to 50% of the speed of light. Therefore, the optimal 1 mile round trip delay is about 22 milliseconds, and the best 3 mile round trip delay is about (four) milliseconds. Therefore, a single server on the US coast will be too far apart to serve the client on the other coast (which may be up to 3 miles away) with the desired delay. However, as illustrated in Figure 13a, if the colocation service 21 is located at the center of the United States (e.g., Kansas, Nebraska, etc.), the distance to any point in the continental United States is about i5〇〇. ^ Miles or less than 150 miles 'The round trip internet latency can be as low as 32 milliseconds. Referring to Figure 4b, note that although the worst case delay allowed by the user ISP 453 is 25 milliseconds, it is generally observed that in the case of DSL and cable modem systems, a delay of approximately 10-15 milliseconds is observed. Further, Fig. 4b assumes that the maximum distance from the user premises 211 to the colocation center 210 is 1 mile. Thus, in the case of the typical IS5 millisecond user ISp round trip delay used and the maximum internet distance of 15 mile for a 32 millisecond round trip delay, the input device 421 is actuated from the user. The total round trip delay from the moment to the response on the display device 422 is 136,720. Doc -98 · 200941233 1 + 1+ 15+32 + 1+ 16+6+8=80ms. As a result, an 80-millisecond response time is typically achieved over an Internet distance of 15 miles. This will allow any user location in the US mainland with a sufficiently short user ISP delay of 453 to be accessed in a centrally located single server center. In another embodiment illustrated in Figure 13b, the colocation service 21" server centers HS1-HS6 are strategically located around the United States (or other geographic area), a particular larger colocation service server center Positioned close to high population centers (eg, HS2 and HS5). In one embodiment, the server centers G hS1-HS6 exchange f signals via the network 13 (1), and the network 13G1 may be an internet or private network or a combination of the two. In the case of multiple server centers, users with high user Isp delays 453 can be served at lower latency. The distance on the S-Internet is indeed a factor that affects the latency of travel through the Internet, but sometimes other factors that are largely unrelated to latency also work. Sometimes the packet flow is routed to the distance 〇 ^ via the Internet and returned again, resulting in a delay from the long loop. Sometimes there is a routing device on the path that does not operate properly, causing delays in transmission from the towel. There are sometimes traffic that overloads the path, which introduces delays. In addition, sometimes there is a failure to prevent the user's lsp from routing to a given destination. Thus, while the universal Internet usually provides connections from one point to another with fairly reliable and optimal routing and latency, this fairly reliable and optimal route and delay is largely determined by distance ( Especially in the case of long-distance connections that are routed to the outside of the user's local area), but the reliability and delay are not guaranteed and often cannot be 136720. Doc -99· 200941233 The user's place is reached to a given destination on the universal Internet. In one embodiment, when the user client 415 is initially connected to the colocation service 210 to play a video game or use an application, the client initiates and each of the available colocation service server centers HS1_HS6 at startup. One communication (eg 'using the techniques described above). If the delay is low enough for a particular connection, use the connection. In one embodiment, the client and all of the courier service feeders or colocation services are configured to select the colocation service with the lowest latency connection. The client can select the service center with the lowest latency connection, or the server center can identify the server center with the lowest latency connection and provide this information (for example, in the form of an internet address) to the client. The client 415 can be redirected if the specific colocation service server center is overloaded and/or the user's game or application can tolerate the delay to another, less loaded colocation service terminator center. Go to another host agency service center. In this case, the game or application being executed by the user will be temporarily placed on the server 402 at the user's overloaded server center, and the game or application status data will be transferred to another colocation service. The server at the server center is 4〇2. The game or application will then restart. In one embodiment, the colocation service 21 will wait until the game or application reaches a natural pause point (eg, between levels in the game or after the user initiates a "save" operation in the application). Transfer. In yet another embodiment, the colocation service 21 will wait until the user activity ceases for a specified period of time (eg, ^ 136720. Doc -100· 200941233 clock) and the receiver will start transmission at that time. As described above, in one embodiment, the colocation service 210 subscribes to the Internet bypass service 440 of Figure i4 in an attempt to provide a guaranteed delay to its client. The Internet Bypass Service, as used herein, provides a service for private network routes with guaranteed features (e.g., latency, data rate, etc.) from one point to another on the Internet. For example, if the host hosting service 210 is using the AT&T DSL service provided by San Francisco to receive a large amount of traffic from the user (rather than to the central office based on AT&T San Francisco), The colocation service 210 will then lease from the service provider between one or more of the central office based in San Francisco and the server center for the colocation service 210 (possibly AT&T itself or another A high-capacity private data connection for a provider). Then, if all the hosted service server centers HS1-HS6 go through the universal Internet to Sanbu... (7), the route using the user of at&t DSL causes excessive delay, and the shot is changed to use private funds# connection. Although private data connections are usually more expensive than routes over a common Internet, as long as they maintain a small percentage of the colocation service 21 to the user, the total cost impact is low and the user will experience more consistent Service experience. In the event of a power failure, the server center often has two spare power layers. The first layer is typically the backup power from the battery (or from an alternate ready-to-use energy source, such as a flywheel that remains operational and attached to the generator) It immediately supplies power when the mains fails, and keeps the server running. If the power failure is temporary and the mains return quickly (eg 136720. Doc 200941233 For example, within one minute), the battery is required to keep the center of the feeder running. However, if the power failure lasts for a long period of time, the generator (e.g., 'diesel power) is typically activated to replace the battery and the generator can operate as long as it has fuel. These generators are extremely expensive because they must be capable of generating the power typically obtained from the mains of the server. In one embodiment, each of the 'hosting services Cong-HS5 shares user data with each other to have power in a server center _ > pauses in ongoing games and applications' And then the game bite status data is transmitted from each of the feeding machines 4〇2 to the feeding device 402 at the other server center, and then the user is notified to each user to guide the communication to the user. New feeding machine. Assuming that such occasions occur occasionally, the user is transferred to a hosted service server center that is not capable of providing the best latency (ie, the user will only have to tolerate the duration of the higher latency duration power failure (1) may be Accepted, it will allow the user to have a much wider range of options. For example, a user who is on the East Coast of the United States will be able to sleep at 11:3 〇ρΜ Users on the West Coast began to peak in video game usage at 8:3QpMi. If there is a power failure in the hosted service server center on the West Coast, other hosted service server centers may not There are sufficient West Coast Feeding Machines for handling all users. In this case, some users can be transferred to the hosted service feeder center with available feeders 402 on the East Coast, and for The only consequence of the user: will be a higher delay. Once the user loses power from the 5 servers, the server center can then start its server and set 136720. Doc 200941233 is shut down in an orderly manner to shut down all equipment before the battery (or other immediate power backup) is exhausted. In this way, the cost of the generator for the servo center can be avoided. In an embodiment, during the time of severe loading of the colocation service 21 (or due to peak user loading, or because one or more feeder centers fail), based on the game the user is using. Or the application's delay requires the user to be transferred to another server center. Therefore, users of games or applications that require low latency will be given the preference for available low latency server connections with limited © supply. Hosted Hosting Service Features Figure 15 illustrates an embodiment of the components of the server center for the hosted hosting service 210 utilized in the following description of the features. As with the colocation service 210 illustrated in the figure, unless otherwise conditional, the components of the server center are controlled and coordinated by the colocation service 21〇 control system 〇1. The incoming Internet traffic 15〇1 from the user client 415 is directed to the incoming route 1502. Typically, incoming Internet traffic 1501 will enter the server center via a high-speed fiber connection to the Internet, but any network connectivity component with sufficient bandwidth, reliability, and low latency will suffice. Inbound routing 1502 is a network (which can be implemented as an Ethernet, fiber channel network, or via any other transport component) switch and a system that supports the routing servers of the switches, which arrives Packets are routed and routed to the appropriate application/game server 1521_1525. In one embodiment, the 'delivery to a particular application/game server's packet indicates its own usage 136720. Doc -103- 200941233 A subset of the data received by the client and/or can be translated/changed by other components in the data center, such as 'network connection components such as gateways and routers. In some cases, the packet is routed to more than one server 1521-1525 each time, for example, if the game or application is executing concurrently on multiple servers. The RAID arrays 15 11-1512 are connected to the incoming routing network 1502 such that the application/game server 1521_1525 can read the raid arrays 1511-1512 and the write arrays 1511_1512. In addition, an array 1515 (which can be implemented as a plurality of RAID arrays) is also coupled to the routing router 1502' and the data from the RAID array 1515 can be read from the application/game feeders 1521-1525. Incoming routing 15〇2 can be implemented in a variety of prior art network architectures (including tree-structured switches, inbound Internet traffic 15〇1 at its root); in a mesh structure interconnecting all of the various devices Implemented; or implemented as a sub-network sequence of interconnects (concentrated traffic among interworking devices and centralized traffic isolation among other devices). One type of network is configured as S AN, which, although commonly used for storage devices, can also be used for general purpose data transfer between devices. Also, the application/game servers 1521-1525 can each have multiple network connections to the incoming route 15〇2. For example, servers 1521-1525 may have a network connection to a subnet attached to RAID arrays 1511-1512 and another network connection to a subnet attached to other devices. The application/game servers 1521-1525 can be configured identically, somewhat differently, or all differently, as previously described with respect to the server 402 in the embodiment illustrated in the drawings. In one embodiment, each user is typically at least one application/game servo 136720 when using the colocation service. Doc 200941233 1521-1525. For the sake of simplicity of description, it will be assumed that a given user is using the application/game server 1 52 1, but multiple servers can be used by one user, and multiple users can share a single application/game server. 1521-1525. The user's control input sent from the client 415 (as previously described) is received as an incoming Internet service 15〇1 and routed to the application/game server 1521 via the incoming route 1502. The application/game server 1521 uses the user's control input as a control input to the game or application executing on the server and calculates the next frame of the video and © associated audio. The application/game feeder 1521 then outputs the uncompressed video/audio 1529 to the shared video compression 1530. The application/game server can output uncompressed video via any component (including one or more ultra-high speed Ethernet connections), but in one embodiment, the video is output via a DVI connection, and the audio and Other compression and communication channel status information is output via a universal serial bus (USB) connection. Shared Video Compression 1530 compresses uncompressed video and audio from the app/game vocabulary mi·® 1525. This compression can be implemented entirely in hardware or in hardware that executes the software. There may be a dedicated compressor for each application/game server 1521-1525, or if the compressor is fast enough, a given compressor may be used to compress video from more than one application/game server 1521-1525 / Audio. For example, when added, the video frame time is 16. 67 milliseconds. If a compressor is capable of compressing a frame within i milliseconds, then the compressor can be used to compress from up to 16 applications/games 136720 by taking input from one server after the other. Doc • 105· 200941233 1521-1525 video/audio, which stores the state of each video/audio compression process and switches the background as it cycles through the video/audio stream from the server. This results in substantial cost savings for the compressed hardware. Because the different word processors will complete the frame at different times, in one embodiment, the compressor resources are in a shared storage component (eg, RAM, flash memory) having a state for storing each compression process. In the shared pool 1530, and when the server 1521-1525 is complete and ready to be compressed, the control component determines which compressed resource is available at the time, provides the server with the status of the compressed resource and the state to be compressed. Uncompressed video/audio frame. Note that one of the states of the compression process for each server includes information about the compression itself, such as the decompressed frame buffer data of the previous frame (which can be used as a reference for the P image block), Resolution of video output; quality of compression; image block structure; allocation of bits per image block; compression quality, audio format (eg stereo, surround sound, Dolby® AC-3). However, the compression process status also includes information about the following communication channel status: peak data rate 941' and whether the previous frame (as illustrated in the figure) is currently being output (and therefore the current frame should be ignored), and potentially Is there a channel characteristic that should be considered in compression (such as excessive packet loss) affecting compression decisions (eg, in terms of the frequency of I-picture blocks, etc.). Since the peak data rate 941 or other channel characteristics change over time, as determined by the application/game server 1521-1525 that supports each user monitoring the data sent from the client 415, the application/game The server 1521-1525 sends the relevant information to the shared hardware compression 136720. Doc •106· 200941233 1530. The shared hardware compression 1530 also encapsulates the compressed video/audio using components such as those previously described, and, where appropriate, applies the FEC code, copies specific material, or takes other steps to adequately ensure video. The audio data stream is received by the client 4155 and is decompressed with a viable high quality and reliability. Some applications (such as those described below) require the video/audio output of a given application/game server 1521_1525 to be available at multiple resolutions (or in multiple other formats). If the application/game server 1521-1525 thus notifies the shared hardware compression 153 resource, the uncompressed video audio 1529 of the application/game server 1521_1525 will be in a different format, with different resolutions and/or Simultaneous compression in different packet/error correction structures. In some cases, some compression resources may be shared among multiple compression processes that compress the same video/audio (eg, in many compression algorithms) there are steps to scale the image to multiple sizes before applying compression. If you need to output images of different sizes, this step can be used to servo several compression processes at the same time). In other cases, separate compression resources will be required for each format. In any event, the compressed video/audio 1539 for all of the various resolutions and formats required for a given application/game server 1521_1525 (one or more) is simultaneously output to the outgoing route 154(). In the embodiment, the output of the compressed video/audio 1539 is in the sUDp format, so it is a unidirectional packet stream. The outgoing routing network 1540 contains a series of routing feeders and switches, the 136720. Doc 200941233 Series Routing Servers and Switches direct each compressed video/audio stream to the intended user via the outgoing Internet service 1599 interface, which is typically connected to the optical interface of the Internet Other destinations and/or return to delay buffer 1515, and/or return to incoming route 15〇2, and/or output via a private network (not shown) for video distribution. Note (as described below): Outbound Routing 1540 can simultaneously output a given video/audio stream to a destination. In one embodiment, this uses Internet Protocol (IP) multicasting to implement a given O UDP stream intended for simultaneous streaming to multiple destinations, and the broadcast is routed by the outgoing route (10). The feeder and the exchanger are repeated. The plurality of destinations of the broadcast may be via the Internet to the user terminals 415 of the plurality of users, to the plurality of application/game servers 152M525 via the routing route 15〇2, and/or to one or more Delay buffer 1515. Thus, the output of a given feeder 1521_1522 is compressed into one or more formats and each compressed stream is directed to one or more destinations. Additionally, in another embodiment, if multiple application/game feeders 1521-1525 are simultaneously used by a user (eg, in a parallel processing configuration for generating 3D output with complex scenes) and each If a server generates a portion of the obtained image, the video output of the plurality of servers 1521-1525 may be combined into a combined frame by the shared hardware compression 153, and the combined frame is processed as described above from the other side. It seems like it comes from a single application/game server 1521-1525. Note that 'in one embodiment, 'a copy of all video generated by the application/game server 1521-1525 (at least at the resolution of the video viewed by the user or higher resolution) is recorded in the delay buffer Η" The time is up to 136720. Doc -108- 200941233 Less than a certain number of minutes (15 minutes in one embodiment). This allows each user to "rewind" the video from each session to check previous work or performance (in the case of the game). Thus, in one embodiment, each compressed video/audio output 1539 stream routed to user client 415 is also multicast to delay buffer 1515. When the video/audio is stored on the delay buffer 1515, the directory on the delay buffer 1515 provides the network address and delay of the application/game server 152 1-1525, which is the source of the delayed video/audio. A reference between the location of the delayed video/audio © can be found on the buffer 1515. The live, instantly viewable, instantly playable game application/game server 1521-1525 can be used not only to execute a given application or video game of the user, but also to establish support for hosting via the host A user interface application for the hosted service 210 of the navigation and other features of the service. A screen shot of the user interface application is shown in Figure 16 ("Game Viewfinder" screen). This particular *user interface screen allows the user to view 15 games played (or delayed) by other users on the spot. Each of the "thumbnail" video windows (e.g., 1600) is a live video window in motion that displays a video from a user's game. The view shown in the thumbnail may be the same view that the user is looking at, or it may be a delayed view (eg, if the user is playing a fighting game, the user may not want other users to see where they are hidden and It can choose to delay any view of its game play for a period of time (eg '1G minutes). The view can also be a camera view of the game different from the view of any user. The user can pass 136720 based on various criteria. Doc -109- 200941233 Menu selection (not shown in this description) to select the game to be viewed at the same time. As a small sample of exemplary choices, the user may select a random selection of games (such as those shown in Figure 16), all categories of games (both played by different players), only top players of the game, games A player of a given level, or a lower level player (for example, if the player is learning the basics), a player who is a "partner" (or a competitor), a game with the largest number of viewers, and the like. Note that usually, each user will decide whether the video from their game or application © can be viewed by others' and if so, decide which other people can view and when they can be viewed by others, and decide whether the video can only have Viewing β in the case of a delay The application/game server 1521-1525 that produces the user interface screen shown in Figure 16 is being requested from each user (the application/game server 1521-1525 is requesting the use) The application/game server 1521-1525 sends a message to acquire the 15 video/audio feeds. The message is sent via the incoming route 1502 or another network. This message will include the size and format of the requested video/audio and will identify the user viewing the user interface screen. A given user may choose to select the "Piracy" mode and not allow any other user to view the video/audio of their game (from their view point or from another view point)' or as described in the previous paragraph, the user may choose Allow viewing of video/audio from its games, but delay the video/audio being viewed. The user application/game server 1521_1525 (which receives and accepts requests for its video/audio to be viewed) will therefore acknowledge to the requesting server and will also notify the shared hardware compression 153 that 136720 is required. Doc •110· 200941233 Generates an additional compressed video stream of the requested format or screen size (assuming the format and the size of the screen are different from the format and screen size already generated), and it also displays the destination of the compressed video ( That is, if the requested video/audio is only delayed, the requesting application/game server 1521-1525 will be notified as such, and it will be found in the directory on the delay buffer 1515. The location of the video/audio and the network address of the application/game server 1521_1525, which is the source of the delayed video/audio, and the delayed video/audio from the delay buffer 1515. Once all such requests are generated, Processing, the video stream of up to 15 live broadcast thumbnail sizes is routed from the outgoing route 1540 to the incoming route 15〇2, to the application/game server 1521_1525 that generates the user interface screen, and will be served by the servo Decompressed and displayed. The delayed video/audio stream may be oversized, and if so, the application/game server 1521_ 1525 will decompress the The stream is scaled down to the thumbnail size. In one embodiment, the request for audio/video is sent to a central "management" service similar to the "hosted service control system" of Figure (Figure 15 Not shown) (and managed by a central "management" service), the central "management" service then redirects the requests to the appropriate application/game server 1521-1525. Furthermore, in an embodiment, it may not be required Request, because the thumbnails are "pushed" to the users of their users who are allowed to. All of the simultaneously mixed audio from 15 games may produce harsh sounds. The user can choose to mix all the sounds together in this way. (Maybe to get the feeling of "喧嚣" generated by all the actions being viewed), or the user can choose to listen to the audio from one game at a time. Single tour 136720. Doc 111 200941233 The choice of play is done by moving the yellow selection box 1601 to a given game (the yellow frame can be moved by using the arrow keys on the keyboard, by moving the mouse, by moving the joystick or by pushing This is done by a direction button on another device such as a mobile phone. Once a single game is selected, it is only played from the audio of the game. Also, show game information 16〇2. In the context of this game 'for example, the publisher logo (r EA)) and the game logo "Need for Speed Card Canyon" and the orange bar indicate the number of people who play or view the game at a particular time under relative conditions ( In this case, many, so the © game is "hot". In addition, a "status" is provided indicating that there are 8.5 different instantiations in which 145 play players are actively playing Need for Speed games (ie, the game can be played by individual player games or multiplayer games), and there are 68〇 One viewer (this user is one of them). Note that such statistics (and other statistics) are collected by the hosted service control system 4.1 and stored on the RAID arrays 15 11 - 1512 for maintaining the hosted service 21 operations. Used to properly bill users and pay fees to publishers who provide content. - Some statistics are recorded as a result of actions performed by service control system 40, and some statistics are served by individual applications/games. The switches 1521-1525 report to the service control system 401. For example, when the game is being viewed (and when the game is stopped viewing), the application/game server 152 executing the game viewfinder application 152 1525 sends a message to the colocation service control system 401 to enable The colocation service control system 401 can update how many games are in the view of the statistics. Some statistics can be used by user interface applications such as this game viewfinder application. 136720. Doc -112- 200941233 If the user clicks the start button on their input device, they will see the thumbnail video in the yellow box and the thumbnail video will be live broadcast to full screen size. This effect is shown in the process of Figure 17. Note that the size of the video window 1700 is increased. In order to implement this effect, the application/game server 1521-1525 requests the full screen size of the game routed to it from the application/game server 1521-1525 executing the selected game (by the user's display device 422) Resolution of the video stream. The game-enabled application/game server 1521-1525 notifies the shared hardware compressor 1530 that a copy of the thumbnail size of the game is no longer needed (unless another application/game server 1521-1525 requires such a thumbnail), And then it directs the shared hardware compressor 1530 to send a copy of the full screen size of the video to the application/game server 1521_1525 that amplifies the video. The user playing the game may or may not have a display device 422 having the same resolution as the display device of the user who zooms in on the game. In addition, other viewers of the game may or may not have the same display 422 as the user who zoomed in on the game (and may have different audio playback components, such as stereo or surround sound). Therefore, the shared hardware compressor 153 determines whether a suitable compressed video/audio stream that satisfies the requirements of the user requesting the video/audio stream has been generated, and if the appropriate compressed video/audio stream does exist, the shared hard The volume compressor 1530 notifies the routing 1540 to route the duplicate of the stream to the application/game server 1521_1525 that amplifies the video, and if the appropriate compressed video/audio stream does not exist, the compressed video is suitable for the user. Another copy of the application/game server 1521_1525 that directs the routing to send the stream back to the incoming route 1502 and the video. 136720. Doc -113· 200941233 This server that now receives the full screen version of the selected video will decompress the full screen version and gradually scale it up to full size. Figure 18 illustrates how the screen looks after the game is fully enlarged to full screen and the game is displayed at full resolution of the user's display device 422 (as indicated by the image pointed by arrow 1 800). The application/game server 1521-1525 executing the game viewfinder application sends a message to other application/game servers 1521-1525 that provide thumbnails to indicate that the thumbnails are no longer needed and host the service control servo to the host 4〇1 sends a message to © means that no other games are viewed. At this point, the only display that appears is the top overlay of the top of the screen, which provides information and menu controls to the user. Note that as the game progressed, the audience grew to 2, 5, 3 viewers. In the case of so many viewers, there are necessarily many viewers with display devices 422 that have the same or close resolution (each application/game server 1521-1525 has a proportionally adjusted video) For the ability to adjust the fit). 〇 Because the game shown is a multiplayer game, the user can decide to join the game at some point. The colocation service 21 may or may not allow users to join the game for a variety of reasons. For example, the user may have to pay for the game and choose not to pay, the user may not have enough levels to join the particular game (eg, for other players, it will not be competitive), or use The Internet connection may not have a sufficiently low latency to allow the user to play (for example, there is no latency constraint for viewing the game, so the view can be viewed remotely without delay (actually, Play on another continent), but for 136720. Doc -114- 200941233 For games to play, the delay must be low enough for the user (a) to enjoy the game' and (b) to be in the same position as other players who may have lower latency connections. If the user is not allowed to play, the application/game server 152ΐ_ι525 providing the user interface of the game viewfinder user interface will request the host escrow service to control the feeding machine 4 0 1 start (ie, 'position and start) An application/game server 1 52 J _ 1525 suitably configured to play a particular game to load the game from the RAID arrays 1511-1512, and then the colocation service control server 401 will direct the routing 1502 Transfer the control signal from User© to the application/game server of the current hosted game and now the hosted game application/game server will guide the shared hardware compression 1530 self-compression from the hosted game viewfinder The video/audio of the application/game server of the app is switched to compress the video/audio of the app/game pad from the current hosted game. The vertical synchronization of the game viewfinder application/game service with the new application/game server of the colocation game is not synchronized, and thus there may be a time lag between the two synchronizations. Since the shared video compression hardware 1530 will begin to compress the video after the application/game server 1521-1525 completes the video frame, the first frame from the new server can be completed earlier than the full frame time of the old server. The first frame from the new server may be before the transmission is completed by the previously compressed frame (for example, consider the transmission time of GPU: 992: if the uncompressed frame 3 963 is completed earlier than the frame time In half, it will impact transmission time 992). In this case, the shared video compression hardware 1530 will ignore the first frame from the new server (e.g., ignore (974) frame 4 964), and the client 41 5 will come from the old servo 136720. Doc -115- 200941233 The last frame of the device maintains an extra frame time, and the shared video compression hardware 1530 will begin to compress the next frame time video from the new application/game server of the colocation game. For the user, the transition from one application/game server to another application/game server will be visually seamless. The colocation service control server 4 then switches the application/game server 1521·1525, which has notified the courier game viewfinder, to the idle state until it is needed again. The user can then play the game. In addition, the 'exceptional game will be played immediately on the Sense© (because the game has been loaded from the 1^1 car column 1511-512 to the app/game server 1521_1525 at a rate of one billion bits per second). ), and the game will be loaded with the ideal drive, register configuration (in the case of wind WS2) along with the operating system configured for the game to the server that is exactly suitable for the game Other applications that are not likely to compete with the operation of the game are executed on the server. Also, as the user progresses through the game, each of the segments of the game will be loaded at a rate of one billion bits per second (i.e., loaded in 8 seconds and one billionths) from the ® RaID array 1511· The 1512 is loaded into the server and due to the huge storage capacity of the RAm array 1511-15 12 (because it is a shared resource for many users, it can be very large, but still cost effective), allowing pre-computed geometry settings Or other game clip settings and store them on the RAID arrays 1511-1512 and load them very quickly. In addition, because the hardware configuration and computational capabilities of each application/game servo n 1521_1525 are known, pixel and vertex shading can be pre-computed. Therefore, the game can be launched almost instantly, it will be executed in an ideal environment, 136720. Doc -116 - 200941233 and the subsequent clips will be loaded almost instantly. However, in addition to these advantages, the latter will be able to view other people playing the game (via the previously described game viewfinder, and other components), and both determine whether the game is interesting, and if so, then watch others. Learning skills. In addition, the user will be able to present the game instantly without having to wait for a large download and/or installation, and the user will be able to play the game instantly (perhaps on a less expensive trial basis or on a longer term basis) . In addition, the user will be able to play the game on a windyws © PC, Macintosh, on a television, at home, on the go, and even on a mobile phone with a sufficiently low latency no-wire connection. In addition, this can be done without having a copy of the game.

As previously stated, the user may decide not to allow his game play to be viewed by others to allow his game to be viewed after a delay, to allow his game to be viewed by a selected user, or to allow his game to be viewed by all users. In the embodiment, the video/audio is stored in the delay buffer 1515 for 15 minutes, and the user will be able to "rewind" and view his previous game play, and pause the game to slowly play back the game. Fast forward the game, etc., as it can be seen while watching τν with a digital video recorder (DVR). Although the user is playing the game in this example, the same "DVR" capability is available if the user is using the application. This may be useful in verifying previous work and in other applications as detailed below. In addition, if the game is designed to have the ability to fall back based on the use of game state information, so that the camera view can be changed, etc., the "3D DVR" capability will also be supported, but it will need to design the game to support "3D 136720. Doc -117- 200941233 earning ability. The "nen" ability to use the delay buffer 1515 will work in conjunction with any game or application (of course, limited to the video generated when using a game or application), but in the case of a game with 3D 11¥11 capability. The user can control the 3" "traversal" of the previously played clip and cause the delay buffer 1515 to record the video and record the game state of the game segment. Therefore, the specific "traversal" is recorded as compressed video, but since the game state will also be recorded, different crossings may be at a later date in the same segment of the game. As described below, users on the colocation service 210 will each have a user page where the user can post information about himself and other materials. One of the things that the user will be able to announce is a video clip from a game that has been saved. For example, if the user has overcome the particular difficulty of the game, the user may just "rewind" to the location where he or she has achieved great results in the game, and then instruct the colocation service 210 to Video clips of duration (eg, 3 seconds) are saved on the user's user page for viewing by other users. The only thing to do to implement the application/game server 1521_1525 that the user is using is to play back the video stored in the delay buffer 1515 to the RAID array 1 5 11 -1 5 12 and then edit the video clip. Indexed on the user's user page "If the game has 3D DVR capabilities" as described above, the user can also record the game state information required by the 3D DVR and make it available to the user's user page. In the case where the game is designed to have "bystanders" in addition to active players, 136720.doc 118- 200941233 (that is, users who can travel in the 3D world without observing and observe the action) The game viewfinder app will enable users to join the game as a bystander and player. From the point of view of the self-examination, there is no difference in the way that the host is a bystander rather than an active player. Load the game onto the app/game pad 1521 1525 and the user will control the game (eg, control the virtual camera viewing the world). The only difference will be the user's gaming experience. Multiple User Collaboration Another feature of the colocation service 210 is the ability of multiple users to collaborate while viewing live video (even with very different devices for viewing). This is useful when playing games and when using an application. Many PCs and mobile phones are equipped with video cameras and have the ability to perform instant video compression (especially when image hours are small, small cameras are available, can be attached to the TV, and many hardware compression devices for compressing video or using video compression) One of them is not difficult to implement instant compression. In addition, many PCs and all mobile phones have microphones, and the headphones are available with a microphone. The combination has local video/audio compression capabilities (specifically, use this article The cameras and/or microphones of the described low latency video compression technology will enable the user to transmit video and/or audio along with the input device control data from the user premises 211 to the colocation service 21 . When the technology is used, the ability illustrated in Figure 19 can be achieved: the user can have his video and audio 19GG appear on the screen of another user's game or application. This example is a multiplayer game. Teammates collaborate in the car. The video/audio of 136720.doc -119- 200941233 can only be selectively viewed/heard by their teammates. Since there is effectively no delay in using the techniques described above, the player will be able to talk to each other or exercise without noticeable delays in real time. This video/audio integration is achieved by taking the camera/microphone from the user. The compressed video and/or audio is completed as an incoming Internet service 1501. The incoming route 1502 then routes the video and/or audio to an application/game that is permitted to view/hear video and/or audio. Servers 1521-1525. Then select the user of the respective application/game server I52 using video and/or audio to decompress the video and/or audio and integrate it as needed to appear in the game or application. Within the program, such as illustrated by 1900. The example in Figure 19 shows how to use the collaboration in a game, but the collaboration can be an extremely powerful tool for the application. Consider a situation where one large building is being The architect in Chicago designed the New York City for a New York-based real estate developer, but the decision involved the move and happened to be at the Miami Airport Investors, and need to make decisions about specific design elements of the building (how they relate to the buildings in the vicinity) to meet both investors and real estate developers. Assume that the construction company has attached to the PC in Chicago. High-resolution cameras for cameras, real estate developers have laptops with cameras in New York, and investors have mobile phones with cameras in Miami. Construction companies can use colocation services 210 to host highly realistic A powerful architectural design application for 3D reproduction' and its access to a large database of buildings in New York City and a library of buildings being designed. The architectural design application will be available in the application / 136720.doc -120- 200941233 Execution is performed on one of the game servers 1521-1525 (or on several if it requires a large amount of computing power). Each of the three user orders at the disparate location will be connected to the colocation service 21, and each will have a simultaneous view of the video output of the architectural design application, but it will be targeted for each The user has a given device and network connectivity characteristics that are appropriately sized by the shared hardware compression 1530 (for example, a building company can see 256 〇 xl 44 〇 6 〇 fps display via a 2 〇 commercial internet connection, New York Real Estate Developers can see © 1280x720 60 fps images via a 6 Mbps connection on their laptops, and they can see 32〇χΐ8〇6〇fps images via the #(四)mu 250 Kbps cellular data connection. Hearing other parties' voices (which will be handled by any of the many widely available conference call suite software in the application/game server i52i_ 1525) and actuated via a button on the user input device Users will be able to use their local camera to make video appear. With the conference = line, the architect will be able to showcase the architecture through a photorealistic 3D rendering Rotate and make it look like what the building looks like when another building in the area traverses, and all parties will see the same video at the display resolution of each party. The local end used by any party Any one of the devices can handle 3D animation with this realism without problems, let alone download or even store the huge library needed to reproduce the surrounding buildings in New York City from the point of view of each of the users. Although it is far away, and although it is a disparate native device, it will simply have a seamless experience at a difficult level. Furthermore, when one wants its face to be better able to convey its emotional state, it can do so In addition, if the real estate 136720.doc -121 - 200941233 developer or investor wants to control the building program and use its own input device (which is keyboard, slippery, keypad or touch glory), then it can be It can respond without a perceived delay (assuming its network connection is not and has a reasonable delay). For example, in the case of a mobile phone;:: The phone is connected to the side of the airport, then its It has a very low latency. But if it is manufactured using bee m Lumei now available off the ❹

Will suffer an lag that is noticeable. However, for most of the purposes of the conference (where investors are watching architects controlling building crossings or talking about video conference calls), even honeycomb delays should be acceptable. Finally, at the end of the cooperative conference call, the real estate developer and investor will make their comments and stop the colocation service, and the construction company will be able to "rewind" the video of the conference that has been recorded on the delay buffer 1515 and Review comments, facial expressions and/or actions applied to the 3D model of the building during the meeting. If there are specific segments that it wishes to save, then the segments of the video/audio can be moved from the delay buffer 1515 to the raid array 1511-1512 for archival storage and later playback. Also, from a cost perspective, if an architect only needs to use New York City's computing power and a large database for a 15-minute meeting, then it only needs to pay for the time when the resources are used, without having to have a high The ability of the workbench does not have to buy an expensive copy of the large database. Video-rich community services Hosting services 210 are an unprecedented opportunity to build video-rich community services on the Internet. The exemplary user page of the game player on the exhibition in Figure 2 is for the host hosting service. Like the game viewfinder application 136720.doc -122- 200941233 The same program as the user page is an application executed on one of the application/game servers 1521_ 1525. All thumbnails and video windows on this page show a constantly moving video (if the clip is short, it loops). By using a video camera or by uploading video, the user (the user name is "KILLHAZARD") can publish his own video 2 (other users can view the video ρ and the video is stored on the rAID array 1511-1512). Also, 'When other users come to the KILLHAZARD user page, if KILLHAZARD is using the colocation service 21〇 at this time, it will show the video of the live broadcast of whatever it is doing 2 (assuming its permission The user viewing the user page views it. This will request the KILLHAZARD to be active from the service control system 401 by the application/game server 152ΐ_ι 525 hosted by the host page application (and if so, request it) The application/game server 1521_1525 is being used. Then, using the same method used by the game viewfinder application, the compressed video stream of the appropriate resolution and format is sent to the application executing the user page application. / Game server 1521-1525 and display it. If the user selects a live broadcast game with KILLHAZARD The window and then appropriately click on its input device, the window will zoom in (again using the same method as the game viewfinder application), and the live video will be viewed by the user's display device 422 (suitable for Viewing the characteristics of the user's Internet connection) Filling the screen. This is a key advantage over the prior art method: the user who views the user's page can see the game played live by the user, and does not have the ability to Play the game's local computer or game console. It is 136720.doc -123- 200941233 users have an excellent opportunity to see the users on the user page displayed as "active", and it is a good view. Users may wish to try a better game. The video clips recorded or uploaded by the camera of KILLHAZARD's partner 2002 are also displayed on the user page, and below each video clip is a line indicating whether the partner is playing online. The text of the game (for example, six-sh〇t is playing the game "Eragon" and MrSnuggles99 is offline, etc.). By clicking on a menu item (not shown), the 'Billing Video Clip' is switched from displaying the recorded or uploaded® video to the partner of the game currently playing the colocation service 210 at the moment in the game. The live video of the content. Therefore, it becomes a game viewfinder that is grouped by the partner. If you select the game of the partner and the user clicks on the game, the game will be enlarged to the full screen and the user will be able to watch the full screen. The game played live on the spot. Again, the user of the game viewing the partner does not own the copy of the game, nor does it have the local computing/game console resources for playing the game. The game view is effective instantaneous. As previously described above, when a user plays a game on the colocation service 210, the user can "rewind" the game and find the video clip he wishes to save, and then save the video clip to his user page. . These are called "Brag Clips". Video clips 2003 are self-reward clips 2〇〇3 saved by KILLHAZARD from previous games played by them. The number 2004 shows how many times the self-reviewed clip has been viewed, and when the self-reward clip is viewed. The user has an opportunity to rate it, and the number of icons 2005 of the orange keyhole shape indicates how high the rating is. When the user views the 136720.doc -124- 200941233 user page, the self-review clip 2003 continually loops along with the rest of the video on the page. If the user selects and clicks on one of the self-review clips 2〇〇3, it is zoomed in to present the self-reward clip 2003, and DVR control that allows playback, pause, rewind, fast forward, step, and the like. The self-reward clip 2003 playback is loaded with the compressed video clip stored on the rAIE) array 15 i_ 15 12 by the application/game server 1521_ 1525 by the application/game server 1521_ 1525 and decompressed and played back. Implementation. © Self-recruiting clip 2003 can also be a 3D DVR" video clip from a 3D DVR-capable game (ie, a sequence of game states from a game that can be played back and allows the user to change the camera's view point) „ in this situation Next, game state information is also stored in addition to the specific "crossing" compressed video recordings that the user performs while recording the game segment. When the user page is being viewed and all thumbnails and video windows are constantly looping, the 3D dvr self-reward clip 2003 will cause the user to record the self-reward clip 2003 of the compressed video when the user records the "travel" of the game segment. Constantly circulate. However, when the user selects the 3D DVR self-reward clip 2003 and clicks the 3D DVR self-reward clip 2〇〇3, the user will be able to click to give it in addition to the DVR control that allows the playback of the compressed video self-reward clip. Button for 3D DVR capabilities of game clips. It will be able to independently control the camera "crossing" during the game segment, and if it wishes (and the user of the user page so allows), it will be able to record the alternate self-reward clip in the form of compressed video. The alternative self-reward clip "crossing" will then be available to other viewers of the user's page (immediately, or after the owner of the user's page has I36720.doc •125· 200941233 with the opportunity to verify the clip ). This 3D DVR Self-Camp Clip 2003 capability is enabled by launching a game that will replay the recorded game state information on another application/game server 152 1525. Since the game can be launched almost instantaneously (as previously described), it is activated (its playback is limited to the game state recorded by the self-reward clip) and then the user is allowed to record the video to the delay buffer 1515. It is not difficult to use the camera to "cross" at the same time. Once the user completes the "crossing", the game is deactivated. © From the user's point of view, launching the “crossing” with the 3D DVR Self-Camp Editing 2003 is no more difficult than controlling the Dvr control of the linear self-reward clip 2〇〇3. It does not know the game or even know how to play the game. It is only a virtual camera operator staring at the 3D world during the game segment recorded by another operator. The user will also be able to add their own audio to the self-reward clip (or record or upload from the microphone). In this way, a self-reward clip can be used to generate custom animations using characters and actions from the game. This dynamic production technique is often referred to as "machinima". As the user progresses through the game, they will reach different skill levels. The played game reports the results to the service control system 4 and these skill levels will also be displayed on the user page. Interactive animated ads Online ads have evolved from text to still images, video, and now to interactive snippets, often implemented using animated, thin-clients like Adobe Flash. The reason for using the thin client is that the user 136720.doc -126- 200941233 is delayed and less impatient with the privilege of selling the product & or service. Also, the thin client is executed on a very low performance PC, and therefore advertisers can have a high level of confidence: interactive ads will work properly. The legacy, such as Adobe Flash's animated streamlined client, is limited in the degree of interactivity and the duration of the experience (to reduce download time). Figure 21 illustrates an interactive advertisement in which the user selects the exterior and interior colors of the car as it rotates in the showroom, while the instant ray tracing shows how the car does not look. Then the user chooses the avatar to drive the car, © And then the user can use the car for driving on a race track or through a country other than M〇naco. The user can select a larger engine or a better tire and then see how the changed configuration affects the car's ability to accelerate or maintain stability. Of course, 'advertising is effective for cutting-edge 3D video games. However, for such advertisements that can be played on a PC or video game console, it will require 100 MB downloads, and in the case of a PC, it may require the installation of a special drive and may lack sufficient CPU or GPU computing power is not implemented at all. Therefore, such advertisements are impractical in prior art configurations. In the colocation service 210, the advertisements are placed almost immediately, and are preferably executed regardless of the capabilities of the user's client 415. As a result, it delivers more quickly than a streamlined, user-based interactive ad, with a richer experience and a higher level of reliability. Streaming geometry RAID arrays 15 11-15 12 and incoming routing 1502 can provide such fast data rates with such low latency during instant animations that the design is likely to depend on 136720.doc -127· 200941233 RAID Array 1511- 15 12 and the routing route 15〇2 to reliably deliver geometric video games and applications directly in the middle of the game play or in the application during live action (eg, traversal with complex databases). In prior art systems, such as the video game system shown in the figure, the large number of storage devices available (especially in practical home devices) is so slow that it is not possible to stream geometry during game play (except for the required In addition to the slightly predictable geometry, for example, in a driving game where a designated road exists, the geometry of the building used to enter the field of view can be reasonably properly predicted and a large number of storage devices can be searched for in advance. The location where the geometry is located. But in complex scenes with unpredictable changes (for example, in a battle scene with complex characters around) 'If the RAM on the PC or video game system is completely filled for the current view The geometry of the object, and then the user suddenly turns his character to view his character, if the geometry is not preloaded into the RAM' then there may be a delay before the geometry can be displayed. In 210, the RAID arrays 1511-1512 can exceed the speed stream data of the same speed Ethernet speed. Under the san network, there is a speed of 10 billion bits per second that is better than 10 ten. 3⁄4 7L Ethernet or better than other network technologies. 1 billion bit / The second will load 1 billion bytes of data in less than a second. In the 60 fps frame time (16.67 milliseconds), you can load about 170 million bits (21 MB) of data. Of course, even in RAID In the configuration, the rotating media will still incur a delay greater than one frame time, but the flash memory based on the flash memory will eventually be larger than the rotating 136720.doc -128- 200941233 media RAID array and will not This high latency is incurred. In one embodiment, a very low latency access is provided using a cache of large RAM writes. Therefore, at a sufficiently high network speed and a large amount of memory with low enough latency, The geometry and the GPU can process the 3D data to quickly stream the geometry to the application/game server 152, 1525. Thus in the example given above, the user suddenly turns his character and Looking back, you can load everyone behind them before the character finishes spinning. The geometric shape of the object, and therefore, for the user, it seems that it is in a real-life photo-realistic world with live action. As discussed earlier, the last of photo-realistic computer animations One of the boundaries is the human face, and because the human eye is sensitive to incompleteness' The slightest error from a photo-realistic face can lead to negative reactions from the viewer. Figure 22 shows the use of ContourTM authenticity capture. Technology (the following is the subject of the application in the application: No. 10/942,609, "Apparatus and method for capturing the ❿ motion of a performer"; September 10, 2004 /942,413 "Apparatus and method for capturing the expression of a performer"; February 25, 2005 曰 Application No. 11/066,954 r Apparatus and method for improving marker identification within a motion capture system"; March 10, 2005 Application No. 11/077,628 "Apparatus and method for performing motion capture using shutter synchronization Application No. 11/255,854, October 20, 2005, "Apparatus and 136720.doc -129- 200941233 method for performing motion capture using a random pattern on capture surfaces"; June 7, 2006 曰 Application No. 11/449 "System and method for performing motion capture using phosphor application techniques", No. 11/449,043, "System and method for performing motion capture by strobing a fluorescent lamp"; June 7, 2006 Japanese Application No. 1/449, 127 "System and method for three dimensional capture of stop-motion 〇animated characters", each of which has been captured by the assignee of this CIP application How the live broadcast performance results in a very smooth capture surface that achieves a high polygon count tracking surface (i.e., the polygon motion accurately follows the movement of the face). Finally, photo-realistic results are produced when the live broadcast of the live video is mapped onto the tracking surface to create a textured surface. While current GPU technology is capable of rendering many polygons in the tracking surface and texture and illuminating the surface in real time, if each of the polygons and textures changes between the frames (which will produce the most photorealistic results), then All available RAM for modern PC or video game consoles will be quickly consumed. Using the stream geometry technique described above, the geometry is continuously fed into the application/game server 1521-1525 so that it can continually animate a photo-realistic face to allow for almost indistinguishable The video game of the facial face of the live action becomes practical. Integration of Linear Content and Interactive Features 136720.doc -130- 200941233 Movies, TV shows and audio materials (collectively "Linear Content") are widely available in many forms for home and office users. Linear content is available on physical media such as CD, DVD, HD_DVD, and Blu-ray media. It can also be recorded by DVRs from satellite and cable broadcasts. In addition, it can be available via pay-as-you-go (ppv) content on satellite and TV, and video on demand (VOD) on the power meter τν. Increasingly linear content is available via the Internet for downloading content and strings > available. Today, there really is nowhere – you can experience the location of all the features associated with linear media. For example, DVDs and other video optical media typically have interactive features that are not available at other locations (such as director reviews, "tact" clips, etc.). Online music sites have cover art and song information that are not available on 匸^^, but not all CDs are available. And websites associated with television shows often have additional features, blogs, and sometimes comments from actors or creators. In addition, in the case of many movies or sports events, there are often issues that are often issued along with linear media (in the case of a movie) or (in the case of sports) can be closely linked to real-world events (eg, player transactions). Video game. The colocation service 210 is well suited for delivering linear content when linking disparate forms of related content together. Indeed, delivering a movie movie is less challenging than delivering a highly interactive video game, and the colocation service 210 can deliver linear content to a variety of devices in a home or office, or to a mobile device. Figure 23 shows an exemplary user interface page for a colocation service 21' which shows the choice of linear content. 136720.doc -131- 200941233 However, unlike most linear content delivery systems, the colocation service 210 is also capable of delivering relevant interactive components (eg, menus and features on DVDs, interactive overlays on HD-DVDs). And the Adobe Flash animation on the website (as explained below)). Therefore, the client device 415 limits no restrictions on which features are available. In addition, the colocation system 210 is capable of dynamically and instantly linking linear content to video game content. For example, if a user is watching a Quidditch game in a Harry Potter movie and decides that he is willing to try to play Quldditcl1, it can simply click the button and the movie will pause and it will be immediately delivered to Harry Potter. The Quidditeh clip of the video game. playing

After the Quidditch game, another click on the button and the movie will start again. In the case of photo-realistic graphics and production techniques, the video captured by the camera cannot be distinguished from the live action characters, when the user performs the video from the Quidditch game in the live action movie to the colocation service. When the Quidditch game in the game changes (as described in this article), the two scenes can't actually be distinguished. This is a new creative option for directors of both linear content and interactive (eg video game) content, as the lines between the two worlds become indistinguishable. Using the colocation service architecture shown in Figure 14, the control of the virtual camera in the 3D movie can be provided to the viewer. For example, in a scenario that occurs within a train, it will be possible to allow the viewer to control the virtual camera as the story progresses and to take care of the train. This assumes that all 3D objects in the train ("assets") and enough 136720.doc -132-200941233 computing power levels are available to instantly reproduce these scenes as well as the original movie. And even for non-computer-generated entertainment, there are very stimulating interactive features that can be provided. In terms of climbing, the 2〇〇5 film “Pride and Prejudice” has many scenes in the beautifully decorated old British mansion. For a particular building scene, the user can pause the video and then control the camera to patrol the building, or possibly the surrounding area. To accomplish this, a camera with a fisheye lens can be carried through the building 'when it tracks its position' very similar to implementing the prior art eight-company QuickTime VR. The various frames will then be converted, since the I image is not distorted, and then it is stored on arrays 1511 - 1512 along with the movie and is played back when the user chooses to continue the virtual tour. In the case of a sporting event, a live event (e.g., a basketball game) can be streamed via the colocation service 21 for viewing by the user (as it would be for a typical TV). After the user watches a particular play, the video game of the game (finally the basketball player looks realistic with the real player-like photo) can catch up with the player starting in the same position, and the user (possibly each (four) one player) can be re-played Play to see if it can do better than these players. The colocation service 210 described in this article is extremely suitable for supporting this future world because it can withstand unrealisticity and cannot be installed in the home or most office backgrounds. Computational power and large-capacity storage resources, and its computing resources are always up-to-date (in the case of the latest computing hardware available)' but in the family background, there will always be older generations of pi and video The family of the game. In addition, in the host hosting service 2 γ, the use of 136720.doc • 133· 200941233 is concealed of all this computational complexity, so even if the user may be using a very sophisticated system, from the user's point of view, Changing the channel on TV is generally simple. In addition, the user will be able to access all of the computing power and computing power that will result from any client 415. Multiplayer games to the extent that the game is multiplayer, then it will be able to communicate not only via the 1502 network to the application/game server 152b 1525 but also by borrowing

Communicated by the network bridge to an internet (not shown) having a server or gaming machine that is not executing in the colocation service 21〇. When playing multiplayer games on a computer on a universal Internet, the application/game server 1521-1525 will have the benefit of extremely fast access to the Internet (when the game is executed on a server in the home) In comparison, but it will be limited by the capabilities of other computers playing games on slower connections, and potentially by the game servers on the Internet to be designed to accommodate the least commonalities (the game servers will be relative) The factual limitations of slow consumer Internet connections on home computers). However, when playing a multiplayer game entirely in the colocation service 21 server center, a great difference can be achieved. Hosting the game for the user's game - app/game feeding n 1521_1525 will be connected to other apps/games server 1 521 _ 1 525 and by extremely high speed, very low latency connectivity and huge, very The Fast Storage Array Host hosts any feeder connected to the central control of the multiplayer game. For example, if an ultra-high-speed Ethernet network is used to route the 1502 network, the application/game feeders 1521-1525 will be communicated to each other and communicated to potentials at a billionth of a second. Only ι 136720.doc • 134· 200941233 milliseconds or less than 1 millisecond delay under the host to host any server for central control of multiplayer games. In addition, 11 8 11) arrays 1511_1512 will be able to respond very quickly and then transfer data at a rate of one billion bits per second. As an example 'If the user customizes the character in terms of appearance and clothing so that the character has a large number of geometric shapes and behaviors that are unique to the character, in front of the game client that is limited to execution on the PC or game console in the home Under the technical system, if a character enters the field of view of another user, the user will have to wait until a long slow download is completed in order to load all geometry® shape and behavior data into their computer. Within the colocation service 210, the same download can be superior to the ultra-high speed Ethernet from the rAID array 1511_1512 at a rate of one billion bits per second. Even if the home user has an 8 Mbps Internet connection (which is extremely fast based on today's standards), the ultra-fast Ethernet network is 100 times faster. Therefore, it takes less than a second to spend a minute on a fast internet connection on a super-fast Ethernet. Top Player Grouping and Koi Race @ Host Escrow Service 210 is perfect for tournaments. Since no game is executed in the local client, there is no chance for the user to cheat. Again, due to the ability of the output route 1540 to multicast UDP streams, the colocation service 210 can simultaneously broadcast larger tournaments to thousands of people in the audience. In fact, when there are so popular that thousands of users are receiving the same stream of specific video streams (such as 'showing a larger tournament view'), the video stream is sent to a content delivery network (CDN) (such as Akamai). Or Limelight) may be more efficient for distribution to a large number of client devices 415. 136720.doc 135· 200941233 When using a CDN to display a top-level player grouping game viewfinder page, a similar level of efficiency can be obtained. For larger tournaments, live celebrity commentators can be used to provide comments during the special event. Although a large number of users will be watching the larger championships and a relatively small number will be played in the tournament. The audio from the celebrity commentator can be routed to the application/game server 1521-1525 that hosts the host in the tournament and hosts any of the bystander mode copies of the game in the tournament, and the audio can be added Recorded on the game audio. The video of the celebrity commentator can be overlaid on the game (and possibly just on the bystander view). The main transport protocol and hypertext transfer protocol (Ηττρ) of the webpage loading acceleration global information network is conceived and defined in an era in which only commercial high-speed Internet connections are used, and online consumers use dial-up data machines or ISDN. in. At this point, the "gold standard" for fast connections is the Ding Hao line, which symmetrically provides a 15 Mbps data rate (ie, equal data rates in both directions). Today, the situation is completely different. The average home connection speed through a DSL or cable modem in a large number of developed worlds has a much higher downlink data rate than the Dingsian line. In fact, in some parts of the world, fiber-to-the-curb is introducing data rates of up to 50 Mbps to 1 Mbps into the home. Unfortunately, HTTP has not been architected (and not implemented) to make effective use of these rapid speed improvements. The website is a collection of files on the remote server. Quite simply, 'HTTP requests the first slot, waiting to download the file, and then 136720.doc • 136· 200941233 θ bundle first slot' waiting to download the broadcast. In fact, HTTP allows for open connections (that is, each time more than one file is requested), and only the agreed-up # standard (and the desire to prevent the web server from being overloaded) allows only very open connections. In addition, because web pages are constructed in a way that browsers are often unaware of multiple simultaneous pages that can be used for immediate downloading (i.e., it becomes apparent only after parsing a page: a new slot such as an image needs to be downloaded). Therefore, the broadcasts on the website are essentially loaded one by one. In addition, due to the request and response protocol used by HTTP, there is a delay of approximately 1 〇 milliseconds associated with each file loaded (accessing a typical US network server). In the case of a relatively low-speed connection, this is not the (four) NPC (four) question, because: the download time of the file itself determines the waiting time of the web page. However, as the connection speed increases (especially in the case of complex web pages), the question arises. In the example shown in Figure 24, a typical commercial website (this particular ❹(4) is from a larger athletic shoe trademark) is shown. There are M slots on the website. The draft includes HTML, CSS, (4), Acid, JavaSedpiF丨_ Slots, and includes video content. Before the live web page (that is, the user can click on it and start using it), a total of i 5 Μ bytes must be carried. There are many reasons for a large number of tenders. First, it is a complex and sophisticated web page, and secondly, it is a web page that is dynamically combined based on information about the user accessing the page (eg, 'Which country does the user come from, what language, whether the user previously made a purchase Etc.) and download different slots depending on all of these factors. However, it is still a very typical commercial web page. 136720.doc -137- 200941233 Figure 24 shows the amount of time that elapses before the live web page is live as the connection speed increases. At a connection speed of 2.4 Mbps at 1.5 Mbps, a conventional network feeder with a conventional web browser is used to take a shift before the live web page is broadcast. At 12 Mbps connection speed 2402, the load time is reduced to 65 seconds, or approximately doubled. However, at a connection speed of 2403 at 96 Mbps, the load time is reduced to only about 5.5 seconds. The reason for this is that at such high download speeds, the time to download the files themselves is minimal, but the delay of approximately 1 millisecond per file is maintained, resulting in a delay of 54 files * 1 〇〇 milliseconds = 54 seconds. ® So, no matter how fast the connection to the home, this site will always take at least 5.4 seconds before the live broadcast. Another factor is that the server side is queued up; each HTTP π request is added after the queue, so on a busy word server, this will have a significant impact because for each to be obtained from the web server A small project 'HTTP request needs to wait for it to return. One way to solve these problems is to discard or redefine Ηττρ. Alternatively, it may be possible for site owners to merge their files into a single file (for example, in Adobe Flash format). But as a practical matter, this division and many others have invested heavily in their website architecture. In addition, there are 12-100 Mbps connections, but most families still have slower speeds, and HTTP does work well at slow speeds. An alternative is to host the web browser host on the application/game server 1521-1525 and host the file host for the web server on the RAID array 1511_1512 (or potentially host the host) Hosting the web browser application/game server 1521_1525 in RAM or on the local storage). Due to the route through the port 15〇2 (or to the local storage 136720.doc -138· 200941233

The very fast interconnection of the device is not a delay of loo milliseconds per file under HTTP, but a minimum delay per file under HTTP. Then, instead of allowing the user in the home to access the webpage via HTTP, the user can access the webpage β via the client 415, and then even under the 15 Mbps connection (because the webpage does not require a large amount of bandwidth for its videoconferencing) ), the webpage will also be live broadcast in less than 1 second per line of 2400. In essence, there will be no delay before the web browser executing on the application/game server 1521_1525 displays the live broadcast page, and there will be no detectable before the user terminal 415 displays the video output from the web browser. The delay to arrive. When the user uses the mouse to search the webpage and/or type a word on the webpage, the user's input information is sent to the web browser executed on the application/game server 1521-1 525, and the web browser A response will be responded accordingly. The disadvantage of this method is that the money is being transmitted (4) to the video material, and the bandwidth is used, even if the web page becomes static. This can be remedied by configuring Compression II to transmit data only when (and if) the web page changes and then only transmit the data to the portion of the page where the change occurred. When there are some web pages with a constantly changing flash slogan, etc., the web pages tend to be annoying, and unless there is a reason to move something (eg, a video clip): then: the regular web page is static of. For these webpages, it may be possible to transmit less data for the following two services (there is no need to stream the actual displayed image, no simpler to flip the image) Large objects that are never viewed (for example, Roll 136720.doc -139- 200941233 Therefore, using the colocation service 210 to host the old web pages, the web page loading time can be reduced to open the web page similar to changing the TV. The degree of the channel: effectively and instantly broadcast the webpage lively. Promoting the debugging of games and applications as previously described, 'video games and applications with instant graphics are very complex applications and usually when they are released to the field. In the middle, it contains defects. Although the software developer will get feedback on the defect from the user, and it may have some © way to return the machine state after the crash, but exactly identify what caused the game or instant It is very difficult for an application to crash or not perform properly. When a game or application is executed in the colocation service 21, the game or should The video/audio output of the program is constantly recorded on the delay buffer 1515. In addition, the watchdog process executes each application/game server 1 52 1 -1 525 'The watchdog process will host the service control to the host System 401 periodically reports that application/game server ι521_1525 is executing smoothly. If the watchdog process fails to report, server control system 401 will attempt to communicate with application/game server 1521_1525, and if successful, No matter what machine state is used for the *T. Any information that is available is sent to the software developer along with the video/audio recorded by the delay buffer 1515. Therefore, when the game or application software developer self-hosting service 210 When notified of the crash, it gets a record of the frame that caused the crash. This information can be extremely valuable in tracking down defects and fixing them. 136720.doc •140- 200941233 Also note 'when the application / Game server 152 1-1 525 crashes, restarts the server at the most recent restartable time, and provides the message to the user Apologize for technical difficulties. Resource Sharing and Cost Savings The systems shown in Figures 4a and 4b provide multiple benefits for both end users and game and application developers. For example, typically, home and office clients The system (for example, a PC or game console) is only in use for a small percentage of the week. According to the Nieisen Entertainment "Live Jump Player Benchmark Study" (http://www.prnewswire.com/cgi - bin/StOrieS.Pl?ACCT=104&STORY=/www/story/10_05·2006/ 〇〇〇44461 15&EDATE=) 2nd, 6th, 1st, 5th, 5th, communication, active gamers average weekly It takes 14 hours to play on the video game console and play on the handheld device for about 17 hours a week. The report also states that active gamers average 13 hours a week for all game play activities, including consoles, handheld devices, and PC game play. Considering the higher number of consoles, the game play time is 24*7 = 168 hours a week, and he suggests that in the active gamer's home, the video game console is only at -17/168 = 10% 2 hours. Medium in use. Or, at 9G% of the time, the video game console is idle. Given the high cost of video game consoles, and the ability to manufacture these devices, this is a very inefficient use of expensive resources. PCs in the industry usually only use non-portable PCs that are often used in high-end applications, such as Autodesk Maya, in one-hours per week. / 杳 s - some businesses operate at all hours and holidays and some r L (for example, .« ( t_x ^ α 豕 for portable Pc working at night) is at 136720.doc -141 - 200941233 ::2: Use of rice cookers 'but most commercial activities tend to be in a given business', A concentrates on about 9 Am to 5 周一 from Monday to Friday

For example, lunch) 'and because most jobs are used: when Pc is actively used', it follows: desktop PCs use the slant to follow these hours of operation. If it is assumed that the five days of the week from 9 AM to 5 PM use PC 1 strangely, it will imply that the PC is utilized in 24 hours of the week. High-performance desktops are very expensive investments for business, and this reflects very low utilization. Schools that teach on a desktop can use the computer for a week or even less and although it varies depending on the number of hours taught, most of the teaching occurs during the hours of the week-week. Therefore, in general, pc and video game consoles are only used during the hour of the week. Notably, because many people work in business or at school during the non-holiday week-to-week hour, these people usually don’t play video games during these hours, and therefore #真真玩视频When playing, it is usually during other hours (such as evenings, weekends, and holidays). Given the configuration of the colocation service shown in Figure 4a, the usage patterns described in the two paragraphs above result in (4) non-effectiveness (4). It will be apparent that there are limits to the number of users that can be serviced by the colocation service 210 at a given time, especially if the user desires immediate responsiveness for complex applications, such as sophisticated 3D video games. However, unlike video game consoles in the home or PCs that are used commercially (which are typically placed most of the time), the server 4〇2 can be reused by different users at the same time. For example, a high-performance dual cpu and dual Gpu 136720.doc -142- 200941233 and a large amount of RAM high-performance server 4〇2 can be utilized by commercial and school in non-false 9 9 AM to 5 PM, but by the cutting edge Video game players are used at night, on weekends and on holidays. Similarly, low-performance applications can be utilized by businesses and schools during the business hours on low-performance servers 402 with Celer〇n cpu, no Gpu (or very low-end GPU) and limited rams. Using a low-performance server during business hours 4 〇 2 ^ In addition, in the case of the colocation service configuration described herein, resources are effectively shared among thousands (if not millions) of users. In general, online services only use a small percentage of their total user base to use the service at a given time. If you consider the Nielsen video game usage statistics listed previously, it's easy to understand why. If the active player plays the console game for only 17 hours a week, and if the peak usage time of the game is assumed to be at night (5-12 AM, 7*5 days = 35 hours/week) and weekend (8 AM-12 AM, During a typical non-working, non-commercial hour of 16*2=32 hours/week, there is 35+32=65 peak hours for a 17-hour game. It is difficult to estimate the exact peak user load on the system for a number of reasons: some users will play during the extra-peak time, there may be a peak time in the user's cluster, and the peak time may be affected by the game being played. Types (for example, children's games will probably play at an earlier time in the evening) and other effects. However, assuming that when the player is likely to play the game, the average number of hours played by the player is much less than the number of hours between the players, then only the number of users of the colocation service 210 will use the host generation at a given time. Tube service 210. For this analysis, the peak load will be assumed to be ns%. Therefore, only 12.5% of the computing, compression and bandwidth resources are used to support 136720.doc • 143· 200941233 at a given time, resulting in only 2.5% of hardware cost to support a given user due to resource reuse. Play one of the performance games given levels. In addition, assuming that some games and applications require more computing power than others, then resources can be dynamically allocated based on the game being played by the user or by the application executed by the user. Therefore, users who choose low-performance games or applications will be assigned low-performance (lower-cost) servers 4〇2, and users who choose high-performance games or applications will be assigned high-performance (more expensive) feeders. 402. In fact, a given game or application may have a lower performance and higher performance area of the game or application, and the user can switch from one server 402 to another between games or applications. 402 ' is performed on the lowest cost server 402 that keeps the user in need of a game or application. Note that RAID array 405, which is much faster than a single disk, will be used by even low performance server 402, which has the benefit of faster disk transfer rates. Thus, the average cost per feeder 402 across all played games or applications used is much less than the cost of playing most of the expensive servers 402 of the highest performance game or application, however, even the low performance server 402 Disk performance benefits will also be obtained from RAID array 405. In addition, the server 402 in the colocation service 210 may be only a PC motherboard that does not have a disk or peripheral interface (different from the network interface), and just can be integrated down to a fast network just to the SAN 403. A single wafer of the interface. Also, the RAID array 405 may be shared among a much larger number of users than in the presence of a disk. Thus, the cost per disk of each active user will be much less than that of a disk drive. All of this device will likely reside in a rack in the environment of the server room controlled by 136720.doc 200941233. If the server 402 fails, it can be easily repaired or replaced at the host escrow service 21 。. In contrast, a pc or game console in a home or office must be strong enough to be spared from reasonable wear and tear to prevent a single appliance from being hit or dropped, requiring at least one drive, and must be spared. Under adverse environmental conditions (for example, being forced into a heated AV cabinet with other appliances), a service guarantee is required 'must be packaged and shipped, and sold by a retailer that may receive retail profits. In addition, the pC or game console must be configured to meet the peak performance of the most computationally intensive game or application that will be used at some point in the future, even for lower performance games or applications (or games or applications). It is also possible to play most of the time. In addition, if a pc or console fails, it can be a costly and time consuming process (which adversely affects manufacturers, users, and software developers). Therefore, it is assumed that the system shown in Figure 4a provides the user with an experience equivalent to the experience of the local computing resource for the user to experience the computing power in the home, office or school, via Figure 4a. The architecture shown in the ® provides a much lower computational power. Eliminate the need for upgrades. ‘Users no longer have to worry about upgrading PCs and/or consoles to play new games or handle new applications with higher performance. Any game or application on the colocation service 210 (regardless of the type of server 402 that the game or application requires) can be used by the user and all games and applications are executed in close proximity (ie, fast) Loaded from the RAID array 405 or the local storage on the server 402) and suitably with the latest updates and defects 136720.doc -145- 200941233 fix (ie, the software developer will be able to choose to execute the given game) Or the ideal server configuration of the application server 402, and then the server 402 is configured with the best driver, and then over time, the developer will be able to provide updates, bug fixes, etc. to the host generation over time. Manage all copies of the game or application in service 210). In fact, after the user starts using the colocation service 21, the user may find that the game and the application continue to provide a better experience (for example, via update and/or defect repair) and may be the following: user one year Later, it was discovered that new games or applications can be used to take advantage of the computing technology (for example, a higher-performance GPU) (which does not even exist in a year), so it is impossible for the user to purchase A year ago, the technology to play games or execute applications a year later. Because the computing resources for playing games or executing applications are invisible to the user (i.e., from the user's point of view, the user only chooses a game or application that is close to being executed immediately).

The channel on the TV), so the user's hardware will be "upgraded" even if the user does not even know about the upgrade. Eliminating the need for backups Another big problem for users in schools and homes is for commercial use. If the disk fails, or if there is an inadvertent erasure, the information stored in the local PC or video game console (for example, in the case of the console, the user's game results and ratings) may be lost. There are many dependencies available for manual or automatic backup, and the game console status can be uploaded to an online server for backup to a safe and organized one, but usually the local backup Re-addressing another local disk (or 136720.doc • 146-200941233 eight non-volatile storage devices), and often makes backups for online services due to the slow upstream speed available via typical low-cost internet connections limited. In the hosted service 21 of Figure 4a, the data stored in the rAID array can be configured using prior art RAID configuration techniques well known to those skilled in the art, such that when the disk fails, The material is not lost and the technician at the server center hosting the failed disk will be notified, and then the technician will replace the disk, which will then be automatically updated to cause the RAID array to tolerate the failure again. In addition, because the drives are close to each other with a fast local network via the SAN 403, all disk systems are configured to be periodically backed up to secondary storage in the server center (which can be stored in It is not difficult to relocate at the center of the server or easily. From the point of view of the hosted service user, the data is always completely secure and never has to be considered for backup. Access to the presentation Users often want to try out a game or application before purchasing a game or application. As previously described, there are demonstrations (the "verb" form means a trial demo version, and the demo version is also called a "demo", but as a noun) a prior art component of a game and application, but each of them Subject to restrictions and/or inconvenience, the use of colocation services 21 is easy and convenient for the user to try out the demo. In effect, the user performs the presentation and trials the presentation via a user interface, such as the user interface described below. The presentation will load almost instantly onto the server 402 suitable for the presentation, and it will execute exactly like any other game or application. Regardless of whether the presentation requires a very efficient server 136720.doc -147- 200941233 402 is still a low-performance server 402, and regardless of the type of home or office client used by the user, from the perspective of the user, the presentation Will work. A software publisher of a game demo or application demo will be able to precisely control how long the demo and trials are allowed for the user to try, and of course, the demo may include providing the user with a full version of the game or application being demonstrated. The user interface element of the access opportunity. Because the presentation may be below cost or free of charge, some users may try to use repeated presentations (especially repeating the potentially interesting ® game demos). The colocation service 210 can use various techniques to limit the presentation usage for a given user. The most straightforward method is to establish a user ID for each user and limit the number of times the given user id is allowed to play the presentation. However, the user can set multiple user IDs, especially if they are free. One technique for solving this problem is to limit the number of times a given user 41 5 can play a presentation. If the client is a standalone device, the device will have a sequence number' and the colocation service 210 can limit the number of times the presentation can be accessed by the client with the serial number. If the client 415 is executing on a PC or other device's software, then the host can be assigned a serial number by the host server and the serial number is stored on the PC and the serial number is used to limit the presentation usage, but the PC is assumed to be available to the user. Reprogramming, and the sequence number is erased or changed, then another option is that the colocation service 210 maintains the PC network adapter media access control (MAC) address (and/or other machine specific identifiers, such as hard Record the number of the disc player, etc.) and limit the use of the demo to the MAC address. It is assumed that the MAC address of the network adapter can be changed, however, this is not an extremely simple method. Another method is to limit the number of times a presentation can be played to a given IP address. 136720.doc -148- 200941233 Although the IP address can be reassigned periodically by cable modems and DSL providers, it does not occur very frequently in practice, and if θ can be determined (eg, by contacting isP) The ip system is located in a block of IP addresses for residential DSL or cable data access, and typically a small number of presentations for a given home can be established. Also, there may be multiple devices in the home after a NAT router sharing the same IP address, but typically in a residential setting, there will be a limited number of such devices. If the IP address is in the servo commercial block, a larger number of presentations for the business can be established. However, after ^ ©, all combinations of the previously described methods are the best way to limit the number of presentations on the PC. While there may be no very simple way to make the determined and technically skilled user likely to be limited in the number of replayed presentations, creating a large number of obstacles can create enough obstacles to make the majority? 〇 Users are not worth the trouble to use the demo system, and instead, they use the demo when they want to try out new games and applications. Benefits to School, Business, and Other Structures Significant benefits especially occur in the commercial use of the system shown in Figure 4a,

Schools and other institutions. Businesses and schools have substantial costs associated with installing, maintaining, and upgrading PCs, especially when it comes to implementing PCs such as Maya's high-performance applications. As stated previously, PCs are typically only used in the hour-to-week rate of 77. And in the home, the cost in an office or school environment with the ability to position the quasi-performance is much higher than in the center of the ship. In the case of larger businesses or schools (eg, large universities), the IT department of these entities sets up the server center and maintains the electronic version accessed via the LAN-level connection and the remote 136720.doc • 149-200941233. It is actual. There are many solutions for remote access to a computer via a LAN or via a private high frequency wide connection between offices. For example, 'through Microsoft's Windows terminal server, or through a virtual network e-application (such as VNC from ReaivNc Ltd) or through a thin client-side component from Sun Microsystems, users can Get remote access to a PC or server with a range of qualities in graphical response time and user experience. In addition, these self-managed server centers are typically dedicated to a single business or school, and therefore cannot be used to exploit the same computing resources at different times of the week in disparate applications (eg, entertainment and business applications). The overlap of use. Therefore, many businesses and schools lack the scale, resources, or expertise to independently set up a server center with a network connection to each user's LAN speed. In fact, a large percentage of schools and businesses have the same Internet connections as homes (eg, DSL, cable modems). However, such organizations may still have the need for very high performance calculations (either periodically or periodically). For example, a small construction company may have only a small number of architects, with relatively modest computing needs when designing, but it may periodically require very efficient 3D calculations (for example, when building for users) When the buckle of the new architectural design is crossed,). The system shown in Figure 4a is extremely suitable for such organizations. These organizations only need to provide the same type of network connection (e.g., DSL, cable modem) to the home and are typically very inexpensive. It can be used at low cost.乍^Client 415, or no Pc at all, and using the low-cost dedicated device 136720.doc 200941233 which only implements control signal logic 413 and low-latency video decompression 412. These features may be stolen or against pc. Schools with problems with damaged special components are particularly attractive. This configuration solves many of the problems for such organizations (and many of these advantages are also shared by home users who perform general purpose computing). For example, 'cost (which ultimately must be passed back to the user in some form to have a viable business) may be much lower because (4) the computing resource is shared with other applications that have different peak usage times during the week, (8) These organizations may obtain (and incur costs) access to high-performance computing resources only when needed' (4) such organizations do not have to provide resources for backing up or otherwise maintaining high-performance computing resources. Elimination of piracy In addition, games, applications, interactive movies, etc. may no longer be pirated as they are today. Since the game is executed at the server center, the user does not have access to the basic code, so there is no piracy. Even if the user is going to copy the source code, the user cannot execute the code on a standard game console or home computer. This opens up markets around the world where standard video games are not available (such as 'China'). Resale of games that have already been used is also impossible. For game developers, there are fewer market discontinuities as they are today. With the new generation of technology forcing users and developers to upgrade and the game developers are dependent on the current situation of the timely delivery of the hardware platform. 'The escrow service can be gradually updated as the game requirements change over time. 2 1 0 . Streaming Interactive Video 136720.doc • 151 - 200941233 The above description provides low latency _ stream interactive video based on the Universal Internet (which also implicitly includes audio along with video, as used herein) The novel basic concepts enable multiple applications. Prior art systems that provide streaming video over the Internet have only those applications that can be implemented by high latency interactions. For example, basic playback controls (e.g., pause, rewind, fast forward) for linear video work properly at high latency, and it is possible to select among the online video feeds. Moreover, as previously stated, the nature of some video games allows them to be played with high latency. However, the high latency (or low compression ratio) of the prior art method for streaming video severely limits the potential application of streaming video or narrows its deployment to a specialized network environment, and even in such environments. The prior art also introduced a substantial burden on the Internet. The techniques described in this article open the door to a variety of applications that may be possible under low-latency, slaughter interactive video over the Internet, especially when enabled via consumer-grade Internet connectivity. Client device

In fact, with the use of Figure 4C, it is sufficient to provide enhanced by any amount of effective force, any amount of fast storage, and extremely fast network connection between powerful servers. The user experience 'enables a new era of computing. Others from the ^ other 'because the bandwidth requirements do not increase with the computing power of the system θ (four) growth (that is, because the bandwidth requirements are only about display resolution, quality and frame rate), so the network The road connectivity is ubiquitous. The problem is that typical wireless consumers (9) will be able to pass through a widely distributed low-latency wireless phantom, and (4) and have a sufficiently high frequency I for the needs of the W-foot 422. 136720.doc -152- 200941233 What is necessary for business applications is a thick client (such as performing bias,

LinuX, 〇SX, etc. PC or mobile phone) is even a thin client (such as Adobe Flash or Java). The emergence of streaming interactive video has led to a reconsideration of assumptions about the structure of the computing architecture. This is an example of a colocation service 210 server center embodiment shown in FIG. For the delay buffer buffer and / or group video 1550 video path feedback loop, where the application / game word processor Ο

The multicast streaming video output of the 152U525 is routed back to the application/game cage 521_1525 via the path or via the path 1551 after a selectable delay. This can be achieved by a variety of practical applications (such as those described in Figures 6 and 17 and Figure 2) that are not possible or feasible by the prior art word processor or the local computing architecture. The more general architectural feature, the feedback loop 155〇 provides recursiveness for streaming interactive video levels' because the video can be looped indefinitely when the application needs video. There are many application possibilities that have never been available before. Another key architectural feature is that the video stream is unidirectional 1;] 〇 1 > stream. This effectively results in any degree of multicasting of streaming interactive video (compared to a two-way stream such as TCP/IP streaming that will increase over the network from back and forth as the number of users increases) The traffic is stagnant). Multicast is an important force in the servo 17 because it allows the system to respond to the growing need of Internet users (and indeed the world's population) to communicate on a one-to-many or even many-to-many basis. Again, the examples discussed in this article that illustrate the use of both interactive video recursion and multicast (such as Figure 136720.doc -153 - 200941233 16) are only the central end of the very large iceberg. In one embodiment, the various functional modules and associated steps described herein may be comprised of specific hardware components (such as 'special application integrated circuits ("ASICs) that are used to perform the fixed-line logic of the steps. ")) or by any combination of stylized computer components and custom hardware components. In an embodiment, the modules can be implemented in a TMS320x architecture such as the Texas Instruments (e.g., 'TMS32〇C6〇〇〇, tmS32()C5(9)〇,

❹, etc.) can be programmed on a digital signal processor ("DSp"). A variety of different DSPs can be used while still adhering to these basic principles. Embodiments may include various steps as set forth above. These steps may be embodied in a machine executable finger 7 that causes a general purpose or special purpose processor to perform particular steps. Various components (such as computers, camcorders, hard drives, input devices) that are not related to these basic principles have been omitted from the figure to avoid mixing the relevant aspects. The elements of the subject matter disclosed may also be provided as a machine-readable medium for storing machine-executable instructions. A machine-readable medium can include, but is not limited to, a flash memory, a compact disc, a CD-ROM, a DVD R〇M, a RAM, an EPROM, an EEPR®, a magnetic or optical card, a broadcast medium, or other suitable for storing electronic instructions. A type of machine readable medium. For example, the present invention can be downloaded as a computer program that can be accessed from a remote computer via a communication link (eg, a data modem or a network connection) by means of a data signal embodied in a carrier wave or other communication medium ( For example, the server) is transferred to the requesting computer (eg, the client). It should also be understood that the elements of the subject matter disclosed may also be provided as computer program products I36720.doc -154- 200941233^, which may be included in the readable medium of the device, and the instructions may be private. The machine can be used to program a computer (for example, or other electronic devices) to perform a sequence of operations. It is carried out by the combination of hardware and enthalpy into the water. Machine readable medium 2) Soft disk, CD, and CD-ROM = ❹

: should, clerk, magnetic or optical card, _ body or other type of media/machine readable medium suitable for storing electronic instructions. For example, the elements of the disclosed subject matter can be downloaded as a computer program product, wherein the program can be transmitted via a communication key (eg, a data machine or a network connection) by means of a data signal embodied in a carrier wave or other communication medium. Transferred from the remote computer or electronic device to the request process. In addition, while the disclosed subject matter has been described in connection with the specific embodiments, numerous modifications and variations are Accordingly, the specification and drawings are to be regarded as illustrative rather BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 illustrates the architecture of a prior art video game system. 2a through 2b illustrate a high level system architecture in accordance with an embodiment. Figure 3 illustrates the actual, rated, and required data rate for communication between the client and the server. Figure 4a illustrates a colocation service and a client used in accordance with an embodiment. Figure 4b illustrates an exemplary delay associated with communication between the client and the colocation service. Figure 4c illustrates a client device in accordance with an embodiment. 136720.doc -155- 200941233 Figure 4d illustrates a client device in accordance with another embodiment. Figure 4e illustrates an example block diagram of the client device of Figure 4c. Figure 4f illustrates an example block diagram of the client device of Figure 4d. Figure 5 illustrates an example of video compression that may be used in accordance with an embodiment. Figure 6a illustrates one example of video compression that may be used in another embodiment. Figure 6b illustrates the peak in the rate of the resource associated with transmitting a low complexity, low motion video sequence. Figure 6c illustrates the peak in the data rate associated with transmitting a high complexity, high motion video sequence. Figures 7a through 7b illustrate an example video compression technique used in an embodiment. Figure 8 illustrates an additional example video compression technique used in an embodiment. Figures 9a through 9c illustrate example techniques for use in mitigating data rate peaks in an embodiment. Figures 10a through 1B illustrate an embodiment of efficiently packaging a block of image images within a packet. Figures 11a through 15 illustrate an embodiment using a forward error correction technique. Figure 12 illustrates an embodiment of compression using a multi-core processing unit. Figures 13a-13b illustrate geolocation and communication between colocation services, in accordance with various embodiments. Figure 14 illustrates an exemplary delay associated with communication between the client and the colocation service. 136720.doc • 156· 200941233 Figure 15 illustrates an example host hosting service server central architecture. Figure 16 illustrates an example screen shot of an embodiment of a user interface including a plurality of live video windows. Figure 17 illustrates the user interface of Figure 16 after selection of a particular video window. Figure 18 illustrates the user interface of Figure 17 after a particular video window has been enlarged to full screen size. Figure 19 illustrates example collaborative user video material overlaid on a multiplayer game screen.

Figure 20 illustrates an example user page for a game player on a colocation service. Figure 21 illustrates an example 3D interactive advertisement. Figure 22 illustrates an example sequence of steps for surface capture for a (four) live broadcast performance to produce a photorealistic image with a textured surface. Figure 23 illustrates an example user interface page that allows selection of & τ % in the linear media.

Figure 24 is a graph showing the amount of time elapsed before the live broadcast of the web page and the starting speed of the connection speed. [Main component symbol description] 100 101 102 103 104 105

CPU/GPU random access memory (RAM) display device (SDTV/HDTV or computer monitor) hard disk drive optical media drive / optical media Zhao / CD player network connection 136720.doc -157- 200941233 〇❹ 106 game control 205 Client Device/Client/Client Platform 206 Internet 210 Hosting Service/Hosting System 211 User Location 220 Game or Application Software Developer 221 Input Device 222 Display Device/Monitor or TV 301 nominal maximum data rate 302 available maximum data rate 303 required data rate 401 colocation service control system / colocation service control server 402 server 403 storage area network (SAN) 404 video compression logic / video compression 405 Independent Redundant Disk Array (RAID) 406 Control Signal 406a Control Signal 406b Control Signal 409 Routing Logic 410 Internet 412 Low Latency Video Decompression Logic 413 Control Signal Logic 136720.doc -158- 200941233 415 420 421 422 440 441 442 443 Ο 444 451 452

453 454 455 456 457 458 462 463 464 465 466 468 Client/Client Device/Customer Computer Display Device Input Device Display Device/Monitor Internet Bypass Service Presence WAN Interface Firewall/Router/NAT (Network Address Translation) Device WAN interface is used to send control signals for one-way transmission time. Round trip routing via user location / user location routing arrow / round trip delay / user ISP universal internet delay one-way routing maximum one-way delay arrow /Compressed Video Decompression Ethernet Hub/Internet Connection HDMI (High Definition Multimedia Interface), Connector Display Capability Client Device/User Side Panel Glasses

Display device / SDTV (standard definition TV) or HDTV 136720.doc -159- 200941233 〇❹ 469 (high definition television) / monitor input device 475 client device 476 flash memory 477 video camera 478 display device 479 blue Bud Input Device / Wired USB Input Device 480 Bus 481 Ethernet Interface 482 WiFi Subsystem 483 Control CPU / Control Processor 484 Bluetooth Interface 485 USB Subsystem 486 Video Decompressor 487 Video Output Subsystem 488 Audio Decompression Subsystem 489 Audio Output Subsystem 490 HDMI Interface 491 DVI-I 492 S-Video 493 Composite Video 494 Digital Interface 495 Stereo Analog Interface 497 Ethernet 136720.doc -160· 200941233 ❹

499 Power 501 Uncompressed Video Frame 502 Uncompressed Video Frame 503 Uncompressed Video Frame 511 Compressed Frame 512 Compressed Frame 513 Compressed Frame 520 Compression Logic 559-561 Uncompressed video frame 611 I frame 612-613 P frame 620 compression logic 621 rated maximum data rate 622 maximum data rate available / available data rate 623 I required peak data rate / peak data rate 624 video Stream data rate / video stream data rate sequence 633 I frame peak 634 video stream data rate sequence / video stream data rate 635 P frame peak 636 P frame peak 670 B frame 671 I frame 701-760 uncompressed Frame 711-770 R frame 136720.doc .161 · 200941233

721-770 Uncompressed frame 731-780 R frame 805 Mobile character 811 R frame 812 R frame 922 Available maximum data rate 934 Video stream data rate 941 Peak data rate / peak maximum data rate 942 2 times peak data Rate 943 3 times peak data rate 944 4 times peak data rate 952 frame 2 times peak 953 flattened 2 times peak 954 frame 4 times peak 955 flattened 4 times peak 961 uncompressed frame 1 962 without Compressed frame 2 963 Uncompressed frame 3 964 Uncompressed frame 4 965 Uncompressed frame 5 966 Uncompressed frame 6 967 Uncompressed frame 7 968 Uncompressed Box 8 969 Uncompressed frame 9 136720.doc 162- 200941233

970 uncompressed frame 10 981 compressed frame 1 982 compressed frame 2 983 compressed frame 3 985 compressed frame 5 986 compressed frame 6 987 compressed frame 10 991 Transmission time (xmit time) 992 Transmission time 993 Transmission time (2 times peak value) 995 Transmission time 996 Transmission time (4 times peak value) 997 Transmission time 1001-1005 Packet 1010 Image block package package logic 1100 I image block 1101 Forward error Correction Code (FEC) 1103 I Image Block 1104 Forward Error Correction Code (FEC) 1105 Forward Error Correction Code (FEC) 1110 Audio Packet 1111 FEC Code 1112 Audio Packet 1113 FEC Code 136720.doc -163- 200941233

1120 User input command 1121 FEC code 1122 User input command 1123 FEC code 1200 Multicore and / or multiprocessor 1201-1208 Core 1205 4x2 configuration / uncompressed frame 1300 Server Center / Hosting Service 1301 Network 1501 Internet access service 1502 Incoming routing/incoming routing network 1511 RAID array 1512 RAID array 1515 RAID array/delay buffer 1521-1525 Application/game server 1529 Uncompressed video/audio 1530 Video Compression/Common Set/Common Hardware Compression/Common Hardware Compressor/Common Video Compression Hardware 1539 Compressed Video/Audio Output 1540 Itch Routing/Output Routing Network 1550 Delay Buffer and/or Group Video/ Feedback loop 1551 after the road 1552 Path 1599 Itching Internet traffic 136720.doc 200941233

1600 "thumbnail" video window 1601 yellow selection box 1602 game information 1700 video window 1800 arrow 1801 over 1900 video and audio 2000 user's own video 2001 live video 2002 partner 2003 video clip / self-reward clip / 3DDVR self-reward Clip 2004 Digital 2005 Illustration 2400 Line 2401 1.5 Mbps Connection Speed 2402 12 Mbps Connection Speed 2403 96 Mbps Connection Speed HS1-HS6 Host Managed Service Server Center 136720.doc 165-

Claims (1)

200941233 X. Patent application scope: A computer-implemented method, comprising: strategically locating a plurality of application servers; positioning a heart at a different geographic location to receive an application from a client based on an application requested by the client; and - an online application Request; type of program and bonding delay The client request is forwarded based on at least the delay request of the requested application. (4) A plurality of applications should be applied to the server center. 2. As requested. Method, wherein the application is an interactive video game 136 136720doc.doc