NO20101412A1 - A system for graphics and computer gaming experiences on the Internet - Google Patents

Abstract

The invention relates to a system for remote play and local play of computer games via the Internet. The invention includes a presentation system (shown in FIG. 1) that buckles in a Web browser plug-in, a remote playback window, and or in the computer games. Computer games can be executed on servers and downloaded to the user's computer. The Web browser plug-in can be used to remotely play local video games. A desktop application can be used to download, synchronize, and manage game resources locally (eg 3D models, etc.). A remote play window can also be used to remotely play and locally play computer games.

Description

INTRODUCTION

The invention generally relates to information technology systems, and more specifically to a system for providing high quality interactive user interface, computer game and remote gaming experiences on the Internet.

BACKGROUND

There is a good deal of prior art within the technical area of the invention. Some examples of this are:

o Systems for high-quality interactive user interfaces on the Internet:

o Microsoft Silverlight and Adobe Flash

o The HTML 4.01 standard and the HTML 5.0 proposal

o the MPEG-4 standard

o System for providing distraction-free content in a Flash-based gaming environment (US patent application number 2010/0175058 Al)

o Systems for remote gaming experiences on the Internet:

o OnLive

Existing technology is mainly focused on presenting graphics and multimedia for Web browsers (e.g. Silverlight, Flash, HTML and MPEG4), and offering pure remote playback over the Internet (e.g. OnLive). A problem with existing techniques, in relation to publishing computer game-like websites (i.e. websites that look like and function like "main title" computer games do today), is that computer game resources (i.e. 3D models, textures, movies, scripts, code, etc.) have a large file size (often several gigabytes). It also causes problems such as that it will take a long time to download the files (e.g. 10 GB takes approximately an hour to download on a 3 Mbps network), the files will take up a lot of time on the user's computer (e.g. .100 websites with 10 GB each, will take up 1 TB of disk space), synchronization of the content will also become a problem as it is updated by the content provider, and securing the content for children and young people is a further, and persistent, problem for Internet content (i.e. ensure that children and young people are not exposed to harmful content, such as pornography, violence, "sexual predators," etc.). Remote gaming solutions (such as OnLive) circumvent many of these issues by having the computer games execute on servers, so that a thin client software application is used to play the computer games over the Internet (i.e. an audiovisual stream is sent from the computer games, and a "user input" power is sent from the thin client). However, remote gaming solutions are completely dependent on the Internet connection between the server and the end user. The transfer rate will limit the quality of the audiovisual stream. The delay rate ("latency") will cause slowness in the gaming experience (eg the game character jumps one second after keyboard "h" was pressed). Individually, the existing techniques are too weak to realize high-quality computer gaming experiences on the Internet. We are not aware of existing techniques for combining the aforementioned techniques to realize high-quality computer gaming experiences on the Internet.

SUMMARY OF THE INVENTION

It is an aim of the present invention to provide a system for solving the above-mentioned problems in prior art.

The features defined in the independent requirements attached characterize this system.

The present invention provides a system that consists of a Web browser plug-in (or Web browser, media player, or the like) that contains a presentation system for multimedia, 2D/3D graphics, computer games, and remote gaming; a desktop application; and/or a separate client program/window for remote playback, which can also contain the presentation system.

BRIEF DESCRIPTION OF THE FIGURES

The invention will further be described in detail with reference to the figures, where:

FIG. 1 shows the system architecture of the multimedia presentation system used in our Web browser plug-in.

FIG. 2 shows the API ("application programming interface") of our Web browser plug-in.

FIG. 3 shows the elements of the file format of our Web browser plug-in.

FIG. 4 shows a user interface for "safer content."

FIG. 5 shows a user interface for a remote play client software application.

FIG. 6 shows a user interface for interactive remote playing of computer games over the Internet.

FIG. 7 shows a user interface for a "desktop" software application.

FIG. 8 shows the contents of an example GXML file ("MyScene.gxml").

FIG. 9 shows the contents of an example Java file for the GXML file above ("MyScene.java").

FIG. 10 shows the contents of an example GXML 3D graphics stage.

FIG. 11 shows the contents of an example Java file for a GXML computer game page ("MyGame.java").

DETAILED DESCRIPTION

The idea behind the invention is mainly to study whether it is possible to create websites in the same way as computer games are created today. Today's "main title" computer games represent the best in graphics and multimedia experiences on today's computers. Websites on the Internet have traditionally used Web browser plug-ins for the presentation of graphics and multimedia. Adobe Flash has had the strongest market position since the late 90s. Microsoft Silverlight is a contender that has arrived in recent years. Many other Web browser plug-ins exist for various purposes (eg QuickTime, WildTangent, Viewpoint, Cult 3D, PDF Viewers, and more). Existing Web browser plug-ins are very limited in terms of offering user experiences of the same quality as "main title" computer games. Technically speaking, there is no reason why a computer game should not be able to execute just as well in a Web browser plug-in

as in a window (eg in Windows). Computer games typically execute as a separate process with a separate window in Windows. The computer game can start threads, allocate memory, use DirectX, etc

further. A Web browser plug-in usually executes in the same process as the Web browser and can start threads, allocate memory, and use DirectX in the same way as an application. Transferring resource files (3D models, textures, etc.) is an obvious problem. It is also clear that synchronization is a problem. One of the main ideas of the invention is to solve the problems of transferring and synchronizing resource files by offering users a "desktop application" that executes on the user's machine as a service (ie in the "system tray" or service). When the user is on a (ordinary HTML) website that offers Web browser-based computer game experiences, the user can e.g. choose to try the computer game with remote play, or choose to download the computer game to your own computer. If the user chooses to download to their own computer, the Web browser plug-in will communicate this to the desktop application, which starts the download process (provided the user has installed the Web browser plug-in

the component and the desktop application). The desktop application will first retrieve an XML list of all the files to be downloaded and/or synchronized, after which the files will be downloaded. The amount of bandwidth to be used and similar can be configured. The files can of course be compressed (e.g. zip/rar and the like) and encrypted (e.g. RSA and the like). With this solution, the user can close the Web browser or navigate to other pages while the computer game content is being downloaded. When the desktop application has finished downloading the files, the user will be notified (e.g. with a pop-up window near the system tray). The user can then start the computer game by returning to the website of the computer game, or open the desktop application from the system tray (or the start menu etc.) and start the computer game from there. The desktop application will contain a list of all user-registered computer game websites, where the user can start local computer games in the Web browser (or in a local client), start remote playing of the computer games, as well as manage the computer games (delete, allocate, synchronize, compress, backup, etc.) . The desktop application can also include a number of additional functionalities, such as

"instant messaging, remote desktops, FTP tools, cache, licenses, heads-up display tools," and so on. Remote play will typically take place in the Web browser plug-in, in a separate window in the desktop application, or in a separate remote play application. A serious problem on the Internet today is that children and young people are exposed to harmful experiences. According to the Norwegian Media Authority's report on

children and digital media 2010, 14% of children and young people (9 to 16 years) have received sexual requests on the Internet in the last year in Norway. 22% of children and young people have been to pornographic websites. 7% have been sent pornographic images and videos by someone on the Internet. 8% have been asked if they could send nude pictures of themselves via the Internet in the past year. 23% have been asked if they could send personal information about themselves on the Internet, and 5% sent this information. More than 60% of children and young people have received training in safe use of the Internet. There are serious problems related to children's and young people's use of computer games. 30% of children (9 to 12 years) and 57% of young people (13 to 16 years) have played computer games that have an age limit of 18 years and older in Norway. There are many ways to access such games, such as piracy, downloading, etc. It is strange that existing Internet products do not include techniques to

protect children and young people from such. One of the main ideas with our techniques is that we will require all content to be given a minimum age rating ("age rating"). Content providers who do not "rate" their content or who "rate" incorrectly will be denied access to our techniques. Our Web browser plug-in will ("default") block all content rated 12 years and older. Proof of age must be provided to access rated content (e.g. driving license and/or tests). Proof of age and identification must also be presented to gain access to "instant messaging" and such communication functionalities. This procedure will ensure that all content distributed via our techniques is safe for children and young people. There exists a wide range of different Web filters to block harmful content on the Internet. The problem with these is that they have to be installed on the user's machine. Many parents do not know about the problem or that Web filters exist, our technique has the advantage that it works independently of the parents' knowledge and willingness to protect children and young people against harmful experiences on the Internet. Another problem with the existing techniques is that Web filters do not work with all Web browsers. Young people can install these (eg Google Chrome) to bypass the Web filters. our technique does not have this limitation (as the Web browser plug-in works the same in all Web browsers). A problem with existing remote playback techniques (eg OnLive) is that they are limited by the Internet connection (ie transfer speed, "latency," quality of service, etc). Many users of computer games often use high screen resolution (often high definition), multiple screens, and more recently also stereoscopic viewing (eg with 3D glasses with "active shutters" or "polarized" techniques). Real-time transmission of such audiovisual streams over the Internet

requires very high bandwidth. Computer games also often have an action that requires a quick reaction from the user. Latency in the network connection will delay this experience, so that many computer games (e.g. Trackmania) will not be usable with remote play. In addition, there are other problems, such as packet loss and the like. Our techniques aim to take advantage of a combination (or a "hybrid solution") of providing video game website experiences in

The Web browser with an in-house developed Web browser plug-in, and offering computer game website experiences in (and outside) the Web browser with remote gaming in (and outside) the Web browser plug-in. Our solution involves using our Web browser plug-in, desktop application, and remote gaming client on the end user's computer, as well as having computer games stored and executed on servers. The main idea of our solution is that users with an ordinary Internet connection (by today's standards) can try the computer game with remote play (to see if it is something they are interested in spending more time on). If they are interested, they can also choose to download the computer game to their own computer for better quality. The solution can also easily be used to create "hybrid games" where parts of the computer game are made with remote play (e.g. "game environment, windows, portals, displays," etc.) and other parts are made with locally stored content. -

The main idea in relation to industrial exploitation in Norway is mainly in relation to developing and offering "computer game websites," which use our Web browser plug-in, desktop application, and remote gaming techniques to produce and offer websites (on the Internet), which look which and works in the same way as (in relation to the user experience) "main title" computer games do today. As of today, text, video and simple multimedia content are mainly used for the development of websites. We believe that computer game-based experiences are a natural next step, in the future, for content on the Internet/Web. Our techniques represent specific practices for providing computer game websites. We plan to offer products, services, etc., to create such computer game websites for companies, organizations, individuals, etc., in Norway. The web browser plug-in software is illustrated in FIG. 1, which shows the system architecture of the multimedia presentation system used in our Web browser plug-in. The plug-in itself is implemented as an ActiveX control (ref. Microsoft) for Microsoft Internet Explorer 3.0 and later versions, as well as Netscape NSAPI (or Gecko) for all other Web browsers (e.g. Firefox, Opera, Google Chrome, Safari, with more). Other APIs can also be easily used, our techniques are not limited to a specific operating system or hardware device, but are available for all types of operating systems, hardware devices, networks, etc. We describe the details of the figures later in the following paragraphs. FIG. 2 illustrates the API used to create (or program) the content displayed in our Web browser plug-in. In the same way that Microsoft Silverlight uses the Microsoft .NET API for programming the content displayed in Silverlight, our Web browser plug-in also has a corresponding API. In Adobe Flash, the "scripting language" Actionscript and a Flash script API are used. our plug-in uses the Java 2.0 programming language for programming its content (note that the plug-in itself is not made in Java). Other programming languages and scripts may be used in the future. FIG.

3 illustrates the elements (or components) in the file format of the content displayed in our Web browser plug-in. Microsoft Silverlight uses the XAML file format, which is actually defined entirely based on the API (declaratively). Adobe Flash uses the file formats SWF and MXML Our format is a separate format in XML and binary form (eg our Norwegian patent no. 318686). FIG. 3 illustrates the naming of the XML elements. Attributes and functions are described later in the following

the paragraphs. FIG. 4 illustrates a "dialog box" (or window) that appears when the user navigates to a web page that contains "rated" content that has a higher age limit than the plug-in is configured for (e.g. content is rated at 18 years age limit). The user can choose to display the page without the rated content, log in to view rated content (with proof of age), or to cancel the loading. FIG. 5 illustrates the user interface of the remote play client. The user selects a computer game (or server with computer games), configurations (e.g. username, password, audio-visual compression bitrates, image size scaling, rate, port, packet size, etc.) and clicks "Connect" to connect to the server ( "Refresh" updates the list of servers/computer games). FIG. 6 illustrates the user interface (or window) that appears after that

the remote play client has connected to the server. This window displays the video images from the computer game in the client area ("Game View") of the window. User input (keyboard, mouse, "game pad, joystick," etc.) is captured and sent to the server (in real time). The user interface in FIG. 6 can be a separate window, a change to the previous interface, full screen, separate program, or the like. FIG. 7 illustrates the "desktop application," which is typically available in the system tray.

This application can also be a service in the operating system (e.g. NT service) with an associated interface in the start menu, in the control panel, or similar. FIG. 7 contains menus, toolbars, and "tabs," but this may vary between implementations. "Game sites" (the name can of course be changed) shows a list of computer games that have been registered. This list is used (by the user) to manage locally downloaded games in resources, start these games in the Web browser, start remote play in the Web browser or remote play client (FIG. 5), and the like. The user interfaces may also contain other or use a different "layout." our industrial exploitation naturally has an embodiment consisting of a Web browser plug-in (or Web browser, media player, or the like) that contains the system in FIG. 1 to 3, for playback of local computer games and remote gaming, desktop application, and a separate client program/window for remote gaming (which may also possibly contain the system in FIG. 1 to 3). There are many advantages to our techniques (and embodiments): The network issues of remote play are overcome (in a user-friendly way) by allowing users to "try out" the game with remote play (or just use remote play if the user is happy with it), and then download the computer game locally if it is interesting and desirable. Hybrid solutions are also possible with our solution, where remote gaming and local computer game resources can be combined to create and offer the best possible user experience that is adapted to the user's computer and network capacity (as well as cost-effective considerations for content providers), our solution also makes it easy to combine this with other multimedia, graphics, etc. to create "in-game heads-up displays," textual content, and the like in computer game experiences.

FIG. 1 illustrates the architecture of the presentation system used by the Web browser plug-in and possibly by the remote play client. In addition to the system, the Web browser plug-in can contain its own functionalities that will also vary between platforms, operating systems and Web browsers (e.g. programmatic interface to JavaScript in HTML, HTML OBJECT and EMBED tags, update and installation functionality, etc.). The web browser plug-in, the presentation system, the remote play client and the desktop application are mainly implemented in C++, and C where necessary. The presentation system is conceptually created by a programmer who uses an integrated development tool ("integrated development environment (IDE)") (1000) to develop a multimedia application (Web browser plug-ins, heads-up displays, and/or computer games) (1001 ). The multimedia application typically includes a Java 2.0 virtual machine (such as the Sun Java 2.0 Micro Edition virtual machine) (1002). It is also possible to use other virtual machines or "script engines" (eg Lua). GxmlSG

(1003) contains the C++ classes for the "scene graph." GxmlShow (1004) contains the classes

to the main engine, game engine, rendering code, audiovisual functionality, and system code. GxmllO (1006) contains the classes for reading and writing GXML files (ie the file format) and associated utility code. GxmlJNI (1005) contains Java and C/C++ code for the bridge between the Java and C++ code (ie "Java native interface (JNI)," ref. Java 2.0). GxmlSystem (1007) contains system-specific functionality. The presentation system consists of the parts (1003)

to (1007). Parts (1000) to (1002) are included to facilitate understanding, and are not intended to limit the invention to just the development process for creating such a multimedia application. Different GxmlJNI (1005) code libraries must be developed specifically for each virtual machine.

FIG. 2 illustrates the APF ("application programming interface") of the presentation system (ie the API that content developers use to program the content to be displayed). We have used Java 2.0 package and class conventions in FIG. 2, but other programming and scripting languages can also be used with corresponding packages and classes. GxmlJNI (1005) can typically be automatically generated from this API (ie GxmlJNI is a wrapper between C++

and Java). We will hereafter call the APF the "GXML API" for ease of discussion, but the name may change in the future. The GXML API consists (at the top level) of traditional multimedia functionality in the packages (2000) to (2004). Corresponding functionality is found in Microsoft Silverlight and Adobe Flash. The (2005) to (2008) packages contain classes for creating video game experiences. Microsoft XNA has similar functionalities (but not Silverlight and Flash). The package gxml.sg (2000) contains classes that contain and manage the GXML scene graph, as well as the Web browser (Browser). The package gxml.show (2001) contains classes that manage the main engine (Engine) and the rendering (the other classes). The package gxml.av (2002) contains audiovisual functionality. The package gxml.sys (2003) contains system functionalities. The package gxml.graphics2d (2004) contains 2D graphics functionalities. The gxml.game (2005) package contains the computer game engine and related functionalities. The package gxml.graphics3d (2006) contains 3D graphics functionalities. The package gxml.game.input (2007) contains user input classes (keyboard, mouse, game pad, direct input, etc.). The package gxml.game.math

(2008) contains mathematics functionalities. Ordinary multimedia presentations inherit the gxml.sg.Scene class. Video game-based presentations inherit the gxml.game.Game class. FIG. 8 illustrates an example GXML file for ordinary (ie 2D and 3D Internet multimedia content). The file contains a "movie" with a "scene" containing an image, a button, and a freehand drawing ("path"). FIG. 9 illustrates an example Java file for interaction code for the GXML file in FIG. 8. The code points the Web browser to "http://ntnu.no" when the user clicks on the button. FIG. 10 illustrates an example "stage" (in a scene) for 3D graphics. The stage is saved in the file "can.x" as a DirectX X file (other 3D file formats can also be used). FIG. 11 illustrates an example Java file for a computer game website. The example contains the code to display a blue background. The code points the Web browser to "http://ntnu.no" when the user clicks the "Back button" on the game pad. Both "raster" and "ray-tracing" techniques can be used. FIG. 3 illustrates the XML elements of our XML-based GXML format. The format can also be stored in binary form (see e.g. Norwegian patent no. 318686). The GXML format (XML-based) starts

with one gxml element (3000), which contains one head element (3100), one movie element

(3200), and possibly other elements (3600). The gxml element has the version attribute, which indicates the version of the GXML format. The Head element (3100) currently has no other attributes, and contains header elements (3101) to (3112). The Title element contains the title of

GXML content. An example GXML file is illustrated in FIG. 8. Base, meta, and link elements

(3102) to (3104) are the same as in HTML version 4.01. SaferContent (3105) can describe the content with regard to safer content for children and young people. The attribute ageRating specifies the minimum age limit. The href attribute specifies a URL to additional information (eg, keys for decrypting encrypted content). SceneHeader (3106) describes a scene stored in the file, or an external file. The name attribute specifies the name of the scene. The src attribute specifies a URL to the external file. The attribute ageRating specifies the minimum age limit. The bitratelDs attribute specifies an appropriate bitrate interval for the content. The attribute languagelDs specifies a set of suitable natural languages for the content (e.g. Norwegian). The screenlDs attribute specifies a set of appropriate screens (eg 1024 x 768) for the content. The machinelDs attribute specifies a set of appropriate hardware configurations (eg Quad Core) for the content. The ResourceHeader (3107) to ProgramResourceHeader (3111) elements are respectively for general resources, images, XML data, text, and program code stored in the file, or an external file. These elements all have the name, src, and ageRating (SceneHeader) attributes. ImageResourceHeader (3109) can additionally have attributes width, height, and bitCount, which indicate the width, height, and bit-depth of the image. The Header section can also contain other elements (3112). None of the header elements or attributes are required ("mandatory"), they are used freely. Elements and attributes can also be changed and deleted in relation to new needs. The Movie element (3200) currently has no attributes, and contains scenes (3300) and resources

(3400) to (3405). It is also possible to create the GXML format without the movie element, in such a way that scenes and resources are entered under the gxml element (3000). The Movie element is essentially a grouping element (for easier overview). The scene element (3300) contains a scene that can be presented to the user. The scene element (3300) contains the attributes name which specifies a name, autoSize which specifies automatic adjustment of the size of the scene to the container (0 is none, 1 is horizontal, 2 is vertical, and 3 is both), width is the width of the scene, height is the height, mousePointer specifies the mouse pointer, backColor is the background color, antialias specifies whether "antialiasing" should be used, quality specifies the quality level, framesPerKSec is the number of frames per 1000 seconds, tag is additional data, ageRating is the minimum age limit, and dataSource is the data source for elements. The bitrealDs, languagelDS, screenlDs, and machinelDs attributes are the same as iSceneHeader. The program element (3301) contains program code for the scene. The src attribute specifies a URL to the binary code, and srcCode specifies a URL to the source code. Active and placeable elements (3302, 3304, 3305, 3306, 3308, 3309, 3310, 3311, 3312, 3313, 3314, 3315, 3316, 3320, 3321, 3327, and 3328) have name attributes that specify a name, x ( or left) which is the x-position, y (or top) which is the y-position, width is the width, height is the height, enabled indicates whether the element is active, visible indicates whether the element is visible, anchorEdge indicates which edges in the container the element is anchored to (i.e. each edge of the element can automatically follow the "resizing" of the edges of the container), rotation specifies rotation (in degrees "clockwise"), transparency (or opacity, opposite) specifies transparency, mousePointer specifies mouse pointer, backColor specifies background color, tag specifies additional data, and style specifies style. The active and placeable elements typically have other attributes as well. The Group element (3302) has surface which indicates whether the element has its own rendering surface, dataSource indicates a data source for the elements, and ageRating is the minimum age limit for the content. The layer element (3303) has name, enabled, visible, tag, bitrelDs, languagelDs, screenlDs, machinelDs, style, dataSource, and ageRating (which is described earlier). The Image element (3304) has src indicating a URL to the content and ageRating. Text (3305) has src, autoSize, and dataSource. Video (3306) has src, srclmage which is a URL to an image that is displayed when video data is not available, ageRating, and autoStart which specifies whether the video should start playing automatically. Audio (3307) is a non-visual audio element that has name, src, ageRating, x (or left), y (or top), enabled, autoStart, tag, and style. Button (3308) is an interactive button. RadioButton (3309) contains grouped "radio" buttons, and has dataSource. CheckBox (3310) contains a checkbox, and has value indicating the state, as well as dataSource. VSIider (3311) contains a vertical slider, and has rangeMin which specifies the minimum value, rangeMax which specifies the maximum value, and value which specifies the value. HSIider (3312) contains a horizontal slider, and has rangeMin, rangeMax, and value. VScrollBar (3313) contains a vertical scroll control, and has rangeMin, rangeMax, and value. HScrollBar (3314) contains a horizontal scroll control, and has rangeMin, rangeMax, and value. Path (3315) contains a freehand drawing, and has d which contains datapath (ref. the SVG standard), pathLength which specifies the length of the path, fill which specifies the fill (i.e. brush), and stroke which is outlined (i.e. pen) . ComboBox

(3316) is a "combo box," and has text which is the text, appearance which specifies how it looks (0 is "dropdown combo," 1 is "dropdown list," and 2 is "listbox"), value is the choice , and dataSource is the data source. Rect (3317) is a rectangle, and has name, x (or left), y (or top), width, height, rx which specifies the x-radius for rounded corners, ry which specifies the y-radius for rounded corners, rotation, enabled, visible, transparency, mousePointer, anchorEdge, tag, style, fill, and stroke. Ellipse (3318) has the same attributes, but cx and cy are the x/y center of the ellipse, and rx and ry are the x/y radii of the ellipse. Line (3319) has the same attributes (except cx, cy, rx, and ry), and has xl and yl specifying the x/y position of the start point, and x2 and y2 specifying the position of the end point. Polyline (3320) is a "polyline" with points that indicate the points of the lines, and strokes. Polygon (3321) is a "polygon" with points, fili, and stroke. BrushSolidColor (3322) is a single-colored "brush" and has name, color, transparency, and tag. Brushlmage (3323) is an "image brush" and has name, src, titleMode which specifies the "title mode" of the "brush" (0 is none, 1 is title, 2 is flipX, 3 is flipY, and 4 is flipXY) , size specifying the size of the brush, transform specifying a 3x3 transformation matrix, color specifying a ("default") single color, transparency, and tag. BrushLinearGradient (3324) is a linear gradient "brush" and has name, units that indicate "gradient units" (0 is userSpaceOnUse and 1 is objectBoundingbox), startPt indicates gradient vector starting point, endPt indicates gradient vector end point, transparency, and tag. BrushRadialGradient (3325) is a radial gradient "brush" and has name, units, center which is the center point for the gradient, rx and ry indicate the x and y radius, origin indicates the gradient origination points, transparency, and tag. GradientStop (3326) is a gradient stop and has offset that specifies the position of the gradient stop, color that specifies the gradient color, and transparency. Stage (3327) is a stage for displaying 3D content and has src, ageRating, inPlace which specifies in-place rendering, autoStart, and loop which specifies whether the element should loop. Light (3333) is used in a stage for 3D lighting and has type indicating the type of light, diffuse which is a diffuse color, specular which is a specular color, ambient is an environmental color, position is a 3D position, and direction is a 3D direction for the light. Viewpoint (3334) is used in a stage to define a camera and has fieldOfView which is the field of view, position, lookAt which is the direction of view, and up which indicates the up direction of the camera. Stage (3327) may contain other elements

(3335) . Glyphs (3328) contains a set of "glyphs" used for "fixed text" and has src, fontRenderingEmSize which is the font size, styleSimulations which is style flags, bidiLevel which is "bi-directional layout level," indices is a string of glyph specifications , and text which is a text string. Animation (3329) is used for animations and has name, autoStart, loop, frameCount which is the number of "frames," framesPerKSec is "framerate," referenceClock is the reference clock (eg another element), referenceClockSkew is the offset of the clock, attrList is a list of element attributes to animate (eg Mylmage.x), and tag. Ouch

(3336) (or keyframe, frame, frm, key, k, f, kf, etc.) is an "animation item" and has key indicating the key frame position, attr is the attribute (index in attrList), val is the value, effect which indicates whether interpolation is to be used, and notify which indicates whether a program code event is to occur. Animation (3329) can have other elements (3337). Effect (3330) is a shader and/or fx effect for a visual element and has src, progress which indicates progression, duration which indicates duration, loop, visible, clock which is the clock of the effect, and tag. Param (3338) is a parameter to the effect implementation and has name which indicates the name of the parameter, and value which is the value. Effect (3330) may have other elements (3339). Style (3331) is a style for other elements and has name, src, and targetType indicating the type of the elements that can be modified by the style. Si (3340) (or param, item, attr, etc.) specifies parameters for the style and has attr (or name) which specifies the name of the attribute, and val (or value) which is the value. Style (3331) can have other elements (3341). Timer (3332) is a "timer" and has name, left (or x), top (or y), enabled, interval which is the interval, and tag. Scene (3300) may also contain other elements (3342). The resources (3400) to (3404) refer to or contain images, text, program code, XML data, and general resources, respectively. The resources have the attributes name, src, and preload which indicate whether the resources should be loaded immediately. There is also the possibility of other resource elements (3405). The Movie element (3200) can have other elements (3500). The gxml element (3000) can have other elements (3600). Elements and attributes are only simply explained in this overview. Many of the attributes and elements have a similar meaning as in HTML, SVG, Adobe Flash, Microsoft Silverlight, DirectX, etc. Colors use 4 bytes, alpha, red, green, and blue as "#aarrggbb" (eg #ff0000ff is blue). The presentation system of FIG. 1 may seem relatively extensive and complicated, but it is straight forward for an expert with good knowledge and long experience in C/C++ programming, Microsoft COM, Java virtual machines, DirectX, OpenGL, video coding, and shader languages to implement the system. It is recommended that the system be implemented as a C/C++ library. The virtual machine should also be structured as a C/C++ library. It is recommended that the thread handling in the virtual machine be changed so that the thread for main execution is removed and replaced with a DoIterate() function that is called periodically by the Engine class in (1004). This will make syncing much easier. However, this is not a requirement, and the invention will also cover virtual machines that use normal thread-based execution. GxmlJNI (1005) can be generated automatically as this is "brigde/wrapper" code between Java and C++. The Java packages and classes in FIG. 2 has functions marked as "native" (ref. Java).

These "native" functions must be implemented in C/C++. Java virtual machines usually contain documentation on how such "Java Native Interfaces" can be generated. GxmllO

(1006) consists of the main classes Reader, Writer, Gxml20, and Tools. Reader can e.g. easily created with Expat to parse a GXML file and instantiate a scene graph GxmlSG (1003). Writer creates a GXML file from a scene graph GxmlSG (1003). Gxml20 contains version-specific parsing functionality for the Reader for GXML 2.0. GxmlSystem (1007) contains system code (eg system information, zlib compression/decompression, MD5 hash, JPEG, PNG, GIF, DirectX and OpenGL helper functionality, etc.). GxmlSG (1003) is the scene graph. The same elements and attributes as in FIG. 3 is used in GxmlSG (1003). The classes will also implement functions (eg Image can contain code to decompress JPEG, PNG, and GIF images). We will describe "public" functions of the scene graph in the review of FIG. 2 in the next paragraph. GxmlShow (1004) contains the main engine Engine (or Engine2D), the game engine Game (or Engine3D), Graphics2D, Graphics3D, audiovisual classes, system classes, and rendering classes (Show classes). The Engine class is relatively simple and consists of a GXML Reader, a Java VM instance, and a Show class. The Engine class also has a URL attribute, and a DoIterate() function. A computer program (e.g. a Web browser plug-in) can use the Engine class as the main interface to the presentation system. The URL of a GXML file must be specified. DoIterate() must be called periodically (eg as frequently as possible). The Engine instance will in DoIterate() download the GXML file and parse it with Reader. The Java program code will become available, and Engine injects this into the Java VM and starts it. The Show instance can be used by the Engine to render the scene graph. GxmlShow (1004) implements various Show classes (eg ShowDX8, ShowDX9, ShowDXIO, ShowDXll, ShowGL, etc.). Show can also have an abstract class that is used by Engine, and from which the Show classes inherit. The show classes can contain all graphics library specific code to render the scene graph. Each element (eg Image) is given a vertex buffer (in eg DirectX). The image itself is rendered as a texture on the vertices. Shaders can be used to manipulate the rendering (ie effects). The Game class is similar to Engine, but does not load a GXML file. The rendering is done 100% in Java. FIG. 2 illustrates the Java API of the presentation system. The package gxml.sg (2000) contains the scene graph. The scene graph's C/C++ representation is at (1003) in FIG. 1, and the GXML file format representation is in FIG. 3. Most of the elements in FIG. 3 are found in (1003) and (2000), as they are described with attributes earlier. The Java API uses "get" and "set" functions (common in Java 2.0) to read and change the contents of attributes (eg Image: :setX(int v)). The classes Video, Audio, and Animation have the functions play (or start), pause, stop, etc. The Browser class represents the Web browser (for the Web browser plug-in). This class can be used to control the Web browser, and has the function navigate, which can be used to navigate to a new website. The classes in gxml.sg (2000) have several functions, but it becomes too detailed for the description (and "scope") of the invention. It is also relevant for some implementations that the scene graph for the format is a separate package in addition to gxml.sg. The package gxml.show (2001) contains the Engine class, which can control the main engine, and has the function play to play a specific scene, as well as functions (eg setShow) to change the show module used (eg to ShowDX8 , ShowDX9, ShowGDI, etc). The package gxml.show (2001) also contains the classes ShowDX, ShowGL, ShowGDI, and ShowPIX which are used for graphics library specific rendering (the names may vary). The classes EffectDX and EffectGL are used to control DirectX and OpenGL effects respectively. The effect classes have the functions play (or start), pause, and stop for controlling animated effects. The package gxml.av (2002) contains audiovisual functionality. The FilterGraph class contains GPU/shader-based filter graphs. The FilterSource class represents the reading and decompression of audiovisual files (codec) and has the functions open, close, play (or start), pause, and stop. The FilterEffect class represents an audiovisual GPU effect and has the functions play (or start), pause, and stop. The FilterDRM class represents the encryption and decryption of an audiovisual file or stream. The AudioCue class represents lists of sound effects and music. The Audio3D class represents playback of 3D audio. WaveBank represents a collective sound. The audiovisual classes will typically have similar functions as in Microsoft DirectX, Windows Media Format, Windows Media Rights Manager, and .NET. The package gxml.sys (2003) contains system functionality. The Convert class contains standard Java conversion functions such as toLong and so on. The Debug class contains debugging functions such as print to write to a debugging console. The Cache class is used for temporary local storage. The System class contains system-specific information and functionality such as getCPUCount and the like. The Network class contains network functionality. The Storage class is used for permanent local storage. The package gxml.graphics2d (2004) contains functionality for 2D graphics. The classes are equivalent to the classes found in Microsoft GDI+ with functions. The package gxml.game (2005) contains classes for creating computer game web pages. The Game class is used to implement the computer game (a derived class is implemented). The GameWnd class is used to control the window in which the computer game is displayed (e.g. Web browser plug-in window or separate window) and has functions such as setFullScreen for full screen display. The ContentManager class is used to load and manage the content (e.g. 3D models) and has functions such as e.g. load. The package gxml.game.graphics3d (2006) contains 3D graphics functionality. The classes are equivalent to the classes found in Microsoft XNA with functions. The Graphics3D class is equivalent to GraphicsDevice in XNA. Classes for which no functions are specified have no specific functions. Functions may be added later (and are outside the "scope" of the invention). Indirect/referenced functions (eg GDI+ and XNA) are also outside the scope of the invention. Features that are outside the "scope" are mentioned only to facilitate a third party to implement the invention. All combinations (within "scope") are therefore part of the invention as they naturally constitute different embodiments of the invention.

FIG. 4 illustrates the user interface displayed (by Web browser plug-in or similar) when a GXML file contains "rated" content (eg an element has a minimum age limit of 18). The user interface may, for example, be displayed as a Windows dialog box window as illustrated in FIG. 4 (title bar, minimize button, close/exit button, and the like). Other types of user interfaces that are (or will become) common in operating systems, platforms, Web browsers, and the like, for the user interface, are also covered by the invention. The user interface contains information for the user, a "combobox" (or other list) to choose what

to be done (e.g. "show without 'rated' content" as illustrated in FIG. 4), username, password, a check box for whether the message should always be displayed, a configuration button, an "ok" button (or similar) and a cancel button. The configuration button displays a dialog box (or window, Web page, or similar) for configuring user accounts, passwords, and the like. Each user to access "rated" content must provide proof of age (e.g. tests and driver's license). Username and password for an age-verified user account must be provided for access to "rated" content. The main idea of this user interface (and the solution) is to protect children and young people from harmful content on the Internet (no active intervention from parents is assumed). Parents may be asked not to create usernames and passwords to help children and youth access "rated" content (ie help them circumvent the system).

FIG. 5 and 6 illustrate the remote play client user interface used on the user's computer to remotely play computer games (or specifically Web-based computer games) over the Internet. The user interface may, for example, be displayed as a Windows dialog box window as illustrated in FIG. 5 and 6 (title bar, minimize button, close/exit button and similar window elements). Other types of user interfaces that are (or will become) common in operating systems, platforms, Web browsers, and the like, for the user interface, are also covered by the invention. FIG.

5 shows the user interface that appears at start-up (of the program) and possibly after a remote play session is finished. The user interface contains information, and the user specifies a server (or computer game) in the "combo box" (or other list). The user interface may also contain configuration controls to configure such things as the bitrate used for audio-visual compression, full screen, "3D vision, grab input, in-game cursor," packet size, image scaling, frame rate, network port, username, and password. This list is an example and can of course be changed for different implementations. The "Connect" button (or similar) is used to start the remote play session (against the server). The Refresh button (or similar) is used to update the list of servers (or similar). The cancel button (or similar) is used to close/exit the user interface/client. The remote play client shows the user interface in FIG. 6 when the remote play session executes (except in full screen mode where the game view covers the screen). Game View displays the decoded video signal from the server (ie a compressed audio and video stream of the screen images of a computer game executing on the server). The video signal can be compressed with any video and audio codec (eg Microsoft WMV2 and WMA). The remote gaming client captures user input (such as keyboard, mouse, game pad, joystick, Direct Input, or similar) and sends user input data compressed (or uncompressed) to the server (ie the computer game). The remote play client is relatively straight forward and easy to implement. The remote gaming client may also contain the presentation system for visualizing 2D, 3D and multimedia content (with user interfaces) in the game view. The integration of the presentation system into game display is relatively straight forward and simple, and the video stream (ie for the remote play) can e.g. is displayed with a Video element (1003) and (2000) in the presentation system. Components are needed on the server to enable remote play. Program code is needed in the computer game which receives the user input (possibly decompressing this) and injects/handles this in the computer game. The computer game must also contain program code to manage the start and stop of remote play (with configurations), capture ("capture") of screen images and sound in the computer game, compression of audio and video, as well as sending audio/video packets over the network. It is also possible that the presentation system is included in the computer game. Implementation of all this functionality in the computer game is relatively straight forward and simple. Different compression techniques can be used. In DirectX, the various Present functions are a natural place for "capture" of the screen image. The invention also covers computer games (and servers) that have such components for enabling remote gaming. The user interfaces of FIG. 5 and 6 can be put together in many ways, e.g. as two dialog boxes/windows, such as Microsoft Remote Desktop, so that the window in FIG. 5 is changed to the window in FIG. 6 when the user presses "Connect," and so on. It is also possible that other layouts and names are used in the interfaces.

FIG. 7 illustrates the "desktop application" that is typically available in the "system tray." The desktop application will typically start when the computer starts up (as an icon in the system tray, such as e.g. Skype). The desktop application can also be implemented as a service (eg as an NT Service in Microsoft Windows). For example, the user interface may be displayed as a Windows dialog window as illustrated in FIG. 7 (title line, minimize and maximize button, close/end button and similar). Other types of user interfaces that are (or will be) common in

operating systems, platforms, Web browsers, and the like, for the user interface, are also covered by the invention. The menu line (ie Menu, Menu2, Menu3, etc.) contains menu functions such as e.g. exit for terminating the application, backup for taking a backup of

a downloaded computer game (or computer game resources), options to change settings, help to give access to documentation, and about for information about the desktop application. Below the menu bar is the toolbar (ie Button1, Button2, Button3, etc.). The toolbar usually contains frequently used functions (which are also found in the menu). Below the toolbar is a "tab" control that displays main categories of functionality (such as "Game Sites" and others). It is applicable that the menu bar and the toolbar differ when the "tab" control is used (ie the main category functionalities have different menu bars and toolbars). "Game Sites" are shown as selected in FIG. 7. The desktop application may also include a number of main category functionalities (eg "instant messaging, remote desktops, FTP tools, cache, licenses, heads-up display tools," etc.). The "Game Sites tab" contains a list of registered video games (local downloadable video games and/or remote play video games). The list in the figure shows 3 examples of computer games in the list. A "scrollbar" (on the right) can be used to "scroll" in the list. Each computer game in the list is shown with a small "thumbnail" image (Image, Image2, and Image3) that shows a small representation of the computer game. Next to the "thumbnail" image there is an information field (e.g. Information 1) which contains information about the computer game (such as descriptions, publisher, publication date, URL, the number of bytes it takes up on the hard drive, the number of files , usage statistics, etc.). The information field can also contain controls (such as buttons and the like). Typical functions that can be performed on each individual computer game in the list are starting local computer games in a Web browser (or in a local client), starting remote playing of the computer games, as well as managing the computer games (deleting, allocating, synchronizing, compressing, backup, etc.). Computer games can typically be registered in the list from the Web browser plug-in, or from the desktop application directly (e.g. "add game" in the menu). If a computer game is registered in from the Web browser plug-in, it can e.g. happen via a Windows message (in Microsoft Windows), registry, a file, a "remote procedure" call, or similar. The desktop application will retrieve an XML list of all the files to be downloaded and/or synchronized, after which the files will be downloaded. The amount of bandwidth to be used and similar can be configured. The files can of course be compressed (e.g. zip/rar and the like) and encrypted (e.g. RSA and the like). Any structure can be used for this XML file, and e.g. a list of "file" and "folder" elements (including file/folder name, date, bytes, md5/crc, URL, ageRating, etc.) within a "synch" element. When the desktop application has finished downloading, it can send a message back to the Web browser plug-in and/or remote play client (progress messages can also be sent).

Although the invention has been described with reference to one embodiment, those skilled in the art will realize that variations in form/detail can be made without deviating from the intent in the scope of the invention as defined in the appended claims. The specific details above are intended to be illustrative only and the scope of the invention is defined by the appended claims. For example, the invention can be practiced as remote gaming in a Web browser and the invention can be integrated into operating systems, Web browsers, hardware, and computer games.

Claims (21)

1. System for remote playing and/or local playing of computer games via the Internet, characterized by a presentation system consisting of an API class for presenting 2D content or 3D stages, an API class for presenting and playing local computer games and/or with one or more remote gaming sessions in such a computer game; and a desktop application (7000) containing a list of "game sites" for computer game resources that can be used by the Web browser plug-in to present/play local computer games with local play sessions and/or with remote play sessions. 2. System according to claim 1, characterized in that the presentation system can use Microsoft DirectX 9, 10 and 11 for rendering/visualization of 2D and 3D graphics. GXML file or code select the version. 3. System according to claim 1, characterized in that the presentation system displays a user interface when it has read GXML-formatted content that contains "rated" content with a minimum age limit of 12 years and older. FIG. 4 shows this user interface, but design/content may vary. 4. System according to claim 1, characterized in that the server system where the computer game executes also contains the presentation system for "in-game heads-up display" graphic interfaces (1003). 5. System according to claim 1, characterized in that the presentation system on the end user's machine contains the entire API (2000) to (2006). 6. System according to claim 1, characterized in that the presentation system on the end user's machine is used in a Web browser plug-in in a Web browser. 7. System according to claim 1, characterized in that the presentation system on the end user's machine is used in a remote playback window or in full screen (6000). The remote play window can be opened as part of a computer program window (5000), from a desktop application (7000), Web browser, Web browser plug-in, from the start menu, "system tray" or the desktop of the operating system. 8. System for publishing computer games via the Internet, characterized by computer games execute on servers, the computer games contain the presentation system used to present 2D and 3D graphics in the computer game, possibly in addition to other 2D and 3D graphics, and specifically use only certain parts of GxmlSG (1003) and GxmlShow (1004); a remote play window, optionally in full screen, (6000) is displayed on the end user's machine and uses the presentation system to present 2D and 3D graphics in the computer game, an audiovisual stream from the servers is displayed in the 2D and 3D graphics; and computer game resource files, which contain at least one 3D model, are stored on the end user's machine, and are used by the presentation system to visualize graphics. 9. System according to claim 8, characterized in that the remote play window is opened as part of a software application for remote play of computer games (5000). The remote play window can be a version of the user interface in (5000), or displayed in full screen mode. 10. System according to claim 8, characterized by Microsoft DirectX 11 being used for visualization with raster-based techniques. 11. System according to claim 8, characterized by "ray-tracing" techniques being used for visualization. 12. System according to claim 8, characterized in that the remote play window displays a user interface when the computer game content is "rated" to a minimum age limit of 12 years and older. FIG. 4 shows the content of an embodiment of this user interface, but the design/content may vary. 13. System according to claim 8, characterized in that the presentation system on the servers contains the entire API (2000) to (2006). 14. System according to claim 8, characterized in that a desktop application (7000) is used to contain a list of "game sites" for computer game resources that can be used by the remote play window to present/play local computer games with local play sessions and/or remote play sessions. 15. Method for remote playing via the Internet of computer games with 3D models created by the users, characterized in that the method includes: a desktop application that executes on a user machine uploads a 3D model to a File Transfer Protocol (FTP) server, and the 3D model is to be used to be loaded and displayed in a computer game or other application with 3D graphics support, called the "computer game"; at a later time, a user initiates a recording of game data i from the computer game, which executes on a server, where the game data recording contains at least one image, a video frame, a sound sample, or a user input sample, and where the computer game displays the 3D model; recording game data from the game, and aggregating the recording data to produce a data representation suitable for transmission over an Internet Protocol network; transmitting the data representation over an Internet Protocol network to the user; receiving the data representation in a web browser, web browser plug-in, or other software or hardware application with support for web browsing, called "web client"; processing of the data representation in the web client; display or playback of at least one image, one video frame, one sound sample, and/or one user input sample from the data representation in the web client; the web client simultaneously captures user input which is transmitted aggregated to the server over an Internet Protocol network; and the user input is received on the server, processed and injected into the computer game. 16. Method according to claim 15, characterized by the fact that the web client is implemented with HTML and JavaScript. 17. Method according to claim 16, characterized by the fact that the game display uses JPEG, PNG or HTML canvas images. 18. Method according to claim 15, characterized in that the web client's configuration controls can be changed arbitrarily. 19. Method according to claim 15, characterized in that the game display uses the presentation system. 20. Method according to claim 15, characterized in that the desktop application uses the user interface in FIG. 7. 21. Method according to claim 15, characterized in that the game display is implemented with Adobe Flash, Microsoft Windows Media Player, Silverlight, or Apple QuickTime for images, video and/or sound.