WO2006041694A2

WO2006041694A2 - Method and system for integrating capture events in a user interface-based application

Info

Publication number: WO2006041694A2
Application number: PCT/US2005/034877
Authority: WO
Inventors: Robert P. Morris
Original assignee: Scenera Technologies, Llc
Priority date: 2004-10-04
Filing date: 2005-09-28
Publication date: 2006-04-20
Also published as: EP1800242A2; CN101124824A; EP1800242A4; JP2008516311A; WO2006041694A3

Abstract

A method and system for submitting from an electronic device multimedia data captured by the electronic device to a web application in a server includes providing a user interface in the electronic device through which the web application is accessed and enabling the user interface to detect and identify a multimedia capture event originating from the electronic device. After such an event is detected and identified, the user interface automatically processes multimedia data captured during the multimedia capture event, and transmits the multimedia data to the web application.

Description

METHOD AND SYSTEM FOR INTEGRATING CAPTURE EVENTS IN A USER INTERFACE- BASED APPLICATION

FIELD OF INVENTION

The present invention relates to user interfaces, and more particularly to a method and system for submitting from an electronic device multimedia data captured by the electronic device to a web-based application.

BACKGROUND OF THE INVENTION

Traditionally, services offered by a software application are utilized only by a user who has access to the computer system on which the application was stored. In a network, where several computer systems are allowed to communicate with one another and/or with a server, an application residing in one computer system or in the server can be utilized by the other computer systems in the network.

Networks have advanced from local private intranets to public networks, such as the Internet. Here, computer systems connected to the Internet are adapted to communicate with other computer systems through a user interface, e.g., a web browser or a remote desktop. In the case of the remote desktop, the computer system is referred to as a remote terminal. Services offered by an application residing in a server or a computer system are available to another computer system through its user interface, e.g., browser, as web applications. Thus, a computer system need only provide a user interface in order to access a web application, while the web application itself is managed from a central location and not locally. This is particularly advantageous for portable, handheld electronic devices, such as digital cameras, telephones, and PDAs, because memory in these devices is generally limited. Frequently, web applications prompt the user to perform an action, such as entering a credit card number, selecting a button, or checking a box. Typically, the user then performs the action by pressing a button displayed on the screen, using a pointer to check the box, or using a keypad to enter data. The interface, e.g., browser, is capable of detecting the user's action, referred to as an input event. Thus, when the user enters his or her credit card number using the keypad, the browser detects the input event as a keystroke and processes the data associated with the keystroke accordingly.

As stated above, a portable handheld electronic device that is web- enabled can access web applications via a user interface, e.g., web browser. In addition, many handheld electronic devices are equipped with multimedia capture subsystems that allow the device to capture and store live image and audio data, referred to as multimedia data.

If a web application prompts the user to submit multimedia data, e.g., an image, the user must first capture and store the image in a file system, then browse the file system to locate the image, and then select and upload the image through some mechanism, such as an HTML form, file transfer operation or email. This process is tedious and time consuming, particularly if the user is utilizing a handheld electronic device to access the web application. In that situation, browsing through what could be hundreds of images stored in the file system for the requested image is difficult and time consuming given the limited navigation capabilities and processing speed.

Accordingly, what is needed is a method and system for simplifying the process for submitting from a handheld electronic device multimedia data captured by the handheld electronic device to a web application. The present invention addresses such a need.

BRIEF SUMMARY OF THE INVENTION

The present invention provides a method and system for submitting from an electronic device multimedia data captured by the electronic device to a web application in a server. The method includes providing a user interface in the electronic device through which the web application is accessed and enabling the user interface to detect and identify a multimedia capture event originating from the electronic device. After such an event is detected and identified, the user interface automatically processes multimedia data captured during the multimedia capture event, and transmits the multimedia data to the web application.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a system diagram according to a preferred embodiment of the present invention. FIG. 2 is a flowchart for submitting multimedia data to a web application from a web browser according to a preferred embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to user interfaces, and more particularly to a method and system for submitting multimedia data captured by a handheld capture device to a web based application. The following description is presented to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a patent application and its requirements.

Various modifications to the preferred embodiments and the generic principles and features described herein will be readily apparent to those skilled in the art. For example, although the preferred embodiment is implemented in a web browser application, the principles of the present invention can also apply to remote desktop technology. Thus, the present invention is not intended to be limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and features described herein.

According to the preferred embodiment of the present invention, a user interface utilized to access a network, such as the Internet, recognizes an input event that is a multimedia capture event, e.g., the pressing of a shutter, or the pressing of a record button. The multimedia data corresponding to the capture event is then passed directly to the user interface as input data. The multimedia data is then processed and submitted to a web application at a server via the network. By enabling the user interface to detect multimedia capture events and to receive directly the corresponding multimedia data, the user is not required to perform additional steps beyond capturing the image, movie clip, or audio in order to submit the multimedia data to a web application. This significantly simplifies the process and saves time because the user is not required to browse through the file system to retrieve the desired multimedia data.

FIG. 1 is a block diagram illustrating a system environment in accordance with a preferred embodiment. According to the present invention, an electronic capture device 10 is coupled to a server 50 via a network 30, such as the Internet. The electronic capture device 10 is preferably a web-enabled digital camera, mobile camera-phone, video camera, or some other web-enabled electronic device that is capable of capturing live image and audio data, i.e., multimedia data. The server 50 includes a web server 52 and a plurality of web applications 54.

The electronic capture device 10 includes a multimedia capture subsystem 12, and a user interface which is used to access the network 30. In the preferred embodiment, the user interface is a web browser 14, although the following principles can also be applied to a remote client provided by a remote desktop application and similar user interfaces. The multimedia capture subsystem 12 manages a capture event, such as taking a picture, recording sound, or making a movie. The image or audio data captured during the capture event is typically processed, named and then stored in a file 15 associated with the captured event in a file system 13. The multimedia capture subsystem 12 and file system 13 are well known to those skilled in the art.

The browser 14 provides an interface between the server 50 and the user 20. A web application 54 on the server 50 transmits web pages to the browser 14 for display to the user 20. In some circumstances, the web page requests input from the user 20 through what is known as a form. The form includes one or more form elements, each of which define a valid input type and other associated attributes. The form is a well known construct to those skilled in the art of creating interactive web pages using any markup language, such as HTML, XHTML, WML and the like.

The browser 14 includes an input handler 16 coupled to a form manager 17 for processing responses from the user. The input handler 16 detects and identifies the type of input event the user 20 has performed in response to the request. For example, input events include moving the mouse, right or left clicking, and keypad entries. The form manager 17 interprets the form element that is presently in focus and processes the input event from the input handler 16 according to the instructions associated with the form element.

Currently, the input handler 16 detects user 20 input events originating from a keystroke (e.g., pressing a navigation key), and events originating from a pointing device(e.g., moving a mouse or right/left clicking). With conventional browsers and input handlers, if the form element requests image data or other data generated from the multimedia capture subsystem 12, the user 20 must take the picture (i.e., capture the image), browse through the file system 13, locate the file 15 associated with the image, and select the file 15. This process is tedious and time consuming, particularly for small electronic devices 10 that have limited user interface capabilities.

According to the preferred embodiment of the present invention, the browser 14 is coupled directly to the multimedia capture subsystem 12, such that input events that are multimedia capture events can be passed directly to the input handler 16. The input handler 16 of the present invention is enhanced to detect and identify multimedia capture events originating from the multimedia capture subsystem when the browser 14 is focused on a form element. By recognizing such capture events as valid input events, the multimedia data captured during a capture event can be automatically associated with the form element, if appropriate, without additional manual steps.

In a preferred embodiment, existing file input tags can be used and no modifications to the form manager 17 are necessary. For example, consider the following form created with standard HTML: Example 1 : <FORM action=http://server.dom/cqi/hanclle enctype="multipart/form-data" method="post"> <P>

Provide pictures of damage: <INPUT type- Tile" name="name_of_files"> </P>

</FORM>

Here, the web page instructs the user 20 to submit images of damage for purposes of filing an insurance claim, for example. The form element defines an input type as a "file." In conventional systems, the user 20 is required to capture and store the image in a file 15, browse the file system 13 for the image file 15, and select the image file 15 using a pointer or navigation key.

According to the preferred embodiment of the present invention, the user 20 can also input an image file by focusing the electronic capture device 10 on the damaged product and pressing the shutter button to capture an image of the damaged product. Because the input handler 16 is capable of detecting and identifying a multimedia capture event, e.g., pressing the shutter button, as an input event, it accepts the multimedia data captured during the event and passes it to the form manager 17. The form manager 17 determines that the multimedia data conforms with the input type for the form element, e.g., a "file," and then processes the data according to the form.

In another preferred embodiment, a new "capture" type is defined for the input type of the form. Here, the form manager 17 is modified slightly to recognize the "capture" type as specifically requiring multimedia data from a multimedia capture event. In example 2, a form according to this preferred embodiment follows:

Example 2:

Provide pictures of damage: <INPUT type="capture" name-'name_of_files"> </P> </FORM>

Here, the form element defines a new input type, a "capture" type. According to the present invention, the "capture" input type explicitly requires multimedia data from the multimedia capture subsystem. In contrast to the form element in Example 1 , where any type of "file" is acceptable, the form element in Example 2 is restricted to one input type, a "capture" type.

In yet another preferred embodiment, a new input attribute is provided, entitled "validlnputTypes," that restricts the source of input for the file input type.

Here, the form manager 17 is modified slightly to interpret the "validlnputTypes" attribute as specifically requiring data from a particular source, e.g., the file system 13 or the multimedia capture subsystem 12. In example 3 below, a form according to this preferred embodiment follows: Example 3:

<FORM action=http://server.dom/cq.i/handle enctype-'multipart/form-data" method="post"> <P> Provide pictures of damage: <INPUT type-'file" validlnputTypes="file, capture" name="name_of_files">

</P> </FORM> Here, in this example, the attribute "validlnputTypes" informs the form manager 17 that inputs originating from the file system 13 and from the multimedia capture subsystem 12 are valid input sources. Thus, the form in Example 3 is less restrictive than the form in Example 2, but more restrictive than the form in Example 1.

The form elements described above exemplify how the browser is modified to integrate multimedia capture events into form elements. In all cases, the input handler is enhanced to detect and identify multimedia capture events from the multimedia capture subsystem. Those familiar with browser technology will readily appreciate that the form manager may or may not be modified and that the modifications described above in Example 2 and Example 3 are not exhaustive.

Other kinds of input types and input attributes can be introduced to provide additional features related to the multimedia capture events. For example, a new input attribute that indicates the number of images expected or allowed can be provided. New input types can define certain characteristics of the multimedia data, such as minimum resolution and formatting. In addition, the file input form element can be extended to specify a destination (URL) for the images rather than posting them as part of the form. The extended form element can also be utilized to delay the transmission of the image(s) so that the form can be posted quickly.

In addition, according to another preferred embodiment of the present invention, the browser 14 is enhanced to support new scripts that are directly responsive to multimedia capture events. Here, one or more script interpreters 18 and event handlers (not shown) are modified to handle input events originating from the multimedia capture subsystem 12. In Example 4 below, a form according to this preferred embodiment follows: Example 4:

Provide pictures of damage: <INPUT type="file" name="name_of_files" </P>

</FORM> Here, a new script event, "onCapture," is defined as an input attribute in the form element. When the input handler 16 identifies a multimedia capture event corresponding to the "onCapture" event, the script interpreter 18 is invoked, and the script interpreter 18 calls an identified subroutine, "preplmageO," which is typically defined in the page. The multimedia data captured during the capture event would be made available to the called routine as a parameter or through a well known script API call, such as "getl_astCapturedlmage()." The multimedia data is then processed according to the subroutine before the data is posted. According to the preferred embodiment, a variety of multimedia capture events can be defined, each having one or more associated script events. For example, a multimedia capture event can include operating the multimedia capture subsystem in a "Live View" mode, and an associated script can display the live view on the browser 14. In another example, the capture event can be a "preCapture" event where the user 20 focuses on an object, but has not yet captured the image. The associated script event, e.g., "onFocus," could adjust certain settings related to the image. In yet another example, the capture event can be a "postCapture" event, and the associated script can display a thumbnail of the captured image on the browser 14. Those skilled in the art would readily appreciate that other multimedia capture events exist and that scripts can be associated therewith according to the preferred embodiment of the present invention.

FIG. 2 is a flowchart illustrating a process for handling a form element in a form according to the preferred embodiment of the present invention. Referring to FIG. 1 and FIG. 2, the process begins when focus is set on the form element in the form (step 200). The input handler 16 detects and identifies a multimedia capture event from the multimedia capture subsystem (step 202) and passes control to the form manager 17. The form manager 17 parses the form element and determines whether the form element is compatible with the multimedia capture event. For example, the form manager 17 can analyze the input type, such as the "file" and "capture" input types described above, and the input attributes of the form element to determine whether the form element is capable of accepting the multimedia data associated with the capture event. If the multimedia capture event is not compatible (step 204), the form manager ignores the input event and waits for the next input event from the input handler 16 (step 205).

If the form element is compatible with the multimedia capture event, the form manager 17 automatically associates a reference to the multimedia data captured during the multimedia capture event with the form element (step 206). In a preferred embodiment, an identifier for the image data or some other event parameter, as opposed to the actual image data, is associated with the form element. After the reference is associated with the form element, the form manager

17 determines whether a script event, e.g., "onCapture," is present as an input attribute (step 208). If the script event is not present, the form manager 17 waits for a next input event (step 205). If the script event is present, the form manager 17 invokes the corresponding script interpreter 18, which executes the specified script or subroutine (step 210). Once the subroutine is completed, the form manager 17 waits for the next input event (step 205). When the user is done entering data into the form, the user selects a submit type button on the form and the data is submitted to the web application 54 in the server 50 according to well known processes in the art. According to aspects of the present invention, the input handler 16 in the browser 14 is enhanced to detect and identify multimedia capture events such that multimedia data can be automatically associated with a compatible form element in focus in a form. By integrating multimedia capture events with the browser 14, the process for submitting multimedia data captured during the capture event is significantly simplified.

Although the preferred embodiments described above are directed to a browser, those skilled in the art would readily appreciate that the principles discussed above would also be applicable to remote display technology other than a browser. In addition, although the form examples are directed to HTML, the same principles would apply to any markup language, e.g., WML and

XHTML, that defines forms and form elements.

The present invention has been described in accordance with the embodiments shown, and one of ordinary skill in the art will readily recognize that there could be variations to the embodiments, and any variations would be within the spirit and scope of the present invention. Accordingly, many modifications may be made by one of ordinary skill in the art without departing from the spirit and scope of the appended claims.

Claims

CLAIMS What is claimed is:

1. A method for submitting from an electronic device multimedia data captured by the electronic device to a web application in a server comprising: providing a user interface in the electronic capture device for accessing the web application; enabling the user interface to detect and identify a multimedia capture event originating from the electronic device; after the multimedia capture event has been detected and identified, automatically processing by the user interface multimedia data captured during the multimedia capture event; and transmitting the multimedia data to the web application via the user interface.

2. The method of claim 1 , wherein providing the user interface includes providing a web browser for displaying and interpreting an interactive web page from the server, wherein the interactive web page includes a form element in a form that allows the user to submit requested data to the server.

3. The method of claim 2, wherein the form element includes at least one attribute related to the data to be submitted via the form element.

4. The method of claim 3, wherein the at least one attribute includes a capture input type, wherein the capture input type explicitly requires multimedia data.

5. The method of claim 3, wherein the at least one attribute defines one or more valid sources from which the data can originate.

6. The method of claim 3 further including: after detecting and identifying the multimedia capture event, determining by the browser whether the multimedia data captured during the multimedia capture event is compatible with the at least one attribute defined in the form element; and if the multimedia data is not compatible, ignoring the multimedia capture event.

7. The method of claim 6, wherein if the multimedia data is compatible, processing the multimedia data captured from the multimedia capture event includes automatically associating the multimedia data with the form element.

8. The method of claim 3 further including defining a script event directly related to the multimedia capture event, such that the script event is responsive to the multimedia capture event.

9. The method of claim 8, wherein the at least one attribute in the form element includes the script event related to the multimedia capture event.

10. The method of claim 9, wherein processing the multimedia data captured from the multimedia capture event includes executing instructions associated with the script event.

11. A handheld electronic capture device comprising: a user interface for accessing a web application in a server; and an input handler in the user interface for detecting and identifying a multimedia capture event originating from the electronic capture device, wherein after the input handler detects and identifies the multimedia capture event, the user interface automatically processes multimedia data captured during the multimedia capture event and transmits the multimedia data to the web application.

12. The device of claim 11 , wherein the user interface is a web browser for displaying and interpreting an interactive web page from the server, wherein the interactive web page includes a form element in a form that allows the user to submit requested data to the server.

13. The device of claim 12, wherein the form element includes at least one attribute related to the data to be submitted via the form element.

14. The device of claim 13, wherein the at least one attribute includes a capture input type, wherein the capture input type explicitly requires multimedia data.

15. The device of claim 13, wherein the at least one attribute defines one or more valid input sources from which the data can originate.

16. The device of claim 13, wherein the browser further includes a form manager for determining whether the multimedia data captured during the multimedia capture event is compatible with the at least one attribute defined in the form element, wherein if the multimedia data is not compatible, the form manager ignores the multimedia capture event.

17. The device of claim 16, wherein if the multimedia data is compatible, the form manager automatically associates the multimedia data with the form element.

18. The device of claim 13 further including means for defining a script event directly related to the multimedia capture event, such that the script event is responsive to the multimedia capture event.

19. The device of claim 18, wherein the at least one attribute in the form element includes the script event related to the multimedia capture event.

20. The device of claim 19, wherein the browser processes the multimedia data captured from the multimedia capture event by executing instructions associated with the script event.

21. A computer readable medium containing program instructions for submitting from an electronic device multimedia data captured by the electronic device to a web application in a server comprising: providing a user interface in the electronic capture device for accessing the web application; enabling the user interface to detect and identify a multimedia capture event originating from the electronic device; after the multimedia capture event has been detected and identified, automatically processing by the user interface multimedia data captured during the multimedia capture event; and transmitting the multimedia data to the web application via the user interface.

22. The computer readable medium of claim 21 , wherein providing the user interface includes providing a web browser for displaying and interpreting an interactive web page from the server, wherein the interactive web page includes a form element in a form that allows the user to submit requested data to the server.

23. The computer readable medium of claim 22, wherein the form element includes at least one attribute related to the data to be submitted via the form element.

24. The computer readable medium of claim 23, wherein the at least one attribute includes a capture input type, wherein the capture input type explicitly requires multimedia data.

25. The computer readable medium of claim 23, wherein the at least one attribute defines one or more valid sources from which the data can originate.

26. The computer readable medium of claim 23 further including: after detecting and identifying the multimedia capture event, determining by the browser whether the multimedia data captured during the multimedia capture event is compatible with the at least one attribute defined in the form element; and if the multimedia data is not compatible, ignoring the multimedia capture event.

27. The computer readable medium of claim 26, wherein if the multimedia data is compatible, automatically associating the multimedia data with the form element.

28. The computer readable medium of claim 23 further including defining a script event directly related to the multimedia capture event, such that the script event is responsive to the multimedia capture event.

29. The computer readable medium of claim 28, wherein the at least one attribute in the form element includes the script event related to the multimedia capture event.

30. The computer readable medium of claim 29, wherein processing the multimedia data captured from the multimedia capture event includes executing instructions associated with the script event.

31. A method for associating multimedia data captured by an electronic device with a form comprising: providing a form element in the form that is compatible with a multimedia capture event; detecting and identifying a multimedia capture event originating from the electronic device; and in response to detecting and identifying the multimedia capture event, automatically associating multimedia data captured during the multimedia capture event with the form element.

32. A system for associating multimedia data captured by an electronic device with a form comprising; a browser in the electronic device¹ for receiving and displaying the form, wherein the form includes a form element that is compatible with a multimedia capture event; and an input handler in the browser for detecting and identifying a multimedia capture event originating from the electronic device, wherein in response to the input handler detecting and identifying the multimedia capture event, the browser automatically associates multimedia data captured during the multimedia capture event with the form element.