HK1203671B - Portable touch screen device, method, and graphical user interface for using emoji characters - Google Patents

Description

Portable touch screen device, method and graphical user interface using emoticons

Divisional Application Instructions

This application is a divisional application of the Chinese invention patent application with the application date of October 12, 2009, application number 200980152822.9 , and invention name “Portable touch screen device, method and graphical user interface using emoticons”.

Technical Field

The disclosed embodiments relate generally to portable electronic devices that provide emoji characters for electronic messages and documents, and more particularly, to portable electronic devices with touch screen displays that provide emoji characters for electronic messages and documents.

Background Art

As portable electronic devices become more miniaturized and the number of functions performed by a given device increases, designing a user interface that allows users to easily interact with multi-function devices becomes a significant challenge. This challenge is particularly evident for handheld portable devices with touch-sensitive displays, which have screens much smaller than those of desktop or laptop computers. This situation is particularly disadvantageous because the user interface is the means by which the user receives not only content but also responses to user actions or behaviors attempted by the user, including accessing features, tools, and functions of the device. According to reports, some portable communication devices (e.g., mobile phones, sometimes referred to as cell phones, cellular phones, and mobile phones) have resorted to adding more buttons, increasing the density of buttons, overloading buttons with functions, or using complex menu systems to allow users to access, store, and manipulate data. These conventional user interfaces often result in users having to memorize complex key sequences and menu hierarchies.

Many conventional user interfaces, such as those that include physical buttons, are also inflexible. This can hinder configuration and/or adjustment of the user interface by an application running on the portable device or by the user. When coupled with the time-consuming requirement of memorizing multiple key sequences and menu hierarchies and the difficulty of activating a desired button, this inflexibility can frustrate most users.

In recent years, portable electronic devices that support the use of emoticons have been developed. Emoticons are often used in online forums, email messages, instant messages (e.g., SMS text messages), notes, and online games. There are a large and growing number of emoticons. However, in existing portable electronic devices with small displays, displaying, organizing, and using a large number of emoticons is quite cumbersome.

Therefore, there is a need for a portable electronic device having a touch screen display with a more transparent and effective user interface for using emoticons. Such an interface allows a user to perform tasks involving emoticons more quickly and efficiently, thereby conserving power and increasing the time between battery charges in the portable electronic device. In addition, such an interface increases the effectiveness and user satisfaction of the portable electronic device.

Summary of the Invention

The above shortcomings and other problems associated with user interfaces for providing and using emoticons on portable devices are reduced by the disclosed portable multifunction device. In some embodiments, the device has a touch-sensitive display (also known as a "touch screen") with a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction groups stored in the memory for performing various functions. In some embodiments, a user interacts with the GUI primarily through finger contacts and gestures on the touch-sensitive display. In some embodiments, the functions may include telephony, video conferencing, email, instant messaging, blogging, digital photography, digital video, web browsing, digital music playback, and/or digital video playback. Instructions for performing these functions may be included in a computer program product (e.g., a computer-readable storage medium having instructions stored therein) configured to be executed by one or more processors.

According to some embodiments, a computer-implemented method is performed on a portable electronic device having a touch screen display. The computer-implemented method includes simultaneously displaying: a character input area operable to display text character input and emoticon character input selected by a user; a keyboard display area; and a plurality of emoticon category icons. As used in the specification and claims, text characters refer to alphanumeric characters, Chinese characters (sinographs), Japanese kanji or kana symbols, and/or other written human language characters. As used in the specification and claims, emoticon characters refer to still and animated picture characters that are not text characters in a written human language. The emoticon category icons correspond to the plurality of emoticon categories. The computer-implemented method further includes detecting a gesture on a responsive emoticon category icon from a plurality of emoticon category icons; and in response to detecting the gesture on the corresponding emoticon category icon, simultaneously displaying: a first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, wherein the corresponding emoticon category includes a plurality of emoticon character key subsets configured to be individually displayed as a sequence of emoticon character key subsets in the keyboard display area; and a plurality of subset sequence marker icons for the corresponding emoticon category, wherein the subset sequence marker icons provide information related to the number of subsets in the corresponding emoticon category and the position of the displayed subsets in the sequence of emoticon character key subsets in the keyboard display area. The computer-implemented method further includes: while displaying the first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, detecting the gesture in the keyboard display area; and in response to detecting the gesture in the keyboard display area: replacing display of the first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area with display of a second subset of emoticon character keys of the corresponding emoticon category, and updating the information provided by the subset sequence marker icons to reflect that the displayed first subset of emoticon character keys has been replaced with the second subset of emoticon character keys.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; one or more processors; a memory; and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include instructions for: simultaneously displaying: a character input area operable to display text character input and emoticon character input selected by a user; a keyboard display area; and a plurality of emoticon category icons, wherein the emoticon category icons correspond to a plurality of emoticon categories. The one or more programs also include instructions for: detecting a gesture on a corresponding emoticon category icon among the plurality of emoticon category icons; in response to detecting a gesture on the corresponding emoticon category icon, simultaneously displaying: a first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, wherein the corresponding emoticon category includes a plurality of emoticon character key subsets configured to be individually displayed as an emoticon character key subset sequence in the keyboard display area; and a plurality of subset sequence marker icons for the corresponding emoticon category, wherein the subset sequence marker icons provide information regarding the number of subsets in the corresponding emoticon category and the position of the displayed subsets in the emoticon character key subset sequence in the keyboard display area. The one or more programs also include instructions for the following actions: detecting a gesture in the keyboard display area while displaying a first subset of emoticon character keys of a corresponding emoticon category in the keyboard display area; and in response to detecting a gesture in the keyboard display area: replacing the display of the first subset of emoticon character keys of the corresponding emoticon category with the display of a second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area; and updating the information provided by the subset sequence marker icon to reflect that the displayed first subset of emoticon character keys is replaced by the second subset of emoticon character keys.

According to some embodiments, a computer-readable storage medium having instructions stored therein is provided, wherein the instructions, when executed by a portable electronic device having a touch screen display, cause the portable electronic device to: simultaneously display: a character input area operable to display text character input and emoticon character input selected by a user; a keyboard display area; and a plurality of emoticon category icons, wherein the emoticon category icons correspond to a plurality of emoticon categories; detect gestures on corresponding emoticon category icons among the plurality of emoticon category icons; and in response to detecting gestures on the corresponding emoticon category icons, simultaneously display: a first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, wherein the corresponding emoticon category includes emoticon character keys configured to be displayed separately as emoticon character key sub-keys in the keyboard display area a plurality of subsets of emoticon character keys of a corresponding emoticon category in a sequence of subsets; and a plurality of subset sequence marker icons of corresponding emoticon categories, wherein the subset sequence marker icons provide information related to the number of subsets in the corresponding emoticon category and the position of the displayed subsets in the emoticon character key subset sequence in the keyboard display area; while displaying a first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, detecting a gesture in the keyboard display area; and in response to detecting the gesture in the keyboard display area: replacing the display of the first subset of emoticon character keys of the corresponding emoticon category with the display of the second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area; and updating the information provided by the subset sequence marker icons to reflect that the displayed first subset of emoticon character keys is replaced by the second subset of emoticon character keys.

According to some embodiments, a graphical user interface (GUI) on a portable electronic device with a touch screen display is provided, comprising: a character input area operable to display user-selected text character input and emoticon character input; a keyboard display area; and a plurality of emoticon category icons. The emoticon category icons correspond to a plurality of emoticon categories. Each emoticon category includes a plurality of emoticon character key subsets configured to be individually displayed as a sequence of emoticon character key subsets in the keyboard display area. The GUI further comprises: a first subset of emoticon character keys for the corresponding emoticon category; a second subset of emoticon character keys for the corresponding emoticon category; and a plurality of subset sequence marker icons for the corresponding emoticon category. The subset sequence marker icons provide information regarding the number of subsets in the corresponding emoticon category and the position of the displayed subsets in the sequence of emoticon character key subsets in the keyboard display area. The character input area, the keyboard display area, and the plurality of emoticon category icons are displayed simultaneously. In response to detecting a gesture on a corresponding emoticon category icon among the plurality of emoticon category icons, the first subset of emoticon character keys for the corresponding emoticon category and the plurality of subset sequence marker icons for the corresponding emoticon category are simultaneously displayed in the keyboard display area. In response to detecting a gesture in the keyboard display area while displaying a first subset of emoticon character keys of a corresponding emoticon category in the keyboard display area, replacing display of the first subset of emoticon character keys of the corresponding emoticon category with display of a second subset of emoticon character keys of the corresponding emoticon category, and updating information provided by the subset sequence marker icon to reflect that the displayed first subset of emoticon character keys is replaced by the second subset of emoticon character keys.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; a device for simultaneously displaying the following items: a character input area operable to display text character input and emoticon character input selected by a user, a keyboard display area, and a plurality of emoticon category icons, wherein the emoticon category icons correspond to a plurality of emoticon categories. The portable electronic device also comprises: a device for detecting a gesture on a corresponding emoticon category icon among the plurality of emoticon category icons; a device for simultaneously displaying the following items in response to detecting a gesture on the corresponding emoticon category icon: a first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, wherein the corresponding emoticon category includes a plurality of emoticon character key subsets configured to be individually displayed as an emoticon character key subset sequence in the keyboard display area, and a plurality of subset sequence marker icons of the corresponding emoticon category. The subset sequence marker icons provide information related to the number of subsets in the corresponding emoticon category and the position of the displayed subsets in the emoticon character key subset sequence in the keyboard display area. The portable electronic device also includes: a device for detecting a gesture in the keyboard display area while displaying a first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area; and a device for performing the following actions in response to detecting a gesture in the keyboard display area: replacing the display of the first subset of emoticon character keys of the corresponding emoticon category with the display of the second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, and updating the information provided by the subset sequence marker icon to reflect that the displayed first subset of emoticon character keys is replaced by the second subset of emoticon character keys.

According to some embodiments, a computer-implemented method is performed on a portable electronic device having a touch screen display while the portable electronic device is in a user interface lock mode. The method includes: receiving an incoming message from a party to a user of the portable electronic device while the touch screen display is in a power-saving state; and in response to receiving the incoming message: powering on the touch screen display and displaying at least a portion of the incoming message from the party to the user on the touch screen display. The method also includes: detecting one or more predetermined finger gestures on the touch screen display that correspond to emoticons; detecting a predetermined user action that initiates sending a response to the incoming message; and in response to detecting the predetermined user action that initiates sending a response to the incoming message, sending a response to the incoming message. The response includes emoticons corresponding to the one or more predetermined finger gestures detected on the touch screen display.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; one or more processors; a memory; and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include instructions for the following actions: while the portable electronic device is in a user interface lock mode and while the touch screen display is in a power saving state, receiving an incoming message from one party to a user of the portable electronic device; in response to receiving the incoming message: powering on the touch screen display; and displaying at least a portion of the incoming message from the party to the user on the touch screen display; detecting one or more predetermined finger gestures on the touch screen display corresponding to emoticons; detecting a predetermined user action to initiate sending a response to the incoming message; and in response to detecting the predetermined user action to initiate sending a response to the incoming message, sending a response to the incoming message. The response includes emoticons corresponding to the one or more predetermined finger gestures detected on the touch screen display.

According to some embodiments, a computer-readable storage medium having instructions stored therein is provided that, when executed by a portable electronic device having a touch screen display, causes the portable electronic device, while the portable electronic device is in a user interface lock mode, to: receive an incoming message from a party to a user of the portable electronic device while the touch screen display is in a power-saving state. The instructions further cause the portable electronic device, while the portable electronic device is in the user interface lock mode, to: power on the touch screen display and display at least a portion of the incoming message from the party to the user on the touch screen display; detect one or more predetermined finger gestures on the touch screen display that correspond to emoticons; detect a predetermined user action that initiates sending a response to the incoming message; and, in response to detecting the predetermined user action that initiates sending a response to the incoming message, send a response to the incoming message. The response includes emoticons corresponding to the one or more predetermined finger gestures detected on the touch screen display.

According to some embodiments, a graphical user interface on a portable electronic device with a touch screen display is provided, comprising: at least a portion of an incoming message from a party to a user of the portable electronic device. While the portable electronic device is in a user interface lock mode and while the touch screen display is in a power-saving state, in response to receiving an incoming message from a party to the user of the portable electronic device: powering on the touch screen display, and displaying at least a portion of the incoming message from the party to the user on the touch screen display. While the portable electronic device is in the user interface lock mode, detecting one or more predetermined finger gestures on the touch screen display that correspond to emoticons. While the portable electronic device is in the user interface lock mode, in response to detecting a predetermined user action that initiates sending a response to the incoming message, sending a response to the incoming message. The response includes emoticons corresponding to the one or more predetermined finger gestures detected on the touch screen display.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; and, while the portable electronic device is in a user interface lock mode: means for receiving an incoming message from a party to a user of the portable electronic device while the touch screen display is in a power-saving state; in response to receiving the incoming message: means for powering the touch screen display; and means for displaying at least a portion of the incoming message from the party to the user on the touch screen display; means for detecting one or more predetermined finger gestures on the touch screen display corresponding to emoticons; means for detecting a predetermined user action that initiates sending a response to the incoming message; and means for sending a response to the incoming message in response to detecting the predetermined user action that initiates sending the response to the incoming message. The response includes emoticons corresponding to the one or more predetermined finger gestures detected on the touch screen display.

According to some embodiments, a computer-implemented method is provided, comprising: on a portable electronic device having a touch screen display, while the portable electronic device is in a user interface lock mode: receiving an incoming message from a party to a user of the portable electronic device while the touch screen display is in a power-saving state; in response to receiving the incoming message: powering the touch screen display, displaying at least a portion of the incoming message from the party to the user and a plurality of predetermined responses to the incoming message on the touch screen display, wherein the predetermined responses include a plurality of emoticons; detecting a predetermined user action for selecting a predetermined response from the plurality of predetermined responses; detecting a predetermined user action for initiating sending of the selected predetermined response to the incoming message; and in response to detecting the predetermined user action for initiating sending of the selected predetermined response to the incoming message, sending the selected predetermined response to the incoming message, wherein the selected predetermined response includes an emoticon from the plurality of emoticons.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include instructions for the following actions: while the portable electronic device is in a user interface lock mode: while the touch screen display is in a power saving state, receiving an incoming message from a party to a user of the portable electronic device; in response to receiving the incoming message: powering the touch screen display, displaying at least a portion of the incoming message from the party to the user on the touch screen display, and displaying a plurality of predetermined responses to the incoming message, wherein the predetermined responses include a plurality of emoticons; detecting a predetermined user action for selecting a predetermined response from the plurality of predetermined responses; detecting a predetermined user action for initiating sending of the selected predetermined response to the incoming message; and in response to detecting the predetermined user action for initiating sending of the selected predetermined response to the incoming message, sending the selected predetermined response to the incoming message. The selected predetermined response includes an emoticon from the plurality of emoticons.

According to some embodiments, a computer-readable storage medium having instructions stored therein is provided that, when executed by a portable electronic device having a touch screen display, causes the portable electronic device, while the portable electronic device is in a user interface lock mode, to: receive an incoming message from a party to a user of the portable electronic device while the touch screen display is in a power-saving state; in response to receiving the incoming message: powering the touch screen display, displaying at least a portion of the incoming message from the party to the user on the touch screen display, and displaying a plurality of predetermined responses to the incoming message, wherein the predetermined responses include a plurality of emoticons; detecting a predetermined user action for selecting a predetermined response from the plurality of predetermined responses; detecting a predetermined user action for initiating sending of the selected predetermined response to the incoming message; and in response to detecting the predetermined user action for initiating sending of the selected predetermined response to the incoming message, sending the selected predetermined response to the incoming message, wherein the selected predetermined response includes an emoticon from the plurality of emoticons.

According to some embodiments, a graphical user interface on a portable electronic device with a touch screen display is provided, comprising: at least a portion of an incoming message from a party to a user of the portable electronic device, and a plurality of predetermined responses to the incoming message. The predetermined responses include a plurality of emoticons. While the portable electronic device is in a user interface lock mode: while the touch screen display is in a power saving state, in response to receiving an incoming message from the party to the user of the portable electronic device: powering on the touch screen display, displaying at least a portion of the incoming message from the party to the user, and a plurality of predetermined responses to the incoming message on the touch screen display; detecting a predetermined user action for selecting a predetermined response from the plurality of predetermined responses; and in response to detecting a predetermined user action for initiating sending of the selected predetermined response to the incoming message, sending the selected predetermined response to the incoming message. The selected predetermined response includes an emoticon from the plurality of emoticons.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; and, while the portable electronic device is in a user interface lock mode: means for receiving an incoming message from a party to a user of the portable electronic device while the touch screen display is in a power-saving state; in response to receiving the incoming message: means for powering the touch screen display; means for displaying at least a portion of the incoming message from the party to the user on the touch screen display; and means for displaying a plurality of predetermined responses to the incoming message, wherein the predetermined responses include a plurality of emoticons; means for detecting a predetermined user action for selecting a predetermined response from the plurality of predetermined responses; means for detecting a predetermined user action for initiating sending of the selected predetermined response to the incoming message; and means for sending the selected predetermined response to the incoming message in response to detecting the predetermined user action for initiating sending of the selected predetermined response to the incoming message. The selected predetermined response includes an emoticon from the plurality of emoticons.

In accordance with some embodiments, a computer-implemented method on a portable electronic device having a touch screen display includes: while the portable electronic device is in a user interface lock mode: while the touch screen display is in a power-saving state, receiving an incoming message from a party to a user of the portable electronic device; in response to receiving the incoming message: powering the touch screen display and displaying at least a portion of the incoming message from the party to the user on the touch screen display; detecting a predetermined user action that initiates display of a plurality of predetermined responses to the incoming message, wherein the predetermined responses include a plurality of emoticons; in response to detecting the predetermined user action that initiates display of the plurality of predetermined responses to the incoming message, displaying the plurality of predetermined responses; detecting a predetermined user action that selects a predetermined response from the plurality of predetermined responses; detecting a predetermined user action that initiates sending of the selected predetermined response to the incoming message; and in response to detecting the predetermined user action that initiates sending of the selected predetermined response to the incoming message, sending the selected predetermined response to the incoming message, wherein the selected predetermined response includes an emoticon from the plurality of emoticons.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include instructions for: while the portable electronic device is in a user interface lock mode; while the touch screen display is in a power-saving state, receiving an incoming message from a party to a user of the portable electronic device; in response to receiving the incoming message: powering the touch screen display and displaying at least a portion of the incoming message from the party to the user on the touch screen display; detecting a predetermined user action that initiates display of a plurality of predetermined responses to the incoming message, wherein the predetermined responses include a plurality of emoticons; in response to detecting the predetermined user action that initiates display of the plurality of predetermined responses to the incoming message, displaying the plurality of predetermined responses; detecting a predetermined user action that selects a predetermined response from the plurality of predetermined responses; detecting a predetermined user action that initiates sending the selected predetermined response to the incoming message; and in response to detecting the predetermined user action that initiates sending the selected predetermined response to the incoming message, sending the selected predetermined response to the incoming message. The selected predetermined response includes an emoticon from the plurality of emoticons.

According to some embodiments, a computer-readable storage medium having instructions stored therein, the instructions, when executed by a portable electronic device having a touch screen display, causes the portable electronic device, while the portable electronic device is in a user interface lock mode, to: receive an incoming message from a party to a user of the portable electronic device while the touch screen display is in a power-saving state; in response to receiving the incoming message: powering the touch screen display and displaying at least a portion of the incoming message from the party to the user on the touch screen display; detecting a predetermined user action that initiates display of a plurality of predetermined responses to the incoming message, wherein the predetermined responses include a plurality of emoticons; in response to detecting the predetermined user action that initiates display of the plurality of predetermined responses to the incoming message, displaying the plurality of predetermined responses; detecting a predetermined user action that selects a predetermined response from the plurality of predetermined responses; detecting a predetermined user action that initiates sending of the selected predetermined response to the incoming message; and in response to detecting the predetermined user action that initiates sending of the selected predetermined response to the incoming message, sending the selected predetermined response to the incoming message. The selected predetermined response includes an emoticon from the plurality of emoticons.

According to some embodiments, a graphical user interface on a portable electronic device with a touch screen display is provided, comprising: at least a portion of an incoming message from a party to a user of the portable electronic device, and a plurality of predetermined responses to the incoming message. The predetermined responses include a plurality of emoticons. While the portable electronic device is in a user interface lock mode: while the touch screen display is in a power saving state, in response to receiving an incoming message from the party to the user of the portable electronic device: powering on the touch screen display and displaying at least a portion of the incoming message from the party to the user on the touch screen display; in response to detecting a predetermined user action that initiates display of a plurality of predetermined responses to the incoming message, displaying the plurality of predetermined responses; detecting a predetermined user action for selecting a predetermined response from the plurality of predetermined responses; and in response to detecting a predetermined user action that initiates sending of the selected predetermined response to the incoming message, sending the selected predetermined response to the incoming message. The selected predetermined response includes an emoticon from the plurality of emoticons.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display, and, while the portable electronic device is in a user interface lock mode: means for receiving an incoming message from a party to a user of the portable electronic device while the touch screen display is in a power-saving state; in response to receiving the incoming message: means for powering the touch screen display, and means for displaying at least a portion of the incoming message from the party to the user on the touch screen display; means for detecting a predetermined user action that initiates display of a plurality of predetermined responses to the incoming message, wherein the predetermined responses include a plurality of emoticons; in response to detecting the predetermined user action that initiates display of the plurality of predetermined responses to the incoming message, means for displaying the plurality of predetermined responses; means for detecting the predetermined user action that selects a predetermined response from the plurality of predetermined responses; means for detecting the predetermined user action that initiates sending of the selected predetermined response to the incoming message; and in response to detecting the predetermined user action that initiates sending of the selected predetermined response to the incoming message, means for sending the selected predetermined response to the incoming message, wherein the selected predetermined response includes an emoticon from the plurality of emoticons.

According to some embodiments, a computer-implemented method on a portable electronic device having a touch screen display is provided, comprising: detecting a user-defined gesture on the touch screen display while the portable electronic device is in a user-defined gesture recording mode; generating a representation of the user-defined gesture; associating the representation of the user-defined gesture with an emoticon character; and storing the representation of the user-defined gesture and the association of the representation of the user-defined gesture with the emoticon character.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include instructions for: detecting a user-defined gesture on the touch screen display while the portable electronic device is in a user-defined gesture recording mode; generating a representation of the user-defined gesture; associating the representation of the user-defined gesture with an emoticon; and storing the representation of the user-defined gesture and the association between the representation of the user-defined gesture and the emoticon.

According to some embodiments, a computer-readable storage medium having instructions stored therein is provided, which, when executed by a portable electronic device having a touch screen display, causes the portable electronic device to: detect a user-defined gesture on the touch screen display while the portable electronic device is in a user-defined gesture recording mode; generate a representation of the user-defined gesture; associate the representation of the user-defined gesture with an emoticon character; and store the representation of the user-defined gesture and the association of the representation of the user-defined gesture with the emoticon character.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; a device for detecting a user-defined gesture on the touch screen display while the portable electronic device is in a user-defined gesture recording mode; a device for generating a representation of the user-defined gesture; a device for associating the representation of the user-defined gesture with an emoticon character; and a device for storing the representation of the user-defined gesture and the association of the representation of the user-defined gesture with the emoticon character.

According to some embodiments, a computer-implemented method on a portable electronic device with a touch screen display is provided, comprising: receiving an electronic message. The electronic message comprises: instructions for launching a geo-mapping application, location data representing a location on a map, and emoticon data representing an emoticon character. The emoticon data is associated with the location data. The method further comprises: displaying the electronic message on the touch screen display; detecting a predetermined user action on the displayed electronic message for launching the geo-mapping application; and in response to detecting the predetermined user action for launching the geo-mapping application: displaying a map in the geo-mapping application; and displaying the emoticon character at a location on the map corresponding to the location data.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include instructions for the following actions: receiving an electronic message, wherein the electronic message includes: instructions for launching a geo-mapping application, location data representing a location on a map, and emoticon data representing an emoticon character. The emoticon data is associated with the location data. The one or more programs also include instructions for the following actions: displaying the electronic message on the touch screen display; detecting a predetermined user action on the displayed electronic message for launching the geo-mapping application; and in response to detecting the predetermined user action to launch the geo-mapping application: displaying a map in the geo-mapping application, and displaying an emoticon character at a location on the map corresponding to the location data.

According to some embodiments, a computer-readable storage medium having instructions stored therein is provided that, when executed by a portable electronic device having a touch screen display, causes the portable electronic device to receive an electronic message. The electronic message includes: instructions for launching a geo-mapping application; location data representing a location on a map; and emoticon data representing an emoticon character. The emoticon data is associated with the location data. The instructions further cause the portable electronic device to: display the electronic message on the touch screen display; detect a predetermined user action on the displayed electronic message for launching the geo-mapping application; and, in response to detecting the predetermined user action for launching the geo-mapping application, display a map in the geo-mapping application and display the emoticon character at a location on the map corresponding to the location data.

According to some embodiments, a graphical user interface on a portable electronic device with a touch screen display is provided, comprising an electronic message, a map, and an emoticon. The electronic message is received by the portable electronic device. The electronic message comprises: an instruction for launching a geographic mapping application; location data representing a location on the map; and emoticon data representing an emoticon. The emoticon data is associated with the location data. The electronic message is displayed on the touch screen display. In response to detecting a predetermined user action in the displayed electronic message for launching the geographic mapping application: displaying a map in the geographic mapping application; and displaying the emoticon at a location on the map corresponding to the location data.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; means for receiving an electronic message, wherein the electronic message comprises: an instruction for launching a geo-mapping application, location data representing a location on a map, and emoticon data representing an emoticon character, wherein the emoticon data is associated with the location data. The device further comprises: means for displaying the electronic message on the touch screen display; means for detecting a predetermined user action on the displayed electronic message for launching the geo-mapping application; and in response to detecting the predetermined user action for launching the geo-mapping application: means for displaying a map in the geo-mapping application, and means for displaying the emoticon character at a location on the map corresponding to the location data.

Thus, the present invention provides a simple, effective, and easy-to-use interface for using emoticons on a portable electronic device with a touch-sensitive display. This interface conserves power by enabling users to complete tasks more quickly, thereby increasing the time between battery charges. This interface also improves user satisfaction with the portable electronic device.

According to some embodiments, a computer-implemented method is provided, comprising: at a portable electronic device having a touch screen display, simultaneously displaying: a character input area operable to display text character input and emoticon character input selected by a user, a keyboard display area operable to display text character keys and emoticon character keys, a first plurality of emoticon character keys in the keyboard display area, and a plurality of emoticon category icons in the keyboard display area, wherein the emoticon category icons correspond to a plurality of emoticon categories; detecting a gesture on a corresponding emoticon category icon among the plurality of emoticon category icons; in response to detecting a gesture on the corresponding emoticon category icon: maintaining display of the character input area; maintaining display of the plurality of emoticon category icons in the keyboard display area; stopping displaying the first plurality of emoticon character keys in the keyboard display area; and simultaneously displaying: a first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, wherein the corresponding emoticon category includes a gesturing icon configured to be displayed on the keyboard display a plurality of emoticon character key subsets displayed separately as emoticon character key subset sequences in the keyboard display area; and a plurality of subset sequence marking icons of corresponding emoticon categories, wherein the subset sequence marking icons provide information related to the total number of subsets in the corresponding emoticon category configured to be displayed separately and the position of the displayed subsets in the emoticon character key subset sequence in the keyboard display area; while displaying a first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, detecting a gesture in the keyboard display area; and in response to detecting a gesture in the keyboard display area: maintaining the display of the character input area; maintaining the display of the plurality of emoticon category icons in the keyboard display area; replacing the display of the first subset of emoticon character keys of the corresponding emoticon category with the display of the second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, and updating the information provided by the subset sequence marking icons to reflect that the displayed first subset of emoticon character keys is replaced by the second subset of emoticon character keys.

According to some embodiments, a portable electronic device is provided, comprising: a touch screen display; one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: simultaneously displaying a character input area operable to display text character input and emoticon character input selected by a user, a keyboard display area operable to display text character keys and emoticon character keys, a first plurality of emoticon character keys in the keyboard display area, and a plurality of emoticon categories in the keyboard display area icon, wherein the expression category icon corresponds to a plurality of expression categories; detecting a gesture on a corresponding expression category icon among the plurality of expression category icons; in response to detecting the gesture on the corresponding expression category icon: maintaining the display of the character input area; maintaining the display of the plurality of expression category icons in the keyboard display area; stopping displaying the first plurality of expression character keys in the keyboard display area; and simultaneously displaying: a first subset of the expression character keys of the corresponding expression category in the keyboard display area, wherein the corresponding expression category includes a plurality of expression character key subsets configured to be individually displayed as an expression character key subset sequence in the keyboard display area; and a plurality of subset sequence marker icons of the corresponding expression category, wherein the subset sequence marker icons provide a first subset of the expression character keys of the corresponding expression category Information related to the total number of subsets configured for separate display and the position of the displayed subsets in the emoticon character key subset sequence in the keyboard display area; detecting a gesture in the keyboard display area while displaying a first subset of emoticon character keys of a corresponding emoticon category in the keyboard display area; and in response to detecting a gesture in the keyboard display area: maintaining display of the character input area; maintaining display of multiple emoticon category icons in the keyboard display area; replacing display of the first subset of emoticon character keys of the corresponding emoticon category with display of the second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, and updating information provided by the subset sequence marker icon to reflect that the displayed first subset of emoticon character keys is replaced by the second subset of emoticon character keys.

According to some embodiments, a device is provided, comprising: a device for simultaneously displaying, at a portable electronic device having a touch screen display, a character input area operable to display text character input and emoticon character input selected by a user, a keyboard display area operable to display text character keys and emoticon character keys, a first plurality of emoticon character keys in the keyboard display area, and a plurality of emoticon category icons in the keyboard display area, wherein the emoticon category icons correspond to a plurality of emoticon categories; a device for detecting gestures on corresponding emoticon category icons among the plurality of emoticon category icons; a device that is enabled in response to detecting a gesture on a corresponding emoticon category icon, comprising: a device for maintaining display of the character input area; a device for maintaining display of the plurality of emoticon category icons in the keyboard display area; a device for stopping displaying the first plurality of emoticon character keys in the keyboard display area; and a device for simultaneously displaying: a first subset of emoticon character keys of a corresponding emoticon category in the keyboard display area, wherein the corresponding emoticon category includes a key configured to be displayed on the keyboard display A plurality of emoticon character key subsets are separately displayed as emoticon character key subset sequences in an area; and a plurality of subset sequence marker icons of corresponding emoticon categories, wherein the subset sequence marker icons provide information related to the total number of subsets in the corresponding emoticon category configured to be separately displayed and the position of the displayed subsets in the emoticon character key subset sequence in the keyboard display area; a device for detecting gestures in the keyboard display area while displaying a first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area; and a device that is enabled in response to detecting a gesture in the keyboard display area: a device for maintaining the display of a character input area; a device for maintaining the display of a plurality of emoticon category icons in the keyboard display area; a device for replacing the display of the first subset of emoticon character keys of the corresponding emoticon category with the display of the second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, and a device for updating the information provided by the subset sequence marker icons to reflect that the displayed first subset of emoticon character keys is replaced by the second subset of emoticon character keys.

According to some embodiments, a graphical user interface is provided on a portable electronic device having a touch screen display, a memory, and one or more processors for executing one or more programs stored in the memory, the graphical user interface comprising: a character input area operable to display text character input and emoticon character input selected by a user, a keyboard display area operable to display text character keys and emoticon character keys, a first plurality of emoticon character keys in the keyboard display area, a plurality of emoticon category icons in the keyboard display area, wherein the emoticon category icons correspond to a plurality of emoticon categories, wherein the respective emoticon categories include a plurality of emoticon character key subsets configured to be individually displayed as a sequence of emoticon character key subsets in the keyboard display area, and ：A character input area, a keyboard display area, a first plurality of emoticon character keys, and a plurality of emoticon category icons are displayed simultaneously; in response to detecting a gesture on a corresponding emoticon category icon among the plurality of emoticon category icons: maintaining the display of the character input area; maintaining the display of the plurality of emoticon category icons in the keyboard display area; stopping displaying the first plurality of emoticon character keys in the keyboard display area; and simultaneously displaying a first subset of emoticon character keys of the corresponding emoticon category and a plurality of subset sequence marker icons of the corresponding emoticon category in the keyboard display area, wherein the subset sequence marker icon provides information about the total number of subsets in the corresponding emoticon category that are configured to be displayed separately and the position of the displayed subsets in the emoticon character key subset sequence in the keyboard display area; and in response to A gesture is detected in the keyboard display area while a first subset of emoticon character keys of a corresponding emoticon category is displayed in the display area: the display of the character input area is maintained; the display of multiple emoticon category icons in the keyboard display area is maintained; the display of the first subset of emoticon character keys of the corresponding emoticon category is replaced with the display of the second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area, and the information provided by the subset sequence marker icon is updated to reflect that the displayed first subset of emoticon character keys is replaced by the second subset of emoticon character keys.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the above-described embodiments and additional embodiments of the present invention, reference is made to the following detailed description in conjunction with the accompanying drawings, in which like reference numerals refer to corresponding parts throughout.

1A and 1B are block diagrams illustrating portable multifunction devices with touch-sensitive displays in accordance with some embodiments.

FIG2 illustrates a portable multifunction device with a touch screen in accordance with some embodiments.

3 illustrates an exemplary user interface for unlocking a portable electronic device in accordance with some embodiments.

4A-4B illustrate exemplary user interfaces for an applications menu on a portable multifunction device in accordance with some embodiments.

5A-5G illustrate exemplary user interfaces for setting up and organizing emoticon keys on a portable electronic device equipped with a touch-sensitive display in accordance with some embodiments.

6A-6E illustrate exemplary user interfaces for navigating between subsets of emoticon characters within a particular emoticon category and entering emoticon characters on a portable electronic device equipped with a touch-sensitive display in accordance with some embodiments.

7A-7E illustrate exemplary user interfaces on a portable electronic device equipped with a touch-sensitive display for entering emoticon characters and sending electronic messages including emoticon characters in accordance with some embodiments.

8A-8C illustrate exemplary user interfaces for reconfiguring emoticon keys in a software keyboard on a portable electronic device equipped with a touch-sensitive display in accordance with some embodiments.

9A-9H illustrate exemplary user interfaces for responding to messages with emoticons while a portable electronic device equipped with a touch-sensitive display is in a user interface lock mode in accordance with some embodiments.

10A-10E illustrate exemplary user interfaces for assigning user-defined gestures to emoticons on a portable electronic device equipped with a touch-sensitive display in accordance with some embodiments.

11A-11B illustrate exemplary user interfaces for using emoticons as location indicators on a map via an electronic message in accordance with some embodiments.

12A-12F are flow charts illustrating methods of operating an emoticon keyboard on a portable electronic device having a touch-sensitive display in accordance with some embodiments.

13 is a flow diagram illustrating a method for responding to a message with an emoticon while a portable electronic device equipped with a touch-sensitive display is in a user interface lock mode in accordance with some embodiments.

14 is a flow diagram illustrating a method for responding to a message with an emoticon while a portable electronic device equipped with a touch-sensitive display is in a user interface lock mode in accordance with some embodiments.

15 is a flow diagram illustrating a method for responding to a message with an emoticon while a portable electronic device equipped with a touch-sensitive display is in a user interface lock mode in accordance with some embodiments.

16 is a flow chart illustrating a method of assigning user-defined gestures to emoticons on a portable electronic device equipped with a touch-sensitive display in accordance with some embodiments.

17 is a flow diagram illustrating a method for using emoticons as location indicators in a map via an electronic message, in accordance with some embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, those skilled in the art will readily appreciate that the present invention can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

It should also be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish elements from each other. For example, a first gesture may be referred to as a second gesture, and similarly, a second gesture may be referred to as a first gesture without departing from the scope of the present invention.

The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context otherwise specifically indicates, the singular forms "a", "a kind of" and "the" are also intended to include plural forms. It should also be understood that the terms "and/or" used herein refer to and include any and all possible combinations of one or more of the items listed in association. It should also be understood that the terms "comprising" and/or "including" used in this specification stipulate the existence of stated features, integers, steps, operations, elements and/or parts, but do not exclude the existence or addition of one or more other characteristics, integers, steps, operations, elements, parts and/or their groups.

As used herein, the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [stated condition or event] is detected" may be interpreted to mean "upon determination" or "in response to determining" or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]," depending on the context.

Embodiments of portable multifunction devices, user interfaces for these devices, and associated processes for using these devices are described. In some embodiments, the devices are portable communication devices such as mobile phones that also include other functions such as PDA and/or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPod and iPod devices from Apple Computer, Int. of Cupertino, California.

In addition to a touch screen or a virtual click wheel displayed on a touch screen, a user interface may also include a physical click wheel. A click wheel is a user interface component that can provide navigation commands based on the angular displacement of the wheel or the point of contact between the wheel and the user of the device. A click wheel can also be used to provide a user command corresponding to the selection of one or more items, for example, when the user of the device presses at least a portion of the wheel or the center of the wheel. Alternatively, breaking contact with the click wheel image on the touch screen surface can indicate a user command corresponding to a selection. For simplicity, the following discussion uses a portable multifunction device including a touch screen as an exemplary embodiment. However, it should be understood that some of the user interfaces and related processes can be applied to other portable devices.

The device may support various applications such as one or more of the following: a phone application, a video conferencing application, an email application, an instant messaging application, a blogging application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.

Various applications that can be executed on the device can use at least one common physical user interface device, such as a touch screen. One or more functions of the touch screen and corresponding information displayed on the device can be adjusted and/or changed from one application to the next and/or within a corresponding application. In this way, the common physical architecture of the device (such as a touch screen) can support various applications with intuitive and transparent user interfaces.

The user interface may include a soft keyboard embodiment for inputting or operable to input text characters and emoticon characters. The soft keyboard embodiment may include standard (QWERTY) and/or non-standard configurations of symbols displayed on icons of keyboards, such as those described in U.S. patent application Ser. No. 11/459,606, "Keyboards For Portable Electronic Devices," filed on July 24, 2006, and U.S. patent application Ser. No. 11/459,615, "Touch Screen Keyboards For Portable Electronic Devices," filed on July 24, 2006, the entire contents of which are incorporated herein by reference. Keyboard embodiments may include a smaller number of icons (or soft keys) than the number of keys in existing physical keyboards, such as those used in typewriters. This makes it easier for a user to select one or more icons in the keyboard, and thereby select one or more corresponding symbols. The keyboard embodiment may be adaptive. For example, the displayed icons may be modified based on a user action, such as selecting one or more icons and/or one or more corresponding symbols. One or more applications on a portable device may utilize common and/or different keyboard embodiments. Thus, the keyboard embodiment used may be customized for at least some of the applications. In some embodiments, one or more keyboard embodiments may be customized for the respective user. For example, one or more keyboard embodiments may be customized for a respective user based on the respective user's word usage history (dictionary, slang, personal usage). When using soft keyboard embodiments, some of the keyboard embodiments may be adjusted to reduce the likelihood of user error when selecting one or more icons and, thereby, one or more symbols. Soft keyboards for inputting or operable to input emoticon characters are discussed in greater detail below.

Now turning our attention to embodiments of the device. Figures 1A and 1B are block diagrams illustrating a portable multifunction device 100 having a touch-sensitive display 112 according to some embodiments. Touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and may also be referred to as or referred to as a touch-sensitive display system. Device 100 may include memory 102 (which may include one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, peripheral interfaces 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input/output (I/O) subsystem 106, other input or control devices 116, and external ports 124. Device 100 may include one or more optical sensors 164. These components may communicate over one or more communication buses or signal lines 103.

It should be understood that device 100 is only one example of a portable multifunction device 100, and that device 100 may have more or fewer components than shown, may combine two or more components, or may have a different configuration or arrangement of components. The various components shown in Figures 1A and 1B may be implemented in hardware, including one or more signal processing and/or application specific integrated circuits, software, or a combination of hardware and software.

The memory 102 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Access to the memory 102 by other components of the device 100, such as the CPU 120 and the peripheral interface 118, may be controlled by a memory controller 122.

Peripherals interface 118 couples the device's input and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and/or sets of instructions stored in memory 102 to perform various functions of device 100 and process data.

In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 may be implemented on a single chip, such as chip 104. In some other embodiments, they may be implemented on separate chips.

RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts electromagnetic signals into/from electrical signals and communicates with communication networks and other communication devices via the electromagnetic signals. RF circuitry 108 may include well-known circuitry for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, and memory. RF circuitry 108 may communicate with networks and other devices via wireless communications, such as the Internet, also known as the World Wide Web (WWW), intranets, and/or wireless networks, such as cellular telephone networks, wireless local area networks (LANs), and/or metropolitan area networks (MANs). Wireless communication may use any of a number of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed Downlink Packet Access (HSDPA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and/or IEEE 802.11n), Voice over IP (VoIP), Wi-MAX, protocols for email (e.g., Internet Message Access Protocol (IMAP) and/or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP)), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Services (IMPS), and more. Service (IMPS)) and/or Short Message Service (SMS)) or any other suitable communication protocol including communication protocols that have not been developed as of the filing date of this document.

The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves that can be heard by a person. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral interface 118 for processing. The audio data can be retrieved from and/or transmitted to the memory 102 and/or the RF circuit 108 via the peripheral interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212, FIG. 2). The headset jack provides an interface between the audio circuitry 110 and a removable audio input/output peripheral, such as an output-only headset or a headset having both output (e.g., a headset for one or both ears) and input (e.g., a microphone).

The I/O subsystem 106 couples input/output peripherals on the device 100, such as the touch screen 112 and other input/control devices 116, to the peripherals interface 118. The I/O subsystem 106 may include a display controller 156 and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive and send electrical signals to and from the other input or control devices 116. The other input/control devices 116 may include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller(s) 160 may be coupled to (or not coupled to) any of a keyboard, an infrared port, a USB port, and a pointing device such as a mouse. The one or more buttons (e.g., 208, FIG. 2 ) may include up/down buttons for volume control of the speaker 111 and/or microphone 113. The one or more buttons may include a push button (e.g., 206, FIG. 2 ). As described in U.S. patent application Ser. No. 11/322,549, “Unlocking a Device by Performing Gestures on an Unlock Image,” filed Dec. 23, 2005, which is incorporated herein by reference in its entirety, a quick press of the push button can release a lock on the touch screen 112 or initiate a process of using gestures on the touch screen to unlock the device. A longer press of the push button (e.g., 206) can turn the power of the device 100 on or off. The user can customize one or more functions of the button. The touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

The touch-sensitive touch screen 112 provides an input interface and an output interface between the device and a user. The display controller 156 receives and/or sends electrical signals to/from the touch screen 112. The touch screen 112 displays visual output to the user. The visual output may include graphics, text, icons, videos, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output may correspond to user interface objects.

The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that receives input from the user based on tactile and/or sensory contact. The touch screen 112 and display controller 156 (together with any associated modules and/or sets of instructions in memory 102) detect contact (and any movement or breakage of contact) on the touch screen 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

Although other display technologies may be used in other embodiments, the touch screen 112 may use LCD (liquid crystal display) technology or LPD (light emitting polymer display) technology. The touch screen 112 and the display controller 156 may detect contact and any movement or cessation thereof by using any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112. In an exemplary embodiment, projected mutual capacitance sensing technology such as that found in the iPhone and iPod from Apple Computer, Inc. of Cupertino, California is used.

The touch-sensitive display in some embodiments of the touch screen 112 may be similar to the multi-touch sensitive tablets described in U.S. Patent 6,323,846 (Westerman et al.), U.S. Patent 6,570,557 (Westerman et al.), and/or U.S. Patent 6,677,932 (Westerman), and/or U.S. Patent Publication 2002/0015024A1, each of which is incorporated herein by reference in its entirety. However, the touch screen 112 displays visual output from the portable device 100, whereas a touch-sensitive tablet does not provide visual output.

The touch-sensitive display in some embodiments of the touch screen 112 can be as described in the following applications: (1) U.S. patent application No. 11/381,313, filed on May 2, 2006, “Multipoint Touch Surface Controller”; (2) U.S. patent application No. 10/840,862, filed on May 6, 2004, “Multipoint Touchscreen”; (3) U.S. patent application No. 10/903,964, filed on July 30, 2004, “Gestures For Touch Sensitive Input Devices”; (4) U.S. patent application No. 11/048,264, filed on January 31, 2005, “Gestures For Touch Sensitive Input Devices”; (5) U.S. patent application No. 11/038,590, filed on January 18, 2005, “Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices”; (6) U.S. Patent Application No. 11/228,758, filed on September 16, 2005, “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. Patent Application No. 11/228,700, filed on September 16, 2005, “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. Patent Application No. 11/228,737, filed on September 16, 2005, “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; (9) U.S. Patent Application No. 11/367,749, filed on March 3, 2006, “Multi-Functional Hand-Held Device”. The entire contents of all of these applications are incorporated herein by reference.

The touch screen 112 may have a resolution exceeding 100 dpi. In an exemplary embodiment, the touch screen has a resolution of approximately 160 dpi. The user may contact the touch screen 112 using any suitable object or appendage, such as a stylus or finger. In some embodiments, the user interface is designed to operate primarily through finger-based contacts and gestures, which are less precise than stylus-based input due to the larger contact area of a finger on the touch screen. In some embodiments, the device translates the rough finger-based input into a precise pointer/cursor position or command for performing the user's desired action.

In some embodiments, in addition to the touch screen, device 100 may include a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, unlike a touch screen, a touchpad is a touch-sensitive area of the device that does not display visual output. The touchpad can be a touch-sensitive surface separate from touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

In some embodiments, device 100 may include a physical or virtual click wheel as input control 116. A user can navigate between or interact with one or more graphical objects (hereinafter referred to as icons) displayed on touch screen 112 by rotating the click wheel or by moving a point of contact with the click wheel (e.g., movement of the point of contact is measured by angular displacement relative to a center point of the click wheel). The click wheel can also be used to select one or more of the displayed icons. For example, a user may press at least a portion of the click wheel or an associated button. User commands and navigation commands provided by the user via the click wheel may be processed by input controller 160 and one or more of the modules and/or sets of instructions in memory 102. For a virtual click wheel, the click wheel and click wheel controller may be part of touch screen 112 and display controller 156, respectively. For a virtual click wheel, the click wheel may be an opaque or semi-transparent object that appears and disappears on the touch screen display in response to user interaction with the device. In some embodiments, the virtual click wheel is displayed on the touch screen of the portable multifunction device and is operated by user contact with the touch screen.

Device 100 also includes a power system 162 for powering the various components. Power system 162 may include a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuitry, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components related to the generation, management, and distribution of power in a portable device.

The device 100 may also include one or more optical sensors 164. Figures 1A and 1B illustrate an optical sensor coupled to the optical sensor controller 158 in the I/O subsystem 106. The optical sensor 164 may include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives ambient light projected through one or more lenses and converts the light into data representing an image. In conjunction with the imaging module 143 (also known as a camera module), the optical sensor 164 can capture still images or video. In some embodiments, the optical sensor is located on the back of the device 100, opposite the touchscreen display 112 on the front of the device, so that the touchscreen display can be used as a viewfinder for still and/or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, so that the user's image can be obtained for video conferencing while viewing other video conference participants on the touchscreen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor within the device housing), so that a single optical sensor 164 can be used with the touchscreen display for both video conferencing and still and/or video image acquisition.

Device 100 may also include one or more proximity sensors 166. Figures 1A and 1B illustrate proximity sensors 166 coupled to peripherals interface 118. Alternatively, proximity sensors 166 may be coupled to input controller 160 in I/O subsystem 106. Proximity sensors 166 may operate as described in U.S. patent application Ser. No. 11/241,839, “Proximity Detector In Handheld Device”; U.S. patent application Ser. No. 11/240,788, “Proximity Detector In Handheld Device”; U.S. patent application Ser. No. 11/620,702, “Using Ambient Light Sensor To Augment Proximity Sensor Output”; U.S. patent application Ser. No. 11/586,862, “Automated Response To And Sensing Of User Activity In Portable Devices”; and U.S. patent application Ser. No. 11/638,251, “Methods And Systems For Automatic Configuration Of Peripherals,” the disclosures of which are incorporated herein by reference in their entireties. In some embodiments, when the multifunction device is near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112. In some embodiments, when the device is in the user's pocket, bag, or other dark area, the proximity sensor keeps the screen off to prevent unnecessary battery drain while the device is locked.

Device 100 may also include one or more accelerometers 168. Figures 1A and 1B illustrate accelerometer 168 coupled to peripherals interface 118. Alternatively, accelerometer 168 may be coupled to input controller 160 in I/O subsystem 106. Accelerometer 168 may operate as described in U.S. Patent Publication No. 20050190059, “Acceleration-based Theft Detection System for Portable Electronic Devices,” and U.S. Patent Publication No. 20060017692, “Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer,” both of which are incorporated herein by reference in their entireties. In some embodiments, information is displayed on the touch screen display in either a landscape or portrait orientation based on analysis of data received from the one or more accelerometers.

In some embodiments, the software components stored in memory 102 may include an operating system 126, a communication module (or a set of instructions) 128, a touch/movement module (or a set of instructions) 130, a graphics module (or a set of instructions) 132, a text input module (or a set of instructions) 134, a global positioning system (GPS) module (or a set of instructions) 135, and an application (or a set of instructions) 136.

The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDWOS, or an embedded operating system such as VxWorks) includes various software components and/or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

The communication module 128 facilitates communication with other devices on one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and/or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are suitable for coupling with other devices directly or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, similar to, and/or compatible with the 30-pin connector used on iPod (a trademark of Apple Computer, Inc.) devices.

Contact/movement module 130 can detect contact with touch screen 112 (in conjunction with display controller 156) and other touch-sensitive displays (e.g., a touchpad or physical click wheel). Contact/movement module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger-down event), determining whether there has been movement of a contact on touch screen 112 and tracking the movement (e.g., detecting one or more finger-drag events), and determining whether contact has ceased (e.g., detecting a finger-up event or contact break). Contact/movement module 130 receives contact data from touch screen 112. Determining the movement of a contact point—represented by a sequence of contact data—can include determining the scalar velocity (magnitude), vector velocity (magnitude and direction), and/or acceleration (change in magnitude and/or direction) of the contact point. These operations can be applied to a single contact (e.g., a one-finger contact) or to multiple simultaneous contacts (e.g., "multi-touch"/multi-finger contacts). In some embodiments, contact/movement module 130 and display controller 156 also detect contact on a touchpad. In some embodiments, contact/movement module 130 and controller 160 detect contact on a click wheel.

The contact/motion module 130 can detect gestures input by the user. Different gestures on the touch screen 112 have different contact patterns. Therefore, gestures can be detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger down event, followed by detecting a finger up event at the same location (or substantially the same location) as the finger down event (e.g., at the location of a button-like user interface object such as a text character key or an emoticon character key in a soft keyboard). As another example, detecting a finger tap gesture on the touch screen includes detecting a finger down event, followed by detecting one or more finger drag events, and then followed by detecting a finger up event. In some embodiments, the contact/motion module 130 detects user-defined gestures.

The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112, including components for changing the intensity of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including but not limited to text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic may be assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like that specify the graphics to be displayed, along with coordinate data and other graphic attribute data, if necessary, and then generates screen image data for output to the display controller 156.

The text input module 134, which may be a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input). The text input module 134 is also configured to provide an emoticon soft keyboard for entering emoticon characters in various applications (e.g., email 140, IM 141, and any other application that uses emoticon character input). In some embodiments, the text input module 134 stores data representing a soft keyboard for entering text and an emoticon soft keyboard.

The GPS module 135 determines the location of the device and provides this information for various applications (e.g., to the phone 138 for location-based dialing, to the camera 143 as picture/video metadata, and to applications that provide location-based services such as a weather widget, a local yellow pages widget, and a map/navigation widget).

Application 136 may include the following modules (or sets of instructions), or a subset or superset thereof:

Contacts module 137 (sometimes called an address book or contact list);

Telephone module 138;

Video conferencing module 139;

Email client module 140;

Instant messaging (IM) module 141;

Training support module 142;

A camera module 143 for still and/or video images;

Image management module 144;

Video player module 145;

Music player module 146;

Browser module 147;

Calendar module 148;

Widget module 149 which may include a weather widget 149-1, a stock widget 149-2, a calendar widget 149-3, an alarm clock widget 149-4, a dictionary widget 149-5 and other widgets obtained by the user, as well as a user-generated widget 149-6;

A widget generator module 150 for making user-generated widgets 149-6;

Search module 151;

A video and music player module 152 that merges the video player module 145 and the music player module 146;

Notes module 153;

Map module 154; and/or

Online video module 155.

Examples of other applications 136 that may be stored in memory 102 include other word processing applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice reproduction.

In conjunction with the touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contacts module 137 can be used to manage an address book or contact list, including: adding name(s) to the address book; deleting name(s) from the address book; associating phone numbers(s), email addresses(s), physical addresses(s), or other information with names; associating images with names; categorizing and sorting names; providing phone numbers or email addresses to initiate and/or facilitate communications by telephone 138, video conferencing 139, email 140, or IM 141; and the like.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 can be used to enter a series of characters corresponding to a phone number, access one or more phone numbers in address book 137, modify an entered phone number, dial the corresponding phone number, conduct a call, and disconnect or hang up when the call is complete. As described above, wireless communications can use any of a number of communication standards, protocols, and technologies.

In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact module 130, the graphics module 132, the text input module 134, the contact list 137, and the telephone module 138, the video conferencing module 139 can be used to initiate, conduct, and terminate a video conference between a user and one or more other participants.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 can be used to generate, send, receive, and manage emails. In conjunction with image management module 144, email client module 140 makes it easy to generate and send emails with still or video images captured by camera module 143. In addition, text input module 134 works in conjunction with email client module 140 to facilitate the insertion of emoticons into emails.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant messaging module 141 can be used to enter a series of characters corresponding to an instant message, modify previously entered characters, transmit a corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receive instant messages, and view received instant messages. In some embodiments, transmitted and/or received instant messages may include graphics, photos, audio files, video files, and/or other attachments supported in MMS and/or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both phone-based messages (i.e., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS). In addition, text input module 134 works in conjunction with instant messaging module 141 to facilitate the insertion of emoticons into corresponding instant messages.

In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the music player module 146, the training support module 142 can be used to generate training (e.g., with time, path, and/or calorie burn goals); communicate with training sensors (sports devices); receive training sensor data; calibrate sensors for monitoring training; select and play music for training; and display, store, and transmit training data.

In conjunction with touch screen 112, display controller 156, optical sensor(s) 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 can be used to capture still images or videos (including video streams) and store them in memory 102, modify the characteristics of still images or videos, or delete still images or videos from memory 102.

In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 can be used to manage, modify or manipulate, annotate, delete, present (e.g., in a digital slide show or album), and store still and/or video images.

In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, audio circuitry 110, and speaker 111, video player module 145 may be used to display, present, or play back video (e.g., on the touch screen or on an externally connected display via external port 124).

In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, music player module 146 allows a user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files. In some embodiments, device 100 may include the functionality of an MP3 player such as an iPod (trademark of Apple Computer, Inc.).

In combination with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, browser module 147 can be used to browse the Internet, including searching for, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to the web pages.

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email module 140, and browser module 147, calendar module 148 can be used to generate, display, modify, and store calendars and calendar-related data (e.g., calendar entries, to-do lists, etc.). In addition, text input module 134 works in conjunction with calendar module 148 to facilitate the insertion of emoticons into calendar entries, to-do lists, etc.

In conjunction with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the widget module 149 is a micro-application that can be downloaded and used by the user (e.g., the weather widget 149-1, the stock widget 149-2, the calculator widget 149-3, the alarm clock widget 149-4, and the dictionary widget 149-5) or a micro-application generated by the user (e.g., the user-generated widget 149-6). In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets).

In conjunction with the RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134 and browser module 147, the widget generator module 150 can be used by a user to generate widgets (e.g., turning a user-specified portion of a web page into a widget).

In conjunction with the touch screen 112, display controller 156, contact module 130, graphics module 132 and text input module 134, the search module 151 can be used to search the memory 102 for text, music, sound, images, videos and/or other files that match one or more search criteria (e.g., one or more user-specified search terms).

In conjunction with the touch screen 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the note module 153 can be used to create and manage notes, to-do lists, etc. In addition, the text input module 134 works together with the note module 153 to facilitate inserting emoticons into notes, to-do lists, etc.

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 can be used to receive, display, modify, and store maps and map-related data (e.g., driving directions; data about stores and other points of interest at or near a particular location; and other location-based data). In addition, text input module 134 works in conjunction with map module 154 to facilitate inserting emoticons into a map.

In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 allows a user to access, browse, receive (e.g., by streaming and/or downloading), play back (e.g., on the touch screen or on an externally connected display via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.246. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video. Additional descriptions of online video applications can be found in U.S. Provisional Patent Application No. 60/936,562, filed on June 20, 2007, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11/968,067, filed on December 31, 2007, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are incorporated herein by reference in their entirety.

Each of the modules and applications identified above corresponds to a set of instructions for performing one or more functions described above. These modules (i.e., multiple sets of instructions) do not need to be implemented as separate software programs, processes, or modules, and thus, various subsets of these modules can be combined or reconfigured in various embodiments. For example, the video player module 145 can be combined with the music player module 146 into a single module (e.g., video and music player module 152, Figure 1B). In some embodiments, the memory 102 can store a subset of the modules and data structures identified above. Furthermore, the memory 102 can store additional modules and data structures not described above.

In some embodiments, device 100 is a device that performs operations of a predetermined set of functions on the device exclusively through touch screen 112 and/or touch pad. By using the touch screen and/or touch pad as the primary input/control device for operation of device 100, the number of physical input/control devices (such as push buttons and dials, etc.) on device 100 can be reduced.

A predetermined set of functions that can be performed exclusively via the touch screen and/or touchpad include navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, causes the device 100 to navigate to a general menu, a main menu, or a root menu from any user interface that may be displayed on the device 100. In these embodiments, the touchpad may be referred to as a "menu button." In some embodiments, the menu button may be a physical push button or other physical input/control device, as an alternative to the touchpad.

Figure 2 illustrates a portable multifunction device 100 with a touch screen 112 according to some embodiments. The touch screen can display one or more graphics within a user interface (UI) 200. In this and other embodiments, a user can select one or more of the graphics, for example, by contacting or touching the graphics with one or more fingers 202 (not drawn to scale). In some embodiments, selection of one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, contact can include gestures such as one or more taps, one or more swipes (right to left, left to right, up and/or down), and/or scrolls (right to left, left to right, up and/or down) of a finger in contact with the device 100. In some embodiments, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture that waves over an application icon does not select the corresponding application when the gesture corresponding to selection is a tap.

The device 100 may also include one or more physical buttons, such as a “home” or menu button 204. As described above, the menu button 204 may be used to navigate to any application 136 in a set of applications executable on the device 100. Alternatively, in some embodiments, the menu button may be implemented as a soft key in the GUI on the touch screen 112.

In one embodiment, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for powering the device on/off and locking the device, a volume adjustment button(s) 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking/charging external port 124. The push button 206 can be used to power the device on/off by pressing and holding the button for a predetermined time interval; to lock the device by pressing and releasing the button before a predetermined time interval has elapsed; and/or to unlock the device or initiate the unlocking process. In an alternative embodiment, the device 100 can also receive voice input for activating or deactivating certain functions via the microphone 113.

Figure 3 shows an exemplary user interface for unlocking a portable electronic device according to some embodiments. In some embodiments, user interface 300 may include the following elements or a subset or superset thereof:

Unlock image 302 that is moved by finger gesture to unlock the device;

An arrow 304 providing a visual cue for the unlock gesture;

Channel 306 provides additional prompts for the unlock gesture;

Time 308;

Day 310;

Date 312; and

Wallpaper image 314.

In some embodiments, the device detects contact with the touch-sensitive display (e.g., contact by a user's finger on or near the unlock image 302) while the device is in the user interface lock state. The device moves the unlock image 302 in accordance with the contact. If the detected contact corresponds to a predetermined gesture, such as moving the unlock image on channel 306, then the device transitions to the user interface unlock state. Conversely, if the detected contact does not correspond to a predetermined gesture, then the device remains in the user interface lock state. As described above, the process of using gestures on a touch screen to unlock the device is described in U.S. patent application Ser. No. 11/322,549, “Unlocking A Device By Performing Gestures On An Unlock Image,” filed on December 23, 2005, and U.S. patent application Ser. No. 11/322,550, “Indication Of Progress Towards Satisfaction Of A User Input Condition,” filed on December 23, 2005, the entire contents of which are incorporated herein by reference.

Figures 4A and 4B illustrate exemplary user interfaces for a menu of applications on a portable multifunction device according to some embodiments. In some embodiments, user interface 400A includes the following elements or a subset or superset thereof:

Signal strength indicator(s) 402 for wireless communication(s), such as cellular and Wi-Fi signals;

Time 404;

Battery status indicator 406;

A tray 408 with icons for frequently used applications, such as:

• Phone 138 which may include an indicator 414 of the number of missed calls or voicemail messages;

An email client 140 that may include an indicator 410 of the number of unread emails;

Browser 147; and

Music player 146; and

Icons for other applications, such as:

IM141;

Image management 144;

Camera 143;

Video player 145;

Weather 149-1;

· Stock 149-2;

Training support 142;

Calendar 148;

Calculator 149-3;

Alarm clock 149-4;

Dictionary 149-5; and

User-generated widgets 149-6.

In some embodiments, user interface 400B includes the following elements, or a subset or superset thereof:

402, 404, 406, 141, 148, 144, 143, 149-3, 149-2, 149-1, 149-4, 410, 414, 138, 140, and 147 above;

Map 154;

·Notes 153;

• Settings 412, described further below, providing access to settings for the device 100 and its various applications 136;

A video and music player module 152, also known as an iPOD (trademark of Apple Computer, Inc.) module 152; and

• Online video module 155 , also referred to as YouTube (trademark of Google, Inc.) module 155 .

In some embodiments, UI 400A or 400B provides integrated access to widget-based applications and non-widget-based applications. In some embodiments, all widgets, whether user-generated or not, are displayed in UI 400A or 400B. In other embodiments, activating the icon of user-generated widget 149-6 results in another UI that includes the user-generated widget or an icon corresponding to the user-generated widget.

In some embodiments, the user can rearrange the icons in UI 400A or 400B, for example, by using the process described in U.S. patent application Ser. No. 11/459,602, “Portable Electronic Device With Interface Reconfiguration Mode,” filed on July 24, 2006, which is incorporated herein by reference in its entirety. For example, the user can move application icons into or out of tray 408 by using finger gestures.

5A-5G illustrate exemplary user interfaces for providing and organizing emoticon keys on a portable electronic device 100 equipped with a touch-sensitive display 112 in accordance with some embodiments. A summary of these figures is provided herein.

FIG5A shows a user interface 500A in an instant messaging application 141. As shown in this example, a character input area 5000 is provided for displaying or being operable to display text character input and emoticon character input selected by a user. As described above, as used in the specification and claims, text characters refer to alphanumeric characters, Chinese characters (sinograph), Japanese kanji or kana symbols and/or other written human language characters, while emoticon characters refer to still and animated picture characters that are not text characters in written human languages. A keyboard display area 5005 that may include multiple emoticon category icons 5010 is also provided in the user interface 500A. The emoticon category icons 5010 correspond to multiple emoticon characters, such as the most recently used, seasonal categories of people, nature, objects, travel, symbols and/or emoticons. In the exemplary embodiment displayed in UI 500A, emoticon category icon 5010-1 corresponds to the most recently frequently used emoticon category, emoticon category icon 5010-2 corresponds to people, emoticon category icon 5010-3 corresponds to nature, emoticon category icon 5010-4 corresponds to objects, emoticon category icon 5010-5 corresponds to travel, and emoticon category icon 5010-6 corresponds to symbols.

UI 500A also includes exemplary emoticon character keys 5015 of corresponding emoticon categories being displayed within keyboard display area 5005. In exemplary UI 500A, since the most recently used emoticon category icon 5010-1 is currently selected, the emoticon character keys being displayed (i.e., 5015-1, 5015-2, and 5015-3) correspond to the emoticon category 'most recently used.' The currently selected emoticon category icon 5010 is typically visually distinguished from the unselected emoticon category icons by being highlighted or having its background color changed relative to the unselected emoticon category icons.

In some embodiments, based on an algorithm that analyzes recent usage and the frequency of use of already used emoticons, the emoticon key in the emoticon category corresponding to emoticon category icon 5010-1, i.e., the emoticon most frequently used recently, is selected for display. The algorithm generates a history list of emoticon usage. Based on the position of the entry in the history list, each entry in the history list is given a relative weight. For example, the most recently used emoticon receives the highest weight, the second most recently used emoticon receives a smaller weight than the most recently used emoticon, the third most recently used emoticon receives a smaller weight than the second most recently used emoticon, and so on. Then, for each different emoticon, a total score is determined by combining (e.g., adding up) the relative weights of the emoticon. For example, for the emoticons occupying the second, tenth, and twelfth positions in the history list, the total score of the emoticon is the sum of the relative weights given to the second, tenth, and twelfth positions in the history list. The emoticon with the highest total score is then displayed (e.g., in descending order of score) in the emoticon category 'most frequently used recently'.

In this exemplary embodiment, UI 500A also includes a keyboard selection icon 5020. In some embodiments, keyboard selection icon 5020 is displayed simultaneously with a plurality of emoticon category icons 5010, for example, in a predetermined area 5009 along the bottom of the touch screen display. In some embodiments, keyboard selection icon 5020 is used to replace the display of emoticon character keys with the display of a text character input keyboard in keyboard display area 5005 (e.g., 5055, FIG. 5G ). In addition, UI 500A includes a send icon 5025 that, when activated (e.g., by a finger tap gesture), initiates the sending of an electronic message entered in area 5000. UI 500A also includes a delete icon 5011 that, when activated (e.g., by a finger tap gesture), deletes one or more characters in input area 5000.

UI 500B ( FIG. 5B ) shows a keyboard display area 5005 including a subset sequence marker icon 5040 for a corresponding emoticon category, here, person, which is visually indicated as selected by highlighting emoticon category icon 5010-2. Subset sequence marker icon 5040 provides information about the number of subsets in the currently displayed emoticon category and the position of the displayed subsets in the sequence of subsets of emoticon character keys in keyboard display area 5005. Here, subset sequence marker icon 5040 indicates that there are six subsets of emoticon character keys in the person emoticon category and that the first subset in the person emoticon category is currently being displayed. As another example, see the following discussion of subset sequence marker icon 6000 in FIGS. 6A-6C .

UI 500C, UI 500D, UI 500E, and UI 500F shown in Figures 5C-5F respectively illustrate examples of user interfaces with other exemplary emoticon categories and emoticon character keyboards.

UI 500G ( FIG. 5G ) shows a text character input keyboard 5055 in keyboard display area 5005 . In some embodiments, keyboard 5055 includes a text character input category icon 5060 that, when activated (e.g., by a finger tap gesture on the icon), causes the display of an alternative text character keyboard with numeric keys and other text character input keys. In some embodiments, keyboard 5055 includes a mixed-purpose key 5062 that is operable to enter a space character and also operable to confirm auto-completion and/or auto-correction when typing text characters. For example, some embodiments allow the use of spoken text when typing Japanese language text. As the device evaluates the text characters entered by the user, it presents one or more recommended completion candidates for the typed text. For example, if the user types the characters "ra," "rain" is recommended as a completion candidate. Then, when the completion candidate "rain" is displayed (not shown), activating mixed-purpose key 5062 (e.g., by a finger tap gesture on key 5062) confirms that "rain" is the desired text to be entered, and the device inserts the string "rain" into character input area 5000 . In some embodiments, as completion candidates, Chinese characters (sinograph) or Japanese kana characters can be presented. In some embodiments, key 5055 includes a mixed purpose key 5063 that can be operated to input a return character and can also be operated to confirm the automatic completion of an action. For example, if a user types the character "goo", then as an action completion candidate, the device will recommend going to the Google search engine. Then, activating the mixed purpose key 5063 (for example, by a finger tapping gesture on key 5063) will cause the device to open the browsing module 147 and display the web page of the Google search engine. The keyboard 5055 also includes a keyboard selection icon 5020. In addition, the keyboard 5055 includes text character input keys 5065 such as 5065-1 ("Q"), 5065-2 ("A") and 5065-3 ("S") etc.

6A-6E illustrate exemplary user interfaces for navigating between subsets of emoticon characters within a particular emoticon category and entering emoticon characters on a portable electronic device 100 equipped with a touch-sensitive display 112 in accordance with some embodiments. A summary of these figures is provided here.

UI 600A ( FIG. 6A ) shows a sequence of subset marker icons 6000-1, 6000-2, and 6000-3 for a currently displayed emoticon category, here, nature, which is visually indicated as being selected by highlighting the emoticon category icon 5010-3. The subset sequence marker icons 6000-1, 6000-2, and 6000-3 provide information about the number of subsets in the currently displayed emoticon category and the position of the displayed subsets in the sequence of subsets of emoticon character keys in the keyboard display area 5005. For example, in UI 600A, the number of marker icons 6000 indicates that there are three subsets in the emoticon category (nature). Additionally, the visual highlighting of icon 6000-1 relative to icons 6000-2 and 6000-3 indicates that the first subset 6015 in the category is being displayed.

In some embodiments, when a gesture (e.g., gesture 6010 ( FIG. 6A ), a horizontal or substantially horizontal finger tap gesture) is detected in the keyboard display area 5005 , the first subset 6015 of emoticon character keys is replaced by the display of the second subset 6020 ( FIG. 6B ) of emoticon character keys in the keyboard display area 5005 . Additionally, in some embodiments, the information provided by the subset sequence marker icon is updated to reflect that the displayed first subset 6015 of emoticon character keys has been replaced by the second subset 6020 of emoticon character keys. In this example, because the first subset 6015 of emoticon character keys in the natural category is displayed, the subset sequence marker icon 6000-1 is highlighted in the UI 600A. When the first subset 6015 of emoticon character keys is replaced by the second subset 6020 ( FIG. 6B ), the subset sequence marker icon 6000-2 is highlighted instead. In some embodiments, a subset sequence marker icon corresponding to a subset of the currently displayed emoticon character keys is highlighted or visually distinguished, while another subset sequence marker icon corresponding to a subset of the displayed emoticon character keys stops being highlighted or visually distinguished.

UI 600B ( FIG. 6B ), discussed to some extent above, illustrates detecting another gesture (e.g., gesture 6025, a horizontal or substantially horizontal finger tap gesture) in keyboard display area 5005. In response to detecting gesture 6025, the second subset 6020 of emoticon character keys is replaced by a display of a third subset 6040 of emoticon character keys ( FIG. 6C ) in keyboard display area 5005. Accordingly, as shown in FIG. UI 600C ( FIG. 6C ), subset sequence marker icon 6000-3 is highlighted in place of subset sequence marker icon 6000-2.

UI 600C ( FIG. 6C ) illustrates detecting another gesture in keyboard display area 5005 (e.g., gesture 6030, a horizontal or substantially horizontal finger tap gesture that is substantially opposite in direction to gesture 6025). In response to detecting gesture 6030, the third subset 6040 of emoticon character keys is replaced with the display of the second subset 6020 of emoticon character keys ( FIG. 6B ) in keyboard display area 5005. Accordingly, subset sequence marker icon 6000-2 is highlighted, replacing subset sequence marker icon 6000-3 ( FIG. 6B ).

UI 600C ( FIG. 6C ) also shows a user gesture 6045 (e.g., a finger tap gesture) on emoticon key 6050. UI 600D ( FIG. 6D ) shows a key selection icon 6060 being displayed while the finger contact in user gesture 6045 is still on emoticon key 6050 (shown in FIG. 6C ). Key selection icon 6060 shows the key that would be selected if the user's finger leaves (breaks contact with) the touch screen display at the current finger contact location on the touch screen display. In some embodiments, an image 6061 of the selected emoticon character, in this case, the emoticon character corresponding to emoticon key 6050, is displayed within key selection icon 6060.

UI 600E ( FIG. 6E ) shows an image 6070 of a selected emoticon character, in this case, the emoticon character corresponding to emoticon key 6050 , displayed within character input area 5000 in response to detecting gesture 6045 .

7A-7E illustrate exemplary user interfaces for inputting emoticon characters and sending electronic messages including emoticon characters on a portable electronic device 100 equipped with a touch-sensitive display 112 in accordance with some embodiments. A summary of these figures is presented here.

UIs 700A-700C illustrate user selection of emoticons in a manner similar to that described above. In UI 700A, emoticon key 7000 is selected in response to detecting user gesture 7001 (e.g., a finger tap gesture). UI 700B ( FIG. 7B ) illustrates key selection icon 7010 being displayed while the finger used for gesture 7001 is still in contact with the area corresponding to emoticon key 7000.

UI 700C ( FIG. 7C ) shows an image 7025 of a selected emoticon character displayed within character input area 5000 in response to detecting gesture 7001. In response to detecting a user gesture (e.g., finger tap gesture 7026, FIG. 7D ) on send icon 5025 ( FIG. 7D ), the text characters and emoticon characters entered in area 5000 are sent as an electronic message to the other party (e.g., to John Appleseed, FIG. 7D and FIG. 7E ).

In this example, the emoticon characters are input and sent in an instant message. However, it will be appreciated by those skilled in the art that the emoticon characters may also be input and sent in electronic messages and files such as other types of emails 140.

8A-8C illustrate exemplary user interfaces for reconfiguring emoticon keys in a software keyboard on a portable electronic device 100 equipped with a touch-sensitive display 112, according to some embodiments. This process provides a user with a simple and effective mechanism to rearrange emoticon keys on a portable electronic device (e.g., to optimize emoticon key positions for more efficient user input).

UI 800A ( FIG. 8A ) shows a keyboard display area 5005 having a plurality of emoticon character keys 8002 including emoticon keys 8001-1 and 8001-2. The user interface is configured to detect a predetermined user action (e.g., pressing and holding finger contact 8005 on any emoticon character key 8001 for more than a predetermined period of time) for initiating a predetermined user interface reconfiguration process.

UI 800B ( FIG. 8B ) shows that in some embodiments, when a predetermined user action 8005 is detected, the emoticon character keys 8001 within the keyboard display area 5005 vibrate in an area substantially centered around the respective average position of each respective emoticon character key (e.g., vibrations 8020-1, 8020-2, 8020-3, etc.). These vibrations provide a simple visual indicator that the device is now in interface reconfiguration mode and that the vibration icons can be rearranged.

UI 800B also shows that in some embodiments, the device detects a point of contact 8050 by a user with the touch screen display 112 at a first location corresponding to a first emoticon character key 8001-2 in the plurality of emoticon character keys 8002, and detects movement of the point of contact 8050 to a second location 8025 on the touch screen display 112. In some embodiments, the user interface will display the first emoticon character key 8001-2 moving to the second location 8025 on the touch screen display 112 based on the detected movement of the point of contact 8050.

In UI 800C ( FIG. 8C ), the user interface fixes the position of first emoticon character key 8001-2 at a second position in response to detecting a second predetermined user action for terminating the predetermined user interface reconfiguration process (e.g., detecting activation of home button 204). In the example shown, the end result is that emoticon character keys 8001-1 and 8001-2 have swapped positions in UI 800C when compared to UI 800A.

9A-9H illustrate exemplary user interfaces for responding to messages with emoticons while the portable electronic device 100 equipped with the touch-sensitive display 112 is in user interface lock mode in accordance with some embodiments. A summary of these figures is provided here.

UI 900A ( FIG. 9A ) illustrates an exemplary user interface in user interface locked mode. See also the description of FIG. 3 above. In user interface unlocked mode, device 100 is in its normal operating state, detecting and responding to user input corresponding to interactions with the user interface. Device 100 in unlocked mode can be described as an unlocked device 100. Unlocked device 100 detects and responds to user input for navigating the user interface, entering data, and activating or deactivating functions. In contrast to user interface unlocked mode, in user interface locked mode, only a restricted subset of control icons is displayed, and device 100 responds only to a restricted set of finger gestures. For example, only unlock image 302, arrow 304, and channel 306 (UI 900A) may be displayed along with wallpaper image 314. Additionally, as described below, a small number of predetermined responses, including emoticons, may be displayed, and/or a small number of gestures may be used to respond to messages received while in user interface locked mode.

UI900B (Figure 9B) shows receiving an incoming message from a party 2210 (here, "Jane Doe") to a user of a portable electronic device while the user interface is in a user interface lock mode, where at least a portion of the incoming message 2212 from the party 2210 is displayed to the user on the touch screen display 112.

UI900C (Figure 9C) shows a user interface for detecting one or more predetermined finger gestures (e.g., tap gestures 9100-1 and 9100-2 and swipe gesture 9100-3) on the touch screen display 112 that correspond to emoticon characters. In some embodiments, as described later with reference to Figures 10A-10E and Figure 16, a correspondence between these one or more predetermined finger gestures and emoticon characters is established by assigning one or more predetermined finger gestures to the emoticon characters. In some embodiments, the user interface also detects a predetermined user action to initiate sending a response to the incoming message 2212 (e.g., swipe gesture 2213), and then sends a response to the incoming message 2212, wherein the response includes the emoticon characters corresponding to the one or more predetermined finger gestures (e.g., 9100-1, 9100-2, and 9100-3) detected on the touch screen display 112.

UI 900D ( FIG. 9D ) illustrates that in some embodiments, a user interface may display multiple predetermined responses 9110 for an incoming message 2212, wherein the predetermined responses (e.g., 9110-1, 9110-2, and 9110-3) include multiple emoticons (e.g., 9111-1, 9111-2, and 9111-3). In UI 900D, each predetermined response 9110 includes a single emoticon 9111. More generally, other predetermined responses 9110 may include more than one emoticon, and some predetermined responses 9110 may not include any emoticons (not shown).

UI900E (FIG. 9E) illustrates detecting a user action (e.g., a tap gesture 9120) to select a predetermined response (e.g., response 9110-1). The user can quickly select from a list of predetermined responses, including emoticons, and send the selected response without unlocking the device. In some embodiments, detecting the selection of predetermined response 9110 also initiates the sending of the predetermined response without detecting any further action by the user. In some embodiments, after detecting the selection of predetermined response 9110, another predetermined user action is detected to initiate the sending of the response, such as detecting activation of the home button 204, detecting activation of a "send" button (not shown), or detecting a sliding finger gesture 2213 that appears to move the unlock image 302 across the channel 306 (in this case, the "slide to view" message in the channel 306 may become a "slide to send a response" message or a similar message, FIG. 9E).

UI 900F ( FIG. 9F ) shows a user interface in the messaging application after the device is subsequently unlocked, showing a predetermined response 9110 - 1 that is sent when the device is in a locked state.

UI 900G and UI 900H (Figures 9G and 9H, respectively) are similar to UI 900E and UI 900F (Figures 9E and 9F, respectively), except that predetermined responses 9110-4, 9110-5, and 9110-6 include only emoticons instead of a combination of text and emoticons. Similarly, predetermined response 9110-4 sent in Figure 9H corresponds to predetermined response 9110-4 in Figure 9G and includes only emoticons 9225.

10A-10E illustrate exemplary user interfaces for assigning user-defined gestures to emoticons on a portable electronic device 110 equipped with a touch-sensitive display 112 in accordance with some embodiments. A summary of these figures is given here.

UI 1000A ( FIG. 10A ) illustrates an exemplary user interface for recording a user-defined gesture. UI 1000A includes a record icon 1001, an associated icon 1010, a save icon 1015, and a cancel icon 1016. Upon detecting a user gesture 1020 on record icon 1001, the user interface enters a gesture recording mode. UI 1000B ( FIG. 10B ) illustrates a user-defined gesture 1030 detected on touch screen 112 while in gesture recording mode. In some embodiments, in response to detecting a gesture on associated icon 1010 (e.g., a finger tap gesture 1040 , FIG. 10C ), a process is initiated for associating a representation of user-defined gesture 1030 with an emoticon, which may include displaying one or more user interfaces with an emoticon key (e.g., UI 1000D , FIG. 10 ). In some embodiments, the user interface with the emoticon character key is similar to UI 500A-500F (Figures 5A-5F), but the user interface is used to associate emoticons with user-defined gestures 1030 rather than for selecting emoticon characters to be entered in an electronic message or file. In response to detecting a gesture on an emoticon character key (e.g., a finger tap gesture 1070 on emoticon key 1060, Figure 10D), a representation of user-defined gesture 1030 is associated with (or temporarily associated with, pending user confirmation of) the emoticon character corresponding to emoticon character key 1060. In some embodiments, a confirmation UI (e.g., UI 1000E, Figure 10E) displays the emoticon character 1075 that will be associated with the user-defined gesture 1030 upon detection of activation of the save icon 1015 (e.g., via a finger tap gesture 1080, Figure 10E). In response to activation of the cancel icon 1016 (e.g., via a finger tap gesture), the process of assigning the user-defined gesture to the emoticon character terminates.

Figures 11A-11B illustrate exemplary user interfaces for using emoticons as location indicators on a map via electronic messaging, according to some embodiments. A summary of these figures is provided here.

UI 1100A ( FIG. 11A ) shows an electronic message 2216 displayed in an instant messaging application 141. The electronic message includes an instruction to launch a geo-mapping application, location data representing a location on a map, and emoticon data representing an emoticon character. The emoticon data is associated with the location data. In response to detecting a predetermined user action on the message (e.g., a finger tap gesture 1105 on a link 2220 to Lake Tahoe), the device displays a map in the geo-mapping application 154 (e.g., map 1115 , FIG. 11B ), and the device displays an emoticon character 1120 ( FIG. 11B ) at a location on the map 1115 corresponding to the location data.

Some of the user interfaces in Figures 5A-11C are used to illustrate the processes described in Figures 12A-12F and 13-17 below.

Figures 12A-12F are flow charts illustrating a method for operating an emoticon keyboard on a portable electronic device with a touch-sensitive display according to some embodiments. Method 1200 is performed on a portable electronic device with a touch screen display (e.g., portable multifunction device 100). The method provides a more efficient way to organize and select emoticons on a portable electronic device, thereby conserving power, increasing the time between battery charges, and improving user satisfaction with the device.

The portable electronic device simultaneously displays (1202) a character input area (e.g., area 5000, FIG. 6A ), a keyboard display area (e.g., area 5005, FIG. 6A ), and a plurality of emoticon category icons (e.g., icon 5010, FIG. 6A ). The character input area 5005 is used to display or is operable to display text character input and emoticon character input selected by the user.

The emoticon category icons correspond to a plurality of emoticon categories (e.g., 5010-1 to 5010-6, FIG. 6A ). In some embodiments, the plurality of emoticon category icons include a recently frequently used emoticon category icon (1204) corresponding to a plurality of emoticon characters that have been frequently selected by the user recently (e.g., recently frequently used emoticon category icon 5010-1, FIG. 6A ). In some embodiments, the plurality of emoticon category icons include a season emoticon category icon (1206) corresponding to a plurality of emoticon characters associated with a particular season (e.g., spring, summer, autumn, or winter). In some embodiments, the plurality of emoticon category icons include a holiday emoticon category icon (1208) corresponding to a plurality of emoticon characters associated with a particular holiday (e.g., Christmas or New Year's Day).

In some embodiments, a plurality of emoticon category icons 5010 are displayed ( 1210 ) in keyboard display area 5005 ( FIG. 6A ).

The device detects (1212) a gesture on a corresponding emoticon category icon 5010 in the plurality of emoticon category icons. In some embodiments, the gesture can be a finger tap gesture (e.g., tap gesture 5007 on category icon 5010-3, FIG. 5A ).

In response to detecting a gesture on a corresponding emoticon category icon, the device simultaneously displays (1214) a first subset of emoticon character keys of the corresponding emoticon category (e.g., subset 6015, FIG. 6A ) and multiple subset sequence marker icons of the corresponding emoticon category (e.g., subset sequence marker icon 6000, FIG. 6A ) in the keyboard display area 5005.

The corresponding expression categories include multiple subsets of expression character keys, which are configured to be displayed individually as a sequence of subsets of expression character keys in the keyboard display area. For example, the natural expression category in Figures 6A-6C includes three subsets of expression character keys, which are configured to be displayed individually as a sequence of subsets of expression character keys in the keyboard display area 5005, namely, the first subset 6015 in Figure 6A, the second subset 6020 in Figure 6B, and the third subset 6040 in Figure 6C.

The subset sequence marker icons provide information about the number of subsets in the corresponding expression category and the position of the displayed subsets in the subset sequence of the expression character keys in the keyboard display area. For example, in Figures 6A-6C, subset sequence marker icons 6000-1, 6000-2, and 6000-3 indicate that there are three subsets of expression character keys in the natural expression category 5010-3. In addition, when the first subset 6015 is displayed, icon 6000-1 is highlighted (Figure 6A); when the second subset 6020 is displayed, icon 6000-2 is highlighted (Figure 6B); and when the third subset 6040 is displayed, icon 6000-3 is highlighted (Figure 6C). In some embodiments, the subset sequence marker icons have similar geometric shapes such as circles, squares, rectangles, or stars.

In some embodiments, the emoticon character keys in at least one emoticon category are sorted based on emoticon character usage data (1216). In some embodiments, the emoticon character usage data includes emoticon character usage data from a software application running on the portable electronic device (1218). In some embodiments, the emoticon character usage data includes emoticon character usage data received (e.g., downloaded) from a service provider of the portable electronic device (1220). In some embodiments, the emoticon character keys in at least one emoticon category are sorted based on emoticon character popularity data (1222).

In some embodiments, a subset sequence marking icon is displayed ( 1224 ) in the keyboard display area (eg, subset sequence marking icon 6000 is displayed in area 5005 , FIG. 6A ).

When a first subset of emoticon character keys of a corresponding emoticon category is displayed in the keyboard display area, the device detects (1226) a gesture in the keyboard display area. For example, the device detects a horizontal or substantially horizontal finger tap gesture 6010 (Figure 6A) in area 5005.

In response to detecting a gesture in the keyboard display area: the device replaces display of a first subset of emoticon character keys of the corresponding emoticon category in the keyboard display area with display of a second subset of emoticon character keys of the corresponding emoticon category, and updates information provided by the subset sequence marker icon to reflect that the displayed first subset of emoticon character keys has been replaced with the second subset of emoticon character keys (1228). For example, in response to detecting a horizontal finger tap gesture 6010 in the keyboard display area 5005, the device replaces display of the first subset 6015 (FIG. 6A) of emoticon character keys of the natural expression category in the keyboard display area 5005 with display of the second subset 6020 (FIG. 6B) of emoticon character keys of the natural expression category, and updates information provided by the subset sequence marker icon 6000 to reflect that the displayed first subset 6015 of emoticon character keys has been replaced with the second subset 6020 of emoticon character keys. Updating the information provided by the subset sequence marker icons may include, but is not limited to, highlighting or visually distinguishing a subset sequence marker icon corresponding to a second subset of emoticon character keys and ceasing to highlight or visually distinguish another subset sequence marker icon corresponding to a first subset of emoticon character keys (e.g., in FIG. 6B , highlighting icon 6000-2 and ceasing to highlight icon 6000-1).

In some embodiments, replacing the display of the first subset of emoticon character keys of the corresponding emoticon category with the display of the second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area includes displaying (1230) an animation of moving the first subset of emoticon character keys outside the touch screen display. For example, between Figures 6A and 6B, an animation of the subset 6015 moving outside the touch screen display 112 can be displayed.

In some embodiments, replacing the display of the first subset of emoticon character keys of the corresponding emoticon category with the display of the second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area includes displaying (1232) an animation of moving the second subset of emoticon character keys on the touch screen display. For example, between Figures 6A and 6B, an animation of moving the subset 6020 on the touch screen display 112 can be displayed.

In some embodiments, replacing the display of the first subset of emoticon character keys of the corresponding emoticon category with the display of the second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area includes displaying an animation of moving the first subset of emoticon character keys out of the touch screen display and displaying an animation of moving the second subset of emoticon character keys on the touch screen display (1234). For example, between FIG. 6A and FIG. 6B , an animation of subset 6015 moving off the touch screen display 112 and subset 6020 moving on the touch screen display 112 can be displayed.

In some embodiments, the device detects a gesture in the keyboard display area while displaying a second subset of emoticon character keys of a corresponding emoticon category in the keyboard display area. For example, the device detects a horizontal or substantially horizontal finger tap gesture 6025 in area 5005 ( FIG. 6B ) while displaying subset 6020. In response to detecting the gesture in the keyboard display area (e.g., gesture 6025), the device replaces display of the second subset of emoticon character keys of the corresponding emoticon category in the keyboard display area with display of a third subset of emoticon character keys of the corresponding emoticon category (1236). Furthermore, in response to detecting the gesture in the keyboard display area (e.g., gesture 6025), the device updates the information provided by the subset sequence marker icon to reflect that the displayed second subset of emoticon character keys has been replaced with the third subset of emoticon character keys (1236). For example, in response to detecting the horizontal finger tap gesture 6025 in the keyboard display area 5000, the device replaces display of the second subset 6020 ( FIG. 6B ) of the emoticon character keys of the natural expression category with display of the third subset 6040 ( FIG. 6C ) of the emoticon character keys of the natural expression category in the keyboard display area 5005, and updates the information provided by the subset sequence marker icons 6000 to reflect that the displayed second subset 6020 of emoticon character keys has been replaced by the third subset 6040 of emoticon character keys. The updated information may include, but is not limited to, highlighting or visually distinguishing the subset sequence marker icon corresponding to the third subset of emoticon character keys and ceasing to highlight or visually distinguish another subset sequence marker icon corresponding to the second subset of emoticon character keys (e.g., in FIG. 6C , highlighting icon 6000-3 and ceasing to highlight icon 6000-2).

In some embodiments, the gesture detected while displaying the second subset of emoticon character keys is a swipe gesture in a first direction (e.g., a right-to-left swipe gesture 6025, Figure 6B). In response to the device detecting a second swipe gesture on the touch screen display in a direction substantially opposite to the first direction in the keyboard display area (e.g., a left-to-right swipe gesture 6030, Figure 6C), the device replaces display of a third subset of emoticon character keys of the corresponding emoticon category in the keyboard display area with display of a second subset of emoticon character keys of the corresponding emoticon category, and updates the information provided by the subset sequence marker icon to reflect that the displayed third subset of emoticon character keys has been replaced by the second subset of emoticon character keys (e.g., highlighting or visually distinguishing the subset sequence marker icon corresponding to the second subset of emoticon character keys and ceasing to highlight or visually distinguish another subset sequence marker icon corresponding to the third subset of emoticon character keys) (1238). For example, in response to detecting a swipe gesture 6030 (Figure 6C) on the touch screen display in the keyboard display area 5005 in a direction substantially opposite to the direction of gesture 6025 (Figure 6B), the device replaces the display of a third subset 6040 of emoticon character keys in the natural expression category with the display of a second subset 6020 of emoticon character keys in the natural expression category in the keyboard display area 5005, and updates the information provided by the subset sequence marker icon 6000 to reflect that the displayed third subset of emoticon character keys is replaced by the second subset of emoticon character keys (e.g., highlighting or visually distinguishing the subset sequence marker icon 6000-2 corresponding to the second subset 6020 of emoticon character keys and ceasing to highlight or visually distinguish the subset sequence marker icon 6000-3 corresponding to the third subset 6040 of emoticon character keys).

In some embodiments, the device displays a keyboard selection icon (e.g., icon 5020, FIG. 6B ). In some embodiments, keyboard selection icon 5020 is displayed simultaneously with multiple emoticon category icons 5010 (e.g., along the bottom of the touch screen display in a predetermined area 5009). The device detects a gesture on the keyboard selection icon (e.g., a finger tap gesture 6055, FIG. 6B ). After detecting the gesture on the keyboard selection icon, the device replaces display of a second subset of emoticon character keys for the corresponding emoticon category in the keyboard display area with display of a text character input keyboard (1240). For example, after detecting tap gesture 6055 on keyboard selection icon 5020, the device replaces display of a second subset 6020 ( FIG. 6B ) of emoticon character keys for the natural expression category in the keyboard display area 5005 with display of a text character input keyboard 5055 ( FIG. 5G ). In some embodiments, in response to detecting gesture 6055, the second subset 6020 of emoticon character keys is replaced with a text character input keyboard 5022. In some embodiments, one or more intermediate text character input keyboard selection UIs are shown before a particular text character input keyboard (eg, an alphanumeric keyboard, a kana keyboard, or a Kanji keyboard) is selected and displayed in area 5005 .

In some embodiments, the device receives one or more words entered by a user via a text character input keyboard (e.g., keyboard 5055, FIG. 5G ). The device displays the one or more words entered by the user in character input area 5000. In response to receiving the one or more words entered by the user via the text character input keyboard, the device determines recommended emoji characters based at least in part on the one or more words entered by the user, and the device displays the recommended emoji characters on the touch screen display (1242). The recommended emoji characters are determined based on the words entered by the user, rather than on punctuation entered by the user. For example, if the user types the word "rain," emoji characters representing an umbrella and/or raindrops may be recommended. Similarly, if the user types the word "fly," emoji characters representing an airplane may be recommended. In some embodiments, a single emoji character is recommended. In other embodiments, multiple emoji characters are recommended. In some embodiments, the device detects a gesture on the touch screen display (e.g., a tap gesture on the space bar in the text character input keyboard or a tap gesture on a recommended emoji character). In response to detecting the gesture, the device inserts and displays the recommended emoticon character (1244) after the one or more words entered by the user in the character input area.

In some embodiments, the device displays a keyboard selection icon 5020 (Figure 5G) while displaying a text character input keyboard 5055 in keyboard display area 5005. Then, upon detecting a gesture, such as a finger tap gesture, on keyboard selection icon 5020, the device replaces the display of text character input keyboard 5055 with a second subset 6020 of emoticon character keys for the corresponding emoticon category (Figure 6B) and a corresponding subset sequence marker icon 6000 in the keyboard display area (1246). In some embodiments, in response to detecting the gesture, the text character input keyboard 5055 is replaced with the second subset 6020 of emoticon character keys. In some embodiments, one or more intermediate emoticon keyboard selection UIs are shown before selecting and displaying a particular subset of emoticon character keys in area 5005. In some embodiments, in response to detecting the gesture on keyboard selection icon 5020, the device switches between displaying the text character input keyboard and the most recently displayed subset of emoticon character keys.

In some embodiments, the device detects a gesture on a corresponding emoji character key in the second subset of emoji character keys. In response to detecting the gesture on the corresponding emoji character key in the second subset of emoji character keys, the device inserts and displays the corresponding emoji character in the character input area (1248). For example, in response to detecting a tap gesture on an emoji character key in the second subset 6020 of emoji character keys, the device inserts and displays the corresponding emoji character in the character input area 5000. Figures 6C-6E illustrate similar insertion and display of emoji characters from the third subset 6040 of emoji character keys in the input area 5000. Additional emoji characters can be inserted and displayed in the character input area 5000 in response to detecting gestures on additional emoji character keys.

In some embodiments, the device detects a predetermined user action for initiating a predetermined user interface reconfiguration process (e.g., pressing and maintaining finger contact on an emoticon character key for more than a predetermined time period). In response to detecting the predetermined user action, the device changes the position of two or more emoticon character keys in a second subset of emoticon character keys. The change includes changing the position of each emoticon character key in the two or more emoticon character keys with respect to a corresponding average position, which is different from the corresponding average position of the other emoticon character keys in the two or more emoticon character keys (1249). For example, in response to detecting a finger contact on any emoticon character key 8001 (Figure 8A) for more than a predetermined time period (e.g., 1 second), the device changes the position of all emoticon character keys in a subset 8002 of emoticon character keys (Figure 8A, corresponding to subset 6020 in Figure 6B).

In some embodiments, each respective emoticon character key of the two or more emoticon character keys oscillates (1250) within a region substantially centered about the respective average position of the respective emoticon character key (as shown in FIG. 8B).

In some embodiments, the device detects a user's point of contact with the touch screen display at a first location corresponding to a first emoticon character key in the second subset of emoticon character keys, and also detects movement of the point of contact to a second location on the touch screen display. In response to both detections, the device displays movement of the first emoticon character key to the second location on the touch screen display based on the detected movement of the point of contact (1252). For example, in response to detecting a user's point of contact 8050 (Figure 8B) with the touch screen display at a location corresponding to emoticon character key 8001-2 and also detecting movement of the point of contact 8050 to the second location 825 on the touch screen display, the device displays movement of emoticon character key 8001-2 to the second location 825 on the touch screen display based on the detected movement of the point of contact.

In some embodiments, the device fixes the position of the first emoticon character key (e.g., 8000-2, Figure 8C) at a second position in response to detecting a second predetermined user action for terminating the predetermined user interface reconfiguration process (e.g., detecting activation of the home button 2004).

13 is a flow chart illustrating a method for responding to messages with emoticons while the portable electronic device 100 equipped with the touch-sensitive display 112 is in a user interface lock mode in accordance with some embodiments. The method 1300 provides a simple and intuitive way for a user to quickly respond to incoming messages while the device is locked, thereby conserving power and increasing the time between battery charges.

Operations 1302-1314 are performed while the portable electronic device 100 is in a user interface lock mode (1301). While the touch screen display is in a power saving state, the device receives (1302) an incoming message from a party to a user of the portable electronic device. In the power saving state, the touch screen is blank (displays nothing) and is not touch sensitive.

In response to receiving an incoming message, the device powers on the touchscreen display (1304) and displays at least a portion of the incoming message from the party to the user on the touchscreen display (e.g., message portion 2212, FIG. 9B ). The device detects (1306) one or more predetermined finger gestures on the touchscreen display that correspond to emoticon characters (e.g., gestures 9100-1, 91002-, and 9100-3 in FIG. 9C ). The device detects (1308) a predetermined user action that initiates sending a response to the incoming message (e.g., such as detecting activation of the home button 204, detecting activation of a "Send" button (not shown), or detecting a swipe gesture 2213, FIG. 9C ). In response to detecting the predetermined user action that initiates sending a response to the incoming message, the device sends (1310) a response to the incoming message. The response includes emoticon characters corresponding to the one or more predetermined finger gestures detected on the touchscreen display. For example, predetermined finger gestures 9100-1, 91002-, and 9100-3 in FIG. 9C may correspond to a smiley face emoticon character. A method of associating an expression with one or more predetermined finger gestures is described below with respect to Figures 10A-10E and 16.

In some embodiments, in response to detecting a predetermined user action to initiate sending a response to the incoming message, the device returns (1312) the touch screen display to the power saving state. In some embodiments, within a predetermined time period (e.g., 5-15 seconds) after sending the response to the incoming message, the device returns (1314) the touch screen display to the power saving state.

14 is a flow chart illustrating another method of responding to messages with emoticons while the portable electronic device 100 equipped with the touch-sensitive display 112 is in a user interface lock mode in accordance with some embodiments. The method 1400 provides a simple and intuitive way for the user to quickly respond to incoming messages while the device is locked, thereby conserving power and increasing the time between battery charges.

While the portable electronic device 100 is in user interface lock mode, actions 1402-1414 are performed (1401). While the touch screen display is in a power saving state, the device receives (1402) an incoming message from a user of the portable electronic device. In the power saving state, the touch screen is blank (displays nothing) and is not touch sensitive.

In response to receiving the incoming message, the device powers on the touch screen display and displays at least a portion of the incoming message from the party to the user (e.g., message portion 2212, FIG. 9D ) and a plurality of predetermined responses to the incoming message (e.g., responses 9110 in FIG. 9D or FIG. 9G ) on the touch screen display (1404). The predetermined responses 9110 include a plurality of emoticons (e.g., emoticon 9111 in FIG. 9D or only the emoticons in response 9110 in FIG. 9G ).

The device detects (1406) a predetermined user action for selecting a predetermined response from a plurality of predetermined responses (e.g., finger tap gesture 9120 on response 9110-1 in FIG. 9E or finger tap gesture 9210 on response 9110-4 in FIG. 9G ). The device detects (1408) a predetermined user action for initiating sending a response to the incoming message. In some embodiments, detecting selection of predetermined response 9110 also initiates sending of the predetermined response without detecting any other user action. In some embodiments, after detecting selection of predetermined response 9110, other predetermined user actions are detected that initiate sending of the response, such as detecting activation of home button 204, detecting activation of a “Send” button (not shown), or detecting a swipe finger gesture 2213 that appears to move unlock image 302 across channel 306 (in which case, the “Slide to View” message in channel 306 may become a “Slide to Send Response” message or similar message, FIG. 9E ).

In response to detecting a predetermined user action to initiate sending a response to the incoming message, the device sends (1410) a response to the incoming message.The selected predetermined response 9110 includes an emoticon character (e.g., 9111) from a plurality of emoticons (1410).

In some embodiments, in response to detecting a predetermined user action to initiate sending of a selected predetermined response to an incoming message, the device returns (1412) the touch screen display to the power saving state. In some embodiments, within a predetermined time period (e.g., 5-15 seconds) after sending the selected predetermined response to the incoming message, the device returns (1414) the touch screen display to the power saving state.

15 is a flow chart illustrating another method for responding to messages with emoticons while the portable electronic device 100 equipped with the touch-sensitive display 112 is in a user interface lock mode in accordance with some embodiments. The method 1500 provides a simple and intuitive way for the user to quickly respond to incoming messages while the device is in a locked state, thereby conserving power and increasing the time between battery charges.

Operations 1502-1518 are performed while the portable electronic device 100 is in a user interface lock mode (1501). While the touch screen display is in a power saving state, the device receives (1502) an incoming message from a party to a user of the portable electronic device. In the power saving state, the touch screen is blank (displays nothing) and is not touch sensitive.

In response to receiving the incoming message, the device powers on the touch screen display (1504) and displays at least a portion of the incoming message from the party to the user on the touch screen display (e.g., message portion 2212, Figure 9B).

The device detects (1506) a predetermined user action (e.g., a finger tap on a reply icon (not shown) or a double tap gesture on touch screen display 112) that initiates display of a plurality of predetermined responses 9110 to the incoming message. The predetermined responses include a plurality of emoticons (e.g., emoticon 9111 in FIG. 9D and only emoticons in response 9110 in FIG. 9G ). In response to detecting the predetermined user action that initiates display of a plurality of predetermined responses to the incoming message, the device displays (1508) the plurality of predetermined responses (e.g., responses 9110 in FIG. 9D or FIG. 9G ).

The device detects (1510) a predetermined user action for selecting a predetermined response from a plurality of predetermined responses (eg, finger tap gesture 9120 on response 9110-1 in Figure 9E or finger tap gesture 9210 on response 9110-4 in Figure 9G).

The device detects (1512) a predetermined user action that initiates sending of the selected predetermined response to the incoming message. In some embodiments, detecting selection of the predetermined response 9110 also initiates sending of the predetermined response without detecting any further action by the user. In some embodiments, after detecting selection of the predetermined response 9110, other predetermined user actions that initiate sending of the response are detected, such as detecting activation of the home button 204, detecting activation of a "Send" button (not shown), or detecting a swipe finger gesture 2213 that appears to move the unlock image 302 across the channel 306 (in which case the "Slide to View" message in the channel 306 may become a "Slide to Send Response" message or similar message, FIG9E ).

In response to detecting a predetermined user action that initiates sending of a selected predetermined response to the incoming message, the device sends (1514) the selected predetermined response to the incoming message. The selected predetermined response includes an emoticon character from a plurality of emoticons.

In some embodiments, in response to detecting a predetermined user action to initiate sending of a selected predetermined response to an incoming message, the device returns 1516 the touch screen display to the power saving state. In some embodiments, within a predetermined period of time (e.g., 5-15 seconds) after sending the selected predetermined response to the incoming message, the device returns 1518 the touch screen display to the power saving state.

16 is a flow chart illustrating a method, in accordance with some embodiments, for assigning a user-defined gesture to an emoticon on a portable electronic device 100 equipped with a touch-sensitive display 112. The method 1600 provides a simple and intuitive way for a user to associate a user-defined gesture with an emoticon character for later recall, thereby enabling the user to type an emoticon character (e.g., an emoticon character that the user frequently uses) on the device without having to use an emoticon keyboard.

The device detects (1602) a user-defined gesture (e.g., gesture 1030, FIG. 10B ) on a touch screen display while the portable electronic device is in a user-defined gesture recording mode. The device generates (1604) a representation of the user-defined gesture. The representation may include absolute or relative positions of finger contacts in the gesture and relative or absolute movement of finger contacts in the gesture. The device associates (1606) the representation of the user-defined gesture with an emoticon character (e.g., as described above with reference to FIG. 10C-10E ). The device stores (1608) the representation of the user-defined gesture and the association of the representation of the user-defined gesture with the emoticon character.

In some embodiments, after storing the representation of the user-defined gesture and the association of the representation of the user-defined gesture with the emoticon character, the device detects a gesture on the touch screen display and determines whether the detected gesture corresponds to the stored representation of the user-defined gesture. If the detected gesture corresponds to the stored representation of the user-defined gesture, the device inserts the emoticon character into an email or file, such as an instant message (e.g., a text message) or an email message (1610).

Figure 17 is a flow chart illustrating a method for using emoticons as location indicators in a map via an electronic message in accordance with some embodiments. Method 1700 is a quick and easy way to provide location data for emoticons on a map that augments the information in the corresponding electronic message.

A device (e.g., portable device 100 or desktop computing device) receives (1702) an electronic message that includes instructions for launching or operable to launch geographic mapping application 154, location data representing a location on a map, and emoticon data representing an emoticon character. The emoticon data is associated with the location data (e.g., such that an emoticon character corresponding to the emoticon data is displayed on the map at a location corresponding to the location data). In some embodiments, the electronic message is a text message with a link to open the map application (1704).

The device displays (1706) an electronic message on the touchscreen display (e.g., message 2216, FIG. 11A ). The device detects (1708) a predetermined user action on the displayed electronic message for launching a geo-mapping application (e.g., a tap gesture 1105 on a link 2220 to Lake Tahoe in message 2216, FIG. 11A ). In response to detecting the predetermined user action to launch the geo-mapping application, the device displays (1710) a map in geo-mapping application 154 (e.g., map 1115, FIG. 11B ) and displays an emoticon character (e.g., emoticon character 1120, FIG. 11B ) at a location on the map corresponding to the location data.

For purposes of explanation, the above description has been given with reference to specific embodiments. However, the above illustrative discussions are not intended to be exhaustive or to limit the invention to the precise forms disclosed. In light of the above teachings, many modifications and variations are possible. The embodiments have been chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling those skilled in the art to best utilize the invention and the various embodiments with various modifications as are appropriate to the particular application contemplated.

Claims (24)

1. A computer-implemented method, comprising: In portable electronic devices with touchscreen displays: Simultaneously display: It can be operated to display a character input area for text character input and emoji character input selected by the user. The keyboard display area can be operated to show text character keys and emoji character keys. The first plurality of emoji character keys in the keyboard display area, and The keyboard display area contains multiple emoji category icons, wherein the emoji category icons correspond to multiple emoji categories; Detect gestures on the corresponding emoji category icons among the multiple emoji category icons; In response to detecting the gesture on the corresponding emoji category icon: Maintain the display of the character input area; Maintain the display of the multiple emoji category icons in the keyboard display area; Stop displaying the first plurality of emoji character keys in the keyboard display area; and simultaneously display: A first subset of the emoji character keys of the corresponding emoji category in the keyboard display area, wherein the corresponding emoji category includes multiple subsets of emoji character keys configured to be displayed separately as a sequence of emoji character key subsets in the keyboard display area; and Multiple subset sequence marker icons for the corresponding emoji category, wherein the subset sequence marker icons provide information related to the total number of subsets configured to be displayed separately in the corresponding emoji category and the position of the displayed subset in the emoji character key subset sequence in the keyboard display area; While displaying a first subset of the emoji character keys for the corresponding emoji category in the keyboard display area, gestures are detected in the keyboard display area; and In response to the detection of the gesture in the keyboard display area: Maintain the display of the character input area; Maintain the display of the multiple emoji category icons in the keyboard display area; The first subset of the emoji character keys for the corresponding emoji category is replaced by the display of a second subset of the emoji character keys for the corresponding emoji category in the keyboard display area, and Update the information provided by the subset sequence marker icons to reflect that the first subset of the displayed emoji character keys is replaced by the second subset of the emoji character keys. 2. The computer-implemented method according to claim 1, wherein the plurality of emoji category icons includes recently and frequently used emoji category icons corresponding to a plurality of emoji characters recently and frequently selected by the user. 3. The computer-implemented method according to claim 1, wherein the emoji character keys in at least one emoji category are sorted according to emoji character usage data. 4. The computer-implemented method of claim 3, wherein the emoji usage data includes emoji usage data from a software application running on the portable electronic device. 5. The computer-implemented method of claim 3, wherein the emoji usage data includes emoji usage data received from the service provider of the portable electronic device. 6. The computer-implemented method according to claim 1, wherein replacing the display of the first subset of the emoji character keys of the corresponding emoji category with the display of the second subset of the emoji character keys of the corresponding emoji category in the keyboard display area comprises: An animation is displayed showing the first subset of emoji keys being removed from the touchscreen display. 7. The computer-implemented method according to claim 1, wherein replacing the display of the first subset of the emoji character keys of the corresponding emoji category with the display of the second subset of the emoji character keys of the corresponding emoji category in the keyboard display area comprises: An animation showing the second subset of moving emoji character keys on the touchscreen display. 8. The computer-implemented method according to claim 1, wherein replacing the display of the first subset of the emoji character keys of the corresponding emoji category with the display of the second subset of the emoji character keys of the corresponding emoji category in the keyboard display area comprises: An animation is displayed showing the first subset of the emoji keys being moved off the touchscreen display, and an animation is displayed showing the second subset of the emoji keys being moved on the touchscreen display. 9. The computer-implemented method according to claim 1, wherein the subset sequence marker icon is displayed in the keyboard display area. 10. The computer-implemented method according to claim 1, comprising: While displaying a second subset of the emoji character keys for the corresponding emoji category in the keyboard display area, gestures are detected in the keyboard display area; and In response to detecting the gesture in the keyboard display area: The display of the second subset of the emoji character keys of the corresponding emoji category is replaced by the display of the third subset of the emoji character keys of the corresponding emoji category in the keyboard display area; and The information provided by the subset sequence marker icons is updated to reflect that the second subset of the displayed emoji character keys is replaced by the third subset of the emoji character keys. 11. The computer-implemented method of claim 10, wherein the gesture detected while displaying the second subset of emoji character keys is a waving gesture in a first direction; and the method further comprises: Detecting a second waving gesture on the touchscreen display in the keyboard display area in a direction substantially opposite to the first direction; and In response to detecting the second waving gesture in the keyboard display area: The display of the third subset of the emoji character keys of the corresponding emoji category is replaced by the display of the second subset of the emoji character keys of the corresponding emoji category in the keyboard display area; and The information provided by the subset sequence marker icons is updated to reflect that the third subset of the displayed emoji character keys is replaced by the second subset of the emoji character keys. 12. The computer-implemented method according to claim 1, comprising: Display keyboard selection icon; Detecting gestures on the keyboard selection icons; and In response to detecting a gesture on the keyboard display icon, the display of the second subset of the emoji character keys of the corresponding emoji category and the plurality of emoji category icons are replaced by the display of the text character input keyboard in the keyboard display area. 13. The computer-implemented method according to claim 12, comprising: Receive one or more words input by the user via the text character input keyboard; Display the one or more words entered by the user in the character input area; and In response to receiving one or more words input by the user via the text character input keyboard: Based at least in part on the one or more words input by the user, a recommended emoji character to be inserted after the one or more words input by the user is determined; and The recommended emoji characters are displayed on the touchscreen display. 14. The computer-implemented method according to claim 13, comprising: Detecting gestures on the touchscreen display; and In response to the detection of the gesture, the recommended emoji character is inserted and displayed after the user enters one or more words in the character input area. 15. The computer-implemented method according to claim 12, comprising: While displaying the text character input keyboard in the keyboard display area, the keyboard selection icon is also displayed; Detecting gestures on the keyboard selection icons; and In response to detecting the gesture on the keyboard selection icon, a display is simultaneously displayed that replaces the text character input keyboard, consisting of the second subset of the emoji character keys of the corresponding emoji category in the keyboard display area, the corresponding subset sequence marker icon, and the plurality of emoji category icons in the keyboard display area. 16. The computer-implemented method according to claim 1, comprising: Detect gestures on the corresponding emoji keys in the second subset of the emoji key set; and In response to a gesture detected on the corresponding emoji key in the second subset of emoji key keys, the corresponding emoji character is inserted and displayed in the character input area. 17. The computer-implemented method according to claim 1, comprising: Detect the scheduled user action used to initiate the scheduled user interface reconfiguration process; and In response to detecting the predetermined user action, the positions of two or more emoji character keys in the second subset of emoji character keys are changed, wherein the change includes: changing the position of each of the two or more emoji character keys with respect to a corresponding average position, the corresponding average position being different from the corresponding average positions of the other emoji character keys in the two or more emoji character keys. 18. The computer-implemented method of claim 17, wherein each corresponding emoji key of the two or more emoji character keys oscillates in a region substantially centered on the corresponding average position of the corresponding emoji character key. 19. The computer-implemented method according to claim 17, comprising: Detect the user's contact point with the touchscreen display at a first position corresponding to a first emoji key in the second subset of emoji key sets, and detect the movement of the contact point to a second position on the touchscreen display; and In response to the detection of the contact point and the detection of movement of the contact point, the movement of the first emoji key to the second position on the touch screen display is displayed according to the detected movement of the contact point. 20. The computer-implemented method according to claim 19, further comprising: In response to detecting a second predetermined user action for terminating the predetermined user interface reconfiguration process, the position of the first emoji key is fixed at the second position. 21. The computer-implemented method of claim 1, wherein the keyboard display area is operable to display the text character keys in the text character input keyboard in lieu of the display of a corresponding subset of the emoji character keys and the plurality of emoji category icons. 22. The computer-implemented method of claim 1, wherein the gesture detected in the keyboard display area while displaying the first subset of the emoji character keys of the corresponding emoji category in the keyboard display area is a waving gesture. 23. A portable electronic device, comprising: Touchscreen display; One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for: Simultaneously display: Capable of operating to display the character input area for text and emoji character input selected by the user. The keyboard display area, which can be operated to show text character keys and emoji character keys, The first plurality of emoji character keys in the keyboard display area, and The keyboard display area contains multiple emoji category icons, wherein the emoji category icons correspond to multiple emoji categories; Detect gestures on the corresponding emoji category icons among the multiple emoji category icons; In response to detecting the gesture on the corresponding emoji category icon: Maintain the display of the character input area; Maintain the display of the multiple emoji category icons in the keyboard display area; Stop displaying the first plurality of emoji character keys in the keyboard display area; and Simultaneously display: A first subset of the emoji character keys of the corresponding emoji category in the keyboard display area, wherein the corresponding emoji category includes multiple subsets of emoji character keys configured to be displayed separately as a sequence of emoji character key subsets in the keyboard display area; and Multiple subset sequence marker icons for the corresponding emoji category, wherein the subset sequence marker icons provide information related to the total number of subsets configured to be displayed separately in the corresponding emoji category and the position of the displayed subset in the emoji character key subset sequence in the keyboard display area; While displaying a first subset of the emoji character keys for the corresponding emoji category in the keyboard display area, gestures are detected in the keyboard display area; and In response to the detection of the gesture in the keyboard display area: Maintain the display of the character input area; Maintain the display of the multiple emoji category icons in the keyboard display area; The first subset of the emoji character keys for the corresponding emoji category is replaced by the display of a second subset of the emoji character keys for the corresponding emoji category in the keyboard display area, and Update the information provided by the subset sequence marker icons to reflect that the first subset of the displayed emoji character keys is replaced by the second subset of the emoji character keys. 24. An apparatus for using emoji characters, comprising: In portable electronic devices with touchscreen displays, A device for simultaneously displaying the following: Capable of operating to display the character input area for text and emoji character input selected by the user. The keyboard display area, which can be operated to show text character keys and emoji character keys, The first plurality of emoji character keys in the keyboard display area, and The keyboard display area contains multiple emoji category icons, wherein the emoji category icons correspond to multiple emoji categories; A device for detecting gestures on corresponding expression category icons among the plurality of expression category icons; A device activated in response to detecting the gesture on the corresponding emoji category icon, comprising: Device for maintaining the display of the character input area; A means for maintaining the display of the plurality of emoji category icons in the keyboard display area; A means for stopping the display of the first plurality of emoji character keys in the keyboard display area; and A device for simultaneously displaying the following: A first subset of the emoji character keys of the corresponding emoji category in the keyboard display area, wherein the corresponding emoji category includes multiple subsets of emoji character keys configured to be displayed separately as a sequence of emoji character key subsets in the keyboard display area; and Multiple subset sequence marker icons for the corresponding emoji category, wherein the subset sequence marker icons provide information related to the total number of subsets configured to be displayed separately in the corresponding emoji category and the position of the displayed subset in the emoji character key subset sequence in the keyboard display area; A means for detecting gestures in the keyboard display area while displaying a first subset of the emoji character keys of the corresponding emoji category in the keyboard display area; and A device activated in response to detecting the gesture in the keyboard display area, comprising: Device for maintaining the display of the character input area; A means for maintaining the display of the plurality of emoji category icons in the keyboard display area; A means for replacing the display of the first subset of the emoji character keys of the corresponding emoji category with the display of the second subset of the emoji character keys of the corresponding emoji category in the keyboard display area, and A means for updating information provided by the subset sequence marker icons to reflect that the first subset of displayed emoji keys is replaced by the second subset of emoji keys.