HK1217793B - User interface for a computing device - Google Patents

Description

User interfaces for computing devices

Technical Field

The present invention relates to a user interface for a computing device; in particular, to a touch-based device such as a smartphone or tablet device.

Background Art

We now use many different types of computing devices (e.g., TVs, laptops, tablets, phones). Even when from the same company, their interfaces can be very different. This makes it more difficult for users to quickly transition between these types of devices; it requires users to learn new ways to interact with them, which can be frustrating, especially for users who are less technologically savvy or simply unwilling to invest the time and effort to learn new interaction techniques. It also makes the process of designing for these types of devices slower and riskier than necessary, as new interaction design methods must be devised for each new type of device. And it makes it more difficult for developers to write applications for each device, as they cannot reuse their software much across each different form factor. This inefficiency arises in part because the underlying operating systems of these different types of devices are not common or shared; furthermore, a user interface designed for a larger device (e.g., a laptop) is unusable if presented on a smartphone—a significant challenge for interaction designers working with the smaller screen of a typical smartphone.

Overall, the interaction design problem of creating a single, unified family of interfaces that: (a) enables fast, one-handed operation on a smartphone; (b) scales effectively from a smartphone to a tablet device where two-handed touch interaction is typical; (c) scales smoothly to accommodate keyboard and pointer input typical of a PC; and (d) scales to a television form factor that can be used with a simple remote control is substantial and has not been successfully solved to date.

This patent specification describes not only various ideas and features, but also their inventive expressions. Accordingly, a portion of the disclosure of this patent document contains material to which copyright is claimed, and a reminder is hereby given: Copyright is owned by Canonical Limited 2012 and 2013 (e.g., pursuant to 17 U.S.C. 401). Copyright protection is claimed for all screenshots, icons, look and feel, and all other protectable expressions associated with the user interface and user interaction illustrated and described in this patent specification.

The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the WIPO or U.S. Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. No licenses, express or implied, under any copyright are granted.

EP2059868B1 and EP2059868(A2) disclose a portable electronic device with a touch screen display for photo management, and also disclose a computer-implemented method, wherein the portable electronic device displays an array of thumbnail images corresponding to a set of photographic images. Upon detecting user contact with a corresponding thumbnail image in the array, the device replaces the displayed array of thumbnail images with a photographic image selected by the user. The user-selected photographic image is displayed at a larger size than the corresponding thumbnail image. In response to a scroll gesture, the portable device displays a different photographic image to replace the user-selected photographic image. The scroll gesture comprises a substantially horizontal movement of the user contacting the touch screen display.

Summary of the Invention

In a first aspect, a smartphone, tablet device, or other computing device is provided, comprising:

(a) a touch-sensitive display;

(b) one or more processors;

(c) computer memory;

(d) one or more computer programs stored in computer memory, configured to be executed by the one or more processors and comprising instructions for detecting a swipe from each and any of the four edges of the touch-sensitive display and, in response to the detected swipe, causing the device to operate in a manner that depends on the particular edge (e.g., left, right, top, or bottom) from which the swipe originated.

This document will describe innovations in the following areas:

A. Edge gestures

B. Ranged Gesture

C. Welcome screen

D. Unlocking mechanism

E. Ubuntu search experience

F. Home screen in Ubuntu

G. Ubuntu "Side-Stage"

H. Scalable and responsive Ubuntu UI

In this section we provide an overview of these innovations.

A. Edge Gestures

A central invention in the Ubuntu User Experience (UX) is the design of the role each edge plays in system interaction, settings, applications, and controls. In the Ubuntu UX, the layout of screens is designed to ensure:

- Smooth scalability from phone to tablet, to PC, and to TV

- Efficient touch interaction that maximizes the screen space available for content and applications and eliminates the need for physical or software "system buttons"

The best expression of this design is “edge magic.” In Ubuntu UX, and especially in the touch version of Ubuntu UX, the edges of the screen play a key, unique, and innovative role.

A.1 Driving application startup or switching from the left or right edge

Swiping in from the left edge is always used to access apps. This is easy to do in both phone (one-handed) and tablet (two-handed) interfaces, which provide more space for apps than the bottom edge of a phone (such as WebOS) and take up a relatively small amount of screen space compared to the horizontal taskbar at the bottom of the screen (such as MacOS or Windows 7). Favorite and running apps can be accessed directly through the "Launcher," which is a vertical bar of app icons accessed through the left edge (e.g., a short swipe in from the left edge). In this document, we will refer to the launcher as both a "launcher" and an "app launcher." Further apps can be reached through the "Home Screen" app page (which is a grid of apps on the device that indicates which apps are running and which are closed; and also displays apps potentially relevant to the user that can be obtained from the server). This Home Screen app page can also be accessed directly from the left edge (e.g., by a longer swipe in from the left edge). Thus, the left edge serves as a consistent gateway to any app. The launcher bar also includes an icon for the "Home screen"; the Home screen displays icons for frequently used applications, the user's recent calls made or received (or missed calls, so that the user can easily return a call by simply touching the item listing the missed call to initiate the return call directly). The Home screen is also fully customizable, so it can include, for example, "Recently Added Music"; New Movies; Online Radio; Wikipedia; Online Stores, and more.

In contrast, swiping in from the right edge accesses the previously used but still running app. It provides a convenient way to return to the previous app. Using the opposite edge to "switch to the next app" and "switch back to the previous app" (that is, swiping in from the left edge to access the launcher bar, then the user's finger moves up or down to release on a specific app icon in the launcher to switch to that app; conversely, swiping in from the right edge switches back to the previous app) is unique to Ubuntu and provides usability benefits: people learn the behavior quickly and it feels very natural.

A.2 Reflects system settings and status on the top edge

The top edge houses a collection of status indicators and provides access to system-wide settings and features (icons corresponding to network, battery, clock and calendar, sound, etc. are typically placed on the right side of the top edge) as well as system or system-wide capabilities (such as messaging and search; icons for these are typically placed on the left side of the top edge).

A.3 Accessing App Controls via the Bottom Edge

Ubuntu uniquely defines the bottom edge of the screen as a way to access controls corresponding to applications - typically, these controls are implemented in various forms, specific to each application: for example, playback controls and timeline in a video player application, a toolbar with action buttons on an image editor application, a game menu for a gaming application are all examples of controls that can be included in a sidebar, window, or area revealed by a brief upward swipe from the bottom edge.

Using one edge consistently as the gateway to app controls enables app developers to remove these controls from their interfaces by default, since users will know the standard way to access them. This has the benefit of giving more space for app content, making Ubuntu feel roomier than before on smaller screens like phones.

The control in question can be a button, or it can be voice control. For example, the Ubuntu HUD is a searchable command interface that is also invoked via the bottom edge. The HUD allows the user to access application controls that cannot be accommodated in a simple touch toolbar or other chrome-based ("on-screen buttons") interface. The mechanism is called a HUD ("heads-up display") because it displays information to the user without forcing the user to shift their primary focus of interest, similar to a traditional HUD.

This document is written from the perspective of a left-to-right language user. In the case of a right-to-left language, the UX can be reversed horizontally (so that the left side becomes the right side). The design of edge behavior and conventions is a key innovation at the heart of the Ubuntu UX innovation for touch devices. It distinguishes Ubuntu from previous systems, even those that used the edge for other purposes. For example, the Nokia N9 uses a universal edge gesture to return to the home screen, and the Palm webOS phone uses the bottom edge to access favorite apps, but Ubuntu provides substantially richer use models for the scarce real estate of the screen edge, and combines them in unique ways to produce a more usable, feature-rich system than any previous system.

B. Long distance gestures

Another important innovation in Ubuntu UX for touch devices is the use of long-range gestures, meaning gestures that can have multiple different outcomes depending on their end position or length of travel. We show that not only is a single progressive gesture, such as sliding a control along a slider, variable in outcome, but gestures that can have any number of widely varying outcomes depending on the scope of the gesture (i.e., distance, length, direction, path, start point, end point).

Long-range gestures are important in the Ubuntu UX because they allow a single resource (the edge of the screen) to become a gateway to multiple different results or capabilities. They also allow users to navigate through all of its capabilities in a faster and more fluid manner, where multiple "steps" or "taps" or "swipes" can be combined into a single directional gesture by more sophisticated users.

Long gestures can contain various stages (an extend gesture takes you through a series of modes, an end gesture selects a specific action from the mode you're currently in, or allows you to continue making choices with subsequent gestures). Long gestures can also contain forking paths, where the gesture can flow in a series of different directions to expose different options.

B.1 Application edge (left edge) long distance gesture

A swipe across the left edge is a long-distance gesture with several stages.

An initial left edge swipe will simply reveal the launcher. The initial prompt for the launcher can be dismissed by reversing the swipe back across the edge or releasing very early in the prompt. However, a smaller swipe is sufficient to reveal the launcher with favorite apps. Releasing at this stage keeps the launcher visible for manipulation with further touches, swipes, or taps.

A longer swipe will select the most recent apps on the launcher (that is, the app corresponding to the icon closest to the touch path of the user's finger, plus the Home screen icons), and moving the contact or touch point up and down allows the user to select a specific app in addition to the app closest to their finger when it initially crossed the edge; apps that have been selected (but not yet actually selected) will typically be displayed with icons larger than their adjacent icons. This approach allows for smooth selection of apps to launch or switch to on the launcher. Releasing the touch on the selected icon corresponding to a specific app will select that app.

In a long gesture, pausing on an icon in the launcher will display the app's Quicklist, allowing the long gesture to fork and follow the items in the Quicklist. This allows for smooth access to a specific item in a specific app's Quicklist in a single gesture that starts with an edge swipe. An app's Quicklist is a specific feature in an app that allows users to quickly access a feature, function, or control of the app. For example, if the app is an email messaging app, the Quicklist could be to start composing a new email message.

If the gesture is extended quickly across the launcher to the right, instead of pausing on an icon, a new stage is reached, which reveals the home screen's application page. This stage provides access to applications that are not in the launcher (where favorite or running applications are located) or even not installed.

Thus, long-reach gestures allow the left edge to provide comprehensive access to apps on the device—favorites, running apps, installed apps, and apps not yet installed.

B.2 Right edge long distance gesture

The right edge has a long-reach gesture that controls access to the previously focused application (i.e., the application the user was previously using) on a touch device, such as a phone or a tablet device with a visible "side stage" (the "side stage" is described more fully later in this document).

The initial stage of a swipe gesture across the right edge will display a hint about the previous app, or a hint about an app that had no previous focus. Releasing during this early part of the gesture will have no effect, returning the system to its state before the hint—to prevent unintended actions—which is switching to the previous app.

Continuing the gesture will confirm displaying the previous app as the focused app on a phone, or in the "side stage" of a tablet device (see below for a description of how the "side stage" can display apps independently of what's shown on the rest of the display).

On tablet devices where the side stage is present, a right edge swipe initially gives a prompt at the side stage, then confirms displaying the side stage as an overlay, and finally confirms splitting the screen between the main stage and the side stage.

B.3 System (top) edge long distance gestures

The top edge provides access to system services, settings, and search via long-reach gestures.

The Ubuntu UX top edge reach gestures are unusual in that they allow very quick and fluid access to information that powers core system indicators at the top edge on the right side of the screen.

On existing phones, it's difficult to interact with system indicators. On iPhone iOS 6 and earlier, you need to find and then switch to a special app, the Settings app, which provides access to system settings (and possibly app settings).

On Ubuntu, the settings are easily accessible via the top edge. They're on the right side of the top edge, so an initial swipe across the right portion of the top edge (for a full-screen app) reveals the system indicators (if they're off-screen) and selects the nearest system indicator. This can be reversed by reversing direction and swiping back up past the top edge to leave the screen. At this point, the long-range gesture can be bifurcated—moving left or right (without lifting the finger) enables selection of a specific system indicator, and moving down confirms focus on the selected indicator.

This allows:

- Quickly access any specific system setting or indicator;

- Quickly view various system information (e.g. "Which network am I on, and what new messages have arrived?")

Additionally, the left portion of the top edge is dedicated to search. Therefore, initiating a gesture at the top left of the screen provides access to various system search options.

B.4 Application Control (Bottom) Edge Long Distance Gestures

The bottom edge in Ubuntu provides access to and control over application controls. Given that various applications may follow very different designs for very different purposes, the nature of these controls may also vary. But in general, swiping the bottom edge should hide and reveal appropriate forms of controls for the application. These controls may be toolbars, or playback controls (play, pause, rewind, forward), or a timeline, or may involve changing the state of the application (for example, pausing a game and presenting menu options).

The bottom edge also has a long-range gesture—an initial swipe across the edge gives a hint about a control, which can be reversed by changing direction and returning off the bottom edge, or by releasing the gesture before the hint becomes a confirmation reveal for the control. Once the control is revealed, the gesture can be continued and a new stage will be revealed, including the ability to launch the HUD (a text- and voice-searchable control system for apps). Releasing the gesture on a HUD icon or at one of the other long-range endpoints of the gesture confirms the result.

C. Welcome screen

When a user turns on the device, the device provides a beautiful and sophisticated reflection of the user and their activities in a live welcome screen, which is a data visualization of the user's personal or device-specific data.

C.1 Data visualization that is different from application information or wallpaper

Displaying data visualizations on the welcome screen is innovative. On existing phones and tablets, it's common to have a personal image ("wallpaper") on the lock screen, or in some cases, to indicate the status of an app ("5 new messages"). Ubuntu is also innovative in that the data visualization can be a composite of data from multiple sources—from apps, online services, and devices—rendered in an artistic way.

The data visualization may or may not be interactive and may or may not trigger the launch of an application or content to unlock the device based on the interaction.

The data visualization may be combined with or part of a personal artwork or wallpaper, and the color palette may be drawn from the artwork to combine the visualization and wallpaper in an aesthetically appealing way.

Data visualizations can be chosen from a variety of options or randomly selected, or dynamically updated by the operator, OEM (or any other partner) or Ubuntu itself. It is also possible for users to design and install their own visualizations.

D. Unlocking mechanism

This "welcome screen" serves a similar purpose to the "lock screen" found on existing phone and tablet devices—it prevents inadvertent disclosure of sensitive information and prevents inadvertent input when the device is unnoticed and perhaps touched inside a bag or pocket. However, it is unique and innovative in several ways:

- In existing lock screens, the device behaves very differently when locked and unlocked. For example, the button that normally takes you to the home screen no longer does so when the device is locked. On Ubuntu Phone and Tablet devices, which use the edges of the screen to navigate between apps, controls, and settings, the Welcome screen behaves like any other app. To "unlock" the device, you use one of the edges (i.e. swipe in from there) to navigate directly to the app or setting you want. On a traditional phone, you need to unlock before you can do anything. On Ubuntu, you can start doing anything and it only asks for access controls when needed.

- When the Welcome screen is displayed, it's possible to launch any favorite app. In existing lock screens, there will be a handful of apps (which may be customizable) that can be launched when locked, but the mechanisms for launching them will be very different when locked than when unlocked. For example, on the iPhone 4S, there is an "unlock" mechanism (a sliding switch) and a mechanism for launching the camera when locked, but these mechanisms are completely different from the normal mechanisms for launching the camera app from the Home screen. The Ubuntu approach is much more consistent between locked and unlocked states.

- On Ubuntu Phone and Tablet devices, based on edge gestures, there's a clear distinction between unlocking to launch a favorite app, unlocking to the home screen, unlocking to the app that was running when the phone was locked, and unlocking to modify system settings. On the existing lock screen, "unlock" can take you to the home screen or to an app, depending on what you were doing when the phone was locked.

BRIEF DESCRIPTION OF THE DRAWINGS

Some implementations of the present invention will be described below with reference to the accompanying drawings, in which:

FIG. 1 illustrates an example of a core layout of a user interface for an application on a screen of a computing device.

FIG2 illustrates an example of a core user interface layout that is responsive and scalable across devices.

FIG3 shows an example of a smartphone user interface in which bottom controls are provided.

FIG. 4 shows several examples of smartphone user interfaces in which bottom controls are provided.

FIG. 5 shows an example of a tablet device user interface in which bottom controls are provided.

FIG6 shows a sequence of device screen configurations from (A) to (E) regarding a HUD interface instance.

FIG. 7 illustrates a sequence of device screen configurations from (A) to (D) regarding a HUD interface instance.

FIG8 illustrates a device screen configuration for an example HUD interface, showing a search input box and an application stack.

FIG. 9A shows an example in which search is brought down by swiping down from the upper left corner in the home screen.

FIG. 9B shows an example of organized search results.

FIG. 10 shows an example in which the “Favorite People” column in (A) collapses when scrolling down to leave the collapsed column “Favorite People” in (B).

FIG. 11 shows an example in which the "Favorite People" column in (A) collapses when scrolling down to leave the collapsed column "Favorite People" in (B).

FIG. 12 shows an example in which the “Running Applications” column in (A) collapses when scrolling down to leave the collapsed column “Running Applications” in (B).

FIG. 13 shows an example in which the “Recent” column in (A) collapses when scrolling down to leave the collapsed column “Recent” in (B).

FIG. 14 shows an example in which the “Recent Contact” field in (A) collapses when scrolling down to leave the collapsed field “Recent Contact” in (B).

FIG. 15 shows an example in which the "Favorites" column in (A) collapses when scrolling down to leave the collapsed column "Favorites" in (B).

FIG. 16 shows an example in which the “Recent” column in (A) collapses when scrolling down to leave the collapsed column “Recent” in (B).

FIG. 17 shows an example in which the “Recent” column in (A) collapses when scrolling down to leave the collapsed column “Recent” in (B).

FIG. 18 shows an example in which, in the sequence from (A) to (I), a message list is brought down as a finger swipes downward from a message icon.

FIG. 19 shows an example where, in the sequence from (A) to (B), a message list is brought down as a finger swipes downward from a message icon.

FIG. 20 shows an example of a pre-defined list of messages from which quick replies may be selected for sending.

Figure 21 shows an example (A) in which an area is expanded and the user can choose to send a text message or return a call, and an example (B) in which a text box for entering a text message is opened within a message in a message list.

FIG. 22 shows an example where, in the sequence from (A) to (C), a note is expanded when tapped, and a touch screen keyboard is provided for editing the note content.

FIG. 23 shows an example where, in the sequence from (A) to (B), a note is expanded when tapped, and a touch screen keyboard is provided for editing the note content.

Figure 24 shows an example in (A) where the conversation field page displays missed calls and messages, and in (B) where SMS messaging is separated into its own application called Messaging and phone calls are no longer displayed as part of the conversation history.

FIG. 25 shows an example where the sequence from (A) to (D) shows an example of a launcher bar revealed after a brief swipe from the left edge of the screen.

Figure 26 shows an example where the sequence from (A) to (C) shows an example of a launcher bar scrolling down (from (A) to (B)) and an application icon revealing the name of the application (in this example for the application "Phone", from (B) to (C)).

FIG27 shows an example where long pressing a launcher icon displays the application name.

FIG28 shows an example where long pressing a launcher icon displays the application name.

Figure 29 shows an example of a sequence of screen images, including (A) to (E). In (A), the user selects the "Notebook" application from the launcher bar. In (B), the screen displays the Notebook application screen output. In (C), the user is swiping their finger across the screen from the right edge of the screen to return to the previous "Events" application. In (C) and (D), the screen output from the previous "Events" application is returning to the screen from the right. In (E), the screen output from the "Events" application is displayed.

FIG. 30 shows an example in which the Contacts screen changes to a call screen to the Contacts screen in the sequence from (A) to (B).

FIG31 shows an example where, for the sequence from (A) to (B), a long swipe from the left edge of the screen brings out the app page.

Figure 32 shows an example where, for the sequence from (A) to (D), a left swipe from the right edge of the screen brings up the home page.

Figure 33 shows an example where, for the sequence from (A) to (C), a left swipe from the right edge of the screen brings up the home page.

FIG. 34 shows an example in which, for the sequence from (A) to (E), successive setting pages are brought out starting from (A).

FIG. 35 shows some examples in which, for screens (B) to (E), starting from screen (A) can bring out the setting screens shown in (B) to (E).

FIG. 36 shows an example of a battery settings screen on a tablet computer screen.

Figure 37 shows an example of a sequence of screen images, including (A) to (C). In (A), output from the "Events" application is displayed. In (B), the "Notes" side stage is being pulled into view by the user swiping their finger horizontally from the right edge of the screen. In (C), the "Notes" side stage has been pulled into view.

FIG38 shows an example of a welcome screen including an infographic in which each dot in a circle represents a day of the month, and the highlighted dot is the current day.

FIG39 shows an example of a welcome screen on a tablet device.

DETAILED DESCRIPTION

One implementation of the present invention is Ubuntu Touch (Ubuntu Touchscreen Edition). This section will describe Ubuntu Touch in more detail.

A.1 “Edge Gestures”

In Ubuntu, swiping from an edge causes the device to behave in a manner (e.g., displaying a user interface (UI) graphic or one or more objects, opening an application, opening a control pane, invoking a function, etc.) depending on the specific edge (e.g., left, right, top, or bottom) from which the swipe originates. The richness and intuitive simplicity of edge swipe behaviors in Ubuntu are key elements. Ubuntu uses many different gestures—one of which is a single, touch-based interaction with the device's touchscreen.

This characterization can be rephrased in the form of a problem and its solution. The problem statement is: How can we design a device interface that maximizes the space available for content and applications on the screen?

Ubuntu's "edge gestures" illustrate many of the solutions to this problem. For example, in Ubuntu, a swipe from the right edge takes you back to the previous app. So, if you're in the Facebook app but need to quickly check something (like a diary event), it's easy to launch the diary app by briefly swiping from the left edge to bring in the launcher bar, then swipe down to release on the diary app, check the event, and then swipe from the right edge to get back to the Facebook app. This combination of quickly opening an app (or delegating a task to it) by swiping from the left edge and then returning to the previous app by swiping from the right edge makes the interaction very fast, efficient, and intuitive.

This specific feature can also be rephrased in the form of a problem and its solution. The problem statement is: How can you design a device interface that allows for quick, one-handed launching of new applications and returning to previous applications? The solution can be generalized to a smartphone, tablet, or other computing device in which the "Start/Launch" and "Back" functions are controlled by swipe gestures from opposite edges of the screen. In one implementation, Start is initiated with a swipe from the left edge; this brings up the Launcher. Conversely, a swipe from the right edge brings up the previously opened applications, thereby acting as a "Back" function. Swiping further to the left from the right edge causes the user to scroll back through all previously used applications in the order they were last used.

Launcher-based unlocking gives us the ability to: unlock directly to the launcher to launch any favorite app; unlock to the quick list (that is, a specific feature in the app, for example if the app is an email messaging app, the quick list could be to start composing a new email message).

Another Ubuntu feature is the presence of system status and function icons at the top right of the screen, such as time and date, volume, network, messaging, and battery. An icon can be selected by moving a finger across this row, and then swiping down from it (or simply touching it and pulling down) reveals a screen listing the settings associated with that icon, allowing for quick updates without leaving the current app. For example, there's a network icon; swiping to it and then swiping down from it reveals traditional network parameters, such as "Airplane Mode," enabling WiFi, and checkboxes for the 3G network to join and the carrier to use. Users can quickly check and change any of these options without leaving their current app. This differs from how settings are accessed on an iPhone, for example, where users must leave the currently active app, find a specific icon for phone-related settings, and then tap on it to view them. The solution implemented in the Ubuntu UI enhances the simplicity of the phone and adheres to the Ubuntu UI theme. Furthermore, by reducing "excise," it provides a more efficient user experience. (Additional work is unnecessary work required to achieve UX goals—it’s the burden or “additional work” on top of the idealized, perfectly frictionless user interaction with optimal cognitive load.)

By simply swiping it up, the status icons can be hidden, giving the app 100% of the screen, allowing people to fully immerse themselves in the content. A short swipe down will reveal the status icons again. This is very useful on small-screen devices such as smartphones.

This feature can be rephrased in the form of a problem and its solution. The problem statement is: How can a user be enabled to check and change system parameters on the phone without leaving the primary application the user is running? The solution can be generalized to a smartphone (or other personal computing device) where the user can select a system status icon near the top edge and swipe down from there (e.g., directly touch the icon and then pull down) to reveal a screen listing the setting(s) associated with the icon, allowing the user to quickly update the setting(s), and then swipe back up the settings page to show the current application still running. Likewise, one can reverse this approach and swipe up from the bottom edge to reveal a screen listing the setting(s) associated with the icon.

Another Ubuntu feature is how it optimizes the space given to apps' on-screen controls. In Ubuntu, these controls are placed at the bottom edge and are only revealed by swiping up from the bottom edge, and then hidden by swiping down on them all the way to the bottom edge. This design is about space management, and the result is a primary visual experience that ensures 100% of the screen is dedicated to apps most of the time, and that these controls are quickly and intuitively available, but only when needed.

This approach is used not only within apps, but also outside of them, whether the user is viewing their home screen, welcome screen, or perhaps flipping between open apps. In all cases, the appropriate controls are revealed by briefly swiping up from the bottom edge. Saving this space gives a cleaner, simpler look and presents important information to the user only when they specifically need it.

This feature can be rephrased in the form of a problem and its solution. The problem statement is: How do you reconcile the need to provide application controls that the user can interact with while not having those controls take up valuable screen real estate? The solution can be generalized to a smartphone, tablet, or other computing device where the application controls are placed at an edge (e.g., the bottom edge), but are revealed only by a brief upward swipe from that edge and then hidden by a downward swipe.

A.2 Summary of Edge Behavior

To summarize some of the key edge cases:

-Swipe from the left edge to bring up the launcher; swipe further to bring up the home page of running apps.

-Swipe from the right edge to bring up recent apps; swipe further to bring up earlier apps.

-Swipe from the top edge to bring up the system status parameters corresponding to the system icons displayed on the right side of the top edge. Swipe up to hide them again.

-Swipe from the bottom edge to bring up app-specific controls and parameters for voice control for that app. Swipe down to hide them again.

By hiding these items until a swipe reveals them, the screen is kept uncluttered and the maximum amount of screen real estate can be dedicated to running apps. The phone itself feels larger and has more room than phones running competing operating systems.

As mentioned before, some of these gestures are "long reach gestures" (see Section C, Swipes and Long Reach Gestures in Ubuntu) in that by continuing to swipe you will bring up further (but related) functions/screens and so on.

It's also possible to speak control inputs to an app (see "HUD Parameters" below); a long swipe upwards from the bottom edge opens a voice-enabled HUD search interface for that app.

A.3 Edge Interactions in the Welcome Screen

Ubuntu's comprehensive use of edge gestures—where each edge has a unique purpose that is consistent across all apps and whether the device is locked or not—creates usability issues for new users, who may not understand the importance of the edges, their functionality, or how to use them. And because there's a deliberate absence of "chrome" (buttons or other visual indicators of the edges) that could detract from the aesthetics and spaciousness of the interface, users have little guidance in exploring the edges.

Ubuntu has implemented several solutions to this problem:

- Accelerometer-based edge notifications on the welcome screen

- On devices with accelerometers or other sensors to track movement, it's possible to display visual cues at the edges of the screen as the phone is moved. For example, shaking the phone could reveal a small portion of the launcher at the left edge of the screen, as if it had just "loosened" off the screen, and shaking it to allow it to slightly "rock" back onto the screen. Similar cues could draw the user's attention to the right, bottom, and top edges, each of which has a specific purpose in the Ubuntu user experience.

- Help activated by touching Explore

- When faced with a "blank" or "buttonless" welcome screen, users will often touch an image or data visualization, or swipe, tap, or rub the welcome screen to see what happens. Unlocking the screen by such gestures is undesirable because it may allow unintentional unlocking. Ubuntu's design solves this problem by displaying a progress bar that measures the number of such gestures, the full completion of which triggers a "help" experience that familiarizes the user with the edge-driven Ubuntu experience, including the ability to play "edge learning" games.

-Edge Learning Games

- One way to familiarize users with an edge-based user experience is to provide games that can be launched from a help experience, encouraging users to use the edge in the way they will need to in order to get the most out of the device.

A.4 Main edge gesture concepts

In this section, we summarize the main "edge gesture" concepts. These concepts are as follows:

A smartphone, tablet, or other computing device comprising:

(a) a touch-sensitive display;

(b) one or more processors;

(c) computer memory;

(d) one or more computer programs stored in computer memory, configured to be executed by the one or more processors and comprising instructions for detecting a swipe from each and any of the four edges of the touch-sensitive display and, in response to the detected swipe, causing the device to operate in a manner that depends on the particular edge (e.g., left, right, top, or bottom) from which the swipe originated.

Other optional implementation features include the following, each of which can be combined with any other feature:

- The device functions in a manner dependent on the particular edge swiped from by displaying a particular type of UI graphic, object, or set of objects dependent on the edge swiped from.

-The device always responds to detected swipes in a consistent manner, regardless of what the device is displaying.

- The device detects a swipe in from one edge of the screen and, in response to the detected swipe, causes the device to open an application launcher that is a collection or group of application icons, or to start or switch to an application, either directly or after further intermediate steps.

- The device detects a swipe in from the edge of the screen and, in response to the detected swipe, causes the device to switch back to the previously open application.

- In response to detecting a swipe-in from the left edge, the device opens an application directly or after further intermediate steps, and in response to a swipe-in from the right edge, the device displays a previously used but still running application directly or after further intermediate steps.

- If the device is displaying a welcome screen, then in response to a swipe-in from the right edge, the device displays the device home page, either directly or after another intermediate step.

- Swiping right from the left edge initiates "Start" to bring up the app launcher; conversely, swiping left from the right edge brings up the previously opened app, acting as a "Back" function.

In response to a further left swipe from the right edge, the instructions cause the device to scroll back through all previously used applications in the order in which they were last used.

- The device can detect when the user selects a system status icon located near one edge, and in response to a swipe away from the icon (e.g., the user directly touches the icon and then pulls down), the instructions cause the device to reveal a screen listing the setting(s) associated with the icon, thereby allowing the user to quickly update the setting(s) and then swipe the settings page back to the edge to display the current application still running. The system status icon can be placed at the top edge of the display, such as on the right side of the top edge.

-These instructions cause the device to display application controls in response to a brief upward swipe from an edge, to display application controls near the edge when an application is running and controlling the device display, and to hide application controls by swiping back to the edge. Application controls may be placed at the bottom edge of the display.

- In response to a swipe from a message icon, such as an envelope, the instructions cause the device to display a panel or window or area listing one or more messages.

- In response to the user selecting a message in a panel, window or area listing message(s), then displaying a response panel, window or area.

-If the selected message is for a missed call reminder, the response panel, window or area includes buttons or icons that enable the user to do one or more of the following: call the person who left the message back; compose and send a reply message to the person; select and send a pre-defined reply to the person.

- In response to detecting that the user pauses on an icon for an application in the application launcher, the device displays a "quick list" corresponding to the application, which is one or more user interface objects associated with features, functions, or controls of the application.

- A swipe-in from an edge must start at or outside the edge of the touch screen display.

The device includes an accelerometer, the processor receives movement signals from the accelerometer, and in response to these signals exceeding a certain threshold, displays a small portion of the screen that is different from the currently displayed screen as a prompt to the user. The threshold may be associated with shaking the device.

Other aspects are:

A method comprising the following steps performed at a computing device with a touch-sensitive display:

(i) detecting a swipe from each and any of the four edges of the touch-sensitive display and responding to the detected swipe;

(ii) causing the device to behave in a manner that depends on the particular edge (eg, left, right, top, or bottom) from which the swipe-in originates.

- The device behaves in a manner that depends on the particular edge from which the swipe-in originated, by displaying a particular type of UI graphic, object, or set of objects that depends on the edge from which the swipe-in originated.

A non-transitory computer-readable storage medium storing one or more programs comprising instructions that, when executed by a computing device with a touch-sensitive display, causes the device to:

(i) detecting a swipe from each and any of the four edges of the touch-sensitive display and responding to the detected swipe;

(ii) Operate in a manner that depends on the specific edge (eg, left, right, top, or bottom) from which the swipe-in originates.

A smartphone, tablet, or other computing device comprising:

(a) a touch-sensitive display;

(b) one or more processors;

(c) computer memory;

(d) one or more computer programs stored in computer memory, configured to be executed by the one or more processors and comprising instructions for detecting a swipe from any two opposing edges of the touch-sensitive display and, in response to the detected swipe, causing the device to function in a manner that depends on the particular edge from which the swipe originated;

And wherein in response to a swipe in from the left edge, the device opens an application directly or after further intermediate steps, and in response to a swipe in from the right edge, the device displays a previously used but still running application directly or after further intermediate steps.

- Bring up the app launcher from the left edge by swiping right from the left edge to initiate Start.

- In response to a further left swipe from the right edge, the device scrolls back through all previously used apps in the order in which they were last used.

- Swiping right from the left edge initiates "Start" to bring up the app launcher; conversely, swiping left from the right edge brings up the previously opened app, acting as a "Back" function.

- A swipe-in from an edge must start at or outside the edge of the touch screen display.

In response to detecting a swipe, two or more different results or functions are triggered or invoked depending on the extent of the detected swipe. The detected extent can be a function of one or more of the following swipe parameters: distance, length, direction, path, end point, or destination.

- A swipe from an edge results in a first result, and if the user continues to swipe in from the edge, a different result results. These two or more different results or functions are typically related or associated results.

- If the swipe continues to a predefined area on the screen to select that area, it results in a specific result.

- A swipe-in from the left edge is detected further to the right, causing the device to display a home screen including icons corresponding to all running applications.

- Lifting off the touch screen is a confirmation gesture.

- A swipe involves the user's finger continuously contacting the touch screen and moving across the screen in one or more directions.

A method comprising the following steps performed at a computing device with a touch-sensitive display:

(i) detecting a swipe from any two opposing edges of the touch-sensitive display and, in response to the detected swipe, causing the device to function in a manner that depends on the particular edge from which the swipe originated;

(ii) and wherein in response to a swipe in from the left edge, the device opens an application directly or after another intermediate step, and in response to a swipe in from the right edge, the device displays a previously used but still running application directly or after another intermediate step.

B.1 Long-distance gestures

Ubuntu uniquely organizes sets of actions into "long-range gestures," so that you can access multiple actions from a particular gesture, based on how many of that gesture you've completed.

There are often sets of related actions, for example:

-The launcher is displayed;

-Shows a specific item in the launcher

- Displays a list of shortcuts corresponding to specific items in the launcher

-Shows all running applications

-Shows all installed and available apps.

Each of these actions is associated with one or more applications. In Ubuntu, a swipe across the left edge can take you to each of these actions, depending on how many swipes you make. The actions are organized into "long-range gestures." An initial swipe across the left edge reveals the launcher (which shows icons for all running and favorite applications), continuing the gesture allows you to select a specific item on the launcher (to launch or switch to it), pausing the touch on an application's icon opens a shortcut list for that application, and continuing further to the right (in the same fluid gesture) shows all installed applications (which are shown as part of the "Apps" page of the Home screen; the Home screen also shows other available but not yet installed applications).

Most traditional touch interfaces work in such a way that a gesture or movement on the screen has one and only one meaning. For example, on an iPhone, swiping your finger from left to right or right to left lets you move between pages of apps. Ubuntu does things differently and combines several meanings into a single gesture. The result of a gesture depends on how much of the gesture is completed.

For example, removing the unlock feature from a phone, using long-distance gestures eliminates the added effort. The result is not only fewer steps, but also a reduced amount of time to reach a specific view. For example, compared to the way iPhones and Android phones work, which require multiple steps to reach different views, the Ubuntu UI only takes one step to reach different views, thus reducing the additional effort when starting an application, changing from the welcome screen to any other view, and switching between applications; all of this can be accomplished with a single gesture. In one move, an application can be found and launched from any view on the phone. Once the desired application is found using the long-distance gesture, simply releasing your finger on the application icon will initiate the launch of that application. This is different from other phones, as iPhones, Windows 8, and Android, for example, require a single tap on an application icon to launch it; there are no other gestures for launching applications on these phones.

With long-range gestures, physical buttons on the phone are no longer necessary and can therefore be removed. Using the iPhone as an example, which only has one button to take the user back to the home screen or to swap between apps, this is no longer necessary with Ubuntu as all of these things can be done with a simple swipe of a finger.

There are three different forms of long-distance gestures in Ubuntu:

- Progressive non-selective gestures - trigger different consequences or outcomes as the gesture progresses; the specific area on the screen where the user terminates the gesture is irrelevant.

- Progressive selective gestures - trigger different consequences or results as the gesture progresses; the specific area on the screen where the user terminates the gesture is crucial, such as if the user travels to and selects a specific area, a specific result is triggered.

-Selective gestures

This characterization can be rephrased in the form of a problem and its solution. The problem statement is: there are more interesting "actions" than simple and memorable gestures, so how can we design effective touch-centric interfaces? The solution can be generalized to a smartphone (or other personal computing device), where gesture input depends on the scope (e.g., distance, length, approach, path, starting point, end point, or destination) of a single gesture, with different (but usually related) consequences triggered depending on that scope.

Long-distance gestures can be accompanied by a reduction in the size of the traversed page—for example, as you swipe from the left to the right edge, the size of the app page minimizes as the swipe continues.

B.2 Main long-distance gesture concepts

In this section we will summarize the main "long-distance gesture" concepts. These concepts are as follows:

A smartphone, tablet, or other computing device comprising:

(a) a touch-sensitive display;

(b) one or more processors;

(c) computer memory;

(d) one or more computer programs stored in a computer memory, configured to be executed by the one or more processors and comprising instructions for detecting a single gesture input and, in response to detecting the single gesture input, causing two or more different results or functions to be triggered or invoked, depending on the scope of the detected gesture input.

Other optional implementation features include the following, each of which can be combined with any other feature:

- The detected range is a function of one or more of the following gesture input parameters: distance, length, direction, path, end point, or destination.

-The gesture input is a swipe from the edge.

- A swipe from an edge results in a first result, and if the user continues to swipe in from that edge, it results in a different result.

- Two or more different outcomes or functions are related or have related consequences.

- If the gesture input continues to a predefined area on the screen to select that area, causing a specific result.

- A sweep from one edge leads to different results than a sweep from a different edge.

- A brief swipe from the left edge is detected, causing the device to display or reveal the app launcher, which is a collection of app icons.

- Detecting that the brief swipe continues further to the right causes the device to display the home screen including icons for all running applications.

- A brief swipe away from a system information icon corresponding to a system information function causes a pane, window, or area related to the system information function to be displayed, and continuing the swipe expands the pane, window, or area to include user-selectable parameters corresponding to the function. The system information icon may be located on the right side of the top edge.

- When an app is open, a brief swipe from the edge causes functions related to the app to be displayed, and continuing the swipe to select a defined target area causes the search function to be invoked. The brief swipe can be an upward swipe from the bottom edge.

- The device is a tablet device, and wherein a brief swipe from an edge causes display of an area (a "side stage") that can display any application, utility, or function independently of content displayed on the rest of the screen, and wherein continuing the swipe causes display of a previously opened application. The brief swipe may be from a right edge.

- In response to detecting that the user pauses on an application's icon in the application launcher, the device displays a "quicklist" for the application, which is one or more user interface objects associated with features, functions, or controls of the application.

- Lifting off the touch screen is a confirmation gesture.

- Single gesture input involves the user's finger continuously contacting the touch screen and moving across the screen in one or more directions.

Other aspects are:

A method comprising the following steps performed at a computing device with a touch-sensitive display:

(a) detecting a single gesture input, and in response to detecting the single gesture input;

(b) causing two or more different results or functions to be triggered or invoked, depending on the extent of the detected gesture input.

A non-transitory computer-readable storage medium storing one or more programs comprising instructions that, when executed by a computing device with a touch-sensitive display, causes the device to:

(a) detecting a single gesture input, and in response to detecting the single gesture input;

(b) causing two or more different results or functions to be triggered or invoked, depending on the extent of the detected gesture input.

C.1 Welcome Screen

This welcome screen is a pictorial data visualization or infographic that graphically displays the user's personal data; in one implementation, it is a circular arrangement of a large number of dots, for example, 30 small white dots, each dot representing an activity or state (user-selectable or customizable), where the activity associated with a particular dot is represented by the size of the semi-transparent circle adjacent to that dot. The overall impression is organic, not unlike a flower.

Each point could relate to, for example, tweets received, emails received, the number of miles walked today, the number of minutes remaining on a call, the number of hours remaining on a break, and so on. Data visualization can work in a variety of ways. For example, the system could cycle through different points every few seconds, labeling the meaning of the current point with a brief text description inside the circle, and subtly changing the color palette from one point to the next. Or each point could represent a specific day of the month, and the screen could cycle through different variables corresponding to today's point every few seconds or longer.

Thus, the welcome screen evolves and updates itself regularly, its appearance being completely personal and unique to a particular user - notably because the size of each overlapping semi-transparent circle associated with a particular point is specific to a particular user at a particular time. But this welcome screen is designed not to overwhelm the user with information (as might be the case with other operating systems such as Windows Phone), notably because the core information content (specifically the size of the circle associated with any given point) is graphical rather than textual, and can therefore be understood very quickly, unlike tile-based systems such as Windows Phone and Windows 8, which primarily convey information through text rather than graphics. With the simple interface that is maintained, the Ubuntu Phone welcome screen still manages to display much more information than, for example, the iPhone's welcome screen.

Thus, the Ubuntu Welcome screen manages to capture a lot of data visually without using a lot of text, making the information intake easier to understand and thus allowing the user to get an overview of interesting and important data faster with minimal cognitive load. This can be compared to the iPhone screen, where various random notifications are placed in a list one after another, making the process of understanding what important information the screen actually contains tedious and slow.

The default Ubuntu welcome screen leverages the power of visual cognition. The eye quickly learns where to find specific information on the screen, and which graphic circles represent what data. This way, for a quick overview, the phone user saves time by not having to read at all, simply identifying the different-sized circles on the screen.

This specific feature can be rephrased in the form of a problem and its solution. The problem statement is: How can a user be provided with quickly understandable key performance or status information corresponding to multiple parameters of their phone without requiring the user to touch-interact with their phone in any way?

The solution can be generalized as a smartphone, tablet, or other computing device that displays a welcome screen with a (potentially dynamic) infographic or (potentially dynamic) data visualization representing each of several variables related to the device or one or more of its users. The variables can be a composite of data from multiple sources, applications, online services, and the device.

One implementation is a smartphone, tablet, or other personal computing device that displays a welcome screen with a dynamic data visualization or info graphic that represents each of several variables related to the state of the device (such as tweets received, emails received, distance walked today; minutes remaining for talk time; hours remaining for rest, or anything else that is of interest to the user) by assigning to each variable (a) a location in a geometric or other pattern or patterns and (b) a shape adjacent to the location of the variable in the pattern(s), the size of the shape changing with the value of the variable.

In one implementation, the geometric pattern is a circle, and each shape is a (typically shaded) portion of a circle, such as the head or petals of a flower. Thus, if a user receives 10 tweets but 20 emails, the portion of the shaded circle associated with tweets will be smaller than the portion of the shaded circle associated with emails. And as tweets are received, the size of the associated shaded circle will increase. Facebook comments can be another variable; a large number of Facebook comments will be rewarded by a rapid expansion of the associated shaded circle. Many other data visualization variations are possible.

Likewise, users will quickly become familiar with the location of different variables within a geometric pattern (e.g., a circle)—for example, the remaining talk time variable may be a point at the very top of the circle, so users will quickly learn to glance at the size associated with that variable to get a sense of the amount of talk time remaining. In Apple iOS, for example, this information is not accessible until after many interaction steps (unlocking the phone, finding and selecting the "Settings" icon, finding and selecting the "General" option, finding and selecting the "Usage" option).

Ubuntu is designed as a multi-user platform, and Ubuntu for phones and tablets demonstrates this by allowing users to open a welcome screen. On tablets, users are displayed as a list along with an anonymous guest account. On phones, the list of users can be displayed on demand (e.g., using the bottom edge of the welcome screen). In this way, a single device can support multiple users and multiple user accounts for different purposes (such as work and home).

The first impression of a new interface can be crucial for the acceptance of a new interface by a broad user base; the welcome screen provides a lively, intriguing, and attractive interface that users want to explore further; it is the first step that draws users into the device. It allows users to understand the status of key variables associated with the phone with a quick glance.

C.2 Main Welcome Screen Concepts

In this section we will outline the main Welcome screen concepts. These concepts are as follows:

A smartphone, tablet, or other computing device comprising:

(a) a touch-sensitive display;

(b) one or more processors;

(c) computer memory;

(d) one or more computer programs stored in a computer memory, configured to be executed by the one or more processors and comprising instructions for displaying a welcome screen having a pictorial information graphic or data visualization that graphically represents each of a number of variables related to the device or one or more users thereof.

Other optional implementation features include the following, each of which can be combined with any other feature:

- The infographic or data visualization is dynamic.

- The information graphic or data visualization is dynamically updated in real time

- The variables are a composite of data from multiple sources, applications, online services and devices.

- The data visualization is personalized and unique to the user of the device.

-The data visualization is specific to a specific time period, such as each day of a month.

- The data visualization loops or can be looped to cover different types of variables.

- The variables include one or more of the following: tweets received, emails received, number of distances walked today; minutes remaining for call time; hours remaining for break time, or anything else of interest to the user.

-Users can define different activities and states to include in the welcome screen.

- The device graphically represents each of a number of variables by assigning to each variable a position, shape, space or volume in a geometric or other pattern or patterns.

- The device graphically represents each of several variables by assigning to each variable a shape, the size of which varies with the value of the variable.

The device graphically represents each of several variables by assigning to each variable a color that changes hue depending on the value of the variable.

- The geometric pattern is a circle, and each shape is a portion of a shaded circle.

- The data visualization is personalized and unique to a specific user at a specific time.

- The data visualization is personalized and unique to a specific user at a specific time and place.

- The Welcome screen is the default screen that is displayed after the device has been idle for a predefined period of time or is first turned on, and is not the Home screen.

-The information graphic or data visualization graphically represents each of a number of variables related to the device (such as tweets received, emails received, the number of distance walked today; the remaining minutes of talk time; the remaining hours of rest, or anything else that is of interest to the user) by assigning to each variable (a) a position in a geometric or other pattern or patterns and (b) a shape adjacent to the position of the variable in the pattern(s), the size of which varies with the value of the variable.

- A swipe in from an edge unlocks the phone or allows access to the phone, and additionally causes the device to function (e.g., display the screen or portion of the screen, or invoke a function) in a manner that depends on the specific edge (e.g., left, right, top, or bottom) from which the swipe originated.

-From the welcome screen, by swiping from the right edge the user can unlock to the previously used state, that is, the state before the device was locked, and then, depending on the user's settings, the device will prompt for a personal identification code (PINCODE) or passphrase before allowing the user to enter.

- From the Welcome screen, the user can swipe from the left edge to reveal the launcher, showing favorite and running apps that can be launched directly, and may or may not be prompted for a PIN or passphrase for use depending on the previously launched app.

- Launching the camera app doesn't require entering a PIN or other authentication, while email clients do.

-From the Welcome screen, users can swipe from the right edge to bring up the Home page.

-When the Welcome screen is displayed, users can swipe down from the top to reveal system indicators.

Other aspects are:

A method comprising the following steps performed at a computing device with a touch-sensitive display:

(a) Displaying a welcome screen having a pictorial information graphic or data visualization that graphically represents each of a number of variables related to the device or one or more of its users.

A non-transitory computer-readable storage medium storing one or more programs comprising instructions that, when executed by a computing device with a touch-sensitive display, cause the device to:

A welcome screen is displayed having a pictorial information graphic or data visualization that graphically represents each of several variables related to the device or one or more users thereof.

D.1 Unlocking Mechanism

Traditional phone interfaces require a specific lock screen—specifically, a screen that is displayed when the phone is not in use; the lock screen displays some type of lock that can be unlocked by a deliberate gesture by the user (in Apple iOS, the lock screen includes a small white "slide to unlock" tile that the user slides along a channel to unlock the device). Thus, the lock screen prevents unintentional use of the phone and is considered a necessary requirement.

But Ubuntu doesn't need a dedicated unlock screen, because the screen edge in Ubuntu can have a specific purpose: a swipe from the edge not only unlocks the phone, but also initiates another useful process that enables quick and efficient use of the device. So for Ubuntu, after the device returns to a locked state (e.g., displaying the welcome screen), there is no such dedicated lock screen - unintentional activation is unlikely to occur because it would require a swipe from the edge to unlock the device. Alternatively, the device can be unlocked simply by swiping from the edge.

From the welcome screen, the user can "unlock" to the previously used state (i.e., before the device was locked) by swiping from the right edge - depending on user settings, this will prompt for a PIN or passphrase before allowing the user in. There's also the possibility of integrating biometric identification (such as voice or fingerprint - if the fingerprint sensor is integrated into the front of the device, the user can swipe their finger across the fingerprint sensor and continue swiping in from the left edge to open the launcher, all in a single, fluid motion).

Additionally, from the welcome screen, the user can swipe from the left to reveal a launcher (with favorite and running apps) that can be launched directly and (depending on the previously launched app) may or may not be prompted for a PIN or passphrase for use. For example, launching a camera app may not require a PIN, while an email client would.

Additionally, when in the welcome screen, the user can swipe from the top to reveal system indicators, some of which may again prompt for a personal identification code or passphrase in order to gain full access thereto.

Thus, from the Welcome screen, a brief swipe from the left edge brings up all your favorite apps as a list of app icons in the launcher; you can slide your finger up and down along the list of app icons in the launcher bar and release on the app you want to launch, thereby unlocking the device and launching the app in question. The launcher bar can also include apps that are not favorites but are currently running, giving quick access to them (once an app is no longer running, it disappears from the launcher bar). In addition, if there is enough space to show all favorite and running apps, non-favorite apps are displayed as a stack of icons that the user can drag out to expand, much like an accordion.

Therefore, the most important unlocking action is a brief swipe in from the left to drag open the launcher bar and slide up or down the launcher bar to the desired app, and then release to confirm. This is just a single, fluid action, where the finger is held on the screen until released on the icon of the app you want to launch (or the home screen icon).

Swiping from other edges can also unlock: a brief swipe in from the right edge unlocks the phone and brings up the last app the user was working in (a certain level of security will then be applied - for example, if the last app was designated as a low-security app, such as a music player, that app can be opened immediately; for more sensitive apps, such as a contacts list, some kind of user authentication will be required to proceed to that app).

A brief downward swipe through the top right edge provides access to system settings and status. For example, a user can swipe through the top edge to invoke system settings, then swipe left or right to select the "Network" icon from among the various system information icons on the top right edge, and then swipe down to reveal the "Network" parameter information that the user can then access, all without lifting their finger - i.e., a single, fluid gesture.

Other system information icons in the top right include: Messages (which allows the user to select a message by tapping on it, and then directly return a call or reply to the message by tapping the "Call Back" or "Message" button); Sound (which allows the user to mute or change the volume); Battery (which allows the user to see the remaining battery and change the screen brightness).

The design choices about how to unlock the phone go hand in hand with the elegant, simple design of the Ubuntu Welcome screen and the entire Ubuntu experience. The approach taken here is that less is actually more.

As mentioned above, the difference compared to other smart phones is that there is no object on the welcome screen that can be moved and in a sense requires interaction with it. For example, compared to the iPhone where the user must swipe the interaction bar from left to right to unlock the phone, for the new Ubuntu, this visualization of the unlocking mechanism is no longer necessary. Any kind of unlocking appearance object in the UI is removed, and Ubuntu uses different edges of the screen to reach other views on the phone instead. In this way, the Ubuntu UI fully utilizes the screen space and removes the intermediate step of having to perform a specific task in order to unlock the phone.

Using the edges to reach other views of the phone might sound similar to how Android (from 2010) unlocks its phones by dragging your finger down on the screen, which in turn opens new windows, or to how Windows 8 unlocks phones by briefly flicking in an upward direction. A key difference with Ubuntu Phone is that you can use any edge of the phone, not just dragging your finger down or briefly flicking in one direction, and each swipe from one edge unlocks to a different state (e.g., a swipe from the left edge unlocks to favorite apps via the launcher; a swipe from the right edge unlocks to the previous app).

Therefore, there is no need for a separate lock screen with an associated unlocking movement, because the unlocking movement in Ubuntu phones is a simple swipe in from the edge. We can compare this process more formally to Apple iOS 6 and earlier versions, where the user first unlocks by (i) touching down and then (ii) dragging the unlocking white rectangle along its path and then (iii) releasing, with the phone optionally displaying a keypad into which the user must (iv) enter their numeric passcode grid, at which point (v) the device displays a grid of available apps, and the user then (vi) taps the desired app icon to open the app (or if the app icon is not immediately visible, the user must (vii) swipe through successive pages of icons until they find the desired icon). Thus, in Apple iOS, the user typically goes from a locked phone to a phone with the desired app open and running, typically six or seven separate steps. But with Ubuntu, the phone is unlocked, opened with a single, smooth movement of a finger across the phone surface, with an optional keypad for authentication. This is much faster and can also be done one-handed by swiping and sliding with your thumb or finger.

Ubuntu therefore uniquely offers its full launcher from the default screen (e.g., the screen that's displayed when there's been no activity for a defined time—specifically, the phone's welcome screen). This means you can unlock and (with a single gesture) go directly to any of your favorite apps. On previous phones, you needed to unlock and then pick a favorite app to launch. Some lock screens have dedicated shortcuts to take you to the camera or similar vendor-selected app collections, but Ubuntu is the first to offer all of a user's favorite apps directly from the default (e.g., welcome) screen.

The fact that Ubuntu has a launcher bar corresponding to the user's favorite applications can also reduce overhead ("overhead" is unnecessary work required to achieve the UX goal - it is a burden or "overhead" on top of the idealized perfect frictionless user interaction with optimal cognitive load. After owning a phone for a while, the user may end up with several pages of different applications, making it easy to forget which application can be found on which page. By having a favorite application section on the phone, the user can quickly and easily open the application they want.

Ubuntu also provides a "Quick List" menu for each item on the launcher. These items are also available directly from the lock screen, making it possible to go directly to the specific screen for any favorite item in the launcher without having to first unlock the device, then select the item, and then select the screen or action corresponding to that item. From a usability perspective, this once again demonstrates the high efficiency, simplicity, and reduced overhead on the phone.

Furthermore, on other devices, the same view is always displayed after unlocking the phone; it's either the main view or the view that was active when the screen was turned off. This is not the case on Ubuntu. Since the extra procedure of having to unlock the phone is removed, different views can be accessed in the same way as if the phone were not locked at all. As a result, the Ubuntu phone allows users to get to their desired view immediately, eliminating multiple unnecessary intermediate steps, thus eliminating additional work by swiping in from the appropriate edge (and by swiping the appropriate length or range if using the "long reach gesture").

The unlock feature can be reformulated in the form of a problem and its solution. The problem statement is: How can a user get to their favorite apps on their phone from the phone's lock screen as quickly as possible? The solution can be generalized to a smartphone (or other personal computing device) where a swipe in from the edge unlocks the phone and also causes the device to display content (e.g., an app launcher bar, a home screen, one or more apps) that the user can then immediately interact with.

In one implementation, a brief swipe in from the left edge will open a launcher bar (featuring the user's favorite apps), and the user can then slide their finger up or down along the launcher bar to the desired app and then confirm (e.g., by releasing on it) the desired app to launch it. This approach works well for apps that don't require any user authentication, such as browsers. For other apps, such as Facebook, the user will have to authenticate in the normal way before entering the app.

One of the interactive features we described in this section is how a brief swipe in from the left edge brings up the launcher bar, but if the user continues to swipe in from the left edge, the launcher disappears and the apps section of the home screen is displayed instead. This is an example of what we call a "long reach gesture."

D.2 Main unlocking mechanism concepts

In this section we will summarize the main unlocking mechanism concepts. These concepts are as follows:

A smartphone, tablet, or other computing device comprising:

(a) a touch-sensitive display;

(b) one or more processors;

(c) computer memory;

(d) one or more computer programs stored in computer memory, configured to be executed by the one or more processors and comprising instructions for detecting a swipe in from one or more edges and subsequently unlocking the screen or making the device accessible in response to the detected swipe.

Other optional implementation features include the following, each of which can be combined with any other feature:

-The screen from which you unlock your device is the Welcome screen.

-From the welcome screen, users can unlock to the previously used state, that is, the state before the device was locked, by swiping from one edge (such as the right edge).

-Depending on user settings, a swipe from Edge will prompt for a PIN or passphrase before allowing the user in.

-From the welcome screen, the user can swipe from an edge (such as the left edge) to reveal a launcher that displays favorite and running applications that can be launched directly and may or may not prompt for a PIN or passphrase depending on the previously launched application.

- Launching the camera app doesn't require entering a PIN or other authentication, while email clients do.

-When the welcome screen is displayed, the user can swipe from the edge, such as swiping down from the top edge, to reveal system indicators.

- The device can be accessed or unlocked from at least the Welcome screen using a swipe in from one or more edges, and the swipe can be continued in a single gesture to directly open any application in the collection of favorite applications.

- There is no dedicated specific unlocking action that is only used to unlock or access the device, instead each and every unlocking or accessing action is not only used to unlock or access the device, but also serves a second purpose.

-The second purpose can be one of the following: opening the application launcher; opening the home screen or the application page of the home screen; opening the search bar; displaying information related to one or more system information functions; displaying application-specific functions or icons; displaying the previously opened screen.

- There is no general access control interaction to unlock or access the device. Instead, access control interaction is only required for those features, settings, or apps that explicitly require it.

- There are multiple features, settings, or apps that do not require access control interaction and can be opened or invoked using just a swipe-in from an edge gesture, regardless of the state of the device.

-The UI graphic displayed after unlocking is one of the following: application launcher bar, home screen, one or more applications, system icons, and application-related functions.

-A brief swipe in from the left edge will open a launcher bar that features at least the user's favorite apps, and the user can then slide their finger up or down along the launcher bar to the desired app and then confirm (e.g., by releasing on it) the desired app to launch it.

-The launcher bar includes the user's favorite apps and also all running apps.

- A swipe from any two or more different edges will unlock from any screen.

-Swiping from any three or more different edges will unlock from any screen.

-Swiping from any of the four edges will unlock from any screen.

- Open apps directly from the locked screen with a single gesture by touching and releasing the screen.

- Multiple public (eg, not password-locked) apps, such as a browser, camera, etc., can be opened with a single gesture (eg, touch and release movement).

-A quick list item corresponding to one or more items can be opened through a single touch and release movement. The quick list corresponding to an application is a specific feature in an application that enables the user to quickly access features, functions or controls corresponding to the application.

-Depending on which edge you unlock from, the device unlocks to different states.

- Unlock is for a system message including a message list, allowing the user to select a message by tapping on it, and then directly return the call or reply to the message by tapping the "Call Back" or "Message" button.

- Unlock is for system information including sound parameters, allowing users to mute or change the volume.

-Unlock is for system information including battery parameters, allowing users to see the remaining power and change the screen brightness.

- A swipe in from an edge unlocks the device or allows access to the device and, in addition, causes the device to display graphical user interface objects depending on the particular edge (eg, left, right, top, or bottom) from which the swipe originated.

Other aspects are:

A method includes the following steps, performed at a computing device with a touch-sensitive display: detecting a swipe in from one or more edges, and then unlocking or enabling access to the device from the screen in response to the detected swipe.

A non-transitory computer-readable storage medium storing one or more programs comprising instructions that, when executed by a computing device with a touch-sensitive display, causes the device to detect a swipe in from one or more edges and subsequently unlock the screen or enable access to the device in response to the detected swipe.

E.1 Ubuntu Search Experience

Searching for content and apps, both online and on your device, is central to the Ubuntu UX. Searching happens through the Home screen, which is the front end to a series of search and data presentation tools called "Scopes."

The Ubuntu search experience can be accessed from any page of the home screen, which accesses the search for the domain that is driving that page. The main page of the home screen defaults to a universal search system, which will implement intelligent search within all available search domains. Therefore, the main page on the home screen will search for anything on the device or online (for example, if appropriate, in the online merchant store; if the phone is supplied by the network operator, the search can also appear from the operator's own store or associated merchants). This is a global search, covering, for example, applications, content, locally stored files, remotely stored files on the network, and products and services from third parties.

Ubuntu itself will figure out what you're looking for and where best to search. So, if you type "Tolkien" into the search bar, a remote server to which your phone, tablet, etc. is connected will return videos for "The Hobbit" and "The Lord of the Rings" for purchase. The home screen could show the local theater showing those films; it could also display books written by Tolkien, and so on. For each of these items, the user can tap to select and be taken to the online merchant's website to complete the purchase, or the system can be configured to initiate the purchase by tapping a "Buy" button on the screen. Another variation shows the user books, etc. from different online merchants, allowing the user to choose which merchant to buy from.

All of these results are now in one place, intelligently and automatically categorized, and all found by the Ubuntu operating system rather than by applications like browsers. This is a feature that, for example, operators can be 100% confident will be offered to their customers because it is an integrated part of the operating system, unlike even very widely used search engines like Google running in widely used browsers like Safari - which do not in any case intelligently and automatically categorize search hits into categories relevant for mobile searches (Ubuntu also uses location-based filters so that, for example, you are only presented with cinemas showing Tolkien films that are reasonably close to your current location - any other shopping, dining, culture and entertainment results can also be location-filtered by Ubuntu).

Ubuntu's search bar or tool is a convenient tool from a user's perspective because of its easy access on the home screen. The term "bar" will be used to refer to any screen or tool related to search, including screens or tools that simply receive user input defining a search and/or display or cause results to appear. It is not limited to screens or tools that have a bar-shaped data entry field.

Ubuntu's UI makes the search tool more efficient because no matter what application is currently focused, getting to it is a single step—swiping down from the left portion of the top edge always invokes the search experience. Long-reach gestures enable the selection of a specific search scope (the default is the primary scope, which searches everything).

The user types in what they want to search for, and Ubuntu takes care of the rest. This can be compared to iOS, for example, where getting to the search tool can be multiple steps, like pressing the home button twice, or flicking through several app pages before arriving at the page with the actual tool. After typing in what they want to search for, on the iPhone, if the user wants to search for something online, they must take additional steps.

The Ubuntu UX is clean and simple, but underneath it's essentially a PC, so the full range of Ubuntu services is available. For example, every application has a built-in cloud (using Ubuntu One), so all your photos and settings are automatically backed up, all your music is backed up and searchable and available online from any Ubuntu device you log into; you can share content via the web with just one click.

E.2 Ubuntu HUD Search Bar

The Ubuntu HUD is a unique and innovative mechanism for supporting complex controls on touch applications in Ubuntu. It is a largely search-driven experience that allows users to access controls for applications that cannot be accommodated in a simple touch toolbar or other chrome-based ("on-screen buttons") interfaces. The mechanism is called a HUD ("Heads-Up Display") because it displays information to the user without forcing the user to divert from their primary focus of interest, similar to a traditional HUD.

The HUD can be invoked via a long gesture from the bottom edge. This is consistent with Ubuntu's use of the bottom edge for application controls, as the HUD is an extension of the visible ("chrome") application controls that are also accessed via the bottom edge.

The HUD presents recent actions in the form of a history of recent or most popular actions, frequent actions in the form of a toolbar, and some standard actions such as "Exit", "Undo", and "App Settings". The HUD also provides a search capability that allows intent-driven exploration of application capabilities. Search can be performed through text input or voice input. Thus, the HUD allows you to "say what you intend to do" and have the application suggest potential actions for you.

In Ubuntu, it is possible to equip user interactions with a HUD that reports searches, results, and user actions, and then remotely analyze the data to prioritize the most relevant results. Ubuntu can also automatically sort search results based on a specific user's previous search history.

In the HUD, actions can have parameters that determine their effect, so that the user does not need to call a dialog box (for example, from a drop-down menu) and delegate the task to that separate dialog box. Pickers, sliders, inputs and other widgets can be integrated into the HUD to provide input for these parameters. This approach can also support voice. The HUD thus combines the capabilities of both traditional menus and traditional dialog boxes into a touch-friendly mechanism that is unique and distinctive to Ubuntu. When searching in the HUD, it is useful to be able to search for actions across various running applications or system services. The HUD "app stack" across the top of the HUD shows running applications that have potential matches for the current HUD search and provides the ability to see which applications have matching actions and jump directly to them. Users can search among multiple applications as before, but now can explicitly select which matching applications will be displayed.

This characterization can be rephrased in the form of a problem and its solution. The problem statement is: How can we provide an easily accessible (and often unified) search that provides results in a way that does not overwhelm the user?

The solution can be generalized as:

A smartphone, tablet, or other computing device in which the device operating system enables a user to reach a search bar from a screen other than the home screen with a single touch gesture on the touch screen.

Ubuntu also improves search relevance with a synonym/fuzzy matching engine that matches user input with additional keywords/dictionaries/vocabularies/lexicons. Ubuntu can also search within system services ("screenshots") and running application actions.

The design of using Ubuntu on the bottom edge for HUD invocation means that users can access the HUD with the same gesture for applications in both the main and side platforms of the tablet device.

E.3 Main Ubuntu Search and HUD Concepts

In this section we will summarize the main Ubuntu search and HUD concepts. These concepts are as follows:

A smartphone, tablet, or other computing device comprising:

(a) a touch-sensitive display;

(b) one or more processors;

(c) computer memory;

(d) an operating system computer program stored in computer memory, configured to be executed by the one or more processors and comprising instructions for detecting a swipe gesture and displaying a search bar in response to the detected swipe gesture.

Other optional implementation features include the following, each of which can be combined with any other feature:

-Reach the search bar by starting with a swipe from the edge.

- The swipe is a long swipe from the left edge, bringing up the Home screen, which features a search bar at the top of the Home screen.

- The search bar allows for universal searches across multiple data sources.

- These data sources are internal and external to the device.

-Reach the search bar with a brief downward swipe from the top edge.

- There is a specific search bar that is limited to making features and actions available within the open application appear.

- The search bar is reached by swiping up from the bottom edge to a defined area, and an icon or symbol is displayed at the area once the area is reached by the swipe gesture.

- In response to detecting selection of the defined area, for example by the user releasing contact upon reaching the area, the device displays the search bar together with items for controlling features or functions of the open applications.

-Users can search for features or actions in the current application through text input or voice control.

- The search results will also list which other apps include the feature or action being searched for.

- Other applications that include the searched feature or action are displayed as a collection of icons.

- Any application that appears in a search can be jumped to (eg, launched or accessed directly from the search results) by the user selecting the application's icon.

-The collection of apps that appear in search (+ indicators and other sources of action) is now visible as the "App Stack", which is a collection of app icons displayed across the top of the search bar, allowing users to search across multiple apps as before, but now can explicitly choose which matching apps will be displayed from.

- Improved unified search relevance with a synonym/fuzzy matching engine that matches user input with attached keywords/dictionaries/vocabularies/lexicons.

- User interactions with search results may be instrumented or analyzed and may be analyzed remotely, enabling automatic re-ranking of search results to prioritize the most relevant results based on historically determined usage analytics.

-Search results can be automatically sorted based on a specific user's previous search history.

- The ability to see which applications, etc. have a match for any search and jump directly to the application.

-Home screen includes a unified search bar.

-Swipe gestures can be made and detected when the device is not displaying the Home screen.

-The search bar can be reached by long swiping up from the bottom edge.

- The search bar can be reached by swiping from the edge (such as a long swipe from the left edge), which brings up the home screen's app page, which features an app search at the top of the page.

- The search bar can be reached by swiping from the search bar icon (such as the magnifying glass).

-An icon corresponding to the search bar is located at the top edge.

-The search function corresponding to general search operates on multiple data sources both inside and outside the device, and the results are automatically categorized into multiple categories, such as people, places, videos, books, cinemas, etc., and ranked by relevance using factors such as location, previous search history, social graph, etc.

-Multiple menu functions or features can be searched in a search bar and appear in the user interface area, panel, or window that includes the search bar, and all variable parameters corresponding to the functions or features can be selected or controlled from within the user interface area, panel, or window, so that the user does not need to call a separate dialog box (for example, from a drop-down menu) and delegate the task to the separate dialog box.

Other aspects are:

A method includes the following steps, performed at a computing device with a touch-sensitive display: detecting a swipe gesture, and displaying a search bar in response to the detected swipe gesture.

A smartphone, tablet device, or other computing device in which any one or more menu functions may appear in a search bar, and all variable parameters corresponding to the function may be selected or controlled from within the search bar, thereby eliminating the need for the user to invoke a dialog box (e.g., from a drop-down menu) and delegate the task to the separate dialog box.

A smartphone, tablet, or other computing device in which you can jump directly to (e.g., launch or access directly from search results) any application that appears in a search.

A smartphone, tablet, or other computing device wherein unified search relevance is improved using a synonym/fuzzy matching engine that matches user input with additional keywords/dictionary/vocabulary/lexicon.

F.1 Home Screen in Ubuntu

The launcher (arranged as a bar or column) typically includes 10 icons. At the bottom are icons corresponding to the Home screen. If the user selects this icon (e.g., by dragging out the launcher with a short swipe from the left edge, then swiping down to the bottom "Home" icon, then releasing on the Home icon), the Home screen is displayed. The Apps page can be reached with a long swipe from the left edge; this takes you to the Apps screen, which includes a complete list of all apps on the phone; a right flick across the top of this screen takes you to the Home screen.

The home screen displays icons for frequently used apps, calls you've recently made or received (or missed calls, so you can easily return a call by simply touching the item listing the missed call). The home screen is also fully customizable, so it can include, for example, "recently added music"; newly released movies; online radio; Wikipedia; online stores, and so on. In fact, users can choose from hundreds of different filters to populate their home screen. In a typical installation, the home screen is divided into several sections that the user reaches by flicking up or down on the screen; these sections are: frequently used apps; favorite people; recently contacted people; recent music; and popular online videos.

Likewise, intermediate steps and additional work are reduced. From the home screen, you can immediately interact with missed calls or other recent calls, for example, if you want to return a call. The user scrolls to the "Recent Calls" section by flicking the screen up or down. This section lists the names of the people whose calls were missed. The user touches the name, and a button labeled "Call" appears—if the user touches the button, a voice call is automatically placed to that person. This simple and efficient process can be contrasted with a more traditional approach, where you have to find a phone icon, tap it, then find a recent phone call tab, tap it, and then finally tap on the contact you want to call.

Since the home screen is very easy to reach by swiping from the left edge of the phone, there is no need for a dedicated hardware home button, unlike in iOS, or soft buttons as in Android. Furthermore, in iOS and Android, this dedicated home button is required whenever the user wants to switch apps - the user must first press the home button, which then brings up a grid of apps for the user to tap to select. But in Ubuntu, switching apps is done in a single fluid swipe gesture as described above, either from the left edge (see the previous section entitled "B. Swipe to unlock Ubuntu") or from the right edge (see the later section entitled "E. More edge behaviors in Ubuntu").

If you're on the Home screen and swipe sideways, you'll see the information you use most often, such as a page with the people you communicate with most often (each list can be accompanied by Twitter or Facebook feedback data); another page with the music you listen to most often; and another page with the apps you use most often. You can jump to any of these pages with a single gesture. A full swipe across the screen from the left edge opens the Apps page, which lists all installed apps and available apps.

In short, in a typical setup, swiping right from the home screen takes you to the home screen's Apps page; swiping further right takes you to Videos. Swiping left takes you to People (Contacts), and swiping further right takes you to Music.

On the Apps page, you'll find icons for all installed apps, as well as apps available for download—so finding apps is consistent across any Ubuntu device; specifically, there are sections for running apps, frequently used apps, installed apps, and apps available for download. Instead of having to navigate to a dedicated app store, all apps are gathered in one place. This makes navigating apps more logical for users, reducing mental effort and making the experience easy and effortless.

Ubuntu supports web, HTML5 and native applications. Web applications (Facebook, Gmail, Twitter, Spotify, etc.) can send notifications and use system features (for example, the Facebook application can use the messaging menu like a native application).

This home screen feature can be reframed in the form of a problem and its solution. The problem statement is: How do you get to the "home" screen without needing a "home" button that takes up valuable screen or housing real estate? The solution can be generalized to a smartphone (or other personal computing device) where swiping in from the edge brings up a home screen that, for example, lists all available apps. A physical "page" button is no longer needed.

In one implementation, the swipe is a long swipe in from the left edge; a confirmation gesture (e.g., release) after the Home screen appears causes the Home screen to be displayed after confirmation; if the user swipes back to the left without confirmation, the Home screen disappears. An additional variation is a short swipe in from the left edge, which brings up a launcher bar of favorite apps and selectable icons corresponding to the Home screen, which the user swipes to and selects by releasing on the Home screen icon. Unlike other soft Home screen icons, this is only called up when specifically needed, but calling it up and selecting it is a very fast, smooth, one-handed operation.

F.2 Main Home Screen Concepts

In this section we will summarize the main "Home Screen" concepts. These concepts are as follows:

1. A smartphone, tablet device, or other computing device comprising:

(a) a touch-sensitive display;

(b) one or more processors;

(c) computer memory;

(d) one or more computer programs stored in computer memory, configured to be executed by the one or more processors and comprising instructions for detecting a swipe-in from an edge of a display and, in response to the detected swipe, causing display of a home screen that lists running applications and user notifications, such as missed calls, regardless of which screen the device was displaying immediately before the swipe-in from the edge.

Other optional implementation features include the following, each of which can be combined with any other feature:

-Swiping in from the edge brings up the Home screen and only requires a single gesture to be detected.

-Swiping in from the edge brings up the Home screen and requires detecting a single gesture and confirming or selecting the action.

- The swipe is a long swipe in from the left edge.

- A short swipe from the left edge brings up the app launcher, and continuing that swipe into a long swipe results in displaying the home screen apps page.

- If a reversal of direction of the long swipe is detected, gradually dismissing the home screen as the swipe continues in its reversed direction back to the left edge.

- Home screen content is user customizable.

- The Home screen includes one or more of the following: Recently Added Music, New Released Movies, Online Store.

- The home screen includes a missed call notification and enables the user to return the call without reassigning the task to the phone app.

-The device does not have a dedicated physical "Home" button.

- The Home screen includes an Apps section that lists all installed apps and also has apps available for download.

-Swiping right from the Home screen takes you to the Home screen Apps page; swiping further right takes you to Videos.

-Swiping left will take you to People (Contacts) and further swiping right will take you to Music.

-A brief swipe from the left edge brings up the launcher bar, which includes icons for various apps plus icons for the home screen.

Other aspects are:

A method comprising the steps of detecting a swipe-in from an edge of the display and, in response to the detected swipe, causing display of a home screen that lists running applications and user notifications, such as missed calls, regardless of which screen the device was displaying immediately before the swipe-in from the edge.

G.1 Ubuntu "side platform"

Ubuntu uniquely places each phone application on the right edge of the touch screen (tablet device or touch PC). This provides it with a phone-shaped space that can be touched with the right hand and thumb, and its touch method is exactly the same as those applications that will be touched on the phone. The right-hand feel of the applications on the tablet device and the phone is exactly the same. Each indication is located on the right edge of both the tablet device and the phone screen. This phone-shaped space (which is called a "side stage") can display any running application (including any phone application), thereby allowing multitasking across two parts of the display - the main part runs an application, and the side stage runs different applications (such as phone applications) that are displayed as covering the first application (for example, applying a translucent effect to the side stage so that some aspects of the underlying application running in the main part are just visible), or using a split-screen format. In addition, since the side stage has the same format and ratio as the smartphone screen, there is no problem of stretching or reformatting the phone application to run correctly in the side stage.

As an additional feature, the right edge behavior of the phone screen is defined as "previous app" (i.e. a brief swipe from the right edge brings up the previous app the user was previously using), and on tablet devices it is defined as "reveal phone space" (i.e. a brief swipe from the right edge on a tablet device reveals the "side table"), or if the swipe continues further, displays the previous app on top of the phone space and replaces any content within it. This provides consistency across the right edge of phones and tablets - a swipe from the right edge on either device will reveal the previously used app.

This feature can be rephrased in the form of a problem and its solution. The problem statement is: How do you allow phone applications to run well as part of a tablet or touch PC environment? The solution can be generalized to a computing device, such as a tablet or PC, where, in response to a swipe in from the right edge, phone applications are displayed in a generally rectangular (e.g., phone-shaped) space on the right edge or side of the device (tablet or touch PC). This allows applications to be touched with the right hand and thumb in exactly the same way as those applications would be touched on a phone.

G.2 Main side platform concepts

In this section we will summarize the main "edge gesture" concepts. These concepts are as follows:

A computing device, such as a tablet device or a PC, comprising:

(a) a touch-sensitive display;

(b) one or more processors;

(c) computer memory;

(d) one or more computer programs stored in computer memory, configured to be executed by the one or more processors and comprising instructions for displaying one or more applications, utilities, or functions in a window, pane, or area on the right side of the device in response to detecting a swipe-in from the right edge.

Other optional implementation features include the following, each of which can be combined with any other feature:

- The window, pane or area is rectangular, such as the shape of a phone display.

- The window, pane or area (which is called a "side stage") can display running applications (including any phone applications), utilities or functions, allowing multitasking across two parts of the display.

-The main part runs one application and the side stage runs a different application or utility or function.

-The main portion runs any of the following: a video player; a web browser; a productivity application such as a word processor; a social networking application; a search engine; and the side portion runs any of the following: a text messaging application; a phone dialer function; an email application; an instant messenger; a system setup utility.

- The side stage has the same format and proportions as the smartphone screen, so there is no need to scale or reformat phone apps to run correctly within the side stage.

-The app is a phone app.

- The application running in the rectangular space covers (with a semi-transparent effect) the application running on the main part.

-The screen is divided into two parts, each part runs a different application.

-A swipe from the right edge reveals the window, pane, or area.

-Continue swiping from the right edge to display previous apps.

Other aspects are:

A method includes the following steps, performed at a computing device with a touch-sensitive display: in response to detecting a swipe-in from a right edge, displaying one or more applications, utilities, or functions in a window, pane, or area on the right side of the device.

In the previous chapters we focused on Ubuntu for phones and tablets, but a key attribute of Ubuntu is that it is a single operating system with an interface that scales smoothly from phone to tablet to desktop to TV.

H.1 Scalable and Responsive Ubuntu UI

The touch-enabled version of Ubuntu complements and completes the family of interfaces that operate on the unified single Ubuntu operating system. These interfaces are:

- Ubuntu TV - ideal for interacting at a distance with a simple (six-button, four-edge) remote control unit

Ubuntu Desktop - ideal for content creation and serious computing tasks using a keyboard and a pointer like a mouse or trackpad

Ubuntu Phone and Tablet devices - ideal for touch-based one-handed (phones) or two-handed (tablets) operation, with a unique mechanism to access the richness of desktop applications

A major benefit of this integrated design of user interface families is that it is possible to have a single OS that provides the appropriate interface for all of these form factors, which in turn enables the creation of integrated devices that provide alternate interfaces for different form factors when docked to appropriate accessories.

It also becomes possible to write a single application that runs on many different device types, or that transitions smoothly from one form factor to another when the device is docked to an accessory that provides the means to present the interface of the different form factors. For example, this enables a smartphone or tablet to be used as a PC when connected to a keyboard, mouse, and monitor; the smartphone or tablet then provides all the computing resources of a traditional PC, giving enterprise customers a single device that can provide all the functions of a phone and a PC (we call this the "super phone" category).

As a result, an Ubuntu superphone or tablet device can provide a rich and complete desktop environment; including access to Windows applications through integrated thin client capabilities.

Device consolidation significantly reduces the complexity and therefore the cost of management, testing, and support because a single version of the operating system can be used across many devices, and fewer devices are needed overall. OS consolidation avoids fragmentation—for example, in the Windows ecosystem, there are Windows PCs, Windows RT, and Windows Phones, all with varying degrees of compatibility and different management processes.

This integrated design means that Ubuntu running on all of these devices has the full capabilities of a PC OS (operating system). Therefore, Ubuntu Phone apps can take advantage of all the CPU cores and the phone's GPU (graphics processing unit)'s fully native OpenGL (Open Graphics Library) and OpenGL ES (OpenGL for embedded systems), just as if they were Ubuntu PC apps. In contrast, apps written for iOS and macOS operate under very different constraints, tools, and capabilities.

The Ubuntu approach exemplifies the benefits of consolidation—the same unified operating system for all device types, and a family of related interfaces optimized for each category but sharing many common features. The technical benefits of a unified, single OS for all device types are particularly attractive to application developers (who can write their apps once and have them run on phones, tablets, desktops, TVs, and any other computing device) and to large enterprise customers and network operators (who can manage Ubuntu phones with the exact same tools they already use for Ubuntu on servers, the cloud, and desktops).

Ubuntu's UI is arranged in a specific way so that it has distinct and unique benefits when translating across multiple form factors (phones, tablets, PCs, and TVs). The arrangement of items (indicators, menus, launchers, notifications, sidebars, and use of screen edges) in each of these form factors is optimally consistent across the board. This arrangement ensures that a device can be transformed from one form factor (e.g., a tablet) to another (e.g., a PC) with minimal disruption or change in item placement and behavior.

Considering the user experience, there is a clear advantage in that the Ubuntu UI is scalable across multiple devices without changing the look or controls, thus maintaining consistency across its products. After learning the interface on one device, users can continue working on another device without any effort, saving time on not having to learn another interface with its own controls; the result is a user-friendly and efficient overall experience.

Not only is the look and feel of the interface the same across devices, but the actual system is also the same. Being able to use your phone as a computer is smart and convenient, and makes it easy for users to take what they need with them wherever they are.

This feature can be rephrased in the form of a problem and its solution. The problem statement is: How do you design a UI that "scales" smoothly from a phone, to a tablet, to a desktop, and to a TV? By smooth scaling we mean that the presentation and placement of UI elements perturb or change as they scale from a phone, to a tablet, to a desktop, and to a TV. The solution can be generalized to a computing device where certain core interactive elements remain at the same screen location as they scale from a phone, to a tablet, to a laptop, to a desktop, to a TV, providing a consistent UI across all screen sizes and device types.

The core interactive elements may include:

- The launcher bar (which is placed vertically down the left edge and is revealed by a short swipe in from the left edge); so the app launcher is always on the left side on all form factors (phone, tablet, TV, and PC). The left edge is the long edge on phones, but the short edge on tablets and desktops; but whether using a phone or tablet, it is generally the most intuitive way to get started.

-HUD search bar (which is placed horizontally at the top and is revealed by selecting the home screen)

-System indicator icon in the upper right corner

- and keeping the right edge free of interactive elements (this is for example to accommodate a "side stage" area in tablet devices to run phone apps, see above - G.1 Ubuntu Side Stage).

This mechanism is consistent across all form factors (smartphones, tablets, desktops, TVs).

H.2 resolution independence in Ubuntu UI

Devices running Ubuntu can range in size from small phones to 50" TVs, and in resolution from low-resolution phones to high-resolution monitors or tablets.

These devices can also have very different resolutions and pixel densities, ranging from 320x240 to 3840x2160 and 100dpi to 400dpi and higher.

Ubuntu has a framework for handling UIs across these sizes and resolutions to maintain a unified experience. This is achieved by defining a resolution-independent unit that is large enough to also help with consistent alignment of UI units.

The display is divided into a grid both horizontally and vertically, and instead of pixels, UI element sizes are usually defined in terms of grid units ( units.dp(x) ).

Depending on pixel density and viewing distance, each device has its own unique ratio of actual pixels per grid cell defined in the system settings corresponding to that particular device.

For example, a 4.7" phone with a 1280 x 720 display may have 18 pixels/grid unit. A 10" 2560 x 1600 tablet device may have 20 pixels/grid unit. A 5" phone with a 1920 x 1080 display may have 27 pixels/grid unit. The exact px/GU setting will be defined individually for each specific device.

Depending on the size and orientation of the device, there are different numbers of grid cell rows and columns. For example, there are typically 40 columns on a phone in portrait mode, and 128 columns on a 2560 x 1600 10" tablet in landscape mode.

By avoiding defining sizes in terms of actual pixels, our UI units automatically scale between devices — and maintain a similar perceived size across devices.

Grid units also help align content consistently – it’s easy to define consistent margins and align UI element sizes and positions in a visually pleasing way.

Finally, grids are valuable when communicating between designers and developers. App designers can start with one of our templates, which already has a defined grid, and align all UI elements to that grid. When the design is finished in the design software, saving the design with the grid overlaid clearly indicates the size and positioning of each element within the grid cells, eliminating the need to write detailed instructions for the developer. This reduces the margin of error in UI implementation from the developer's perspective.

H.3 Automatic scaling of design assets

For our framework it is possible to use vector or bitmap assets. The nature of vector assets (SVG - Scalable Vector Graphics) is resolution independent and scales perfectly to any size and resolution.

For bitmap assets, it is possible to create only one set of raw bitmap assets at ultra-high resolution (30px/g) and save them as "filename@30.png". Our framework will automatically downscale these assets (using high-quality filtering algorithms with minimal visual data loss) to the correct pixel density for each device - so that developers only need to prepare one set of assets.

But if you want to prepare different levels of detail for different displays, it is also possible to save separate assets at different px/GU ratios (i.e. asset1@12.png, asset1@18.png, asset1@30.png). The framework will automatically pick the file that is closest to the px/GU settings of the current device and scale it if necessary to exactly match the px/GU.

H.4 Main Scalable UI Concepts

In this section we will summarize the main scalable UI concepts. These concepts are as follows:

A computing device comprising:

(a) displays;

(b) one or more processors;

(c) computer memory;

(d) one or more computer programs stored in computer memory, configured to be executed by the one or more processors and comprising instructions for generating UI elements designed to remain at the same screen location as the device size scales from smartphone to tablet, to laptop, to desktop, to television, thereby providing a consistent UI across all screen sizes and device types.

Other optional implementation features include the following, each of which can be combined with any other feature:

- The core interactive element comprises a launcher bar comprising application icons that can be selected to open or switch to associated applications.

- The launcher bar is placed vertically down along the left edge and is revealed by a short swipe in from the left edge.

-The core interactive elements include a search bar.

-The search bar is placed horizontally at the top of the screen.

-The search bar is revealed by selecting the Home screen.

- Show the home screen if a long (e.g., greater than a certain length) swipe from the left edge of the device is detected.

-The core interactive elements include system indicator icons.

-System indicator icons are located along the top edge of the screen.

- The core interactive elements include keeping the right edge free of interactive elements.

Other aspects are:

A method comprising the following steps, performed at a computing device with a touch-sensitive display, of generating UI elements designed to remain at the same screen location as the device size scales from a smartphone, to a tablet, to a laptop, to a desktop, to a television, thereby providing a consistent UI across all screen sizes and device types.

A smartphone, tablet, or other computing device comprising:

(a) a touch-sensitive display;

(b) one or more processors;

(c) computer memory;

(d) one or more computer programs stored in computer memory, configured to be executed by the one or more processors and comprising instructions for generating UI elements designed with logical grid cell sizes that are independent of screen resolution.

- Depending on pixel density and viewing distance, each device has its own unique ratio of actual pixels per logical grid cell defined in the system settings corresponding to that specific device.

The steps, techniques and processes described above can be implemented using standard, well-known programming techniques. Part of the novelty of the embodiments described above lies not in the particular programming techniques, but in the use of the described steps, techniques and processes to achieve the described results. The software programming code that forms part of the present invention is typically stored in a permanent, non-transitory storage device. The software programming code can be embodied on any of a variety of known media for data processing systems, such as a disk, hard drive or CD ROM. The code can be distributed on such media, or can be distributed from the memory or storage device of one computer system to other computer systems over some type of network for use by users of such other systems. The techniques and methods for embodied software program code on physical media and/or distributing software code over a network are well known and will not be described further here.

It should be understood that each unit shown in the figure or various combinations of units in the figure can be implemented by a general and/or special-purpose hardware-based system that implements the specified functions or steps, techniques and processes, or by various combinations of general and/or special-purpose hardware and computer instructions.

These program instructions can be provided to a processor to produce a machine, such that the instructions executed on the processor produce means for performing the functions specified in the diagram. The computer program instructions can be executed by a processor to cause the processor to perform a series of operational steps to produce a computer-implemented process, such that the instructions executed on the processor provide steps for performing the functions specified in the diagram. Accordingly, the drawings support combinations of means for performing the specified functions, combinations of steps for performing the specified functions, and program instruction means for performing the specified functions.

I. Detailed description of the accompanying drawings

A core user interface layout corresponding to an application on a screen of a computing device, such as a portable computing device, is provided. The computing device may be, for example, a tablet computer or a smart phone. A menu bar is provided at the top of the screen. Below the menu bar, a header is provided, which may include a page title. Below the header, a dividing line is provided. Below the dividing line, a content area is provided. When fully exposed, a toolbar is provided at the bottom of the screen. The core layout provides a visual identifier. An example is shown in FIG1 .

The core user interface layout can be responsive and scalable between devices, from phones to tablets to PCs to TVs. An example of a user interface suitable for use on a variety of devices with a wide range of screen sizes, such as TVs, laptops, tablets, and smartphones, is shown in FIG2 .

A special framework can be provided for a unified user interface experience across devices of different sizes and resolutions. Instead of using pixels for measurement, the display can be divided into a grid, and the dimensions of user interface (UI) elements can be collectively defined in terms of grid cells. Depending on the pixel density and viewing distance, each device has its own unique ratio of actual pixels per grid cell. Thus, resolution independence is provided in the user interface.

In the phone user interface example, bottom controls are provided. In one example, bottom controls for the browser (e.g., "Back," "Forward") are revealed by a brief upward swipe from the bottom edge of the screen. An exemplary user interface screen is shown in FIG3 . Additional examples (A) and (B) are shown in FIG4 , each corresponding to viewing a contact. Bottom controls (e.g., "Back," "Edit") are revealed by a brief upward swipe from the bottom edge of the screen. Bottom controls may also be provided in a picture view and may be revealed by a brief upward swipe from the bottom edge of the screen.

In the tablet device user interface example, bottom controls are provided. In one example, a brief upward swipe from the bottom edge of the screen reveals the bottom controls for the browser (e.g., "Back," "Forward"). In one example, a brief upward swipe from the bottom edge of the screen reveals the bottom controls for the gallery (e.g., "Back," "Forward"). An example is shown in FIG5 . In one example, a brief upward swipe from the bottom edge of the screen reveals the bottom controls for the picture view. In one example, a brief upward swipe from the bottom edge of the screen reveals the bottom controls for the conversation view in the side stage.

A searchable command interface is provided. This mechanism is called a HUD ("heads-up display") because it displays information to the user without forcing the user to stray from their primary focus of interest, similar to a traditional HUD. In one example, a HUD is provided on a smartphone. In one example, to reach the HUD for an application, the user begins by swiping upward from the bottom of the screen to reveal the bottom controls and HUD icons. Releasing a finger over the HUD icon reveals the HUD and its available features, for example, to search for controls or actions within the application. To hide the HUD again, the user can swipe downward from the top of the HUD. An example is shown in FIG6 . FIG6 shows a sequence of device screen configurations, from (A) to (E). In the example of FIG6 , the display in (A) includes an image. In (B), the bottom controls are displayed in response to an upward swipe from the bottom of the screen, and the HUD icons are displayed. Releasing a finger over the HUD icon results in the HUD interface in (C). In (D), the user begins hiding the HUD by swiping from the top of the screen, and the HUD interface moves downward in response to the user swiping. In (E), return to the screen of (A).

In one example, a HUD is provided on a tablet device. To reach the application's HUD, the user begins by swiping upward from the bottom of the screen to reveal the bottom controls and HUD icons. Releasing their finger over the HUD icon reveals the HUD and its available features, such as changing the color balance. To hide the HUD again, the user can swipe downward from the top of the HUD. An example is shown in FIG7 . FIG7 illustrates a sequence of device screen configurations, from (A) to (D). In the example of FIG7 , the display in (A) includes an image. In (B), the bottom controls are displayed in response to an upward swipe from the bottom of the screen, and the HUD icons are displayed. Releasing their finger over the HUD icon results in the HUD interface on the left side of the screen in (C). In (D), the user's swipe from the top of the screen hides the HUD (the (D) screen image is rotated 90 degrees compared to (A) to (C)), while the HUD interface moves downward in response to the user's swipe. In the final screen (not shown), the user returns to the screen in (A).

In one example, a HUD is provided on a device such as a smartphone, tablet, laptop, desktop monitor, or television screen. In the HUD, the user can search for features of the current application by text input or by voice control. The search will display available content, but will also display in which other open applications the feature is available. To display other applications with the same feature, the user interface can use an application stack, which can be viewed as a collection of icons at the top of the HUD. An example is shown in Figure 8. The search input box and application stack are shown in Figure 8.

A search function is provided. In one example, on the home screen, swiping down from the upper left corner brings up the search function. An example is shown in FIG9A . From here, local searches can be performed and global searches can be performed across the Internet. For easier overview of the results, the search results can be organized into appropriate categories. An example of organized search results is shown in FIG9B .

A home page for a smartphone is provided. The home page can be such that scrolling down on the home page collapses columns that were scrolled out of view, and when scrolled back into view, the columns can be expanded again. The home page can be such that swiping from right to left on the screen brings up the application page, and swiping from left to right brings up the social domain. An example is shown in FIG10 , where the "Favorites" column in (A) collapses when scrolling down to leave the collapsed "Favorites" column in (B).

A home page for a tablet device is provided. The home page can be such that scrolling down on the home page collapses columns that were scrolled out of view, and when scrolled back into view, the columns can be expanded again. The home page can be such that swiping from right to left on the screen brings up the application page, and swiping from left to right brings up the social domain. An example is shown in FIG11 , where the "Favorites" column in (A) collapses when scrolling down to leave the collapsed "Favorites" column in (B).

An application page for a smartphone is provided. The application page can be configured such that scrolling down on the application page collapses columns that were scrolled out of view, and when scrolled back into view, the columns can be expanded again. The application page can be configured such that swiping from right to left on the screen brings up a video page, and swiping from left to right brings back the home page.

An application page for a tablet device is provided. The application page can be configured such that scrolling down on the application page collapses the columns that were scrolled out of view, and when scrolled back into view, the columns can be expanded again. The application page can be configured such that swiping from right to left on the screen brings up the video page, and swiping from left to right brings back the home page. An example is shown in FIG12 , where the "Running Apps" column in (A) collapses when scrolling down to leave the collapsed "Running Apps" column in (B).

A video page for smartphones is provided. The video page can be configured so that scrolling down on the video page collapses columns that were scrolled out of view, and when scrolled back into view, the columns can be expanded again. The video page can be configured so that swiping from left to right on the screen brings back the application page.

A video page for tablet devices is provided. The video page can be configured such that scrolling down on the video page collapses the columns that were scrolled out of view, and when scrolled back into view, the columns can be expanded again. The video page can be configured such that swiping from left to right on the screen brings back the application page. An example is shown in FIG13 , where the "Recent" column in (A) collapses when scrolling down to leave the collapsed "Recent" column in (B).

A social domain page for smartphones is provided. The social domain page can be configured so that scrolling down on the page collapses columns that are scrolled out of view, and when scrolled back into view, the columns can be expanded again. Swiping from left to right on the screen can bring up the music page. Swiping from right to left can return to the home page. An example is shown in FIG14 , where scrolling down collapses the "Recent Contact" column in (A) to leave the collapsed "Recent Contact" column in (B).

A social domain page for tablet devices is provided. The social domain page can be configured so that scrolling down on the page collapses columns that are scrolled out of view, and when scrolled back into view, the columns can be expanded again. Swiping from left to right on the screen can bring up the music page. Swiping from right to left can return to the home page. An example is shown in FIG15 , where the "Favorites" column in (A) collapses when scrolling down, leaving the "Favorites" column collapsed in (B).

A music page for a smartphone is provided. The music page can be configured so that scrolling down on the music page collapses columns that are scrolled out of view, and when scrolled back into view, the columns can be expanded again. Swiping from right to left on the screen can bring back the social domain page. An example is shown in FIG16 , where scrolling down collapses the "Recent" column in (A) to leave the collapsed "Recent" column in (B).

A music page for tablet devices is provided. The music page can be configured so that scrolling down on the music page collapses columns that are scrolled out of view, and when scrolled back into view, the columns can be expanded again. Swiping from right to left on the screen can bring back the social domain page. An example is shown in FIG17 , where the "Recent" column in (A) collapses when scrolling down, leaving the "Recent" column collapsed in (B).

A message page for a smartphone is provided. A user can bring down a list of messages by swiping a finger downward from the message icon in the upper right corner of the screen. An example is shown in FIG18 , where in the sequence from (A) to (I), a list of messages is brought down as a finger is swiped downward from the message icon.

A message page for a tablet device is provided. A user can bring down a list of messages by swiping down from the message icon in the upper right corner of the screen. An example is shown in FIG19 , where in the sequence from (A) to (B), a message list is brought down as a finger is swiped down from the message icon.

A message page for smartphones is provided. This message page allows for message interaction. For example, a user can select from a drop-down message list to directly return a missed call, compose a message, or send a quick reply using a pre-defined message. From this message list, a variety of different message types can be replied to, including (but not limited to) missed Skype calls and messages, emails, and Facebook messages. An example is shown in FIG20 , which shows a list of pre-defined messages from which a quick reply can be selected for sending.

A messaging page is provided for tablet devices. This messaging page allows for messaging interactions. For example, users can select from a drop-down message list to directly return a missed call, compose a message, or send a quick reply using a pre-defined message. From the message list, users can respond to a variety of different types of messages, including (but not limited to) missed Skype calls and messages, emails, and Facebook messages. In one example, users can reply to emails.

A message page for a smartphone is provided. An accordion function can be provided, wherein when a message is selected, the messages above and/or below it are pushed aside to make room for an area in which a reply can be written. Examples are shown in Figure 21. In one example (see, for example, (A)), an area is expanded from which the user can select whether to send a text message or return a call. In one example (see, for example, (B)), a text box for entering a text message is opened within a message in a message list. In another example, an area is opened from which a quick reply text can be selected from a list of pre-prepared replies.

A message page for a tablet device is provided. An accordion feature can be provided, wherein when a message is selected, the messages above and/or below it are pushed aside to make room for an area in which a reply can be written. In one example, an area is expanded from which a user can select whether to send a text message or return a call. In one example, a text box for entering a text message is opened within a message in a message list. In another example, an area is opened from which a quick reply text can be selected from a list of pre-prepared replies.

A note page for a smartphone is provided. An accordion function can be provided, wherein when a note is tapped, it expands (e.g., like an accordion) for easier editing. An example is shown in FIG22 . In FIG22 , in the sequence from (A) to (C), a note expands when tapped, and a touch screen keyboard is provided for editing the note content.

A note page for tablet devices is provided. An accordion feature can be provided, wherein when a note is tapped, it will expand like an accordion for easier editing. The note can expand when tapped, and a touch screen keyboard can be provided for editing the note content. An example is shown in Figure 23. In Figure 23, in the sequence from (A) to (B), the note expands when tapped, and a touch screen keyboard is provided for editing the note content.

A conversation history page for smartphones is provided. In one implementation, the conversation history page displays missed calls and messages. An example is shown in FIG24(A). In an alternative implementation, SMS messaging is separated into its own application called Messaging, and phone calls are no longer displayed as part of the conversation history. An example is shown in FIG24(B).

A conversation bar page for a tablet device is provided. In one implementation, the conversation bar page displays missed calls and messages.

A launcher bar for a smartphone is provided. The launcher bar can be used from all views of the phone, the welcome screen, the home screen, or within an application. It can be revealed by a brief swipe from the left edge of the screen. Holding a finger on an application icon can reveal the name of the application. The launcher bar can display favorite and open applications. An example is shown in Figure 25. In Figure 25, the sequence from (A) to (D) shows an example of revealing the launcher bar after a brief swipe from the left edge of the screen. An example is shown in Figure 26. In Figure 26, the sequence from (A) to (C) shows an example of the launcher bar scrolling down (from (A) to (B)) and an application icon revealing the name of the application (from (B) to (C), corresponding to the application "Phone" in this example).

A launcher bar for smartphones is provided. In an alternative implementation, a long press on a launcher icon displays the application name and may display a list of shortcuts. An example is shown in FIG27 for the application "Browser." An option may be provided to remove a launcher icon from the launcher. It is also possible to reorder the launcher icons by long pressing an icon and then dragging and dropping it to a new position in the launcher.

A launcher bar is provided for tablet devices. The launcher bar is available from all views of the tablet device, the welcome screen, the home screen, or within an application. It can be revealed by a brief swipe from the left edge of the screen. The name of the application can be revealed by holding a finger on the application icon. An example is shown in Figure 28. The launcher bar can display favorite and open applications.

A launcher bar for smartphones is provided. It is possible to go back between applications. To start a favorite application, the user can select an application from the launcher bar. To return to a previous application, the user can swipe a finger across the screen from the right edge of the screen. An example is shown in Figure 29, which gives a sequence of screen images from (A) to (E). In (A), the user selects the application "Notebook" from the launcher bar. In (B), the screen displays the notebook application screen output. In (C), the user swipes a finger across the screen from the right edge of the screen to return to the previous application "Events". In (C) and (D), the screen output from the previous application "Events" is returning to the screen from the right. In (E), the screen output from the application "Events" is displayed.

A navigation function for smartphones is provided. In some implementations, a user swipes a finger horizontally in a page header (e.g., where contacts are displayed) to bring up a navigation function screen that helps the user navigate and change screens with a simple finger tap. An example is shown in FIG30 , where, in the sequence from (A) to (B), the contacts screen changes to a call screen corresponding to the contacts screen.

A left edge swipe feature is provided for smartphones or tablets. On both phones and tablets, a long swipe from the left edge of the screen will bring up the app page, except when in the home screen. An example is shown in Figure 31 for the sequence from (A) to (B).

A right edge swipe feature is provided for smartphones. When viewing the Welcome screen, swiping left from the right edge of the screen brings up the Home page. An example is shown in Figure 32 for the sequence from (A) to (D). In all views except the Welcome screen, swiping left from the right edge of the screen brings up previously used and open apps.

A right edge swipe feature is provided for tablet devices. As on a phone, swiping left from the right edge of the screen brings up previously used and open apps within the tablet's side stage. In all views except the Welcome screen, and when the side stage is in view, swiping left from the right edge of the screen brings up previously used and open apps. When viewing the Welcome screen, swiping left from the right edge of the screen brings up the Home page. An example is shown in Figure 33 for the sequence from (A) to (C).

Provided are arrival indicators for smart phones. In one example, swiping down from an indicator icon brings down settings related to that icon. Without releasing a finger when swiping down, moving the finger horizontally allows the user to select what indicator will be displayed. Tapping on the indicator bar displays a prompt about the indicator that will be displayed. It is also possible to navigate between indicator screens by tapping on the tab header to reveal more indicator tabs and then selecting one of them by tapping. An example is shown in Figure 34. A sequence from (A) to (E) is shown in Figure 34. In (A), swiping down from the sound icon provides screen (B). Without releasing a finger when swiping down, moving the finger horizontally allows the user to select what indicator will be displayed, thereby making moving the finger horizontally provide a screen sequence from (B) to (E). Figure 34 shows an example of available views from swiping down from the top right edge of the screen (where each indicator is displayed) (and then swiping sideways, as mentioned above).

In one implementation, on a smartphone, an overview of settings can be reached by tapping on the top of the screen (where the indicator icon is found). This brings a settings screen into view, which can redirect the user to other settings views. A "settings overview" screen is provided. The indicators can be accessed by swiping down and sideways from the top edge. It is also possible to navigate between indicator screens by tapping on the tab headers to reveal more indicator tabs, and then selecting one of them by tapping. Detailed system settings can be accessed through a dedicated settings app, which can be accessed by tapping on shortcuts on certain individual indicator screens, and can also be accessed by opening the system settings app from the app domain.

In one implementation, tapping on the icon at the top of the screen will provide the corresponding settings screen. An example of settings for a smartphone is shown in Figure 35. A sequence from (A) to (B) is shown in Figure 35. In (A), tapping the sound icon at the top of the screen (where the sound icon is found) provides screen (B). Starting from screen (A), tapping on the network icon at the top of the screen will provide screen (C). Starting from screen (A), tapping on the battery icon at the top of the screen will provide screen (D). Starting from screen (A), tapping on the date and time icon at the top of the screen will provide screen (E).

Reach indicators for tablet devices are provided. In one example, swiping down from an indicator icon (e.g., in the upper right corner of the screen) brings down settings related to that icon. Without releasing the finger when swiping down, moving the finger horizontally allows the user to select what indicator will be displayed. Tapping on the indicator bar displays a prompt about the indicator that will come down. It is also possible to navigate between indicator screens by tapping on the tab header to reveal more indicator tabs and then selecting one of them by tapping. Figure 36 shows an example of a battery settings screen on a tablet computer screen.

In one implementation, on a tablet device, the overview of settings can be reached by tapping on the top of the screen (where the indicator icon is found). This brings the settings screen into view, which can redirect the user to other settings views.

A tablet device side stage is provided on the tablet computer screen. In one example, the tablet device side stage can be pulled into view by the user swiping his finger horizontally from the right edge of the screen. An example is shown in Figure 37, which gives a sequence of screen images from (A) to (C). In (A), output from the "Events" application is displayed. In (B), the "Notes" side stage is being pulled into view by the user swiping his finger horizontally from the right edge of the screen. In (C), the "Notes" side stage has been pulled into view. In one example, the tablet device side stage can be viewed in two different ways. This can be either on top of the main view or through a split screen, which will resize the main view to fit into the new space provided.

The tablet's sidebar supports multitasking. For example, you can browse music or watch videos while making a phone call, or view your gallery while writing a text, changing system settings, or taking a note. You can do many things simultaneously. The tablet's sidebar allows you to swipe from the right edge of the screen to navigate through previous apps within the sidebar.

A welcome screen is provided on a smartphone. The welcome screen can include an infographic where each dot in a circle represents a day of the month, with the highlighted dot being the current day. In one example, the highlighted circles surrounding the center circle represent the number of tweets received during a specific time period. An example is shown in FIG38 . Other data can also be depicted in this manner.

A welcome screen is provided on the tablet device. The welcome screen can be customized by the background image, colors, and data displayed. An example is shown in FIG39 . The tablet device can have multiple accounts, including a private account and a guest account that anyone can use. The guest account may not have an information graphic on the welcome screen.

Notice

It should be understood that the above-mentioned arrangements are merely for the purpose of illustrating the application of the principles of the present invention. Without departing from the spirit and scope of the present invention, many modifications and replacement arrangements may be envisioned. Although the above description of the present invention is fully described above in conjunction with the most practical and preferred embodiments currently considered to be the present invention (a plurality of) and in the accompanying drawings, it will be appreciated by those skilled in the art that many modifications may be made without departing from the principles and concepts of the present invention as set forth herein.

Claims (31)

1. A smartphone, tablet, or other computing device, comprising: (a) A touch-sensitive display; (b) One or more processors; (c) Computer memory; (d) A computer program stored in the computer memory, configured to be executed by the one or more processors and comprising instructions for: Detecting a scan from each and any one of the four edges of the touch-sensitive display, and responding to the detected scan by causing the device to operate in a manner dependent on the specific edge from which the scan originates; and A welcome screen is displayed that features dynamic pictorial information graphics or dynamic data visualizations, which graphically represent each of a plurality of variables that are in real time relevant to the device or one or more of its users. These variables are a composite of data from multiple sources, applications, online services, and the device. Each variable is assigned a customized and unique position in a geometric pattern or one or more other patterns that form at least a portion of the pictorial information graphics or data visualizations. The welcome screen prevents unintentional input when the device is not being observed. The customized unique location indicates to the user the type of variable with a changed state, as well as the shape, space, or volume of the graphic at the customized unique location that indicates the changed state of the variable to the user in real time. The user unlocks the device or allows the device to access and expose an application launcher from the welcome screen by scanning from the first edge. The application launcher selectively runs applications that can be launched directly by continuing the scanning from the first edge, and may or may not prompt for a personal identification number or password depending on the previously launched application. The continued scanning from the first edge was not associated with any notification. 2. The device of claim 1, wherein the device operates in a manner dependent on a specific edge from which the scan originates, by displaying a UI graphic, object, or set of objects of a specific type depending on the edge from which the scan originates. 3. The device of claim 1, wherein the device always responds to the detected sweep in a consistent manner, regardless of what the device is displaying. 4. The device of claim 1, wherein it detects a scan from an edge of the touch-sensitive display, and in response to the detected scan, causes the device to directly or after a further intermediate step to open an application launcher as a set or group of application icons, or to start or switch to an application. 5. The device of claim 1, wherein it detects a scan from one edge of the touch-sensitive display, and in response to the detected scan, causes the device to switch back to a previously opened application. 6. The device of claim 1, wherein, in response to a scan from the left edge, the device opens an application directly or after a further intermediate step, and in response to a scan from the right edge, the device displays a previously used but still running application directly or after a further intermediate step. 7. The device of claim 1, wherein if the device is displaying a welcome screen, the device displays the device main page directly or after a further intermediate step in response to a scan from the right edge. 8. The device of claim 6, wherein the application launcher is brought out by initiating a "start" via a rightward swipe from the left edge; conversely, a previously opened application is brought out by a leftward swipe from the right edge to serve as a "back" function. 9. The device of claim 8, wherein, in response to a further leftward sweep from the right edge, the instruction causes the device to scroll backward through all previously used applications in the order in which each application was last used. 10. The device of claim 1, wherein the device can detect when a user selects a system status icon located near an edge, and in response to a swipe away from the system status icon, the instruction causes the device to display a screen listing one or more settings associated with the icon, thereby allowing the user to quickly update the one or more settings, and subsequently swipe the screen of the settings back to the edge to display the currently running application. 11. The device of claim 10, wherein the system status icon is placed on the top edge of the touch-sensitive display. 12. The device of claim 1, wherein the instruction causes the device to display application controls in response to a brief upward swipe from an edge, to display the application controls near the edge while the application is running and controlling the touch-sensitive display, and to hide the application controls by swiping back to the edge. 13. The device of claim 12, wherein the application control is located at the bottom edge of the touch-sensitive display. 14. The device of claim 1, wherein, in response to a swipe from a message icon, the instruction causes the device to display a panel, window, or area listing one or more messages. 15. The device of claim 14, wherein, in response to a user selecting a message in the panel, window, or area listing one or more messages, a response panel, window, or area is subsequently displayed. 16. The device of claim 15, wherein if the selected message is for a missed call alert, the response panel, window, or area includes a button or icon that enables the user to: call back the person who left the message; compose and send a reply message to that person; select and send a pre-defined reply to that person. 17. The device of claim 4, wherein, in response to detecting that a user pauses on an icon of an application in the application launcher, the device displays a "shortcut list" corresponding to the application, the shortcut list being one or more user interface objects associated with features, functions, or controls of the application. 18. The device of claim 1, wherein the scan from one edge must begin at the edge of the touch-sensitive display or from outside the touch-sensitive display. 19. The device of claim 1, further comprising instructions for detecting a single gesture swipe input from an edge, and for triggering or invoking two or more distinct results or functions in response to the detection of the single gesture input, depending on the range of the detected gesture input. 20. The device of claim 19, wherein the detected range is a function of one or more of the following: distance, length, direction, path, endpoint, or destination. 21. The device of claim 20, wherein a sweep from one edge results in a first result, and if the user continues to sweep from said edge, it results in a different result. 22. The device of claim 1, further comprising instructions for detecting a scan from one or more edges, and subsequently unlocking or granting access to the device from the touch-sensitive display in response to the detected scan. 23. The device of claim 22, wherein the screen from which the device is unlocked is a welcome screen. 24. The device of claim 1, further comprising instructions for detecting a swipe gesture from an edge and for displaying a search bar in response to the detected swipe gesture. 25. The device of claim 1, further comprising instructions for detecting a swipe in from an edge of the touch-sensitive display, and for causing a main screen to be displayed in response to the detected swipe, the main screen listing running applications and user notifications, regardless of which screen the device was displaying immediately before the swipe in from the edge. 26. The device of claim 1, further comprising instructions for displaying one or more applications, utilities, or functions in a window, pane, or area on the right side of the device in response to detecting a scan from the right edge. 27. The device of claim 26, wherein the window, pane, or area is rectangular. 28. The device of claim 27, wherein the window, pane, or area can display a running application, utility, or function, thereby allowing multitasking across two portions of the display. 29. The device of claim 28, wherein, of the two parts, the main part runs an application, and the window, pane, or area runs a different application, utility, or function. 30. The device of claim 1, further comprising instructions for generating UI elements designed to remain in the same screen position as the device size scales from smartphones to tablets, laptops, desktops, and televisions, thereby providing a consistent UI across all screen sizes and device types. 31. A method comprising the following steps implemented at a computing device having a touch-sensitive display: (i) Detecting a scan from each and any one of the four edges of the touch-sensitive display, and responding to the detected scan; (ii) causing the device to operate in a manner dependent on a specific edge from which the scan originates; and (iii) Displaying a welcome screen with dynamic pictorial information graphics or dynamic data visualization, the dynamic pictorial information graphics or dynamic data visualization graphically representing each of a plurality of variables that are in real time related to the device or one or more of its users, the plurality of variables being a composite of data from multiple sources, applications, online services and the device, each variable being assigned a customized unique position in a geometric pattern or one or more patterns that form at least a portion of the pictorial information graphics or data visualization, the welcome screen preventing unintentional input when the device is not being noticed; The customized unique location indicates to the user the type of variable with a changed state, as well as the shape, space, or volume of the graphic at the customized unique location that indicates the changed state of the variable to the user in real time. The user unlocks the device or allows the device to access and expose an application launcher from the welcome screen by scanning from the first edge. The application launcher selectively runs applications that can be launched directly by continuing the scanning from the first edge, and may or may not prompt for a personal identification number or password depending on the previously launched application. The continued scanning from the first edge was not associated with any notification.