WO1996041311A2 - Stereoscopic interactive painting - Google Patents

Abstract

Disclosed is an illustrator- or painter-operated, computer-mediated system for creating binocular, stereoscopic images from 2-D and 3-D image elements created by the operator or imported into the system. Depth effects are synthesized by creating parallax offsets which are applied to the display positions of the various elements in the view presented to each eye. The magnitude of the offset for each object is related to the depth of that object, the direction of the offset is related to which eye the view is intended for. A 3-D input device is described as a system component which may be used to control such a system. Additional techniques are disclosed whereby digital information so created may be displayed, broadcast, or utilized in a virtual reality system.

Description

AN INTERACTIVE COMPUTER SYSTEM FOR CREATING THREE-DIMENSIONAL IMAGE INFORMATION AND FOR CONVERTING TWO-DIMENSIONAL IMAGE INFORMATION

FOR THREE-DIMENSIONAL DISPLAY SYSTEMS

TECHNICAL FIFXD

The instant invention relates to an operator's interactive computer workstation for the conversion of two- dimensional motion pictures, video or other image information for three-dimensional display or viewing systems, as well as for the creation and integration of three-dimensional image information.

PRIOR ART

There exist a number of true three-dimensional, stereographic, binocular display systems, including: separate CRT displays for each eye, often routed through prisms or by mirrors, or mounted in a helmet sometimes with orientation sensors; and glasses with red and green or mutually polarized filters over the left and right eyes, for use with film or video displays.

Many video displays, including home televisions, work with an interlaced raster image where each image frame is divided into two fields, with the even lines displayed first followed by the odd lines (or vise versa). For US and Japanese systems the frame rate is 30/sec and the field rate is 60/sec; in Europe the rates are 25 and 50 respectively; for many industrial applications the rate is doubled to 60 and 120 respectively; and, many other rates are available or possible.

It is thus possible to form a stereoscopic 3-D display by creating a left and right image pair interleaved into the even and odd fields (or visa versa) of a video display, and viewing that display through LCD shuttered glasses which are flickered (180 degrees out of phase for each eye) in synchrony to the field rate of an interlaced video display, separating two 'field' images from a single video 'frame' signal.

US patent 4,952,594, of which the applicant is a co-inventor, describes a method for creating and modifying 3-D program material derived from already existing 2-D motion picture films and video, and other information; including a system to produce such material. 'Paint' and similar interactive programs permit an artist or operator to create 2-D paintings, drawings, illustrations and business, technical and industrial graphics on computer systems.

SUMMARY OF INVENTION

The instant invention relates to techniques and devices for creating and/or delivering 3-D (stereoscopic, binocular, holographic, serial or otherwise) image information which may be derived from 2-D images or image elements converted to 3-D by the system, 3-D image elements created by the system, or integrated with pre-existing 3D image information created independent of the system.

The primary system component is an interactive 3-D computer paint system which shares many elements with 2-D computer paint systems, but which extends those elements suitably to deal with 3-D image information. User interface elements such as menus, buttons, palettes, cursors, and information windows may be placed in the

Z or depth direction (perpendicular to the plane of the display screen) as well as in X (horizontal) and Y (vertical); these elements may also have size or 'thickness' in the Z direction. In addition to image information (R, G & B or indexed colors) depth information may be stored with pixels in order to implement 3-D painting and processing functions. In this way, when further painting is to be performed at a particular pixel, the depth to be painted and the depth of what is already stored there may be compared; and, painting would only be accomplished if the new information was closer to the viewer than (and should thus obscure) what was already there.

Further, when painting, or other image manipulation or processing, functions are to be performed, they are performed on two separate images; one for the left eye and one for the right. However, the application of the function in each picture is offset in the horizontal direction by the appropriate parallax offset. The two images may then be interleaved for 3-D display. Thus, many standard 2-D painting and image processing algorithms may be applied, separately to each picture, without having to re-design or re-program them for interleaved 3-D images. Techniques are disclosed for preparing such images for display with anaglyphic, LCD-shuttered, or new types of flickering glasses; or for broadcast by dual channel video systems. In addition, it is disclosed how such information may be adapted for use with virtual reality systems.

Additionally, system components are disclosed which permit the input, by a human operator, of more positional information than simply X and Y position, yet which may be constructed by combinations of simple X-Y transducers or other input elements used in combination.

BRIEF DESCRIPTION OF DRAWINGS Figure 1 depicts the use of a standard paint system and a specialized 'disinterleaving' imaging function to create stereoscopic images.

Figure 2 depicts a 'disinterleaved' 4 x 4 pixel brush. Figure 3 depicts two-ball (3-D) mouse operational modes.

Figure 4 shows the parallax offsets for near, far and screen depth information in the left and right images of a 3-D paint system.

SCOPE OF INVENTION The instant invention relates to an operator's interactive computer workstation for the conversion of two- dimensional motion pictures, video or other image information for three-dimensional display or viewing systems. It may also be employed for the creation and processing of new 3-D image information, and for the integration of either type of image information with three-dimensional image information imported into the system.

For example (as described in patent 4,925,294), when converting a particular still frame of a 2-D motion picture to 3-D, gaps between image parts may develop. In the patented system, functions are provided that permit the operator to fill gaps in the converted motion picture by borrowing image information from other parts of the frame or from other frames in the sequence, or to create 'fill-in image information' by utilizing functions similar to known 2-D 'computer paint systems'. In the absence of a 2-D motion picture frame to convert, such functions may be utilized to create new 3-D images. As such, the system may be used as a '3-D Paint System' for creating non-computed (i.e., operator or 'hand' generated) 3-D paintings, technical illustrations, business graphics, or other operator specified graphics. In addition, such a system could also be used for adding image information to existing true 3-D images, including 3-D photographs and computationally generated binocular 3-D graphics. For example, circles, arrows, scales, axes and text notes may be drawn 'within the 3-D space' of a 3-D video display, in much the same way that a marking pen can be used to annotate on the surface of a standard 2-D photograph. The technologies described herein may also be used, either in an illustrator operated system or more automatic system, for purposes of converting 2-D medical images to 3-D. In this way, traditional diagnostic and other medical images containing complex data may use the added depth dimension to enhance visualization and increase comprehension.

The embodiments described herein employ LCD shuttered glasses, synchronized to the field rate of an interlaced video display. Such a system may be used for both design and production of displayable and recordable end product on video. Similar systems may be designed around other stereoscopic or holographic display technologies; or, the system as described herein may be used by the operator to design images that will later be produced for display, viewing or recording by any 3-D means currently in use or later developed. Such variations are considered to be within the scope of the instant invention.

A number of system components or techniques mentioned herein — such as: stereoscopic systems of various kinds; computer graphic devices, including frame buffers; interactive computer graphic software systems such as 'paint systems'; or, techniques, such as 'Z-buffering' — already exist and may be the subject of patents or other publications. The literamre on such elements is well developed and the reader is directed to that literature for additional details of implementation. Although such components play a significant role in the instant invention, it is not the purpose of this application to claim the invention of such basic components or techniques. Rather, the subject of the instant application are composite designs and processes, which employ and combine such existing components into novel and useful systems or for new purposes, as well as numerous additional technical details which, in and of themselves, are novel. A number of techniques will be described and/or claimed as being embodied in either digital hardware or software. It is often possible to simulate or mimic the functionality of special-purpose hardware, within an environment of general purpose hardware, by the coding of control software. It is intended that all of the techniques and functional structures described herein may be embodied as either: specially constructed hardware; general purpose hardware with the appropriate software; or, any combination of the two. Similarly, a number of electronic functions may be embodied as either digital or analog circuitry; and, various display technologies — such as photographic or other film, holographic film or display, lenticular screens, HDTV, plasma panels, stationary or moving LEDs or LCDs, etc. — may be substituted for video displays. It is intended throughout that these and similar substitutions, combinations or hybridizations are within the scope of the instant invention, even though not specifically stated below. The term '3-D Computer Graphics' is often used to describe the rendering of a 2-D projection of a 3-D or

'solid' mathematical model onto the plane of the computer display screen. Such images can be quite sophisticated, taking complex lighting, shading, shadows and reflections into account, and the computational techniques for such rendering are highly developed. However, such '3-D Graphics' are 3-D only in the computational sense and result in a single 'monocular' 2-D image. In contrast, as used herein, the term 3-D will generally be used to mean binocular, stereoscopic or holographic images which exhibit depth characteristics when viewed.

It is also possible to create two 2-D computer rendered images from 3-D models, such that the perspective view of each corresponds to the view as seen by the left and right eye of the viewer. Such image pairs may be displayed by true 3-D display systems to achieve synthetic stereoscopic imagery. Such displays are sometimes used for entertainment purposes, but more often for industrial applications such as CAD or flight simulators. Such straight forward 'solid model' rendering — even if made in stereoscopic pairs — is, generally, not being claimed as part of the instant invention. Rather, the instant invention relates to an interactive system which permits the operator to use 'painterly' functions to create 'true 3-D' images, exhibiting a sense of depth.

Lastly, discussion of vertical parallax will be omitted here, for the sake of simplicity and clarity. However

— especially for objects close to the viewer, on the order of the interocular distance — noticeable vertical parallax or differential obscuring may come into effect. The same principles and techniques described herein for the horizontal case may be applied in the perpendicular direction, as and if desired, and are considered to be within the scope of the instant invention.

DESCRIPTION OF INVENTION As described in applicant's US patent 4,925,294, 2-D image elements may be relatively offset and combined with other image information elements to create 3-D images. A computerized system for creating such images is described herein.

In one basic embodiment, the instant invention may be implemented using standard, off-the-shelf hardware and software components, with the addition of specialized software functions and the careful application of production techniques. A standard 'paint' program, operating on a standard interlaced video computer display (frame buffer), may be used to convert separate 2-D image elements into a 3-D composite as follows.

In Figure IA, the rearmost image element (10, a hollow square in this example), called plane zero or PO, is brought into the frame buffer by running a standard 'image load' function of the paint software, or may be created using computer paint functions. In general, this PO element should contain image information (even if it is only a

'blank' background) to fill the entire area of interest; but, this area may be somewhat smaller than the entire display, as will be made clear below, to take into account left/right image shift.

When PO is viewed through LCD shuttered glasses, the even/left image field and odd/right image field will register perfectly. There will be no differential shift or parallax and, thus, no depth effect. This 'neutral' depth plane will appear to be at the plane of the front of the CRT which will also be perfectly registered between the left- and right-eye views.

At this point, a special software function is invoked. If the paint system software that is used does not permit calling a user specified outside function, it may be necessary to exit the paint software, run this special function, and then re-enter the paint program. This function will shift the even/left-eye field lines of the image in one direction by an amount SO and the odd/right-eye field lines in the other direction, generally by the same amount. See Figure

IB. (Note: the even/left — odd/right relationship may be as stated or reversed to odd/left even/right; only consistency is important.) This will cause the image plane to appear to move closer to (if SO is 'positive') or further from (if SO is 'negative') the viewer, by an amount DO, due to the parallax between the two interleaved images. The larger SO is made, the more the image plane will appear to be moved in the direction perpendicular to the CRT plane, the 'Z' direction. For the purpose of this discussion, the rearmost planes will be moved progressively further away from the viewer at each stage.

In Figure IC, the standard paint software is then used to load or paint a next-closer 2-D image plane, PI, (11, a vertical bar in this example) which may also have been derived by separating it from a 2-D motion picture frame, as in applicant's patent '425. This and later, more forward elements will, in general, be loaded with part of the image area 'transparent' (or painted/created with 'holes') allowing PO or other relatively rearward planes to be seen through them. When images are stored as files, transparency is often denoted by storing a particular numerical 'color' at those portions, often (0) for indexed or 'color-mapped' images, or (0,0,0) for full-color, R-G-B images. In Figure ID, the composite image, consisting of P0 offset by SO and appealing to be DO behind PI, is now offset in a similar fashion by the special software function, by an amount SI. Now P0 is offset a total of SO + SI, appearing DO behind PI and DO + Dl behind the face of the CRT; PI is offset by SI appearing Dl behind the face of the CRT. Note that, for certain circumstances, the apparent distances (Ds) may not, in fact, add linearly; but, the distances do accumulate, and the relative ordering of the planes is as described.

The process is repeated and many 2-D image planes are, thus, stacked-up, back-to-front, to form a multi- planed, 3-D image when viewed through the flickering glasses. The back-to-front order guarantees that obscuring of rearward layers, which will be different in the left and right images due to parallax offsets, is done correctly.

By progressively moving the composite image away from the viewer, or into the CRT display, the last, forward-most image plane will be at the front of the CRT and the full composite image will appear to be 'inside' the video display, similar to looking into a fish tank. By running the odd/even differential shifting function one last time, by an amount equal to SO + SI + S2 + ... + Sn-1, but in the opposite direction, the rearmost image plane may be restored to the plane of the CRT face. See Figure IE. Thus, the composite image will then appear to come out of the CRT into 'room space'. This restoration process may be run part way and arbitrarily set any position in the composite image (even between planes containing images) as the 'neutral' plane, without relative shift between the odd and even image fields. In shifting image elements left and right, some image part may 'fall off the edge' of the frame buffer and be lost. As the image (which is spread, in opposite directions horizontally on alternate lines to create parallax, for alternating field stereo) is brought close to the edge of the screen, the field for one eye or the other will 'fall off the edge' first. This will leave part of the image visible on only alternate lines, for a single eye, creating a ragged edge. Alternately, depending upon the details of the hardware and software used, the image information may 'wrap- around' and incorrectly re-appear at the opposite edge of the frame buffer.

For this reason, the width of the area of the display used for imaging in this embodiment may best be made less, by the amount of the maximum shifts involved, than the maximum width of the display. To do this, the Z- Buffer may be initialized to a horizontally α minishing wedge to avoid images with ragged edges.

If the niinimum increment is 2 pixels (a minimum of one offset in each direction) then to avoid ragged edges while painting, the last two usable pixel columns on each side must be avoided by me next farthest plane. This restriction continues, for further planes, loosing two additional on each side. This is true, starting from the '0th' or neutral plane, for both into and out of the screen, although parallax is reversed between the two cases.

Alternately, an extra wide frame buffer, providing non-visible pixels beyond the edges of the display area, may be employed and would permit the storage and restoration of those parts of image planes temporarily pushed off the edge of the display screen by the software shift function. Such a frame buffer would also avoid errors which could result from temporary wrap-around caused by shifting or unshifting the composite image. The AT-VISTA is an off-the-shelf display buffer for the IBM-PC that may be software configured to provide such capabilities.

If the functionality of a paint system may be modified or if a new '3-D Paint System' is programmed, the "layer-shift-layer-shift-layer . . . " process may be incorporated into the program. Alternately, this process may be replaced by a more convenient functionality. This cumbersome process is required because a standard paint system 'paints' or performs its other imaging functions at the 'neutral' plane, where the odd and even image fields are not relatively shifted. In a preferred embodiment of the instant invention, the paint brush (and all other painting, graphic and imaging function such as: draw a line, circle, square, polygon, text or other graphic element; load, move, copy, or modify a portion of image; as well as all other traditional 2-D paint and graphic functions) would, itself, be programmed to have its odd and even image fields offset by a variable amount.

Figure 2 shows a small square 4x4 pixel brush. In 2A the brush is set for the zero or neutral plane with no relative offset of the odd and even lines of the brush image. In 2B the odd and even lines are offset for plane 1; in 2C for plane -1; in 2D for plane 2; in 2E for plane 20. Although much more convenient to use, this technique does have a drawback to implementation. The separated brush and interleaved left and right images require that any painting function be applied separately to the odd lines and even lines of the frame buffer contents. Programming algorithms that are applied to non-continuous areas of frame buffer memory is both a conceptual and a computational complication. Additionally, the contents of many frame buffers are manipulated by transferring rectangular areas of the image to the computer for modification and then remrning the data to display memory. The overhead of initiating a transfer is often very high compared to the transfer of each additional pixel. A 4x4 brush would require a single transfer of 16 pixels back and forth. A separated brush as shown in Figure 2E would require 2 transfers of 12 pixels (21 and 22) or a single rectangle containing 176 pixels (23).

A third embodiment that eliminates or reduces me above problems can be implemented in some configurable display boards such as Truevision's AT- VISTA, which includes a 'row table' feature. These devices permit the programmed specification of which memory line will be displayed on the screen for each display line. In that case the left and right images can each be placed in continuous rectangular blocks of the display's memory. The display processor is programmed, however, to display these two images in interleaved fashion. The advantage is that all functions and algorithms can then be programmed to operate on (two) continuous rectangles of memory, rather than having to be made more complex in order to work with two interleaved views. Many standard, well-known and well- understood, and 'built in' functions can then be used as in standard computer paint systems, including: drawing of lines, polygons, etc.; digital filters, anti-aliasing and otiier image processing functions; color and pattern filling; etc. However each function will have to be applied twice, once to each image, and all X-coordinates must be offset in each image by a parallax value appropriate to the depth plane at which the function is to be applied.

Another advantage of this method is that different types of interleaved and non-interleaved display formats can be accommodated for different display systems from the same frame buffer memory content, just by changing the table that maps memory to display. For example, switching between left/even right/odd configuration and a left/odd right/even configuration only requires that the table entries be shifted down or up by one line; also, a pad entry may be made at the top or bottom, and one display line lost at the bottom or top. Similarly other display formats, such as those required for the StereoGraphics brand of high frame rate alternating stereo displays can also be easily programmed.

This third scheme is the preferred embodiment of the instant invention.

With the later embodiments described above, the artist would be able to position the brush (or other imaging function) along the Z (depth) axis as well as the X (left to right) and Y (top to bottom) axes. In addition, the brush

(or other imaging function) itself may have a depth 'thickness' or size, as well as a position.

The brush position, or size, along the Z-axis would be displayed as a cursor, positionable in all three dimensions, and may be controlled by any appropriate computer control mechanism now in use or later developed, including: 1. A depth palette may be provided, much like a standard color palette, that allows the artist to pick off a depth;

2. The above may be in the form of a depth 'ruler', as described below;

3. The artist may pick off a depth by pointing at a particular item or spot in the image;

4. A numerical value may be typed in at a keyboard; 5. A pen with a variable pressure tip (rather than the standard click-on tip) may be used to specify depth;

6. Other physical input transducers may be used, such as a knob, foot pedal, joystick, 'data glove', 3-D input device as described below, or other control or position sensor; or,

7. A graphic of a knob, slider or other control icon may be controlled by the artist; and additionally,

8. The computer may be programmed to compute a depth by: a depth value, depth tag, or other depth specification, stored with an image element; interpolation, either spatially over a frame, or temporally between frames; by image analysis of any type; or, as described below, by consulting a depth or Z-buffer.

Since the left and right pictures and the left and right brush are interleaved for display, into the odd and even fields of a video frame, moving up or down by a single display line reverses the relationship between which brush field is displayed in which picture. Thus, if, on even vertical cursor positions, the left brush is painted into the left image and the right brush is painted into the right image, on odd vertical cursor positions, the right brush is painted into me left image and me left brush is painted into the right image. However, the parallax shift also must be alternated or the brush will flip in and out of the screen on alternate lines.

If the above is termed Field Mode; a Frame Mode is also possible. In Frame Mode, the cursor position may only move in multiples of two lines vertically and, thus, the relationship between brush halves and picture halves is always preserved.

In particular, it should be noted that image processing related 2-D painting functions, such as filtering, averaging, smearing, blurring or sharpening functions, usually are programmed to be applied to the pixels in a rectangular window or arbitrarily shaped matte, or to those pixels under the shape of the brush. For 3-D painting, these 2-D algorithms can be apphed to a single plane, or may be generalized to be applied to all the pixels in a 3-D rectangular solid, other arbitrarily shaped 3-D volume or, in particular, within the volume of a 3-D brush shape.

The image processing algorithms may be generalized to 3-D form as well; for example, 'blurring' or

'softening' functions are often implemented as algorithms that "average the pixel values (often with weighting factors) within a given area" which, when generalized, would become "average all pixel values within a given volume". Alternately, one may choose to apply traditional 2-D image processing algorithms to individual 2-D planes that are 'stacked' in 3-D space, working only on the pixels at a specified depth or range of depths. In that case, when the computational kernel (neighborhood of pixels used to compute a particular pixel value) comes to an edge of an object in a particular plane, non-standard options arise. For example, those pixels that are in the kernel but not on a specified plane in question may or may not effect the computation; and may or may not be effected by the computation. Directional algorithms, for example bas relief, may or may not be reversed in the two eye views.

Generally, based upon the depth at which a digital filter is to be applied, the center of the kernel will be horizontally displaced by the parallax offset differently for each eye view. In general, the differential layering of image elements in the two eye views will yield different results when digital filters are applied. Alternately, if the 'hidden' pixels of an image element on a particular plane are still available to the computer, the digital filter kernel can be applied to both the visible and obscured pixels of a plane.

Additional discussion of the "depth (D- or Z-) buffer": The above describes a convenient usable system for creating and modifying stereoscopic images. However, the 3-D effect depends not only on horizontal parallax, but on the proper obscuring (which is subtly different for each eye) of rearward image elements by forward image elements. If the visual cues of parallax and obscuring are inconsistent, confusion results and the perception of 3-D is destroyed. By painting from back to front, one may guarantee that this aspect is properly handled; but, this is inconvenient and would interfere with the creative process by unduly restricting the freedom to paint and modify an image in arbitrary order. An alternate embodiment, wherein depth information, in addition to display color information, is stored for image pixels, takes care of this problem and is described below.

For many standard computer graphic display devices — full color displays — memory planes are provided for storing Red, Green and Blue values at each pixel; each pixel may have an independently specified color.

Alternately, for color-mapped devices, a single number is stored at each pixel and is translated into Red, Green and

Blue values by using it as an index into tables of Red, Green and Blue values — called color look-up tables or color maps — with all pixels sharing a common index displayed as the same color.

In addition to die Red Green and Blue (or Index) information for each pixel, it is common to store additional information in other memory planes. One, three, eight, sixteen or more memory planes (corresponding to the number of bits per pixel) of additional information may be provided, with eight being very common for moderate cost computer displays; often with 32 bits per pixel being assigned as eight bits each for Red, Green, Blue and Alpha (a non-displayed channel). The Alpha channel is traditionally used for a number of purposes which can include: overlay information (e.g., text or menus); video keying or (soft) matting of the computer display signal with another signal; masking off parts of the display from further processing; temporary storage of items such as menus, icons or palettes — or, of the parts of images obscured by those items; and, pixel transparency.

For computer rendering of 2-D images of '3-D solid' (but not stereoscopic) mathematical models, depth information is sometimes stored for each pixel to facilitate the proper display of many intersecting objects. Rather than trying to compute the correct intersection of diverse objects, the pixels of each object calculated are only put into the display if they are closer to the viewer than the equivalent pixels already contained in the frame buffer, thus automatically creating the proper intersection between, and obscuring of, objects without their direct computation. Such a depth memory plane is often called a Z-buffer. It is most often 16 or more bits deep, in order to provide enough resolution (2 raised to the 16th power, or roughly 64 mousand divisions) for the proper mathematical placement of objects in the computer's simulated 3-D (but still displayed as 2-D) environment.

While a 16-bit buffer may be used in the embodiment of the instant invention under discussion, an eight-bit Alpha channel may also be used to store depth information to great advantage, even though eight bits provide a resolution of only 256 values. Eight bits of non-display memory is routinely available for computer paint systems while 16 extra bits is not. A detailed discussion follows. While 256 divisions may not be sufficient to specify the depth of a particular object in a simulated maύiematical space, in the instant invention 2-D image layers are often stacked up and differentially shifted, to create a stereographic display. These 2-D image layers may be ordered and each assigned a depth or D value. This D value does not need to represent an exact depth value, just an ordering. The differential odd/even field left/right offsets will have to be calculated correctly for an exact depth value; but for obscuring, only an order is required. Thus, the much cruder resolution afforded by 8 bits can be used for the instant invention. For die purposes of this application, the use of a depth channel will be referred to as a D-buffer.

In the simplest embodiment, when loading or creating a layer, in addition to the differential offset function, for each pixel, the pixel value will only be used to replace the R-G-B values already stored for that pixel in the frame buffer if the new pixel's depth is closer to the viewer and, therefore, should obscure that which is behind it. For more flexibility, the depth range behind, equal to, and in front of the selected depth for painting may be treated separately and pixel-by-pixel replacement for each range may be individually permitted or not. Not all combinations will make sense, but may still be used for effect. Additionally, individual deptii planes, or ranges of depths may be protected from replacement or not. (See section on "Protection".)

For each possible D-buffer value, the value may represent the actual depth. Alternately, a number may be associated witii each D-buffer value which in mm indicates the actual depth which will be used to offset the pixels of that plane; in effect a depth look-up table or depth map, analogous to a color map. (Note that non-standard hardware or special software may be required to implement some of the following techniques.) Thus, the 256 planes specifiable with an 8-bit D-buffer do not need to be evenly spaced, and may be positioned with arbitrary resolution in the depth direction. When D-buffering is to be done on a point-by-point basis for generally shaped objects, rather than for just parallel planar elements, non-linear spacing may be particularly useful.

In general, this approach is useful and eliminates much of the reason why 16 bits are often considered as required for Z-buffering. Sixteen bits are usually required because the space between 'nose close' and (for all intents and purposes) infinity needs to be divided into many segments if they are to be of uniform size. If too few segments are used, an aliased or chunky effect will result when trying to effect intersections between objects. For example, if working from the screen plane into room space, the relative left and right shift required for two objects separated by a few inches will be very great if the two objects are close to the viewer; but, the left and right shift required for two objects separated by a few inches will be virtually identical if the two objects are far from the viewer. Therefore, to correctly correspond to roughly uniform parallax offsets (the S shifts) between consecutive D-values: close (to die viewer) consecutive D- values may be separated by inches or even fractions of inches; consecutive mid-range values may be separated by several inches to feet; and, consecutive far-range values may be separated by many feet in 'scene space' . Uniformly spacing by equal 1/D values in the D direction is one way that may be used to advantage for simulating true 3-D perspective. In this way, a relatively small number of possible D values may be used effectively.

So far, only planes parallel to the screen have been discussed. In addition, as described in US Patent number 4,925,294, certain stereographic elements will not be in planes parallel to the CRT faceplate. They may be tilted planes or 3-D 'sculpted' elements. In tiiat case tiiere may be several planar elements witii most of the D-values reserved for non-parallel or non-planar elements. As an example, if three depth values were reserved for three background planar elements, and three more for three for foreground planar element, that would still leave about 250 values for tilted-planar or sculpted elements.

Another embodiment provides additional flexibility. Assume that a D-buffer containing 8 bits per pixel is available, although the same basic idea could be used for other numbers of bits per pixel. The 8 bits will be split into two segments: the upper or most significant 3 bits (providing 8 possible numbers 0 through 7) will be reserved as a 'tag'; the lower or least significant 5 bits (providing 32 possible numbers 0 through 31) will be used for a 'value'. In this example, tag 0 will be reserved for planar objects parallel to the screen. For planar elements the value will represent the depth of that plane and can be used, as is, for obscuration purposes. However, for flexibility and added precision and, in particular, for interaction with otiier non-parallel-planar elements, die actual depth value used to create parallax may be stored in a look-up table. Each of die 32 entries in that table would be indexed to one of the 32 possible values for planar entries as described above. The depth assigned to a particular pixel would thus be derived by using the value stored at tiiat pixel as an index into the deptii table. Thus, all the pixels associated with a planar element could be moved by changing a table entry rather than by changing die value at each pixel. More than one planar element could tims be assigned the same depth. If overlap occurs, then an ambiguous situation results. The tag numbers could be used to arbitrate the conflict. Alternately a second table (an O-look-up table) could be constructed to store ordering or obscuring information to be used all the time, or only in conflict situations. In general two separate 'depth' designations can be maintained: one, a 'distance' value would be used to determine parallax offsets; the other, an 'ordering' or 'priority' value would be used to determine which objects obscure other objects.

Tags 1 through 7 would each be reserved for a tilted or 'thick' image element. In tiiese cases, die 32 possible values could be interpreted differently than as described above, with several alternatives: 1. The 32 values associated witii a particular tag could themselves be interpreted as actual depth values.

2. The 32 values associated widi a particular tag could themselves index a table of actual depdi values (as described above), providing an image element whose individual pixels were distributed at up to 32 depths. A second order table could also be used.

3. The 32 values could index a table of relative depth values, providing (probably fairly small) deptii offsets from a positioning depth value also associated witii die tag. The deptii value for each pixel of the image element is calculated by adding die offset table entry associated witii die value for that pixel, from the table associated witii the pixel's tag, to the deptii position also associated with that tag. In this way the element, once its 'deptii shape' at each pixel were defined, could be repositioned just by changing the one position depth rather than by changing the entire table or, worse, the depth (tag + value) stored at each pixel of the element. The position could be a single value representing the back, front or middle of the element. It may also consist of several numbers representing, for example, front and back, or position and thickness. In this way the element could be bodi positioned and stretched in the depth direction by changing only two values.

Otiier variables associated witii a tag could indicate how die table values were to be interpreted: as actual values, table indexes, or relative offsets (1, 2 and 3 above); positive, negative or signed; linear, 1/linear, exponential or other non-linear function; etc. The interpretation of the values may also change as die position brings the element closer to or further from the viewer. That, in mm, may depend on die focal length of a virtual camera or an actual camera used to photograph a flat image to which depth is to be added.

Image elements that are planar, but tilted with respect to the face of the display, may be handled as thick or sculpted elements as described above. However, particularly for tilted planes that extend over a wide range of depth values, die limited number of possible values may produce a series of strips parallel to the face plate, rather than a smooth plane. Such planes may be marked as such witii their tag and handled as a special case. By storing positions for the edges or comers of a rectangle, and applying the parallax rules to those positions, a pair of continuous smooth image elements may be created for the left- and right-eye views by interpolating between those edges or comers. Because of the simple geometry associated with tilted planes, a precise depth for each pixel may be easily calculated for the purposes of proper obscuring with other image elements stored with D-values at each pixel. By using a matte (perhaps stored in the now unused value portion, or stored as a separate raster image element), or by designating one 'transparent color', the image element associated with the tilted plane need not be rectangular, but may be given an arbitrary outline. Those pixels on the plane, but outside the matte, would thus be ignored for both display and obscuring.

Video displays in most computers, and die data files meant to be displayed tiiereupon, are often in the range of 512 pixels horizontally by 486 lines vertically. Other often used horizontal resolutions are 640, 768 (or 756, with six null, black or transparent pixels on each side) or even 1024 or 1280. Straightforward embodiments use offsets on pixel boundaries and, therefore, the minimum horizontal parallax shift between planes adjacent in the depth direction is one pixel (usually two pixels, one each in opposite directions for the left and right images) and, particularly at the lower end of these horizontal resolution figures, this does not permit separate planes or parts of objects to be closely spaced.

The current embodiment of the system utilizes the AT-VISTA display board which can be configured at many different resolutions, including a relatively high 1008 pixels horizontally (which may be increased to a nomrnal 1536). When importing images painted, rendered or scanned at more usual resolutions, data pixels are repeatedly loaded into the higher resolution display's memory locations. For example, with 512-pixel resolution images, into a 1536 pixel wide screen, each pixel would be displayed three times; and for 768-pixel resolution images each pixel is displayed twice. With 1024-pixel resolution images each data pixel is used for 3/2 display pixels, which is implemented by loading data pixel N in display pixel M, data pixel N+ 1 in display pixel M+2, and a 50/50 mix of data pixels N and N+ 1 in display pixel M+ 1. A similar interpolative scheme is used to load 640-pixel resolution data into the 1536-pixel resolution display. Each group of 5 data pixels (A dirough E) must be apportioned into 12 display pixels (1 through 12) as follows:

1 = A 2 = A 3 = 0.4A+0.6B

4 = B 5 = 0.8B+0.2C 6 = C

7 = C 8 = 0.2C+0.8D 9 = D

10 = 0.6D+0.4E 11 = E 12 = E

Similar interpolative schemes can be used to covert between other resolutions. It should also be noted that when data witii relatively wide pixels is to be displayed in a frame buffer of relatively narrow pixels, it may be parallax offset by multiples of the narrower display pixel, rather than the wider data pixel, to achieve finer depth effects.

The reverse process can be used to derive interleaved or separate left and right images from the higher resolution 3-D display. For example three pixels of die 1536-pixel resolution image would be combined and averaged to a single storage pixel of a 512-pixel resolution data file.

The contradictory needs for high-resolution (many small pixels) to achieve small parallax offsets, and low- resolution (fewer larger pixels) for economy of storage and display, may both be addressed by a frame store where pixels of different sizes may be displayed at different parts of the same image at the same time. Such a frame store is described in applicant's co-pending US patent application number 07/077,916, of which this application is a continuation-in-part.

Information may be loaded into a display, either from a pre-existing image or from the 'brush' of a paint system; it may be loaded from information at the same resolution as displayed or (as described above) from a different resolution tiian displayed. In any of these cases the straight-forward approach leads to a miiumum parallax offset between planes of one pixel which may provide widely spaced planes or parts of objects in synthesized 3-D space.

However, an interpolative technique, similar to that used to display information stored at different resolutions, may be used to provide sub-pixel positioning and, thereby, very fine and accurate parallax shifts which result in fine and accurate positioning in the depth direction.

As an example here, consider display and data of the same resolution; an existing display line segment consisting of 9 pixels (a) through (i); and a new data display segment consisting of 5 pixels (A) through (E). Table 1 shows how the new information is combined with the old information for five cases: case 1 , with the new information obscuring pixels (c) dirough (g) of the old information but with no parallax offset (new information on the same plane as the old); cases 2 and 3, with additional left and right parallax shifts of one pixel; or, cases 4 and 5, with additional left and right shifts of 1/lOth pixel. Thus, parallax offsets, and tiiereby depth plane spacing, of 1/10 (or any fraction) of that normally achievable by display hardware of a particular resolution may be implemented by such a software technique.

TABLE 1

POSITION CONTENTS CONTENTS CONTENTS CONTENTS CONTENTS

LETTER CASE 1 CASE 2 CASE 3 CASE 4 CASE 5

a a a a a a b b A b . lA + .9b b c A B c .9A + .1B .9A+. lc d B C A .9B+ .1C .1A+ .9B e C D B .9C + .1D .1B + .9C f D E C .9D+.1E .1C + .9D g E g D .9E+. lg .1D + .9E h h h E h . lE+.9h i i i i i i

However, two problems result from such sub-pixel positioning techniques. First, it is not clear what depth should be stored witii a pixel whose display contents consists of a combination of two pixels at different depths.

Second, it is complicated, if not impossible, to correctly 'un-do' , erase, or further process pixels at the edges of such objects whose display contents consists of combinations of more man one object. This leads to the next embodiment of the instant invention.

In order to avoid the above mentioned and similar problems, each separately painted plane, sub-plane, or other non-planer object, may be maintained as a separate object in separate buffers or sections of memory, other than the display's own memory. In this way, objects may then be properly parallax offset shifted, and left and right composite images built up in the portion of memory used by the display. With this scheme (as opposed to keeping only the composite image, only in display memory) objects may be moved in X, Y or Z without leaving 'holes' as objects reveal what is behind tiiemselves. Additionally, when sub-pixel shifts, or other techniques such as 'anti- aliasing' , result in display pixels tiiat are composites of data pixels from more than one object, there is no problem due to not being unable to disentangle the composite information. Since the entire display image is reconstructed from the separate objects, the appropriate display pixel compositing is just re-computed from scratch.

An additional advantage is that objects may be made transparent or painted transparently; or, because objects have deptii associated with them, a light source(s) may be specified and correctly calculated shadows may be cast in 3-D-sρace by some or all objects, to add realism to the final image. If either of these processes is done when just a composite image is stored, it is difficult, if not impossible, to undo transparency or shadow casting. Shadow casting would probably be implemented as a post-painting process. However, if the objects are stored separately, as described above, not only can tiiey be moved and re-composited, but they can be re-composited with transparency and shadows.

Off-die-shelf high-speed image processing sub-systems such as the Intel DVI board for the IBM-PC are available. Such devices can achieve real-time or near-real-time compositing of many separate and separately offset image elements into a unified display image, or left and right pair of images.

Alternately, instead of software compositing die final image in display memory, it is also possible that each object may be kept in its own display buffer, witii the composite image created only on die fly by properly offsetting and mixing the output signals from the multiple display buffer sections.

An additional advantage associated with this technique is that for many situations only a single copy of each object must be stored (with depth specification) and may be shifted in opposite directions prior to compositing for each of the left and right images. If alternate lines of an object/image are to be shifted in opposite directions for interleaved display, fine vertical detail may broken up into the two eye views, which may interfere with perception.

Low-pass filtering in the vertical direction is one way to lessen such effects. Alternately, each source line may be used in each eye, although this will stretch the image vertically.

In practice, most of the area of the left and right images composited from many objects will not be changing during die painting process; only those pixels under the brash or operational area will be effected. However, when an object is moved in any direction, much or all of the images will have to be recomposited. To organize such operation, the human operator may be presented widi two portions of an interactive program: one geared to painting or creating objects; and, another geared toward re-arranging objects and fine-tuning the compositing process.

Fine-tuning the compositing process may include various options as to how object boundaries are handled. Particularly when using sub-pixel parallax shifts, pixels at the boundaries of objects may contain data from more than one object, with more than one deptii. Visibly, anti-aliasing or mixing of the visible data from multiple objects, in a pro rαtα fashion, may be used at die edges. Alternately, die visible information from the object that is closest, or the object that covers most of the pixel may be used for die entire pixel. An additional problem is to determine what depth value is to be used and stored (for offset or obscuring purposes) with such composite pixels. Again, the deptii value of a dominant pixel may be used, or the depths from several component pixels may be used. If a deptii value is to be derived from a composite, simple or weighted averaging of the several component deptii values may be used.

Alternately, at tiiese edges where the proper deptii value is less certain, a technique may be adapted from applicants co-pending application number 07/435,487, where color information was made less saturated or 'pure' as the edges of objects were approached. Iπ this case the depth information will be made less 'pure' or 'fuzzier' by adding a random variation or noise to those pixels at or near the boundary of two objects separated in depth. In general, the closer to the edge, the less pure the depth will be. In practice, this will amount to picking deptiis for edge pixels tiiat are randomly scattered between the near and far depths of: the component pixels, for anti-aliasing techniques or micro-positioning; or the bordering objects for other situations.

A PREFERRED EMBODIMENT A '3-D Computer Paint System' has been programmed in the computer language 'C by the applicant as a preferred embodiment of the instant invention. The program operates on the AT- VISTA display card installed in an IBM-PC/AT type computer.

The display is configured for 1008 (nominally 1024, and may be increased to a nominal 1536) horizontal by 486 vertical resolution, permitting a fine degree of horizontal parallax shifting (and, therefore, depth spacing) but configured for display on and recording by standard broadcast video equipment; as well as virtually all available electronic LCD-shuttered 3-D glasses.

As explained more fully above, the 'row table' feature of the AT- VISTA can be re-configured for five display modes: left/right in odd/even video fields, for 60Hz operation; right/left in odd even video fields, for 60Hz operation; left/right in over/under full frame, witii a second blanking interval, for 120Hz operation; right/left in over/under full frame, with a second blanking interval, for 120Hz operation; left image on top, right image on bottom, for glassesless viewing of both images separately (a debugging mode).

Facilities are provided for importing/exporting both 2-D and 3-D images, full screen or partial windows, at various resolutions (512, 640, 756/768, 1008/1024, 1512/1536) with proper aspect ratio conversion. In order to simplify programming and operation, both the 'canvas' upon which the image is painted and the various brushes consist of continuous left and right parts; the interleaved display, when used, is only a function of the 'row table' feature. Each application of the brush, text or other function is thus carried out twice, once for each image, with the appropriate parallax offsets added to whatever normal 2-D positioning data is obtained from the mouse, data tablet or keyboard. This paint system is supplied witii an unusually large number of user adjustable parameters and options, and virtually all such options may be separately adjusted for the left and right canvases. Both field and frame cursor movement, as described above, are available.

At each pixel, eight bits each are stored for Red, Green and Blue visual information. The fourth eight-bit channel is used as follows. Seven bits are used for 'Z' or deptii information. Of the 128 values thus possible, 101 are used: 50 into the screen; 50 out of the screen, into room space; and, die plane of the screen (no parallax offset).

If the paint system is left with the plane of the screen selected, it may thus be used as a standard — but, very flexible — 2-D paint system. The eighth bit of the fourth channel is reserved as a protection bit, whose use will be described below. The Red, Green, Blue, Depth and Protection data may each separately (and separately for each of the left and right canvases) be selected for writing to the frame buffers memory, or masked from writing to memory. In this embodiment, where input comes from a 2-D data tablet, the user selects a single plane at a time upon which painting is to occur, or to which a flat or multi-depth function is to be referenced. Alternately, when connected to a 3-D input device, the selected plane may also be dynamically and continuously selected along with the X and Y position.

The 3-D paint system has many usual elements, menus of buttons, color and brash palettes, sub-menus, etc. but these are displayed and work in a uniquely 3-D way.

All buttons and palettes float upon a menu plane which, in mm, floats above the canvas; this effect is achieved by relative parallax offset. Similarly the text of the button floats above the plane of the button. In addition, the buttons cast drop shadows in 3-D. Normally drop shadows are dark areas the same shape as the casting object offset down and right (or up, or left) and obscured by the casting object. In this case, in addition to the drop shadow offset, parallax offsets are added to the left and right shadows to make them appear behind die casting object.

Similarly, elements cast shadows onto palettes, palettes cast shadows onto the menu plane, etc. Whenever selected or 'hit', buttons and other elements animate by depressing and returning as appropriate by altering the parallax shift of the selected element, through a number of positions, over several frame-times.

For certain functions windows open into the menu, and otiier user interface elements or messages are revealed. One such window is the 'ruler' a device with a set of tics for each plane that indicate the depth value of the selected plane (with progressively longer tics at intervals of 5, 10 and 50), whether that plane has been previously selected for painting, or whether that plane is currently set as 'protected' or masked from painting.

It should be noted diat the Apple Macintosh computer, the IBM Windows operating system and other systems provide so-called Graphic User's Interfaces (GUIs) which incorporate menus, buttons and stacked windows on a 'desktop'. The desktop often gets crowded and confusing as many 2-D elements partially obscure each other.

It is potentially useful to use the stereoscopic display techniques described herein to make such GUIs into 3-D user interfaces to assist users navigating through such crowded interfaces.

All options and features may be selected for the right channel, left channel, or both; and may be copied right-to-left and left-to-right. Different colors and brush halves are displayed for each eye as appropriate.

The cursor is positioned in Z as well as X and Y by the use of parallax offsets. In addition, flat rectangular cursors used to delineate areas for processing, storing, editing, etc. have been generalized to 'bricks' which delineate a rectangular solid space in the 3-D canvass.

Various methods are available to protect particular pixels from being effected by die painting process. Each of these various types may be selectively enabled or disabled for each of the two images.

As stated above, die eighth bit of the deptii channel is reserved as a pixel-by-pixel protection bit. If this option at painting is selected during a painting function, the protection bit is turned on for each pixel changed. A complementary function prevents such protection tagged pixels from being painted over.

By selecting botii die rning on of the protection bit by painting, and the use of the protection bit to prevent painting over, at the same time, a very useful function results. If one has selected a partial tinting pamt function

(e.g., tint all pixels hit by the brash 10% green) then repeatedly going over the same pixel will tint it more and more heavily. This may in fact be what is desired. But, if it is desired to uniformly tint an area to the same degree, each pixel should be effected only once; this is very hard to do free-hand. This technique guarantees that each pixel is effected no more than once. Using more bits for a 'protection plane' provide the ability to count the number of times a pixel has been painted or 'hit' and tiius more functional flexibility. For example, such a protection counter could be used by die program to permit a pixel to be tinted heavily on its first hit, and progressively less heavily tinted on subsequent hits. Or to only perform a function after some threshold of hits has been reached.

By turning on the protection bit when painting, but masking the writing of Red, Green, Blue and Depth values, the operator may paint an area of 'protection from further painting' (Fixing) without effecting the contents on the screen or depth values. Areas may be similarly 'unfixed' by painting with the protection bit turned off and, again, masking Red, Green, Blue and Depdi.

Similarly, if Red, Green and Blue are masked off, but Deptii is not, an 'invisible sheet of glass' at a particular deptii may be created which will protect everything behind it from being painted by a brush behind (further than) the plane of the 'glass' but which may be painted over by a brush closer than the plane of the glass. Otiier types of protection are provided. Each plane may be explicitly protected by selecting it as such on the ruler. Facilities are provided to turn on or off specific values or ranges and complex combinations may be built up this way. Then pixels are painted, or not, based on their deptii value rather than on the state of their own protection bit (although both methods may be used simultaneously). Pixels may also be protected because their display (Red, Green, Blue) contents are a specific value or in a specific range of color values. Also, options are provided to restrict application of the painting function to: all depth values, one depth value, a range of depth values, all but one deptii value, all but a range of deptii values.

Lastly, when painting at a particular plane, it may be separately specified whether painting will or will not effect pixels that are: closer, further, or equal to die plane being painted. The usual mode is tiiat pixels that are closer will not be affected. These last two restrictions based on deptii may be used in tandem. But, for some types of functions the first is applied prior to calculation (culling the points considered for die calculation of a digital filter, over a 3-D kernel, for example), and the second after calculation (culling which points are to be updated on the screen).

In this paint system brushes may be any shape within a 16 x 16 pixel rectangle. However, like the canvas itself, each brush is 'disinterleaved into a right and left half, each only 8 pixels high by 16 pixels wide (128 pixels in all, for each half). The brush itself may have a single bit, or 8, 24 or 32 bits of information stored at each pixel.

One may chose to use the two halves of the brash as a left/right pair or have either half repeated for both halves of the canvas. Each brush half may also be independently med on or off for painting.

The paint system has many different painting modes, some of which require complex calculation at each pixel effected. Therefore, in order to keep the operation of the program running as fast as possible, unnecessary calculations must be avoided.

Part of painting is deteπnining which pixels are excluded by die shape of the brash within its rectangle, by the various protection schemes, by the restriction of depth values, restriction of color values, or by the requirements of the particular painting mode (for example, luminance paint only paints if the pixel to be painted is brighter than the pixel contents already at diat position). Two schemes are described below.

First, an essentially one-bit version of the brush may be kept, which operates as a mask or picture of which pixels are to be effected. At the beginning of each application of the brash, the various protection schemes are checked and bits in die mask are mmed off if protection will eliminate that pixel from being effected. Only those pixels still represented in the reduced brash mask are then applied to die canvas. The problem with this scheme is that the various protection schemes and many elements of the paint amount to a decision being carried out (is the protection bit on, is the deptii value separately protected, etc.). While finding that a bit is mmed off in the one-bit image of the brash eliminates the need to proceed further with a particular test, the test of whether each of the 128 pixels (for each brash half) are on or off in the brush image must be repeatedly tested for each pass of protection or final painting as the brash is laid down once. One may test 128 pixels for each of M tests or one may perform M test for each of 128 pixels, the result is still 128 times M tests.

To get around mis problem the instant systems instead keeps each brush as a list of pixels. Each list of bytes starts with a counter that tells how many entries are in the list: between 0 (no pixels in the brash) and 128 (all pixels on). Then the list follows with that many entries listing the location (in raster order) of each pixel in the brush. In raster order, from left to right, top to bottom, the upper left comer is 0, the lower right comer 127. Since a rectangle of image is brought into or taken from the computer's memory by the AT- VISTA display as a linear array in raster order, this scheme works well.

Thus, if the brush contains very few pixels, the list is short and very few pixels must be considered. No time is wasted repeatedly considering and then rejecting as not valid die many 'off pixels in a brash image. As each protection scheme is applied, rejected pixels are removed from the list since, once rejected for one reason, it is no longer necessary to determine if a pixel is to be rejected for a second reason as well. The list is shortened by taking die last entry, placing it in the rejected pixels position, and decrementing the first list entry (the counter) by one. In this way, whether a pixel is 'on' or 'off never needs to be considered; off pixels are never on the list for consideration to begin with. The list can only get shorter as pixels are rejected by various tests.

The 3-D paint system has many different paint modes; some are similar to 2-D paint systems; others are unique to this program. Standard painting may be achieved by using a one-bit 'rubber stamp-like' brash with a color chosen from the palette. Alternately a brush may be a 'picture' with Red, Green and Blue (and even Transparency or Depth) values stored at each pixel. This bush may man be applied to die canvas in a straightforward manner, or my be used to tint (change hue but not luminance) or shade (change luminance but not hue) die canvas. The entire result may also be modulated by a transparency factor which keeps some proportion of the old pixel's display value. In addition to the otiier protection schemes described above, several versions of 'luminance' paint are available which limit painting to pixels whose new value will be above (or below) a threshold based on an average of the Red, Green and Blue values; a television luminance calculation; or separate comparisons for each of the Red, Green and Blue plane values. Other types of painting perform various mathematical calculations (e.g., logical AND, OR, XOR, and their complements, addition, subtraction, multiplication (witii and without clipping), etc.) on the current and brash pixels data and use the results to replace the old pixel (again in regular, tint, shade or transparent fashion).

Still other types of painting perform a calculation based only on the canvas pixels' values, based on some parameters. For example, several image processing functions can be attached to die tip of the brush to selectively adjust level, gain, contrast, saturation, color correction, etc. on a free-hand basis.

Otiier types look to a neighborhood around the affected pixel and perform various digital filters that may blur, soften, sharpen, etc. the image; or, may smear pixel values in the direction of the motion of the brush.

'Air' brushing is a technique that lays down a transparent, soft-edged swipe of color in two dimensions on the canvas. There is also 'Spray' painting that does so but with a thick curved tube (by use of deptii values and parallax offset of individual pixels) several planes thick; it has a dense center with a 'haze' around it. A similar tube¬ like mode witii a 'hot' center and softer 'glow' is dubbed 'neon' paint. In these 'thick' painting modes die depth values may be updated, creating a substantial object. Alternately, depth values may be left unchanged so tiiat no record of the spray's existence would be made, it would be 'insubstantial'. Additionally, the core may be made substantial with the 'glow' or 'haze' made insubstantial. In addition to arbitrary 16 by 16 pixel brashes. Larger computational brushes are available. A 'disk' is round witii uniform density and a sharp but anti-aliased edge; a 'cone' is round witii a dense center which falls off to transparent in a controlled fashion; a 'ball' is like a cone but, again, with depth values that give it thickness and roundness in the Z direction. Storing transparency values for individual brash pixels is used to implement the soft edges or glows described above. One last painting mode will be described; Shape paint painting is done on die surface of a 3-D shape already in die frame buffer, changing the surface look but not the depth shape. A color and brush are picked as usual. However, instead of picking a depth and painting at a particular parallax offset, when Shape paint mode is picked, pairs of pixels with the appropriate cursor Y value are compared. When two pixels (in the two images) are found whose stored depth values are die same and tiieir relative parallax offset consistent with that deptii value as applied to die non-parallax-adjusted X value of the cursor, that pixel pair has the right X, Y and 'virtual' Z to be painted.

'Solid' objects may be built up from contours at different depth planes (like a contour map) or may be rendered by otiier software and imported, or rendered by the 3-D paint system. In addition, libraries of precomputed shapes may be made available to die operator, much like 2-D clip-art. In any of those cases, shape paint will permit the surface of those objects to be re-painted without disturbing their shape. The ability to create 3-D objects by stacking painted contours, as well as to import and combine 3-D objects from other sources, and to 'Shape paint' provide a flexible mechamsm for a painter or illustrator to create 3-D objects and environments. One particularly useful application for this system will be the creation of such objects and environments for inclusion in Virtual Reality simulation systems. These 'illustrations with depth' can be converted by software utilities to the appropriate file formats for such VR rendering systems. Although the depth planes are noticeably separated, lines and other primitives (rectangles, polygons, etc.) may be drawn to smoothly span the distance between planes. A 3-D line or polygon may be drawn by adding appropriate parallax offsets to the end-points or vertices of the object (different for the left and right image) and using existing or well known algorithms for drawing die adjusted image twice: once for the left, once for the right. A problem results in that while the image may appear to be smoothly spanning a 3-D space, the depdi values stored for those visible pixels will be quantized or 'stairstepped'. This simation may be kept in mind as a limitation of the system or, alternately, the deptii values may be 'dithered'. Dithering randomly selects one value from the closest available quantized depdi values above and below die desired deptii value. The probability of selecting one or the other is adjusted depending upon how close each available value is to the desired value. In this way a 'fuzzy' rather than stairstepped depdi image is created and when further painting or other image functions interact with the object, a fuzzy but smooth looking intersection with the new object results.

If 16-bits are available for depdi buffering, the entire issue becomes moot because enough resolution in the depth direction is available for smooth deptii specification without ditiiering.

ADDITIONAL FEATURES. TECHNIQUES AND EMBODIMENTS

It must be understood diat the parallax shifts associated with the left and right eye views, in effect, change the point of view (POV) of the viewer of the multi-plane image. By adding coordinated, rather than opposite, shifts, the POV of the two-eyed viewer may be shifted. Several useful variations result.

First, even without glasses or any kind of binocular stereoscopy, the POV may be shifted in 'wiggling' motion to produce 'motion (serial) parallax' which can convey much depdi information. Although the perceived wiggling of the image can be annoying, and therefore is of limited use for long entertainment presentations. For short term 'bringing to life' of stacked multi-plane images, such as those created by die instant system, this method can be very useful to convey 3-D information without any glasses. For, scientific, industrial, military, educational or medical imaging, 3-D visualization of data can be achieved without glasses. The applicant has found that five such images — with no parallax offset (N), slight offsets to the right and left (r and 1), and greater offsets to the right and left (R and L) — a strong 3-D effect can be achieved by alternating them in die following order: L, 1, N, r, R, r, N, 1, L, 1, N, r, R, r, N, 1, etc.

As explained briefly above, the applicant has used die Intel DVI board to composite images in real- or near- real-time utilizing the multi-plane techniques described herein. Such a system can be programmed to update the POV, or parallax shifts between planes, to implement the above. Additionally, when coupled to input from a head- position-tracking device, the POV can be updated continuously to create a display from the appropriate perspective as the viewer moves around die room; this is a very powerful 3-D (though not binocular) effect. The combination of 3-D glasses and the head-tracker (or a head-mounted display) produce an even more powerful effect. In that case, not only head position but orientation may be captured and each eye may be displayed witii proper horizontal and vertical parallax offsets, since head orientation may tilt the eyes.

Such a system is a low-end alternative to costly full-immersion helmet-based Virtual Reality displays. When using such a system and drastically changing the viewer's POV (particularly when the multi-plane image is derived from a 2-D image or film/video sequence as described in applicant's patent number 4,925,294) holes in the image may develop, and the reverse sides of objects may need to be displayed. It is just such as system as described herein that will be used to fill tiiose holes or 'paint' die reverse sides of objects so that not only will 2-D images and motion sequences be able to be adapted to 3-D stereoscopic displays, but may also be adapted for use with Virtual Reality systems.

Systems have been disclosed, such as that by NYIT, where two images may be combined in such a way that by proper frequency filtering either one may be displayed. This has been disclosed as intended to be utilized to transmit two entirely independent image streams. However, it is also possible to transmit die left image as one image stream and die right image as die otiier stream. Alternately, the single center image stream may be sent and a second, perhaps lower-quality, image stream (by the above or any other method of hidden information) may contain depdi information only. Then, as described above, die deptii image may be used to distort the single image, once to the left, once to the right, to create stereoscopic paired image streams. When distorting such a single image into left and right images, (or as described elsewhere in the instant application) holes may develop as shown below with B to the rear of A and C.

BBBB Original AAAAA CCCCC

BBBB BBBB

Right AAAAA** CCCCC Left AAAAA **CCCCC

Such holes (**) may be automatically covered by stretching the available material at either end, or by interpolating the material on either side of the holes. In one approach the holes may be filled with values derived from interpolating from A to C. Alternately, the hole in the right image may be covered with material interpolated between A and B; and, die hole in the left image may be covered witii material interpolated between B and C.

Also, frame-rate-doubling televisions are now commercially available. These take the 60 fields of a standard television signal and by the use of a frame store, show each twice. Such a framestore device may be easily modified to solve the flicker problem associated witii 3-D electronic LCD-shuttered glasses which deliver only 30 fields to each eye. Similarly, because die odd lines go the one eye, and the even lines to die odier, vertical resolution to each eye is halved. A television witii a frame store device would be able to fill in tiiose gaps during display. Application of the applicant's variable Spatial-Temporal Signature (STS) to this technique will prove useful. See applicant's serial number 07/077,916.

Additionally, a new type of low-cost electronic stereo glasses may be used where small lights (or LEDs) are flickered, alternating with each eye in synchrony to the alternating left/right pattern. The light for one eye is on while the image for the other eye is displayed. Altiiough not bright enough to completely disturb vision, die small flash in one eye will suppress the perception of the unwanted image in diat eye. The light may be directly flashed into the visual field or, preferably, through a diffusion element. Alternately, half-reflective lenses may be used in diese glasses which will reflect the internal light flash but otherwise pass the television image to the eye. There will also have to be an opaque barrier to prevent cross-talk between the two lights and two eyes.

Composite color systems display, poorly, red/green anaglyphic stereographic images due to the encoding/decoding of color information at reduced bandwidth. Due to die nature of the raster pattern this blurring, smearing, or lack of resolution is a problem in the horizontal direction but not die vertical.

What must be eliminated in order to overcome this limitation inherent to composite color systems is the variation in the hue of the color signal rather than the luminance content of the color signal. It is, thus, possible to send a high-definition monochrome signal which might just as easily be monochromatically green or red as 'black & white'.

In its simplest embodiment, dierefore, red fields would be alternated with green fields in order to provide thirty fields per second (e.g. , for NTSC) to each eye separately. However, this reduces die flicker rate for each eye below the perceivable threshold, as witii 60 Hz field sequential LCD shuttered glasses.

However, since separation is done passively, by color, rather than actively, with an LCD shutter, the present system can increase the 'switching rate' as follows. For each field red lines and green lines will be alternated. Due to die raster pattern adjacent vertical information is widely spaced in time and therefore changing from monochromatic red to monochromatic green between lines does not effect die detail information. The switching rate is, thus, increased from 60 Hz to 15.75 KHz, far too fast for current LCD shutter technology. (Of course, once high-speed LCD shutter or otiier active technology is developed, all of the techniques herein may be applied to full color actively switched video signals. This being in contrast to current efforts which are toward increasing the field rate radier tiian parsing the field into relatively small left and right portions.) What this will do is reduce die effective vertical resolution to each eye to approximately 120 visible lines per field or 240 per frame. However, this is expected to be much less annoying tiian the flicker. The reduced vertical resolution anomaly may be lessened in two ways.

First, the choice of which lines are red and which lines are green may be varied field to field, frame to frame, or on a longer basis cycle. Second, lines containing both red and green may be inserted into the sequence to fill in 'holes' that would otherwise develop in one eye or the other.

Additionally, segments of less t an one line, but sufficiently long to avoid major degradation due to composite encoding/decoding may also be used and would contain monochromatic red or monochromatic green information, possibly with the addition of segments containing both red and green information.

The distribution over the frame of red and green (and optionally red and green combined) segments could be varied in a non-trivial manner according to instant inventor's co-pending bandwidth compression patent application number 07/077,916. Further, a method exists for transmitting full-color television to be used witii anaglyphic glasses whereby

'red green fringes' or some other red/green image information is incorporated into die color signal for deptii perception purposes. All of the above techniques may be similarly applied to such signals as well.

Further, instant inventor suggests an alternative whereby monochromatic red, monochromatic green and full color segments of one line or less are alternated, with all the variations described above, for the reception of both color and 3-D information on a composite monitor.

To achieve color perception the red and green images can be derived as die red and green components of a full color image. These may be used instead or in combination with red and green tinted monochrome signals. Further, die blue color information may be mixed in to either the red or green side or both. In general the red and green color components, red and green tinted monochrome components, a blue component, and full color signals may be mixed to achieve an appropriate "look" for each scene of a motion picture sequence, much as standard

"color correction" is done.

It is also noted tiiat some anaglyphic glasses use red and blue rather than red and green. The equivalent process can be described by "swapping" die words green and blue in the discussion herein. Further, anaglyphic glasses of other color combinations, in particular tuned to the NTSC I or Q hues, can be employed with those colors substituted for red and green.

Generally, left and right image pairs are alternated as video fields or frames. However, interleaving of smaller image sections can be used to advantage for improved perception. Individual lines or sub-lines can be alternated from die left and right images. Faster flickering electronic LCD-glasses, than currently available on a low- cost basis, would be required to decode such an image. However, if using passive red/green anaglyphic glasses, flickering is not an issue.

The problem with using red/green anaglyphic images with television is mat the color encoding process lowers the resolution of the color portion of the signal. However, if whole individual lines (or large line parts) of the image were uniformly tinted red (for left) or green (for right) these large sections would be displayed at high resolution because only their luminance portion would be changing, not the color portion of the signal. Thus, high- resolution anaglyphic images could be broadcast or taken from videotape.

For either of the techniques described above, die way diat die left/right information is distributed over the frame may be changed from frame to frame in a non-trivial manner, as a way to improve perception. This technique, called a variable STS, is described in some detail in applicant's co-pending application number 07/077,916.*** See Figure 5 for a basic high-level system diagram of the above. As described above, die 3-D paint system will operate at a continuously and dynamically selected depth witii the addition of a 3-D input device. Such a device is described in the following pages.

For inputting positional information to digital computers, mice (basically upside-down trackballs) operate witii a single roller-ball or optical sensor and give two axis inputs: X (left-to-right) and Y (away /toward die user). One, two or more buttons allow for selection type input as well. Flat X-Y data tablets provide similar functionality, with one on/off 'button' function often tied to the pen tip.

For many 'flat screen oriented' applications this is adequate. However, for otiier applications additional input axes are advantageous. Tablet and pen units exist where tip pressure (rather than just on/off contact) is measured and can be used for a third axis. However, such a solution is costly, delicate and takes up a large amount of desk space. Further, for many, especially for 'non-computer-artists' who are the vast majority of users, a mouse type device is preferable to a tablet and pen.

The present discussion relates to a class a mouse-like input devices diat employ multiple roller-balls (or optical sensors) to provide additional input axes and a versatile user interface.

If a second ball is added to the underside of a mouse, input on a third axis is possible — a twist about die Z axis; i.e., the axis perpendicular to the two axes of normal mouse movement. Such input could be used as a rotational parameter, for example, to point a screen cursor in a particular direction, not just position the cursor on the X-Y plane; or, for other rotational information, such as to rotate 3-D objects in a computer graphic or CAD system. Alternately, the input may be used as s magnitude parameter, for example to translate along the Z-axis, e.g., clockwise (CW) twist for up (with respect to the desktop, or out with respect to the screen) and counterclockwise (CCW) twist for down (or in). Or, it may be used for sign or selection information for a 'double button' like a two way switch or knob, e.g., CW twist for 'plus' or select, CCW twist for 'minus' or cancel.

The two-ball mouse comprises two trackballs mounted fore and aft to (of) the center point of the Y-axis. (Alternately, the two balls could be mounted to the left and right of the center of the X axis; but, this wide mouse would probably be harder to hold.) For normal X (side-to-side) and Y (up/down) operation, the two balls operate in tandem, giving (dieoretically, and at least approximately) equal output. For a twist, CW as seen from above, the rear ball will register a positive or rightward X output, die rear ball a negative or leftward X output. A CCW twist will produce complementary output.

The output of both balls can be sent on to the host computer for integration there, or they may be integrated by die circuit in die two-ball mouse for transmission of X, Y and Z data (along witii the button hit information as well).

The two-ball mouse's software driver will pass on die X, Y, Z and button-state information to applications programmed to properly interpret it. Alternately, in a standard mouse compatibility mode, only die standard X, Y and button information would be presented to die application.

In mouse-like manner, the two-ball mouse may be lifted, unwound and replaced on the pad for continued twisting. However, an alternate interpretation of the input by the software, either in the mouse or in the host computer device driver, would allow for a 'wind-up mode'. Like a mechanical watch stem, die CW (or CCW, perhaps chosen by whichever move started the sequence) twists would accumulate, die CCW would be ignored. Alternately, the CW and CCW twists could be summed for a magnitude, die sign being specified, e.g., by a 'button' or by which twist direction started the sequence. Which twist mode is in operation may be specified by the user via a button or a menu selection, or may be determined by the application depending on which software function were in operation. In particular, software may allow the user to mm off wind-up mode by making a normal X or Y axis motion; pausing longer than some threshold between twists; making repeated turns in the same direction, or further than the normal back-and-fortii wind-up twists; or by some other programmed gesture.

The advantage of such a wind-up mode is diat it allows the mouse (and more importantly the user's wrist) to twist back and forth, without lifting the mouse for die unwanted directional twists, and to accumulate a total rotation much further than the range of the human wrist would allow.

A similar mode could also be implemented to avoid having to go through the 'drag-lift-drag-lift-drag' sequence for long X and Y motions. However, due to ergonomics, this feamre is much more important for the much more restricted rotational movements.

In the twist mode described above, die axis of twist is through a point between the two balls, and the front and rear balls act in opposition. In an alternate 'pin and wag' mode, die axis of twist would pass dirough the center of either the front or rear trackball. In such a mode die pinned ball would see (theoretically, or at least approximately) no movement and the other ball would 'wag' left and right, along the X axis with a slight curve, for balls mounted front and rear. So, in addition to twist CW/CCW, one could pin the head ball and 'wag die tail' back and fourth, or pin the tail and wag die head back and fourth.

As a way to implement a variation on the 'pin and wag' mode, a 'rocker' pin or ball would be placed between the two trackballs. This pivot would extend somewhat beyond die two trackballs and would allow die user to pivot to have either the front or rear ball contact the mouse pad, without the other making contact. In that way, die contacted ball could be 'wagged' around the pivot, in a manner similar to above but with a smaller pivot radius.

But, for even more flexibility the pivot point would have a small roller-ball in its tip, much like a large-bore, inkless, ballpoint pen refill. As such, the rear (or front) ball and pivot could be moved left and right along the X axis, much like the wagging mode; or, back and fourth along the Y axis.

A spring load in die pivot point would allow the user to apply a small amount of additional pressure to have both the front and rear balls contact the mouse pad at the same time. In that case, the dual-ball X and Y motions, as well as the twist mode, would be available.

Thus, tiiere would be a total of at least seven axes with the dual trackball mouse, as opposed to only two (X and Y) witii a standard mouse. These are: X and Y pivoted for the front only, X and Y pivoted for rear only, X and Y with botii balls contacting and moving in a coordinated way and, lastly, twisting with both balls contacting in an anti-coordinated way.

In addition to increasing the number of input axes from 2 to 7, a dual trackball mouse would permit die new gestures to be added to the X-Y positioning and button clicking of the standard mouse. As on example, a 'wrist flick' could be used to implement a number of useful functions such as a 'page flip' to browse through a text file as if it were a book. A flick consisting of a short CW motion followed by a return CCW motion would constitute a jump ahead one page command; a CCW-CW combination, a mm back one page. A(n adjustable) threshold of, say, about 5 degrees would prevent accidental page turning from minor jiggling of the mouse. A larger flip would mm 2, 5 or 10 pages (again, perhaps user selectable) and a very large flip would mm 50 or 100 pages. The degree of 'largeness' would be increased by: increasing the CW/CCW rotation to 10, 15 or more degrees; increasing the speed of die flip; increasing the time between the CW and CCW phases of the flip, or between two successive flips; or, to make a number of flips in quick succession with 1, 2 or 3 flips indicating 1, 10 or 100 pages, for example.

Many other gestures and axis combinations can be developed with such a device and the proper software. For example, in a computer painting or drawing program, calligraphic writing may be implemented by drawing in X and Y as usual and, in addition, controlling the angle or tilt of die 'pen' by twisting the mouse. Similarly, for 3-D modeling or sculpting, an oriented chisel-like function may be implemented.

Or, using the two trackballs independently, witii the rocker, two separate objects may be controlled simultaneously without having to switch back and forth between functions, using a button click. By pressing harder, and having botii balls contact the mouse pad, die two objects may also be moved togedier in a coordinated manner.

More than two roller-balls may be used for additional input axes and increased system flexibility. Many configurations are possible and witiiin the scope of the present invention but, in particular, four- and five-ball configurations will be discussed blow.

Consider a simation where four balls are placed at the four compass points (East, equal 0 degrees, is to die direct right). All balls would act in unison and would thus provide only X, Y and twist axes; no improvement over two roller-balls. However, if an extended, rolling pivot were provided in between the four roller-balls, each of the four balls could be made to contact the mouse pad independently, by tilting the whole mouse to the right, left, away or toward die user. Each of the four balls would provide input along the X and Y axes, for a total of eight axes.

However, now consider the situation where the central pivot ball were also an active input roller-ball. When used with each of the four comer balls, the central ball would provide a pair of balls and this be able to provide a twist axis as well as X and Y, for a total of 12 axes. Additionally (perhaps, depending upon the amount that the central ball is extended and the dexterity of die user) by balancing carefully, so that none of die comer balls touched the pad, the central ball alone could provide an additional set of X and Y axes, for a total of 14.

Such an embodiment may be particularly useful for systems used to design and produce 3-D computer animation. In such systems it is often required to specify (at least for key frames) the X, Y and Z positions as well as rotations around the X, Y and Z axes, for both objects and the theoretical 'camera' . These four groups of 3 parameters may each be assigned to the four comer roller-balls, each with 3 axes. Thus, without having to switch between modes, all of the required positioning information may be controlled gesturally. The same comer roller-balls may be used to enter other information in otiier modes; mode-switching would be achieved with the active center ball, or otherwise. It should be noted that for optical mice, (where an optical sensor counts a number of lines passed over on the mouse pad, instead of a roller ball physically rolling over the pad) a small contact switch would be mounted for each sensor to sense which of the four 'balls' were being tilted toward for selection. Or, some odier set of orientation sensing devices, for example, mercury switches, would be used.

An additional variation provides extra stability to the embodiment described above. Consider the simation where the East or right roller-ball is chosen, by pivoting the entire assembly, on the central ball, to the right. In that orientation, the device is not entirely stable; it may, additionally, be tilted North or South, or balanced in between.

When tilted North and East botii balls touch the pad (as well as die central ball) and a stable configuration is achieved; similarly, for South and East.

Thus, if (instead of North, South, East and West) die NE, NW, SE, and SW directions were used, four stable configurations would be achieved. Although each of the four comer balls would be used twice, each of the pairs is unique and, therefore, no confusion would exist. An added benefit of this embodiment is that, since two corner balls are used in each mode, X, Y, and twist axes are accessible for each of the NE, NW, SE and SW comers, for a total of twelve axes without having an active center roller-ball; 14 axes witii an active center.

In practice, the entire design may be rotated 45 degrees, so that the active comer roller-balls would be located at die NE, NW, SE and SW comers, with stable tilt configurations at N, S, E and W.

Of course, with a dexterous user, all eight directions (N, NE, E, SE, S, SW, W, NW) may be used for a total of 20 axes without an active central ball; 26 with.

In addition to die specific configurations, uses and gestures described above, with additional input axes available, the possibilities for games, 3-D animation, computer aided design, machine or robot control, or otiier user interfaces are greatly increased.

Finally, all of the principles and techniques described above can also be applied to multiple transducers on other computer input devices. For example, two (or more) pen/stylus or puck/cursor receptors may be put in the same housing and queried (in parallel or serially) to generate not only position information but orientation information as well; or, two trackballs may be mounted in such a way as to be covered by die palm and thus input direction information, not just X and Y positional information.

It will thus be seen that the objects set forth above, among those made apparent from the preceding description, are efficiently attained and certain changes may be made in carrying out the above method and in the construction set forth. For example, the substitution or combination of various technologies, including but not limited to: analog and digital circuitry; special purpose circuitry and general purpose circuitry running software; various

3D displays, such as tiiose employing passive or active glasses, lenticular screens, CRT displays, holographic film, virtual reality and serial motion parallax; trackballs, mice, styluses and data tablets, or other position or input sensors in coordinated use as described herein; and, various ways of storing image and deptii information. Accordingly, it is intended diat all matter contained in the above description or shown in the accompanying figures shall be interpreted as illustrative and not in a limiting sense.

While there has been shown and described what are considered to be preferred embodiments of the invention, it will, of course, be understood that various modifications and changes in form or detail could readily be made widiout departing from the spirit of the invention. It is, therefore, intended tiiat the invention be not limited to die exact form and detail herein shown and described, nor to anything less than the whole of the invention herein disclosed as hereinafter claimed.

I claim:

Claims

An improved method for creating image information by using a computer paint system, wherein painting functions may include any and all such functions available in standard computer paint and similar systems including but not limited to image loading, compositing, modification, processing, editing, as well as brush- oriented painting and pen-oriented drawing functions, wherein said improvement comprises the creation of three-dimensional image information, wherein said three-dimensional creation is effected by die maintenance of a plurality of image information memories and the application of at least one painting function to at least two of said image information memories and wherein the application of said at least one painting function in a first of said image information memories is offset with respect to the application of the same painting function in a second of said image information memories.

2. A method for integrating 2D and 3D image elements by utilizing the method of claim 1.

3. A method for modifying a 3D image by utilizing the method of claim 1.

4. A mediod as in claim 1 wherein said offset may be adjusted by a fraction of a pixel for at least one image information value by the application of interpolation to apportion said image information value to more than one image information memory location.

5. A method as in claim 1 wherein a painting function is truncated in such a manner that for progressively larger offsets an extreme edge of die area to which said painting function may be applied is progressively reduced in order to avoid die application of said painting function to the image intended to be displayed to one eye without it also being applied to the image intended to be displayed to the other eye.

6. A method as in claim 1 wherein said plurality of image information memories are each uniquely associated with at least one of a plurality of image information elements, and left and right images are created by applying parallax offsets to at least some of said plurality of image information elements, and wherein said parallax offsets are determined as a function of depth information associated with at least some of said image information elements.

7. A method as in claim 6 wherein said offsets may be adjusted by a fraction of a pixel for at least one image information memory value by the application of interpolation to apportion said image information value to more than one image information display pixel.

8. A method as in claim 1 wherein said plurality of image information memories comprises two, a left image information memory and a right image information memory, and said offset is binocular parallax offset.

9. A method as in claim 8 wherein said left and right image information memories are each maintained as separate contiguous areas of memory and three-dimensional combination is effected only upon display.

10. A mediod as in claim 8 wherein said left and right image information memories are arranged in an interleaved fashion within a single information memory.

11. A method as in claim 8 wherein an image element is loaded into botii left and right image information memories without offset, the contents of said image information memories are then parallax shifted with respect to each other, and subsequently an additional image element is loaded into both left and right image information memories without offset.

12. An improved method for creating image information by using a computer paint system, wherein painting functions may include any and all such functions available in standard computer paint and similar systems including but not limited to image loading, compositing, modification, processing, editing, as well as brash- oriented painting and pen-oriented drawing functions, wherein said improvement comprises the storage of depth information in addition to standard image information.

13. A method as in claim 12 wherein the obscuring of one image information element by another is based on said depth information for at least some of said plurality of image information elements.

14. A method as in claim 12 wherein said deptii information is referenced to determine if a selected painting function is to be applied to a selected area of image information.

15. A method as in claim 12 wherein an additional improvement comprises at least one function to paint with a brush that applies both color value to a color image memory plane and deptii value to a depth image memory plane.

16. A method as in claim 12 wherein said depth information is sub-divided into deptii tag and deptii value information and d e depth specification to be used is derived as a function of the deptii tag and deptii value.

17. A method as in claim 12 wherein an additional improvement comprises that for at least one image processing function the set of pixels input to the image processing function and die set of pixels effected by die output of the image processing function are dependent upon the deptii information associated witii those pixels.

18. A method as in claim 12 wherein an additional improvement comprises the inclusion in said created image of shadows cast by at least one first element onto at least one second element whereby the shadow casting function is calculated based upon said depth information.

19. A method as in claim 12 wherein an additional improvement comprises the ability to separately inhibit writing to various information planes which may include without limitation red, green, blue, color index, depth, obscuration and protection.

20. A method as in claim 19 wherein an additional improvement comprises at least one function, such as 'painting a sheet of glass', that updates the depth image memory value without updating the color image memory value.

21. A method as in claim 19 wherein an additional improvement comprises at least one function, such as shape paint, that updates die color image memory value without updating the deptii image memory value.

2 A method for creating virtual reality databases whereby at least some three-dimensional image is created by die mediod of claim 12, and virtual reality shape information is derived from the deptii information of said tiiree-dimensional image.

23. A method as in claim 22 whereby, in addition, virtual reality texmre information is derived from the color information of said tiiree-dimensional image.

24. A method as in claim 23 applied to die frames of a motion picture.

25. A method as in claim 24 wherein additional depdi, position or shape information is automatically derived by computer analysis of the geometry of elements contained in die motion picture.

26. A method for modifying a stereoscopic image by altering the parallax shifts of image elements inherent within the image.

27. An improved method for creating image information by using a computer paint system, wherein painting functions may include any and all such functions available in standard computer paint and similar systems including but not limited to image loading, compositing, modification, processing, editing, as well as brush- oriented painting and pen-oriented drawing functions, wherein said improvement comprises d e application of a dis-interleaved painting function.

28. An improved method for applying paint system brash-oriented functions; whereby for active/existing pixels within a brash shape, often a bounding rectangle, the value of the image under each brush pixel is replaced by some function of die brush pixel value, or its neighborhood, and die value of die image under the brash pixel, or its neighborhood; and whereby prior to affecting said replacement, for each pixel within the brash shape, various conditions must be tested for and met; which conditions may include but are not limited to criteria of values for color, protection, depth or obscuration, for the brash pixel value, image pixel value or some combination of the two; which may result in the necessity to perform as many as P x C tests where P is the number of pixels within said brush shape, and C is die number of conditions to be tested for including die existence/activation of any pixel within said brash shape; wherein the improvement comprises: the maintenance of said brush as an array of pixel values and the maintenance of a list of address indices; and whereby non-active/non-existent pixels which are within die bounding shape do not appear on the list; and the further manipulation of said list whereby when a pixel fails any one test which is critical it is removed from said list and need not be considered for additional tests.

29. An improved mediod for applying paint system brush-oriented functions wherein the improvement comprises that during the execution of at least some paint function an image protection plane, of at least one bit, is maintained such that said image protection plane may be both wπtten into and referenced during painting and whereby the action of the paint function on the image is modulated based upon the values already stored in said image protection plane.

30. An improved method for applying paint system brush-oriented functions wherein the improvement comprises the application of image processing functions, which are normally only applied by a program to an entire image or within a bounding shape, including but not limited to contrast correction, color correction, gamma correction, gain, offset, and ma and logical functions, to the pixels under die brash being used by die operator.

31. An improved method of presenting a Graphic User Interface on a computer comprising any of the standard set of elements common to Graphic User Interfaces including but not limited to: buttons, menus, palettes, slidebars, check boxes, frames and otiier widgets, as well as windows which may, in mm, contain text, still or moving images or other information, any of which may include drop shadows and any of which may animate; where die improvement comprises the display of at least one of said elements in stereographic format.

32. A memod as in claim 31 wherein said Graphic User Interface is employed as an interface to a paint system.

33. A method as in claim 31 wherein said Graphic User Interface is employed as an interface to a microprocessor-based windowing system.

34. A method as in claim 31 wherein said Graphic User Interface is employed as an interface to a visual programming language system.

35. A method for displaying information with the perception of deptii comprising the synthesis of a plurality of images, each of said images constructed from a different viewing position, and displaying said plurality of images in a oscillatory sequential manner.

36. An improved method for the operator input of sealer information to a computer wherein the improvement comprises the employment of a plurality of transducers which may be used in a coordinated fashion for specification of a first set of variables and in an anti-coordinated fashion for specification of a second set of variables.

37. An improved device embodying die method of claim 36 for the operator input of sealer information to a computer wherein the improvement comprises the incorporation of a plurality of transducers which may be used in a coordinated fashion for specification of a first set of variables and in an anti-coordinated fashion for specification of a second set of variables.

38. A device as in claim 37 providing a third uncoordinated mode of operation whereby at least one first transducer receives null input and at least one second transducer receives input resulting in specification of a third set of variables.

39. A device as in claim 37 wherein the plurality of transducers comprises at least two translation transducers which when used in a coordinated fashion specify translation and when used in an anti-coordinated fashion specify rotation.

40. A device for the operator input of sealer information to a computer wherein the improvement comprises die incorporation of a plurality of rotational transducers at least one of which specifies more than one axis of information.

41. A device as in claim 40 wherein the plurality of rotational transducers comprises two balls.

42. A device as in claim 40 wherein the plurality of rotational transducers comprises four balls at the comers of a square and one pivot ball in the center and where at least the four comer balls provide active input data.

43. An improved method for interpreting mput from a device for the operator input of sealer information to a computer whereby the improvement comprises accumulating the input in one direction and ignoring input in the other direction.

44. An improved mediod for controlling sequential frames of information presented on a computer screen, wherein the improvement comprises die interpretation of input from a device for the operator input of sealer information to a computer, such that the number of frames between the current displayed frame and die next frame displayed is directly dependent upon input from the device by utilization of a page flip paradigm.

45. An improved method for displaying diagnostic and odier medical images which may include without limitation x-ray, CAT, NMR, endoscopic, photographic, video, computer or film images, wherein the improvement comprises the creation or modification of deptii information to be incorporated into said medical images.

46. A device for viewing sequentially interleaved binocular information comprising a set of glasses incorporating light sources which may be directed to each eye separately wherein the light sources for the left and right eye are alternated in such a manner that the light source is illuminated for the eye opposite to that for which an image being displayed is intended.

47. A method for compatible stereoscopic broadcast comprising: the broadcast of a two-dimensional image which may be displayed on a standard-type television display; additionally, the broadcast of deptii information which is not visibly displayed on a standard television; the distortion of said two-dimensional image into two images, at least one of which has been modified witii respect to said two-dimensional image based upon said depdi information; and, the stereoscopic display of said two images.

48. A mediod as in claim 47 wherem said depth information is encoded into a video blanking interval prior to the two-dimensional image which is to be modified based upon said deptii information.

49. A method for the creation of a stereoscopic display comprising the interleaving of at least two images, wherein at least some of the interleaved segments are less than one video field.

50. A improved method for the creation of an anaglyphic stereoscopic display on a composite video monitor, wherein the improvement comprises the interleaving of segments tinted by a first color only with segments tinted by a second color only.

51. A method as in claim 50 wherein die distribution of those segments tinted by a first color only and tiiose segments tinted by a second color only is varied on a frame-to-frame basis by a variable STS scheme.

52. An improved metiiod for creating three-dimensional images wherein the improvement comprises that for areas of ambiguous or uncertain depdi, such as boundaries between objects separated in the depdi direction, the deptii value utilized is modulated by a random-noise-like function.

53. A system embodying die mediod of claim 1.

54. A system embodying die metiiod of claim 6.

55. A system embodying die method of claim 7.

56. A system embodying the method of claim 12.

57. A system embodying the method of claim 22.

58. A system embodying die metiiod of claim 25.

59. A system embodying die mediod of claim 45.

60. A product produced by the method of claim 1.

61. A product produced by die method of claim 12.

62. A product produced by die mediod of claim 17.

63. A product produced by die method of claim 22.

64. A product produced by die mediod of claim 24.

65. A product produced by the method of claim 35.

66. A product produced by die mediod of claim 45.

67. A product produced by die method of claim 46.

68. A product produced by the method of claim 51.

69. A product produced by the method of claim 1 and embodied as a stereoscopic pair of film images.

70. A product produced by the method of claim 12 and embodied as a video signal carried by an information bearing medium.

71. A product produced by the method of claim 46 and embodied as a video signal carried by an information bearing medium.

72. A product produced by the metiiod of claim 49 bearing medium. and embodied as a video signal carried by an information

73.

An improved metiⁱod for the creation of an anaglyphic stereoscopic display on a composite video monitor, wherein the improvement comprises the interleaving of segments tinted by a first color only witi_i segments tinted by a second color only.

74. A method as in claim 1 wherein the distribution of those

75.