JP6526051B2 - Image processing apparatus, image processing method and program - Google Patents

Description

  The present invention relates to an image processing technique for displaying image data on a head mounted display or the like.

  A technique is known in which an observer views an image using a head mounted display (hereinafter, HMD) fixed to the observer's head. In recent years, glasses-type displays and the like are also known. The image data displayed on the HMD is obtained by cutting out an image in the entire circumferential direction each time, and the original image is also called an environment map. The environment map can be acquired, for example, by photographing using a fisheye lens or a method of imaging light reflected by a mirror ball with a camera.

  Patent Document 1 discloses a method of detecting the motion of the observer using a sensor incorporated in the HMD, and scrolling and displaying the displayed image according to the operation corresponding to the movement of the head of the observer. ing.

JP, 2013-254251, A

  In recent years, as resolution of display images has been increased, the capacity of image data necessary for display has also increased, and it takes time to read out and decode image data for display on the HMD. Therefore, according to the method disclosed in Patent Document 1, even if the head moves in a direction desired to be viewed by the observer, image data desired to be viewed may not be displayed smoothly.

  Therefore, the present invention aims to display image data according to the line of sight of the observer more naturally according to the movement of the observer by predicting the area to be viewed according to the movement of the observer.

In order to solve the above problems, the present invention is an image processing apparatus for displaying image data in accordance with the gaze direction of the observer, which is storage means for storing a plurality of environment maps having different resolutions; Based on acquisition means for acquiring information indicating movement as movement information, holding means for holding current area information indicating the position of the current area displayed to the observer, and based on the current area information and the movement information, Target area determination means for determining a target area in the environment map, path area determination means for determining an area including a path whose line of sight direction changes from the current area to the target area as a path area, and a plurality of environment maps Among them, setting means for setting resolutions respectively corresponding to the path area and the target area, and the setting means Generating means for reading out image data corresponding to each of the path area and the target area from an environment map of a specified resolution, and generating image data for displaying on the path area and image data for displaying on the target area possess the door, the motion information includes acceleration information comprising information indicating the acceleration in a lateral direction of the information and the vertical direction showing the acceleration, the setting means, towards the longitudinal acceleration is higher than the lateral acceleration The second resolution lower than the first resolution is set as the resolution corresponding to the path area if the lateral acceleration is larger than the longitudinal acceleration if the lateral acceleration is larger than the longitudinal acceleration. It is characterized by

  The present invention can display image data more naturally according to the movement of the observer.

FIG. 1 is a block diagram showing the configuration of an image processing apparatus. The figure which shows the structure of a head mounted display (HMD). The figure which shows the structure of an environment map. The flowchart of an environmental map display process. Flow chart of high resolution environment map display processing. Conceptual diagram of the prefetching method. Flow chart of high resolution environment map prefetching process. The flowchart of the resolution selection process at the time of prefetching. 10 is a flowchart of environment map display processing using a cache. 10 is a flowchart of environment map display processing involving movement. FIG. The flowchart of environment map pre-reading processing.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

First Embodiment
(overall structure)
In the first embodiment, an image processing apparatus for reading out an environment map of a plurality of resolutions according to acceleration information and displaying it on a head mounted display (HMD) will be described as an example. The present invention is not limited to the HMD, but can be applied to a similar device such as a glasses-type display.

  FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus applicable to the first embodiment. The image processing apparatus includes a CPU 101, a main memory 102, a storage device 103, a GPU 104, and a bus 106. First, the CPU 101 executes arithmetic processing and various programs. The main memory 102 holds programs and data necessary for image processing. The main memory 102 is also used as a work area for image processing by the CPU 101. The storage device 103 is a device for storing an image processing program and a plurality of image data, and for example, a hard disk drive (HDD) is used. The GPU 104 is connected to an HMD 105, which is an external display device for displaying image data and a GUI, by an I / F unit (not shown). The GUI 104 develops image data to be displayed and transfers the image data to the HMD 105. Note that an I / F connecting the GPU 104 and the HMD 105 can exchange data using, for example, infrared communication or a wireless LAN. A bus 106 is a bus connecting the above-described components. In addition, although there are various components other than the above regarding the system configuration, since it is not the gist of the present embodiment, the description thereof will be omitted.

  FIG. 2 is a diagram schematically illustrating the configuration of the HMD 105. As shown in FIG. As shown in FIG. 2A, the HMD 105 has a display unit 201, and the display unit 201 further includes an acceleration sensor 203, a geomagnetic sensor 204, and a GPS sensor 205. The acceleration sensor 203, the geomagnetic sensor 204, and the GPS sensor 205 are sensors for acquiring motion information indicating the motion of the observer wearing the HMD 105. The acceleration sensor 2013 detects a lateral direction in which the observer's head moves to the left and right, and a vertical acceleration moving in the vertical direction, and outputs it as acceleration information. The geomagnetic sensor 204 detects the direction of the geomagnetism to calculate the orientation of the HMD 105, and outputs angle information indicating the direction in which the observer is watching. A GPS (Global Positioning System) sensor 205 measures the position of the HMD 105 on the earth and acquires position information. In addition, since the fixing unit 202 fixes the display unit 201 to the observer's head, the fixing unit 202 is structured such that the observer can put it on his / her ear. That is, in conjunction with the movement of the head of the observer, each sensor incorporated in the display unit 201 detects the position and the direction.

  A more detailed configuration of the display unit 201 is shown in FIG. FIG. 2B schematically shows the internal configuration of the display unit 201 when the HMD 105 shown in FIG. 2A is viewed from above. The display unit 201 has a structure in which the entire visual field of the observer can be covered by the liquid crystal display 206 by passing the lens 207. The liquid crystal display 206 displays image data. The liquid crystal display 206 displays image data obtained by cutting out a part of an environment map consisting of images in all directions. This allows the viewer to view a realistic image as if at the center position of the environment map. Image data to be displayed is transferred from the image processing apparatus 400.

  FIG. 3 is a diagram for explaining an environment map. FIG. 3A shows the coordinate system of the environment map with respect to the viewpoint position. As shown in FIG. 3A, the viewing direction from the viewpoint position 301 of the observer can be expressed using polar coordinates (θ, φ). An environment map is obtained by quantizing the polar coordinates (θ, φ) and replacing information for each quantized polar coordinates (θ, φ) with image data. It is also called an omnidirectional image or an omnidirectional image. Each pixel constituting the environment map holds a brightness vector, the magnitude of the brightness vector represents a brightness value, and the direction of the brightness vector always points to the viewpoint position. That is, the environment map is luminance information of the entire periphery incident on the viewpoint position 301 of the observer. As data methods for holding the environment map, there are the equidistant cylinder map method shown in FIG. 3B, the spherical map method shown in FIG. 3C, and the like. In the equidistant cylinder map method shown in FIG. 3B, luminance information is held by converting the image data into a two-dimensional rectangular image data with θ as the horizontal axis and φ as the vertical axis. In the spherical map method shown in FIG. 3 (c), an axis from the center of the circle to the arc is φ in the environment map, and the rotation direction from the center of the circle is θ in the environment map. And the luminance information is held. Both the sine cylinder map method and the spherical map method cause distortion because a spherical environment map is projected on a plane. Therefore, in both methods, it is necessary to perform projective transformation when generating image data to be displayed in the HMD.

(Configuration of image processing apparatus)
FIG. 4 shows the detailed configuration of the image processing apparatus 400 applicable to the first embodiment. In the present embodiment, each configuration will be described by way of an example in which the configuration is implemented by software.

  The image processing apparatus 400 according to the present embodiment cuts out the image data of the display candidate from the environment map of the plurality of resolutions stored in the storage device 103 according to the operation of the HMD 105 to generate display image data. FIG. 5 shows a plurality of environment maps. The environment maps in this embodiment hold three different resolution environment maps: a high resolution environment map 501, a medium resolution environment map 502, and a low resolution environment map 503. In addition, the environment map in this embodiment presupposes that it is an equidistant cylinder map system. The environment map of each resolution is divided into block images 504 and stored in the storage device 103 as different files. Therefore, the image processing apparatus 400 can read out only block images of the required resolution.

  The detection result acquisition unit 401 acquires, from the HMD 105, motion information such as acceleration information detected by the acceleration sensor 203 and angle information detected by the geomagnetic sensor 204. The current area determination unit 409 specifies the area currently viewed by the observer as the current area 601 based on the angle information. FIG. 6A shows a diagram for explaining the current area.

  The target area determination unit 403 acquires the position of the current area from the current area 409. Further, when acquiring new acceleration information, the target area determination unit 403 predicts the direction in which the observer intends to view the current area 601 from now as shown in FIG. 6A, and determines the target area 602. .

  When the target area 602 is determined, the path area determination unit 404 determines the path area 603 based on the current area 601 and the target area 602. The path area 603 is an area that enters the view of the observer until the line of sight moves to the target area 602 that the observer intends to view.

  The resolution information acquisition unit 402 acquires a resolution that can be displayed on the display unit 201 of the HMD 105 as resolution information. The required resolution setting unit 405 calculates the required resolution of display image data to be transmitted next based on the acceleration information obtained from the detection result acquisition unit 401 and the resolution information obtained from the resolution information acquisition unit 402. Specifically, the resolution information acquisition unit 402 sets the required resolution of the image data to be displayed in the route area according to the acceleration in two directions indicated by the acceleration information. Further, the necessary resolution of the image data to be displayed in the target area is set according to the resolution information.

  The display image generation unit 407 extracts the environment map of the resolution to be displayed from the environment maps stored in the environment map acquisition unit 406, cuts out the area, and projective converts the cut environment map, thereby displaying the display image data. Generate Even if the display image data is displayed in the same size area, the image data generated by cutting out the environment map with higher resolution has a larger amount of data, and the data transfer also takes time. The display image generation unit 407 in the present embodiment first generates display image data from the environment map with the lowest resolution when the viewer starts viewing the video. Thereafter, the display image generation unit 407 generates display image data in accordance with the movement of the HMD 105. When motion is detected in the HMD 105, display image data to be displayed in the route area and display image data to be displayed in the target area are generated and sequentially output. In addition, it is assumed that the size (field angle information) of the image data to be displayed is set in advance in the display unit 201 of the HMD 105 connected to the display image generation unit 407.

  The transmitting unit 407 transfers display image data to be displayed on the display unit 201 to the HMD 105 side.

(Flow of processing in image processing apparatus)
The flow of processing in the image processing apparatus 400 will be described using FIG. 7. The CPU 100 implements the image processing apparatus 400 by reading and executing the program of the flowchart shown in FIG. 7 from the ROM 102.

  First, when the observer wears the HMD 105 and viewing is started, the environment map acquisition unit 406 reads the low resolution environment map 503 from the storage device 103 and deploys it in the main memory 102 in step S701.

  In step S702, the current area determination unit 409 determines, based on the angle information (θ, φ) acquired by the detection result acquisition unit 401 from the geomagnetic sensor 204 of the HMD 105 and the field angle information of the display unit 201 stored in advance. Determine the current area that the observer is currently observing. As described above, luminance information for each angle is held in advance in the environment map. Therefore, in the present embodiment, the current area 601 is set by associating the angle information (θ, φ) with the coordinate system of the environment map, and setting an area according to the angle of view.

  In step S703, the display image generation unit 407 refers to the low resolution environment map 503, cuts out the current area, and executes projective transformation for displaying the luminance information of the cut out area on the display unit 201 of the HMD 105. The low resolution display image data generated by the display image generation unit 407 is transferred to the HMD 105 via the transmission unit 408. Since the low resolution environment map is already expanded in the main memory 102, it can be processed at high speed. In addition, since the low resolution environment map has a small data capacity and does not have a high processing load, it is possible to display image data of the area viewed by the viewer at high speed.

  In step S705, the detection result acquisition unit 401 determines whether new acceleration information (aθ, aφ) is acquired from the acceleration sensor 203 of the HMD 105. If the acceleration information is not updated, the process proceeds to step S706, and if new acceleration information is acquired, the process proceeds to step S711. Note that the detection result acquisition unit 401 acquires acceleration information (aθ, aφ) at predetermined time intervals, and if (aθ, aφ) = (0, 0), the process proceeds to step S 706 and (aθ, aφ) φ If (0, 0), the process may proceed to step S711.

  In step S706, when acceleration information (aθ, aφ) is not acquired after transmitting display image data to the current area 601, there is no movement in the head of the observer, and the same current area 601 is continuously viewed. Means Therefore, the necessary resolution calculation unit 405 calculates a resolution that can be displayed in the current area 601 and that requires new generation of display image data. The required resolution information calculation unit 405 acquires resolution information indicating the highest resolution that can be displayed on the display unit 201 of the HMD 105 from the resolution information acquisition 402. The required resolution calculation unit 405 determines the highest resolution among the environment maps as the required resolution below the maximum displayable resolution indicated by the resolution information.

  In step S 707, the display image generation unit 407 refers to the required resolution obtained from the required resolution calculation unit 706, and determines whether it is necessary to generate display image data with a higher resolution than the low resolution environment map. If the required resolution is equal to or lower than the low resolution environment map 503, the resolution can not be changed, and the process advances to step S721. If the required resolution is higher than the resolution of the low resolution environment map 503, the process proceeds to step S708.

  In step S 708, the display image generation unit 407 reads from the storage device 103 only the block image corresponding to the current area 601 from the environment map according to the required resolution, and expands the image in the main memory 102.

  In step S709, the display image generation unit 407 cuts out the current region 601 from the read block image, and projective converts the cut out image data to generate display image data.

  In step S710, the display image generation unit 407 outputs the display image data to the transmission unit 408, and the transmission unit 408 transfers high resolution display image data by the HMD 105. As a result, when the observer continues to view the current area 601, it is possible to view a higher resolution image.

  Next, a process when the observer's head moves and new acceleration information (aθ, aφ) is acquired from the acceleration sensor 203 of the HMD 105 (or (aθ, aφ) a (0, 0)) will be described. In step S711, the target area determination unit 403 determines the target area 602 based on the acceleration information (aθ, aφ). Because of its structure, the line of sight of the human being is limited by the movable range in the θ and φ directions as shown in FIG. 6 (c). Therefore, the target area determination unit 403 predicts the movement of the observer's head from the movement range of the head held in advance and the acceleration information (aθ, aφ) as shown in FIG. An area 602 is set. The target area is calculated from the movement amount moved by the acceleration (aθ, aφ) for a given time Δt seconds from the central coordinates (θc, φc) of the current area. The central coordinates (θt, φt) of the target area after Δt seconds are

により算出できる。この中心座標から画角に合わせた領域を設定する。Δtは、人間の頭部を稼働する平均時間として1秒を設定しておく。なお、（θt、φt）が頭部の可動範囲を超える場合は、可動範囲内にターゲット領域を再設定する。

It can be calculated by From this center coordinate, set an area in accordance with the angle of view. Δt is set to 1 second as an average time to operate the human head. When (θt, φt) exceeds the movable range of the head, the target area is reset within the movable range.

  In step S 712, the route area calculation unit 404 sets a route area 603 connecting two areas from the current area 601 and the target area 602. The path area is determined by calculating a block image group that covers all areas in which the current area and each vertex of the target area are connected by a straight line.

  In step S713, the display image generation unit 407 cuts out the image data of the target area 602 and the route area 603 from the low resolution environment map 503 on the main memory 102, and performs projective transformation.

  In step S714, the display image generation unit 602 sequentially outputs the display image data of the path area 603 and the display image data of the target area 602 to the transmission unit 408. The transmission unit 408 controls the display unit 201 to display low-resolution display image data in accordance with the angle information.

  In step S715, the required resolution calculation unit 405 calculates the required resolution. The required resolution for the target area 602 is determined in the same manner as step 706. Details of the calculation of the required resolution for the route area 603 in step S715 will be described later.

  In step S 716, as in step S 707, the display image generation unit 716 refers to the required resolution obtained from the required resolution calculation unit 706 and determines whether it is necessary to generate display image data of higher resolution than the low resolution environment map. Determine If it is determined that the display image data has been changed, the process proceeds to step S717.

  The processing of the display image generation unit 407 in steps S717 to S719 is similar to that of steps S708 to S710.

  In step S 720, the current area determination unit 409 determines the position of the current area 601 based on the angle information (θ, φ) transmitted from the geomagnetic sensor 204 of the HMD 105 at predetermined intervals and the position information transmitted from the GPS sensor 205. Update.

  In step S721, the image processing apparatus 400 determines whether the viewing in the HMD 105 has ended, and in the case of ending the process, the processing in the image processing apparatus 400 is completed. If viewing is continued, the process returns to step S705, and display image data is generated according to the movement of the observer's head.

  FIG. 8 is a flowchart of the calculation process of the required resolution by the required resolution acquisition unit 402 in S715. In step S801, it is determined whether the resolution of the display unit 201 of the HMD 105 is equal to or greater than a predetermined threshold R. If the resolution of the display unit 201 is equal to or greater than the threshold R, the process proceeds to step S802. If the resolution of the display unit 201 is less than the threshold R, the process proceeds to step S805.

  In step S802, based on the acceleration information (aθ, aφ) obtained from the HMD 105, it is determined whether the absolute value of the acceleration is less than a predetermined threshold A. When the movement of the head is fast, the observer can infer that the path area until the observer's line of sight reaches the viewing direction can not be seen as much video. Therefore, if the absolute value of the acceleration is equal to or greater than the threshold A, the process proceeds to step S805, and a relatively low resolution is set for the route area. If it is determined in step S802 that the absolute value of the acceleration is less than the threshold A, the process proceeds to step S803.

  In step S803, the magnitudes of aθ and aφ are compared. If the size of aφ is less than the size of aφ, the process proceeds to step S804. If the size of aθ is greater than or equal to the size of aφ, the process proceeds to step S805. In step S804, the same resolution as the resolution corresponding to the target area 602 is set as the resolution for the path area 603. That is, as the resolution for the route area 603, the highest resolution among the environment maps that can be displayed by the display unit 201 is determined. In step S805, as the resolution for the path area 603, a resolution in which the one-step resolution is lower than the resolution corresponding to the target area 602 is set. For example, in the present embodiment, when the same resolution as the high resolution environment map 501 is determined for the target area 602, the same resolution as the medium resolution environment map 502 is determined as the necessary resolution for the path area 603. This takes account of the need to look ahead over a wide area, as the human head has a wider range of movement in the θ direction. That is, when the head moves largely in the θ direction, the data capacity is reduced by lowering the resolution as the size of the display image data to be generated becomes large.

  As described above, according to the present embodiment, the area to be observed by the observer wearing the HMD 105 is predicted in advance to generate display image data. Since the display image data generated based on the prediction is generated in advance of the image to be displayed on the display unit 201 of the HMD 105, the display unit 201 displays a high resolution image in real time according to the movement of the observer can do.

Second Embodiment
In the first embodiment, the display image data generated by the image processing apparatus 400 is displayed on the display unit 201 of the HMD 105 as needed. In the second embodiment, a method of caching (temporarily holding in the main memory 102) and deleting the image data prefetched as the target area 603 will be described. The same components as those of the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  FIG. 9 is a diagram for explaining the movement characteristic of the human head. First, it is assumed that the head 901 is facing the viewpoint position 902. Next, as shown in FIG. 9B, the head 901 rotates in the θ direction, and the viewpoint position moves to the viewpoint position 903. When the neck reaches a position where it can not be rotated any more, as shown in FIG. 9C, the head 901 is reversely rotated, and there is a characteristic that it is likely to return to the original viewpoint position 902. In the present embodiment, a method of caching image data using this motion characteristic will be described.

  FIG. 10 is a flowchart showing the entire process in the second embodiment. In step S1001, the image processing apparatus 400 acquires acceleration information detected by the acceleration sensor 203 from the HMD 105.

  In step S1002, the image processing apparatus 400 executes the processing from step 701 to step S704 according to the angle information, and generates display image data in the current area 601 viewed by the observer. Also, by executing the processing from step S711 to step S719 according to the acceleration information, the display image data of the target area 602 and the display image data of the route area 603 are generated according to the acceleration direction, and the main memory Cache to 102. The image processing apparatus 400 causes the main memory 102 to hold display image data until at least the direction indicated by the acceleration information acquired from the HMD 105 is reversed, that is, until the acceleration information in the reverse direction is input. The display image data corresponding to each area is sequentially transferred to the HMD 105 in step S1003, and the display unit 210 displays the display image data in the order received in step S1004. At this time, the display image data remains held in the main memory 102.

  In step S1005, the HMD 105 acquires new acceleration information, and transfers the new acceleration information to the image processing apparatus 400. In step S1006, display image data is generated based on the acceleration information acquired from the HMD 105. The process of calculating the target area, the path area, and the required resolution is performed in the same manner as in the first embodiment. However, when the display image data to be generated is cached in the main memory 102, the display image data is read out from the main memory 102 without newly cutting out the environment map and performing projection conversion. If the display image data to be generated is not stored in the main memory 102, display image data corresponding to the target area and the route area is generated as in the first embodiment. In step S1008, display image data is displayed on the display unit 201 of the HMD 105. In step S1009, the data is deleted from the main memory 102.

  As described above, the display image data generated by prefetching is cached in consideration of the possibility that the display image data generated once is viewed again according to the movement characteristic of the human head. As a result, it is possible to display a high resolution image in real time with less processing load in real time according to the movement of the observer.

Third Embodiment
In the second embodiment, the method of controlling the display image data according to the movement characteristic of the human head has been disclosed. In the third embodiment, a process of switching the environment map in accordance with the movement (walking) of the observer will be described. The same components as those of the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  FIG. 11 is a schematic view illustrating a method of switching an environment map in accordance with human walking. It is assumed that the observer 1101 first observes the environment map A. When the observer 1101 advances, it is necessary to change the view observed by the observer 1101. Therefore, it is necessary to switch from the environment map A to the environment map B. At this time, it is assumed that the observer 1101 walks while gazing at a certain gaze point 1102. As described in the above embodiments, when viewing a video in the same viewing direction, display image data generated based on the low resolution environment map is displayed first, and then the high resolution environment map is displayed. It switches to the display image data generated based on this. Therefore, the observer 1101 already observes the high-resolution environment map A, and as a result of walking, when switching to the low-resolution display image data of the environment map B based on the position information and the angle information, the sense of discomfort is felt. I feel it. Therefore, a method for reducing this feeling of incongruity is required.

  FIG. 12 is a flowchart showing the entire process in the third embodiment. In step S 1201, the display unit 201 of the HMD 105 displays display image data generated based on the high-resolution environment map A at the gaze point 1002. In step S1002, the walking of the observer 1101 is detected using acceleration information by the acceleration sensor 203, angle information by the magnetic sensor 204, and position information by the GPS sensor 205, and the movement information is transmitted to the image processing apparatus 400 as movement information. .

  In step S1003, the environment map B corresponding to the position after the observer 1101 has moved according to the movement information (direction, distance) is selected. In step S1004, the HMD 105 transmits the resolution of the environment map A at the currently displayed gaze point 1002 to the image processing apparatus 400. In step S1005, the HMD 105 matches the display image data generated based on the environment map A at the currently displayed gaze point 1002 with the movement information of the observer 1101 under the control of the CPU (not shown). Projective transformation and display. On the other hand, in step S1006, the image processing apparatus 400 sets the resolution of the environment map B to be read in accordance with the resolution displayed by the HMD 105. In step S1007, the image processing apparatus 400 reads a block image of the current area centered on the gaze point 1002 of the environment map B of the resolution set in step S106 from the step storage device 103 and develops the block image in the main memory 102. In step S1008, the display image generation unit 407 of the image processing apparatus 400 performs projection conversion on the read image data of the current area to generate display image data. In step S1009, the display unit 201 of the HMD 105 displays the display image data generated in step S1008.

  As described above, even when the observer moves, it is possible to display a high-resolution image in real time without the observer feeling discomfort due to the change in resolution. Further, even when the environment map corresponding to the viewpoint position is changed due to the change of the position of the observer, it is possible to efficiently generate high resolution image data.

<Other Embodiments>
In the above embodiment, in order to detect the movement of the head (or the line of sight) of the observer, an example in which the acceleration of the head is detected using an acceleration sensor mounted on the HMD 105 and the acceleration information is used is described. However, the method of acquiring the observer's movement is not limited to this. For example, it may be a method of detecting the velocity when the head moves instead of the acceleration, or information represented by a vector instead of the acceleration information (aθ, aφ) may be used as the information indicating the movement of the head. .

  The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  The present invention is not limited to the above embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Accordingly, the following claims are attached to disclose the scope of the present invention.

Claims (9)

An image processing apparatus for displaying image data according to a gaze direction of an observer,
Storage means for storing a plurality of environment maps of different resolutions;
Acquisition means for acquiring information indicating the movement of the observer as movement information;
Holding means for holding current area information indicating the position of the current area displayed to the observer;
Target area determining means for determining a target area in the environment map based on the current area information and the motion information;
Route area determination means for determining an area including a path in which the sight line direction changes from the current area to the target area as a path area;
Setting means for setting resolutions respectively corresponding to the path area and the target area among the plurality of environment maps;
The image data corresponding to the path area and the target area are read out from the environment map of the resolution set by the setting means, and the image data to be displayed on the path area and the image data to be displayed on the target area It has a generating means for generating,
The motion information includes acceleration information including information indicating lateral acceleration and information indicating longitudinal acceleration.
The setting means is configured to set the first resolution when the longitudinal acceleration is larger than the lateral acceleration, and to be lower than the first resolution when the lateral acceleration is larger than the longitudinal acceleration. An image processing apparatus , wherein a second resolution is set as a resolution corresponding to the path area . The setting means is further based on said magnitude of the absolute value of the acceleration information, the image processing apparatus according to claim 1, characterized in that to set the resolution corresponding to the path area. The said setting means sets the resolution corresponding to the said target area | region as said 1st resolution, when the acceleration of the vertical direction is larger than the acceleration of the horizontal direction. Image processing device.   The image processing apparatus according to any one of claims 1 to 3, wherein the path area determination unit sets an area narrow in the vertical direction and wide in the horizontal direction as the path area.   5. The setting method according to any one of claims 1 to 4, wherein the setting means stores in advance the highest resolution which can be displayed by the display means for displaying the image data, and sets the highest resolution as the resolution corresponding to the target area. The image processing apparatus according to one of the items. And a holding unit that holds the image data generated by the generation unit.
The acquisition means acquires information indicating the moving direction of the observer as the movement information.
The holding unit holds the image data until the acquisition unit acquires motion information in a direction opposite to the direction indicated by the motion information acquired when the image data is generated. The image processing apparatus according to any one of claims 1 to 5 . Furthermore, the acquisition unit acquires movement information indicating information related to the movement of the observer as the movement information;
The image processing apparatus according to any one of claims 1 to 6, wherein the generation unit switches an environment map to be read according to the movement information.   A computer program which causes a computer to function as the image processing apparatus according to any one of claims 1 to 7 by causing the computer to read and execute the program. An image processing method for displaying image data according to a gaze direction of an observer,
An acquisition step indicates the movement of the observer, obtained as acceleration information motion information comprising information indicating information and longitudinal acceleration indicating the acceleration in a lateral direction,
Holding the current area information indicating the position of the current area displayed to the observer;
A determining step of determining a target area in the environment map based on the current area information and the motion information;
Determining an area including a path in which the sight line direction changes from the current area to the target area as a path area;
Setting the resolution corresponding to each of the path area and the target area among a plurality of environment maps of different resolution stored in the storage means;
The image data corresponding to the path area and the target area are read out from the environment map of the resolution set by the setting means, and the image data to be displayed on the path area and the image data to be displayed on the target area And generating steps to generate
The setting step is the first resolution when the longitudinal acceleration is larger than the lateral acceleration, and is lower than the first resolution when the lateral acceleration is larger than the longitudinal acceleration. Setting the second resolution as the resolution corresponding to the path area .

Images