WO2018025825A1 - Image capture system - Google Patents

Abstract

Provided is an image capture system which facilitates an image capture operation for a full spherical image at the field of a facility, a real-estate property and the like, and which makes it possible to delete an unwanted portion, such as the image of the photographer, from an acquired full spherical image by simple image processing. The image capture system is provided with: an image capture device which generates, in a single image capture operation, an image in which a subject included in the surrounding omnidirectional 360° range is captured; a portable terminal including an image data acquisition unit which acquires image data of a plurality of images generated by having the image capture device capture the same subject a plurality of times, the plurality of images having different positions of an object other than the subject with respect to the image capture device; and a server including an image processing unit which generates an image from which an image of the object is deleted by synthesizing the plurality of images.

Description

Imaging system

The present invention relates to an imaging system that generates an omnidirectional image.

In recent years, image display technology of virtual reality (virtual reality, hereinafter also referred to as "VR") is used in various fields such as games and entertainment, learning support, vocational training, and business. In the VR, usually, a glasses-type or goggle-type display device called a head mounted display (hereinafter also referred to as “HMD”) is used. The user can recognize an image having a three-dimensional effect by wearing the HMD on the head and viewing the screen built in the HMD with both eyes. The HMD incorporates a gyro sensor and an acceleration sensor, and the image displayed on the screen changes according to the movement of the user's head detected by these sensors. As a result, the user can experience as if it were embedded in a three-dimensionally displayed image (virtual space).

An image to be a material of VR is acquired by capturing an image of a real subject with a camera, configuring it with computer graphics, or combining both. Conventionally, when acquiring a panoramic image with a wide angle of view, images of a plurality of frames generated by repeating imaging while changing the direction of the camera have been stitched together. On the other hand, recently, a camera (hereinafter, also referred to as a "all-sky camera") capable of picking up an image (so-called all-sky image) in which a 360.degree. Reference 1 and 2) are put to practical use. In the case of using the omnidirectional camera, it is possible to easily and quickly acquire an image containing substantially the entire subject around the camera.

JP, 2013-198062, A JP, 2016-103806, A

As an example of use in the business of VR, there is an example of having the customer experience the preview of the facility such as a wedding hall or the inside view of the real estate property by the customer instead of taking the customer to the site. In these cases, an image for VR can be easily created by imaging each room of a facility or a real estate property with an omnidirectional camera. On the other hand, when using an omnidirectional camera, there is a problem that a photographer operating the omnidirectional camera may be reflected in an image. Therefore, in the related art, it has been necessary to set up the omnidirectional camera at the site, move the photographer to the next room or hide behind objects, and remotely perform imaging. In particular, when a plurality of staff members including a photographer are present at a site, it may be difficult to secure a hiding place at the time of imaging, and even one person may have to recapture imaging when it is reflected in the image, which takes time and effort. There was also.

With respect to such a problem, Patent Document 2 is an imaging device that simultaneously captures a plurality of images for generating a omnidirectional image, and is generated from a captured image captured by the imaging device or the captured image. In the generated image, it is determined whether or not the image of the support supporting the imaging device is within a predetermined range, and based on the determination result, guide information for changing the position of the viewpoint at which the camera captures an image A technology to notify is disclosed. That is, in Patent Document 2, by notifying the photographer of the guide information, an image suitable for removing the image of the photographer is acquired. Thereby, an area (photographer's area) where the photographer is photographed is extracted from the acquired image, and the photographer is arranged by arranging a texture image based on another area (for example, the ground area where the ground is photographed) to this area. Has erased the image.

However, in the case of Patent Document 2, since the photographer has to adjust his / her position and the position of the camera according to the guide information, the imaging operation is troublesome. Further, image processing for extracting a photographer area and a ground area from the acquired image is also complicated.

The present invention has been made in view of the above, and can easily perform an imaging operation of an omnidirectional image at a site such as a facility or a real estate property, and an image of a photographer from the acquired omnidirectional image It is an object of the present invention to provide an imaging system capable of erasing unnecessary portions such as by simple image processing.

In order to solve the above-mentioned subject, the imaging system which is one mode of the present invention generates the picture in which the subject included in the range of 360 degrees of all directions of a circumference was photographed by one imaging operation, and the above-mentioned imaging device An image data acquisition unit configured to acquire image data of a plurality of images generated by imaging the same subject a plurality of times in the plurality of images, wherein positions of objects other than the subject with respect to the imaging device are different; And an image processing unit configured to generate an image in which the image of the object is erased by combining a plurality of images.

The imaging system includes a server that manages an image, and a portable terminal connected to the server via a communication network, the image data acquisition unit is provided in the portable terminal, and the image processing unit is Provided in a server, the portable terminal transmits image data of the plurality of images acquired by the image data acquisition unit to the server, the server receives image data of the plurality of images, and the image data The image processing unit may execute image processing based on the above.

In the imaging system, the image processing unit may further generate image data for virtual reality by using image data of an image from which an image of the object has been erased.

The imaging system includes a server that manages an image, and a portable terminal connected to the server via a communication network, and the image data acquisition unit and the image processing unit are provided in the portable terminal. The portable terminal may transmit the image data of the image from which the image of the object generated by the image processing unit is erased to the server.

In the above imaging system, the server may further include a second image processing unit that generates image data for virtual reality using image data of an image from which the image of the object has been erased.

In the imaging system, the portable terminal includes a display unit, an input device for inputting a signal according to an operation performed from the outside, and one of the plurality of images displayed on the display unit. When a signal for selecting at least one region in the one image is input from an input device, the signal in the other image of the plurality of images for the at least one region in the one image is input. And a display control unit that causes the display unit to display an area corresponding to the at least one area in an overlapping manner, and further, to the server, positional information indicating the position of the at least one area in the one image. The image processing unit may transmit, and in addition to the image data of the plurality of images, the position information may be used to generate an image in which the image of the object is erased.

In the above imaging system, the portable terminal includes a display unit, an input device for inputting a signal according to an operation performed from the outside, and a display control unit which causes one of the plurality of images to be displayed on the display unit. And the image processing unit adds the image data of the plurality of images when the signal for selecting at least one area in the one image is input from the input device. Position information representing a position of the at least one region in one image may be used to generate an image in which the image of the object is erased.

In the imaging system, the portable terminal acquires image data of the other image from the imaging device when a signal for selecting at least one region in the one image is input from the input device. Also good.

In the above imaging system, the mobile terminal changes the position of the object after the first imaging is finished, a display unit, and a first imaging instruction screen instructing to execute the first imaging by the imaging device. The display control unit may further include: a display control unit that causes the display unit to sequentially display a second imaging instruction screen that instructs the imaging device to execute a second imaging operation.

In the imaging system, the image processing unit extracts an area in which a change in pixel value occurs among the plurality of images, and a plurality of pixel values of the area in one image of the plurality of images are detected. The pixel value of the area in another image among the images of

The imaging system further includes a display unit, an input device for inputting a signal according to an operation performed from the outside, and a display control unit for displaying one of the plurality of images on the display unit. The image processing unit may be configured to input a plurality of pixel values of the at least one region in the one image, when a signal for selecting at least one region in the one image is input from the input device. The pixel value of the area corresponding to the at least one area in the other image among the images of.

The imaging system further includes a display unit, an input device for inputting a signal according to an operation performed from the outside, and a display control unit for displaying one of the plurality of images on the display unit. The image processing unit extracts at least one region in which a change in pixel value occurs among the plurality of images, and the display control unit causes the display unit to display the one image displayed on the display unit. At least one mark respectively representing the at least one region extracted by the processing unit is superimposed and displayed, and the image processing unit is a signal for selecting at least one of the at least one mark from the input device. When it is input, the pixel value of the area corresponding to the at least one mark in the one image corresponds to that in another of the plurality of images. It may be replaced with the pixel values of the range.

The imaging system further includes a display unit, an input device for inputting a signal according to an operation performed from the outside, and a display control unit that superimposes the plurality of images and causes the display unit to display the images. The processing unit may make the area transparent when a signal for selecting an area in the top layer image displayed on the display unit among the plurality of images is input from the input device.

In the imaging system, the display control unit may cause the display unit to display an image obtained by projecting the one image on a plane.

In the imaging system, the display control unit may cause the display unit to display an image obtained by performing coordinate conversion of the one image into an orthogonal coordinate system.

The imaging system includes: a storage unit storing image data of an image specifying a position of an imaging location; and a mapping processing unit mapping image data of the plurality of images to image data of an image specifying the position You may provide further.

According to the present invention, by causing the imaging device to capture the same subject a plurality of times, a plurality of images different in the position of an object other than the subject with respect to the imaging device are acquired, and the object is eliminated by combining these images. Because the generated image is generated, it is possible to easily perform the imaging operation of the omnidirectional image at the site, and unnecessary portions such as the image of the photographer from the acquired omnidirectional image can be simply processed. It becomes possible to erase.

It is a schematic diagram which shows schematic structure of the imaging system which concerns on the 1st Embodiment of this invention. It is a block diagram which shows schematic structure of the server shown in FIG. It is a block diagram which shows schematic structure of the portable terminal shown in FIG. It is a schematic diagram which illustrates the external appearance of the imaging device shown in FIG. It is a block diagram which shows schematic structure of the imaging device shown in FIG. It is a flowchart which shows operation | movement of the imaging system which concerns on the 1st Embodiment of this invention. It is a schematic diagram which illustrates the flyer of real estate property. It is a schematic diagram which illustrates the identification code reading screen displayed on a portable terminal. It is a schematic diagram which illustrates the imaging place selection screen displayed on a portable terminal. It is a schematic diagram which illustrates the imaging place confirmation screen displayed on a portable terminal. It is a schematic diagram which illustrates the imaging instruction screen (the 1st time) displayed on a portable terminal. It is a schematic diagram which illustrates the mode of the 1st imaging. It is a schematic diagram which illustrates the image acquired when imaging is performed by the positional relationship shown in FIG. It is a schematic diagram which illustrates the imaging directions screen (after the 2nd time) displayed on a portable terminal. It is a schematic diagram which illustrates the mode of the 2nd imaging. It is a schematic diagram which illustrates the image acquired when imaging is performed by the positional relationship shown in FIG. It is a schematic diagram which illustrates the confirmation screen displayed on a portable terminal. It is a schematic diagram which illustrates the imaging place change screen displayed on a portable terminal. It is a schematic diagram which illustrates the imaging completion confirmation screen displayed on a portable terminal. It is a schematic diagram which illustrates the transmission confirmation screen displayed on a portable terminal. It is a schematic diagram for demonstrating the process which erase | eliminates a photographer's image. It is a schematic diagram which illustrates the image by which the image of the photographer was deleted. It is a schematic diagram which illustrates the operation screen for making a user select the unnecessary area | region in a 1st modification. It is a schematic diagram for demonstrating the image processing in a 1st modification. It is a schematic diagram which illustrates the operation screen for making a user select the unnecessary area | region in a 2nd modification. It is a schematic diagram which illustrates the operation screen for making a user select the unnecessary area | region in a 2nd modification. It is a schematic diagram which illustrates the operation screen for making a user select the unnecessary area | region in a 3rd modification. It is a schematic diagram which illustrates the operation screen for making a user select the unnecessary area | region in a 4th modification. It is a flowchart which shows operation | movement of the imaging system which concerns on the 2nd Embodiment of this invention. It is a schematic diagram which illustrates the unnecessary field selection screen displayed on a personal digital assistant. It is a schematic diagram which illustrates the imaging instruction screen displayed on a portable terminal. It is a schematic diagram which illustrates the image confirmation screen displayed on a portable terminal. It is a flow chart which shows operation of an imaging system concerning a 3rd embodiment of the present invention. It is a schematic diagram for demonstrating the image processing in the 3rd Embodiment of this invention. It is a schematic diagram for demonstrating the image processing in the 3rd Embodiment of this invention.

Hereinafter, an imaging system according to an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited by these embodiments. Further, in the descriptions of the respective drawings, the same parts are denoted by the same reference numerals.

In the following, as an example, a case of generating an inside-looking image of a real estate property as an image to be a material of virtual reality (hereinafter referred to as “VR”) will be described.

First Embodiment
FIG. 1 is a schematic view showing a schematic configuration of an imaging system according to a first embodiment of the present invention. The imaging system 1 shown in FIG. 1 includes a server 10 that manages information on real estate properties (hereinafter referred to as “property information”) and an image on the real estate properties, and a mobile terminal that acquires image data of real estate properties and transfers it to the server 10 20 and an imaging device 30. Among these, the server 10 and the portable terminal 20 are connected via the communication network N. The portable terminal 20 and the imaging device 30 may be directly connected by wire or wireless, or may be connected via the communication network N.

As the communication network N, for example, an Internet line, a telephone line, a LAN, a dedicated line, a mobile communication network, a communication line such as WiFi (Wireless Fidelity), Bluetooth (registered trademark), or a combination thereof is used. The communication network N may be wired, wireless, or a combination of these.

FIG. 2 is a block diagram showing a schematic configuration of the server 10. The server 10 is a device that accumulates and manages image data of an image in which a real estate property is photographed, and is configured by, for example, a computer. Here, the server 10 does not necessarily have to be configured by one computer, and for example, the server 10 may be configured by a plurality of computers connected to the communication network N. As shown in FIG. 2, the server 10 includes a communication unit 11, a storage unit 12, and an arithmetic unit 13.

The communication unit 11 is an interface that connects the server 10 to the communication network N and communicates with other devices connected to the communication network N. The communication unit 11 is configured using, for example, a soft modem, a cable modem, a wireless modem, an ADSL modem, or the like.

The storage unit 12 is a computer-readable storage medium such as a semiconductor memory such as a ROM or a RAM or a hard disk, and includes a program storage unit 121, an image data storage unit 122, and a property information database 123.

The program storage unit 121 stores, in addition to the operating system program and the driver program, application programs for executing various functions, various parameters used during execution of these programs, and the like. Specifically, the program storage unit 121 manages property information, and performs image processing based on the image data acquired by the imaging device 30 and transferred via the portable terminal 20, thereby processing the image data of the VR image. And a program for causing the calculation unit 13 to execute a series of operations for managing the image data in association with property information.

The image data storage unit 122 is image data of an image showing a real estate property, image data of an image obtained by constructing this image for VR (hereinafter referred to as “VR image data”), and image data of a floor plan of real estate property Hereinafter, "lay-out data" is stored.

The property information database 123 stores, in the image data storage unit 122, information on properties for which image data is stored. Specifically, the property information is identification information (property ID) for identifying the property information, the location of the property, and an image in which the image data, VR image data, and floor plan data of the property are stored. It includes an address or the like representing a storage area on the data storage unit 122.

The arithmetic unit 13 is configured by using, for example, a central processing unit (CPU) or a graphics processing unit (GPU), and reads various programs stored in the program storage unit 121 to centrally control the respective units of the server 10. At the same time, the imaging system 1 executes various arithmetic processing for managing the image data of the property. Specifically, the calculation unit 13 includes a property information registration unit 131, a property information extraction unit 132, a property information update unit 133, and an image processing unit 134.

The property information registration unit 131 registers property information in the property information database 123 based on the information input via the communication network N. Specifically, when information such as the location or floor plan of a real estate property, rent or selling price is input, the property ID is issued, and the input information is linked to the property ID and recorded in the property information database 123 Do.

When the property ID is input through the communication network N, the property information extraction unit 132 extracts property information associated with the property ID from the property information database 123.
The property information updating unit 133 updates property information in accordance with the information input via the communication network N.

The image processing unit 134 executes predetermined image processing on the image of the real estate based on the image data input via the communication network N. Specifically, the image processing unit 134 includes an image combining unit 135 and a VR image data generating unit 136.

The image combining unit 135 combines a plurality of images acquired by capturing the same subject a plurality of times to obtain an unnecessary object reflected in these images, that is, an object other than the subject to be captured. Erase the image.

The VR image data generation unit 136 generates image data for VR based on the image from which the image of the unnecessary object has been erased by the image synthesis unit 135.

FIG. 3 is a block diagram showing a schematic configuration of the portable terminal 20. As shown in FIG. The mobile terminal 20 is a device for controlling a series of operations for imaging using the imaging device 30 at the site of a real estate property, such as a smartphone, a tablet terminal, a mobile phone, and a personal digital assistant (PDA), for example. It is comprised by the portable apparatus which can transmit / receive information via the communication network N. FIG.

As shown in FIG. 3, the mobile terminal 20 includes a communication unit 21, an input unit 22, a display unit 23, a reading unit 24, a storage unit 25, and an arithmetic unit 26.

The communication unit 21 is an interface that connects the portable terminal 20 to the communication network N and communicates with the server 10 and other devices. The communication unit 21 communicates with the imaging device 30 via a communication cable or by wireless communication. The communication unit 21 is configured using, for example, an ISDN modem, an ADSL modem, a cable modem, an optical modem, a soft modem, and the like.

The input unit 22 is an input device such as an operation button or a touch panel provided on the display unit 23, and inputs a signal corresponding to an operation performed from the outside to the calculation unit 26.

The display unit 23 is, for example, a liquid crystal display or an organic EL display, and displays an icon for a user to perform an input operation, a text message, an image, and the like.

The reading unit 24 is, for example, a camera incorporated in the mobile terminal 20, and has a function of recognizing an identification code such as a two-dimensional code. The portable terminal 20 does not necessarily have to have a built-in camera. When the portable terminal 20 does not have a built-in camera, a reader device capable of recognizing the identification code may be externally attached.

The storage unit 25 is a computer-readable storage medium such as a semiconductor memory such as a ROM or a RAM, and includes a program storage unit 251 and an image data storage unit 252.

The program storage unit 251 stores, in addition to the operating system program and the driver program, various application programs, various parameters used during execution of these programs, and the like. Specifically, the program storage unit 251 stores an application program for causing the computing unit 26 to execute a series of operations for acquiring image data by causing the imaging device 30 to perform imaging of a real estate property.

The image data storage unit 252 stores floor plan data of real estate properties transmitted from the server 10 and image data generated by the imaging device 30 performing imaging.

Arithmetic unit 26 is configured using, for example, a central processing unit (CPU) or a graphics processing unit (GPU), and reads various programs stored in program storage unit 251 to control each part of portable terminal 20 in an integrated manner. At the same time, a series of operations for acquiring image data of a real estate property from the imaging device 30 are executed. Specifically, the calculation unit 26 includes an identification information acquisition unit 261, a display control unit 262, an image data acquisition unit 263, and a mapping processing unit 264.

When the reading unit 24 recognizes an identification code, the identification information acquisition unit 261 acquires information such as a property ID represented by the identification code. Here, as the identification code, an optically readable code such as a bar code or a two-dimensional code is used.

The display control unit 262 causes the imaging device 30 to image the same subject multiple times, thereby displaying on the display unit 23 during a series of imaging operations for acquiring a plurality of images in which the position of an object other than the object with respect to the imaging device 30 is different. The screen to be displayed is generated and displayed on the display unit 23.

The image data acquisition unit 263 takes in the image data output from the imaging device 30, and causes the image data storage unit 252 to store the image data of the images acquired by imaging the same subject a plurality of times in association with each other.

The mapping processing unit 264 acquires floor plan data of a real estate property from the server 10, and performs mapping of the image data acquired by the image data acquisition unit 263 from the imaging device 30 on the floor plan data.

FIG. 4 is a schematic view illustrating the appearance of the imaging device 30. As shown in FIG. FIG. 5 is a block diagram showing a schematic configuration of the imaging device 30. As shown in FIG. The imaging device 30 is a so-called all-sky camera capable of imaging 360 degrees of all directions around it by one imaging operation, and as shown in FIG. 4, two super wide-angle lenses 30a attached to a housing 30c, 30b, and two imaging units (a first imaging unit 31a and a second imaging unit 31b) that each include a CCD image sensor or a CMOS image sensor, and generate and output image data of a color image.

Each of the super wide-angle lenses 30a, 30b has an angle of view of at least 180 °. The first imaging unit 31a is provided such that a light receiving surface for receiving light from a subject faces the outside through the super wide-angle lens 30a. In addition, an optical system such as a prism is disposed between the super wide-angle lens 30a and the first imaging unit 31a to make the light collected by the super wide-angle lens 30a incident on the light receiving surface of the first imaging unit 31a. ing. Thereby, a hemispherical image having a horizontal angle of view of 180 ° and a vertical angle of view of 180 ° can be acquired by the first imaging unit 31a. The super wide-angle lens 30 b and the second imaging unit 31 b also have the same configuration.

By combining the two hemispherical images acquired by the first imaging unit 31a and the second imaging unit 31b, it is possible to acquire an image in which all directions in the periphery are taken over 360 °. Such an image is also called an omnidirectional image.

As illustrated in FIG. 5, the imaging device 30 includes a communication unit 32, a storage unit 33, an operation unit 34, and a control unit 35 in addition to the first imaging unit 31 a and the second imaging unit 31 b.
The communication unit 32 is an interface that connects the imaging device 30 to the portable terminal 20 via a communication cable or by wireless communication.

The storage unit 33 is, for example, a semiconductor memory such as a ROM or a RAM, and temporarily stores image data output from the first imaging unit 31a and the second imaging unit 31b.

The operation unit 34 is an input device such as a push button, for example, and inputs an imaging instruction to the control unit 35 in response to an input operation (shutter operation) of the user. Note that the input of the imaging instruction to the imaging device 30 may be performed by remote operation from the portable terminal 20 connected to the imaging device 30, in addition to directly operating the operation unit 34.

The control unit 35 is configured using, for example, a central processing unit (CPU) or a graphics processing unit (GPU), controls the operation of the imaging device 30, and outputs each of the first imaging unit 31a and the second imaging unit 31b. Predetermined image processing is performed on the image based on the selected image data. Specifically, the control unit 35 performs image processing such as white balance processing and gamma processing on hemispherical images based on the image data output from the first imaging unit 31a and the second imaging unit 31b, respectively, and further, And an image processing unit 351 that generates an omnidirectional image by combining the hemispherical images.

As such an imaging device 30, for example, an all-sky digital camera (product name: THETA) manufactured by Ricoh Co., Ltd. can be used.

Next, the operation of the imaging system 1 according to the present embodiment will be described. FIG. 6 is a flowchart showing the operation of the imaging system 1.

Prior to imaging a real estate property, the photographer obtains a property ID of the property to be imaged. The method of obtaining the property ID is not particularly limited, but as an example, as shown in FIG. 7, a flyer printed with an identification code (for example, a two-dimensional code) m11 representing the property ID may be obtained. In addition to the identification code m11, information such as the floor plan m12 of the property and the location of the property are printed on the flyer M1.

First, in step S20, the photographer activates an imaging application on the portable terminal 20, and the reading unit 24 reads the identification code m11 printed on the leaflet M1. FIG. 8 is a schematic view illustrating an identification code reading screen displayed on the portable terminal 20 after activation of the imaging application. First, the photographer holds the reading unit 24 over the identification code m11 printed on the leaflet M1 in accordance with the message m21 displayed on the identification code reading screen M2.

In response to this, the portable terminal 20 reads the identification code m11, accesses the server 10 by the URL included in the read information, and transmits the property ID included in the read information to the server 10 (step S21). In addition, the method of transmitting property | object ID to the server 10 is not limited to this. For example, the photographer may directly input the property ID to the portable terminal 20 and transmit it.

In step S10, the server 10 receives the property ID. In the subsequent step S11, the property information extraction unit 132 of the server 10 searches the property information database 123, and extracts property information corresponding to the received property ID.

Subsequently, in step S12, the property information extraction unit 132 of the server 10 acquires the address of the storage area in which the floor plan data is stored from the extracted property information, and the floor plan data from the image data storage unit 122 according to this address. It reads out and transmits to the portable terminal 20.

In step S22, the portable terminal 20 receives the floor plan data, and displays the floor plan on the display unit 23 (step S23). FIG. 9 is a schematic view illustrating an imaging location selection screen on which a floor plan is displayed. FIG. 10 is a schematic view illustrating a confirmation screen of the selected imaging location.

The photographer follows a message m32 displayed on the imaging location selection screen M3 shown in FIG. 9 and performs a predetermined operation (for example, a tap operation) at a position (for example, LDK) on the floor plan m31 corresponding to the location to be imaged Do. The portable terminal 20 selects a position on the floor plan m31 at which the operation is detected as an imaging location (step S24), and displays the pin m33 at this position (see FIG. 10).

In addition to the floor plan m31 and the pin m33, the imaging location confirmation screen M4 shown in FIG. 10 displays a message m34 for confirming the imaging location, a confirmation button m35, and a change button m36. When detecting a predetermined operation (for example, a tap operation) on the confirmation button m35, the portable terminal 20 determines the selected imaging location. Conversely, when detecting a predetermined operation (same as above) on the change button m36, the portable terminal 20 cancels the selection of the imaging location and displays the imaging location selection screen M3 shown in FIG. 9 again.

In the subsequent step S25, the portable terminal 20 acquires an image by causing the imaging device 30 to perform the first imaging. Specifically, as illustrated in FIG. 11, the portable terminal 20 first displays an imaging instruction screen (first time) M5 on which a message m37 instructing execution of imaging is displayed. When the photographer performs a shutter operation on the imaging device 30 according to the message m37, the imaging device 30 starts imaging and generates image data of a omnidirectional image in which a subject around the imaging device 30 is captured. The image data is transferred from the imaging device 30 to the portable terminal 20 via a communication cable or wireless communication.

Here, when imaging is performed, preferably, the imaging device 30 may be fixed using a tripod or the like. More preferably, the shutter operation may be performed from a position slightly away from the imaging device 30 by using the self-timer function of the imaging device 30 or by performing remote operation from the portable terminal 20 or the like. This is to reduce composition change and blurring during multiple imagings as much as possible.

FIG. 12 is a schematic view illustrating the state of the first imaging at the selected imaging location (for example, LDK). FIG. 13 is a schematic view illustrating the first image acquired when imaging is performed in the positional relationship shown in FIG. FIG. 13 shows a state in which a part of the subject captured in the omnidirectional image (for example, the subject captured in the area of the hemisphere) is projected on a plane.

It is preferable to perform imaging in a state where the imaging device 30 is installed at the center of the room as much as possible, as shown in FIG. In FIG. 12, the photographer 2 performs imaging in a state of standing on the left side of the imaging device 30 in the drawing. Therefore, as shown in FIG. 13, the image 2 'of the photographer is reflected at the left position of the image I1.

In the subsequent step S26, the portable terminal 20 acquires an image by causing the imaging device 30 to perform the second and subsequent imaging. Specifically, the portable terminal 20 first displays an imaging instruction screen (second and subsequent times) M6 illustrated in FIG. In the imaging instruction screen M6, a message m38 instructing movement of only the photographer to perform a second imaging is displayed without moving the camera (imaging device 30). According to this message m38, the photographer moves himself while maintaining the position of the imaging device 30, and when the shutter operation is performed in this state, the imaging device 30 starts imaging and the subject around the imaging device 30 is Generate image data of the photographed omnidirectional image. This image data is also transferred from the imaging device 30 to the portable terminal 20.

FIG. 15 is a schematic view illustrating the state of the second imaging. FIG. 16 is a schematic view illustrating a second image acquired when imaging is performed in the positional relationship shown in FIG. FIG. 16 shows a state in which a part of the subject captured in the omnidirectional image (for example, the subject captured in the area of the hemisphere) is projected on a plane. In FIG. 15, the photographer 2 performs imaging in a state where the imaging device 30 stands on the right side of the figure. Therefore, as shown in FIG. 16, the image 2 'of the photographer is reflected at the position on the right side of the image I2.

Note that, when imaging is performed by a plurality of staff members including the photographer, each staff also moves between the imaging operations each time as in the case of the photographer 2. In short, it is only necessary to acquire a plurality of images having different relative positions between the imaging device 30 and the staff.

FIG. 17 is a schematic view illustrating the imaging confirmation screen displayed on the portable terminal 20. The imaging confirmation screen M7 illustrated in FIG. 17 includes a message m39 asking whether imaging is successful, a success button m40, and a redo button m41. When detecting a predetermined operation (for example, a tap operation) on the redo button m41, the portable terminal 20 erases the image data regarding the imaging location transferred from the imaging device 30. In this case, the photographer can redo imaging in a place where the pin m33 is displayed on the floor plan m31.

On the other hand, when the portable terminal 20 detects a predetermined operation (same as above) for the success button m40, the image data storage unit 252 associates image data generated by performing imaging a plurality of times at the imaging location as one group. Remember.

The number of times of imaging in step S26 can be set as appropriate. For example, in the case of automatically performing image processing for deleting the image of the photographer from the omnidirectional image, imaging is performed at least three times in total (including step S25). In addition, in the case of manually selecting the image of the photographer from the omnidirectional image, the imaging may be performed at least twice in total.

In the subsequent step S27, the mapping processing unit 264 maps the image data transferred from the imaging device 30 on the floor plan data. Thereby, the transferred image data is associated with the position specified by the pin m33 on the floor plan m31.

FIG. 18 is a schematic view illustrating an imaging location change screen displayed on the portable terminal 20 after the end of imaging at one imaging location. The imaging location change screen M8 illustrated in FIG. 18 includes, in addition to the floor plan m31, a message m42 for selecting the next imaging location, and an end button m43. When detecting the operation on the floor plan m31, the portable terminal 20 displays an imaging location confirmation screen (see FIG. 10) in which the pin m33 is displayed at the position according to the operation. The photographer captures an image of all the rooms of the real estate property by repeating such an operation.

On the other hand, when detecting the operation on the end button m43, the portable terminal 20 displays an imaging completion confirmation screen illustrated in FIG. The imaging completion confirmation screen M9 shown in FIG. 19 includes a message m44 asking whether imaging of all rooms is completed, a YES button m45, and a NO button m46. When detecting the operation on the NO button m46, the portable terminal 20 displays the imaging location selection screen M3 shown in FIG. 9 again. Thus, the photographer can continue imaging.

On the other hand, when detecting the operation on the YES button m45, the portable terminal 20 subsequently displays a transmission confirmation screen of image data shown in FIG. The transmission confirmation screen M10 shown in FIG. 20 includes a message m47 asking whether to transmit image data, a button m48 for inputting an instruction to transmit now, and a button m49 for inputting an instruction to be transmitted later. And. When detecting the operation on the button m 48, the portable terminal 20 transmits the image data of the property transferred from the imaging device 30 and the floor plan data after the mapping process to the server 10 via the communication network N. On the other hand, when the portable terminal 20 detects an operation on the button m49, it does not transmit image data etc. on the spot, and stores the image data of the property and the floor plan data after the mapping process in the image data storage unit 252 . In this case, the portable terminal 20 transmits the image data and the floor plan data to the server 10 when receiving the transmission instruction separately.

Here, the photographer may transmit the image data and floor plan data of the property to the server 10 at the site of the real estate property, or may transmit when the communication environment with the server 10 is established. For example, when the number of rooms to be imaged is large, it is expected that data transmission will take a long time, so it is preferable to use the latter. Alternatively, the image data may be directly transmitted from the portable terminal 20 to the server 10, or the image data may be transmitted from the portable terminal 20 to the computer via the communication cable, and the image data may be transmitted from the computer to the server 10. Also good.

In step S28, the portable terminal 20 transmits the image data of the property and the floor plan data after the mapping process to the server 10. In step S13, the server receives these data.

In the subsequent step S14, the property information updating unit 133 of the server 10 updates the floor plan data stored in the image data storage unit 122 with the floor plan data after the mapping process received in step S13.

In the subsequent step S15, the image processing unit 134 performs image processing on the image based on the image data of the property received in step S13. Specifically, the image combining unit 135 performs processing of erasing the image of the photographer from a plurality of omnidirectional images obtained by imaging the same subject (the same room) a plurality of times.

Here, the process of erasing the image of the photographer will be described with reference to FIGS. 13, 16, 21 and 22.
The image I1 shown in FIG. 13 and the image I2 shown in FIG. 16 are images obtained by imaging the same subject while changing the position of the photographer with respect to the imaging device 30. Therefore, in each of the images I1 and I2, only the position of the image 2 'of the photographer reflected in each image is different, and the images of the other objects are common. Therefore, the image 2 'of the photographer can be extracted by detecting the area (for example, the area 3 shown in FIG. 21) in which there is a change between the image I1 and the image I2. Specifically, the pixel value of the pixel located at the coordinate x of the image I1 is p (x), and the pixel value of the pixel located at the coordinate x of the image I2 is q (x) Calculate the difference of A region where this difference is equal to or greater than a predetermined threshold value Th, that is, a region satisfying the following expression (1) is a region where there is a change between the image I1 and the image I2.
| P (x) -q (x) | ≧ Th (1)

Therefore, the image calculated by | p (x) −q (x) | is binarized by the threshold value Th, and contour tracking is performed on the obtained binary image to obtain the image 2 ′ of the photographer and the original subject. Can be separated and extracted.

As the actual processing, using the three omnidirectional images (hereinafter referred to as an image A, an image B, and an image C) obtained by imaging the same subject at least three times, the images A and B and The area of the image of the photographer in the image B is extracted by taking the AND of the image of the photographer extracted from the difference image of and the image of the photographer extracted from the difference image of the image B and the image C. Then, an image of the same area in another image (image A or C) is fitted to the area of the image of the photographer extracted from the image B. That is, the pixel values of pixels in the region of the image of the photographer in image B are replaced with the pixel values of pixels in the corresponding region in another image. Thereby, an image I3 in which the image of the photographer is erased as shown in FIG. 22 can be obtained.

In addition, when the imaging device 30 is fixed using a tripod etc. at the time of imaging, about the area | region directly under the imaging device 30, a difference does not arise between several images. In this case, the pixel value of the pixel in the area where the predetermined range including the area directly under the imaging device 30 is further added to the image from which the image of the photographer has been erased by the image processing as described above Execute image processing to replace with the pixel value of the pixel of. As a result, the entire area has the same texture as that in the vicinity of the area, and the image of the tripod or the like supporting the imaging device 30 can be erased. At this time, the range of the region in which the pixel value is to be replaced may be set in advance.

In the subsequent step S16, the VR image data generation unit 136 generates VR image data using the mapping information in the floor plan data, based on the image data of the omnidirectional image from which the image of the photographer has been deleted. Furthermore, in step S17, the VR image data generation unit 136 stores the generated VR image data in the image data storage unit 122 in association with the property ID. Thus, the operations of the server 10 and the portable terminal 20 end.

As described above, according to the first embodiment of the present invention, the photographer only has to change the position with respect to the imaging device 30 at each imaging location and perform imaging a plurality of times. The imaging can be completed. Further, according to the first embodiment, the image 2 'of the photographer can be erased from the omnidirectional image by simple arithmetic processing among a plurality of images different in the position of the photographer. Therefore, it is possible to easily acquire an omnidirectional image without reflection of the photographer or the like.

In the first embodiment, the imaging is performed twice on the same subject, but the number of imaging times is not limited to two. For example, when a plurality of staff members including a photographer are in the field, the image of the staff can be more reliably erased from the omnidirectional image by performing imaging three or more times while changing the staff positions .

In the first embodiment, the server 10 performs image processing for removing unnecessary portions such as the photographer's image from the omnidirectional image, but the image processing is performed on the portable terminal 20 side. Also good. In the first embodiment, as described above, since unnecessary portions can be deleted from the omnidirectional image by simple arithmetic processing, the image processing can be performed with a small load even in the portable terminal 20. In this case, the image data of the omnidirectional image from which the image of the photographer is erased may be transferred from the portable terminal 20 to the server 10, and the server 10 may generate the VR image data using the received image data as it is .

(First modification)
In the first embodiment, the server 10 automatically performs the image processing for deleting unnecessary areas (hereinafter, also simply referred to as unnecessary areas) such as the image of the photographer from the omnidirectional image, but The above image processing may be performed based on information input from a terminal device that has accessed the server 10. The terminal device includes a communication interface for communicating with the server 10, a display unit, an operation unit having a display control function for controlling screen display in the display unit, etc., and a signal corresponding to an operation performed from the outside. A general-purpose device having an input device (mouse, touch panel, keyboard, etc.) for input can be used. Specifically, a personal computer, a notebook PC, a tablet terminal or the like is used.

FIG. 23 is a schematic view illustrating an operation screen for allowing the user (operator of the terminal device) to select the unnecessary area, which is displayed on the display unit provided in the terminal device that has accessed the server 10. FIG. 24 is a schematic diagram for explaining the image processing in the first modification. In the following example, imaging may be performed at least twice for one imaging location.

The operation screen M11 shown in FIG. 23 is one of the images (see steps S25 and S26 in FIG. 6) acquired by imaging the same subject (one imaging location) multiple times (image in FIG. 23). It includes I1), a message m50 for prompting selection of the unnecessary area, and an OK button m51. Here, in FIG. 23, the image I1 is displayed in a state where a part of the subject captured in the omnidirectional image (for example, the subject captured in the region of the hemisphere) is projected on a plane. In the operation screen M11, the image I1 may be scrolled by performing a predetermined operation (for example, a slide operation) on the image I1 by an input device provided in the terminal device.

The user visually finds the image 2 'of the photographer from the image I1 and visually detects the image 2 of the photographer by a predetermined operation (for example, a drawing operation using a mouse or a touch panel) on the operation screen M11. Enclose the area including '(refer to the enclosing line 4), and select the OK button m51. When the terminal device detects an operation surrounding the area in the image I1 and an operation on the OK button m51 from the signal input from the input device, the terminal apparatus determines that the enclosed area is selected as an unnecessary area, and Position information (coordinate information) indicating the position of the area is transmitted to the server 10. The area to be selected may be one or plural.

In response to this, the image processing unit 134 of the server 10 selects the pixel value of the pixel in the area selected in the image I1 on the basis of the position information received from the terminal device. It replaces with the pixel value of the pixel (refer area | region 4 ') in the corresponding area | region in the image I2 shown at 24. FIG. Thus, an image (see FIG. 22) from which the image 2 'of the photographer is erased can be obtained.

According to this modification, since the user selects an area to be deleted from the image, an image from which the unnecessary area is deleted can be obtained by simpler image processing.

(Second modification)
In the first modification, the unnecessary area such as the image 2 'of the photographer is selected by surrounding it with the locus 4, but the method of selecting the unnecessary area is not limited to this. For example, when the image is to be scrolled by performing a slide operation on the image (a part of the omnidirectional image) displayed on the operation screen, an operation to select the unnecessary area is It is preferable to minimize the operation to the area in the image.

25 and 26 are schematic views illustrating another operation screen for causing the user to select an unnecessary area. These operation screens are also displayed on the display unit provided in the terminal device that has accessed the server 10, as in the first modification.

In the operation screen M12 shown in FIG. 25, when a drawing operation of tracing the arc 5 is performed on the image I1 using an input device such as a mouse or a touch panel, the fan-shaped area 6 including the arc 5 is selected. It is When the user finds the image 2 'of the photographer from the image I1, the user traces the arc 5 near the image 2' of the photographer. Thereby, the fan-shaped area 6 including the image 2 'of the photographer is selected as the unnecessary area. In this case, the user can easily perform the area selection operation without touching the inside of the image I1.

The operation screen M13 shown in FIG. 26 divides the image I1 into a plurality of areas 7a to 7h in advance to display unnecessary areas in units of divided areas. The user performs a predetermined operation (for example, double-click operation or double-tap) on a desired area (for example, the area 7g including the image 2 'of the photographer) of the divided areas 7a to 7h using the input device Do the operation). Thereby, the area where the operation is performed (for example, the area 7g) is selected as the unnecessary area. In this case, the user can easily perform the area selection operation with the minimum operation on the image I1.

Although the image I1 is divided into sectors in FIG. 26, the division method is not limited to this. For example, the image I1 may be divided into sectors and may be divided radially, or may be divided into grids.

(Third modification)
In the first modified example, a circular image obtained by projecting the omnidirectional image on a plane is displayed on the operation screen for causing the user to select the unnecessary region, but the method of displaying the omnidirectional image is the same. It is not limited to.

FIG. 27 is a schematic view illustrating still another operation screen for causing the user to select an unnecessary area. This operation screen is also displayed on the display unit provided in the terminal device that has accessed the server 10, as in the first modification.

On the operation screen M14 shown in FIG. 27, there is displayed a rectangular image I2 'obtained by coordinate-converting an omnidirectional image (for example, an image I2 shown in FIG. 16) into an orthogonal coordinate system. The icons m52 and m53 are displayed on the left and right of the image I2 ′, respectively, and the image I2 ′ is displayed by performing a predetermined operation (for example, click or tap operation) on the icons m52 and m53. You can scroll left and right. Alternatively, the image I2 'may be scrolled by directly performing a predetermined operation (e.g., a slide operation) on the image I2'.

The method of selecting an unnecessary area for the image I2 'is not particularly limited. For example, as in the first modification, the unnecessary area may be selected by surrounding a desired area in the image I2 '. Alternatively, as in the second modification, a rectangular area including a traced line may be selected as an unnecessary area by horizontally tracing the upper side or the lower side of the image I2 ′ or the image I2 ′. . Furthermore, the image I2 'may be divided in advance into a plurality of areas, and the unnecessary area may be selected in the divided area units.

(The 4th modification)
In the first to third modifications, although the user visually finds the image 2 'of the photographer from the image, the candidate area of the image 2' of the photographer is automatically detected, and from among the candidate areas The unnecessary area may be selected by the user. In this case, the image processing unit 134 of the server 10 takes the difference between the plurality of images acquired by imaging the same subject a plurality of times based on the image data transmitted from the portable terminal 20. The candidate area to be seen as the image 2 'of the image is detected in advance. The candidate area can be detected by performing contour tracking on the area satisfying Expression (1), as in the first embodiment.

FIG. 28 is a schematic view illustrating still another operation screen for causing the user to select an unnecessary area. This operation screen is also displayed on the display unit provided in the terminal device that has accessed the server 10, as in the first modification.

On the operation screen M15 shown in FIG. 28, one of a plurality of images (for example, the image I1 shown in FIG. 13 and the image I2 shown in FIG. 16) acquired by imaging the same subject a plurality of times In FIG. 28, an image I1) is displayed. Further, marks 8a and 8b indicating candidate areas detected in the server 10 are superimposed and displayed on the image I1. Among these, the mark 8a corresponds to the image 2 'of the photographer appearing in the image I1, and the mark 8b corresponds to the image 2' of the photographer appearing in the image I2. The type of mark is not limited to the contour of the candidate area as shown in FIG. For example, it may be an object in which the entire candidate area is filled, or may be an arrow pointing to the position of the candidate area.

When detecting an operation on any of the marks 8a and 8b on the operation screen M15, the portable terminal 20 transmits, to the server 10, information indicating the mark (for example, the mark 8a) on which the operation has been performed. In response to this, the image processing unit 134 of the server 10 corresponds the pixel value of the pixel of the area corresponding to the mark 8a in the image I1 displayed on the operation screen M15 to the position in another image I2. Replace with the pixel value of the selected pixel. Thus, an image (see FIG. 22) from which the image 2 'of the photographer is erased can be obtained.

According to this modification, a region where there is a change between a plurality of images acquired by imaging the same subject multiple times is automatically detected as a candidate region, thereby reducing the oversight of unnecessary regions by the user's visual observation. be able to. Further, according to the present modification, since the user is made to select the unnecessary area to be erased from the candidate areas, it is possible to suppress unnecessary image processing.

Second Embodiment
Next, a second embodiment of the present invention will be described.
In the first embodiment, after acquiring a plurality of omnidirectional images by imaging the same subject a plurality of times (see steps S25 and S26 in FIG. 6), the image 2 of the photographer is obtained from the omnidirectional images. Performed image processing to erase unnecessary areas such as'. However, the image processing may be performed on the portable terminal 20 in parallel with the imaging operation. In this case, the operation unit 26 of the portable terminal 20 also functions as an image processing unit by reading the image processing program stored in the program storage unit 251. Hereinafter, the operation of the imaging system 1 in the case of performing image processing in the portable terminal 20 will be described.

FIG. 29 is a flowchart showing the operation of the imaging system according to the second embodiment of the present invention. Among these, steps S10 to S14, S16, S17, and steps S20 to S24 are the same as in the first embodiment.

In step S30 following step S24, the portable terminal 20 acquires an image by causing the imaging device 30 to execute the first imaging. The details of step S30 are the same as step S25 of FIG. 6 (see FIGS. 11 to 13). When the photographer performs a shutter operation of the imaging device 30, the imaging device 30 starts imaging and generates image data of a omnidirectional image in which a subject around the imaging device 30 is captured. This image data is transferred to the portable terminal 20 via the communication cable or by wireless communication.

In the subsequent step S31, the portable terminal 20 displays the image acquired in step S30 on the display unit 23. Specifically, operation unit 26 (display control unit 262) creates the unnecessary area selection screen shown in FIG. 30, and arranges an image based on the image data transferred from imaging device 30 in the screen to display the display unit. Display on 23 The unnecessary area selection screen M16 shown in FIG. 30 includes the image I4 acquired in step S30, a message m60 for causing the user to select the unnecessary area, and an OK button m61.

The photographer visually searches an area to be erased from the image I4, that is, an area (unnecessary area) in which the photographer himself is reflected, and performs an operation of selecting the unnecessary area on the portable terminal 20. The operation of selecting the unnecessary area is not particularly limited. For example, as in the first modification, a method may be employed in which a region including an unnecessary region is surrounded by a line, or in the same manner as in the second modification, a method of tracing the arc of a fan-shaped region including an unnecessary region It is also possible to select one of a plurality of regions divided in advance. Further, in the unnecessary area selection screen M16, as in the third modified example, an image in which the omnidirectional image is coordinate-transformed into an orthogonal coordinate system may be displayed. Furthermore, the region to be selected may be one or more. FIG. 30 shows a state in which an area including the image 2 'of the photographer in the image I4 is surrounded by an encircling line 9. In FIG.

When the portable terminal 20 receives a signal for selecting one or more regions in the image I4 and further detects that the OK button m61 has been subjected to a predetermined operation (for example, a tap operation), the mobile terminal 20 is selected in the image I4. The position information (coordinate information) of the unnecessary area is stored (step S32).

In the subsequent step S33, the portable terminal 20 acquires an image by causing the imaging device 30 to perform imaging again. Specifically, the calculation unit 26 first causes the display unit 23 to display an imaging instruction screen shown in FIG. On the imaging instruction screen M17, a message m62 for instructing the photographer to move and perform imaging is displayed. When the photographer moves himself / herself according to the message m62 and performs the shutter operation of the imaging device 30, the imaging device 30 starts imaging, and an image of the omnidirectional image in which the subject around the imaging device 30 is taken Data is generated and transferred to the mobile terminal 20.

In the following step S34, the portable terminal 20 performs image processing to combine the image acquired in step S30 and the image acquired in step S33. In detail, the calculation unit 26 as an image processing unit acquires the pixel value of the pixel at the position of the unnecessary area stored in step S32 from the image acquired in step S33. Then, the pixel values of the pixels in the unnecessary area in the image I4 acquired in step S30 are replaced with these pixel values.

In the following step S35, the portable terminal 20 displays the image subjected to the image processing in step S34. In detail, the calculation unit 26 creates an image confirmation screen shown in FIG. 32, arranges the image processed image in this screen, and causes the display unit 23 to display the image. An image confirmation screen M18 shown in FIG. 32 includes an image I5 subjected to image processing, a message m64 for allowing the user to confirm the presence or absence of the unnecessary area, and buttons m65 and m66 for inputting the confirmation result of the unnecessary area. . The photographer visually checks the image I5, and when there is no unnecessary area to be further erased, performs a predetermined operation (for example, a tap operation) on the button m65. On the other hand, when there is an unnecessary area to be deleted, a predetermined operation (same as above) is performed on the button m66.

When the portable terminal 20 detects an operation indicating that there is an unnecessary area (step S36: Yes), the operation of the portable terminal 20 returns to step S31. In this case, the portable terminal 20 again creates the unnecessary area selection screen (see FIG. 30), and displays the image I5 on which the image processing has been performed on this screen. Thereby, the photographer can further select an unnecessary area from the image I5.

On the other hand, when the portable terminal 20 detects an operation indicating that there is no unnecessary area (step S36: No), the computing unit 26 (mapping processing unit 264) generates floor plan data of image data corresponding to the image-processed image I5. Are mapped (step S37). Thereby, the image data of the image-processed image I5 is associated with the position specified by the pin m33 on the floor plan m31.

In the subsequent step S38, the portable terminal 20 transmits the image data of the property subjected to the image processing and the floor plan data after the mapping processing to the server 10.

The server 10 receives the image data and the floor plan data of the property (step S13), and updates the floor plan data (step S14). Then, VR image data is generated using the image data subjected to image processing received in step S13 (step S16), and the generated VR image data is stored.

As described above, according to the second embodiment of the present invention, the photographer can proceed with the imaging operation while confirming the presence or absence of the unnecessary region in the image at the site of the property. Therefore, it becomes possible to obtain an image without omission omission of the unnecessary area.

Next, a modification of the second embodiment will be described. In the present modification, the portable terminal 20 (display control unit 262) sets the image of the area corresponding to the unnecessary area in the image acquired in step S33 to the unnecessary area selected in the image I4 (see steps S31 and S32). A simple process of only superposing and displaying a part is performed (steps S34 and S35). Then, in step S38, the portable terminal 20 transmits, to the server 10, the image data before image processing acquired in steps S30 and S33 and the position information (coordinate information) of the unnecessary area stored in step S32. . The server 10 replaces the unnecessary area by replacing the pixel value of the pixel of the unnecessary area in one image with the pixel value of the pixel of the corresponding area in another image based on the received image data and the position information. Execute image processing to delete.

According to this modification, since the imaging operation is performed while confirming the presence or absence of the unnecessary area, it is possible to reliably acquire an image necessary for erasing the unnecessary area and to reduce the calculation load in the portable terminal 20. It becomes possible.

Third Embodiment
Next, a third embodiment of the present invention will be described. FIG. 33 is a flowchart showing the operation of the imaging system according to the third embodiment of the present invention. FIG. 34 and FIG. 35 are schematic diagrams for explaining image processing in the third embodiment of the present invention.

In the imaging system according to the present embodiment, the image processing for deleting an unnecessary area such as the image of the photographer from the omnidirectional image is different from the first and second embodiments. In the following, a case is described in which the mobile terminal 20 performs image processing for deleting an unnecessary area from the omnidirectional image, but after the image processing is stored in the server 10, the terminal device accesses the server 10 You may do it.

In FIG. 33, steps S10 to S14, S16, S17 and steps S20 to S24 are the same as in the first embodiment. In step S40 following step S24, the portable terminal 20 acquires two or more images generated by causing the imaging device 30 to perform imaging a plurality of times at one imaging location. At this time, the position of the unnecessary area is changed between the images by, for example, moving the photographer each time an imaging operation is performed at one imaging location.

In the subsequent step S41, the portable terminal 20 superimposes two or more images photographed at the same place and displays the superimposed image on the display unit 23. At this time, the orientations of the images are aligned so that the positions of subjects other than the unnecessary area overlap between two or more images. FIG. 34 shows a state in which the image I1 shown in FIG. 13 and the image I2 shown in FIG. 16 are superimposed. In FIG. 34, since the image I2 is superimposed on the image I1, the image 2 '(broken line part) of the photographer in the image I1 is not shown on the screen, and the image 2' of the photographer in the image I2 (Solid line part) is reflected on the screen.

In the following step S42, the portable terminal 20 determines whether or not an operation indicating that there is an unnecessary area in the displayed image is detected. In this determination, for example, as in FIG. 32, two buttons for causing the user to input the presence or absence of an unnecessary area are displayed on the screen, and it is detected whether an operation (for example, a tap operation) is performed on any of the two buttons. Can be done by

When detecting an operation indicating that there is an unnecessary area (step S42: Yes), the portable terminal 20 selects a portion (erased portion) to be erased in the image displayed on the screen according to the operation performed on the screen (Step S43). Specifically, when detecting a swipe operation of swiping the screen with the finger, the portable terminal 20 selects the swiped area as the erased portion. For example, in FIG. 34, when an operation of tracing the area A1 including the image 2 'of the photographer of the image I2 is detected, the area A1 is selected as the erasure portion. The erased portion may be on a subject (a door on both sides of the image 2 'of the photographer in FIG. 34) which is originally desired to be left in the image.

In the following step S44, the portable terminal 20 makes the portion selected in step S43 transparent. That is, the transmittance of the selected part is set to 100%. As a result, the lower layer image is displayed on the screen at that portion. For example, as shown in FIG. 35, by making the selected part A2 transparent, the image 2 'of the photographer shown in the image I2 and the image around it disappear from the screen, and the image I1 of the lower layer comes to be visible . Thereafter, the process returns to step S41.

When three or more images are superimposed, when an operation (swipe operation) is further performed to select an erased portion for a portion that has been made transparent, an image (for example, one sheet) that has been subjected to a transparency processing A process is performed to make the selected portion of the image (for example, the second image) directly below the eye image transparent. Thereby, the corresponding portion of the lower layer (for example, the third sheet) is displayed on the screen.

Moreover, in step S42, when the portable terminal 20 does not detect an operation indicating that there is an unnecessary area (step S42: No), the process proceeds to step S37. In subsequent steps S37 and S38, the two or more superimposed images are treated as a set.

As described above, according to the third embodiment of the present invention, the user (photographer) can erase the unnecessary area by intuitive operation on the screen using his / her finger or the like. In addition, even if a subject that should originally be left in the image is selected as the erased portion among the subjects captured in the upper layer image, the same subject captured in the lower layer image is displayed on the screen. Mistakes can be reduced. Therefore, the user can easily erase the unnecessary area by rough operation.

In the above first to third embodiments and their modifications, although the case of generating the image for the inside of the real estate property has been described as an example, the present invention is also applicable to the surroundings by the omnidirectional camera. The present invention can be applied to the case of acquiring various images that want to capture an object at once but do not want to capture the image of the photographer in the image. As a specific example, an image for preview of a facility such as a wedding hall, an image for virtual experience of a sightseeing spot for sightseeing guidance, and the like can be mentioned.

As an example, when acquiring an image for sightseeing guidance, it is difficult to acquire an image of only a tourist facility or a landscape in which a person is not reflected, because a tourist visits a famous tourist spot without exception. In such a case, a plurality of images are acquired by imaging a tourist facility or a landscape a plurality of times at appropriate timings. At this time, the photographer also moves. Then, using these images, processing may be performed to delete an area in which a change occurs between the images, that is, an area in which a tourist, a passerby, or the like is captured. As a result, it is possible to easily acquire an image in which only a tourist facility or a landscape is shown.

The image processing method may be automatically performed in the server 10 as in the first embodiment, or an image displayed on the terminal device as in the first or second modification or the second embodiment. The method may be such that the user selects the unnecessary area, or the user may select the unnecessary area from the candidate areas of the unnecessary area detected in the server 10 as in the fourth modification. Alternatively, as in the second embodiment, imaging, selection of an unnecessary area from an image, and image processing or display of a superimposed image may be repeated at an imaging location. In the latter case, since it is possible to confirm whether or not the unnecessary area can be erased at the place where the imaging is performed, it is possible to suppress the erasure leakage of the unnecessary area due to the image processing. Alternatively, it is possible to reduce the time and effort for performing imaging more in preparation for failure of image processing.

As another example, when taking a memorial photo at a travel destination, a tourist visits a famous tourist spot, so I get an image showing only myself (photographer himself) in the background of a tourist facility or landscape. It is difficult to do. In such a case, the composition of the person to be left in the background and the photograph is determined, the timing is appropriately opened, and imaging is performed multiple times with the same composition. At this time, the photographer does not have to move as well. Then, by using the plurality of acquired images and performing processing to delete the area in which changes occur between the images, it is possible to easily acquire an image in which only the background and the images appear.

In these cases, the image data of the omnidirectional image may be mapped to the image data of the image that specifies the position of the imaging location, like the floor plan data of the real estate property. Specifically, map data, image data of a sketch of a tourist facility, etc. may be mentioned.

The present invention is not limited to the above first to third embodiments and modifications, and various combinations of plural components disclosed in the above first to third embodiments and modifications can be made. Invention can be formed. For example, some components may be excluded from all the components shown in the first to third embodiments and the modification, or the components shown in the first to third embodiments and the modification These may be combined appropriately.

DESCRIPTION OF SYMBOLS 1 imaging system 2 photographer 2 'image of a photographer 3, 4'7a-7h area | region 4, 9 enclosure line 5 circular arc 6 sectoral area 8a, 8b mark 10 server 11 communication part 12 memory | storage part 13 calculating part 20 portable terminal 21 communication Unit 22 Input unit 23 Display unit 24 Reading unit 25 Storage unit 26 Operation unit 30 Imaging device 30a, 30b Super wide-angle lens 30c Housing 31a, 31b Imaging unit 32 Communication unit 33 Storage unit 34 Operation unit 35 Control unit 121 Program storage unit 122 Image data storage unit 123 Property information database 131 Property information registration unit 132 Property information extraction unit 133 Property information update unit 134 Image processing unit 135 Image combining unit 136 VR image data generation unit 251 Program storage unit 252 Image data storage unit 261 Acquisition of identification information Part 262 Display control part 263 Image data acquisition 264 mapping processing unit 351 image processing unit

Claims (16)

An imaging device for generating an image in which an object included in a range of 360 ° in all directions is captured by one imaging operation;
An image data acquisition unit that acquires image data of a plurality of images generated by causing the imaging device to capture the same subject a plurality of times, the images being different in position of an object other than the subject with respect to the imaging device When,
An image processing unit that generates an image in which the image of the object is erased by combining the plurality of images;
An imaging system comprising: A server that manages images,
A mobile terminal connected to the server via a communication network,
Equipped with
The image data acquisition unit is provided in the portable terminal.
The image processing unit is provided in the server,
The portable terminal transmits image data of the plurality of images acquired by the image data acquisition unit to the server.
The server receives image data of the plurality of images, and causes the image processing unit to execute image processing based on the image data.
The imaging system according to claim 1. The imaging system according to claim 2, wherein the image processing unit further generates image data for virtual reality using image data of an image from which the image of the object is erased. A server that manages images,
A mobile terminal connected to the server via a communication network,
Equipped with
The image data acquisition unit and the image processing unit are provided in the portable terminal.
The portable terminal transmits, to the server, image data of an image from which the image of the object generated by the image processing unit is erased.
The imaging system according to claim 1. The imaging system according to claim 4, wherein the server further includes a second image processing unit that generates image data for virtual reality using image data of an image in which an image of the object is erased. The portable terminal is
A display unit,
An input device for inputting a signal according to an operation performed from the outside;
When one of the plurality of images is displayed on the display unit, and a signal for selecting at least one region in the one image is input from the input device, the signal in the one image A display control unit configured to superimpose a region corresponding to the at least one region in another image of the plurality of images on the at least one region and to display the region on the display unit;
, And further transmitting to the server position information representing the position of the at least one region in the one image;
The image processing unit generates the image in which the image of the object is erased, using the position information in addition to the image data of the plurality of images.
The imaging system according to claim 2 or 3. The portable terminal is
A display unit,
An input device for inputting a signal according to an operation performed from the outside;
A display control unit that causes one of the plurality of images to be displayed on the display unit;
And have
The image processing unit adds, to the image data of the plurality of images, the at least one of the one image when a signal for selecting at least one region in the one image is input from the input device. Generating an image in which the image of the object is erased, using position information representing the position of the area;
The imaging system according to claim 4 or 5. The portable terminal acquires image data of the other image from the imaging device when a signal for selecting at least one region in the one image is input from the input device. The imaging system described in. The portable terminal is
A display unit,
A first imaging instruction screen for instructing to execute the first imaging by the imaging device, and changing the position of the object after the first imaging is completed, and performing the second imaging by the imaging device A display control unit that causes the display unit to sequentially display a second imaging instruction screen for instructing the
The imaging system according to any one of claims 2 to 5, further comprising The image processing unit extracts an area in which a change in pixel value occurs among the plurality of images, and the pixel value of the area in one image of the plurality of images is set to one of the plurality of images. Replace with the pixel value of the area in another image
The imaging system according to any one of claims 1 to 5. A display unit,
An input device for inputting a signal according to an operation performed from the outside;
A display control unit that causes one of the plurality of images to be displayed on the display unit;
And further
When the image processing unit receives a signal for selecting at least one region in the one image from the input device, the image processing unit may generate pixel values of the at least one region in the one image by the plurality of pixels. Replace the pixel value of the area corresponding to the at least one area in another image of the images,
The imaging system according to any one of claims 1 to 5. A display unit,
An input device for inputting a signal according to an operation performed from the outside;
A display control unit that causes one of the plurality of images to be displayed on the display unit;
And further
The image processing unit extracts at least one region in which a change in pixel value occurs among the plurality of images,
The display control unit superimposes and displays at least one mark respectively representing the at least one area extracted by the image processing unit on the one image displayed on the display unit.
The image processing unit is configured such that, when a signal for selecting at least one of the at least one marks is input from the input device, a pixel of an area corresponding to the at least one of the marks in the one image. Replace the values with pixel values of corresponding regions in other ones of the plurality of images,
The imaging system according to any one of claims 1 to 5. A display unit,
An input device for inputting a signal according to an operation performed from the outside;
A display control unit configured to superimpose the plurality of images and display the plurality of images on the display unit;
And further
The image processing unit makes the region transparent when a signal for selecting a region in the image of the top layer displayed on the display unit among the plurality of images is input from the input device.
The imaging system according to any one of claims 1 to 5. The imaging system according to any one of claims 6, 7, 11, and 12, wherein the display control unit causes the display unit to display an image obtained by projecting the one image on a plane. The imaging system according to any one of claims 6, 7, 11, and 12, wherein the display control unit causes the display unit to display an image obtained by coordinate-transforming the one image into an orthogonal coordinate system. A storage unit that stores image data of an image that specifies the position of an imaging location;
A mapping processing unit that maps the image data of the plurality of images to the image data of the image specifying the position;
The imaging system according to any one of claims 1 to 15, further comprising:

Images