JP7720535B2 - Photographing system, photographing device, and photographing control method - Google Patents

Description

Article 30, Paragraph 2 of the Patent Act applies 1. Date of exhibition: July 12, 2023 2. Exhibition name and location: GEMBA SOLUTION 2023, Osaka 3. Exhibitors: Toguchi Yota, Horiki Toshio, Hashimura Momo 4. Contents of exhibit: At GEMBA SOLUTION 2023, Toguchi Yota, Horiki Toshio, and Hashimura Momo exhibited a photographic device invented by Toguchi Yota, Horiki Toshio, and Hashimura Momo.

The present invention relates to an imaging system, imaging device, and imaging control method that includes an imaging device that captures images of a measurement target location and a processor that controls the imaging device, in order to perform 3D measurement processing that generates 3D spatial information about the measurement target location.

A 3D measurement technique is known that generates a point cloud representing a measurement target in three-dimensional space based on captured images of the target. In recent years, the SLAM (Simultaneous Localization and Mapping) method has been attracting attention for this type of 3D measurement. SLAM involves having a mobile object (e.g., a worker) hold a camera and capturing images of the target from various angles. Based on these captured images, the method performs self-localization to obtain the device's own position information, and mapping to generate a point cloud representing the target in three-dimensional space.

A known technology related to this type of SLAM method is one that presents areas with insufficient photography to the user using an image of an arrow indicating the direction and speed of movement of the imaging device (sensor) held by the user, or an image that distinguishes between the direction of measured areas and the direction of unmeasured areas (see Patent Document 1).

Patent No. 6489566

According to conventional technology, a tablet PC (the measurement device itself) equipped with a distance sensor and visible camera is used as the imaging device for capturing images of the measurement location, and the user captures the measurement location by holding the imaging device with both hands and walking through the measurement location. The user can also use the touch panel display provided on the imaging device to operate the screen to give instructions and make settings related to the imaging.

However, there are times when a user wants to operate the screen to give instructions or make settings related to shooting while shooting. In this case, operating the screen while shooting is cumbersome when the user holds the camera with both hands, as in conventional technology. In other words, since the user needs to remove the hand holding the camera from the camera in order to operate the screen, it becomes difficult to hold the camera stably and it is difficult to operate the screen.

The primary objective of the present invention is to provide a photography system, photography device, and photography control method that allows users to easily operate the screen on a touch panel display to give instructions and make settings related to photography while holding the photography device stably, even during photography.

The photographing system of the present invention comprises a photographing device that photographs a location to be measured, and a control device connected to the photographing device and that performs three-dimensional measurement processing of the location to be measured , wherein the photographing device comprises a sensor unit including a camera that photographs the location to be measured, a first display unit that displays a screen used in a photographing mode to assist a user in their photographing work and includes a touch panel display that detects screen operations by the user, and a support body that has a grip unit that the user holds in one hand and supports the sensor unit and the first display unit, and the control device comprises a three-dimensional measurement processing unit that performs the three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured based on the photographing data acquired from the photographing device, a second display unit that displays a screen used in a non-photographing mode for performing operations related to the management and processing of the photographing data, and a display control unit that controls the display information of the screens to be displayed on the first display unit and the second display unit, and wherein, upon a user's operation to transition from the non-photographing mode to the photographing mode, the display control unit displays a pause screen on the second display unit and an operation screen on the first display unit that allows the user to at least instruct the start of photographing .

According to the present invention, when a user operates to transition from non-photography mode to photography mode, the display control unit displays a pause screen on the second display unit of the control device and also displays an operation screen on the first display unit of the photography device that allows the user to give at least an instruction to start photography, so that the transition to photography mode in the photography system can be confirmed and screen operations for giving instructions and setting related to photography on the touch panel display can be easily performed with the other hand.

FIG. 1 is an explanatory diagram showing a situation of a photographing operation performed by a user using the photographing system according to the first embodiment; An explanatory diagram showing standard shooting conditions and high-altitude shooting conditions FIG. 1 is a perspective view showing a support for the imaging device; FIG. 10 is a side view showing the expansion and contraction of the support of the imaging device; FIG. 1 is a perspective view showing a sensor unit of an imaging device; A block diagram showing the schematic configuration of an imaging device and a control device. A block diagram showing an overview of the processing performed by the processor of the control device. FIG. 10 is an explanatory diagram showing a list of photographed data displayed on the display of the control device. FIG. 10 is an explanatory diagram showing a detailed image data screen displayed on the display of the control device. An explanatory diagram showing a screen that pops up on the shooting data details screen FIG. 10 is an explanatory diagram showing the transition status of the main part of the imaging data details screen. FIG. 10 is an explanatory diagram showing a setting screen displayed on the display of the control device. FIG. 10 is an explanatory diagram showing a pause screen displayed on the display of the control device. FIG. 10 is an explanatory diagram showing a connection failure notification screen displayed on the display of the control device. FIG. 10 is an explanatory diagram showing a pause screen displayed on a touch panel display of the imaging device. FIG. 10 is an explanatory diagram showing a shooting standby screen displayed on a touch panel display of the imaging device. FIG. 10 is an explanatory diagram showing a screen displayed during imaging on a touch panel display of the imaging device. FIG. 10 is an explanatory diagram showing the transition of the available shooting time guide section on the shooting screen; An explanatory diagram showing the shooting screen with a message window displayed An explanatory diagram showing the contents and timing of messages displayed in the message window FIG. 10 is an explanatory diagram showing a confirmation screen for ending imaging displayed on a touch panel display of the imaging device. FIG. 10 is an explanatory diagram showing a self-location lost notification screen displayed on a touch panel display of the imaging device. FIG. 10 is an explanatory diagram showing a recovery screen displayed on a touch panel display of the imaging device. FIG. 10 is an explanatory diagram showing an enlarged screen of the final position acquisition point displayed on the touch panel display of the imaging device. FIG. 10 is an explanatory diagram showing a recovery failure notification screen displayed on a touch panel display of the imaging device. FIG. 10 is an explanatory diagram showing a connection failure notification screen displayed on a touch panel display of the imaging device. FIG. 10 is an explanatory diagram showing a confirmation screen for ending the imaging mode displayed on the touch panel display of the imaging device. FIG. 10 is a block diagram showing the schematic configuration of an imaging device and a control device according to a second embodiment.

A first invention made to solve the above problem is an imaging system comprising an imaging device that images a measurement target location and a control device connected to the imaging device and that performs three-dimensional measurement processing of the measurement target location , wherein the imaging device comprises a sensor unit including a camera that images the measurement target location, a first display unit including a touch panel display that displays a screen used in an imaging mode to assist a user in their imaging work and that detects screen operations by the user , and a support body that has a grip unit that the user holds in one hand and supports the sensor unit and the first display unit, and the control device comprises a three-dimensional measurement processing unit that performs the three-dimensional measurement processing to generate three-dimensional spatial information of the measurement target location based on imaging data acquired from the imaging device, a second display unit that displays a screen used in a non-imaging mode for performing operations related to managing and processing the imaging data, and a display control unit that controls the display information of the screens to be displayed on the first display unit and the second display unit, and wherein, upon a user's operation to transition from the non-imaging mode to the imaging mode, the display control unit displays a pause screen on the second display unit and an operation screen on the first display unit that at least allows the user to instruct the start of imaging .

According to this, when a user operates to transition from non-photography mode to photography mode, the display control unit displays a pause screen on the second display unit of the control device and also displays an operation screen on the first display unit of the photography device that allows the user to give at least an instruction to start photography, so that the transition to photography mode in the photography system can be confirmed and screen operations for giving instructions and setting related to photography on the touch panel display can be easily performed with the other hand.

In addition, in a second invention, the display control unit is configured to display, as the operation screen, a shooting standby screen, a shooting in progress screen, and a shooting end confirmation screen on the touch panel display, with the screens transitioning sequentially in response to user instructions.

This allows users to properly make the necessary settings and check the settings at each stage: before, during, and after shooting.

In a third aspect of the present invention, the display control unit displays the shooting standby screen including an operation unit for a user to select one of a plurality of shooting modes having different control conditions for the camera.

This allows the user to specify the shooting mode according to the characteristics of the location to be measured before starting shooting. In this case, it may be possible to select either indoor shooting mode or outdoor shooting mode. Furthermore, it may be possible to select either of two outdoor shooting modes depending on whether a neutral density filter is used or not.

In addition, in a fourth invention, the display control unit is configured to display the shooting screen, which includes a first image display frame that enlarges and displays either the image captured by the camera or the shooting path image representing the path traveled by the shooting device, a second image display frame that reduces and displays the other image, and an operation unit that switches the images to be displayed in the first image display frame and the second image display frame.

This allows the user to switch between enlarged and reduced image display as needed. The second image display frame itself may also be the operation unit.

In addition, in a fifth invention, the display control unit is configured to notify the user that the shooting data has been saved and that the shooting standby screen has been returned to when shooting is completed, and to display the shooting completion confirmation screen that includes an operation unit for the user to input their confirmation.

This allows the user to easily confirm that shooting has finished.

In addition, in a sixth invention, the display control unit is configured to measure the speed at which the user moves the sensor unit and display the shooting screen including an image that visualizes whether the speed is appropriate or not.

This allows the user to easily determine whether the speed at which they are moving the sensor is appropriate.

In addition, in a seventh invention, the display control unit displays the shooting-in-progress screen including a predetermined available shooting time when shooting starts, and displays the shooting-in-progress screen including the remaining shooting time when the remaining shooting time, which is calculated by subtracting the elapsed time from the start of shooting from the available shooting time, becomes equal to or less than a predetermined value during shooting.

This allows users to know the remaining shooting time when they start shooting, and also to know the remaining shooting time when the shooting time is running low.

In addition, in an eighth invention, when the display control unit detects that the current position of the imaging device has been lost during the tracking process, it notifies the user of the occurrence of the loss of its own position, presents the user with the last position acquisition point, and displays a screen on the touch panel display urging the user to return to the last position acquisition point.

This means that if the user loses their position, they can quickly return to the last location they obtained and start taking photos again.

In addition, in a ninth invention, the display control unit is configured to detect a connection failure on the communication path between the camera and the touch panel display and the control device , and to display a screen on the touch panel display notifying the user of the occurrence of the connection failure.

This allows the user to easily understand when a poor connection is preventing the imaging device from operating normally.

Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
Fig. 1 is an explanatory diagram showing the state of a photographing operation performed by a user using the photographing system according to the first embodiment, Fig. 2 is an explanatory diagram showing a standard photographing state and a high-altitude photographing state.

As shown in Figure 1, the photography system includes a photography device 1 and a control device 2 (information processing device).

The imaging device 1 includes a sensor unit 11, a display/input panel unit 12, and a rod-shaped support member 13. The sensor unit 11 and display/input panel unit 12 of the imaging device 1 are connected via a first cable 21. The display/input panel unit 12 of the imaging device 1 and the control device 2 are connected via a second cable 22.

The control device 2 is configured as a laptop or tablet PC that can be carried by the user (worker). In the example shown in Figure 1, the control device 2 can be stored in a shoulder bag and carried by the user, but the manner in which the user carries the control device 2 is not limited to this.

The user holds the camera device 1 in their hand and walks through the location to be measured, causing the camera device 1 to take pictures of the location. At this time, the user can change the position (height) of the sensor unit 11 by moving their arm holding the camera device 1.

For example, as shown in Figure 2(A), by extending the arm holding the imaging device 1 slightly forward, the user can take a photograph with the sensor unit 11 positioned at approximately the same height as the user's eye level (standard imaging state). On the other hand, as shown in Figure 2(B), by raising the arm holding the imaging device 1, the user can take a photograph with the sensor unit 11 positioned at a higher level than the user's eye level (high-altitude imaging state).

Next, we will explain the support body 13 of the imaging device 1. Figure 3 is a perspective view showing the support body 13. Figure 4 is a side view showing the expansion and contraction of the support body 13.

As shown in FIG. 3, the support 13 of the imaging device 1 includes a main body 31, an extendable rod 32, a sensor mounting portion 33, a panel mounting portion 34, and a grip portion 35. The display/input panel 12 includes a touch panel display 36.

The panel mounting unit 34 is provided on the main body 31. The panel mounting unit 34 includes a joint 41 (ball joint) that connects the display/input panel unit 12 to the main body 31. This joint 41 allows the orientation of the display/input panel unit 12, i.e., the orientation of the display surface of the touch panel display 36, to be changed. Specifically, the display/input panel unit 12 rotates around two axes A1 and A2 that pass through the center of the joint 41 (ball joint), allowing the display/input panel unit 12 to perform a tilt operation that changes the orientation of the display/input panel unit 12 up and down, and a pan operation that changes the orientation of the display/input panel unit 12 left and right. The panel mounting unit 34 also includes a fixing knob 42. The user can fix the orientation of the display/input panel unit 12 by operating the fixing knob 42. In the example shown in FIG. 3, because the joint 41 is a ball joint, the orientation of the display/input panel unit 12 can be adjusted to any direction within a predetermined angle range.

In this way, the user can adjust the orientation of the display/input panel unit 12 as appropriate, improving the visibility and operability of the touch panel display 36. For example, when changing the angle of the rod-shaped support body 13, such as when shooting at a high altitude, adjusting the orientation of the display/input panel unit 12 in the tilt direction makes it easier to view and operate the screen of the touch panel display 36. Also, for example, when the user holds the imaging device 1 in their right hand, adjusting the orientation of the display/input panel unit 12 in the pan direction so that the screen of the touch panel display 36 faces the left side, where the user's face and left hand, which operates the screen, are located, makes it easier to view and operate the screen of the touch panel display 36.

The sensor mounting section 33 is provided at the tip of the extendable rod section 32. The sensor mounting section 33 has a joint 43 (hinge joint) that connects the sensor unit 11 to the tip of the extendable rod section 32. This joint 43 allows the orientation of the sensor unit 11 to be changed. Specifically, by rotating the sensor unit 11 around the central axis A3 of the joint 43 (hinge joint), the sensor unit 11 can perform a tilting operation that changes the orientation of the sensor unit 11 up and down. A fixing knob 44 is provided on the sensor mounting section 33. The user can fix the orientation of the sensor unit 11 by operating the fixing knob 44.

In the example shown in Figure 3, the display input panel unit 12 can be tilted and panned by the joint 41 made up of a ball joint in the panel mounting section 34, but the display input panel unit 12 may also be tilted and panned by combining multiple hinge joints.

In the example shown in FIG. 3, the joint 43 in the sensor mounting section 33 is a hinge joint, which allows only tilting to change the orientation of the sensor unit 11 up and down, but it may also be possible to pan to change the orientation of the sensor unit 11 left and right. In this case, the joint 43 may be a ball joint or a combination of multiple hinge joints.

The telescopic rod section 32 is configured to be extendable and retractable in the axial direction. In the example shown in Figure 4, the telescopic rod section 32 has a structure in which multiple sheath tubes of gradually decreasing diameter are slidably connected.

By extending or retracting the extendable rod portion 32, the user can change the position of the sensor portion 11 relative to the grip portion 35, which is held by hand. That is, as shown in FIG. 4(A), when the extendable rod portion 32 is extended (extended state), the sensor portion 11 is separated from the grip portion 35. This allows the sensor portion 11 to be positioned at a higher position when photographing at high altitudes (see FIG. 2(B)). On the other hand, as shown in FIG. 4(B), when the extendable rod portion 32 is retracted and most of the extendable rod portion 32 is stored in the main body portion 31 (stored state), the overall length of the imaging device 1 is shortened, making it easier to carry and store the imaging device 1 when not in use.

In this manner, in this embodiment, the user can hold the imaging device 1 stably in one hand by grasping the grip portion 35 with their hand. This allows the user to take pictures while shooting, by changing the orientation of their arm or body holding the imaging device 1 and orienting the sensor unit 11 in any direction. Furthermore, the user can adjust the position of the sensor unit 11 and the orientation of the display input panel unit 12 by extending or retracting the extendable rod, even while shooting.

In this embodiment, the display input panel unit 12 is fixed to the main body unit 31, but the display input panel unit 12 may be fixed midway along the extendable rod unit 32, allowing the position of the display input panel unit 12 relative to the grip unit 35 to be adjusted. Furthermore, the display input panel unit 12 may be fixed to the main body unit 31 via a movable member (not shown) separate from the extendable rod unit 32 to which the sensor unit 11 is fixed at its tip.

Next, we will explain the sensor unit 11 of the imaging device 1. Figure 5 is a perspective view showing the sensor unit 11.

The sensor section 11 includes a sensor unit 51 and a sensor cover 52.

The sensor unit 51 includes a visible camera 53 and a depth sensor 54. The visible camera 53 (color camera) captures color images of the subject. The depth sensor 54 includes an infrared projector 55 and left and right infrared cameras 56 and 57 (stereo cameras). The infrared projector 55 irradiates the subject with infrared light. The left and right infrared cameras 56 and 57 detect the reflected light of the infrared light irradiated on the subject. Note that the depth sensor 54 may be a sensor that can acquire depth information using other methods, such as LiDAR (Light Detection and Ranging), in addition to a stereo camera.

In the sensor unit 51, the visible camera 53, left and right infrared cameras 56 and 57, and the infrared projector 55 are arranged side by side (approximately horizontally). The housing of the sensor unit 51 is elongated in the horizontal direction.

The sensor cover 52 has a cover body 61 and a filter member 62.

The cover body 61 covers the sensor unit 51. During photography, the sensor section 11 may collide with surrounding objects while the photography device 1 is being moved, and the cover body 61 is provided to protect the sensor unit 51 from the impact of such a collision. The cover body 61 has an opening 66 that exposes the cover glass on the front side of the visible light camera 53, infrared projector 55, and infrared cameras 56 and 57.

The filter member 62 is attached to the cover body 61 so as to be slidable in the longitudinal direction of the sensor unit 51, i.e., in the direction of the arrangement of the visible light camera 53, infrared projector 55, and infrared cameras 56 and 57. The filter member 62 has a first ring portion 63 and a second ring portion 64. An ND (Neutral Density) filter 65 (neutral density filter) is attached to the first ring portion 63.

The filter member 62 is slidable between a first position shown in FIG. 5(A) and a second position shown in FIG. 5(B). A user can grip the filter member 62 with their fingers and slide it. In the first position shown in FIG. 5(A), the ND filter 65 does not cover the visible light camera 53. In the second position shown in FIG. 5(B), the ND filter 65 covers the visible light camera 53.

Here, in the first position shown in Figure 5(A), the first ring portion 63 to which the ND filter 65 is attached is positioned between the visible camera 53 and one of the infrared cameras 57, and the second ring portion 64 is positioned to surround the lens of one of the infrared cameras 57. As a result, none of the visible camera 53, infrared projector 55, or infrared cameras 56 and 57 are covered by the filter member 62.

On the other hand, in the second position shown in Figure 5 (B), the first ring portion 63 is positioned corresponding to the visible camera 53, and the second ring portion 64 is positioned between the visible camera 53 and one of the infrared cameras 57. As a result, only the visible camera 53 is covered by the ND filter 65, and neither the infrared projector 55 nor the infrared cameras 56 and 57 are covered by the filter member 62.

When the second ring portion 64 abuts against the inner periphery of the opening in the cover body 61, the filter member 62 is positioned at the first position shown in Figure 5(A). A groove (not shown) is formed on the underside of the cover body 61, extending in the direction of movement of the filter member 62. A protrusion on the filter member 62 fits into this groove, restricting the range of movement of the filter member 62 and positioning the filter member 62 at the second position shown in Figure 5(B).

When photographing outdoors, the user adjusts the filter member 62 to the second position (see FIG. 5(B)) where the ND filter 65 covers the visible camera 53. On the other hand, when photographing indoors, the user adjusts the filter member 62 to the first position (see FIG. 5(A)) where the ND filter 65 does not cover the visible camera 53. This makes it possible to adjust the brightness of the entire captured image by reducing the amount of light input to the camera when photographing outdoors. It also prevents the problem of the entire captured image being tinted purple. Furthermore, because only the visible camera 53 (color camera) is covered by the ND filter 65, it is possible to avoid the problem of the density of the generated point cloud decreasing when the infrared cameras 56 and 57 are also covered by the ND filter 65.

Next, the general configuration of the image capture device 1 and the control device 2 will be described. Figure 6 is a block diagram showing the general configuration of the image capture device 1 and the control device 2. Figure 7 is a block diagram showing an overview of the processing performed by the processor 86 of the control device 2.

As shown in FIG. 6, the sensor unit 11 of the imaging device 1 includes a visible camera 53, a depth sensor 54, an IMU 71 (Inertial Measurement Unit), an input/output interface 72, and a connection terminal 73.

The visible light camera 53 (color camera) takes color images and outputs the captured color images.

The depth sensor 54 outputs depth information (distance image) as the detection result based on images captured by the left and right infrared cameras 56, 57 (see Figure 5).

The IMU 71 detects the motion state of the device itself, specifically, three-dimensional angular velocity and acceleration. Based on the detection results of the IMU 71, the position, orientation, and velocity of the sensor unit 11 can be detected.

The input/output interface 72 exchanges data with the control device 2 via the display/input panel unit 12. Specifically, detection data from the visible camera 53, depth sensor 54, and IMU 71 is transmitted. The input/output interface 72 may be based on the USB (registered trademark) standard.

A first cable 21 that connects the sensor unit 11 and the display/input panel unit 12 is detachably coupled to the connection terminal 73.

The visible camera 53, depth sensor 54, and IMU 71 may not be integrated into the sensor unit 51 (see Figure 5). Alternatively, the depth sensor 54 and IMU 71 may be omitted, with only the visible camera 53 provided. Alternatively, the visible camera 53 may be provided with either the depth sensor 54 or the IMU 71.

The display/input panel unit 12 of the imaging device 1 includes a touch panel display 36, a repeater 74, an input/output interface 75, a first connection terminal 76, and a second connection terminal 77.

The touch panel display 36 displays screens and other information to assist the user in taking photographs, based on control by the control device 2.

The repeater 74 relays data communication between the control device 2 and the sensor unit 11. The repeater 74 may be a hub based on the USB (registered trademark) standard.

The input/output interface 75 inputs and outputs data to and from the control device 2. Specifically, it receives display information, such as screens that assist the user in taking photographs, from the control device 2. The input/output interface 75 may be based on the USB (registered trademark) standard.

A first cable 21 that connects the sensor unit 11 and the display/input panel unit 12 is detachably coupled to the first connection terminal 76. A second cable 22 that connects the display/input panel unit 12 and the control device 2 is detachably coupled to the second connection terminal 87.

The first cable 21 is used for data communication between the control device 2 and the sensor unit 11 via the display input panel unit 12. In this data communication, for example, imaging data (imaging information) output from the sensor unit 11 is transmitted to the control device 2. The imaging data includes the image captured by the visible camera 53, the detection results (distance information) of the depth sensor 54, the detection results of the IMU 71, and the time of imaging. The second cable 22 is used for data communication between the control device 2 and the display input panel unit 12, and data communication between the control device 2 and the sensor unit 11. In data communication between the control device 2 and the display input panel unit 12, for example, display information for the screen to be displayed on the touch panel display 36 is transmitted from the control device 2, and user operation information detected on the touch panel display 36 is transmitted to the control device 2.

The control device 2 includes an input/output interface 81, a display 82, an input device 83, a memory 84, a storage device 85, a processor 86, and a connection terminal 87.

The input/output interface 81 inputs and outputs data to and from the imaging device 1. The input/output interface 81 may be based on the USB (registered trademark) standard.

The display 82 displays screens for managing previously acquired imaging data and screens for setting the operating conditions of the imaging device 1.

The input device 83 is used by the user to perform input operations. The input device 83 may be a keyboard, mouse, touchpad, touch panel, etc. If the control device 2 is configured as a tablet PC, a touch panel display 36 is provided in which the touch panel as the input device 83 and the display panel as the display 82 are integrated.

A second cable 22 that connects the control device 2 and the display input panel unit 12 of the imaging device 1 is detachably connected to the connection terminal 87.

Memory 84 stores programs executed by processor 86, etc.

The storage device 85 stores the shooting data (shooting information) acquired from the image capture device 1. The shooting data includes the image captured by the visible camera 53, the detection results (distance information) of the depth sensor 54, the detection results of the IMU 71, and the shooting time. The storage device 85 also stores point cloud data generated by the processor 86 as the 3D measurement results.

The processor 86 performs various processes by executing programs stored in the memory 84. In this embodiment, the processor 86 performs the three-dimensional measurement process P1.

In the 3D measurement process P1, the processor 16 uses the SLAM (Simultaneous Localization and Mapping) method based on images captured by the visible camera 53, etc., to generate point cloud data (environmental map) as 3D spatial information about the location to be measured. In addition, in the 3D measurement process P1, self-position estimation is performed in addition to the generation of the point cloud data, and the self-position for each measurement, i.e., the position of the shooting point, is obtained.

As shown in Figure 7, the 3D measurement process P1 includes a feature extraction process P11, a tracking process P12, a position and orientation correction process P13, and a point cloud generation process P14.

In feature extraction processing P11, the processor 86 extracts feature information (such as feature points) from the image (frame) captured by the visible camera 53.

In tracking process P12, processor 86 compares the currently extracted feature points with previously extracted feature points to estimate the amount of transition related to the position and orientation of the image capture device 1, and updates the position and orientation trajectory data based on this amount of transition. Note that the position and orientation trajectory data tracks the position and orientation of the image capture device 1, and includes the tracking results of the position and orientation at the time of capture of each captured image (frame), i.e., information regarding the position and orientation of the image capture device 1 at each time of capture.

In the position and orientation correction process P13, the processor 86 corrects the position and orientation trajectory data acquired in the tracking process P12 based on the detection data of the IMU 23. Here, the position and orientation trajectory data is corrected to compensate for the measurement results of areas with few features, such as walls and ceilings.

In the point cloud generation process P14, the processor 86 generates point cloud data based on the distance information from the depth sensor 54 and the position and orientation trajectory data acquired in the tracking process P12 and the position and orientation correction process P13. The point cloud generation process P14 also includes a process (standard point cloud generation process) that generates regular point cloud data acquired as a 3D measurement result, and a process (simplified point cloud generation process) that generates simplified point cloud data that allows the user to check the shooting conditions (point cloud generation conditions). The simplified point cloud generation process must be performed in real time during shooting, but the standard point cloud generation process may be performed after shooting.

Also, as shown in FIG. 6, the processor 86 performs drawing generation processing P2, display information generation processing P3, and display processing P4.

In drawing generation process P2, processor 86 generates a three-dimensional model of the target location based on the point cloud data of the target location generated by three-dimensional measurement process P1, and generates a layout drawing of the target location based on the three-dimensional model of the target location. At this time, a two-dimensional layout drawing (plan view) of the target location is generated by projecting the three-dimensional model onto a horizontal plane. In addition, a three-dimensional layout drawing of the target location is generated by projecting the three-dimensional model based on a specified line of sight direction.

In the display information generation process P3, the processor 86 generates display information for the screen to be displayed on the display 82 of the control device 2. The display 82 displays screens (see Figures 8 to 14) on which the user performs operations such as managing the imaging data, issuing instructions for post-imaging processing (point cloud generation process P14, drawing generation process P2), and setting processing conditions. The processor 86 also generates display information for the screen to be displayed on the touch panel display 36 of the imaging device 1. The touch panel display 36 displays screens (see Figures 15 to 19, 21 to 27) that support the user's imaging work.

In display processing P4, the processor 86 displays a screen (see Figures 8 to 14) on the display 82 of the control device 2 based on the display information generated in display information generation processing P3. The processor 86 also displays a screen (see Figures 15 to 19 and 21 to 27) on the touch panel display 36 of the imaging device 1.

In this embodiment, the shooting standby screen 311 (see FIG. 16), shooting in progress screen 331 (see FIG. 17), and shooting end confirmation screens 351 and 361 (see FIG. 21) are displayed on the touch panel display 36 of the shooting device 1 so as to transition sequentially in response to user instructions. This allows the user to appropriately make the necessary settings and check the screens at each stage before, during, and after shooting.

In addition, in this embodiment, switching between non-photography mode (first mode) and photography mode (second mode) is possible in response to user operation.

In non-photography mode, the display 82 of the control device 2 displays operable screens that allow the user to manage and process the photographed data (see Figures 8 to 14), while the touch panel display 36 of the photographing device 1 displays an inoperable pause screen 301 (see Figure 15).

On the other hand, in photography mode, photography becomes possible using the photography device 1, and operable screens are displayed on the touch panel display 36 of the photography device 1, such as screens to assist the user in the photography process (see Figures 15 to 19, 21 to 27), such as screens for instructing the start and end of photography (see Figures 16 and 17), while an inoperable pause screen 221 (see Figure 13) is displayed on the display 82 of the control device 2.

Also, as shown in FIG. 6, the processor 86 performs a connection failure detection process P5, a speed display process P6, a shooting time display process P7, and a self-position lost recovery process P8.

In the connection failure detection process P5, the processor 86 detects a connection failure on the communication path between the processor 86 and the sensor unit 51 and touch panel display 36, and notifies the user of the occurrence of the connection failure. Specifically, connection failure notification screens 231, 241 (see FIG. 14) are displayed on the display 82 of the control device 2. In addition, connection failure notification screen 411 (see FIG. 26) is displayed on the touch panel display 36 of the imaging device 1.

In the speed presentation process P6, the processor 86 measures the speed at which the user moves the sensor unit 11 based on the detection data (acceleration, angular velocity) of the IMU 71, and visualizes whether the speed is appropriate and presents it to the user. Specifically, a speed bar 334 is displayed on the shooting screen 331 (see Figure 17).

In the shooting time presentation process P7, the processor 86 presents the user with a preset available shooting time when shooting begins. Furthermore, during shooting, the processor 86 presents the user with the remaining shooting time when the remaining shooting time, calculated by subtracting the elapsed time from the shooting time, falls below a predetermined time. Specifically, the available shooting time is presented on the shooting-in-progress screen 331 (see FIG. 18) when shooting begins, and then the remaining shooting time is presented during shooting.

In the self-position lost return process P8, when the processor 86 detects that the current location of the image capture device 1 has been lost (tracking lost) during the tracking process P12, the processor 86 notifies the user that the self-position lost has occurred and presents the user with the last position acquisition point to provide return guidance urging the user to return to the last position acquisition point. Specifically, a self-position lost notification screen 371 (see Figure 22) is displayed, followed by a return screen 381 (see Figure 23) containing an image captured at the last position acquisition point. Furthermore, when the processor 86 detects that the image capture device 1 has returned to the last position acquisition point, the normal tracking process P12 is resumed.

Next, we will explain the imaging data list screen 101 displayed on the display 82 of the control device 2. Figure 8 is an explanatory diagram showing the imaging data list screen 101.

When the photography application is launched on the control device 2, the photography data list screen 101 is displayed on the display 82.

The imaging data list screen 101 has a list display section 102. The list display section 102 displays a list of captured images for each of a plurality of imaging data sets with different measurement locations and imaging dates and times. In the example shown in FIG. 8, the captured images for each imaging data set are displayed sorted by date. The list display section 102 also displays the imaging date and time for each captured image. If the point cloud generation process P14 has been performed, the list display section 102 displays a check mark 111 indicating this, and if the drawing generation process P2 has been performed, a check mark 112 indicating this.

When the user selects a piece of photography data by operating one of the photographed images for that piece of photography data displayed in the list display section 102, the screen transitions to the photography data details screen 121 (see Figure 9) for the selected photography data. At this time, the user can select the photography data they want to delete, or the photography data for which they want to execute the point cloud generation process P14 and drawing generation process P2.

If there is no registered photography data, a message indicating that there is no photography data will be displayed in the list display section 102, such as the words "No photography data available."

The shooting data list screen 101 also has a settings button 103. When the user operates the settings button 103, a settings screen 181 (see Figure 12 (A)) pops up on the shooting data list screen 101.

The shooting data list screen 101 also has a "Start shooting" button 104. When the user operates the "Start shooting" button 104, the mode switches from non-shooting mode to shooting mode. At this time, the display 82 of the control device 2 transitions to a pause screen 221 (see FIG. 13). Meanwhile, the touch panel display 36 of the shooting device 1 transitions from a pause screen 301 (see FIG. 15) to a shooting standby screen 311 (see FIG. 16).

Next, we will explain the shooting data details screen 121 displayed on the display 82 of the control device 2. Figure 9 is an explanatory diagram showing the shooting data details screen 121. Figure 10 is an explanatory diagram showing a screen that pops up on the shooting data details screen 121. Figure 11 is an explanatory diagram showing the transition status of the main parts of the shooting data details screen 121.

The shooting data details screen 121 has an image display section 122 and a shooting date and time display section 123. The image display section 122 plays back the captured images (key frames) included in the shooting data in order. The shooting date and time display section 123 displays the shooting date and time of the captured image displayed in the image display section 122. In the SLAM method, frames in which a significant change in field of view appears are extracted as key frames, and these key frames are reflected in the point cloud.

The shooting data details screen 121 also has a playback operation section 124. The playback operation section 124 has a seek bar 131, a playback stop button 132, a frame advance button 133, and a frame rewind button 134. The user can specify the desired playback start time by operating the slider on the seek bar 131. The user can also instruct the shooting data to be played or stopped by operating the playback stop button 132. The user can also instruct the shooting data to be played or rewinded frame by operating the frame advance button 133 and frame rewind button 134.

The shooting data details screen 121 also has an instruction operation section 125. The instruction operation section 125 has a "Generate point cloud" button 141, a "Display point cloud" button 142, a "Generate drawing" button 143, and a "Display drawing" button 144.

When the user operates the "Generate point cloud" button 141, the point cloud generation process P14 (standard point cloud generation process) begins, and the point cloud generation in progress screen 151 shown in FIG. 10(A) pops up on the shooting data details screen 121. The point cloud generation in progress screen 151 displays a message indicating that the point cloud generation process P14 is being executed. The point cloud generation in progress screen 151 also includes a progress bar 152. The progress bar 152 visualizes the progress of the point cloud generation process P14. The point cloud generation in progress screen 151 also includes a "Cancel point cloud generation" button 153. When the user operates the "Cancel point cloud generation" button 153, the point cloud generation process P14 is canceled and the screen returns to the shooting data details screen 121 (see FIG. 9). When the point cloud generation process P14 is completed, the screen returns to the shooting data details screen 121.

Furthermore, when the user operates the "Point Cloud Display" button 142 on the shooting data details screen 121 shown in Figure 9, the point cloud display application is launched and the point cloud data generated in the point cloud generation process P14 is displayed.

When the user operates the "Drawing Generation" button 143, the drawing generation in progress screen 161 shown in FIG. 10 (B) pops up on the photographic data details screen 121. The drawing generation in progress screen 161 displays a message indicating that the drawing generation process P2 is currently being executed. The drawing generation in progress screen 161 also has a "Cancel drawing generation" button 162. When the user operates the "Cancel drawing generation" button 162, the drawing generation process P2 is canceled and the screen returns to the photographic data details screen 121 (see FIG. 9). When the drawing generation process P2 is completed, the screen returns to the photographic data details screen 121. The drawing generation in progress screen 161 may also display an animation indicating that the drawing generation process P2 is currently being executed.

Furthermore, when the user operates the "Drawing display" button 144 on the shooting data details screen 121 shown in Figure 9, the drawing display application is launched and the drawing generated in the drawing generation process P2 is displayed.

The shooting data details screen 121 also has a "Delete" button 145. When the user operates the "Delete" button 145, a deletion confirmation screen 171 shown in FIG. 10 (C) pops up on the shooting data details screen 121. The deletion confirmation screen 171 displays a message prompting the user to confirm the deletion of the shooting data. The deletion confirmation screen 171 also has a "Delete" button 172 and a "Cancel" button 173. When the user operates the "Delete" button 172, the shooting data is deleted and the screen returns to the shooting data list screen 101 (see FIG. 8). When the user operates the "Cancel" button 173, the screen returns to the shooting data details screen 121 (see FIG. 9).

Here, as shown in FIG. 11, in the instruction operation unit 125, the states of the "Generate point cloud" button 141, "Display point cloud" button 142, "Generate drawing" button 143, and "Display drawing" button 144 change before and after the point cloud generation process P14 and the drawing generation process P2.

First, as shown in Figure 11 (A), before the point cloud generation process P14 is executed, only the "Generate Point Cloud" button 141 is operable, while the "Display Point Cloud" button 142, "Generate Drawing" button 143, and "Display Drawing" button 144 are inoperable. When the point cloud generation process P14 is completed, as shown in Figure 11 (B), a check mark 146 is displayed indicating that the point cloud generation process P14 has been executed. Also, in this state, the "Display Point Cloud" button 142 and the "Generate Drawing" button 143 are operable, while the "Display Drawing" button 144 is inoperable. When the drawing generation process P2 is completed, as shown in Figure 11 (C), a check mark 147 is displayed indicating that the drawing generation process P2 has been executed. Also, in this state, the "Display Drawing" button 144 is operable.

Also, as shown in FIG. 9, the imaging data details screen 121 has a "Return to list" button 126. When the user operates the "Return to list" button 126, the screen returns to the imaging data list screen 101 (see FIG. 8).

Next, we will explain the settings screen 181 displayed on the display 82 of the control device 2. Figure 12 is an explanatory diagram showing the settings screen 181.

When the user operates the settings button 103 on the shooting data list screen 101 (see Figure 8), the settings screen 181 shown in Figure 12 (A) pops up on the shooting data list screen 101.

The settings screen 181 has a save folder setting section 182. The save folder setting section 182 displays a folder path indicating the location where the settings file containing the settings information will be saved. The save folder setting section 182 also has a "Change" button 191. When the user operates the "Change" button 191, a screen (not shown) for specifying a folder pops up, allowing the user to specify the folder in which to save the settings file.

The settings screen 181 also has a camera settings section 183. The camera settings section 183 has input sections 192, 193, and 194 for the resolution of the visible camera 53, the resolution of the depth sensor 54, and the frame rate of the visible camera 53. When the user operates each of the input sections 192, 193, and 194, a pull-down menu (not shown) is displayed. The user can select the resolution of the visible camera 53, the resolution of the depth sensor 54, and the frame rate of the visible camera 53 from the pull-down menu.

The settings screen 181 also has a device information display section 184. The device information display section 184 displays information about the sensor unit 51, such as the name (camera name), serial number, and firmware version of the sensor unit 51, as well as the date and time the previous calibration process was performed. The device information display section 184 also has a "Forced Restart" button 195. When the user operates the "Forced Restart" button 195, the sensor unit 51 is forcibly restarted. The device information display section 184 also has an "Execute" button 196. When the user operates the "Execute" button 196, a device calibration screen 201 shown in FIG. 12(B) pops up on the settings screen 181.

The settings screen 181 also has a "Done" button 185. When the user operates the "Done" button 185, the settings process is executed using the input details on the settings screen 181, and the screen returns to the shooting data list screen 101 (see Figure 8).

As shown in FIG. 12(B), the device calibration screen 201 displays a message urging the user to keep the image capture device 1 stationary in a stable location for a predetermined time for the calibration process, as well as the remaining time. The device calibration screen 201 also displays a progress bar 202 indicating the progress of the calibration process. The device calibration screen 201 also has a "Cancel" button 203. When the user operates the "Cancel" button 203, the calibration process is canceled and the screen returns to the settings screen 181 (see FIG. 12(A)). When the calibration process is completed, the screen returns to the settings screen 181.

Next, we will explain the pause screen 221 and poor connection notification screens 231 and 241 displayed on the display 82 of the control device 2. Figure 13 is an explanatory diagram showing the pause screen 221. Figure 14 is an explanatory diagram showing the poor connection notification screens 231 and 241.

When the user operates the "Start shooting" button 104 on the shooting data list screen 101 (see Figure 8), the mode switches from non-shooting mode to shooting mode.

If the transition to photography mode is successful, the screen will transition to the pause screen 221 shown in Figure 13. The pause screen 221 displays a message indicating that photography mode has been entered, and a message indicating that by performing an operation to end photography mode on the screen displayed on the touch panel display 36 of the photography device 1, it will become possible to display and operate the screen on the display 82 of the control device 2.

On the other hand, if the transition to photography mode fails, a poor connection notification screen 231, 241 (error screen) shown in FIG. 14 will pop up on the photography data list screen 101 depending on the cause of the failure. Here, if a poor connection has occurred with the second cable 22 connecting the display input panel unit 12 of the photography device 1 and the control device 2, the poor connection notification screen 231 shown in FIG. 14(A) will be displayed. On the other hand, if a poor connection has occurred with the first cable 21 connecting the display input panel unit 12 of the photography device 1 and the sensor unit 11, the poor connection notification screen 241 shown in FIG. 14(B) will be displayed.

The poor connection notification screen 231 shown in FIG. 14(A) displays a message stating that the camera cannot transition to shooting mode due to a poor connection between the display/input panel unit 12 of the camera 1 and the control device 2, and a message urging the user to check the location of the poor connection. The poor connection notification screen 231 also has a "retry" button 232 and a "cancel" button 233. The user operates the "retry" button 232 after taking steps to resolve the poor connection of the second cable 22. This causes the process of transitioning to shooting mode to be executed again. On the other hand, if the user operates the "cancel" button 233, the screen returns to the shooting data list screen 101 (see FIG. 8).

The poor connection notification screen 241 shown in FIG. 14(B) displays a message stating that the transition to photography mode is not possible due to a poor connection between the display/input panel unit 12 and the sensor unit 11 of the photography device 1, and a message urging the user to check the location of the poor connection. The poor connection notification screen 241 also has a "retry" button 242 and a "cancel" button 243. The user operates the "retry" button 242 after taking steps to resolve the poor connection of the first cable 21 connecting the display/input panel unit 12 and the sensor unit 11 of the photography device 1. This causes the process of transitioning to photography mode to be executed again. On the other hand, if the user operates the "cancel" button 243, the screen returns to the photography data list screen 101 (see FIG. 8).

Next, we will explain the pause screen 301 displayed on the touch panel display 36 of the imaging device 1. Figure 15 is an explanatory diagram showing the pause screen 301.

In non-photography mode, that is, while the user is operating the screens on the display 82 of the control device 2, such as the photography data list screen 101 (see Figure 8), the photography data details screen 121 (see Figure 9), and the settings screen 181 (see Figure 12), the pause screen 301 is displayed on the touch panel display 36 of the photography device 1.

The pause screen 301 displays a message indicating that the device is in non-photography mode, i.e., that the display 82 of the control device 2 is currently displaying and operating the photography data list screen 101 (see Figure 8). The pause screen 301 also displays a message indicating that the device will transition to photography mode by performing an operation to start photography on the screen displayed on the display 82 of the control device 2.

Next, we will explain the shooting standby screen 311 displayed on the touch panel display 36 of the photographing device 1. Figure 16 is an explanatory diagram showing the shooting standby screen 311.

As shown in Figure 16 (A), the shooting standby screen 311 has an image display section 312. The image display section 312 displays images captured in real time by the visible camera 53.

The shooting standby screen 311 also has a shooting mode selection section 313. When the user operates the shooting mode selection section 313, a pull-down menu 314 is displayed, as shown in FIG. 16(B). The pull-down menu 314 allows the user to select from multiple shooting modes with different control conditions (image signal processing conditions) for the visible camera 53. In the example shown in FIG. 16(B), the user can select from "indoor" (indoor shooting mode), "outdoor" (first outdoor shooting mode), or "outdoor + ND filter" (second outdoor shooting mode) as the shooting mode. Note that each shooting mode uses a different brightness correction table, for example, used in image signal processing (image correction processing).

The shooting standby screen 311 also has a brightness adjustment bar 315. The user can adjust the brightness of the captured image by operating the slider on the brightness adjustment bar 315. Brightness adjustment can be performed separately for each shooting mode.

The shooting standby screen 311 also has a start shooting button 316. When the user operates the start shooting button 316, shooting begins and the screen transitions to the shooting in progress screen 331 (see Figure 17).

The shooting standby screen 311 also has an "End shooting" button 317. When the user operates the "End shooting" button 317, the shooting mode ends, the camera switches to non-shooting mode, and the screen transitions to the pause screen 301 (see Figure 15).

Next, the shooting screen 331 displayed on the touch panel display 36 of the photographing device 1 will be described. Figure 17 is an explanatory diagram showing the shooting screen 331. Figure 18 is an explanatory diagram showing the transition status of the shooting time guide section 336 on the shooting screen 331. Figure 19 is an explanatory diagram showing the shooting screen 331 with a message window 337 displayed. Figure 20 is an explanatory diagram showing the content and display timing of the message displayed in the message window 337.

As shown in Figures 17(A) and (B), the shooting screen 331 has a main window 332 (first image display frame) and a sub-window 333 (second image display frame). The main window 332 and the sub-window 333 have different display magnifications, with the main window 332 displaying an enlarged image and the sub-window 333 displaying a reduced image.

The main window 332 and sub-window 333 also display either a real-time image 341 captured by the visible camera 53 or a photography path image 342 representing the photography path at the measurement target location. The photography path image 342 is obtained by superimposing a line 344 representing the photography path on a point cloud image 343 representing the measurement target location. The point cloud image is an image (rendering) of each point of the generated point cloud data as viewed from a specified viewpoint.

The example shown in Figure 17 (A) is in the standard state. In this case, the photographed image 341 is displayed enlarged in the main window 332, and the photographed path image 342 is displayed reduced in the sub-window 333. On the other hand, the example shown in Figure 17 (B) is in the expanded photographed path state. In this case, the photographed path image 342 is displayed enlarged in the main window 332, and the photographed image 341 is displayed reduced in the sub-window 333.

The user can switch between the standard state shown in FIG. 17(A) and the expanded shooting path state shown in FIG. 17(B) by operating the sub-window 333. That is, when the sub-window 333 is operated in the standard state shown in FIG. 17(A), the state transitions to the expanded shooting path state shown in FIG. 17(B), and the shooting path image 342 is enlarged and displayed in the main window 332. When the sub-window 333 is operated in the expanded shooting path state shown in FIG. 17(B), the state transitions to the standard state shown in FIG. 17(A), and the captured image 341 is enlarged and displayed in the main window 332.

A mesh image may be superimposed on the captured image 341 within the point cloud generation range. The mesh image can be obtained by converting the generated point cloud data into mesh data and then processing the mesh data to image it with the same angle of view as the captured image 341.

The shooting screen 331 also has a speed bar 334. The speed bar 334 visualizes and displays the speed at which the user moves the sensor unit 11. Specifically, the speed is represented by color. For example, the speed bar 334 is drawn with a gradation from green to orange, with the proportion of orange increasing as the speed increases. This allows the user to easily check whether the speed at which the sensor unit 11 is moved is appropriate, and more specifically, whether the speed at which the sensor unit 11 is moved is not too fast. This makes it possible to avoid a situation where the user moves the sensor unit 11 too fast, resulting in blurring and a decrease in the accuracy of the 3D measurement.

The shooting screen 331 also has an elapsed time display section 335. The elapsed time display section 335 displays the time that has elapsed since shooting began.

Also, as shown in FIG. 18, the shooting-in-progress screen 331 displays a shooting time indicator 336. While the elapsed time display 335 is always displayed, the shooting time indicator 336 is only displayed at a predetermined timing for a predetermined period of time. Note that the shooting time indicator 336 is not limited to being displayed only at predetermined timings, but can also be set to be always displayed by user operation.

First, as shown in FIG. 18(A), when shooting starts, the shooting time guide 336 is displayed for a predetermined time (e.g., 5 seconds) from the start of shooting. In this case, the shooting time guide 336 displays the remaining shooting time (e.g., 15 minutes).

Next, as shown in FIG. 18(B), the shooting time indicator 336 is displayed for a predetermined time (e.g., 5 seconds) when the remaining shooting time (the time elapsed since shooting began minus the available shooting time) falls below a predetermined time. In this case, the remaining shooting time is displayed in the shooting time indicator 336. The timing at which the shooting time indicator 336 is displayed is not limited to once. For example, the shooting time indicator 336 may be displayed when the remaining shooting time is 3 minutes and when the remaining shooting time is 2 minutes.

Next, as shown in FIG. 18(C), when the remaining shooting time becomes very short (for example, one minute), the shooting time indicator 336 continues to be displayed until the remaining shooting time runs out. In this case, the remaining shooting time is displayed in the shooting time indicator 336. Furthermore, the shooting time indicator 336 is displayed in a manner different from that shown in FIGS. 18(A) and (B). For example, the background and frame of the shooting time indicator 336 change color, thereby highlighting the shooting time indicator 336.

In addition, as shown in FIG. 19, a message window 337 is displayed on the shooting screen 331. Messages to notify the user are displayed in the message window 337. Specifically, messages regarding shooting instructions and various warnings are displayed.

The example shown in Figure 19 (A) is an information message. Information messages provide users with information that will be helpful for their photography work or information to draw their attention. The example shown in Figure 19 (B) is a warning message. Warning messages prompt users to take action to improve the status of photography work that requires immediate improvement.

The message window 337 may be displayed in a different position on the shooting screen 331 depending on the type of message (information, warning, etc.). For example, in the example shown in FIG. 19(B), the message window 337 for a warning message is displayed at the bottom of the shooting screen 331, similar to the information message shown in FIG. 19(A). However, the warning message may also be displayed in the center of the shooting screen 331 so that the user can immediately check it.

Figure 20 shows examples of information messages and warning messages displayed in the message window 337. Information messages include a message indicating successful recovery from self-position loss, a message indicating completion of position correction processing when loop closing (circular movement) is detected, and a message urging the user to stop the image capture device 1 so that position correction processing can be performed when loop closing is detected. Warning messages include a message urging the user to stop the image capture device 1 if the user moves the image capture device 1 while loop closing is detected and position correction processing is being performed, a message urging the user to move the image capture device 1 more slowly if the user is moving the image capture device 1 too quickly, a message urging the user to change the subject being captured if few feature points are extracted from the captured image, and a message urging the user to maintain an appropriate distance from the subject if they have gotten too close.

Next, we will explain the end-of-shooting confirmation screens 351 and 361 displayed on the touch panel display 36 of the imaging device 1. Figure 21 is an explanatory diagram showing the end-of-shooting confirmation screens 351 and 361.

As shown in FIG. 17, the shooting-in-progress screen 331 has an end shooting button 339. When the user operates the end shooting button 339, a process is executed to end shooting and save the captured data, and a shooting end confirmation screen 351 pops up on the shooting-in-progress screen 331, as shown in FIG. 21 (A).

The end-of-shooting confirmation screen 351 displays a message indicating that shooting has ended and that the shooting data has been saved, and a message indicating that the screen will return to the shooting standby screen 311 (see FIG. 16). The end-of-shooting confirmation screen 351 also has an "OK" button 352. When the user operates the "OK" button 352, the screen transitions to the shooting standby screen 311.

In addition, in this embodiment, if the shooting time (the time elapsed since the start of shooting) exceeds a predetermined available shooting time, shooting is forcibly ended and processing to save the captured data is executed. In this case, as shown in FIG. 21(B), a confirmation screen 361 for ending shooting due to shooting time exceeding is displayed as a pop-up on the shooting in progress screen 331.

The shooting end confirmation screen 361 displays a message indicating that shooting has been forcibly terminated due to the time limit being exceeded and that the shooting data has been saved, and a message indicating that the screen will return to the shooting standby screen 311 (see FIG. 16). The shooting end confirmation screen 361 also has an "OK" button 362. When the user operates the "OK" button 362, the screen transitions to the shooting standby screen 311.

In this embodiment, shooting is forcibly terminated when the shooting time exceeds a specified shooting time, but such forced termination may not be performed, or the user may be able to select a mode in which forced termination is performed or a mode in which forced termination is not performed.

Next, we will explain the screens displayed on the touch panel display 36 of the image capture device 1 when the self-position is lost. Figure 22 is an explanatory diagram showing the self-position lost notification screen 371. Figure 23 is an explanatory diagram showing the recovery screen 381. Figure 24 is an explanatory diagram showing the last position acquisition point enlargement screen 391. Figure 25 is an explanatory diagram showing the recovery failure notification screen 401.

During shooting, a self-position loss (tracking loss) may occur during tracking processing P12 (self-position estimation processing), where the self-position is lost. In this case, it is necessary to return to the last position acquisition point, i.e., the last point where the self-position was correctly estimated, and restart tracking processing P12. Therefore, first the user is notified that a self-position loss has occurred, and then the user is assisted in returning to the last position acquisition point. When it is detected that the user has returned to the last position acquisition point, tracking processing P12 is restarted.

In this embodiment, when self-location loss is detected, a self-location loss notification screen 371 is displayed as a pop-up on the shooting screen 331 on the touch panel display 36 of the image capturing device 1, as shown in FIG. 22 .

The self-location lost notification screen 371 notifies the user that their self-location has been lost and displays a message urging the user to return to the last location acquisition point. The self-location lost notification screen 371 also has an "OK" button 372. When the user operates the "OK" button 372, the screen transitions to the returning screen 381 shown in FIG. 23. At this time, the processor 86 of the control device 2 starts processing to detect that the image capture device 1 has returned to the last location acquisition point.

The example shown in Figure 23(A) is the standard state, similar to the example shown in Figure 17(A), in which an enlarged image 341 captured by the visible camera 53 is displayed in the main window 332 (first image display frame), and a reduced image of the shooting path 342 is displayed in the sub-window 333 (second image display frame). The example shown in Figure 23(B) is the enlarged state of the shooting path, similar to the example shown in Figure 17(B), in which an enlarged image of the shooting path 342 is displayed in the main window 332, and a reduced image of the shooting image 341 is displayed in the sub-window 333.

In addition to the sub-window 333 (second image display frame) in which the captured image 341 and the captured path image 342 are displayed in reduced size, the return screen 381 also has another sub-window 382 (third image display frame). The captured image of the last location acquisition point is displayed in the sub-window 382. This allows the user to easily grasp the last location acquisition point.

In addition, a message window 383 is displayed on the recovery screen 381. Similar to the self-location lost notification screen 371 (see Figure 22), the message window 383 displays a message notifying the user that their own location has been lost and urging them to return to the last location acquisition point.

When the sub-window 382 displaying the captured image of the last location acquisition point is operated, a last location acquisition point enlarged screen 391 pops up on the returning screen 381, as shown in FIG. 24. The last location acquisition point enlarged screen 391 displays an enlarged image of the last location acquisition point. An "x" button 392 is also displayed on the last location acquisition point enlarged screen 391. When the user operates the "x" button 392, the last location acquisition point enlarged screen 391 closes and the screen returns to the returning screen 381.

Here, if the device recovers from losing its own position, i.e., if it has successfully returned to the last location acquisition point, the screen will transition to the shooting screen 331 (see FIG. 17). Note that, upon detecting that the device has returned to the last location acquisition point, the returning screen 381 may display a message informing the user that the device has returned to the last location acquisition point, and then the screen may transition to the shooting screen 331.

On the other hand, if the device is unable to recover from a state where its own position has been lost even after a predetermined time has elapsed, a recovery failure notification screen 401 will pop up on the recovery in progress screen 381, as shown in FIG. 25. The recovery failure notification screen 401 displays a message indicating that recovery from a state where its own position has been lost has failed ("Tracking recovery has failed."). The recovery failure notification screen 401 also has an "OK" button 402. When the user operates the "OK" button 402, shooting is forcibly terminated and the screen transitions to the shooting standby screen 311 (see FIG. 16).

Next, we will explain the poor connection notification screen 411 displayed on the touch panel display 36 of the imaging device 1. Figure 26 is an explanatory diagram showing the poor connection notification screen 411.

During shooting, the first cable 21 connecting the display/input panel unit 12 and the sensor unit 11 may become disconnected. In this case, a poor connection notification screen 411 (error screen) shown in FIG. 26 pops up on the shooting screen 331.

The poor connection notification screen 411 displays a message stating that the sensor unit 11, including the visible camera 53, cannot be detected, and a message urging the user to check the condition of the sensor unit 11 and the cable connecting the display/input panel unit 12 to the sensor unit 11.

The poor connection notification screen 411 also has a "Retry" button 412 and a "Cancel" button 413. The user operates the "Retry" button 412 after resolving the poor connection in the cable connecting the display/input panel unit 12 and sensor unit 11 of the image capture device 1. When the poor connection is resolved, the screen transitions to the image capture standby screen 311 (see FIG. 16). When the user operates the "Cancel" button 413, the screen transitions from image capture mode to list mode. At this time, the touch panel display 36 of the image capture device 1 transitions to the pause screen 301 (see FIG. 15). Meanwhile, the display 82 of the control device 2 transitions from the pause screen 221 (see FIG. 13) to the image capture data list screen 101 (see FIG. 8).

Note that the second cable 22 connecting the display input panel unit 12 of the image capture device 1 and the control device 2 may become disconnected. In this case, the screen will no longer be displayed on the touch panel display 36 of the image capture device 1. Meanwhile, on the display 82 of the control device 2, a screen (not shown) notifying the user of the poor connection pops up on the pause screen 221 (see Figure 13). The screen notifying the user of the poor connection has an "OK" button. When the user operates the "OK" button, the screen transitions to the image capture data list screen 101 (see Figure 8).

Next, we will explain the shooting mode termination confirmation screen 421 displayed on the touch panel display 36 of the photographing device 1. Figure 27 is an explanatory diagram showing the shooting mode termination confirmation screen 421.

When the user operates the "End shooting" button 317 on the shooting standby screen 311 shown in Figure 16, a shooting mode end confirmation screen 421 shown in Figure 27 pops up on the shooting standby screen 311.

The shooting mode termination confirmation screen 421 displays a message asking the user whether or not to terminate shooting mode and return to list mode. The shooting mode termination confirmation screen 421 also has an "OK" button 422 and a "Cancel" button 423. When the user operates the "Cancel" button 423, the screen returns to the shooting standby screen 311 (see FIG. 16). When the user operates the "OK" button 422, the shooting mode is terminated and the screen returns to list mode. At this time, the touch panel display 36 of the shooting device 1 transitions to the pause screen 301 (see FIG. 15). Meanwhile, the display 82 of the control device 2 transitions from the pause screen 221 (see FIG. 13) to the shooting data list screen 101 (see FIG. 8).

Second Embodiment
Next, a second embodiment will be described. It should be noted that the points not specifically mentioned here are the same as those of the previous embodiment. Fig. 28 is a block diagram showing the schematic configuration of an image capture device 1 and a control device 2 according to the second embodiment.

In the first embodiment, the image capture device 1 and the control device 2 are connected by wire, but in this embodiment, the image capture device 1 and the control device 2 are connected wirelessly. Specifically, the image capture device 1 is equipped with a wireless communication unit 78. Furthermore, the control device 2 (information processing device) is equipped with a wireless communication unit 88. The wireless communication unit 78 of the image capture device 1 and the wireless communication unit 88 of the control device 2 communicate wirelessly using an appropriate wireless communication method such as wireless LAN. In the example shown in Figure 28, an input/output interface 79 is provided for inputting and outputting data to and from a sensor unit 11 configured in the same way as in the first embodiment.

When the imaging device 1 and control device 2 are wirelessly connected in this way, the control device 2 can be left stationary in an appropriate location, such as the location to be measured or nearby, during imaging, eliminating the need for the user to carry the control device 2 around with them.

In the first and second embodiments, the imaging system is composed of an imaging device 1 held by the user and a control device 2 carried by the user, but the display input panel 12 of the imaging device 1 may also include all or part of the functions of the control device 2. In this case, the process of generating simple point cloud data (simplified point cloud generation process) for the user to check the imaging situation (point cloud generation situation) requires a relatively small load and needs to be performed in real time during imaging, so it may be performed by the imaging device 1. Furthermore, the process of generating regular point cloud data obtained as a 3D measurement result (standard point cloud generation process) requires a relatively large load and only needs to be performed after imaging, so it may be performed by a separately provided server device with high processing capabilities.

Furthermore, in a configuration in which the image capture device 1 and the control device 2 are wirelessly connected as in the second embodiment, the control device 2 may be configured on-premise or in the cloud. Furthermore, in a configuration in which the image capture device 1, which has some of the functions of the control device 2, is wirelessly connected to a server device, the server device may be configured on-premise or in the cloud.

As mentioned above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can also be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is possible to combine the components described in the above embodiments to create new embodiments.

The imaging system, imaging device, and imaging control method of the present invention have the advantage that the imaging device can be held stably, even while imaging, and screen operations for issuing instructions and making settings related to imaging on the touch panel display can be easily performed. They are useful as imaging systems, imaging devices, and imaging control methods that include an imaging device that images a measurement target location and a processor that controls the imaging device in order to perform three-dimensional measurement processing that generates three-dimensional spatial information about the measurement target location.

1: Photographing device 2: Control device (information processing device)
11: Sensor unit 12: Display input panel unit 13: Support 21: First cable 22: Second cable 31: Main body 32: Telescopic rod unit 33: Sensor mounting unit 34: Panel mounting unit 35: Grip unit 36: Touch panel display 41: Joint 42: Fixing knob 43: Joint 44: Fixing knob 51: Sensor unit 52: Sensor cover 53: Visible camera 54: Depth sensor 61: Cover main body 62: Filter member 65: ND filter 101: Shooting data list screen 121: Shooting data details screen 231: Connection failure notification screen 241: Connection failure notification screen 311: Shooting standby screen 331: Shooting in progress screen 335: Elapsed time display unit 336: Shooting time guide unit 351: Shooting end confirmation screen 361: Shooting end confirmation screen 371: Self-position lost notification screen 381: Recovery in progress screen 411: Connection failure notification screen 421: Shooting mode end confirmation screen

Claims (9)

An imaging system including an imaging device that images a measurement target location, and a control device that is connected to the imaging device and executes three-dimensional measurement processing of the measurement target location ,
The imaging device is
a sensor unit including a camera for capturing an image of a measurement target location;
a first display unit including a touch panel display that displays a screen used in a photography mode that supports a user's photography operation and detects screen operations by the user;
a support body having a grip portion that is held by a user with one hand and that supports the sensor portion and the first display portion;
The control device
a three-dimensional measurement processing unit that performs the three-dimensional measurement processing to generate three-dimensional spatial information of a measurement target location based on the photographing data acquired from the photographing device;
a second display unit that displays a screen used in a non-photographing mode for performing operations related to management and processing of the photographed data;
a display control unit that controls display information of a screen to be displayed on the first display unit and the second display unit,
The display control unit, in response to a user's operation to transition from the non-photography mode to the photography mode, displays a pause screen on the second display unit and an operation screen on the first display unit through which the user can at least instruct the start of photography . The display control unit
2. The photographing system according to claim 1, wherein the operation screens displayed on the touch panel display include a photographing standby screen, a photographing in progress screen, and a photographing end confirmation screen, which are sequentially displayed in response to a user's instruction. The display control unit
3. The photographing system according to claim 2, wherein the photographing standby screen includes an operation section for allowing a user to select one of a plurality of photographing modes having different control conditions for the camera. The display control unit
The photographing system according to claim 2, characterized in that the photographing-in-progress screen includes a first image display frame that enlarges and displays either the image photographed by the camera or a photographing path image representing the path traveled by the photographing device, a second image display frame that reduces and displays the other image, and an operation unit that switches the images displayed in the first image display frame and the second image display frame. The display control unit
The photography system according to claim 2, characterized in that, when photography is completed , the user is notified that the photography data has been saved and that the photography standby screen has been returned to, and the photography completion confirmation screen is displayed, which includes an operation unit for the user to input confirmation. The display control unit
3. The photographing system according to claim 2, wherein the speed at which the user moves the sensor unit is measured and the screen during photographing is displayed including an image that visualizes whether the speed is appropriate. The display control unit
When starting shooting, the shooting screen including a predetermined available shooting time is displayed;
3. The photographing system according to claim 2, wherein the photographing-in-progress screen including the remaining photographing time is displayed when the remaining photographing time, which is calculated by subtracting the elapsed time from the start of photographing from the available photographing time, becomes equal to or less than a predetermined value during photographing. The display control unit
The photographing system according to claim 1, characterized in that when a self-position loss is detected during tracking processing, in which the photographing device loses track of its current position, the system notifies the user of the occurrence of the self-position loss, presents the user with the last position acquisition point, and displays a screen on the touch panel display prompting the user to return to the last position acquisition point. The display control unit
The photographing system according to claim 1, characterized in that a connection failure on the communication path between the camera and the touch panel display and the control device is detected, and a screen notifying the user of the occurrence of the connection failure is displayed on the touch panel display.