JP7722751B2 - Real-time communication support system and method, mobile terminal, server, and program - Google Patents

Description

The present disclosure relates to a real-time communication support system, a real-time communication support method, a mobile terminal, a server, and a program .

In recent years, systems have been proposed for inspecting, monitoring, maintaining, and operating infrastructure and equipment that utilize 3D sensors such as LiDAR (Light Detection and Ranging) and ToF (Time of Flight) cameras. The 3D point cloud data acquired by 3D sensors is a sparse collection of points, and simply looking at it makes it difficult to determine which 3D point cloud corresponds to which structure (building or equipment). Furthermore, unlike regular image data, 3D point cloud data is structural data, so its large data volume makes it difficult to share the situation at a site, etc., represented by the 3D point cloud data, in real time.

Patent Document 1 discloses using a distance sensor to measure the distance to each point within a predetermined area that includes an object. Patent Document 1 also discloses using a three-dimensional object recognition means to convert the measurement data for each point into mesh data, and grouping this mesh data into multiple mesh groups. Patent Document 1 also discloses creating a projection plane perpendicular to the normal vector of each mesh group, and obtaining projection data by projecting each mesh group onto the corresponding projection plane. Patent Document 1 also discloses extracting contour data from the projection data and comparing the contour data with two-dimensional shape data of the object to recognize the position and orientation of the object.

JP 2011-159042 A

When 3D point cloud data is meshed, converted into CAD (Computer Aided Design) data, or polygonized, the structure is represented as an object, the data volume is reduced, and processing speeds for drawing and interaction can be increased. However, meshed objects have lower accuracy than the 3D point cloud itself, making them unsuitable for measuring the details of an object with high precision.

Therefore, an object of the present disclosure is to provide a real-time communication support system, a real-time communication support method, a mobile terminal, a server, and a program that simultaneously display detailed 3D point cloud data and small-capacity mesh data.

The real-time communication support system of the present disclosure includes:
a means for photographing the subject;
means for acquiring 3D point cloud data from the photographed object;
A means for obtaining a mesh screen of a landscape in which everything other than the target is meshed;
means for storing the 3D point cloud data of the object and the mesh view of the scene;
and means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.

In addition, the real-time communication support system of the present disclosure includes:
A means of photographing the landscape, including the subject,
means for determining the object from the photographed scene;
means for acquiring 3D point cloud data from the photographed scene;
a means for generating a mesh screen of the scenery by meshing the 3D point cloud data other than the determined object;
means for storing the 3D point cloud data of the object and the mesh view of the scene;
and means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.

In addition, the mobile terminal of the present disclosure includes:
The mobile terminal includes a means for photographing an object and a means for simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape in which everything other than the object has been meshed.

In addition, the server of the present disclosure
The server comprises a means for acquiring 3D point cloud data from a photographed object, a means for acquiring a mesh screen of a landscape in which everything other than the object has been meshed, and a means for saving the 3D point cloud data of the object and the mesh screen of the landscape.

Further, the real-time communication support method of the present disclosure includes:
photographing an object;
acquiring 3D point cloud data from the photographed object;
A step of obtaining a mesh screen of a landscape in which everything other than the target is meshed;
saving the 3D point cloud data of the object and the mesh view of the scene;
and simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.

In addition, the program of the present disclosure
The program causes a mobile terminal to execute the steps of photographing an object and simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape in which everything other than the object has been meshed.

This disclosure makes it possible to provide a real-time communication support system that simultaneously displays detailed 3D point cloud data and small-volume mesh data.

FIG. 10 is a diagram showing distance measurement in infrastructure inspection etc. according to an embodiment. FIG. 1 is a diagram illustrating a first meshing technique according to an embodiment. FIG. 10 is a diagram illustrating a second meshing technique according to an embodiment. FIG. 1 is a diagram illustrating a configuration of a system according to an embodiment. FIG. 2 is a diagram illustrating a configuration of a mobile terminal according to an embodiment. FIG. 2 is a diagram illustrating a configuration of a server (cloud) according to an embodiment. 10 is a flowchart of a process of the mobile terminal according to the embodiment. 10 is a flowchart of a process of a server (cloud) according to an embodiment. FIG. 10 is a diagram illustrating the application of the meshing process according to the embodiment to a simulation. FIG. 10 is a diagram illustrating the application of the meshing process according to the embodiment to logistics. FIG. 10 is a diagram showing the application of the meshing process according to the embodiment to animal breeding. FIG. 10 is a diagram illustrating the application of the meshing process according to the embodiment to remote control of a construction machine. 1 is a block diagram of a communication support system according to an embodiment;

Embodiments Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and repeated explanations are omitted as necessary.

(Outline of the present embodiment and meshing method)
Fig. 1 is a diagram showing remote measurement in infrastructure inspection etc. according to an embodiment. With reference to Fig. 1, an outline of this embodiment will be described using as an example the maintenance, operation, management, inspection and monitoring of infrastructure facilities in electric power companies and telecommunications companies.

As shown in Figure 1, when a 3D sensor is used to measure the size of utility poles, the length of power lines, or the distance between utility poles and power lines and buildings, the 3D point cloud data can only be viewed on measuring devices such as tablets held by workers at high altitudes. As a result, supervisors who are not at the on-site measurement location, such as at a remote business office, cannot view the 3D point cloud data. Because 3D point cloud data has a large data volume and requires a high processing load, it is difficult to transfer the 3D point cloud data to other workers' devices and display and share it in real time. Furthermore, it is difficult for other workers to understand the situation on-site if the display is entirely 3D point cloud data.

Therefore, by meshing non-target areas to reduce the load, measurement results and comments on them can be shared in real time, making it easier to understand the situation on site. Figure 2 is a diagram showing the first meshing method according to an embodiment. As shown in Figure 2, a pre-prepared mesh of buildings other than the target area can be obtained, and this mesh can be overlaid on the captured 3D point cloud for display. The pre-prepared building mesh may use external data, such as city data from PLATEAU.

Figure 3 is a diagram showing a second meshing technique according to an embodiment. As shown in Figure 3, a 3D point cloud of a landscape including a target is photographed, and the landscape other than the target is meshed, and the mesh and the 3D point cloud are displayed superimposed. This makes it easy to process and apply the results, such as recording them as digital information and adding information on the spot.

(Configuration of the system according to the embodiment)
Fig. 4 is a diagram showing the configuration of a system according to an embodiment. The configuration of the system according to the embodiment will be described with reference to Fig. 4. The system includes a server (cloud) 100 and a plurality of mobile terminals 200.

The server (cloud) 100 is a computer server or a cloud, etc. The server (cloud) 100 is connected to multiple mobile terminals 200 via a wired LAN (Local Area Network), a wireless LAN, an Internet line, a mobile phone line, etc.

The mobile terminal 200 is a personal computer, tablet, or smartphone. The mobile terminal 200 may also be a glasses-type device that realizes VR (Virtual Reality), AR (Augmented Reality), or MR (Mixed Reality). The mobile terminal 200 may also be a 3D display. The mobile terminal 200 may be carried by a user. The mobile terminal 200 may also be a system that exists on-site and exchanges information with the user. For example, the mobile terminal 200 is a system that presents information to the user using a projector and acquires information from the user using a sensor or voice. The mobile terminal 200 acquires image data 300 using a 3D sensor and transmits and receives data in real time with other mobile terminals 200 via the server (cloud) 100. The mobile terminal 200 may only acquire data from the server (cloud) 100, like a supervisor's terminal, without acquiring image data 300 using a 3D sensor.

(Description of a mobile terminal according to an embodiment)
5 is a diagram illustrating a configuration of a mobile terminal according to an embodiment. The configuration of the mobile terminal according to the embodiment will be described with reference to FIG. 5. The mobile terminal 200 includes a data collection unit 210, an information display unit 211, an information assignment unit 212, and a data transmission unit 213.

The data collection unit 210 is a part that has the function of acquiring shooting data 300 acquired by 3D sensors such as LiDAR, ToF cameras, or stereo sensors, RGB cameras, or other sensors, as well as data stored on the server (cloud) 100. The 3D sensor may capture images while the photographer is moving or while the 3D sensor is moving. Multiple 3D sensors may also be used.

The information display unit 211 is a display device such as a liquid crystal display device or an organic EL (Electro Luminescence) display device. The information display unit 211 is connected to the data collection unit and has the function of displaying information acquired by the data collection unit 210. The information display unit 211 displays 3D point cloud data and meshed mesh screen data. The information display unit 211 may be expressed in a virtual space or may be displayed in an easy-to-understand manner using a user's avatar, but the expression method is not limited to these. Furthermore, the information display unit 211 may not only display the 3D point cloud data and meshed mesh screen data, but may also overlay them to make them easier to see. Furthermore, the information display unit 211 may switch between the 3D point cloud data and the meshed mesh screen data and adjust the degree of overlap. Furthermore, the information display unit 211 may select whether to display the 3D point cloud data or the meshed data.

The information assigning unit 212 is connected to the information display unit 211 and has the function of assigning information to data displayed on the information display unit 211. The information assigning unit 212 may update and edit data acquired by the data collection unit 210. The information assigning unit 212 also generates information from other functions of the mobile terminal 200 (such as the camera and GPS (Global Positioning System)) and generates original information by the user. Specifically, the information assigning unit 212 assigns information such as location information such as the latitude and longitude of the location where the image was taken, the direction in which the image was taken, and information identifying the object to be displayed as a 3D point cloud. The information assigning unit 212 also assigns information such as comments about the object. For example, the information assigning unit 212 works in conjunction with a touch sensor, a touch pen, or the like installed on the information display unit 211.

The data transmission unit 213 is connected to the information providing unit 212 and has the function of transmitting the information provided by the information providing unit 212 to the server (cloud) 100. As described above, the data transmission unit 213 is connected to a wireless LAN, a wired LAN, an Internet line, a mobile phone line, or the like.

(Description of Server (Cloud) According to the Embodiment)
Fig. 6 is a diagram illustrating a configuration of a server (cloud) according to an embodiment. The configuration of the server (cloud) according to the embodiment will be described with reference to Fig. 6. The server (cloud) 100 includes a location information acquisition unit 110, a data collection unit 111, an information management unit 112, and a data transmission unit 115. The information management unit 112 includes a mesh processing unit 113 and an information storage unit 114.

The location information acquisition unit 110 is a part that has the function of acquiring location information of the terminal user, the subject to be photographed, etc. The data collection unit 111 is a part that is connected to the data transmission unit 213 of the mobile terminal 200 and has the function of collecting data transmitted from the mobile terminal 200. The location information acquisition unit 110 is connected to the data collection unit 111 and acquires location information of the terminal user, the subject to be photographed, etc. from the mobile terminal 200. The data collection unit 111 acquires mesh data from an external system or acquires 3D point cloud data from the mobile terminal 200.

The information management unit 112 is connected to the data collection unit 111 and receives mesh data or 3D point cloud data. The mesh processing unit 113 of the information management unit 112 has the function of converting mesh data or meshing point clouds other than the target. The mesh processing unit processes mesh data acquired from an external source so that it can be applied to this system. Alternatively, the mesh processing unit 113 meshes 3D point clouds, converts them into polygons, or converts them into CAD data. The mesh processing unit 113 may also perform meshing processing in conjunction with an external system. Here, external systems include CAD systems, as-built drawing management systems, drawing management systems, and GIS (Geographic Information Systems). The meshing process does not limit the objects to be meshed by distinguishing between targets and non-targets. Furthermore, the meshing process may determine whether to mesh and the granularity of the meshing based on processing power, number of point clouds, density, accuracy, error, etc. The granularity of the meshing refers to the number of polygons and the accuracy of the number of operations. In addition, the meshing process may determine the target to be meshed and the granularity of the meshing based on instructions from a person or in cooperation with external data.

The information storage unit 114 of the information management unit 112 is a magnetic recording medium (e.g., a flexible disk, magnetic tape, or hard disk drive), a magneto-optical recording medium (e.g., a magneto-optical disk), or a CD-ROM (Read Only Memory). The information storage unit 114 of the information management unit 112 is also a CD-R or CD-R/W. The information storage unit 114 of the information management unit 112 is also a semiconductor memory (e.g., a mask ROM, a PROM (Programmable ROM), an EPROM (Erasable PROM), a flash ROM, or a RAM (Random Access Memory)). The information storage unit 114 of the information management unit 112 is a part that has the function of storing 3D point cloud data of the target and data of scenery other than the target that has been meshed by the mesh processing unit 113. The information storage unit 114 may store a temporal history of the data and perform meshing processing based on the history and changes over time. The information storage unit 114 may also store meshed data and data that is not subject to meshing and use the data to determine whether or not a data item is subject to meshing. The information storage unit 114 may also store the data and use the data for object detection (searching and matching for objects, etc.) and object tracking. These processes can reduce the weight and simplify the point cloud processing. The mobile terminal 200 may also include a mesh processing unit 113. The mobile terminal 200 may transmit 3D point cloud data, a mesh, or both to the server (cloud) 100.

The data transmission unit 115 is connected to the information management unit 112 and the mobile terminal 200. The data transmission unit 115 has the function of transmitting 3D point cloud data of the object stored in the information management unit 112 and data of scenery other than the object meshed by the mesh processing unit 113 to each mobile terminal 200 in response to a request received from each terminal. The data transmission unit 115 is connected to a wireless LAN, a wired LAN, an Internet line, a mobile phone line, etc.

(Explanation of Flowchart of Mobile Terminal According to the Embodiment)
7 is a flowchart of the process of the mobile terminal according to the embodiment, which will be described with reference to FIG.

First, the mobile device 200 captures an image of an object or an object and a landscape using a 3D sensor or the like (Step A1). Here, a landscape refers to the portion of the image captured by cutting out the object, specifically a factory or warehouse, a zoo or ranch, a construction site, or a location where infrastructure is installed. Furthermore, an object refers to luggage, an animal, a portion of a construction site that the user wishes to check or operate construction machinery on, or infrastructure such as utility poles and power lines. Next, the mobile device 200 acquires data such as location information and the location of the image capture using the data collection unit 210 (Step A2). Next, the mobile device 200 displays the information using the information display unit 211 (Step A3). If data stored on the server (cloud) 100 is displayed instead of data acquired using a 3D sensor, Step A1 may be omitted. The data collection unit 210 may perform point cloud processing, meshing, or preprocessing for point cloud processing and meshing. Next, the mobile terminal 200 determines an object from the photographed scenery as needed by the information providing unit 212, and provides information such as providing a comment (step A4). Finally, the mobile terminal 200 transmits the data to the server (cloud) 100 (step A5).

(Description of Flowchart of Server According to Embodiment)
8 is a flowchart of the processing of the server according to the embodiment, which will be described with reference to FIG.

First, the server (cloud) 100 acquires location information of the terminal and the shooting data (shooting location, shooting direction, shooting settings and conditions, and additional information) using the location information acquisition unit 110 (step B1). Next, the server (cloud) 100 acquires 3D point cloud data of the photographed object or the object and scenery using the data collection unit 111 (step B2). In this step B2, point cloud data from the sensors photographed may be combined, and multiple sensors may be synthesized (registered). Alignment of the point cloud data using location information and external data may be performed in this step B2, or may be performed after meshing.

Next, the server (cloud) 100 acquires a mesh screen using the mesh processing unit 113, or creates a mesh screen by meshing the 3D point cloud data of the scenery other than the determined target (step B3). Meshing may be performed on the mobile terminal 200 at the site, rather than on the server (cloud) 100. The 3D mesh may be created before capturing an image of the target with a 3D sensor and acquiring the 3D point cloud data, or after capturing the 3D point cloud data. The created mesh screen may also be matched with a mesh (polygon, CAD information, model, etc.) stored externally or in the information storage unit 114, and aligned and identified. In this step, this is treated as "meshing." The point cloud processing and meshing processes may involve pre-processing and post-processing, such as noise removal and correction.

Next, the server (cloud) 100 stores all or any necessary processing data in the information storage unit 114 (step B4). For example, the server (cloud) 100 stores the target 3D point cloud data and a mesh screen of the landscape. By updating all or any necessary data (step B5), real-time information is displayed. The mesh granularity and accuracy are adjusted based on human instructions, collaboration with external systems, the real-time nature of internal processing, and resource availability, and real-time information is displayed.

Next, the server (cloud) 100 adds additional information as needed, such as the information added using the mobile terminal 200 in step A4 (step B6). Finally, the server transmits the target 3D point cloud data and data such as a mesh screen of the landscape to each mobile terminal 200 (step B7). The information display unit 211 simultaneously displays the target 3D point cloud data and mesh screen of the landscape sent to each mobile terminal 200.

With this embodiment, when acquiring a point cloud using a 3D sensor to measure the distance to utility poles and power lines, the image can be displayed by meshing everything except the target and showing only the target as a point cloud. This makes the operation screen easier to understand, reduces data volume and processing load, and allows images to be displayed in real time and shared with other devices.

(Description of application of meshing processing of this embodiment to simulation)
9 is a diagram showing the application of the meshing process according to the embodiment to a simulation. With reference to FIG. 9, the application of the meshing process according to the embodiment to a simulation will be described.

As shown in Figure 9, point cloud data is clustered (segmented) or meshed into a model object, which is then used in the simulation. For example, when measuring the distance between power lines or communication lines, if the captured 3D data shows a distance violation, the 3D point cloud or model of the existing power lines can be moved. In this way, it is possible to find a location where the distance violation can be resolved. A function can also be provided to automatically find a location where the distance violation can be resolved.

(Description of application of meshing processing of this embodiment to logistics)
10 is a diagram showing the application of the meshing process according to the embodiment to physical distribution. With reference to FIG. 10, the application of the meshing process according to the embodiment to physical distribution will be described.

As shown in Figure 10, the meshing process according to this embodiment can be applied to logistics and warehouse operations management. For example, the meshing process according to this embodiment can be used in situations such as measuring the size of packages on a conveyor belt in a factory or warehouse or checking for abnormalities. If a 3D sensor were used to acquire and measure all 3D point cloud data for packages in a factory or warehouse, the data volume would be large, the processing load would be high, and because the packages move, it would not be possible to display them in real time. However, by meshing objects other than packages, the data volume is reduced, allowing packages in a factory or warehouse to be displayed in real time and the data to be shared. For example, an event camera can be used in combination to display moving objects as a 3D point cloud and stationary objects as a mesh.

(Description of application of meshing processing according to this embodiment to animal breeding)
11 is a diagram showing the application of the meshing process according to the embodiment to animal breeding. With reference to FIG. 11, the application of the meshing process according to the embodiment to animal breeding will be described.

As shown in Figure 11, the meshing process according to this embodiment can be applied to situations where animals are kept, managed, or monitored. Using a 3D sensor to acquire 3D point cloud data to measure the size of animals at a zoo or farm results in a large data volume, a high processing load, and the animals move, making it impossible to display them in real time. However, by meshing the zoo and farm, which are not the subject of animals, the data volume is reduced, allowing the animals at the zoo or farm to be displayed in real time and the data to be shared. For example, an event camera can be used in combination to display moving objects as a 3-point cloud and stationary objects as a mesh.

(Description of application of meshing processing according to this embodiment to remote operation of construction machinery)
12 is a diagram illustrating the application of the meshing process according to the embodiment to the remote control of a construction machine. The application of the meshing process according to the embodiment to the remote control of a construction machine will be described with reference to FIG.

As shown in Figure 12, the meshing process of this embodiment can be applied to situations where construction machinery is remotely operated at a construction site. When operating construction machinery remotely, attempting to display the entire vast site as a 3D point cloud results in a high processing load. Here, by displaying only the parts of the construction site that the user wants to check or the parts of the construction machinery that the user wants to operate as a 3D point cloud, and displaying the rest as a mesh, the overall picture is easier to grasp, the parts that the user wants to operate can be displayed with high accuracy in real time, and the data can be shared.

Furthermore, some or all of the processing in the above-mentioned server (cloud) 100 and mobile terminal 200 can be implemented as a computer program. Such programs can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Furthermore, the programs may be supplied to a computer by various types of temporary computer-readable media. Examples of the temporary computer-readable medium include an electric signal, an optical signal, and an electromagnetic wave. The temporary computer-readable medium can provide the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

13 is a block diagram of a real-time communication support system according to an embodiment of the present invention, which will be described with reference to FIG.
The real-time communication support system 10 according to this embodiment comprises a means 11 for photographing an object, a means 12 for acquiring 3D point cloud data from the photographed object, a means 13 for acquiring a mesh screen of the scenery in which everything other than the object has been meshed, a means 14 for saving the 3D point cloud data of the object and the mesh screen of the scenery, and a means 15 for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.

This embodiment makes it possible to provide a real-time communication support system that simultaneously displays detailed 3D point cloud data and small-volume mesh data.

The above describes an embodiment of the present invention, but the present invention includes appropriate modifications that do not impair its objects and advantages, and is not limited to the above embodiment.

A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.
(Appendix 1)
a means for photographing the subject;
means for acquiring 3D point cloud data from the photographed object;
A means for obtaining a mesh screen of a landscape in which everything other than the target is meshed;
means for storing the 3D point cloud data of the object and the mesh view of the scene;
and means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.
(Appendix 2)
a mobile terminal including a means for photographing the object and a means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery;
A real-time communication support system as described in Appendix 1, comprising: a server having: means for acquiring 3D point cloud data from the photographed object; means for acquiring a mesh screen of the scenery in which everything other than the object has been meshed; and means for saving the 3D point cloud data of the object and the mesh screen of the scenery.
(Appendix 3)
A means of photographing the landscape, including the subject,
means for determining the object from the photographed scene;
means for acquiring 3D point cloud data from the photographed scene;
a means for generating a mesh screen of the scenery by meshing the 3D point cloud data other than the determined object;
means for storing the 3D point cloud data of the object and the mesh view of the scene;
and means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.
(Appendix 4)
a mobile terminal including: a means for photographing a landscape including the object; a means for determining the object from the photographed landscape; and a means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the landscape;
A real-time communication support system as described in Appendix 3, comprising a server having: means for acquiring 3D point cloud data from the photographed scenery; means for meshing the 3D point cloud data other than the determined object to create the mesh screen of the scenery; and means for saving the 3D point cloud data of the object and the mesh screen of the scenery.
(Appendix 5)
A real-time communication support system according to any one of appendices 1 to 4, comprising means for simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scenery.
(Appendix 6)
The scene is a factory or warehouse and the object is a package, or
the scene is a zoo or farm and the objects are animals, or
The scene is a construction site, and the object is a part of the construction site that the user wants to check or a part of the construction site that the user wants to operate construction equipment on, or
A real-time communication support system according to any one of appendices 1 to 5, wherein the scenery is a location where infrastructure equipment is installed, and the target is the infrastructure equipment.
(Appendix 7)
A mobile terminal comprising: a means for photographing an object; and a means for simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape in which everything other than the object has been meshed.
(Appendix 8)
A server comprising: a means for acquiring 3D point cloud data from a photographed object; a means for acquiring a mesh screen of a landscape in which everything other than the object has been meshed; and a means for saving the 3D point cloud data of the object and the mesh screen of the landscape.
(Appendix 9)
photographing an object;
acquiring 3D point cloud data from the photographed object;
A step of obtaining a mesh screen of a landscape in which everything other than the target is meshed;
saving the 3D point cloud data of the object and the mesh view of the scene;
and simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.
(Appendix 10)
A non-transitory computer-readable medium that records a program that causes a mobile terminal to execute the steps of photographing an object and simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape in which everything other than the object has been meshed.
(Appendix 11)
A non-transitory computer-readable medium that records a program to be executed by a server, including steps of acquiring 3D point cloud data from a photographed object, acquiring a mesh screen of a landscape in which everything other than the object has been meshed, saving the 3D point cloud data of the object and the mesh screen of the landscape.
(Appendix 12)
A mobile terminal comprising: a means for photographing a landscape including an object; a means for determining the object from the photographed landscape; and a means for simultaneously displaying 3D point cloud data of the object and a mesh screen of the landscape other than the object.
(Appendix 13)
A server comprising: a means for acquiring 3D point cloud data from a landscape including a photographed object; a means for meshing the 3D point cloud data of the landscape other than the object to create a mesh screen of the landscape; and a means for saving the 3D point cloud data of the object and the mesh screen of the landscape.
(Appendix 14)
Photographing a landscape including a subject;
determining the object from the captured scene;
acquiring 3D point cloud data from the captured scene;
a step of meshing the 3D point cloud data other than the determined object to create a mesh screen of the scenery;
saving the 3D point cloud data of the object and the mesh view of the scene;
and simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.
(Appendix 15)
A real-time communication support method as described in Appendix 14, which maintains a temporal history of the step of saving the 3D point cloud data of the object and the mesh screen of the scenery, and performs a step of meshing the 3D point cloud data other than the determined object based on the change in the history over time to create a mesh screen of the scenery.
(Appendix 16)
A non-transitory computer-readable medium that records a program that causes a mobile terminal to execute the following steps: photographing a landscape including an object; determining the object from the photographed landscape; and simultaneously displaying 3D point cloud data of the object and a mesh screen of the landscape other than the object.
(Appendix 17)
A non-transitory computer-readable medium that records a program that causes a server to execute the following steps: acquiring 3D point cloud data from a landscape including a photographed object; meshing the 3D point cloud data of the landscape other than the object to create a mesh screen of the landscape; and saving the 3D point cloud data of the object and the mesh screen of the landscape.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above. Various modifications that would be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

This application claims priority based on Japanese Patent Application No. 2022-051437, filed March 28, 2022, the entire disclosure of which is incorporated herein by reference.

10 Real-time communication support system 11 Means for photographing an object 12 Means for acquiring 3D point cloud data 13 Means for acquiring a mesh screen 14 Means for saving 3D point cloud data and a mesh screen 15 Means for simultaneously displaying 3D point cloud data and a mesh screen 100 Server (cloud)
110 Position information acquisition unit 111 Data collection unit 112 Information management unit 113 Mesh processing unit 114 Information storage unit 115 Data transmission unit 200 Portable terminal 210 Data collection unit 211 Information display unit 212 Information assignment unit 213 Data transmission unit

Claims (9)

a means for photographing the subject;
means for acquiring 3D point cloud data from the photographed object;
A means for obtaining a mesh screen of a landscape in which everything other than the target is meshed;
means for storing the 3D point cloud data of the object and the mesh view of the scene;
means for simultaneously displaying the 3D point cloud data of the object and the mesh image of the scenery;
and means for simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scenery . a mobile terminal including a means for photographing the object and a means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery;
The real-time communication support system of claim 1, comprising a server having: means for acquiring 3D point cloud data from the photographed object; means for acquiring a mesh screen of the scenery in which everything other than the object has been meshed; and means for saving the 3D point cloud data of the object and the mesh screen of the scenery. A means of photographing the landscape, including the subject,
means for determining the object from the photographed scene;
means for acquiring 3D point cloud data from the photographed scene;
a means for generating a mesh screen of the scenery by meshing the 3D point cloud data other than the determined object;
means for storing the 3D point cloud data of the object and the mesh view of the scene;
means for simultaneously displaying the 3D point cloud data of the object and the mesh image of the scenery;
and means for simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scenery . a mobile terminal including: a means for photographing a landscape including the object; a means for determining the object from the photographed landscape; and a means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the landscape;
The real-time communication support system of claim 3, comprising a server having: means for acquiring 3D point cloud data from the photographed scenery; means for meshing the 3D point cloud data other than the determined object to create the mesh screen of the scenery; and means for saving the 3D point cloud data of the object and the mesh screen of the scenery. The scene is a factory or warehouse and the object is a package, or
the scene is a zoo or farm and the objects are animals, or
The scene is a construction site, and the object is a part of the construction site that the user wants to check or a part of the construction site that the user wants to operate construction equipment on, or
The real-time communication support system according to claim 1 , wherein the scenery is a location where infrastructure facilities are installed, and the object is the infrastructure facilities. a means for photographing an object, and a means for simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape in which everything other than the object is meshed;
and means for simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scene . A means for acquiring 3D point cloud data from a photographed object, a means for acquiring a mesh screen of a landscape in which everything other than the object is meshed, and a means for saving the 3D point cloud data of the object and the mesh screen of the landscape;
and means for simulating the movement of the 3D point cloud data of the object relative to the mesh view of the scene . photographing an object;
acquiring 3D point cloud data from the photographed object;
A step of obtaining a mesh screen of a landscape in which everything other than the target is meshed;
saving the 3D point cloud data of the object and the mesh view of the scene;
simultaneously displaying the 3D point cloud data of the object and the mesh view of the scene;
and simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scenery . a step of photographing an object; and a step of simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape other than the object, the mesh screen being a mesh of the landscape;
and a step of simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scenery .