WO2022095543A1 - Image frame stitching method and apparatus, readable storage medium, and electronic device - Google Patents

Abstract

Disclosed are an image frame stitching method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring a preview video stream captured by means of moving a panoramic photography device in a preset space; in response to multiple photography instructions received when moving the panoramic photography device, acquiring images of multiple locations in the preset space by means of the panoramic photography device, so as to obtain multiple scene images; estimating corresponding pose information of the multiple scene images on the basis of the preview video stream; and stitching the multiple scene images on the basis of the corresponding pose information of the multiple scene images, so as to obtain a panoramic image of the preset space.

Description

Image frame stitching method and device, readable storage medium and electronic device technical field

The present disclosure relates to computer vision technology, and in particular, to an image frame stitching method and apparatus, a computer-readable storage medium, an electronic device, and a computer program product.

Background technique

With the popularization and application of terminals in people's lives, users can use the terminal to shoot panoramic images. The panoramic image in the related art is based on splicing multiple images to achieve a wide-angle effect to show more scenes. However, for some special scenes, such as repetitive textures in buildings, occlusion of walls, and similar spaces, the problem of wrong stitching often occurs.

SUMMARY OF THE INVENTION

According to an aspect of the embodiments of the present disclosure, there is provided an image frame stitching method, comprising: acquiring a preview video stream captured by moving a panoramic photographing device in a set space; in response to receiving a video stream in the process of moving the panoramic photographing device The obtained multiple shooting instructions, obtain images of multiple positions in the set space through the panoramic shooting device to obtain multiple frames of scene images; estimate the corresponding poses of the multiple frames of scene images based on the preview video stream information; based on the corresponding pose information of the multi-frame scene images, stitching the multi-frame scene images to obtain a panoramic image of the set space.

According to another aspect of the embodiments of the present disclosure, there is provided an image frame splicing apparatus, including a device for implementing the above-mentioned image frame splicing method.

According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, where the computer-readable storage medium stores a computer program, and the computer program is used to execute the above-mentioned image frame stitching method.

According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device includes: a processor; a memory for storing instructions executable by the processor; a processor for reading the executable instructions from the memory, and The instructions are executed to implement the above image frame stitching method.

According to another aspect of the embodiments of the present disclosure, there is provided a computer program product including a computer program, wherein the computer program implements the above-mentioned image frame stitching method when executed by a processor.

The technical solutions of the present disclosure will be further described in detail below through the accompanying drawings and embodiments.

Description of drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification, and are used to explain the present disclosure together with the embodiments of the present disclosure, and do not limit the present disclosure. In the drawings, the same reference numbers generally refer to the same components or steps.

FIG. 1 is a flowchart of an image frame stitching method according to an embodiment of the present disclosure.

FIG. 2 is a flowchart of an image frame stitching method according to yet another embodiment of the present disclosure.

FIG. 3 is a flowchart of an image frame stitching method according to still another embodiment of the present disclosure.

FIG. 4 is a flowchart of an image frame stitching method according to another embodiment of the present disclosure.

FIG. 5 is a schematic structural diagram of an image frame stitching apparatus according to an embodiment of the present disclosure.

FIG. 6 is a structural diagram of an electronic device according to an exemplary embodiment of the present disclosure.

Detailed ways

Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present disclosure, not all of the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited by the example embodiments described herein.

It should be noted that the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

Those skilled in the art can understand that terms such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, etc., and neither represent any specific technical meaning, nor represent any difference between them. the necessary logical order of .

It should also be understood that, in the embodiments of the present disclosure, "a plurality" may refer to two or more, and "at least one" may refer to one, two or more.

It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more in the case of no explicit definition or contrary indications given in the context.

In addition, the term "and/or" in the present disclosure is only an association relationship to describe associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, and A and B exist at the same time , there are three cases of B alone. In addition, the character "/" in the present disclosure generally indicates that the related objects are an "or" relationship.

It should also be understood that the description of the various embodiments in the present disclosure emphasizes the differences between the various embodiments, and the same or similar points can be referred to each other, and for the sake of brevity, they will not be repeated.

Meanwhile, it should be understood that, for the convenience of description, the dimensions of various parts shown in the accompanying drawings are not drawn in an actual proportional relationship.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application or uses in any way.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

Embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal equipment, computing systems, environments and/or configurations suitable for use with terminal equipment, computer systems, servers, etc. electronic equipment include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients computer, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the foregoing, among others.

Electronic devices such as terminal devices, computer systems, servers, etc., may be described in the general context of computer system-executable instructions, such as program modules, being executed by the computer system. Generally, program modules may include routines, programs, object programs, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer systems/servers may be implemented in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

FIG. 1 is a flowchart of an image frame stitching method according to an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic equipment. As shown in FIG. 1 , the image frame splicing method includes the following steps:

S102: Acquire a preview video stream shot by moving the panoramic shooting device in the set space.

The setting space may be an indoor room or an outdoor place. Panoramic shooting device is used to refer to a device provided with a panoramic shooting camera and a controller, wherein the panoramic shooting camera can be a fisheye panoramic camera, a multi-lens panoramic camera, or a mobile client that can generate panoramic shooting effects; the controller can include SLAM ( real-time positioning and map construction) system. The preview video stream is used to represent continuous image frame data generated after the mobile panorama shooting device is initialized.

S104, in response to the multiple shooting instructions received during the process of moving the panoramic shooting device, acquire images of multiple positions in the set space through the panoramic shooting device to obtain multiple frames of scene images.

In the embodiment of the present disclosure, the preview video stream can be viewed in real time through a remote device such as a mobile phone, and a shooting instruction can be sent through the remote device to realize remote control.

S106, estimating corresponding pose information of the multi-frame scene images based on the preview video stream.

The corresponding pose information of the multi-frame scene images is used to represent the displacement and posture of the panoramic photographing device corresponding to the corresponding scene images in the multi-frame scene images.

S108, based on the corresponding pose information of the multi-frame scene images, stitching the multi-frame scene images to obtain a panoramic image of the set space.

For example, take a panoramic shot of the entire source A. First, the preview video stream a1 is obtained by moving the fisheye panoramic camera in the room A1 of the house A. After obtaining the preview video stream a1, the user continuously shoots the room A1 with the fisheye panoramic camera to obtain images of multiple positions of the room A1, and obtains multiple frames of scene images of the room A1. Then, the user continues to move the fisheye panoramic camera to the next room A2, and shoots the room A2 in the same way. After the shooting of all rooms in the house A is completed, the corresponding pose information of the multi-frame scene images of the house A is estimated based on all the preview video streams, and the multi-frame scene images of the house A are determined according to the corresponding pose information. interrelationships between. By stitching together adjacent scene images, a panoramic image of house A can be obtained.

According to the image frame stitching method of the embodiment of the present disclosure, a preview video stream captured by moving a panoramic photographing device in a set space is obtained; in response to a plurality of photographing instructions received in the process of moving the panoramic photographing Obtaining images of multiple positions in the set space to obtain multi-frame scene images; estimating the corresponding pose information of the multi-frame scene images based on the preview video stream; Stitch to get a panoramic image of the set space. The embodiments of the present disclosure can effectively solve the problem of incorrect splicing of scene images in a panoramic image by using the corresponding pose information of multiple frames of scene images. In addition, by using the preview video stream of the panoramic photographing device, the global pose of the panoramic photographing device for photographing the set space can also be estimated, so as to obtain an accurate panoramic image.

In some embodiments, before step S106, it may further include: removing the moving object in the preview video stream to obtain a preview video stream with the moving object removed, then step S106 may further include: based on the preview from which the moving object is removed The video stream estimates the corresponding pose information for multiple frames of scene images.

FIG. 2 is a schematic flowchart of an image frame stitching method according to another exemplary embodiment of the present disclosure. The above-mentioned removal of the moving target in the preview video stream may include the following steps:

S201. Perform moving object detection on a scene image in a preview video stream to determine whether a moving object is detected.

The moving target can be a person or an animal.

S202, in response to detecting the moving target, remove the moving target based on a preset second neural network.

The embodiments of the present disclosure can determine whether there is a moving target through feature point detection. The preset second neural network is used to represent the neural network that detects moving objects and removes moving objects, for example, SSD (single deep neural network model, Single Shot MultiBox Detector), Yolo (you can recognize your model at a glance, You Only look once), Deeplab (a hole convolution model).

The embodiment of the present disclosure can remove redundant moving objects in the scene image, so that the image frame information is more complete and accurate.

FIG. 3 is a schematic flowchart of an image frame stitching method according to another exemplary embodiment of the present disclosure. On the basis of the embodiment shown in FIG. 1 above, step S106 may specifically include the following steps:

S301 , based on the real-time positioning and mapping algorithm and the loop closure detection algorithm, process the motion trajectory of the panoramic photographing device to estimate the pose information of the panoramic photographing device corresponding to the scene image in the preview video stream.

The Live Localization and Mapping (SLAM) algorithm and the loop closure detection algorithm are pre-stored in the Live Localization and Mapping (SLAM) system. The purpose of the real-time localization and mapping (SLAM) algorithm is to estimate the pose at each moment in the motion trajectory of the panoramic shooting device; the purpose of the loop closure detection algorithm is to find out whether the current scene has appeared in history, and if it has occurred, it can be used accordingly. Provide a very strong constraint, that is, correct the trajectory of the panorama shooting equipment with a large deviation to the correct position.

S302 , based on the pose information of the panoramic photographing device corresponding to the scene image in the preview video stream, obtain corresponding pose information of multiple frames of scene images.

Therefore, the embodiments of the present disclosure can use the real-time positioning and mapping algorithm and the loop closure detection algorithm to estimate the pose information of the panoramic shooting device at each moment, so as to realize the estimation of the relative displacement and relative rotation between the scene images, so as to Ensure smooth jumping between scene images of each frame.

In some embodiments, before step S108, the following steps may be further included: acquiring the pose scale of the panoramic shooting device, then step 108 may further include: based on the pose scale of the panoramic shooting device and the corresponding pose information of the multi-frame scene images, Stitches multi-frame scene images.

The pose scale is used to represent the ratio of the on-map distance in the multi-frame scene image to the corresponding actual distance in the set space.

In some embodiments, obtaining the pose scale of the panoramic shooting device may include the following steps: obtaining the pose scale of the panoramic shooting device based on the actual distance between the panoramic shooting device and the fixed reference object; or based on a preset first The neural network processes the preview video stream to obtain the pose scale of the panoramic shooting device.

A fixed reference can be the floor or ceiling of a room. For example, if the distance between the observation point and the ground in the multi-frame scene image is 1, and the actual distance between the fisheye panoramic camera placed on the tripod and the ground is 1.5 meters, the observation point and the multi-frame scene image The ratio between the distance between the ground and the actual distance between the fisheye panoramic camera placed on the tripod and the ground is 1:1.5; or, through the preset first neural network, that is, the neural network that obtains depth information, The preview video stream is processed to determine the pose scale of the fisheye panoramic camera. For example, by inputting the preview video stream data into the convolutional neural network model trained on the test set, the pose scale of the fisheye panoramic camera can be obtained.

In this embodiment of the present disclosure, the pose scale of the panoramic photographing device is obtained by using the actual distance between the panoramic photographing device and the fixed reference object or by inputting the preview video stream into a preset first neural network, so as to determine the position and orientation of the panoramic photographing device in the multi-frame scene images. The distance correspondence between the information and the information in the actual scene.

FIG. 4 is a schematic flowchart of an image frame stitching method according to another exemplary embodiment of the present disclosure. On the basis of the embodiment shown in FIG. 1 above, step S108 may specifically include the following steps:

S401: Determine the splicing sequence of the multi-frame scene images based on the corresponding pose information of the multi-frame scene images.

The splicing sequence of the multi-frame scene images is used to represent the sequence of continuous changes of the pose corresponding to the panoramic photographing device, that is, the change of translation coordinates and the change of rotation coordinates.

S402: Determine a panoramic image of the set space based on the splicing sequence of the multiple frames of scene images.

In some embodiments, if there are scene images with overlapping images in the multiple frames of scene images, image fusion processing is performed on the overlapping part of the images.

For example, based on the splicing sequence of multiple frames of scene images, after the overlapping parts in adjacent image frames are fused, the image frames are spliced together according to the splicing sequence to obtain a panoramic image. In addition, the embodiment of the present disclosure can also project the panoramic image onto a spherical surface, a cylindrical surface or a cube, so as to realize all-round view browsing.

The embodiments of the present disclosure use the pose information of the multi-frame scene images to stitch the multi-frame scene images, which effectively solves the problem that when the panoramic shooting device encounters different panoramic images in a similar space, it is easy to give a wrong estimate, so that a multi-frame scene occurs. The problem of wrong image stitching.

Any image frame stitching method provided by the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any image frame stitching method provided by the embodiments of the present disclosure may be executed by a processor, for example, the processor executes any of the image frame stitching methods mentioned in the embodiments of the present disclosure by invoking corresponding instructions stored in the memory. No further description will be given below.

FIG. 5 is a schematic structural diagram of an image frame stitching apparatus according to an exemplary embodiment of the present disclosure. The apparatus can be set in electronic equipment such as terminal equipment and servers, so as to execute the image frame stitching method of any of the above-mentioned embodiments of the present disclosure. As shown in Figure 5, the device includes:

The first obtaining module 51 is configured to obtain a preview video stream captured by moving the panoramic shooting device in the set space;

The first obtaining module 52 is configured to, in response to multiple shooting instructions received in the process of moving the panoramic shooting device, obtain images of multiple positions in the set space through the panoramic shooting device to obtain Multi-frame scene images;

an estimation module 53, configured to estimate the corresponding pose information of the multi-frame scene images based on the preview video stream; and

The second obtaining module 54 is configured to stitch the multiple frames of scene images based on the corresponding pose information of the multiple frames of scene images to obtain a panoramic image of the set space.

Based on the image frame stitching apparatus provided by the above-mentioned embodiments of the present disclosure, a preview video stream captured by moving a panoramic photographing device in a set space is obtained; The shooting device acquires images of multiple positions in the set space to obtain multi-frame scene images; estimates the corresponding pose information of the multi-frame scene images based on the preview video stream; The images are stitched to obtain a panoramic image of the set space. The embodiments of the present disclosure can effectively solve the problem of incorrect splicing of scene images in panoramic images by using corresponding pose information images of multiple frames of scene images. In addition, by using the preview video stream in the panoramic photographing device, the global pose of the panoramic photographing device for photographing the set space can also be estimated, so as to obtain an accurate panoramic image.

In some embodiments, the estimation module 53 includes:

a removing unit configured to remove a moving object in the preview video stream to obtain a preview video stream with the moving object removed; and

The first estimation unit is configured to estimate the corresponding pose information of the multi-frame scene images based on the preview video stream from which the moving object is removed.

In some embodiments, the removal unit includes:

a first determining unit configured to perform moving object detection on the scene image in the preview video stream to determine whether a moving object is detected; and

The processing unit is configured to, in response to detecting the moving target, remove the moving target based on a preset second neural network.

In some embodiments, the estimation module 53 includes:

The second estimation unit is configured to process the motion trajectory of the panoramic photographing device based on the real-time positioning and mapping algorithm and the loop closure detection algorithm, so as to estimate the motion trajectory of the panoramic photographing device corresponding to the scene image in the preview video stream. pose information; and

The first obtaining unit is configured to obtain the corresponding pose information of the multi-frame scene images based on the pose information of the panoramic photographing device corresponding to the scene images in the preview video stream.

In some embodiments, the second obtaining module 54 includes:

The second obtaining unit is configured to obtain the pose scale of the panoramic photographing device, wherein the pose scale is used to represent the distance on the map in the multi-frame scene images and the corresponding actual distance in the set space distance ratio; and

The stitching unit is configured to stitch the multiple frames of scene images based on the pose scale of the panoramic photographing device and the corresponding pose information of the multiple frames of scene images to obtain a panoramic image of the set space.

In some embodiments, the second obtaining unit is configured to:

Obtain the pose scale of the panoramic photographing device based on the actual distance between the panoramic photographing device and a fixed reference object; or

Based on the preset first neural network, the preview video stream is processed to obtain the pose scale of the panoramic shooting device.

In some embodiments, the second obtaining module 54 includes:

a second determining unit configured to determine the splicing sequence of the multi-frame scene images based on the corresponding pose information of the multi-frame scene images; and

The third determination unit is configured to determine the panoramic image of the set space based on the splicing sequence of the multiple frames of scene images.

In some embodiments, it also includes:

The fusion module is configured to, in response to determining that at least one scene image in the multiple frames of scene images has image overlap, perform image fusion processing on the overlapping portion of the images.

Hereinafter, an electronic device according to an embodiment of the present disclosure will be described with reference to FIG. 6 . The electronic device may be either or both of the first device and the second device, or a stand-alone device independent of them that can communicate with the first device and the second device to receive the collected data from them input signal.

6 illustrates a block diagram of an electronic device according to an embodiment of the present disclosure.

As shown in FIG. 6 , electronic device 60 includes one or more processors 61 and memory 62 .

Processor 61 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in electronic device 60 to perform desired functions.

Memory 62 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory, or the like. The non-volatile memory may include, for example, read only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 61 may execute the program instructions to implement the image frame stitching method and/or the various embodiments of the present disclosure described above. Other desired features. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

In one example, the electronic device 60 may also include an input device 63 and an output device 64 interconnected by a bus system and/or other form of connection mechanism (not shown).

For example, when the electronic device is the first device or the second device, the input device 63 may be the above-mentioned microphone or microphone array for capturing the input signal of the sound source. When the electronic device is a stand-alone device, the input device 63 may be a communication network connector for receiving the collected input signals from the first device and the second device.

In addition, the input device 63 may also include, for example, a keyboard, a mouse, and the like.

The output device 64 can output various information to the outside, including the determined distance information, direction information, and the like. The output devices 64 may include, for example, displays, speakers, printers, and communication networks and their connected remote output devices, among others.

Of course, for simplicity, only some of the components in the electronic device 60 related to the present disclosure are shown in FIG. 6 , and components such as buses, input/output interfaces, and the like are omitted. Besides, the electronic device 60 may also include any other suitable components according to the specific application.

In addition to the methods and apparatus described above, embodiments of the present disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the "exemplary method" described above in this specification The steps in the image frame stitching method according to various embodiments of the present disclosure described in the section.

The computer program product may write program code for performing operations of embodiments of the present disclosure in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc. , also includes conventional procedural programming languages, such as "C" language or similar programming languages. The program code may execute entirely on the user computing device, partly on the user device, as a stand-alone software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.

In addition, embodiments of the present disclosure may also be computer-readable storage media having computer program instructions stored thereon that, when executed by a processor, cause the processor to perform the above-described "Example Method" section of this specification The steps in the image frame stitching method according to various embodiments of the present disclosure described in .

The computer-readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

The basic principles of the present disclosure have been described above with reference to specific embodiments. However, it should be pointed out that the advantages, advantages, effects, etc. mentioned in the present disclosure are only examples rather than limitations, and these advantages, advantages, effects, etc. should not be considered to be A must-have for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of example and easy understanding, but not for limitation, and the above details do not limit the present disclosure to be implemented by using the above specific details.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments may be referred to each other. As for the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

The block diagrams of devices, apparatus, apparatuses, and systems referred to in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will appreciate, these means, apparatuses, apparatuses, systems may be connected, arranged, and configured in any manner. Words such as "including", "including", "having" and the like are open-ended words meaning "including but not limited to" and are used interchangeably therewith. As used herein, the words "or" and "and" refer to and are used interchangeably with the word "and/or" unless the context clearly dictates otherwise. As used herein, the word "such as" refers to and is used interchangeably with the phrase "such as but not limited to".

The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Furthermore, in some embodiments, the present disclosure can also be implemented as programs recorded in a recording medium, the programs including machine-readable instructions for implementing methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

It should also be noted that, in the apparatus, device and method of the present disclosure, each component or each step may be decomposed and/or recombined. These disaggregations and/or recombinations should be considered equivalents of the present disclosure.

The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims (12)

An image frame splicing method, comprising: Get the preview video stream shot by moving the panorama shooting device in the set space; In response to a plurality of shooting instructions received in the process of moving the panoramic shooting device, acquiring images of multiple positions in the set space through the panoramic shooting device to obtain multiple frames of scene images; Estimating corresponding pose information of the multi-frame scene images based on the preview video stream; and Based on the corresponding pose information of the multi-frame scene images, the multi-frame scene images are stitched to obtain a panoramic image of the set space. The method according to claim 1, wherein the estimating the corresponding pose information of the multi-frame scene images based on the preview video stream comprises: removing the moving object in the preview video stream to obtain a preview video stream with the moving object removed; and Based on the preview video stream with the moving object removed, corresponding pose information of the multi-frame scene images is estimated. The method according to claim 2, wherein the removing the moving object in the preview video stream comprises: performing moving object detection on scene images in the preview video stream to determine whether a moving object is detected; and In response to detecting the moving object, the moving object is removed based on a preset second neural network. The method according to any one of claims 1-3, wherein the estimating the corresponding pose information of the multi-frame scene images based on the preview video stream comprises: processing the motion trajectory of the panoramic shooting device based on the real-time positioning and mapping algorithm and the loop closure detection algorithm to estimate the pose information of the panoramic shooting device corresponding to the scene image in the preview video stream; and Based on the pose information of the panoramic photographing device corresponding to the scene image in the preview video stream, the corresponding pose information of the multiple frames of scene images is acquired. The method according to any one of claims 1-4, wherein the multi-frame scene images are spliced based on the corresponding pose information of the multi-frame scene images to obtain a panoramic image of the set space, include: acquiring the pose scale of the panoramic photographing device, wherein the pose scale is used to represent the ratio of the distance on the map in the multi-frame scene images to the corresponding actual distance in the set space; and Based on the pose scale of the panoramic photographing device and the corresponding pose information of the multi-frame scene images, the multi-frame scene images are stitched to obtain the panorama image of the set space. The method according to claim 5, wherein the acquiring the pose scale of the panoramic photographing device comprises: Obtain the pose scale of the panoramic photographing device based on the actual distance between the panoramic photographing device and a fixed reference object; or Based on the preset first neural network, the preview video stream is processed to obtain the pose scale of the panoramic shooting device. The method according to any one of claims 1-4, wherein the multi-frame scene images are spliced based on the corresponding pose information of the multi-frame scene images to obtain a panoramic image of the set space, include: determining a splicing sequence of the multi-frame scene images based on the corresponding pose information of the multi-frame scene images; and Based on the splicing sequence of the multiple frames of scene images, a panoramic image of the set space is determined. The method according to any one of claims 1-7, further comprising: In response to determining that there is image overlap in at least one of the multiple frames of scene images, image fusion processing is performed on the overlapped portion of the images. An image frame splicing device, comprising: a device for implementing the method of any one of claims 1-8. A computer-readable storage medium, wherein the storage medium stores a computer program for performing the method of any one of claims 1-8. An electronic device comprising: processor; and A memory for storing instructions executable by the processor, wherein the executable instructions, when executed by the processor, implement the method of any of claims 1-8. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 1-8.

Images