WO2021258579A1 - Image splicing method and apparatus, computer device, and storage medium - Google Patents

Abstract

The present invention relates to an image splicing method and apparatus, a computer device, and a storage medium. The method comprises: obtaining images to be processed acquired from different visual angles, converting the images to be processed to be in a unified coordinate space, and obtaining converted images corresponding to the images to be processed; determining an overlapping region of the converted images; obtaining spatial domain optical flow information on the basis of pixel points of the converted images in the overlapping region, the spatial domain optical flow information being used for representing the degree of deviation of the pixel points in the overlapping region; and splicing the converted images according to the spatial domain optical flow information. By using the method, the image splicing quality can be improved.

Description

Image splicing method, device, computer equipment and storage medium

This disclosure is filed based on a Chinese patent application with an application number of 202010595138.6 and an application date of June 24, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference into this disclosure.

Technical field

The present disclosure relates to the technical field of image processing, and in particular, to an image splicing method, device, computer equipment, and storage medium.

Background technique

Image stitching technology is a technology that combines several images with overlapping parts into one image. It has important application value in the fields of panoramic imaging, intelligent security, medical image analysis, and virtual reality. Due to the limited field of view of a single camera, it is difficult to achieve imaging of a large field of view or even a full scene of field of view. Based on multiple images with different perspectives or video stream images collected by a single or multiple cameras, they can be spliced into a large-scale scene. Informational image.

In the traditional image stitching method, the image stitching is realized by extracting the feature points of each image to be stitched and matching the feature points. However, when there are short-distance foreground objects or moving objects in the images to be stitched, obvious stitches or ghosts are likely to appear in the images stitched by traditional methods, resulting in poor image stitching quality.

Summary of the invention

Based on this, it is necessary to provide an image splicing method, device, computer equipment, and storage medium that can improve the splicing quality in response to the above technical problems.

According to one aspect of the present disclosure, there is provided an image stitching method, the method including:

Acquiring each to-be-processed image collected from different perspectives, transforming each of the to-be-processed images into a unified coordinate space, and obtaining a converted image corresponding to each of the to-be-processed images;

Determining the overlapping area of each of the converted images;

Obtain spatial optical flow information based on the pixels of each of the converted images in the overlapping area, where the spatial optical flow information is used to characterize the degree of deviation of each pixel in the overlapping area;

According to the spatial optical flow information, stitching each of the converted images.

According to another aspect of the present disclosure, there is provided an image splicing device, the device including:

The image acquisition module is configured to acquire each to-be-processed image collected from different perspectives, convert each of the to-be-processed images to a unified coordinate space, and obtain a converted image corresponding to each of the to-be-processed images;

An overlapping area determining module, configured to determine the overlapping area of each of the converted images;

The optical flow information determining module is configured to obtain spatial optical flow information based on the pixel points of each of the converted images in the overlapping area, where the spatial optical flow information is used to characterize the offset of each pixel in the overlapping area degree;

The image stitching module is used to stitch each of the converted images according to the spatial optical flow information.

According to another aspect of the present disclosure, there is provided a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Acquiring each to-be-processed image collected from different perspectives, transforming each of the to-be-processed images into a unified coordinate space, and obtaining a converted image corresponding to each of the to-be-processed images;

Determining the overlapping area of each of the converted images;

Obtain spatial optical flow information based on the pixels of each of the converted images in the overlapping area, where the spatial optical flow information is used to characterize the degree of deviation of each pixel in the overlapping area;

According to the spatial optical flow information, stitching each of the converted images.

According to another aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and the computer program implements the following steps when executed by a processor:

Acquiring each to-be-processed image collected from different perspectives, transforming each of the to-be-processed images into a unified coordinate space, and obtaining a converted image corresponding to each of the to-be-processed images;

Determining the overlapping area of each of the converted images;

Obtain spatial optical flow information based on the pixels of each of the converted images in the overlapping area, where the spatial optical flow information is used to characterize the degree of deviation of each pixel in the overlapping area;

According to the spatial optical flow information, stitching each of the converted images.

The above-mentioned image splicing method, device, computer equipment and storage medium acquire each to-be-processed image collected from different perspectives, convert each to-be-processed image into a unified coordinate space, and obtain the converted image corresponding to each to-be-processed image; determine each converted image The overlapping area of the image; based on the pixels of each converted image in the overlapping area, the spatial optical flow information is obtained. The spatial optical flow information is used to characterize the offset degree of each pixel in the overlapping area; according to the spatial optical flow information, each conversion is performed After the image is stitched. Since the pixel offsets of objects with different depths of field are inconsistent in different viewing angles, the spatial optical flow information can effectively distinguish the foreground image and the background image in the overlapping area image. When image stitching is performed according to the spatial optical flow information, stitching can be made The thread bypasses the foreground object or moving object, effectively avoiding the seam or ghosting problem that is likely to occur when the stitching thread passes through the foreground object or moving object, thereby improving the quality of image stitching.

Description of the drawings

FIG. 1 is a schematic flowchart of an image stitching method in an embodiment;

Figure 2 shows two images to be processed collected from different perspectives in an embodiment;

Figure 3 shows two images to be processed and their overlapping areas in an embodiment;

FIG. 4 is an overlapping area of converted images corresponding to two images to be processed in an embodiment;

Figure 5 is an energy diagram obtained based on spatial optical flow information in an embodiment;

Fig. 6 is a mask diagram for processing each converted image in an embodiment;

Fig. 7 is a converted image after mask image processing in an embodiment;

Fig. 8 is a converted image after mask image processing in an embodiment;

Figure 9 is an image after stitching in an embodiment;

FIG. 10 is a schematic flowchart of an image stitching method in an embodiment;

Figure 11 is a structural block diagram of an image splicing device in an embodiment;

Figure 12 is an internal structure diagram of a computer device in an embodiment;

Fig. 13 is an internal structure diagram of a computer device in an embodiment.

detailed description

In order to make the objectives, technical solutions, and advantages of the present disclosure clearer, the following further describes the present disclosure in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present disclosure, but not used to limit the present disclosure.

In one embodiment, as shown in FIG. 1, an image stitching method is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to A system that includes a terminal and a server, and is realized through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. In this embodiment, the method includes the following steps S102 to S108.

S102: Obtain each to-be-processed image collected from different perspectives, convert each to-be-processed image into a unified coordinate space, and obtain a converted image corresponding to each to-be-processed image.

Among them, the image to be processed means the image to be spliced, and the images to be processed collected from different perspectives may specifically be images captured by cameras at different positions for the same object, and the images to be processed have a certain overlap.

After each image to be processed is obtained, since the image coordinate system of each image to be processed may be inconsistent, it is necessary to perform image registration on each image to be processed to convert each image to be processed into a unified coordinate space to facilitate subsequent processing. In an embodiment, one of the images to be processed may be selected as a reference image, and other images to be processed are used as images to be registered, and registered with the reference image, so that the images to be processed are placed in the same coordinate space.

Specifically, the image registration process can be as follows: extract feature points of each image to be processed, obtain feature points of each image to be processed, and match the feature points of each image to be registered with the feature points of the reference image, based on the feature The point matching information can obtain the coordinate transformation relationship between each image to be registered and the reference image, and calculate the homography transformation matrix used for the spatial registration transformation of each image to be registered. Based on the homography transformation matrix, the The quasi-image and the reference image are transformed into the same coordinate space.

It should be noted that in the above process, any possible method can be used for feature point extraction, such as the Scale Invariant Feature Transformation Algorithm (SIFT), or any possible method for feature point matching, such as Random Sampling Consensus Algorithm (RANSAC). ), no restrictions here.

S104: Determine the overlapping area of each converted image.

Specifically, after each image to be processed is converted into the same coordinate space, the coordinate position information of the image points in each converted image can be obtained, and the overlapping area can be determined according to the intersection of the coordinate position information of the image points in each converted image.

Taking two to-be-processed images as an example, please refer to FIG. 2, which shows two to-be-processed images collected from different perspectives in an embodiment. The to-be-processed image on the left is denoted by L, and the to-be-processed image on the right is denoted by R. The viewing ranges of the image L and the image R are different and have a certain overlapping area. Please refer to FIG. 3, which shows the overlapping area of the image L and the image R. Convert image L and image R to a unified coordinate space to obtain the corresponding converted image, denoted by L′ and R′ respectively. Specifically, using image L as a reference image, image L′ and image L are the same, and image R is projected In the coordinate space where the image L is located, the image R′ is obtained. Please refer to Figure 4, which shows the overlapping area of the image L′ and the image R′. The left and right images in Figure 4 represent the image R′ and the image L, respectively. ′The image in the overlapping area is represented by R1 and L1 respectively.

S106: Obtain spatial optical flow information based on the pixels of each converted image in the overlap area, where the spatial optical flow information is used to represent the degree of deviation of each pixel in the overlap area.

Optical flow information can be used to characterize the degree of pixel offset. For each image collected from different perspectives, due to distance and parallax, the pixels of foreground objects (including moving objects) in each image will be offset and offset The degree is greater than that of the background object, so the spatial optical flow information can be obtained based on the pixel offset in each image collected from different perspectives.

Taking the image of the overlapping area in Fig. 4 as an example, the optical flow information from image R1 to image L1 can be calculated by the optical flow algorithm to obtain spatial optical flow information. The shift degree of the position of the pixel in the image L1. It should be noted that in the above process, any possible optical flow algorithm can be used to calculate the optical flow information, such as Lucas-Kanade, FlowNet, etc., which are not limited here.

S108, according to the spatial optical flow information, stitch each converted image.

Since the pixel offsets of objects with different depths of field are inconsistent in different viewing angles, the pixel offset of foreground objects (including moving objects) is greater than that of background objects, that is, the greater the spatial optical flow information, the greater the degree of pixel offset corresponding to the foreground object. The greater the probability, the smaller the spatial optical flow information, and the greater the probability that it corresponds to the background object. Therefore, using the spatial optical flow information to stitch each converted image can make the stitching bypass the foreground target.

In the above-mentioned image stitching method, by acquiring each to-be-processed image collected from different perspectives, each to-be-processed image is converted to a unified coordinate space, and the converted image corresponding to each to-be-processed image is obtained; the overlap area of each converted image is determined; The pixels of the converted image in the overlap area obtain spatial optical flow information. The spatial optical flow information is used to characterize the offset degree of each pixel in the overlap area; according to the spatial optical flow information, each converted image is spliced. Since the pixel offsets of objects with different depths of field are inconsistent in different viewing angles, the spatial optical flow information can effectively distinguish the foreground image and the background image in the overlapping area image. When image stitching is performed according to the spatial optical flow information, stitching can be made The thread bypasses the foreground object or moving object, effectively avoiding the seam or ghosting problem that is likely to occur when the stitching thread passes through the foreground object or moving object, thereby improving the quality of image stitching.

In one embodiment, the step of obtaining spatial optical flow information based on the pixel points of each converted image in the overlapping area may specifically include the following steps: obtaining the first pixel of the first pixel point in the overlapping area of the first converted image The position information, and the second position information of the second pixel matching the first pixel in the overlap area of the second converted image, each converted image includes the first converted image and the second converted image; The location information and the second location information obtain the spatial optical flow information.

Take the overlapping area image in Figure 4 as an example, the pixels in the image R1 represent the pixels in the overlapping area of the first converted image, that is, the first pixel, and the first position information indicates that the first pixel is in the image R1 Location information. The pixels in the image L1 represent the pixels in the overlap area of the second converted image (for distinguishing purposes, here are represented by the fourth pixel), the fourth pixel is searched for the pixel that matches the first pixel, That is, the second pixel, and the second position information indicates the position information of the second pixel in the image L1. According to the deviation of the first position information relative to the second position information, the spatial optical flow information is obtained.

In this embodiment, the spatial optical flow information is obtained based on the position offset of the same pixel in the overlapping area images of different viewing angles. Based on the spatial optical flow information obtained in this way, the foreground part and the background part in the overlapping area image can be effectively distinguished. And the spatial optical flow information is not sensitive to brightness, that is, for the overlapping area images under various lighting conditions, the foreground and background parts can be accurately identified, thereby improving the image recognition effect under different lighting conditions.

In one embodiment, the method further includes the following steps: acquiring a first adjacent frame image of the first image to be processed, and a second adjacent frame image of the second image to be processed, each image to be processed includes the first image to be processed and The second image to be processed; the first adjacent frame image and the second adjacent frame image are converted into a unified coordinate space to obtain the first auxiliary image and the second auxiliary image.

Take two images to be processed as an example. The two images to be processed include the first image to be processed and the second image to be processed. Go to the unified coordinate space to obtain the second transformed image. Specifically, the first image to be processed and the second image to be processed may respectively correspond to the t-th frame image collected by the first camera and the second camera, and the first adjacent frame image and the second adjacent frame image may respectively correspond to the first camera With the t-1th frame image collected by the second camera, the first adjacent frame image is converted to the unified coordinate space to obtain the first auxiliary image, and the second adjacent frame image is converted to the unified coordinate space to obtain the second auxiliary image , So that the first converted image, the second converted image, the first auxiliary image, and the second auxiliary image are in the same coordinate space.

In this embodiment, by acquiring adjacent frame images of each image to be processed, the temporal change information of the pixels in each image to be processed can be acquired, providing a larger range of effective information for subsequent calculation of optical flow information, so that the calculated The optical flow information can more accurately reflect the actual offset of the pixels, thereby helping to improve the accuracy of image recognition.

In one embodiment, after obtaining the first auxiliary image and the second auxiliary image, the method may further include the following steps: obtaining first position information of the first pixel in the overlap area of the first converted image, and in the first auxiliary image The third position information of the third pixel matching the first pixel; the first time domain optical flow information is obtained according to the first position information and the third position information; the fourth pixel in the overlap area of the second converted image is obtained The fourth position information of the dot, and the fifth position information of the fifth pixel matching the fourth pixel in the second auxiliary image, each converted image includes the first converted image and the second converted image; according to the fourth The position information and the fifth position information are used to obtain the second time domain optical flow information.

Among them, the first converted image corresponds to the space-converted image of the t-th frame image collected by the first camera, the first auxiliary image corresponds to the space-converted image of the t-1th frame image collected by the first camera, and the second conversion The rear image corresponds to the space-converted image of the t-th frame captured by the second camera. The second auxiliary image corresponds to the space-converted image of the t-1th frame captured by the second camera. The overlap area represents the first converted image. And the overlapping area of the second converted image.

The pixel points in the overlap area of the first converted image, that is, the first pixel point, and the first position information represents the position information of the first pixel point in the first converted image. Find a pixel point that matches the first pixel point from the pixel points of the first auxiliary image, that is, the third pixel point, and the third position information represents the position information of the third pixel point in the first auxiliary image. According to the deviation of the first position information relative to the third position information, the first time-domain optical flow information is obtained.

The pixel points in the overlap area of the second converted image, that is, the fourth pixel point, and the fourth position information represents the position information of the fourth pixel point in the second converted image. Find a pixel point that matches the fourth pixel point from the pixel points of the second auxiliary image, that is, the fifth pixel point, and the fifth position information represents the position information of the fifth pixel point in the second auxiliary image. According to the deviation of the fourth position information relative to the fifth position information, the second time-domain optical flow information is obtained.

In this embodiment, the continuous information of the image in the time domain is fully utilized, and the optical flow information in the time domain is obtained through the position offset of the same pixel in different frames of the image, which is used to reflect the change of the pixel in the time dimension, and is useful for the spatial domain. The optical flow information is supplemented to more accurately reflect the actual offset of the pixel and help improve the accuracy of image recognition.

In one embodiment, the step of stitching each converted image according to the spatial optical flow information may specifically include the following steps: obtaining an energy map corresponding to the overlapping area according to the spatial optical flow information; The images are stitched.

The step of obtaining the energy map corresponding to the overlapping area according to the spatial optical flow information may specifically include: converting the spatial optical flow information to obtain the corresponding spatial energy value; and obtaining the energy map corresponding to the overlapping area based on the spatial energy value. Taking the image of the overlapping area in Figure 4 as an example, after calculating the spatial optical flow information from image R1 to image L1, the spatial optical flow information can be converted. Specifically, the spatial optical flow information can be normalized. After the information is normalized, the corresponding airspace energy value is obtained. The range of the airspace energy value can be 0~1. It should be understood that the greater the airspace optical flow information, the larger the corresponding airspace energy value, and the smaller the airspace optical flow information. The corresponding airspace energy value is smaller. Obtain the energy map corresponding to the overlapping area based on the spatial energy value. Please refer to Figure 5, which shows the energy map obtained according to the spatial optical flow information of the image R1 to the image L1. In this energy map, the closer the energy value is to 0, the more the color Closer to black, the corresponding position is more likely to be the background image; the closer the energy value is to 1, the closer the color is to white, and the more likely the corresponding position is the foreground image.

In this embodiment, the energy map corresponding to the overlapping area is obtained according to the spatial optical flow information. The energy map can accurately reflect the areas corresponding to the foreground image and the background image in the overlapping area, so that the foreground part and the background image in the overlapping area image can be distinguished based on the energy map. In the background part, when image stitching is performed based on the energy map, the stitching line can only pass through the background part, effectively avoiding the seam or ghosting problem that is likely to occur when the stitching line passes through foreground objects or moving objects, thereby improving the quality of image stitching.

In one embodiment, the step of obtaining the energy map corresponding to the overlapping area according to the spatial optical flow information may specifically include the following steps: separate the spatial optical flow information, the first time domain optical flow information, and the second time domain optical flow information. Perform conversion to obtain the corresponding spatial energy value, first time domain energy value and second time domain energy value; based on the spatial energy value, first time domain energy value and second time domain energy value, obtain the energy map corresponding to the overlapping area .

Specifically, the spatial optical flow information, the first time domain optical flow information, and the second time domain optical flow information can be respectively normalized to obtain the corresponding spatial energy value, first time domain energy value, and second time domain energy The range of the spatial energy value, the first time domain energy value, and the second time domain energy value can all be 0 to 1. After obtaining the spatial energy value, the first time domain energy value, and the second time domain energy value, the first time domain energy value and the second time domain energy value can be correspondingly added and averaged to obtain the time domain energy value, Then the time domain energy value and the space domain energy value are correspondingly added and averaged to obtain the final energy value, and based on the final energy value, the energy map corresponding to the overlapping area is obtained.

In this embodiment, the energy map corresponding to the overlapping area is obtained by fusing the spatial optical flow information and the time domain optical flow information, which contains a larger range of effective information. Based on the energy map, the foreground part and the image in the overlapping area can be more accurately distinguished. The background part helps to further improve the quality of subsequent image stitching.

In an embodiment, determining the foreground image and background image of each converted image in the overlap area according to the energy map may be as follows: according to the position of the image point corresponding to the energy value greater than the preset threshold in each energy value in the energy map , Determine the foreground image of each converted image in the overlap area; determine the background image of each converted image in the overlap area according to the position of the image point corresponding to the energy value less than or equal to the preset threshold in each energy value in the energy map.

Specifically, the energy value can range from 0 to 1, and the preset threshold can be set according to actual conditions. For example, it can be set to 0.5, and the image point position corresponding to the energy value greater than 0.5 is considered to be the corresponding image point in the overlap area image. Belonging to the foreground image, the image point corresponding to the energy value less than or equal to 0.5 in the overlapping area image corresponds to the background image.

In this embodiment, the positions of the image points corresponding to the foreground image and the background image in the energy map are distinguished by a suitable energy value threshold, so as to facilitate the distinction and extraction of the foreground image and the background image.

In one embodiment, the step of stitching each converted image according to the energy map may specifically include the following steps: determining the shortest path in the stitching direction according to each energy value in the energy map as the best stitching line; The best stitching line is to stitch each converted image to obtain the stitched image.

Take the energy map in Figure 5 as an example, the splicing direction is up and down, and the shortest path means the shortest path from the top of the energy map to the bottom row. The shortest path can be understood as the path with the smallest sum of energy values. The ground can find the shortest path of the energy map through the dynamic programming algorithm. The shortest path of the energy map shown in Figure 5 is shown by the line on the right in Figure 5.

In this embodiment, the best stitching line is determined according to the shortest path of the energy map in the splicing direction. The shortest path represents the path with the smallest sum of energy values in the splicing direction. The best stitching line determined by the shortest path can bypass foreground objects or moving objects, which can effectively avoid seam or ghosting problems that are prone to occur when stitching passes through foreground objects or moving objects during stitching, thereby improving the quality of image stitching.

In one embodiment, the step of stitching each converted image based on the best stitching line to obtain the stitched image may specifically include the following steps: cropping each converted image based on the best stitching line, and cutting the cropped image The images are stitched together.

Specifically, based on the optimal stitching line, the overlapping area image of each converted image can be divided into two parts, the first part of the image is close to its own non-overlapping area image, and the second part of the image is far away from its own non-overlapping area image, based on the best stitching line Crop each converted image, specifically by cropping the second part of the image in the overlap area of each converted image, and each cropped image represents the remaining image after each converted image is cropped off the second part of the image. . Based on the best stitching line, the mask map used to process each converted image can also be obtained. According to the mask map corresponding to each converted image, each converted image is cropped to obtain the cropped image, and the cropped images are merged , To obtain the stitched image.

Take the optimal stitching line in Fig. 5 as an example. According to the optimal stitching line, the mask map for processing the image L′ and the image R′ can be obtained. The mask map of image R'. The left and right images in Figure 6 respectively represent the mask maps used to process the image L'and the image R', denoted by mask_L and mask_R, respectively. The white area in the mask map mask_L and The black areas correspond to the area to be retained and the area to be removed in the image L′. The gradient area in the mask map mask_L plays a role of feathering, making the transition at the splicing more natural. Please refer to Figure 7, which shows the image L′ The image processed by mask_L is represented by L2. The white area and black area in the mask image mask_R correspond to the areas to be retained and to be removed in the image R′, respectively. The gradient area in the mask image mask_R plays a role of feathering, making the transition at the splicing more natural, see Figure 8. , Shows the image after the image R'is processed by the mask image mask_R, denoted by R2. The image L2 and the image R2 are spliced to obtain a spliced image. Please refer to FIG. 9, which shows the spliced image of the image L2 and the image R2.

In this embodiment, the converted images are stitched based on the best stitching line, which can bypass foreground objects or moving objects, and can effectively avoid the seam or ghosting problem that is likely to occur when the stitching line passes through foreground objects or moving objects during stitching. , Thereby improving the quality of image stitching.

It should be understood that the above embodiment takes the stitching of two images as an example for description, but it is not limited to the stitching of two images, and can also be applied to the stitching of more than two images. The stitching process is similar to the above embodiment. I won't repeat it here.

In one embodiment, as shown in FIG. 10, an image stitching method is provided, and the method includes the following steps S1001 to S1008.

S1001: Obtain each to-be-processed image and adjacent frame images of each to-be-processed image collected from different perspectives, convert each to-be-processed image and each adjacent frame image to a unified coordinate space, and obtain a converted image corresponding to each to-be-processed image and The auxiliary image corresponding to each adjacent frame image.

S1002: Determine the overlap area of each converted image.

S1003: Obtain spatial optical flow information based on the pixel points of each converted image in the overlapping area.

S1004: Obtain time-domain optical flow information based on the pixel points of each converted image in the overlap area and the pixel points of each auxiliary image.

S1005: Convert the spatial optical flow information and the time domain optical flow information to obtain corresponding spatial energy values and time domain energy values.

S1006: Obtain an energy map corresponding to the overlapping area based on the energy value in the space domain and the energy value in the time domain.

S1007: Determine the shortest path in the splicing direction according to each energy value in the energy map as the best suture line.

S1008, based on the best stitching line, stitch each converted image to obtain the stitched image.

For the specific limitations of the above steps S1001 to S1008, reference may be made to the foregoing embodiment, and details are not described herein again. In this embodiment, by fusing spatial optical flow information and temporal optical flow information, it is possible to effectively distinguish foreground objects (including moving objects) and background objects in the overlapping area of each image to be spliced, so as to obtain the ability to bypass foreground objects or motion The best stitching line of the object, to avoid seam or ghosting problems during stitching, and to improve the quality of image stitching.

It should be understood that the method of fusing spatial optical flow information and temporal optical flow information for image splicing can also be applied to video stream splicing. For example, for the first video stream to be spliced and the second video stream to be spliced, a single frame image (for example, the first frame) is extracted from the first video stream to be spliced, and used as the first image to be spliced and the second image to be spliced. Then, the spliced first frame image can be obtained, and in the same way, the spliced nth frame image can be obtained, and based on the spliced frame images, the spliced video stream can be obtained.

It should be understood that although the various steps in the flowcharts of FIGS. 1 and 10 are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless specifically stated in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least part of the steps in Figures 1 and 10 may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution of these steps or stages The sequence is not necessarily performed sequentially, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.

In one embodiment, as shown in FIG. 11, an image splicing device is provided, including: an image acquisition module 1110, an overlapping area determination module 1120, an optical flow information determination module 1130, and an image splicing module 1140, in which:

The image acquisition module 1110 is used to acquire each to-be-processed image collected from different perspectives, convert each to-be-processed image to a unified coordinate space, and obtain a converted image corresponding to each to-be-processed image.

The overlapping area determining module 1120 is used to determine the overlapping area of each converted image.

The optical flow information determining module 1130 is configured to obtain spatial optical flow information based on the pixels of each converted image in the overlapping area, and the spatial optical flow information is used to represent the degree of deviation of each pixel in the overlapping area.

The image stitching module 1140 is used to stitch each converted image according to the spatial optical flow information.

In one embodiment, when the optical flow information determining module 1130 obtains spatial optical flow information based on the pixel points of each converted image in the overlap area, it is specifically configured to: obtain the first pixel of the first converted image in the overlap area The first position information of the dot, and the second position information of the second pixel that matches the first pixel in the overlap area of the second converted image, each converted image includes the first converted image and the second converted image ; Obtain spatial optical flow information according to the first position information and the second position information.

In an embodiment, the image acquisition module 1110 is further configured to: acquire a first adjacent frame image of the first image to be processed, and a second adjacent frame image of the second image to be processed, each image to be processed includes the first image The image to be processed and the second image to be processed; the first adjacent frame image and the second adjacent frame image are converted into a unified coordinate space to obtain the first auxiliary image and the second auxiliary image.

In an embodiment, the optical flow information determining module 1130 is further configured to: obtain the first position information of the first pixel in the overlap area of the first converted image, and the first pixel in the first auxiliary image that matches the first pixel. The third position information of the third pixel; the first time-domain optical flow information is obtained according to the first position information and the third position information; the fourth position information of the fourth pixel in the overlap area of the second converted image is obtained, And the fifth position information of the fifth pixel point matching the fourth pixel point in the second auxiliary image, each converted image includes the first converted image and the second converted image; according to the fourth position information and the fifth position information , To obtain the second time domain optical flow information.

In one embodiment, when the image stitching module 1140 stitches each converted image according to the spatial optical flow information, it is specifically used to: obtain the energy map corresponding to the overlapping area according to the spatial optical flow information; After the conversion, the images are stitched together.

In one embodiment, when the image stitching module 1140 obtains the energy map corresponding to the overlapping area according to the spatial optical flow information, it is specifically used to: convert the spatial optical flow information to obtain the corresponding spatial energy value; based on the spatial energy value, Obtain the energy map corresponding to the overlapping area.

In one embodiment, when the image stitching module 1140 obtains the energy map corresponding to the overlapping area according to the spatial optical flow information, it is specifically used to combine the spatial optical flow information, the first time domain optical flow information, and the second time domain optical flow information. The information is converted separately to obtain the corresponding spatial energy value, the first time domain energy value and the second time domain energy value; based on the spatial energy value, the first time domain energy value and the second time domain energy value, the corresponding to the overlapping area is obtained Energy graph.

In one embodiment, when the image stitching module 1140 obtains the energy map corresponding to the overlapping area based on the spatial energy value, the first time domain energy value, and the second time domain energy value, it is specifically used to: Correspondingly add to the second time-domain energy value and take the average value to obtain the time-domain energy value; add the time-domain energy value and the space-domain energy value correspondingly and take the average value to obtain the final energy value; obtain the final energy value based on the final energy value The energy map corresponding to the overlapping area.

In one embodiment, the image stitching module 1140 is further configured to: determine the foreground image of each converted image in the overlap area according to the position of the image point corresponding to the energy value greater than the preset threshold in each energy value in the energy map.

In an embodiment, the image stitching module 1140 is further configured to: determine the background image of each converted image in the overlap area according to the position of the image point corresponding to the energy value less than or equal to the preset threshold in each energy value in the energy map.

In one embodiment, when the image stitching module 1140 stitches the converted images according to the energy map, it is specifically used to: determine the shortest path in the stitching direction according to the energy values in the energy map as the best stitching line ; Based on the best stitching line, stitch the converted images.

In one embodiment, when the image stitching module 1140 stitches each converted image based on the best stitching line, it is specifically used to: crop each converted image based on the best stitching line, and perform the cropped image Splicing.

For the specific definition of the image splicing device, please refer to the above definition of the image splicing method, which will not be repeated here. Each module in the above-mentioned image splicing device can be implemented in whole or in part by software, hardware, and a combination thereof. The foregoing modules may be embedded in the form of hardware or independent of the processor in the computer device, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the foregoing modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. 12. The computer equipment includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to realize an image stitching method.

In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. 13. The computer equipment includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator's network, NFC (near field communication) or other technologies. The computer program is executed by the processor to realize an image stitching method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, or it can be a button, trackball or touchpad set on the housing of the computer equipment , It can also be an external keyboard, touchpad, or mouse.

Those skilled in the art can understand that the structure shown in FIG. 12 or FIG. 13 is only a block diagram of a part of the structure related to the solution of the present disclosure, and does not constitute a limitation on the computer device to which the solution of the present disclosure is applied. The computer device may include more or fewer components than shown in the figures, or combine certain components, or have a different component arrangement.

In one embodiment, a computer device is provided, including a memory and a processor, and a computer program is stored in the memory, and the processor implements the steps in the foregoing method embodiments when the computer program is executed.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement the steps in the foregoing method embodiments.

It should be understood that the terms "first", "second", etc. in the foregoing embodiments are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage. In the medium, when the computer program is executed, it may include the procedures of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other media used in the various embodiments provided in the present disclosure may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM may be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as the combinations of these technical features are not contradictory, they should be It is considered as the range described in this specification.

The above-mentioned embodiments only express several implementation manners of the present disclosure, and their descriptions are more specific and detailed, but they should not be understood as limiting the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present disclosure, several modifications and improvements can be made, and these all fall within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the appended claims.

Claims (13)

An image stitching method, the method includes: Acquiring each to-be-processed image collected from different perspectives, transforming each of the to-be-processed images into a unified coordinate space, and obtaining a converted image corresponding to each of the to-be-processed images; Determining the overlapping area of each of the converted images; Obtain spatial optical flow information based on the pixels of each of the converted images in the overlapping area, where the spatial optical flow information is used to characterize the degree of deviation of each pixel in the overlapping area; According to the spatial optical flow information, stitching each of the converted images. The method according to claim 1, wherein the obtaining spatial optical flow information based on the pixel points of each of the converted images in the overlapping area comprises: Acquire the first position information of the first pixel of the first converted image in the overlap area, and the first position of the second pixel of the second converted image that matches the first pixel in the overlap area 2. Position information, each of the converted images includes the first converted image and the second converted image; Obtain spatial optical flow information according to the first position information and the second position information. The method according to claim 1, further comprising: Acquire a first adjacent frame image of a first image to be processed, and a second adjacent frame image of a second image to be processed, each of the images to be processed includes the first image to be processed and the second image to be processed ； The first adjacent frame image and the second adjacent frame image are converted into the unified coordinate space to obtain a first auxiliary image and a second auxiliary image. The method according to any one of claims 1-3, further comprising: Acquiring first position information of a first pixel in the overlap area of the first converted image, and third position information of a third pixel in the first auxiliary image that matches the first pixel; Obtaining first time-domain optical flow information according to the first position information and the third position information; Obtain the fourth position information of the fourth pixel of the second converted image in the overlap area, and the fifth position information of the fifth pixel in the second auxiliary image that matches the fourth pixel , Each of the converted images includes the first converted image and the second converted image; Obtain second temporal optical flow information according to the fourth position information and the fifth position information. The method according to claim 4, wherein, according to the spatial optical flow information, stitching each of the converted images comprises: Obtaining an energy map corresponding to the overlapping area according to the spatial optical flow information; According to the energy map, stitching each of the converted images. The method according to claim 5, wherein obtaining the energy map corresponding to the overlapping area according to the spatial optical flow information includes any one of the following two items: the first item: Converting the airspace optical flow information to obtain the corresponding airspace energy value; Obtaining an energy map corresponding to the overlapping area based on the airspace energy value; second section: The spatial optical flow information, the first time domain optical flow information, and the second time domain optical flow information are respectively converted to obtain the corresponding spatial energy value, first time domain energy value, and second time domain energy value; Based on the spatial energy value, the first time domain energy value, and the second time domain energy value, an energy map corresponding to the overlapping area is obtained. The method according to claim 6, wherein, based on the spatial energy value, the first time domain energy value, and the second time domain energy value, obtaining an energy map corresponding to the overlapping area comprises: Correspondingly adding the first time domain energy value and the second time domain energy value and taking an average value to obtain a time domain energy value; Correspondingly adding the time domain energy value and the space domain energy value and taking an average value to obtain the final energy value; An energy map corresponding to the overlapping area is obtained based on the final energy value. The method according to claim 6 or 7, further comprising at least one of the following two items: Determine the foreground image of each of the converted images in the overlap area according to the position of the image point corresponding to the energy value greater than the preset threshold in each energy value in the energy map; Determine the background image of each of the converted images in the overlap area according to the position of the image point corresponding to the energy value that is less than or equal to the preset threshold in each energy value in the energy map. The method according to claim 6 or 7, wherein, according to the energy map, stitching each of the converted images includes: According to the energy values in the energy diagram, determine the shortest path in the splicing direction as the best suture line; Based on the optimal stitching line, stitching each of the converted images. The method according to claim 9, wherein, based on the optimal stitching line, stitching each of the converted images comprises: Based on the optimal stitching line, each of the converted images is cropped, and the cropped images are stitched together. An image splicing device, wherein the device includes: The image acquisition module is configured to acquire each to-be-processed image collected from different perspectives, convert each of the to-be-processed images to a unified coordinate space, and obtain a converted image corresponding to each of the to-be-processed images; An overlapping area determining module, configured to determine the overlapping area of each of the converted images; The optical flow information determining module is configured to obtain spatial optical flow information based on the pixel points of each of the converted images in the overlapping area, and the spatial optical flow information is used to characterize the offset of each pixel in the overlapping area degree; The image stitching module is used to stitch each of the converted images according to the spatial optical flow information. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 10 when the computer program is executed by the processor. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method according to any one of claims 1 to 10 when the computer program is executed by a processor.

Images