WO2022011560A1 - Image cropping method and apparatus, electronic device, and storage medium - Google Patents

Abstract

An image cropping method and apparatus, an electronic device, and a storage medium, which relate to the technical field of image processing. The method comprises: acquiring a color image and a depth image of a target object (S210); performing perspective transformation on the color image to obtain a perspective transformation image, and extracting image boundary information of the target object in the perspective transformation image (S220); determining point cloud boundary information of the target object according to point cloud data generated on the basis of the depth image (S230); obtaining target boundary information on the basis of the image boundary information and the point cloud boundary information (S240); and cropping the color image according to the target boundary information to obtain a cropped image (S250). The method can improve accuracy of image cropping.

Description

Image cropping method and device, electronic device and storage medium technical field

The present disclosure relates to the technical field of image processing, and in particular, to an image cropping method, an image cropping device, an electronic device, and a computer-readable storage medium.

Background technique

With the development of terminal equipment, the image processing capability of the terminal equipment is gradually enhanced. Among them, when cropping an image, a combination of traditional methods and deep learning methods can be used for automatic cropping. However, when the image information is missing or there is interference information, the accuracy of image cropping is low.

SUMMARY OF THE INVENTION

The purpose of the present disclosure is to provide an image cropping method, an image cropping device, an electronic device and a computer-readable storage medium, so as to overcome the problem of low image cropping accuracy due to limitations and defects of the related art to a certain extent .

According to a first aspect of the present disclosure, there is provided an image cropping method, comprising:

Obtain the color image and depth image of the target object;

Perform perspective transformation on the color image to obtain a perspective transformed image, and extract the image boundary information of the target object in the perspective transformed image;

Determine the point cloud boundary information of the target object according to the point cloud data generated based on the depth image;

Based on the image boundary information and the point cloud boundary information, obtain target boundary information;

The color image is cropped according to the target boundary information to obtain a cropped image.

According to a second aspect of the present disclosure, there is provided an image cropping device, comprising:

The image acquisition module is used to acquire the color image and depth image of the target object;

an image boundary information determination module, configured to perform perspective transformation on the color image to obtain a perspective transformed image, and extract the image boundary information of the target object in the perspective transformed image;

a point cloud boundary information determination module, configured to determine the point cloud boundary information of the target object according to the point cloud data generated based on the depth image;

a target boundary information determination module, configured to obtain target boundary information based on the image boundary information and the point cloud boundary information;

An image cropping module, configured to crop the color image according to the target boundary information to obtain a cropped image.

According to a third aspect of the present disclosure, there is provided an electronic device, comprising:

processor; and

a memory configured to store executable instructions for the processor;

Wherein, the processor is configured to perform the above-mentioned image cropping method by executing the executable instructions.

According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned image cropping method when executed by a processor.

Exemplary embodiments of the present disclosure may have at least some or all of the following beneficial effects:

In the image cropping method provided by an exemplary embodiment of the present disclosure, the point cloud boundary information of the target object is determined through the depth image, so that the cropping range of the target object can be obtained. And combined with the image boundary information of the target object in the color image, the target boundary information of the target object is determined. In this way, even if the target object has no obvious lines or interference information, the target boundary information can be accurately obtained. Further, cropping the color image according to the target boundary information can improve the accuracy of cropping, thereby improving user experience.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure;

FIG. 2 shows a flowchart of an image cropping method in an embodiment of the present disclosure;

FIG. 3 shows a flow chart of perspective transformation in an embodiment of the present disclosure;

Fig. 4 shows a kind of schematic diagram of a kind of area detection;

FIG. 5 shows a flow chart of generating point cloud boundary information in an embodiment of the present disclosure;

6 shows a schematic diagram of a target object and a point cloud boundary in an embodiment of the present disclosure;

FIG. 7 shows a schematic diagram of performing line segment detection on an image in an embodiment of the present disclosure;

FIG. 8 shows a schematic diagram of determining a target boundary in an embodiment of the present disclosure;

FIG. 9 shows a schematic diagram of displaying a cropped image in a view mode and an edit mode in an embodiment of the present disclosure;

FIG. 10 shows a schematic diagram of displaying a cropped image in a perspective transformation mode in an embodiment of the present disclosure;

FIG. 11 shows a schematic structural diagram of an image cropping apparatus in an embodiment of the present disclosure.

detailed description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

It should be noted that, in the present disclosure, the terms "comprising", "configured with", and "disposed on" are used to indicate an open-ended inclusive meaning, and refer to elements/components/etc. other than the listed elements/components/etc. Additional elements/components/etc. may also be present; the terms "first," "second," etc. are used merely as labels and not as limitations on the number or order of the objects.

Referring to FIG. 1 , FIG. 1 shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure. It should be noted that the electronic device 100 shown in FIG. 1 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG. 1 , the electronic device 100 may specifically include: a processor 110, a wireless communication module 120, a mobile communication module 130, a charging management module 140, a power management module 141, a battery 142, a USB (Universal Serial Bus, Universal Serial Bus) ) interface 150, antenna 1, antenna 2, internal memory 161, external memory interface 162, display screen 170, sensor module 180, camera module 190, etc.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a video codec, a digital signal processor , baseband processor and/or neural network processor, etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors. The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 for storing instructions and data. The memory may store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 110 . In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 110 . If the processor 110 needs to use the instruction or data again, it can be directly called from the memory, which avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

The display screen 170 is used to display images, videos, and the like.

The sensor module 180 may include a depth sensor, a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a fingerprint sensor, a temperature sensor, a touch sensor, and the like. Among them, the depth sensor is used to obtain the depth information of the scene. In some embodiments, the depth sensor may be disposed in the camera module 190 .

The camera module 190 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the image signal processor to convert it into a digital image signal. The image signal processor outputs the digital image signal to the digital signal processor for processing. The digital signal processor converts the digital image signal into a standard RGB, YUV and other format image signal. In some embodiments, the electronic device 100 may include one or more camera modules 190 .

The technical solutions of the embodiments of the present disclosure are described in detail below.

In the image cropping technology, the image can be cropped based on the boundary information of the object. However, the lack of available boundaries in the image as well as the presence of interfering boundaries results in lower accuracy of image cropping.

In order to solve the above problems, the present disclosure provides an image cropping method and device, an electronic device and a computer-readable storage medium, which can improve the accuracy of image cropping.

Referring to FIG. 2, FIG. 2 shows a flowchart of an image cropping method in an embodiment of the present disclosure, which may include the following steps:

Step S210, acquiring a color image and a depth image of the target object.

Step S220: Perform perspective transformation on the color image to obtain a perspective transformed image, and extract image boundary information of the target object in the perspective transformed image.

Step S230: Determine point cloud boundary information of the target object according to the point cloud data generated based on the depth image.

Step S240, based on the image boundary information and the point cloud boundary information, obtain target boundary information.

Step S250, crop the color image according to the target boundary information to obtain a cropped image.

In the image cropping method of the embodiment of the present disclosure, the point cloud boundary information of the target object is determined by using the depth image, so that the cropping range of the target object can be obtained. And combined with the image boundary information of the target object in the color image, the target boundary information of the target object is determined. In this way, even if the target object has no obvious lines or interference information, the target boundary information can be accurately obtained. Further, cropping the color image according to the target boundary information can improve the accuracy of cropping, thereby improving user experience.

The image cropping method according to the embodiment of the present disclosure will be introduced in more detail below.

In step S210, a color image and a depth image of the target object are acquired.

In this embodiment of the present disclosure, the target object may be any object to be photographed by the user, and may be a person, an animal, or a scene. The electronic device may contain multiple different camera modules, so that different images can be captured for the same target object. For example, one of the camera modules can capture color images (eg, RGB images or YUV images, etc.). Another camera module can capture the depth information of the target object to obtain a depth image. For example, the depth information of the target object can be obtained through a TOF (Time of Flight) sensor. Specifically, the TOF sensor emits modulated near-infrared light, and the near-infrared light is reflected after encountering an object. Get depth information.

TOF sensors are not affected by illumination changes and object textures, and can reduce costs on the premise of meeting the accuracy requirements. With the help of the three-dimensional data of the TOF sensor to assist in obtaining the three-dimensional information of the target object and calculating the perspective transformation matrix, the document scanning (referring to the correction of the perspective relationship) can be independent of the picture information, which greatly improves the application scope of the document scanning. Of course, the depth sensor may also be a structured light sensor or a binocular sensor, etc., which is not limited in the present disclosure.

In step S220, perspective transformation is performed on the color image to obtain a perspective transformed image, and image boundary information of the target object in the perspective transformed image is extracted.

It should be noted that perspective transformation is the process of projecting an image onto a new viewing plane. For example, an object that is a straight line in reality may appear as an oblique line on the image, and the oblique line can be converted into a straight line through perspective transformation.

Referring to FIG. 3, FIG. 3 shows a flowchart of perspective transformation in an embodiment of the present disclosure, which may include the following steps:

Step S310, performing plane detection on the point cloud data generated based on the depth image, and determining a perspective transformation matrix according to the detected plane.

Among them, point cloud data can be generated after the depth image is transformed by coordinates, that is, the three-dimensional coordinates in the depth image are converted into three-dimensional coordinates in the camera coordinate system to generate three-dimensional point cloud data. After that, plane detection can be performed on the 3D point cloud data. For example, RANSAC (Random Sample Consensus) and other methods can be used to perform plane detection on the 3D point cloud data to obtain 3D plane parameters. The 3D plane parameters can be used to represent the detected flat. Afterwards, the perspective transformation matrix can be calculated according to the three-dimensional plane parameters. For example, the perspective transformation matrix can be calculated by the four-point method or the like.

Step S320: Perform perspective transformation on the color image through a perspective transformation matrix to obtain a perspective transformed image.

After the perspective transformation matrix is obtained, for each pixel in the color image, the position coordinates of the pixel are multiplied by the perspective transformation matrix to obtain the position coordinates after perspective transformation. The above process is performed for each pixel to obtain a perspective transformed image.

In the embodiment of the present disclosure, the image boundary information of the target object can be more accurately extracted from the perspective transformed image. In an implementation manner of the present disclosure, line segment detection may be performed on the perspective transformed image, for example, line segment detection may be performed on the perspective transformed image by means of Hough transform or the like. Among them, the Hough transform can not only identify the straight lines in the image, but also any other shapes, such as circles, ellipses, etc. When the line segment information is included in the perspective transformed image, the image boundary information of the target object can be determined according to the line segment information.

Specifically, the present disclosure can also perform validity judgment and classification according to information such as the length and angle of the line segment. For example, shorter segments can be classified as invalid segments, longer segments as valid segments, and so on. The validity judgment can delete the invalid line segment information and keep the valid line segment information. Classification refers to dividing line segments into different categories, for example, line segments can be divided into horizontal line segments, vertical line segments, and so on. The image boundary information of the final extracted target object may contain multiple line segment information.

In another case, if the perspective transformed image does not contain line segment information, the image boundary information of the target object cannot be obtained by the above-mentioned line segment detection method. At this time, the perspective transformation image can be processed based on the region detection algorithm to obtain the main body region image. Specifically, for a perspective transformed image, the saliency distribution can be calculated first, and then the range box can be calculated. The region detection algorithm may adopt an attention region detection algorithm, a main region detection algorithm, or a method based on saliency region detection, etc., wherein the method based on salient region detection includes: DeepGaze, DenseGaze, FastGaze, and the like.

Referring to FIG. 4, FIG. 4 shows a schematic diagram of a region detection. It can be seen that through region detection, a subject region image can be obtained, and the subject region image includes a range frame, and the range frame contains a target object. After that, the image boundary information of the target object can be determined according to the subject area image. It can be understood that the range box in Fig. 4 can be the image boundary of the target object.

It should be noted that, when extracting the image boundary information of the target object in the perspective transformed image, the region detection algorithm can be used after line segment detection and the line segment information is not detected, or the region detection algorithm can be directly used. Not limited. Moreover, when using the region detection algorithm, the perspective transformed image can be processed separately, or the perspective transformed image and the depth image can be processed separately, and fused after processing, which can improve the accuracy of image boundary information determination.

In step S230, point cloud boundary information of the target object is determined according to the point cloud data generated based on the depth image.

In the embodiment of the present disclosure, in addition to determining the image boundary information based on the color image, the point cloud boundary information can also be determined based on the depth image, and the boundary information of the target object can be determined from more levels, so as to improve the accuracy of the final determined boundary information. .

Specifically, for the process of generating point cloud data according to the depth image, reference may be made to the description in step S310. For the process of determining the point cloud boundary information of the target object according to the point cloud data, reference may be made to FIG. 5 . FIG. 5 shows a flowchart of generating point cloud boundary information in an embodiment of the present disclosure, which may include the following steps:

In step S510, after plane detection is performed on the point cloud data, plane point cloud data is obtained.

As mentioned above, after plane detection is performed on the point cloud data, the 3D plane parameters can be obtained. According to the three-dimensional plane parameters, the point cloud belonging to the plane can be extracted to obtain plane point cloud data. It should be noted that the point cloud data belonging to the plane may be data on the plane, or point cloud data on the plane, and point cloud data whose distance from the plane is less than the distance threshold. The distance threshold can be set according to the actual situation. The smaller the distance threshold, the higher the accuracy of image cropping.

Step S520, determining the bounding box of the plane point cloud data.

Specifically, assuming that the detected plane is the target plane, a rectangular coordinate system can be established with the normal of the target plane and the two tangential directions of the target plane as the orthogonal basis, and the bounding box of the plane point cloud data can be calculated in this coordinate system. Bounding box is an algorithm for solving the optimal bounding space of a discrete point set, which can replace complex geometric objects approximately with geometric objects with slightly larger volume and simple characteristics (called bounding boxes). Bounding boxes include: AABB bounding box, bounding sphere, directional bounding box OBB, and fixed-direction convex hull, etc.

The AABB bounding box is the smallest hexahedron that contains the target object and whose sides are parallel to the coordinate axes. When the AABB bounding box is used in the present disclosure, the process of calculating the bounding box is to calculate the maximum and minimum values along each axis of the coordinate system.

Step S530, mapping the bounding box to the perspective transformation image to obtain point cloud boundary information of the target object.

In the embodiment of the present disclosure, after the bounding box is obtained, the point cloud boundary information can be obtained by directly projecting the three-dimensional range (maximum value and minimum value) of the bounding box to the perspective transformed image.

It should be noted that, after the plane point cloud data is obtained, before step S520 is performed, the plane point cloud data can be filtered. Through the filtering process, the noise point cloud data can be filtered out, so that the target plane point cloud data can be obtained. . In an implementation manner of the present disclosure, distance filtering processing may be performed on the plane point cloud data, for example, by counting the density of the point cloud, and filtering out the point cloud with a lower density as noise. That is, the noise point cloud that is misjudged as belonging to the target plane in the plane point cloud data is filtered out by distance filtering.

In another implementation manner of the present disclosure, the plane point cloud data may also be subjected to clustering filtering processing, for example, by clustering the plane point cloud data, to extract the class with the largest number of plane point cloud data. In this way, non-subject point clouds that belong to other objects but are computed because they intersect the target plane can be filtered out. Among them, the clustering algorithm can be K-means clustering algorithm, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm, etc. Of course, distance filtering processing and clustering filtering processing can also be performed on the plane point cloud data, which can further improve the accuracy of the target plane point cloud data compared with performing distance filtering processing or performing cluster filtering processing alone.

Correspondingly, after the filtering process, the bounding box of the point cloud data of the target plane can be determined, so that the accuracy of the bounding box can be improved, and then the accuracy of the boundary information of the point cloud can be improved.

In step S240, target boundary information is obtained based on the image boundary information and the point cloud boundary information.

In this embodiment of the present disclosure, the image boundary information is information used to describe the image boundary, and different image boundary information corresponds to different image boundaries. The point cloud boundary information is the information used to describe the point cloud boundary, and different point cloud boundary information corresponds to different point cloud boundaries. By fusing the image boundary information calculated based on the color image with the point cloud boundary information calculated based on the point cloud data, more accurate target boundary information can be obtained.

Among them, the fusion method can be to take the point cloud boundary as the initial value, and select the boundary closest to the outside of the point cloud boundary within a certain range from the image boundary; it can also be to use the point cloud boundary as the initial value, and select a certain range from the image boundary. The boundary closest to the inner side of the point cloud boundary.

Referring to FIG. 6, FIG. 6 shows a schematic diagram of a target object and a point cloud boundary in an embodiment of the present disclosure. Due to the influence of sensor performance, sensor accuracy, reflection characteristics of objects, etc., there are a lot of noise point clouds in the point cloud data, so the detection algorithm based on the bounding box cannot give a very accurate object range. That is, affected by the noise point cloud, the boundary of the target object cannot be accurately determined according to the point cloud data. It can be seen that there is a large gap between the point cloud boundary and the actual boundary of the target object.

According to the method of the embodiment of the present disclosure, for example, through line segment detection, the detection result shown in FIG. 7 can be obtained, that is, multiple straight lines can be detected. Since the detected line is on the inner side of the point cloud boundary, when the detected line and the point cloud boundary are fused, the point cloud boundary can be used as the initial value, and the most inner side of the point cloud boundary can be selected from multiple straight lines. The boundary of the target after fusion can be seen in Figure 8.

In addition, the point cloud boundary can be used as the initial value, and the line segment with the closest distance to the point cloud boundary and the included angle smaller than the angle threshold, that is, the target line segment, can be selected from the image boundary, and the line segment information corresponding to the target line segment can be used as the target boundary information. The angle threshold can be set according to the actual situation, for example, it can be 10°, 15° and so on. Since an image boundary can include multiple directions, for example, for a rectangular boundary, four different directions are included. Therefore, the line segment with the closest distance to the point cloud boundary and whose included angle is smaller than the angle threshold refers to the line segment with the closest distance to the point cloud boundary in each direction and whose included angle is smaller than the angle threshold. Correspondingly, the line segments finally selected are also line segments in multiple directions, and the line segments in multiple directions constitute the target boundary.

In an implementation manner of the present disclosure, after obtaining the target boundary information, the color image may not be cropped, but the target boundary information may be displayed in the color image for the user's reference, so that the user can manually adjust the color image based on the target boundary information. Image is cropped. That is, the target boundary information is displayed in the color image to assist the user in manual cropping. The user may directly crop according to the target boundary information, or of course, it may not be cropped according to the target boundary information.

In step S250, the color image is cropped according to the target boundary information to obtain a cropped image.

It should be noted that, after obtaining the target boundary information, it can also be directly and automatically cropped to obtain a cropped image. After the cropped image is obtained, the cropped image can also be displayed to the user. As shown in FIG. 9 , when displaying the cropped image, it can also be displayed in view mode, the user can click “manual crop” to enter the editing mode, and the user can further manually crop the image.

In edit mode, users can adjust the crop box, rotate the image, translate the image, stretch the image, apply a perspective transformation to the image, and more. The user can drag the four corners of the cropping frame to adjust the size of the cropping frame, or adjust the size of the cropping frame by dragging the borders of the cropping frame. The user can rotate the image by dragging. When rotating, the rotation angle can be displayed to assist the user to rotate. The image can also be panned by dragging, and the image can be stretched by touching with two fingers. Semi-automatic stretching can also be achieved by means of a sliding bar.

In this embodiment of the present disclosure, the user can click "Perspective Transformation" to enter the perspective transformation mode. As shown in Figure 10, by giving four corner points, the user can manually drag each corner point to realize perspective transformation.

When the user performs an editing operation on the cropped image, the electronic device edits the cropped image in response to the user's editing operation on the cropped image, so that an image that better meets the user's needs can be obtained.

The image cropping method of the embodiment of the present disclosure obtains plane point cloud data by performing plane detection on the point cloud data, and performs perspective transformation on a color image based on the plane point cloud data to obtain a perspective transformed image, so that perspective transformation does not depend on image information. And based on the three-dimensional information of the plane point cloud data, the bounding box is calculated, and the information of the cropping range is obtained according to the projection position of the bounding box in the perspective transformed image. By using the line segment information in the color image, it can further assist to correct the result of point cloud detection and improve the accuracy of automatic cropping. Or by performing region detection on a color image, the subject range can be obtained without line segment information to assist in the range calculation of automatic cropping. Therefore, the present disclosure can accurately obtain the boundary range of the target object and perform automatic cropping when the target object has no obvious lines or any direction information, or even when there is interfering information (such as line segment stripes of the object, etc.). , which can improve the accuracy of cropping. In addition, the user can further perform manual cropping, so as to obtain an image that is more in line with the user's needs.

It should be noted that although the various steps of the methods of the present disclosure are depicted in the figures in a particular order, this does not require or imply that the steps must be performed in that particular order, or that all illustrated steps must be performed to achieve the desired the result of. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be decomposed into multiple steps for execution, and the like.

Corresponding to the above method embodiments, the embodiments of the present disclosure further provide an image cropping apparatus. Referring to FIG. 11 , an image cropping apparatus 1100 includes:

The image acquisition module 1110 is used to acquire the color image and the depth image of the target object;

The image boundary information determination module 1120 is used to perform perspective transformation on the color image to obtain the perspective transformed image, and extract the image boundary information of the target object in the perspective transformed image;

The point cloud boundary information determination module 1130 is configured to determine the point cloud boundary information of the target object according to the point cloud data generated based on the depth image;

a target boundary information determination module 1140, configured to obtain target boundary information based on the image boundary information and the point cloud boundary information;

The image cropping module 1150 is used for cropping the color image according to the target boundary information to obtain a cropped image.

In an exemplary embodiment of the present disclosure, the image boundary information determination module includes:

a line segment detection unit for performing line segment detection on the perspective transformed image;

The image boundary information first determining unit is configured to determine the image boundary information of the target object according to the line segment information when the perspective transformation image contains line segment information.

In an exemplary embodiment of the present disclosure, the image boundary information determination module further includes:

The second image boundary information determining unit is used to process the perspective transformed image based on a region detection algorithm when the line segment information is not included in the perspective transformed image to obtain an image of the main region; and determine the image boundary information of the target object according to the image of the main region.

In an exemplary embodiment of the present disclosure, the image boundary information determination module further includes:

The perspective transformation unit is used to perform plane detection on the point cloud data generated based on the depth image, and determine a perspective transformation matrix according to the detected plane; perform perspective transformation on the color image through the perspective transformation matrix to obtain a perspective transformation image.

In an exemplary embodiment of the present disclosure, the point cloud boundary information determination module includes:

The plane point cloud data acquisition unit is used to obtain plane point cloud data after plane detection is performed on the point cloud data generated based on the depth image;

The bounding box determination unit is used to determine the bounding box of the plane point cloud data;

The mapping unit is used to map the bounding box to the perspective transformation image to obtain the point cloud boundary information of the target object.

In an exemplary embodiment of the present disclosure, the above-mentioned image cropping apparatus further includes:

The plane point cloud data filtering module is used to filter the plane point cloud data after obtaining the plane point cloud data to obtain the target plane point cloud data;

The bounding box determination unit is specifically used to determine the bounding box of the point cloud data of the target plane.

In an exemplary embodiment of the present disclosure, the plane point cloud data filtering module is specifically configured to perform distance filtering processing on plane point cloud data; or perform cluster filtering processing on plane point cloud data; or perform distance filtering processing on plane point cloud data; Distance filtering processing and cluster filtering processing.

In an exemplary embodiment of the present disclosure, the target boundary information determination module includes:

The target line segment selection unit is used to select the target line segment from the line segment represented by the image boundary information, wherein the target line segment is the closest to the line segment represented by the point cloud boundary information and the included angle is smaller than the angle threshold;

The target boundary information determining unit is used for taking the line segment information corresponding to the target line segment as the target boundary information.

In an exemplary embodiment of the present disclosure, the above-mentioned image cropping apparatus further includes:

The image display module is used for displaying the target boundary information in the color image after obtaining the target boundary information, so that the user can crop the color image based on the target boundary information.

In an exemplary embodiment of the present disclosure, the above-mentioned image cropping apparatus further includes:

The image display module is also used for displaying the cropped image to the user after the cropped image is obtained;

The image editing module is used to edit the cropped image in response to the user's editing operation on the cropped image.

In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer executes any of the methods described above.

It should be noted that the computer-readable storage medium shown in the present disclosure can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory, read only memory, erasable programmable read only memory (EPROM) or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, radio frequency, etc., or any suitable combination of the foregoing.

Other embodiments of the present disclosure will readily suggest themselves to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (13)

An image cropping method including: Obtain the color image and depth image of the target object; Perform perspective transformation on the color image to obtain a perspective transformed image, and extract the image boundary information of the target object in the perspective transformed image; Determine the point cloud boundary information of the target object according to the point cloud data generated based on the depth image; Based on the image boundary information and the point cloud boundary information, obtain target boundary information; The color image is cropped according to the target boundary information to obtain a cropped image. The method according to claim 1, wherein extracting the image boundary information of the target object in the perspective transformed image comprises: performing line segment detection on the perspective transformed image; When the perspective transformed image includes line segment information, image boundary information of the target object is determined according to the line segment information. The method according to claim 2, wherein after performing line segment detection on the perspective transformed image, the method further comprises: When the perspective transformed image does not contain line segment information, processing the perspective transformed image based on a region detection algorithm to obtain a subject region image; According to the subject area image, image boundary information of the target object is determined. The method according to claim 1, wherein the performing perspective transformation on the color image to obtain a perspective transformed image comprises: Perform plane detection on the point cloud data generated based on the depth image, and determine a perspective transformation matrix according to the detected plane; Perform perspective transformation on the color image through the perspective transformation matrix to obtain a perspective transformed image. The method according to claim 4, wherein determining the point cloud boundary information of the target object according to the point cloud data generated based on the depth image, comprising: After plane detection is performed on the point cloud data, plane point cloud data is obtained; determining the bounding box of the plane point cloud data; The bounding box is mapped to the perspective transformation image to obtain point cloud boundary information of the target object. The method of claim 5, further comprising: After obtaining the plane point cloud data, filtering the plane point cloud data to obtain target plane point cloud data; The determining of the bounding box of the plane point cloud data specifically includes: A bounding box of the target plane point cloud data is determined. The method according to claim 6, wherein the filtering processing on the plane point cloud data comprises: performing distance filtering processing on the plane point cloud data; or Perform cluster filtering processing on the plane point cloud data; or Perform distance filtering processing and cluster filtering processing on the plane point cloud data. The method according to claim 3, wherein the obtaining target boundary information based on the image boundary information and the point cloud boundary information comprises: Selecting a target line segment from the line segment represented by the image boundary information, wherein the target line segment is the closest to the line segment represented by the point cloud boundary information and the included angle is less than an angle threshold; The line segment information corresponding to the target line segment is used as the target boundary information. The method of claim 1, further comprising: After the target boundary information is obtained, the target boundary information is displayed in the color image, so that the user can crop the color image based on the target boundary information. The method of claim 1, further comprising: After obtaining the cropped image, displaying the cropped image to the user; The cropped image is edited in response to an editing operation of the cropped image by the user. An image cropping device, comprising: The image acquisition module is used to acquire the color image and depth image of the target object; an image boundary information determination module, for carrying out perspective transformation to the color image, obtaining a perspective transformation image, and extracting the image boundary information of the target object in the perspective transformation image; a point cloud boundary information determination module, configured to determine the point cloud boundary information of the target object according to the point cloud data generated based on the depth image; a target boundary information determination module, configured to obtain target boundary information based on the image boundary information and the point cloud boundary information; An image cropping module, configured to crop the color image according to the target boundary information to obtain a cropped image. An electronic device comprising: processor; and a memory configured to store executable instructions for the processor; Wherein, the processor is configured to perform the method of any one of claims 1-10 by executing the executable instructions. A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the method of any one of claims 1-10.

Images