WO2025130944A1 - Picture editing method and system, and related device - Google Patents

Abstract

Provided in the embodiments of the present application are a picture editing method and system, and a related device. On the basis of the understanding of semantic content of a picture and various features, such as a mask feature, a depth feature, a contour feature and a color feature, of an editing area selected by a user, an editing type (such as deletion, dragging, replacement, addition and color adjustment) and editing parameters (such as replaceable content, the target position of the dragging, newly added content and a color adjustment value) are recommended to the user, thereby providing editing ideas for the user. In addition, it is also possible to determine whether an editing instruction is rationally matched with the editing area selected by the user, and not only can the user be prompted that the editing instruction cannot be executed, but the user can also be recommended to select a new editing area, or the user can be assisted in modifying the editing instruction, such that the problem of an editing effect failing to conform to common sense logic due to random input of editing instructions by the user is solved, thereby making it possible to reduce the number of trials and errors when the user edits pictures, and thus significantly improving the image-making efficiency.

Description

Image editing method, related equipment and system

This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on December 22, 2023, with application number 202311795993.1, and priority to the Chinese patent application with the invention name “Picture Editing Method, Related Equipment and System”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of electronic technology, and in particular to a picture editing method, related equipment and system.

Background Art

Currently, many professional image editing tools or applications can assist users in editing and processing local areas of images. When editing images, users usually need to manually use functions such as lasso and box selection to mark and select the area they want to modify in the original image, and then perform operations such as cutting out, redrawing, coloring, or adding new elements to finally obtain the edited image. The entire process has high requirements for professional knowledge and the use of professional tools, and requires a lot of manpower costs to complete high-quality editing tasks, which is not suitable for ordinary users.

Summary of the invention

The embodiments of the present application provide a picture editing method, related equipment and system, which can lower the threshold for picture editing, support user-defined editing instructions, and understand the user's operating intentions, provide users with editing ideas, avoid users from experiencing a lot of trial and error, and significantly improve the efficiency of picture output.

In a first aspect, an embodiment of the present application provides a picture editing method, which can be applied to a picture editing system, and the method may include: displaying a first picture, detecting a user operation on the first picture for selecting an editing area, determining an editing area from the first picture according to the operation position of the user operation on the first picture, and distinguishing and displaying the editing area in the first picture; then, generating recommended editing instructions according to image features of the editing area, and displaying the recommended editing instructions; detecting that a user inputs a first editing instruction for the editing area, the first editing instruction including: a recommended editing instruction; then, performing image editing processing on the first picture according to the first editing instruction; and finally, displaying the first picture after the image editing processing.

In the first aspect, the first picture can be a picture opened by a user through an application for browsing, managing or processing pictures, such as a photo album (also known as a gallery), a photo editing application, a drawing design program, etc. The first picture can be stored on a terminal device such as a mobile phone, or stored on a network.

The method provided in the first aspect is based on the understanding of the semantic content of the original image. It recommends editing instructions to the user according to the image features of the editing area selected by the user, and provides the user with editing ideas. This not only reduces the complexity of use, but also ensures the rationality of the content of the edited image, avoids users from experiencing a lot of trial and error, and significantly improves the efficiency of image output.

In combination with the first aspect, in some embodiments, generating recommended editing instructions based on image features of the editing area may specifically include: using a fused feature vector of multiple image features of the editing area as one of the inputs of the first artificial intelligence algorithm; using one or more preset editing types as the second input of the first artificial intelligence algorithm; obtaining recommended editing instructions through the operation of the first artificial intelligence algorithm, the recommended editing instructions including editing parameters corresponding to the preset editing types; wherein the multiple image features may include multiple items of mask features, depth features, contour features, and color features.

In conjunction with the first aspect, in some embodiments, the preset editing type may include one or more of the following: delete, drag, replace, add, or color adjustment. The editing parameter corresponding to the deletion may include the deleted content, the editing parameter corresponding to the replacement may include the replaceable content, the editing parameter corresponding to the drag may include the drag target position, the editing parameter corresponding to the addition may include the newly added content, the editing parameter corresponding to the color adjustment may include the color adjustment value, etc.

In conjunction with the first aspect, in some embodiments, detecting that the user inputs the first editing instruction for the editing area may specifically include: detecting that the user selects to input a recommended editing instruction, so that the recommended editing instruction selected by the user is determined as the first editing instruction.

In combination with the first aspect, in some embodiments, after identifying the editing area, the picture editing system is not limited to displaying recommended editing instructions, and can also prompt the user to enter editing instructions in the following manner: display a first input box, the first input box can be used to receive voice or text editing instructions; the first editing instructions also include voice or text editing instructions entered through the input box. When prompting the user to enter editing instructions in this manner, detecting that the user enters the first editing instruction for the editing area can specifically include: detecting the voice or text instruction entered by the user in the first input box, and the voice or text instruction being determined as the first editing instruction. That is, the first editing instruction can include the user entering a text instruction in the input box or pressing the voice key to enter a voice instruction.

In conjunction with the first aspect, in some embodiments, after identifying the editing area, the image editing system is not limited to displaying recommended editing instructions, and can also prompt the user to enter editing instructions in the following manner: display one or more preset editing instructions, such as commonly used editing instructions or editing instructions saved in advance by the user. When prompting the user to enter editing instructions in this manner, detecting that the user enters a first editing instruction for the editing area can specifically include: detecting that the user selects to enter a preset editing instruction, and the voice or text instruction is determined to be the first editing instruction.

In combination with the first aspect, in some embodiments, the user operation for selecting an editing area may include: a user operation of selecting a first object in a first picture, the operation position of the user operation on the first picture falls on the first object in the first picture, and the image area where the first object is located is the editing area; the image area where the first object is located is determined by performing image segmentation processing on the first picture.

In combination with the first aspect, in some embodiments, the user operation for selecting the editing area may include: a user operation of drawing the editing area in the first picture.

In combination with the first aspect, in some embodiments, distinctively displaying the editing area in the first picture may include one or more of the following methods: highlighting the outline of the editing area, highlighting the entire editing area, or displaying a dotted frame along the outline of the editing area.

In conjunction with the first aspect, in some embodiments, the method of the first aspect may further include: obtaining preprocessing information of the first image, and determining the editing area from the area indicated by the preprocessing information. In this way, the editing area can be determined based on the preprocessing information of the image, rather than based on image segmentation technology, and repeated online calculations are not required.

In combination with the first aspect, in some embodiments, the preprocessing information may include indication information of multiple regions, wherein the preprocessing information includes coordinates of contour points of each of the multiple regions, a binary image, and a grayscale image, the values of the multiple regions in the binary image are first values, and the grayscale values of the multiple regions in the grayscale image are first grayscale values or first grayscale ranges. In this way, determining the editing region from the region indicated by the preprocessing information may specifically include: determining the region where the operation position of the user operation is located in the multiple regions as the editing region, or determining the region closest to the operation position in the multiple regions as the editing region.

In conjunction with the first aspect, in some embodiments, the preprocessing information may only include indication information of a region, such as coordinates of contour points of a region, a binary image or a grayscale image indicating a region, etc. In this way, determining the editing region from the region indicated by the preprocessing information may specifically include: directly determining the region as the editing region.

In conjunction with the first aspect, in some embodiments, the preprocessing information of the first image may not directly indicate one or more regions, but may include other data, such as multi-layer information, or contour information, or depth information. In this way, the other data may be used to first determine one or more regions, and then the editing region may be determined from the one or more regions.

Methods for determining one or more regions indirectly indicated by the pre-processing information may include, but are not limited to:

If the preprocessing information includes layer information of multiple layers, and the layer information of each layer includes the coordinates of opaque pixels in the layer, then: before determining the editing area from the area indicated by the preprocessing information, the connected opaque pixels in each layer can be determined as an area indicated by the preprocessing information.

If the preprocessing information includes contour information, then: before determining the editing area from the area indicated by the preprocessing information, the area surrounded by the contour indicated by the contour information can be determined as the area indicated by the preprocessing information.

If the preprocessing information includes depth information, then: before determining the editing area from the area indicated by the preprocessing information, pixels with the same or similar depth values may be determined as an area indicated by the preprocessing information based on the depth information.

In conjunction with the first aspect, in some embodiments, the method of the first aspect may further include: before performing image editing processing on the first picture according to the first editing instruction, if it is determined that the first editing instruction is not reasonable to apply to the editing area, re-recommending the editing instruction or re-recommending the editing area. In this way, the problem that the editing effect that does not conform to common sense logic is caused by the user's random input of editing instructions is solved, thereby reducing the number of trial and error times when the user edits the picture.

In combination with the first aspect, in some embodiments, it can be determined whether the first editing instruction is reasonable to be applied to the editing area by comparing the feature vector corresponding to the first editing instruction with the fused feature vector of each image feature of the editing area.

In combination with the first aspect, in some embodiments, re-recommending the editing area may specifically include: traversing the area outside the editing area in the first image, comparing the fused feature vector of the traversed area with the feature vector corresponding to the first editing instruction, finding an area that can be reasonably matched with the first editing instruction, and re-recommending the found area as the editing area.

In combination with the first aspect, in some embodiments, re-recommending editing instructions may specifically include: traversing the editing types and/or editing parameters in the recommendation pool, comparing the feature vectors corresponding to the traversed editing types and/or editing parameters with the fused feature vector of the editing area, finding the editing types and/or editing parameters that can reasonably match the editing area, and modifying the first editing instruction according to the found editing type and/or editing parameters, and re-recommending the modified first editing instruction.

In conjunction with the first aspect, in some embodiments, the editing instructions input by the user for the editing area may involve adding a new object, and the editing instructions involving adding a new object may include: adding, replacing, dragging and other types of editing instructions. Among them, replacing is equivalent to deleting an object from the original image and then adding another object; dragging is equivalent to deleting an object from a certain position in the original image and then adding the object to another position. That is, the editing instruction involving adding a new object may mean that the editing process corresponding to the editing instruction includes: adding a new object to the editing area.

After such image editing processing, in the edited area in the first picture, the depth features of the first pixel area are replaced by the depth features of the new object, and the depth features of the second pixel area remain the depth features of the original object, wherein the perspective relationship of the first pixel area is that the new object is in front of the original object, and the perspective relationship of the second pixel area is that the original object is in front of the new object.

In combination with the first aspect, in some embodiments, performing image editing processing on the first picture according to the first editing instruction may include: correcting image features of the first picture, and regenerating the first picture using the corrected image features of the first picture.

In combination with the first aspect, in some embodiments, the image features may include depth features. Correcting the image features of the first image may specifically include: determining the perspective relationship between the new object and the original object based on the image features of the first image and the editing parameters in the first editing instruction; determining the reference depth of the new object based on the perspective relationship between the new object and the original object, and the depth value of the original object, and then correcting the depth features of the new object using the reference depth; replacing the original depth features of the editing area with the corrected depth features of the new object; wherein, after the depth features of the new object are corrected, the average depth of the new object is close to or equal to the reference depth, and the depth difference between various areas on the new object remains unchanged.

In combination with the first aspect, in some embodiments, the image feature may further include a first image feature, which is an image feature other than a depth feature and may include one or more of the following: a mask feature, a contour feature, and a color feature.

Correcting the image features of the first picture may also include: in the editing area, if the depth feature of a pixel area is replaced with the corrected depth feature of the new object, then using the first image feature of the pixel area on the new object to replace the original first image feature of the pixel area in the editing area.

In the second aspect, an embodiment of the present application provides a terminal device, which may include: a human-computer interaction module, a processor and a memory, wherein the human-computer interaction module is coupled to the processor, and the memory is coupled to the processor; the human-computer interaction module may include input and output components such as a touch screen; wherein the memory may be used to store computer program code, and the computer program code may include computer instructions, and when the processor executes the computer instructions, the terminal device executes the method described in any one or more embodiments of the first aspect mentioned above.

When the terminal device provided in the second aspect has powerful computing capabilities, complex calculations can be performed directly on the computing module (including the processor) of the terminal device. When the terminal device has powerful computing capabilities, the human-computer interaction module and the computing module mentioned in the subsequent embodiments can be deployed on the terminal device, and the steps performed by the two are the steps performed by the terminal device, and the communication or data interaction between the two belongs to intra-device communication.

In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, characterized in that when the instructions are executed on a terminal device, the terminal device executes the method described in any one or more embodiments of the first aspect above.

In a fourth aspect, an embodiment of the present application provides a method for editing an image, which can be applied to a human-computer interaction module. The human-computer interaction module is included in an image editing system, and the image editing system also includes: a computing module.

The method may include: a human-computer interaction module displays a first image, detects a user operation on the first image for selecting an editing area, and displays the editing area in the first image; then, the human-computer interaction module receives a recommended editing instruction sent by a computing module, and displays the recommended editing instruction, which is generated by the computing module according to image features of the editing area; the human-computer interaction module detects that a user inputs a first editing instruction for the editing area, and the first editing instruction includes: a recommended editing instruction; finally, the human-computer interaction module receives the first image after image editing processing sent by the computing module, and displays the first image after image editing processing, and the image editing processing is performed by the computing module according to the first editing instruction.

In the fourth aspect, the first picture may be a picture opened by a user through an application for browsing, managing or processing pictures, such as a photo album (also known as a gallery), a photo editing application, a drawing design program, etc. The first picture may be stored on a terminal device such as a mobile phone, or may be stored on a network. Some technical details of the fourth aspect may refer to any one or more embodiments of the first aspect.

In conjunction with the fourth aspect, in some embodiments, detecting that the user inputs the first editing instruction for the editing area may specifically include: detecting that the user selects to input a recommended editing instruction, so that the recommended editing instruction selected by the user is determined as the first editing instruction.

In conjunction with the fourth aspect, in some embodiments, after identifying the editing area, the human-computer interaction module is not limited to displaying recommended editing instructions, and can also prompt the user to enter editing instructions in the following manner: display a first input box, the first input box can be used to receive voice or text editing instructions; the first editing instruction also includes a voice or text editing instruction entered through the input box. When prompting the user to enter an editing instruction in this manner, it is detected that the user enters the first editing instruction for the editing area, which can specifically include: detecting the voice or text instruction entered by the user in the first input box, and the voice or text instruction is determined to be the first editing instruction. That is, the first editing instruction can include the user entering a text instruction in the input box or pressing the voice key to enter a voice instruction.

In conjunction with the fourth aspect, in some embodiments, after identifying the editing area, in addition to displaying recommended editing instructions, the human-computer interaction module may also prompt the user to enter the editing instruction in the following manner: display one or more preset editing instructions, such as commonly used editing instructions or editing instructions saved in advance by the user. When prompting the user to enter the editing instruction in this manner, detecting that the user enters the first editing instruction for the editing area may specifically include: detecting that the user selects to enter the preset editing instruction, and the voice or text instruction is determined as the first editing instruction.

In combination with the fourth aspect, in some embodiments, the user operation for selecting an editing area may include: a user operation of selecting a first object in a first picture, the operation position of the user operation on the first picture falls on the first object in the first picture, and the image area where the first object is located is the editing area; the image area where the first object is located is determined by performing image segmentation processing on the first picture.

In conjunction with the fourth aspect, in some embodiments, the user operation for selecting the editing area may include: a user operation for drawing the editing area in the first picture.

In combination with the fourth aspect, in some embodiments, distinctively displaying the editing area in the first picture may include one or more of the following methods: highlighting the outline of the editing area, highlighting the entire editing area, or displaying a dotted box along the outline of the editing area.

In combination with the fourth aspect, in some embodiments, after the human-computer interaction module detects that the user inputs a first editing instruction for the editing area, the method may further include: if the first editing instruction is not reasonable to be applied to the editing area, the human-computer interaction module re-recommends the editing area; the re-recommended editing area is found by the calculation module from an area outside the editing area based on the feature vector corresponding to the first editing instruction.

In combination with the fourth aspect, in some embodiments, after the human-computer interaction module detects that the user inputs a first editing instruction for the editing area, the method may also include: if the application of the first editing instruction to the editing area is unreasonable, the human-computer interaction module recommends a modified first editing instruction; the editing type and/or editing parameters of the modified first editing instruction are found by the calculation module from the recommendation pool based on the fused feature vector of the editing area.

In a fifth aspect, an embodiment of the present application provides a method for editing a picture, the method is applied to a computing module, the computing module is included in a picture editing system, and the picture editing system further includes: a human-computer interaction module;

The method may include: a computing module generates a recommended editing instruction based on image features of an editing area of a first image, and sends the recommended editing instruction to a human-computer interaction module, so that the human-computer interaction module displays the recommended editing instruction; then, the computing module receives the first editing instruction sent by the human-computer interaction module, and performs image editing processing on the first image according to the first editing instruction; finally, the computing module sends the first image after image editing processing to the human-computer interaction module, so that the human-computer interaction module displays the first image after image editing processing.

In conjunction with the fifth aspect, in some embodiments, the computing module generates a recommended editing instruction based on the image features of the editing area, which may specifically include: the computing module uses the fused feature vector of multiple image features of the editing area as one of the inputs of the first artificial intelligence algorithm, and uses one or more preset editing types as the second input of the first artificial intelligence algorithm; then, the computing module obtains the recommended editing instruction through the operation of the first artificial intelligence algorithm, and the recommended editing instruction includes the editing parameters corresponding to the preset editing type. Among them, the multiple image features may include multiple features of the mask feature, depth feature, contour feature, and color feature.

In combination with the fifth aspect, in some embodiments, the preset editing type includes one or more of the following: deletion, dragging, replacement, addition, or color adjustment.

In conjunction with the fifth aspect, in some embodiments, the method of the fifth aspect may further include: the computing module obtains preprocessing information of the first image, and determines the editing area from the area indicated by the preprocessing information. In this way, the editing area can be determined based on the preprocessing information of the image, rather than based on image segmentation technology, and repeated online calculations are not required.

In conjunction with the fifth aspect, in some embodiments, the preprocessing information may include indication information of multiple regions, wherein the preprocessing information includes coordinates of contour points of each of the multiple regions, a binary image, and a grayscale image, the values of the multiple regions in the binary image are first values, and the grayscale values of the multiple regions in the grayscale image are first grayscale values or first grayscale ranges. In this way, determining the editing region from the region indicated by the preprocessing information may specifically include: determining the region where the operation position of the user operation is located in the multiple regions as the editing region, or determining the region closest to the operation position in the multiple regions as the editing region.

In conjunction with the fifth aspect, in some embodiments, the preprocessing information may only include indication information of a region, such as coordinates of contour points of a region, a binary image or a grayscale image indicating a region, etc. In this way, determining the editing region from the region indicated by the preprocessing information may specifically include: directly determining the region as the editing region.

In conjunction with the fifth aspect, in some embodiments, the preprocessing information of the first image may not directly indicate one or more regions, but may include other data, such as multi-layer information, or contour information, or depth information. In this way, the other data may be used to first determine one or more regions, and then the editing region may be determined from the one or more regions.

Methods for determining one or more regions indirectly indicated by the pre-processing information may include, but are not limited to:

If the preprocessing information includes layer information of multiple layers, and the layer information of each layer includes the coordinates of opaque pixels in the layer, then: before determining the editing area from the area indicated by the preprocessing information, the calculation module can determine the connected opaque pixels in each layer as an area indicated by the preprocessing information.

If the preprocessing information includes contour information, then: before determining the editing area from the area indicated by the preprocessing information, the calculation module may determine the area surrounded by the contour indicated by the contour information as the area indicated by the preprocessing information.

If the preprocessing information includes depth information, then: before determining the editing area from the area indicated by the preprocessing information, the calculation module may determine pixels with the same or similar depth values as an area indicated by the preprocessing information according to the depth information.

In conjunction with the fifth aspect, in some embodiments, the method of the first aspect may further include: before performing image editing processing on the first picture according to the first editing instruction, if it is determined that the first editing instruction is not reasonable to apply to the editing area, the computing module re-recommends the editing instruction or re-recommends the editing area. In this way, the problem that the editing effect caused by the user's random input of the editing instruction does not conform to common sense logic is solved, thereby reducing the number of trial and error times when the user edits the picture.

In conjunction with the fifth aspect, in some embodiments, the computing module may determine whether it is reasonable to apply the first editing instruction to the editing area by comparing the feature vector corresponding to the first editing instruction with the fused feature vector of each image feature of the editing area.

In combination with the fifth aspect, in some embodiments, re-recommending the editing area may specifically include: the computing module traverses the area outside the editing area in the first image, compares the fused feature vector of the traversed area with the feature vector corresponding to the first editing instruction, finds the area that can be reasonably matched with the first editing instruction, and re-recommends the found area as the editing area.

In combination with the fifth aspect, in some embodiments, re-recommending editing instructions may specifically include: a computing module traversing the editing types and/or editing parameters in the recommendation pool, comparing the feature vectors corresponding to the traversed editing types and/or editing parameters with the fused feature vector of the editing area, finding the editing types and/or editing parameters that can reasonably match the editing area, and modifying the first editing instruction according to the found editing type and/or editing parameters, and re-recommending the modified first editing instruction.

In conjunction with the fifth aspect, in some embodiments, the editing process corresponding to the first editing instruction may include: adding a new object to the editing area. After such image editing process, in the editing area of the first picture, the depth feature of the first pixel area is replaced with the depth feature of the new object, and the depth feature of the second pixel area remains the depth feature of the original object, wherein the perspective relationship of the first pixel area is that the new object is in front of the original object, and the perspective relationship of the second pixel area is that the original object is in front of the new object.

In combination with the fifth aspect, in some embodiments, the computing module performs image editing processing on the first picture according to the first editing instruction, including: the computing module corrects the image features of the first picture, and regenerates the first picture using the corrected image features of the first picture.

In conjunction with the fifth aspect, in some embodiments, the image features may include depth features. The calculation module corrects the image features of the first image, specifically including: the calculation module determines the perspective relationship between the new object and the original object according to the image features of the first image and the editing parameters in the first editing instruction; determines the reference depth of the new object according to the perspective relationship between the new object and the original object, and the depth value of the original object, and then corrects the depth feature of the new object using the reference depth; replaces the original depth feature of the editing area with the corrected depth feature of the new object; wherein, after the depth feature of the new object is corrected, the average depth of the new object is close to or equal to the reference depth, and the depth difference between various areas on the new object remains unchanged.

In combination with the fifth aspect, in some embodiments, the image feature may also include a first image feature, which is an image feature other than a depth feature, and includes one or more of the following: a mask feature, a contour feature, and a color feature.

The calculation module corrects the image features of the first image, and may also include: in the editing area, if the depth feature of a pixel area is replaced with the corrected depth feature of the new object, the calculation module uses the first image feature of the pixel area on the new object to replace the original first image feature of the pixel area in the editing area.

In the sixth aspect, an embodiment of the present application provides a terminal device, which may include: a human-computer interaction module, a processor and a memory, wherein the human-computer interaction module is coupled to the processor, and the memory is coupled to the processor; the human-computer interaction module may include input and output components such as a touch screen; wherein the memory can be used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the terminal device executes the method described in any one or more embodiments of the aforementioned fourth aspect.

Unlike the terminal device provided in the second aspect, which has powerful computing capabilities, the terminal device provided in the sixth aspect does not have powerful computing capabilities, and needs to submit complex computing tasks (such as editing instruction recommendation algorithms, image processing algorithms, image semantic understanding algorithms, etc.) to the cloud through the network for execution and wait for the task execution results returned by the cloud-side server. When the terminal device does not have powerful computing capabilities, the aforementioned human-computer interaction module can be deployed on the terminal device, and the aforementioned computing module can be deployed on the server. The steps executed by the human-computer interaction module are the steps executed by the terminal device, and the steps executed by the computing module are the steps executed by the server. The communication or data interaction between the two belongs to inter-device communication.

In the seventh aspect, an embodiment of the present application provides a server, which may include: a processor and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store computer program code, and the computer program code includes computer instructions, and when the processor executes the computer instructions, the terminal device executes the method described in any one or more embodiments of the aforementioned fifth aspect.

In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, characterized in that when the instructions are executed on a terminal device, the terminal device executes the method described in any one or more embodiments of the aforementioned fourth aspect.

In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, characterized in that when the instructions are executed on a terminal device, the terminal device executes the method described in any one or more embodiments of the aforementioned fifth aspect.

In the tenth aspect, an embodiment of the present application provides a picture editing system, which may include: a terminal device and a server, wherein the terminal device is the terminal device described in the sixth aspect, and the server is the server described in the seventh aspect.

In an eleventh aspect, an embodiment of the present application provides a picture editing system, which may include: a human-computer interaction module and a computing module, wherein:

The human-computer interaction module may be used to display the first image, detect a user operation on the first image for selecting an editing area, and inform the calculation module of an operation position of the user operation on the first image;

The calculation module may be used to determine an editing area from the first image according to an operation position of a user operation on the first image, and inform the human-computer interaction module of the editing area;

The human-computer interaction module may also be used to distinguish and display the editing area in the first picture;

The calculation module may also be used to generate a recommended editing instruction according to the image features of the editing area, and inform the human-computer interaction module of the recommended editing instruction;

The human-computer interaction module can also be used to display recommended editing instructions, then detect that the user inputs a first editing instruction for the editing area, and send the first editing instruction to the calculation module; the first editing instruction includes: a recommended editing instruction;

The calculation module may also be used to perform image editing processing on the first picture according to the first editing instruction, and send the first picture after image editing processing to the human-computer interaction module;

Finally, the human-computer interaction module can also be used to display the first image after image editing.

The image editing system provided in the eleventh aspect can be deployed in the same device (such as a terminal device such as a mobile phone with powerful computing capabilities) or in two devices (a terminal device and a server).

In the picture editing system provided in the eleventh aspect, the human-computer interaction module can execute the method described in any one or more embodiments of the aforementioned fourth aspect, and the computing module can execute the method described in any one or more embodiments of the aforementioned fifth aspect.

In the twelfth aspect, an embodiment of the present application provides a chip system, which is applied to a terminal device. The chip system includes one or more processors, which are used to call computer instructions so that the terminal device can execute the method described in the aforementioned first aspect, or any one or more embodiments of the fourth aspect.

In the thirteenth aspect, an embodiment of the present application provides a chip system, which is applied to a server, and the chip system includes one or more processors, which are used to call computer instructions so that the server can execute the method described in any one or more embodiments of the aforementioned fifth aspect.

In a fourteenth aspect, the present application provides a computer program product comprising instructions. When the above-mentioned computer program product runs on an electronic device, the electronic device can execute the method described in any one or more embodiments of the above-mentioned first aspect, fourth aspect, or fifth aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 shows a picture editing system provided by an embodiment of the present application;

FIG2 shows a terminal device provided in an embodiment of the present application;

FIG3 shows a server provided in an embodiment of the present application;

FIG4 shows the relationship between various method embodiments of the present application;

FIG5 shows a method for editing a picture provided by an embodiment of the present application;

FIG6 shows an example of a user opening a picture;

FIG. 7 shows an example in which a user selects the editing area “sky” in a picture;

FIG8 shows an example of a user drawing a heart-shaped editing area in a picture;

FIG9 shows an example of prompting a user to input an editing instruction by voice or text;

FIG10 shows an example of outputting a recommended editing instruction to a user;

FIG11 shows another example of outputting a recommended editing instruction to a user;

FIG12 exemplarily shows the digitized form of the mask feature of the image;

FIG13 exemplarily shows a visualization form of a mask feature;

FIG14 exemplarily shows a visualization form of a deep feature;

FIG15 exemplarily shows a visualization form of contour features;

FIG16 exemplarily shows a visualization form of a color feature;

FIG17 exemplarily shows the result of executing the editing command “add ‘flying geese’” for the editing area “sky”;

FIG18 exemplarily shows the result after executing the editing instruction “adjust hue: reduce brightness to 82%, reduce saturation to 75%, and change hue to red” for the editing area “sky”;

FIG. 19 exemplarily shows an image of “flying geese”;

FIG20 shows another picture editing method provided by an embodiment of the present application;

FIG. 21 exemplarily shows that the preprocessing information of a picture only includes the coordinate data of the contour points of one area;

FIG. 22 exemplarily shows that the preprocessing information of a picture only includes binary data of one region;

FIG23 exemplarily shows a method of determining a plurality of regions indicated by preprocessing information based on preprocessing information including contour information;

FIG. 24 exemplarily shows a method of determining a plurality of regions indicated by preprocessing information based on preprocessing information including depth information;

FIG. 25 shows another picture editing method provided by an embodiment of the present application;

FIG. 26 shows a method flow for determining whether a first editing instruction and an editing area are reasonably matched, provided in an embodiment of the present application;

FIG. 27 shows an example of prompting the user that the first editing instruction and the editing area are not properly matched;

FIG28 shows an example of prompting the user to change the editing area;

FIG29 shows an example of prompting the user to change the editing instruction;

FIG30 shows a method flow of how to execute an image involving a newly added object provided by an embodiment of the present application;

FIG31 exemplarily shows a situation where the perspective relationship between the new object and the original object in the picture is unreasonable;

FIG32 exemplarily shows a new object image;

FIG33 exemplarily shows a pixel area in an edit area where a new object mask feature cannot be used;

FIG34 exemplarily shows a case where the perspective relationship between the new object and the original object in the picture is reasonable;

FIG35 exemplarily shows the size comparison between the new object image area and the area in the original image where feature replacement (such as depth feature replacement, mask feature replacement, contour feature replacement, etc.) occurs.

DETAILED DESCRIPTION

The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

In order to reduce the complexity of image editing, some simple image editing functions are widely used, such as erasing a local area of the image, moving the position of an object in the image, adjusting the color or brightness of an object in the image, etc. In these image editing functions, users do not need to perform complex editing processes such as cutting out, redrawing, and coloring the local area of the image. They only need to select the target area or target object by clicking, and enter editing instructions such as deletion, color adjustment, and moving the object position. However, this type of image editing function only supports a few inherent editing operations, and does not support user-defined editing operations; moreover, it only supports users to process content that has already appeared in the image, and does not support users to add new content to the image; in addition, it cannot prompt users whether the editing operation is reasonable, for example, "dragging a street bench to the sky" is unreasonable.

The embodiments of the present application provide a picture editing method, related equipment and system, which can lower the threshold for picture editing, support user-defined editing instructions, and understand the user's operating intentions, provide users with editing ideas, avoid users from experiencing a lot of trial and error, and significantly improve the efficiency of picture output.

The picture editing method provided in the embodiment of the present application can be implemented based on the picture editing system 10 shown in Figure 1. As shown in Figure 1, the picture editing system 10 may include a human-computer interaction module 100 and a computing module 200. The human-computer interaction module 100 and the computing module 200 work together to jointly implement the picture editing method provided in the embodiment of the present application.

Among them, the human-computer interaction module 100 can be used as a human-computer interaction interface for users to use the picture editing function. Picture editing programs, such as photo albums (also known as gallery), picture editing applications, drawing design programs, etc., can be run on the human-computer interaction module 100. The human-computer interaction module 100 can provide input capabilities such as touch input, audio input, gesture input, and output capabilities such as display output and audio output, so that users can use picture editing functions by clicking, dragging, text input, gestures, etc., such as adding new objects to the picture, replacing an object in the picture, erasing a local area in the picture, moving the position of an object in the picture, adjusting the color or brightness of an object in the picture, etc. The human-computer interaction module 100 can also be used to communicate with the calculation module 200 to transmit editing parameters (such as the position clicked by the user, the image features of the editing area, etc.) to the calculation module 200, so that the calculation module performs complex calculations involved in the picture editing function according to the editing parameters. What are the editing parameters and what kind of calculations the calculation module performs according to the editing parameters will be described in detail in the following embodiments, and will not be expanded here.

Among them, the calculation module 200 can be responsible for the complex calculations involved in the picture editing function, such as editing instruction recommendation algorithm, image processing algorithm, image understanding algorithm, etc. The calculation module 200 can also be used to communicate with the human-computer interaction module 100 to receive the editing parameters transmitted by the human-computer interaction module 100, and return the processing results to the human-computer interaction module 100, so that the human-computer interaction module 100 provides editing suggestions to the user or displays the edited picture according to the processing results.

The human-computer interaction module 100 and the computing module 200 can be in different devices respectively. For example, the human-computer interaction module 100 can be in a terminal device such as a mobile phone, a tablet computer, a smart screen, a smart watch, etc., and the computing module 200 can be in a device such as a cloud-side server that can provide stronger computing power, or in other terminal devices with surplus computing power. At this time, the communication between the two is the communication between devices, which can be mobile communications such as 2G/3G/4G/5G, wireless fidelity (Wi-Fi) communication, satellite communication and other wireless communications, or Ethernet communication, Universal Serial Bus (USB) communication and other wired communications.

The human-computer interaction module 100 and the computing module 200 may also be integrated into the same device, for example, both are in a terminal device such as a mobile phone, a tablet computer, a personal computer, etc. that has both human-computer interaction capabilities and complex computing capabilities. In this case, the communication between the two is intra-device communication, which can be intra-device communication such as bus communication and shared memory communication.

FIG. 2 exemplarily shows a terminal device 300 provided in an embodiment of the present application.

The terminal device 300 may have both human-computer interaction capability and computing capability. The device type of the terminal device 300 may be any of a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), and smart home devices such as smart large screens, wearable devices such as smart watches and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), in-vehicle devices or smart city devices, etc.

As shown in FIG2 , the terminal device 300 may include: a processor 110, a memory 120, a display 130, a display driver integrated circuit (DDIC) 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a gyroscope sensor 180B, an acceleration sensor 180E, and a touch sensor 180K, etc. The various parts in the terminal device 300 may be connected through a bus.

Among them, the processor 110 can be responsible for providing computing power and can be used as a computing module of the terminal device; the display 130, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193 and other input and output components can be responsible for providing human-computer interaction capabilities and can be used as a human-computer interaction module of the terminal device. When the computing module in the terminal device has powerful computing power, the terminal device 300 can be independently implemented as the picture editing system 10 shown in Figure 1; at this time, the part of the terminal device 300 responsible for providing computing power (such as the processor) can constitute the computing module 200 in the picture editing system 10, and the part of the terminal device 300 responsible for providing human-computer interaction capabilities (such as the display screen, touch sensor, etc.) can constitute the human-computer interaction module 100 in the picture editing system 10. When the computing module in the terminal device does not have strong computing power, the terminal device 300 can also be implemented as only the human-computer interaction module 100 in the picture editing system 10, and the computing module 200 in the picture editing system 10 can be implemented by the cloud-side server.

The processor 110 may be one or more, and they may be integrated into an integrated circuit of a system on chip (SOC). SOC is a system-level chip. The processor 110 may include a central processing unit (CPU), a graphic processing unit (GPU), a neural-network processing unit (NPU), etc. Among them, the CPU may include an application processor (AP), a baseband processor chip (BP), etc., wherein the AP may be responsible for running the operating system, user interface, and application program on the terminal device; the BP may be responsible for receiving and sending wireless signals and managing radio frequency services. The GPU may be responsible for graphics rendering, coloring according to the rendering instructions and data from the CPU, filling, rendering, and outputting materials, etc. The NPU quickly processes input information by drawing on the biological neural network structure, such as the transmission mode between neurons in the human brain, and can also continuously self-learn. The NPU can be used to run artificial intelligence algorithms, such as editing instruction recommendation algorithms, image processing algorithms, image understanding algorithms, etc. The CPU and GPU can be used to render and synthesize the images to be sent to the display 130.

The processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input output (GPIO) interface, a subscriber identity module (SIM) interface, and/or a universal serial bus (USB) interface, etc.

The processor 110 may be provided with a cache memory for storing instructions or data just used or cyclically used by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the cache memory, which can reduce the waiting time of the processor 110 and improve the program running efficiency.

The memory 120 may include a program storage area and a user data storage area, wherein the program storage area may store an operating system and one or more application programs (such as game applications), and the data storage area may store data (such as photos, contacts) created by the user when using the terminal device 300. The memory 120 may be a high-speed random access memory or a non-volatile memory, such as a disk, a flash memory, a universal flash storage (UFS), etc. The memory 120 may also be an external memory card, such as a Micro SD card.

The memory 120 may also store code instructions of the image editing method provided in the embodiment of the present application. When the processor 110 reads the code instructions from the memory 120 and runs the code instructions, the terminal device 300 can execute the steps performed by the human-computer interaction module and/or the computing module in the image editing method provided in the embodiment of the present application.

The memory 120 may also be integrated with the processor 110 in an integrated circuit of a SOC.

As shown in Figure 2, the terminal device 300 can realize the display function through SOC, DDIC 140, and display 130.

The display 130 has multiple refresh rates. The refresh rate indicates the number of times the display screen refreshes the display screen in 1 second. For example, a 60 Hz refresh rate indicates that the display screen refreshes the display screen 60 times in 1 second. The display 130 may use an LTPO display panel, allowing the refresh rate to be reduced to a low refresh rate, such as 10 Hz or 1 Hz, thereby supporting the reduction of the power consumption of the display screen.

Among them, the display driver integrated circuit (DDIC) 140 can be used as the control core of the display 130 to drive the display 130 to work and receive data from the SOC (processor 110), such as image data and some instructions. The DDIC 140 can send driving signals and data to the display panel of the display 130 in the form of electrical signals, and then realize the control of the screen brightness and color, so that image information such as letters and pictures can be displayed on the screen, completing the screen refresh.

The image data of the screen to be displayed sent by the SOC to the DDIC 140 can be sent to the frame buffer for storage to complete the display (or image sending). Then, the DDIC 140 takes out the image data from the frame buffer and drives the display 130 for display.

The wireless communication function of the terminal device 300 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 300 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

The mobile communication module 150 can provide solutions for wireless communications including 2G/3G/4G/5G applied to the terminal device 300. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display 130. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) and the like applied to the terminal device 300. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of the signal, amplify the signal, and convert it into electromagnetic waves for radiation through the antenna 2.

In some embodiments, the antenna 1 of the terminal device 300 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 300 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and/or a satellite based augmentation system (SBAS).

The terminal device 300 can realize the shooting function through ISP, camera 193, video codec, GPU, display 130 and application processor.

ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 300 may include 1 or N cameras 193, where N is a positive integer greater than 1.

Video codecs are used to compress or decompress digital videos. The terminal device 300 may support one or more video codecs. Thus, the terminal device 300 may play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The terminal device 300 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 300 can listen to music or listen to a hands-free call through the speaker 170A.

The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 300 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 300 can be provided with at least one microphone 170C. In other embodiments, the terminal device 300 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 300 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

The earphone interface 170D is used to connect a wired earphone. The earphone interface 170D may be a USB interface, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

The button 190 includes a power button, a volume button, etc. The button 190 can be a mechanical button or a touch button. The terminal device 300 can receive the button input and generate a key signal input related to the user settings and function control of the terminal device 300. The motor 191 can generate a vibration prompt. The SIM card interface 195 is used to connect the SIM card. The SIM card can be connected to and separated from the terminal device 300 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195.

The structure shown in FIG2 does not constitute a specific limitation on the terminal device 300. The terminal device 300 may include more or fewer parts than shown in the figure, or combine some parts, or split some parts, or arrange the parts differently. The various parts shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

FIG. 3 exemplarily shows a server 400 provided in an embodiment of the present application.

The server 400 can provide complex computing capabilities for executing the steps performed by the computing module in the picture editing method provided in the embodiment of the present application.

As shown in FIG. 3 , the server 400 may include: a processor 210 , a memory 220 , an input/output device 230 , a communication module 240 , etc. These components may be coupled via a bus.

The server 400 may have powerful computing resources, and the processor 210 thereon may include one or more processors with powerful computing power, such as a central processing unit (CPU), a neural network processing unit (NPU), a graphics processing unit (GPU), etc.

The processor 210 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input output (GPIO) interface, a subscriber identity module (SIM) interface, and/or a universal serial bus (USB) interface, etc.

The processor 210 may be provided with a cache memory, which may be used to store instructions or data just used or cyclically used by the processor 210. If the processor 210 needs to use the instruction or data again, it may be directly called from the cache memory, which may reduce the waiting time of the processor 210 and improve the program running efficiency.

The processor 210 may also be connected to an external memory. The memory may be a high-speed random access memory or a non-volatile memory, such as a disk, a flash memory, a universal flash memory (UFS), etc. The memory may also be an external memory card, such as a Micro SD card.

The processor 210 is the computing core of the server 400 and has powerful computing capabilities. It is coupled to the memory 220 and can be used to read and execute computer-readable instructions in the memory 220 and run the operating system and various programs. Specifically, the CPU 210 can be used to call a program stored in the memory 220, such as the implementation program of the picture editing method provided in the embodiment of the present application, and execute the instructions contained in the program.

The memory 220 may include a high-speed random access memory, a non-volatile memory, such as a disk, a flash memory or other non-volatile solid-state storage device. The memory 220 may be used to store various software programs and multiple sets of instructions. The memory 220 may store an operating system, such as an operating system such as Linux. The memory 220 may also store one or more programs, such as programs involved in patch production, such as a compiler and a linker. The memory 220 may also store code instructions of the image editing method provided in the embodiment of the present application. When the processor 210 reads the code instructions from the memory 220 and runs the code instructions, the server 400 may execute the steps performed by the computing module in the image editing method provided in the embodiment of the present application.

The input and output devices 230 may include a display screen, a keyboard, a mouse and other devices, which can be used to receive user input and output program execution results to the user.

The communication module 240 may include a wired communication module and a wireless communication module. Among them, the wired communication module may support wired communication protocols, such as universal serial bus (USB), serial port, Ethernet and other protocols, and communicate with other devices through physical communication cables. The wireless communication module may include 2G/3G/4G/5G and other wireless communication modules, Wi-Fi communication modules, etc. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the CPU 210; the wireless communication module may also receive the signal to be sent from the CPU 210, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna.

The structure shown in FIG3 does not constitute a limitation on the server 400. The server 400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

Based on the various products introduced in the aforementioned embodiments, the picture editing method provided by the embodiments of the present application will be described in detail through four embodiments below. These four embodiments may be referred to as Embodiment 1, Embodiment 2, Embodiment 3 and Embodiment 4, respectively. As shown in Figure 4, Embodiment 1 introduces the overall process of the picture editing method provided by the present application, and Embodiment 2 is an alternative to Embodiment 1. Embodiments 1 and 2 introduce two user operation modes, both of which can predict the user's editing intentions and give recommended editing operations. Embodiment 3 introduces how to judge whether the editing operation input by the user is reasonable. Embodiment 4 solves the problem of how to insert the element into the middle of an existing object in the original image when adding or replacing an element. Embodiments 1 and 2 are in a parallel relationship, and Embodiments 3 and 4 are supplements to Embodiments 1 and 2, and are an expanded introduction to certain steps in the overall process.

Embodiment 1

Embodiment 1 introduces the overall process of a user editing a picture through a terminal device. When the terminal device has powerful computing power, complex image processing can be completed directly on the computing module of the terminal device. When the terminal device does not have powerful computing power, the terminal device can submit complex computing tasks (such as complex image processing tasks) to the cloud through the network for execution and wait for the task execution results returned by the cloud server. In the following method embodiments, the technical solution will be described with the human-computer interaction module and the computing module as the execution subject. Then, when the terminal device has powerful computing power, the human-computer interaction module and the computing module can both be located on the terminal device, and the steps executed by the two are the steps executed by the terminal device, and the communication or data interaction between the two belongs to intra-device communication; and when the terminal device does not have powerful computing power, the human-computer interaction module can be located on the terminal device, and the computing module can be located on the server. The steps executed by the human-computer interaction module are the steps executed by the terminal device, and the steps executed by the computing module are the steps executed by the server, and the communication or data interaction between the two belongs to inter-device communication.

As shown in FIG5 , the overall process of the picture editing method provided in the first embodiment may include:

S10-S13: Open the first picture.

Specifically: as described in S10, the human-computer interaction module may detect a user operation of opening the first picture. In response thereto, as described in S11-S12, the human-computer interaction module may obtain the first picture from the storage module. As described in S13, after obtaining the first picture, the human-computer interaction module may display the first picture.

The terminal device may be installed with an application for browsing, managing or processing pictures, such as a photo album (also called a gallery), a photo editing application, a drawing design program, etc. The user may open the first picture through the application.

In the embodiment of the present application, the user operation of opening the first picture can be referred to as the first user operation.

As shown in FIG6 , the first user operation may be, for example, an operation of a user clicking on a thumbnail 61 of the first picture in the album, and in response to the operation, the terminal device displays the original picture 63 of the first picture. Relative to the thumbnail, the original picture may also be called a large picture. Not limited to what is shown in FIG6 , the user operation of opening the first picture may also be a user operation of clicking on a thumbnail in a folder, a user operation of clicking on a thumbnail in a web page, a user operation of clicking on a thumbnail in a chat interface, and so on. The first picture user operation may also be a user operation of clicking on a link of the first picture, a user operation of clicking on a file icon of the first picture, and so on.

The first picture can be stored in a local storage module of the terminal device, or in a network storage module. When the first picture is stored on the network, the human-computer interaction module can download the first picture from the network storage module.

S14-S18: Identify the editing area.

Specifically, as described in S14, the human-computer interaction module can detect the operation of the user selecting the editing area in the first picture. In response to this, as described in S15, the human-computer interaction module can transmit the operation position of the user's operation on the first picture to the calculation module. Then, as described in S16, the calculation module can determine which area in the picture the editing area selected by the user is based on the operation position, and inform the human-computer interaction module of the contour information of the editing area selected by the user as described in S17. In this way, as described in S18, the human-computer interaction module can distinguish and display the editing area in the first picture based on the contour information of the editing area selected by the user. Not limited to the contour information of the editing area, the information returned by the calculation module to the human-computer interaction module in S17 can also be a binary image, a grayscale image, etc. of the editing area, which, like the contour information, can uniquely indicate the editing area in the first picture. In the embodiment of the present application, this information is collectively referred to as the indication information of the editing area.

Here, "distinguishing" means distinguishing the editing area from other areas in the first image, and its implementation means may include but are not limited to: highlighting the outline of the editing area, highlighting the entire editing area, or displaying a dotted frame along the outline of the editing area, etc.

In the embodiment of the present application, the operation of the user selecting the editing area in the first picture can be called the second user operation, and the editing area selected by the user can be called the first editing area.

The second user operation may be an operation of selecting an object in the first picture, such as clicking or long pressing an object, and the first editing area may be the image area where the object selected by the user is located. For example, as shown in FIG7 , in the first picture 71 , the user clicks on the object “sky”, and the first editing area is the image area of “sky”. In response to this, the human-computer interaction module may display a dotted frame on the outline of “sky” to distinguish “sky” from other areas as the first editing area.

The first editing area can be specifically identified by the computing module according to the action position of the second user operation on the first picture. Specifically, the human-computer interaction device can send the action position of the second user operation on the screen to the computing module, and the computing module identifies the first editing area according to the action position. Among them, the identification of the first editing area can be based on the existing image segmentation technology. Image segmentation technology can be used to identify and divide each object instance in the image, and mark the corresponding object category label for each pixel in the image. After using the image segmentation technology to identify and divide each object in the picture, the computing module can determine which object in the picture the user has selected according to the action position of the second operation on the picture, and determine the image area where the object is located as the first editing area. The embodiment of the present application is based on the understanding of the semantic content of the picture, and realizes the prediction of the scope of the editing area selected by the user according to the user operation. In this way, it can support users to simply and efficiently select the editing area in the picture to be edited, without the need for users to achieve this purpose through a series of complex operations such as lasso, cutout, and separation of layers, which not only reduces the complexity of use, but also improves editing efficiency.

One existing image segmentation technique is, for example, the segment anything model (SAM) technique, which is a deep learning model for image segmentation tasks. SAM uses a convolutional neural network (CNN) combined with a Transformer-based architecture to process images in a hierarchical and multi-scale manner. The working principle of SAM can be summarized as follows: SAM uses a pre-trained Vision Transformer (VIT) as its backbone network. The backbone network is used to extract features from the input image. SAM uses a feature pyramid network (FPN) to generate feature maps at multiple scales. FPN is a series of convolutional layers that operate at different scales to extract features from the output of the backbone network. FPN ensures that SAM can identify objects and boundaries at different levels of detail. SAM uses a decoder network to generate a segmentation mask for the input image. The decoder network takes the output of the FPN and upsamples it to the original image size. The upsampling process enables the model to generate a segmentation mask with the same resolution as the input image. SAM also uses a Transformer-based architecture to improve the segmentation results. Transformer is a neural network architecture that processes sequential data, such as text or images, very effectively. Use Transformer-based architectures to improve segmentation results by obtaining contextual information from the input image. SAM uses self-supervised learning to learn from unlabeled data. This involves training the model on a large dataset of unlabeled images to learn common patterns and features in the images. The learned features can be used to improve the performance of the model on specific image segmentation tasks. SAM can perform panoptic segmentation, which involves combining instance and semantic segmentation. Instance segmentation involves identifying and classifying each object instance within an image, while semantic segmentation involves labeling each pixel in the image with a corresponding class label. Panoptic segmentation combines these two methods to provide a more comprehensive understanding of the image.

Another existing image segmentation technology is the segment everything everywhere all at once (SEEM) technology. In terms of model architecture, the SEEM model adopts a common encoder-decoder architecture. It can not only achieve image segmentation, but also support multimodal input, and achieve one-click segmentation of any category of objects in any image through different types of visual prompts and text prompts. Among them, the visual prompt can be a point, a box, a randomly drawn graffiti, a mask, or a reference area of another image selected by the user on the image; and the text prompt can be the type (class) that the user needs to segment or a sentence (sentence) that represents the task. In other words, for an image, the user can click a point, draw a box, or paint a few times in the form of hand-drawing to complete the segmentation of the corresponding object. In addition, the user can also tell the model the name of the object to be segmented in the form of this article or describe it in a sentence, which can also complete the one-click segmentation.

The second user operation is not limited to directly selecting an object, and can also be a user operation of drawing an edit area in the first image. For example, as shown in FIG8 , in the first image 81 , the user draws a heart outline 82 , and the image area inside the heart outline is the edit area selected by the user. The second user operation is not limited to drawing an outline, and can also be an operation of painting out an edit area, etc.

S19: receiving an editing instruction input by the user.

Specifically, the human-computer interaction module may detect that the user inputs a first editing instruction for the first editing area.

In the embodiment of the present application, after the editing area is identified and before S19, the human-computer interaction module may also prompt the user to input an editing instruction.

The methods for prompting the user to input editing instructions may include but are not limited to the following methods:

Method 1. As shown in FIG. 9 , the human-computer interaction module pops up an input box 121 for the user to enter an editing instruction, such as a text or voice instruction, in the input box. For example, the user can enter the following text or voice instruction in the input box: "Add 'flying geese'". In the editing instruction of this example, "add" is the editing type, and "flying geese" is the editing parameter. FIG. 9 is only an example, and the embodiment of the present application does not limit the appearance style, display position, etc. of the input box 121.

When the user is prompted to input an editing instruction in method 1, the first editing instruction may be a text instruction input by the user in an input box or a voice instruction input by pressing a voice key.

Mode 2: The human-computer interaction module may display one or more recommended editing instructions, which are generated by the calculation module according to the image features of the first editing area.

For example, as shown in FIG10 , the editing area selected by the user is a heart-shaped area 131 in the sky, and the recommended editing instruction generated for the editing area is “recommended adding new screen content ‘soaring geese’”, where “add” is the editing type and “soaring geese” is the editing parameter.

For another example, as shown in FIG11 , the editing area selected by the user is the sky area, and the recommended editing instruction generated for the editing area is “Recommended adjustment of hue: reduce brightness to 82%, saturation to 75%, and hue to red”, where “adjust hue” is the editing type, and “reduce brightness to 82%, saturation to 75%, and hue to red” are the editing parameters.

Compared with the prior art method that requires users to independently determine the editing type and editing parameters, the embodiment of the present application is based on the understanding of the semantic content of the original image, and recommends editing types and editing parameters to users according to the image features of the editing area selected by the user, thereby providing users with editing ideas. This not only reduces the complexity of use, but also ensures the rationality of the content of the edited image, avoids users from experiencing a lot of trial and error, and significantly improves the efficiency of image output.

The image features of the first editing area may include, but are not limited to: mask features, depth features, contour features, and color features. These features may be vector features extracted from the first editing area using various existing technologies.

Specifically, the calculation module can use an image segmentation algorithm (such as SAM, SEEM algorithm) to extract the mask features of the image. The mask features can carry the category information corresponding to each pixel in the image. Figure 12 exemplifies the matrix storage form of the mask features of a picture including the sky, castle, mountain, valley, forest and tree. The size of the matrix is the same as the size of the original image, and the coordinates of the matrix correspond one-to-one with the coordinates of the original image. The values in the matrix represent the category to which the pixels at the corresponding position of the original image belong, such as "0" for "sky", "1" for "forest", etc. The storage form of the mask features can be varied, as long as it can reflect the category information of each pixel, and the embodiments of the present application do not limit this. Regardless of the storage form, the mask features can be converted to obtain the visualization results shown in Figure 13.

Taking Figure 10 as an example, when the editing area selected by the user is the heart-shaped area 131, the calculation module can determine that the category of the pixels in the heart-shaped area 131 is "sky" based on the mask characteristics, and then determine the recommended editing type as "add" and the corresponding editing parameter as "flying geese" based on the specific recommendation rule. In this example, the specific recommendation rule includes: when the first editing area is the sky, the editing type is "add" and the editing parameter is "flying geese". In actual applications, the recommendation rules can be set to other, and the embodiments of the present application do not limit this.

The calculation module can also extract depth features using a depth estimation algorithm, extract contour features using an edge detection and contour extraction algorithm, extract color features using a primary color extraction algorithm (such as a Kmeans clustering algorithm), and so on. Taking the picture of the sky, castle, mountain, valley, forest, and tree as an example, the visualization form of the depth feature can be shown in FIG14, the visualization form of the contour feature can be shown in FIG15, and the visualization form of the color feature can be shown in FIG16. FIG16 visualizes the color moment of the picture and intuitively describes the color distribution in the picture through a second-order rectangular form. The storage form of the three features of depth feature, contour feature, and color feature can also be in a matrix form, similar to FIG12, except that the values in the matrix represent different meanings. The matrix value of the depth feature represents the depth value of each pixel, the matrix value of the contour feature represents whether each element is a contour, and the matrix value of the color feature represents the color of each element.

After extracting various features such as mask features, depth features, contour features, and color features of the first editing area, the calculation module can use weighted summation and other techniques to fuse these features to obtain a fused feature vector, and use the fused feature vector as one of the inputs of the specific artificial intelligence algorithm. In addition, the calculation module can also map some editing types into numbers (such as "add" is mapped to 1, and "delete" is mapped to 2) as the second input of the specific artificial intelligence algorithm. Finally, the calculation module can obtain the editing parameters corresponding to various editing types through the calculation of the specific artificial intelligence algorithm. For a certain editing type, if the artificial intelligence algorithm cannot output the editing parameters corresponding to it or the confidence level of the output corresponding editing parameters is too low (such as less than 60%), it can be determined that the editing type is not suitable for the editing area and is not recommended. Here, the specific artificial intelligence algorithm can be a credible model trained with a large number of samples. In its training sample set, each training sample can include an input sample and an output sample, wherein the input sample includes the fused feature vector of the image area and the mapping ID of the editing type, and the output sample includes the editing parameters used when the editing type is applied to the image area. The more reasonable the training sample, the larger the sample set, and the higher the reliability of the trained model.

In some embodiments, the computing module can also generate editing questions based on the image features of the editing area and the preset editing type. The answer to the editing question is how to match the editing parameters of the editing type, and then input it into a large model such as chatGPT to obtain the answer. For example, the editing question generated based on the editing area "sky" and the editing type "add" is "What to add to the sky?" The chatGPT model can output an answer, such as: "Add 'clouds' to the sky", where 'clouds' is the answer to the question by the large model. This example is only used to explain the embodiments of the present application. In actual applications, the questions posed to chatGPT can be more complex and carry more details, such as adding additional picture descriptions.

In some embodiments, the computing module may further traverse the editing instructions in the recommendation pool, and compare the feature vectors corresponding to the traversed editing instructions with the fused feature vectors of the editing area. For details, please refer to S52 in Example 3, to find the editing instructions that can be reasonably matched with the editing area, and finally recommend the editing instructions.

When the user is prompted to input an editing instruction using method 2, the first editing instruction input by the user may be selected from the recommended editing instructions generated by the calculation module, for example, the user clicks on a recommended editing instruction to confirm inputting the recommended editing instruction as the first editing instruction, or the user drags a recommended editing instruction to the first editing area to confirm inputting the recommended editing instruction as the first editing instruction. The example is only used to explain the embodiments of the present application, and may be different in actual applications. The embodiments of the present application do not limit the manner in which the user inputs the first editing instruction.

Mode 3: The human-computer interaction module may display one or more preset editing instructions, such as commonly used editing instructions or editing instructions saved in advance by the user.

When the user is prompted to enter an editing instruction using method 3, the first editing instruction entered by the user may be selected from preset editing instructions displayed by the human-computer interaction module, for example, the user clicks on a preset editing instruction to confirm that the recommended editing instruction is entered as the first editing instruction, or the user drags a preset editing instruction to the first editing area to confirm that the recommended editing instruction is entered as the first editing instruction. The examples are only used to explain the embodiments of the present application, and may be different in actual applications. The embodiments of the present application do not limit the manner in which the user enters the first editing instruction.

In the embodiment of the present application, the above-described methods of prompting the user to input the editing instruction can also be implemented in combination, for example, the human-computer interaction module pops up an input box to prompt the user to input the editing instruction as in method 1, displays the recommended editing instruction as in method 2, and displays the threshold editing as in method 3. The user can see prompts of multiple methods on the interface at the same time, and choose to input the editing instruction according to a certain prompt.

S20-S23: Edit pictures.

In response to the user inputting the first editing instruction for the first editing area, as described in S20, the human-computer interaction module can send the first editing instruction to the calculation module, and then, as described in S21, the calculation module can perform corresponding image editing processing on the first picture according to the first editing instruction, and return the first picture after the image editing processing to the human-computer interaction module as described in S22. In this way, as described in S23, the human-computer interaction module can display the first picture after the image editing processing.

Compared with the image before the image editing process, the image of the first editing area after the image editing process is changed, and the change is determined by the first editing instruction. The editing instruction may include the following content: the editing type and its corresponding editing parameters. Among them, the editing type may be, for example, deletion, dragging, replacement, addition, color adjustment, etc., and the corresponding editing parameters may be, for example, replaceable content, drag target position, newly added content, color adjustment value, etc.

For example, if the editing area selected by the user is "sky" and the first editing instruction is "delete 'dark clouds'", then compared with the image before editing, the "dark clouds" in the "sky" of the first picture after image editing are deleted. For another example, if the editing area selected by the user is "sky" and the first editing instruction is "add 'flying geese'", then compared with the image before editing, the "flying geese" in the "sky" of the first picture after image editing are added.

Assuming that the first picture is Picture 71 shown in Figure 7 and Picture 81 shown in Figure 8, Figure 17 exemplifies the first picture after editing when the first editing instruction is "add 'soaring geese'", and Figure 18 exemplifies the first picture after editing when the first editing instruction is "adjust the color tone: reduce the brightness to 82%, the saturation to 75%, and the hue to red".

When the editing type in the first editing instruction is "add", the computing module can also obtain the image content to be added to the first picture according to the editing parameters in the first editing instruction, such as "soaring geese", and add the image content to the first picture to complete the image editing process corresponding to the first editing instruction (such as "add 'soaring geese'"). The image content can be carried in a material picture, which can come from a network, a terminal device, or the material picture can be generated by the computing module using artificial intelligence. Figure 19 exemplifies the material picture obtained according to the editing parameters ("soaring geese") of the editing instruction "add 'soaring geese'".

Embodiment 2

Embodiment 2 is an alternative to Embodiment 1 and also introduces the overall process of the image editing method. However, in Embodiment 2, the editing area can be determined based on the preprocessing information of the image instead of image segmentation technology, and repeated online calculations are not required.

As shown in FIG. 20 , the overall process of the picture editing method provided in the second embodiment may include:

S30-S33: Open the first picture.

For details, please refer to S10-S13 in the first embodiment, which will not be described in detail here.

S34-S39: Identify the editing area.

Specifically, as described in S34, the human-computer interaction module can detect the operation of the user selecting the editing area in the first picture, such as clicking on an object in the picture. In response to this, as described in S35, the human-computer interaction module can trigger the calculation module to obtain the preprocessing information of the first picture; in addition, as described in S36, the human-computer interaction module can also transmit the operation position of the user's operation on the first picture to the calculation module. In this way, as described in S37, the calculation module can determine which area of the picture the editing area selected by the user is based on the preprocessing information, or further combined with the operation position, and inform the human-computer interaction module of the indication information (such as contour information, binary image, grayscale image) of the editing area selected by the user as described in S38. Then, as described in S39, the human-computer interaction module can distinguish and display the editing area in the first picture according to the contour information of the editing area selected by the user.

For some implementation details of "identifying the editing area", please refer to S14-S18 in Example 1 and its related instructions. For example, the user's operation of selecting the editing area in the first picture can be an operation of clicking or long pressing an object in the first picture. As shown in Figure 7, in the first picture 71, the user selects "sky" as the editing area by clicking the object "sky". For another example, the way to distinguish and display the editing area in the first picture may include but is not limited to: highlighting the outline of the editing area, highlighting the entire editing area, or displaying a dotted box along the outline of the editing area, etc.

In the second embodiment, the preprocessing information may be used to indicate one or more regions. Here, the one or more regions refer to image regions in the first picture, and the image regions in the first picture may be divided into units of objects, and all pixels of an object constitute an image region. The editing region may be determined from the one or more regions. The one or more regions may be regions that the user prefers to edit, or regions that are recommended to edit, or regions that are allowed to edit, and so on. The editing region may be quickly determined based on the preprocessing information without performing image segmentation processing on the first picture.

After the human-computer interaction module detects the user's operation of selecting the editing area in the first picture, the calculation module may first read the preprocessing information:

1. If the pre-processing information only includes indication information of one area, the area is determined as the editing area.

The preprocessing information may only contain the coordinates of the contour points of a region. As shown in FIG21 , the preprocessing information may be in the form of an array in Json format, where an array represents the contour of a region, and each element in the array is a tuple, where the first value of the tuple represents the x-coordinate value of a contour point of the region, and the second value of the tuple represents the y-coordinate value of the contour point. Therefore, the region enclosed by the contour recorded in the array can be determined, and the region can be determined as the editing region.

The preprocessing information is not limited to the Jason format and may also adopt other data formats.

Not limited to the coordinates of the contour points of the region, the data content in the preprocessing information can also be a binary image, a grayscale image, etc.

The preprocessing information may also be a binary image containing only one region. In the binary image, only one region has a first value. The binary image may be exemplified as shown in FIG22, wherein one element corresponds to one pixel, the coordinates of the element may represent the position of the pixel, the value of the element may represent the pixel category of the pixel, and is binarized, such as "1" may represent that the pixel at this position belongs to the "sky", and "0" may represent that the pixel at this position does not belong to the "sky". Based on the binary image shown in FIG22, an area with a value of "1" ("sky" area, the first value is "1") or an area with a value of "0" (non-"sky" area, the first value is "0") may be determined as an editing area.

The preprocessing information may be a grayscale image, in which only one region may have a grayscale value of a specific grayscale value or be within a specific grayscale range. Therefore, the only region may be determined as the editing region based on the grayscale image.

2. If the preprocessing information includes indication information of multiple areas, the editing area can be further determined in combination with the action position of the second user operation on the first picture. Specifically, it can be determined which area of the picture the action position is located in, or which area is closer to, and the area is determined as the editing area. In other words, among the multiple areas indicated by the preprocessing information, the area where the action position is located can be determined as the editing area, or the area closest to the action position among the multiple areas can be determined as the area selected by the user. In addition, the area selected by the user can be determined as the editing area.

Furthermore, after determining the area selected by the user, it is also possible to determine whether there are areas in the remaining areas that overlap more with the selected area. If so, the selected area and the area that overlaps more with it are merged into one area, and the merged area is finally determined to be the editing area. In some embodiments, a non-maximum suppression (NMS) algorithm may be used to merge multiple overlapping areas. In the embodiment of the present application, more overlap may mean that the area of the overlapping part exceeds a specific value, such as 10 pixels (px).

In order to directly indicate multiple regions, the preprocessing information may include the coordinates of the contour points of the multiple regions, wherein the coordinates of the contour of each region may be expressed as an array. The preprocessing information may also be a binary image including multiple regions, wherein the values of multiple regions in the binary image are the first value (such as "1"). For example, "1" indicates that the pixel at this position belongs to "flower", and "0" indicates that the pixel at this position does not belong to "flower". The preprocessing information may also be a grayscale image, wherein the grayscale values of multiple regions in the grayscale image are specific grayscale values or are in a specific grayscale range, for example, there are multiple regions whose grayscale values are in the grayscale range of 0-50.

The preprocessing information is not limited to the coordinates, binary images, and grayscale images of the contour points of the region and can be used to directly indicate multiple regions. The preprocessing information may also have other forms of data content, which is not limited in the embodiments of the present application.

3. The preprocessing information of the first image may not directly indicate one or more regions, but may include other data, such as multi-layer information, contour information, or depth information. However, the calculation module may use the other data to determine one or more regions, and then determine the editing region from the one or more regions based on the method described in 1 or 2 above. That is, the preprocessing information of the first image may indirectly indicate one or more regions.

Methods for determining one or more regions indirectly indicated by the pre-processing information may include, but are not limited to:

3.1 If the preprocessing information includes the layer information of multiple layers, the layer information of each layer may include the coordinates of the opaque pixels in the layer, then for each layer in the layer information, the opaque pixels in the layer that are connected together can be regarded as a region. Furthermore, the layer regions with more overlaps can be merged into one region, for example, multiple overlapping layer regions are merged using the NMS algorithm, and the edited region is determined using the merged one or more regions.

3.2 If the preprocessing information contains contour information (such as Figure 18), the contour information can record the pixels on the contour, then the useless internal contours in the contour information can be filtered out using dilation and erosion techniques, and one or more regions of the contour information that are completely closed can be obtained based on graph theory and other techniques. Among them, dilation and erosion can be used to eliminate noise, segment independent image elements, and connect adjacent elements; graph theory and other techniques can be used for connected domain analysis, which refers to finding independent connected domains in an image and marking them out. The connected domain of an image refers to the region composed of pixels with the same pixel value and adjacent positions in the image. Generally, a connected domain contains only one pixel value. Therefore, in order to prevent the influence of pixel value fluctuations on the extraction of different connected domains, connected domain analysis often processes binarized images.

Still taking the picture including the sky, castle, mountain, valley, forest and tree as an example, FIG23 exemplarily shows a plurality of regions obtained based on the contour information of the picture.

3.3 If the preprocessing information includes depth information (as shown in FIG. 19 ), the depth information may record the depth values of multiple pixels, and the pixels may be divided into different regions according to the depth value of each pixel in the depth information. Specifically, pixels with the same or similar depth values may be divided into the same region.

Still taking the picture including the sky, castle, mountain, valley, forest and tree as an example, FIG24 exemplarily shows a plurality of areas obtained based on the depth information of the picture, wherein the "castle" area 241 may be composed of some pixel points with the same or similar depth values, and the "tower" area 242 may be composed of some pixel points with the same or similar depth values.

In the embodiment of the present application, similar depth values may mean that the difference in depth values does not exceed a specific value, such as 0.1. In a picture, the depth value of each pixel can be expressed in the range of 0 to 1, where 0 represents the depth value of the pixel farthest from the camera that took the first picture, and 1 represents the depth value of the pixel closest to the camera that took the first picture. Without limitation to this, the depth value may also be expressed in other ways, which are not limited in the embodiment of the present application.

The preprocessing information may include multiple contents introduced in 3.1 to 3.3 above at the same time. For example, the preprocessing information may include layer information, depth information, and contour information at the same time. After the areas indicated by the preprocessing information are determined based on these three contents, they can be calibrated with each other to improve the accuracy of recognition. For example, if both the layer information and the depth information indicate the same area, the recognition of the area is often accurate; conversely, if there is a conflict between the areas indicated by the layer information and the depth information, it means that the area recognition based on the depth information or the layer information is inaccurate, and the algorithm can be optimized and re-recognized.

It can be seen that embodiment 2 can determine the editing area from one or more areas directly or indirectly indicated by the pre-processing information, without first using image segmentation processing to identify and divide the image area where each object in the first picture is located and then identify the editing area according to the user's operation position. Therefore, the area that the user wants to edit can be predicted more quickly, and image segmentation calculations can be avoided.

S40: receiving an editing instruction input by a user.

For details, please refer to S19 in Example 1, which will not be described in detail here.

S41-S44: Edit pictures.

For details, please refer to S20-S23 in Example 1, which will not be described in detail here.

Embodiment 3

Embodiment 3 is a supplementary refinement of Embodiments 1 and 2. Before executing the "Edit Picture" step, a step of judging whether the editing instruction input by the user and the editing area selected by the user are reasonably matched (see S41b in FIG. 25) can be added. When the editing area selected by the user and the input editing instruction do not match, not only will the user be prompted that the editing instruction cannot be executed, but the user can also be recommended to select a new editing area, or the user can be helped to modify the editing instruction. Embodiment 3 solves the problem that the editing effect caused by the user's arbitrary input of editing instructions does not conform to common sense logic, thereby reducing the number of trial and error times when the user edits the picture.

Specifically, as shown in FIG26 , the specific implementation of determining whether the first editing instruction and the editing area are reasonably matched may include the following process:

S50. Calculate and obtain the feature vector of the editing parameters and editing type in the first editing instruction.

The editing types in the first editing instruction may be finite and enumerable. Different editing types may be mapped to different identification numbers (IDs) and may thus be represented by corresponding IDs. The computing module may convert the IDs corresponding to the editing types using models such as deep learning algorithms to obtain feature vectors corresponding to the editing types. For example, the ID corresponding to "increase" is "001", which is converted into the following feature vector: [0.5250, 0.7937, 0.1356, 1.4893, -3.9651, 1.5068].

The editing parameters in the first editing instruction (such as "quiet lake") can also be mapped to an ID first and then converted into a feature vector. The difference is that the text content of the editing parameters is unpredictable and not easy to be directly mapped to an ID. In this embodiment, the calculation module can perform word segmentation on the phrase representing the editing parameter, and then find the ID corresponding to each word segmentation result from the preset word list. For example, the editing parameter "quiet lake" can be segmented into: ["quiet", "quiet", "of", "lake", "park"], and the ID array corresponding to the word segmentation result is: [40496,3152,2099,8024,3563,8024,40497,0,0,0]. Among them, 40496 and 40497 respectively represent the start and end of the description phrase of the editing parameter, and 0 represents supplement. The length of the ID array, that is, the number of elements it contains, can be preset to constrain the maximum length of the description phrase of the editing parameter. Then, the calculation module can generate a corresponding feature vector for each ID in the ID array through a model such as a deep learning algorithm. For example, the ID array in the previous example can be converted into the following array of feature vectors: [[0.8838,0.1570,0.5249,...,0.4278,0.1725,0.4225],[1.8143,-0.5514,0.0995,...,-4.7141,-1.3811,-1.1166],[0.0186,3.5949,1.1780,...,1.1433,2.7235,-0.5069],...,[1.1674,0.9497,1.8264,...,1.3671,0.5551,-0.4302] .5551,-0.4302]], where [0.8838,0.1570,0.5249,...,0.4278,0.1725,0.4225] represents the eigenvector of 40496, [1.8143,-0.5514,0.0995,...,-4.7141,-1.3811,-1.1166] represents the eigenvector of 3152, [0.0186,3.5949,1.1780,...,1.1433,2.7235,-0.5069] represents the eigenvector of 2099, ..., [1.1674,0.9497,1.8264,...,1.3671,0.5551,-0.4302] represents the eigenvector of 0. The feature vector is a two-dimensional array, in which each element is a feature vector corresponding to an ID.

The deep learning algorithm model may be, for example, a word vector (Word2Vec) model, a Transformers model, etc.

S51. Extract various features such as mask features, depth features, contour features, color features, etc. of the editing area selected by the user, and fuse these features using weighted summation and other techniques to obtain a fused feature vector.

The fused feature vector corresponding to the edited area can be expressed as a two-dimensional array. For example, the fused feature vector of the "sky" area is: [[0.2611, 1.8726, ..., -0.9721], ..., [1.6888, 2.6287, ..., -5.5910]].

The feature vector corresponding to a feature (such as a deep feature) can be calculated by a deep learning algorithm model, and the deep learning algorithm model can be, for example, a convolutional neural network model (CNN). The fusion of multiple features can be achieved through multiple CNN models and weighted summation. For example, the mask feature shown in Figure 13 is obtained by CNN model 1, the deep feature shown in Figure 14 is obtained by CNN model 2, and the contour feature shown in Figure 15 is obtained by CNN model 3, etc. The dimensions of the feature vectors of these features are the same length, so these feature vectors can be obtained by weighted summation to obtain a fused feature vector.

S52. Use models such as machine learning or deep learning algorithms to compare the feature vector corresponding to the first editing instruction with the fused feature vector of the editing area selected by the user, and determine whether it is reasonable to apply the first editing instruction to the editing area selected by the user.

Specifically, the calculation module can use the feature vector corresponding to the editing type in the first editing instruction as one of the inputs of models such as machine learning or deep learning algorithms, use the feature vector corresponding to the editing parameters in the first editing instruction as the second input of models such as machine learning or deep learning algorithms, and use the fusion feature vector of the editing area as the third input of models such as machine learning or deep learning algorithms. Finally, the calculation module can obtain a judgment result on whether it is reasonable to apply the first editing instruction to the editing area selected by the user through calculations by models such as machine learning or deep learning algorithms.

When it is determined that the first editing instruction is not reasonable to be applied to the editing area selected by the user, the human-computer interaction module can output an error prompt, which can be a visual prompt displayed on the screen, a vibration prompt perceptible by touch, a voice prompt, and so on.

For example, as shown in FIG. 27 , when the editing area selected by the user is “sky” and the editing instruction input is “add ‘quiet lake’”, the human-computer interaction module may display an error prompt 311 to remind the user that adding “quiet lake” to “sky” is not in line with common sense and may not allow the execution of the editing instruction.

For another example, for picture 71 shown in Figure 7, when the editing area selected by the user is "sky" and the editing instruction input is "delete 'sun'", the calculation module can determine that the editing instruction is applicable to the "sky", and if the editing instruction becomes "delete 'mountain'", the calculation module can determine that the new editing instruction is not applicable to the "sky".

For another example, for picture 71 shown in Figure 7, when the editing area selected by the user is "sun" and the editing instruction input is "replace with 'clouds'", the calculation module can determine that the editing instruction is applicable to the editing area, and if the editing instruction becomes "replace with 'trees'", the calculation module can determine that the new editing instruction is not applicable to the editing area.

For another example, for picture 71 shown in Figure 7, when the editing area selected by the user is "tree" and the editing instruction input is "drag to 'forest'", the calculation module can determine that the editing instruction is applicable to the editing area, and if the editing instruction becomes "drag to 'lake'", the calculation module can determine that the new editing instruction is not applicable to the editing area.

The above examples are only used to explain the embodiments of the present application and should not be construed as limiting.

The error prompt 311 shown in FIG. 27 is also only an example. In actual applications, the error prompt may also be in other styles, such as flashing erroneous editing instructions or flashing editing areas. The embodiments of the present application do not limit this.

S53. When it is determined that the first editing instruction is not reasonable to be applied to the editing area selected by the user, re-recommend the editing instruction and editing area.

Specifically, other areas in the first image can be recommended to the user as editing areas based on the editing type and feature vectors corresponding to the editing parameters in the first editing instruction. The calculation module can traverse other areas in the first image, and use the method in S52 to compare the fused feature vectors of other areas with the feature vectors corresponding to the first editing instruction to find an area that can reasonably match the first editing instruction, and then recommend the found area as the editing area. Here, other areas refer to areas outside the editing area selected by the user in the first image.

Specifically, the first editing instruction can be modified according to the editing area selected by the user, such as modifying the editing type and/or editing parameters, so that the modified editing instruction adapts to the editing area selected by the user. Among them, the other area refers to the area outside the editing area selected by the user in the first picture. The calculation module can traverse the editing types and/or editing parameters in the recommendation pool, and use the method in S52 to compare the feature vectors corresponding to the traversed editing types and editing parameters with the fused feature vector of the editing area selected by the user to find the editing type and/or editing parameters that can reasonably match the area selected by the user, and then modify the editing instruction, and finally recommend the modified editing instruction.

For example, as shown in Figure 28, when the editing area selected by the user is "sky" and the editing instruction input is "add 'quiet lake'", the calculation module can determine that: the editing instruction is not applicable to the "sky" area 312, and modify the editing area, using the "forest" area 313 as the new editing area, and finally recommends the user to apply the editing instruction to the "forest" area 313.

For another example, as shown in FIG29 , when the editing area selected by the user is “sky” and the editing instruction input is “add ‘quiet lake’”, the calculation module can determine that: the editing instruction is not applicable to the “sky”, and modify the editing instruction to “add ‘soaring geese’” as a new editing instruction, and finally suggest that the user apply the new editing instruction to the area “sky”.

Embodiment 4

Example 4 is a supplementary refinement of the "editing picture" step in Examples 1 and 2, and introduces how to change the perspective relationship of objects in the original image (i.e., front-to-back relationship, depth of field relationship) so that when new content is added to the original image, the front objects in the original image are not obstructed, thereby ensuring that the perspective relationship between objects in the edited image is reasonable.

In the embodiment of the present application, the editing instructions input by the user for the editing area involve adding new objects, and the editing instructions involving adding new objects may include: adding, replacing, dragging and other types of editing instructions. Among them, replacing is equivalent to deleting an object from the original image and then adding another object; dragging is equivalent to deleting an object from a certain position in the original image and then adding the object to another position. In other words, the editing instructions involving adding new objects may mean that the editing processing corresponding to the editing instructions includes: adding new objects to the editing area.

In this embodiment, the step of "editing the picture" may specifically include: in response to the user inputting a first editing instruction for the editing area, the human-computer interaction module may send the first editing instruction to the computing module. Here, the first editing instruction involves adding a new object to the editing area, and its editing type may be, for example, adding, replacing, dragging, etc. Then, the computing module may perform corresponding image editing processing on the first picture according to the first editing instruction, and return the first picture after the image editing processing to the human-computer interaction module. In this way, the human-computer interaction module can display the first picture after the image editing processing.

As shown in FIG30 , the specific implementation of the image editing performed by the computing module may be as follows (S61-S62):

S61. Extract various image features of the new object image, such as depth features, mask features, contour features, color features, etc., extract various image features of the first image, and modify the various features of the first image in combination with the image features of the editing area and the editing parameters in the editing instructions.

Here, the new object is relative to the original object. The original object refers to the object in the editing area before the new object is added. The new object replaces the original object in the editing area. For the two editing types of adding and replacing, the human-computer interaction module can provide one or more images of the new object indicated by the editing parameters for the user to select the image of the new object. For the editing type of dragging, the new object is actually the object dragged by the user, and its image comes from the first picture, not from outside the first picture.

Modifications to various features are described below.

Correction of deep features

Step 1. Determine the perspective relationship, ie, the front-to-back relationship, between the new object and the original object in the first image according to the image features of the first image and the editing parameters in the first editing instruction, which is reflected in the data as the different depth values.

If the image of the new object is directly used to replace the image of the original object, the perspective relationship between the objects will be unreasonable, and the new object will not be reasonably integrated into the first picture, such as causing the original object in the editing area to be completely blocked by the new object. For example, as shown in Figure 31, if the new object "Quiet Lake" shown in Figure 32 directly replaces the original objects "Tree" and "Valley" in the editing area 317 of the picture 315, it will cause the "Quiet Lake" to block the original object "Tree", resulting in an unreasonable perspective relationship.

The embodiments of the present application may use artificial intelligence algorithms such as image semantic understanding to determine the perspective relationship between the new object and the original object, and based on this, correct the depth features of the image to present a reasonable perspective relationship between objects.

For example, if the first editing instruction for the editing area 317 of the picture 315 in Figure 31 is "add 'quiet lake'", and the image of the new object "quiet lake" is shown in Figure 32; then the computing module can determine based on the masking features of the picture 315: the original objects in the editing area 317 are "tree" and "valley", and the new object "quiet lake" will block the original objects "tree" and "valley". Then, the computing module can determine the front and back relationship between the new object "quiet lake" and the original objects "tree" and "valley" based on artificial intelligence algorithms such as image semantic understanding, such as: the "tree" is in front of the "quiet lake", and the "quiet lake" is in front of the "valley".

Step 2. According to the perspective relationship between the new object and the original object, and the depth value of the original object, the reference depth of the new object is determined, and then the depth feature of the new object is corrected using the reference depth.

First, the reference depth of the new object can be determined by interpolation or other methods. In the above example, assuming that the average depth values of the original objects "tree" and "valley" are 0.6 and 1.0 respectively, then, based on the front-to-back relationship between the new object "quiet lake" and the original objects "tree" and "valley", the reference depth of the "quiet lake" can be determined to be 0.8 through interpolation algorithms, which is greater than 0.6 and less than 1.0. Here, the further back the object is, the greater its depth. In this way, the perspective relationship of the new object in the overall picture of the first image can be made reasonable.

Secondly, the depth feature of the new object is corrected by using the baseline depth of the new object and the depth difference between various regions on the new object.

After the depth feature of the new object is corrected, the average depth of the new object can be close to or equal to the reference depth, and the depth difference between each area remains unchanged. Here, closeness can mean that the difference between the average depth and the reference depth does not exceed a specific depth value, such as 0.05. In this way, it can be ensured that the new object as a whole forms a reasonable perspective relationship with the original object, and the perspective relationship between each element in the new object can be preserved.

Step 3: Replace the original depth features of the edited area with the corrected depth features of the new object to correct the depth features of the first image.

Here, the essence of replacement can be: traverse each pixel area in the editing area, if the new object is in front of the original object in the pixel area, then use the depth data of the pixel area on the new object to replace the original depth data of the pixel area, so as to realize the perspective relationship of the new object in front of the original object; if the original object in the pixel area is in front of the new object, then keep the original depth data of the pixel area, so as to realize the perspective relationship of the original object in front of the new object. The original depth data of a pixel area refers to the depth data of the pixel area on the original object.

The editing area can be divided into a first pixel area and a second pixel area, wherein the perspective relationship of the first pixel area is that the new object is in front of the original object, and the perspective relationship of the second pixel area is that the original object is in front of the new object. The depth features of the first pixel area can be replaced with the depth features of the new object, and the depth features of the second pixel area can maintain the depth features of the original object. In practical applications, the new object may also be entirely in front of the original object, excluding the second pixel area. In this case, the depth features of the entire editing area can be directly replaced with the depth features of the new object.

Therefore, in actual applications, only some pixel areas in the editing area may have depth data replaced, and the perspective relationship of these pixel areas is: the new object is in front of the original object; while other pixel areas may keep the original depth data, and the perspective relationship of these other pixel areas is: the original object is in front of the new object. For example, as shown in Figure 33, in pixel area 319, assuming that the average depth value of the original object "tree" is 0.6, and the reference depth of the new object "quiet lake" is 0.8, it can be seen that in pixel area 319, the original object "tree" is in front of the new object "quiet lake". Therefore, pixel area 319 keeps the depth data, which can ensure that: in area 319, the "tree" in front will not be blocked by the "quiet lake" behind, forming a reasonable perspective relationship.

FIG. 34 exemplarily shows a reasonable perspective relationship formed between the new object “quiet lake” and the original objects “tree” and “valley” through correction of depth features.

Modification of mask features

After the correction of the depth feature, the correction of the mask feature may not directly use the new object mask data to replace the original mask data of the edited area, but only use the new object mask data to replace the original mask data in the area where the depth data has been replaced. In this way, it can ensure that the correction of the mask feature is consistent with the correction of the depth feature, so that the corrected mask feature and depth feature both point to the same object, avoiding conflicts between the two, and ensuring that the semantics of the regenerated image are reasonable.

For example, in FIG33 , the depth data of pixel area 319 in the editing area 317 is not replaced, that is, the depth data of the original object "tree" is still used, so that the original object "tree" is in front of the new object "quiet lake" in the perspective relationship. In this regard, when correcting the mask feature, in area 319, the mask data of the new object "quiet lake" (such as "8") is not used to replace the original mask data of area 319 (such as "5", that is, the mask data of "tree"), but the mask data of the original object "tree" is retained; otherwise, it will lead to a contradiction, that is: from the perspective of the depth feature, area 319 is the original object "tree", but from the perspective of the mask feature, area 319 is indeed the new object "quiet lake".

In other words, within the editing area, if the depth data of a pixel area belongs to the depth data of the new object, and is no longer the original depth data of the pixel area, the mask data of the pixel area on the new object can be used to replace the original mask data of the pixel area. The original mask data of a pixel area refers to the mask data of the pixel area on the original object. Therefore, in actual applications, the area where the mask feature is replaced may only be part of the pixel area in the editing area, and the depth features of this part of the pixel area are replaced with the depth features of the new object.

In the example shown in FIG34 , the area where the mask feature replacement occurs is smaller than the area of the new object “Quiet Lake”. FIG35 simply shows a comparison between the two.

Correction of contour features

Similar to the correction of the mask feature, after the correction of the depth feature, the correction of the contour feature may not directly use the contour data of the new object to replace the original contour data of the edited area, but only use the contour data of the new object to replace the original contour data in the area where the depth data has been replaced. In this way, it can ensure that the correction of the contour feature is consistent with the correction of the depth feature, so that the corrected contour feature and depth feature both point to the same object, avoiding conflicts between the two, and ensuring that the semantics of the regenerated image are reasonable.

In other words, within the editing area, if the depth data of a pixel area belongs to the depth data of the new object, and is no longer the original depth data of the pixel area, the contour data of the pixel area on the new object can be used to replace the original contour data of the pixel area. The original contour data of a pixel area refers to the contour data of the pixel area on the original object.

Similarly, in the example shown in FIG. 34 , the area where the contour feature replacement occurs is smaller than the area of the new object “quiet lake”. FIG. 35 can also be used to illustrate the comparison between the two.

Not limited to mask features and contour features, the correction of other features (such as color features) is also considered, that is: in the editing area, if the depth data of a pixel area belongs to the depth data on the new object, and is no longer the original depth data of the pixel area, then the color features of the pixel area on the new object and other features can be used to replace the original color features and other features of the pixel area.

S62. Regenerate the image using the corrected depth features, mask features, contour features, etc.

Specifically, an artificial intelligence algorithm model can be used to regenerate images, for example, by combining a Stable Diffusion model with a ControlNet model. Stable Diffusion can be used to generate diffusion models for images based on text or images. In the process of generating images, more conditions (such as depth features, mask features, contour features, etc.) can be introduced through the ControlNet model to intervene in the image generation process.

The regenerated image can be shown in the right picture of Figure 34, which realizes the reasonable insertion of new objects (such as "quiet lake") between the original objects (such as "tree" and "valley") in the first image, presenting a reasonable perspective relationship between objects.

An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the steps performed by the human-computer interaction module in the above-mentioned method embodiments, or the steps performed by the human-computer interaction module and the computing module.

The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps performed by the computing module in the above-mentioned method embodiments can be implemented.

The embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps performed by the human-computer interaction module in the above-mentioned various method embodiments, or the steps performed by the human-computer interaction module and the computing module.

The embodiment of the present application also provides a computer program product. When the computer program product is run on a server, the server can implement the steps performed by the computing module in the above-mentioned various method embodiments.

The embodiment of the present application also provides a chip system, which includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps performed by the human-computer interaction module in any method embodiment of the present application, or the steps performed by the human-computer interaction module and the computing module. The chip system can be a single chip, or a chip module composed of multiple chips.

The embodiment of the present application also provides a chip system, which includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps performed by the computing module in any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

The term "user interface (UI), or interface for short) in the specification and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls (controls), also known as widgets, are basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes. For example, XML specifies the controls contained in the interface through nodes such as <Textview>, <ImgView>, and <VideoView>. A node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering. In addition, many applications, such as hybrid applications, usually include web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML defines the elements and attributes of a web page through <p>, <img>, <video>, and <canvas>.

The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims (34)

A method for editing a picture, comprising: Display the first picture; detecting a user operation on the first picture for selecting an editing area; Determining the editing area from the first picture according to the operation position of the user operation on the first picture, and distinguishably displaying the editing area in the first picture; generating a recommended editing instruction according to the image features of the editing area; displaying the recommended editing instructions; detecting that a user inputs a first editing instruction for the editing area, the first editing instruction comprising: a recommended editing instruction; Performing image editing processing on the first picture according to the first editing instruction; The first picture after the image editing process is displayed. The method according to claim 1, wherein generating the recommended editing instruction according to the image features of the editing area specifically comprises: Using a fusion feature vector of multiple image features of the editing area as one of the inputs of the first artificial intelligence algorithm; the multiple image features include multiple features of mask features, depth features, contour features, and color features; Using one or more preset editing types as second input to the first artificial intelligence algorithm; The recommended editing instruction is obtained by the operation of the first artificial intelligence algorithm, and the recommended editing instruction includes editing parameters corresponding to the preset editing type. The method according to claim 1 or 2 is characterized in that the preset editing type includes one or more of the following: deletion, dragging, replacement, addition, or color adjustment. The method as described in any one of claims 1-3 is characterized in that the detecting that the user inputs the first editing instruction for the editing area specifically includes: detecting that the user chooses to input the recommended editing instruction; the recommended editing instruction selected by the user is determined as the first editing instruction. The method according to any one of claims 1 to 4, characterized in that the method further comprises: displaying one or more preset editing instructions; the first editing instruction also comprises the preset editing instruction; The detecting that the user inputs the first editing instruction for the editing area specifically includes: detecting that the user selects to input the preset editing instruction; and the preset editing instruction selected by the user is determined as the first editing instruction. The method according to any one of claims 1 to 5, characterized in that the method further comprises: displaying a first input box, the first input box being used to receive a voice or text editing instruction; the first editing instruction further comprises a voice or text editing instruction input through the input box; The detecting that the user inputs the first editing instruction for the editing area specifically includes: detecting a voice or text instruction input by the user in the first input box; and determining the voice or text instruction as the first editing instruction. The method as described in any one of claims 1-6 is characterized in that the user operation for selecting an editing area includes: a user operation of selecting a first object in the first picture, the operation position of the user operation on the first picture falls on the first object in the first picture, and the image area where the first object is located is the editing area; the image area where the first object is located is determined by performing image segmentation processing on the first picture. The method according to any one of claims 1 to 7, characterized in that the user operation for selecting the editing area includes: a user operation of drawing the editing area in the first picture. The method according to any one of claims 1-8 is characterized in that it also includes: obtaining preprocessing information of the first image, and determining the editing area from the area indicated by the preprocessing information. The method of claim 9, wherein the preprocessing information includes indication information of a plurality of regions, wherein the preprocessing information includes coordinates of contour points of the plurality of regions, a binary image, and a grayscale image, wherein the values of the plurality of regions in the binary image are first values, and the grayscale values of the plurality of regions in the grayscale image are first grayscale values or first grayscale ranges; Determining the editing area from the area indicated by the preprocessing information specifically includes: determining the area where the operation position of the user operation is located among the multiple areas as the editing area, or determining the area among the multiple areas that is closest to the operation position as the editing area. The method as claimed in claim 9 is characterized in that the preprocessing information includes layer information of multiple layers, and the layer information of each layer includes the coordinates of opaque pixels in the layer; the method also includes: before determining the editing area from the area indicated by the preprocessing information, each of the opaque pixels connected in the layer is determined as an area indicated by the preprocessing information. The method according to claim 9 or 11 is characterized in that the preprocessing information includes contour information; the method also includes: before determining the editing area from the area indicated by the preprocessing information, determining the area surrounded by the contour indicated by the contour information as the area indicated by the preprocessing information. The method according to claim 9, 11 or 12 is characterized in that the preprocessing information includes depth information; the method also includes: before determining the editing area from the area indicated by the preprocessing information, pixels with the same or similar depth values are determined as an area indicated by the preprocessing information based on the depth information. The method according to any one of claims 1-13 is characterized in that the distinctive display includes one or more of the following methods: highlighting the outline of the editing area, highlighting the entire editing area, or displaying a dotted frame along the outline of the editing area. The method as described in any one of claims 1-14 is characterized in that it also includes: before performing image editing processing on the first picture according to the first editing instruction, if it is determined that the application of the first editing instruction to the editing area is unreasonable, then re-recommending the editing instruction or re-recommending the editing area. The method as claimed in claim 15 is characterized in that it also includes: judging whether it is reasonable to apply the first editing instruction to the editing area by comparing the feature vector corresponding to the first editing instruction with the fused feature vector of each image feature of the editing area. The method as claimed in claim 15 or 16 is characterized in that the re-recommending the editing area specifically includes: traversing the area outside the editing area in the first image, comparing the fused feature vector of the traversed area with the feature vector corresponding to the first editing instruction, finding an area that can reasonably match the first editing instruction, and re-recommending the found area as the editing area. The method as described in any one of claims 15-17 is characterized in that the re-recommending editing instructions specifically includes: traversing the editing types and/or editing parameters in the recommendation pool, comparing the feature vectors corresponding to the traversed editing types and/or editing parameters with the fused feature vector of the editing area, finding the editing types and/or editing parameters that can reasonably match the editing area, and modifying the first editing instruction according to the found editing type and/or editing parameters, and re-recommending the modified first editing instruction. The method as described in any one of claims 1-18 is characterized in that the editing processing corresponding to the first editing instruction includes: adding a new object to the editing area; in the first picture after the image editing processing, in the editing area, the depth features of the first pixel area are replaced with the depth features of the new object, and the depth features of the second pixel area remain the depth features of the original object, wherein the perspective relationship of the first pixel area is that the new object is in front of the original object, and the perspective relationship of the second pixel area is that the original object is in front of the new object. The method as claimed in claim 19 is characterized in that the image editing processing of the first picture according to the first editing instruction includes: correcting the image features of the first picture, and regenerating the first picture using the corrected image features of the first picture. The method of claim 20, wherein the image feature comprises a depth feature; and the correcting the image feature of the first image comprises: determining a perspective relationship between the new object and the original object according to image features of the first picture and editing parameters in the first editing instruction; Determine a reference depth of the new object according to the perspective relationship between the new object and the original object, and the depth value of the original object, and then use the reference depth to correct the depth feature of the new object; Replacing the original depth feature of the edited area with the corrected depth feature of the new object; After the depth feature of the new object is corrected, the average depth of the new object is close to or equal to the reference depth, and the depth difference between various regions on the new object remains unchanged. The method according to claim 20 or 21, characterized in that the image feature further comprises a first image feature, and the first image feature is an image feature other than a depth feature, and comprises one or more of the following: a mask feature, a contour feature, and a color feature; The correcting the image feature of the first picture further includes: In the editing area, if the depth feature of a pixel area is replaced with the corrected depth feature of the new object, the original first image feature of the pixel area in the editing area is replaced with the first image feature of the pixel area on the new object. A terminal device, characterized in that it comprises: a human-computer interaction module, a processor and a memory, wherein the human-computer interaction module is coupled to the processor, and the memory is coupled to the processor; the human-computer interaction module comprises a touch screen; The memory is used to store computer program codes, and the computer program codes include computer instructions. When the processor executes the computer instructions, the terminal device executes the method as described in any one of claims 1 to 22. A computer-readable storage medium, comprising instructions, characterized in that when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 22. A picture editing method, the method is applied to a human-computer interaction module, characterized in that the human-computer interaction module is included in a picture editing system, and the picture editing system further includes: a calculation module; The method comprises: The human-computer interaction module displays a first picture; The human-computer interaction module detects a user operation for selecting an editing area on the first picture; The human-computer interaction module distinguishably displays the editing area in the first picture; The human-computer interaction module receives the recommended editing instruction sent by the calculation module and displays the recommended editing instruction, where the recommended editing instruction is generated by the calculation module according to the image features of the editing area; The human-computer interaction module detects that a user inputs a first editing instruction for the editing area, the first editing instruction comprising: a recommended editing instruction; The human-computer interaction module receives the first image after image editing processing sent by the computing module, and displays the first image after image editing processing, wherein the image editing processing is performed by the computing module. The method according to claim 25, characterized in that the method further comprises: the human-computer interaction module displays one or more preset editing instructions; The detecting that the user inputs the first editing instruction for the editing area specifically includes: detecting that the user selects to input the preset editing instruction; and the preset editing instruction selected by the user is determined as the first editing instruction. The method according to any one of claims 25-26, characterized in that the method further comprises: the human-computer interaction module displays a first input box, the first input box is used to receive a voice or text editing instruction; The detecting that the user inputs the first editing instruction for the editing area specifically includes: detecting a voice or text instruction input by the user in the first input box; and determining the voice or text instruction as the first editing instruction. The method as described in any one of claims 25-27 is characterized in that the user operation for selecting an editing area includes: a user operation of selecting a first object in the first picture, the operation position of the user operation on the first picture falls on the first object in the first picture, and the image area where the first object is located is the editing area; the image area where the first object is located is determined by performing image segmentation processing on the first picture. The method according to any one of claims 25-28 is characterized in that the user operation for selecting the editing area includes: a user operation of drawing the editing area in the first picture. The method as described in any one of claims 25-29 is characterized in that the distinctive display includes one or more of the following methods: highlighting the outline of the editing area, highlighting the entire editing area, or displaying a dotted box along the outline of the editing area. The method as described in any one of claims 25-30 is characterized in that, after the human-computer interaction module detects that the user inputs a first editing instruction for the editing area, the method further includes: if it is unreasonable to apply the first editing instruction to the editing area, then re-recommended the editing area; the re-recommended editing area is found by the calculation module from the area outside the editing area according to the feature vector corresponding to the first editing instruction. The method as described in any one of claims 25-31 is characterized in that, after the human-computer interaction module detects that the user inputs a first editing instruction for the editing area, the method further includes: if it is unreasonable to apply the first editing instruction to the editing area, then recommending a modified first editing instruction; the editing type and/or editing parameters of the modified first editing instruction are found by the calculation module from the recommendation pool based on the fused feature vector of the editing area. A terminal device, characterized in that it comprises: a human-computer interaction module, a processor and a memory, wherein the human-computer interaction module is coupled to the processor, and the memory is coupled to the processor; the human-computer interaction module comprises a touch screen; The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the terminal device executes the method as described in any one of claims 25-32. A computer-readable storage medium, comprising instructions, characterized in that when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 25 to 32.