WO2025226229A1

WO2025226229A1 - A computer implemented method for assisting a user to create a visual design idea to skecth

Info

Publication number: WO2025226229A1
Application number: PCT/TR2024/050408
Authority: WO
Inventors: Tevfik Metin SEZGIN; Ezgi DEDE
Original assignee: Koc Universitesi
Current assignee: Koc Universitesi
Priority date: 2024-04-24
Filing date: 2024-04-24
Publication date: 2025-10-30
Anticipated expiration: 2026-10-24

Abstract

The present invention relates to a computer implemented method for assisting a user to find a novel idea to create a sketch where by facilitating the creativity cycle in visual design by use of ambiguous sketches and also a system and a medium suitable for such method.

Description

A COMPUTER IMPLEMENTED METHOD FOR ASSISTING A USER TO CREATE A VISUAL DESIGN IDEA TO SKECTH

The present invention relates to a computer implemented method, system and a computer readable medium to assist a user for creating a sketch. The present invention is particularly related to a computer-implemented method that helps the user find new ideas for sketching designs and guides them to improve those.

The first steps of visual design are tricky. Inexperienced designers tend to be stuck with similar ideas. Experienced designers try to overcome fixation by sketching random strokes or ideas, because sketches are naturally ambiguous and interpreting ambiguous shapes improves the chances of finding new ideas. Existing studies tried to use existing sketch datasets, which are not particularly formed to boost creativity with their ambiguity, for stimulating creative ideas. Therefore, there is a need for a unique dataset and a new method that can gives rise to new ideas and guide both experienced and inexperienced designers for more creative designs.

In the state of the art, a tool known as Creative Sketching Partner assists the professional users with the sketches taken directly from Google Quick Draw. Therefore, there is a lack of intentional ambiguity and support for inexperienced users in this solution. Also, another well- known tool named Co-Creative Drawing Agent adds a finished sketch from a sketch dataset directly to the canvas. However, this tool falls short in terms of guiding and stimulating user creativity, as it lacks the ability to offer a comprehensive solution in that regard.

Also, a state-of-the-art patent application no. US2022/0308843A1 discloses a computer implemented method and computer product for creating user experience designs.

Thus, an aspect of the present invention is to provide a method that guides both experienced and inexperienced designers for more creative designs by using ambiguous nature of sketches. Another aim of the present invention is to provide a system and a medium for such method.

The method described in claim 1 of the present invention enables users to discover effective ideas that enhance their creativity by providing tailored ambiguous sketches as stimuli. This method utilizes unfinished sketches as representations of ambiguous sketches. In other words, ambiguous sketches are illustrations that do not depict a single object; rather, they are suitable for being associated to various objects in the minds of different users. As individuals often employ similar shapes to depict various objects while sketching, these recurring shapes are identified by re-processing finished sketches of different objects such as trees, tables, clocks, and more. In the method of the present invention a sketch dataset is obtained which contains plurality of ambiguous sketches and their definitions obtained from different users. When a user seeks assistance while sketching, the created sketch is compared with ambiguous sketches in the design dataset, and the most similar sketch(es), preferably at least three, are determined. Then, the most similar sketch(es) and their definitions are presented to the user as suggestion. Suggestion provides the user with a list of definitions for the most similar sketch(es). This allows both the ambiguous sketches and the accompanying written definitions, which have been found to be highly effective in stimulating creativity, to be presented to the user. Thus, suggestion provides user to notice eluded ideas by prompting the similar sketches and/or different meanings given to similar sketches obtained from participants. Thus, the method of the present invention utilizes ambiguous sketches to facilitate creativity.

In an embodiment of the present invention while forming the sketch data set - an ambiguous sketch is prompted to participants in an online activity, the participants are asked to interpret the ambiguous sketch in a given context and the collected definition is compared with the definitions related to the ambiguous sketch. If it is determined that the answer does not match the predetermined definitions or there is no predetermined-definition for the corresponding ambiguous sketch yet, the answer is added to the sketch dataset as a definition for the corresponding ambiguous sketch. To obtain definitions for each ambiguous sketch, a group of participants are asked to provide definitions for the displayed sketches in a given context. The definitions received for ambiguous sketches are then filtered and rated by a group of raters. If a definition receives a low average score for visual similarity to the interpreted sketch from the rating group, it is excluded from the dataset. The remaining definitions are ordered based on their average visual similarity ratings. As a result, definitions for ambiguous sketches are filtered and sorted by assigning scores that reflect the resemblance of the sketch to the interpreted object.

In an embodiment of the present invention, to generate the ambiguous sketches for the design dataset, a raw doodle dataset (Google Quick Draw) including finished sketches of different objects is utilized. Sketches for each object are redrawn stroke by stroke based on the original order of strokes, and at each step incomplete sketch is saved separately and they are determined as ambiguous sketches until the entire sketch is completed. In a preferred version of this embodiment, determination of incomplete sketch process is applied to 150 sketches per object category of Google Quick Draw, which has 345 categories. Consequently, ambiguous incomplete sketches related to specific objects are collected into respective classes. Then, they are converted into feature vectors. Then, vectors are grouped according to their distance to create groups of visually similar incomplete and ambiguous sketches. Those groups were formed based on the visual features of the incomplete sketches. However, these visually similar ambiguous sketches are not necessarily incomplete representations of the same object. Hence, if the incomplete drawings of different objects share visual similarities, they can belong to the same group. Thus, vectors falling within a cluster represent visual shapes that are ambiguous, as they could be partial drawings of various object categories.

In another embodiment of the present invention, if a group contains sketches from more than 50 different classes or contains sketches from less than 3 different classes, the ambiguous sketch group is excluded. The average vector of each of the remaining groups represents the common visual features of the incomplete/partial sketches in that group. The partial sketches which have the vector with lowest spatial distance to the average vector of the group that it belongs has been selected as the ambiguous shapes representing the visual features of that group.

The computer-readable medium of the present invention comprises instructions for controlling and/or regulating a computer- implemented system. The computer- implemented system of the present invention may be any of a mobile phone, a computer, a tablet, a laptop etc.

By means of the present invention, a method for providing suitable assistance both for experienced and inexperienced users is obtained. Method for assisting users offers new ideas to help inexperienced designers to overcome their design fixation and provides creative ideas depending on the user’s existing sketch. The computer implemented method provides assistance for all users from different skill stages to achieve better results.

Accompanying drawings are given solely for the purpose of exemplifying the technical approach of the present invention whose advantages were outlined above and will be explained hereinafter in brief.

The drawings are not meant to delimit the scope of protection as identified in the claims nor should they be referred to alone in an effort to interpret the scope identified in said claims without recourse to the technical disclosure in the description of the present invention.

Fig. 1 is a view of a basic user interface of the method according to an embodiment of the present invention.

Fig. 2 is another view of a basic user interface of the method according to an embodiment of the present invention. The following numerals are assigned to different parts demonstrated in the drawings:

1. Sketch

2. Suggestion

3. Ambiguous sketch

4. Definition

The computer implemented method for assisting a user to create a sketch (1) comprises the steps of: obtaining a sketch dataset including plurality of ambiguous sketches (3) and plurality of predetermined definitions (4) for each ambiguous sketch (3), obtaining a sketch (1) from the user, upon a request for assistance comparing the sketch (1) with the ambiguous sketches (3) to determine at least one most similar ambiguous sketch (3), prompting a suggestion (2) to the user wherein the suggestion (2) comprises the at least one most similar ambiguous sketch (3) and/or the definitions (4) for the at least one most similar ambiguous sketch (3).

In the method according to the present invention, a set of ambiguous sketches (3) and more than three, but most preferably eight definitions (4) for each ambiguous sketch (3) are obtained for each predetermined object via the sketch dataset. Sketch dataset contains more than three predetermined written definition (4) for each of the ambiguous sketches (3). When a user requests assistance, their sketch (1) is sent to the transformer model, which generates an embedding for the sketch (1). This embedding is then compared with the embeddings of all ambiguous sketches in the sketch dataset (3) and at least one most visually similar ambiguous sketch (3) and more preferably the three most visually similar ambiguous sketches (3) are determined. Thereafter, most visually similar ambiguous sketches (3) and definitions (4) related to the most similar sketches (3) are presented to the user via the suggestion (2). Preferably, suggestion (2) prompts at least two similar ambiguous sketches (3) and definitions (4) for that ambiguous sketches (3). More preferably, suggestion (2) prompts three similar ambiguous sketches (3) and their definitions (4). Suggestions (2), which include three ambiguous sketches (3) have been identified as the most effective in triggering creativity while staying relatively close to the general concept of the sketch (1). Suggestion (2) offers multiple definitions (4) for the ambiguous sketches (3) determined as the most similar to the sketch (1) to be presented to the user. Thus, the suggestion (2) represents creative interpretations of different users and puts a new complexion on the existing sketch (1). Users can seek assistance multiple times for a single sketch (1), and each time they request assistance, the method provides a new suggestion (2) according to the present stage of the sketch (1).

Figures 1 and 2 shows a user interface according to an embodiment of the present invention. In this example, the user aims to design a logo for a firm that develops prosthetic limbs and initially sketches a man with prosthetic limbs. Then, calls for assistance i.e. by clicking on the help button. A suggestion (2) comprising three ambiguous sketches (3) is prompted to inspire the user. When the user clicks on a sketch, the suggestion (2) presents related interpretations as a textual suggestion. The user interprets the sketches and considers the idea of a logo with a spider who has 8 prosthetic legs. Therefore, the user changes the drawing by merging prosthetic limbs idea with the ambiguous sketch that the user interpreted as a spider. Also, when the user clicks on the ambiguous sketch (3) that is interpreted as the spider, a list of definitions related to that sketch, which are taken from the sketch dataset that is formed previously are presented. When the user reads the list, the user realizes that the ambiguous sketch (3) can also be interpreted as a squid. It gives the user a new idea to draw a spider with 6 prosthetic arms and 2 prosthetic tentacles. So, the user is inspired with a new idea for a logo and that is not as cliche as a man with prosthetic limbs.

In an embodiment of the present invention the method also comprises the steps of detecting if the user selects an ambiguous sketch (3) from the suggestion (2) and prompting definitions (4) predetermined for the selected ambiguous sketch (3). The user may select one of the sketches in the suggestion (2), for example, by upon clicking on them. If the user selects an ambiguous sketch (3) from the suggestion (2), only the definitions (4) predetermined for the selected ambiguous sketch (3) are displayed to the user via the suggestion (2). This enables the user to engage in more focused ideas.

In an embodiment of the present invention the method also comprises the steps of prompting an ambiguous sketch (3) to the user, asking the user to explain their sketch (1) and their design ideas, obtaining an answer from the user and comparing the answer with the predetermined definitions (4) for the ambiguous sketch (3), if it is determined that the answer matches with some of the predetermined definitions (4) placing those definitions at the top of the definitions (4).

In an embodiment of the present invention the method also comprises the steps of providing a raw dataset including complete sketches of different objects, and obtaining more than one ambiguous sketch (3) by generating incomplete sketches which are redrawn stroke by stroke based on the original order of strokes of complete sketches and at each step incomplete sketch is saved and determined as an ambiguous sketch (3) until the entire sketch is completed. Each finished sketch is reconstructed by adding strokes in the original drawing order. This allows the sketches to be segmented into their respective parts based on the stroke order, resulting in the acquisition of ambiguous sketches (3). In a preferred version of this embodiment the ambiguous sketches (3) are obtained by clustering visual feature vectors of incomplete sketches. A transformer model is employed to generate embeddings i.e. visual feature vectors for each completion step of every finished sketch.

In an embodiment of the present invention the method also comprises the steps of determining a group for different objects wherein each class contains the ambiguous sketches (3) of the related object, comparing the ambiguous sketches (3) from different classes with each other to determine graphically similar ambiguous sketches (3) from different classes, determining more than one group wherein each group contains the ambiguous sketches (3) from different classes determined as graphically similar. All the ambiguous sketches (3) related to one object form a class. Ambiguous sketches (3) are processed separately to extract their vector representations, which are numerical representations capturing the visual and shape attributes of the sketches. These vectors are then grouped based on their proximity in a multidimensional space to cluster similar shapes together. For example, an unfinished and thus an ambiguous sketch (3) of an alarm clock may resemble an ambiguous sketch (3) of a dog, indicating that the vectors of an ambiguous dog sketch can be similar to those of the ambiguous alarm clock sketch. This enables the identification of meaningful interpretations among the ambiguous sketches (3). Groups are obtained by using proximity algorithms in a multidimensional space, such as Euclidean distance calculations and k-clustering algorithms, to group ambiguous sketches (3) based on their visual similarities.

In an embodiment of the present invention the method also comprises the steps of counting the number of different classes of ambiguous sketches (3) within a group, comparing the counted number of different classes with a maximum grouping value and if the counted number of different classes are determined to be higher than the maximum grouping value, excluding the ambiguous sketches (3) from the design dataset. In a preferred version of this embodiment, the maximum grouping value is determined to be 50. Hence, if a group comprises ambiguous sketches (3) from nearly every class (e.g., sun, ball, wheel, cake, hat, eye, lake...), ambiguous sketches (3) within that group are excluded as they do not offer significant insights. In an embodiment of the present invention the method also comprises the steps of comparing the counted number of different classes with a minimum grouping value and if the counted number of different classes are determined to be lower than the minimum grouping value, excluding the ambiguous sketches (3) from the sketch dataset. In a preferred version of this embodiment, the minimum grouping value is determined to be 3. If a group comprises ambiguous sketches (3) from a few classes this means that the ambiguous sketches (3) represent highly specialized representation of an object. Consequently, their vectors are usually clustered with vectors from one or two categories in the multidimensional space. Again, ambiguous sketches (3) in such group are also excluded as they do not offer significant insights.

In an embodiment of the present invention the method also comprises the step of after obtaining a sketch (1) from the user, prompting at least one question to the user. The first sketch (1) of a user is expected to be a non-creative cliche. To assist the user, several questions are preferably prompted to better understand the intention behind the sketch (1). The preferred questions include: "What were you envisioning while drawing this sketch?", "What sets your design apart from cliches?", and "How is the company that will use this design different from similar companies?" These questions aim to capture the underlying idea of the user's sketch (1) and provide relevant suggestion (2) Additionally, users are encouraged to explore new ideas while responding to these questions, as effective questioning can trigger creative processes.

In an embodiment of the present invention the method also comprises the steps of obtaining at least one intention comprising at least one keyword from the user via questions, extracting the keyword from their intentions, comparing the keyword with definitions (4) for the at least one ambiguous sketch (3) determined as the most similar to the sketch (1) obtained from the user and if at least one definition (4) matches with the at least one keyword, the corresponding definition (4) is distinguished while being prompted via the suggestion (2) from the other nonmatching definitions (4). In this embodiment, users answer to the questions i.e. intention is recorded, converted to text, translated, and coded. At least one keywords is extracted from the answer to focus the analysis. Preferably, keywords are encoded into word2vec or similar embeddings, and their similarity with the pre-determined definitions (4) is assessed. If a similarity between definitions (4) and keyword is determined, the corresponding definition (4) is distinguished, preferably by being indicated with a different color. For example, highly similar definitions (4) are highlighted in red and placed at the top of the written definitions (4) list. A computer- implemented system according to the invention is suitable for carrying out the steps of the method according to the invention.

A computer-readable medium according to the invention contains instructions for controlling and/or regulating a computer-implemented system according to the invention. While certain exemplary preferred embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, which is only limited according to the wording of the claims.

Claims

1. A computer implemented method for assisting a user to create a sketch (1), the method comprising the steps of: obtaining a sketch dataset including plurality of ambiguous sketches (3) and plurality of predetermined definitions (4) for each ambiguous sketch (3), obtaining a sketch (1) from the user upon a request for assistance, comparing the sketch (1) with the ambiguous sketches (3) to determine at least one most similar ambiguous sketch (3), prompting a suggestion (2) to the user wherein the suggestion (2) comprises the at least one most similar ambiguous sketch (3) and/or the definitions (4) for the at least one most similar ambiguous sketch (3).

2. The computer-implemented method according to claim 1 further comprising: detecting if the user selects an ambiguous sketch (3) from the suggestion (2) and prompting definitions (4) predetermined for the selected ambiguous sketch (3).

3. The computer-implemented method according to claim 1 or 2 further comprising: prompting an ambiguous sketch (3) to a group of participants, asking the participant group to define the ambiguous sketch (3), obtaining definitions (4) from the participant group, if it is determined that the answer does not match the predetermined definitions (4) adding the answer to the dataset as a definition (4) for the corresponding ambiguous sketch (3).

4. The computer-implemented method according to any of the above claims further comprising:

Providing a raw dataset including complete sketches of different objects, obtaining more than one ambiguous sketch (4) by generating incomplete sketches which are redrawn stroke by stroke based on the original order of strokes of complete sketches and at each step incomplete sketch is saved and determined as an ambiguous sketch (4) until the entire sketch is completed.

5. The computer-implemented method according to any of the above claims further comprising: determining a class for different objects wherein each class contains the ambiguous sketches (3) of the related object, comparing the ambiguous sketches (3) from different classes with each other to determine graphically similar ambiguous sketches (3) from different classes, determining more than one group wherein each group contains the ambiguous sketches (3) from different classes determined as graphically similar.

6. The computer-implemented method according to claim 5 further comprising:

- counting the number of different classes of ambiguous sketches (3) within a group,

- comparing the counted number of different classes with a maximum grouping value and

- if the counted number of different classes are determined to be higher than the maximum grouping value, excluding the ambiguous sketches (3) from the other ambiguous sketches (3) in the sketch dataset.

7. The computer-implemented method according to claim 6 wherein the maximum grouping value is 50.

8. The computer-implemented method according to claim 5 further comprising:

- comparing the counted number of different classes with a minimum grouping value and

- if the counted number of different classes are determined to be lower than the minimum grouping value, excluding the ambiguous sketches (3) from the other ambiguous sketches (3) in the sketch dataset.

9. The computer-implemented method according to claim 8 wherein the minimum grouping value is 3.

10. The computer-implemented method according to any of the above claims further comprising: after obtaining a sketch (1) from the user, prompting at least one question to the user.

11. The computer-implemented method according to claim 10 further comprising: obtaining at least one intention answer comprising at least one keyword from the user extracting the keyword from the intention and comparing the keyword with definitions (4) for the at least one ambiguous sketch (3) determined as the most similar to the sketch (1) obtained from the user, if at least one definition (4) matches with the at least one keyword, the corresponding definition (4) is distinguished while being prompted via the suggestion (2) from the other non-matching definitions (4).

12. A computer implemented system arranged to perform the method of any of claims 1 to claim 11.

13. A computer-readable medium comprising instructions for controlling and/or regulating a computer- implemented system according to claim 12.