WO2025204235A1 - Information processing system, control method, and program - Google Patents

Abstract

[Problem] To assist a creation of a user in cooking. [Solution] An information processing system provided with a control unit that performs: a process for generating cooking information using a cooking learning model on the basis of instruction information input by a user; a process for generating cooking processing information corresponding to cooking steps included in the cooking information, and integrating the cooking processing information to generate integrated cooking processing information; and a process for presenting the integrated cooking processing information being generated.

Description

Information processing system, control method, and program

This disclosure relates to an information processing system, a control method, and a program.

In recent years, technologies have been disclosed that use cooking robots to automate various cooking processes. For example, Patent Document 1 below discloses technology that senses the biometric reactions of a chef when preparing a dish, and then includes the sensed biometric data in recipe data to recreate a dish that suits the eater's preferences.

JP 2022-63885 A

However, there was a lack of support for users such as chefs and culinary experts when creating new dishes.

This disclosure therefore proposes an information processing system, control method, and program that can assist users in creating dishes.

According to the present disclosure, an information processing system is provided that includes a control unit that performs the following processes: generating cooking information using a cooking learning model based on instruction information input by a user; generating cooking processing information corresponding to the cooking steps included in the cooking information using a cooking processing learning model, integrating the cooking processing information to generate integrated cooking processing information; and presenting the generated integrated cooking processing information.

The present disclosure also provides a control method including: a processor generating cooking information using a cooking learning model based on instruction information input by a user; generating cooking processing information corresponding to cooking steps included in the cooking information; integrating the cooking processing information to generate integrated cooking processing information; and presenting the generated integrated cooking processing information.

Furthermore, according to the present disclosure, a program is provided that causes a computer to function as a control unit that performs the following processes: generating cooking information using a cooking learning model based on instruction information input by a user; generating cooking processing information corresponding to the cooking steps included in the cooking information and integrating the cooking processing information to generate integrated cooking processing information; and presenting the generated integrated cooking processing information.

1 is a diagram illustrating an overall configuration of a cooking information generation system 1 according to an embodiment of the present disclosure. 1 is a block diagram showing an example of the configuration of an information processing device 10 according to the present embodiment. FIG. 2 is a diagram for explaining details of the function of a draft generation unit 112 according to this embodiment. FIG. 10 is a diagram for explaining generation of cooking information according to the present embodiment. 1 is a flowchart showing an example of the overall flow of the cooking information generation system 1 according to the present embodiment. 10 is a flowchart showing an example of the flow of basic information input processing according to the present embodiment. 10A and 10B are diagrams for explaining estimation of dish contents based on a user's dish information according to this embodiment. 10 is a flowchart showing an example of the flow of a recipe plan generation process according to the present embodiment. FIG. 10 is a diagram showing an example of a display screen on which a recipe plan is displayed according to the present embodiment. FIG. 10 is a diagram illustrating a display screen that displays details of a recipe plan. FIG. 10 is a diagram illustrating a display screen that displays details of a recipe plan. 10 is a flowchart showing an example of the flow of a process for generating recipe plans with different levels of freshness according to this embodiment. 10 is a flowchart showing an example of the flow of an adjustment process according to the present embodiment. 10A and 10B are diagrams illustrating an example of adjustment according to a user's cooking environment according to the present embodiment. 10 is a flowchart showing an example of the flow of an element review process according to the present embodiment. 10 is a flowchart showing the flow of the review process for element 1. 10 is a flowchart showing the flow of the review process for element 2. 10 is a flowchart showing the flow of the review process for element 3. 10 is a flowchart showing the flow of the review process for element 4. 10 is a flowchart showing an example of the flow of integration processing according to the present embodiment. FIG. 10 is a diagram showing an example of presentation of integrated cooking and processing information according to this embodiment. 10 is a flowchart showing an example of the flow of a process for presenting integrated cooking and processing information according to the present embodiment. FIG. 10 is a diagram showing an example of a detailed display screen for a selected dish according to this embodiment.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that in this specification and drawings, components having substantially the same functional configuration will be assigned the same reference numerals, and redundant explanations will be omitted.

The explanation will be given in the following order:
1. Overview of a Recipe Information Generation System According to an Embodiment of the Present Disclosure 2. Configuration Example of Information Processing Device 10 3. Operation Processing 3-1. Basic Information Input Processing 3-2. Recipe Draft Generation Processing 3-3. Adjustment Processing 3-4. Element Review Processing 3-5. Integration Processing 3-6. Presentation Processing 4. Others 5. Supplementary Information

<<1. Overview of a Recipe Information Generation System According to an Embodiment of the Present Disclosure>>
An overview of a cooking information generation system according to an embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a diagram showing the overall configuration of a cooking information generation system 1 according to an embodiment of the present disclosure.

As shown in FIG. 1, the cooking information generation system 1 according to this embodiment includes an information processing device 10, a camera 20, an ingredient sensor 22, a laser processing machine 24, a cooking robot 26, and a cutting machine 28.

The information processing device 10 can be connected to and communicate with the camera 20, food sensor 22, laser processing machine 24, cooking robot 26, and cutting machine 28 via network 30. Note that the camera 20, food sensor 22, laser processing machine 24, cooking robot 26, and cutting machine 28 are examples of configurations used to consider the feasibility of cooking and processing information corresponding to cooking processes, and this embodiment is not limited to this.

The information processing device 10 is realized by a PC (personal computer), tablet terminal, smartphone, HMD (head-mounted display), etc. The information processing device 10 creates new dishes in response to instructions (hereinafter also referred to as suggestions) from a user such as a chef, and presents them to the user. The information processing device 10 can reflect the user's suggestions in the creation of the dish as appropriate during the dish creation process. In this embodiment, the information processing device 10 can create recipe data that includes the cooking steps of the dish as the creation of the dish.

Although it has been common practice to use AI (Artificial Intelligence) to generate new recipes using learning models that have learned from existing recipes, cooking is not simply a matter of cutting and pasting multiple recipes. Creating a dish involves a variety of factors, such as changes in ingredients that occur during the cooking process, safety, and design. Furthermore, simply generating new dish images using image generation AI does not go so far as to suggest feasible recipes using actual ingredients.

This disclosure therefore proposes a system to assist users in creating dishes. Specifically, the information processing device 10 uses AI to generate recipes (an example of cooking information), adjusts the recipe in response to user feedback, and considers the feasibility of each cooking step included in the recipe (i.e., generates corresponding cooking processing information), and integrates each piece of cooking processing information to generate integrated cooking processing information (a new recipe, also referred to as an integrated recipe), which is presented to the user. Cooking processing information is, for example, information on processing related to the appearance of ingredients (laser processing, cutting, heating, presentation, etc.).

When generating cooking and processing information, the information processing device 10 may use the results of simulations performed using a camera 20, food ingredient sensor 22, laser processing machine 24, cooking robot 26, cutting machine 28, etc., or the results of actual cooking.

For example, the camera 20 captures an image of the food ingredient F and outputs the captured image to the information processing device 10, the laser processing machine 24, the cooking robot 26, or the cutting machine 28. The food ingredient F may be placed by the cooking robot 26. The food ingredient sensor 22 senses various components (e.g., moisture, sugar content, etc.) or conditions (e.g., temperature) of the food ingredient F, and outputs the sensing data to the information processing device 10, etc.

The laser processing machine 24 can draw letters, patterns, marks, etc. on the food material F by irradiating it with a laser and charring it, or can carve letters, patterns, marks, etc. into the food material F by using a laser to carve the food material F. The cooking robot 26 has one or more robotic arms and can grasp the food material F and place it in a predetermined position, add seasonings, pour sauces, and perform various cooking operations using various cooking utensils. The cutting machine 28 can cut the food material F using a CNC (Computer Numerical Control) milling machine, laser, etc.

The laser processing machine 24, cooking robot 26, and cutting machine 28 are digital fabrication devices that can precisely perform cutting, engraving, heating, laminating, and various cooking processes based on digital data. The laser processing machine 24, cooking robot 26, and cutting machine 28 are examples of processing devices that perform various cooking processes, and the configuration of the cooking information generation system 1 is not limited to this.

The above provides an overview of the cooking information generation system 1 according to one embodiment of the present disclosure. Next, the information processing device 10 that generates cooking information according to this embodiment will be described in detail with reference to the drawings.

<<2. Configuration example of information processing device 10>>
2 is a block diagram showing an example of the configuration of the information processing device 10 according to the present embodiment. As shown in FIG. 2, the information processing device 10 includes a control unit 110, a communication unit 120, an operation input unit 130, a display unit 140, a storage unit 150, and an audio input/output unit 160.

(Communication unit 120)
The communication unit 120 has a transmitter that transmits data to an external device and a receiver that receives data from the external device. For example, the communication unit 120 is communicatively connected to the camera 20 and receives captured images of the food ingredient F from the camera 20. The communication unit 120 is also communicatively connected to the food ingredient sensor 22 and receives sensing data of the food ingredient F from the food ingredient sensor 22. The communication unit 120 is also communicatively connected to various digital fabrication devices (the laser processing machine 24, the cooking robot 26, and the cutting machine 28) and transmits control signals for controlling various cooking operations of the various digital fabrication devices.

The communication unit 120 may communicate with an external device or the Internet using, for example, a wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution)), 5G (fifth generation mobile communication system)), etc.

(Operation Input Unit 130 and Display Unit 140)
The operation input unit 130 accepts operation input by the user and outputs the input information to the control unit 110. The operation input unit 130 may be realized by a mouse, a keyboard, a switch, a button, or the like. The display unit 140 displays various operation screens and a predicted image of a processed result, which will be described later. The display unit 140 may be a display panel such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display. The operation input unit 130 and the display unit 140 may be provided integrally. The operation input unit 130 may be a touch sensor stacked on the display unit 140 (for example, a panel display).

(Control unit 110)
The control unit 110 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the information processing device 10 in accordance with various programs. The control unit 110 is realized by electronic circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, etc. The control unit 110 may also include a ROM (Read Only Memory) that stores programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

The control unit 110 can also function as an agent unit 111, a draft generation unit 112, a display control unit 113, an adjustment unit 114, an element review unit 115, an integration unit 116, and a story generation unit 117.

The agent unit 111 functions as an agent capable of interacting with the user, and acquires appropriate information from the interaction with the user. Information input by the user to the information processing device 10 may be input through user operation from the operation input unit 130, or may be input through interaction with an agent presented by the agent unit 111. The interaction between the user and the agent may be via text or voice.

More specifically, the agent unit 111 may be an AI agent that utilizes large language models (LLMs), etc. This allows the user to input natural language.

The agent unit 111 acquires the user's suggestions (requests) regarding the dish to be created from a dialogue with the user. For example, the user may interact with the agent about the concept of a new dish they want to make, the ingredients they want to use, etc. The agent unit 111 generates and outputs prompts in response to the user's suggestions based on the dialogue with the user. AI may be used to generate the prompts. More specifically, a learning model that has learned about prompt generation may be used. Hereinafter, the prompts that correspond to the user's suggestions and are output from the agent unit 111 are also referred to as suggestions.

The recipe plan generation unit 112 has the function of generating a recipe plan (an example of recipe information). Using a learning model generated by machine learning, the recipe plan generation unit 112 generates one or more recipe plans in response to user suggestions (prompts) input from the agent unit 111. A recipe plan includes at least recipe data for the dish. The recipe plan may also include images of the dish. The generation of a recipe plan will be explained in detail below with reference to Figure 3.

Figure 3 is a diagram for explaining the details of the function of the recipe plan generation unit 112 according to this embodiment. As shown in Figure 3, the recipe plan generation unit 112 includes an ingredient pairing unit 510 and a recipe plan generation unit 520.

The agent unit 111 interacts with the user U, and the prompt 41 generated from the user's suggestions 40 is input to the draft generation unit 112. The user's suggestions 40 include information about the concept of the new dish they want to make and the ingredients to be used. The concept of the dish may include, for example, the impression, atmosphere, style, genre, theme, purpose, and key points of the dish. In addition, requests regarding the design of the dish and the ingredients to be used may also be input as concepts.

Table 1 below shows some examples of concepts. The concept can be the overall image of the dish, or it can be related to the design or ingredients.

The ingredient pairing unit 510 outputs combination ingredients that are suitable for use in combination with the ingredients the user wants to use (the ingredients used). For example, the ingredient pairing unit 510 inputs data on the ingredients used into an ingredient pairing learning model, and outputs ingredients that pair well with the ingredients used or that are compatible at the molecular level. The ingredient pairing learning model is a learning model generated by learning ingredient combinations using machine learning.

The recipe draft generation unit 520 inputs the concept, ingredients used, and ingredient combinations into the cooking learning model as input data, and outputs one or more recipe drafts 42, which are recipe information proposals. The cooking learning model is a model that has learned about cooking in advance through machine learning. The cooking learning model can learn images (captured images) of various dishes, corresponding recipe data, etc. The recipe data to be learned includes the name of the dish, ingredients, cooking steps, cooking utensils used, cooking time, and tags (for example, attribute information such as recipe category, purpose of cooking, genre, suitable season, suitable occasion, target demographic, and other keywords).

Specific examples of recipe categories that can be tagged include: "Suitable seasons" include, for example, spring, summer, autumn, winter, New Year's, cherry blossom viewing, moon viewing, and Christmas. "Suitable occasions" include, for example, entertaining, parties, snacks, late-night snacks, goes well with wine, and anniversaries. "Target demographics" include, for example, athletes, children, adults, and the elderly. "Other keywords" include, for example, gorgeous, colorful, lively, nutritious, low-salt, easy to digest, healthy, relaxing, mood changer, easy, time-saving, nostalgic, Italian, and Chinese. The dish draft output from the cooking learning model includes recipe data and corresponding dish images. The cooking learning model may include a recipe generation AI that generates recipe data and an image generation AI that generates dish images corresponding to the recipe data.

The cooking learning models used by the recipe draft generation unit 520 include a user cooking learning model 521 that has learned the user's cooking, and an other user cooking learning model 522 that has learned the cooking of other users (e.g., other chefs). There may be multiple other user cooking learning models 522. For example, there may be an other user cooking learning model that has learned the cooking of a French chef, or an other user cooking learning model that has learned the cooking of a Chinese chef. The other user cooking learning model 522 may also be an other user cooking learning model that has learned the cooking of many chefs regardless of cooking genre.

The recipe draft generation unit 520 may output a recipe draft using the user's cooking learning model 521, or may output a recipe draft using the other user's cooking learning model 522, or may output a recipe draft using the user's cooking learning model 521 and the other user's cooking learning model 522. The user's cooking learning model 521 and the other user's cooking learning model 522 may be stored in the memory unit 150. For ease of explanation, Figure 3 illustrates the cooking learning models used by the recipe draft generation unit 520 as the configuration of the recipe draft generation unit 520, but it is assumed that the recipe draft generation unit 520 calls up and uses each cooking learning model stored in the memory unit 150.

Furthermore, when generating dish information, the recipe draft generation unit 520 may link with an external database as appropriate to acquire external information (for example, information about the dish, information about ingredients, information related to the input concept, various designs, etc.). Furthermore, the recipe draft generation unit 520 may use design generation AI to acquire a design based on the concept, input the design data into the cooking learning model, and output a dish image decorated based on that design.

The display control unit 113 controls the display of images on the display unit 140. The display control unit 113 controls the display of various input screens on the display unit 140, such as a screen that prompts dialogue with the AI agent, a screen that presents a recipe draft generated by the draft generation unit 112, a screen that presents new recipe information (integrated cooking and processing information) generated by the integration unit 116 (described below), and a screen that presents a new recipe development story generated by the story generation unit 117 (described below).

The adjustment unit 114, element review unit 115, and integration unit 116 will be explained below with reference to Figure 4. Figure 4 is a diagram for explaining the generation of cooking information according to this embodiment.

The adjustment unit 114 adjusts the recipe drafts 42 generated by the draft generation unit 112 and outputs the adjusted recipe drafts to the element review unit 115. The adjustment unit 114 may be input with one or more recipe drafts automatically selected from the recipe drafts 42 generated by the draft generation unit 112, or with one or more recipe drafts selected by the user. As a method of automatic selection, for example, the adjustment unit 114 may assign a score to each recipe draft based on a certain criterion (for example, whether it is close to the characteristics of the user's dish) and select the recipe draft with the highest score. Furthermore, when the user makes a selection, the recipe drafts may be sorted based on scores according to various criteria, making it easier for the user to select.

In the example shown in FIG. 4, a recipe draft 42A selected by the user is input to the adjustment unit 114. The recipe draft 42A includes recipe data 421A and a dish image 422A.

The adjustment unit 114 may receive new user suggestions (prompts based on these suggestions, also simply referred to as suggestions) obtained from the dialogue between the user and the agent from the agent unit 111. The adjustment unit 114 adjusts the recipe draft in accordance with the user suggestions input from the agent unit 111. In doing so, the adjustment unit 114 may use a cooking learning model (user cooking learning model, other user cooking learning model), an ingredient pairing learning model, image generation AI, etc. The cooking learning model is stored in the memory unit 150, and the adjustment unit 114 can call up and use the cooking learning model from the memory unit 150 as needed. Furthermore, it is assumed that such a cooking learning model is the same as the cooking learning model used in the recipe draft generation unit 520 of the draft generation unit 112.

The adjustment unit 114 also functions as an environment adjustment unit that adjusts the recipe draft in accordance with the user's cooking environment information. The adjustment unit 114 may also acquire external information from an external database, etc., as needed, and use the external information to adjust the recipe draft.

Specific examples of adjustments to the recipe draft by the adjustment unit 114 include replacing ingredients used, obtaining new ingredient combinations, changing the cooking utensils used, and modifying the dish image (appearance of the dish).

For example, if a user selects a recipe draft and converses with the agent saying, "This is good, but maybe fruit would be better as a topping," the agent unit 111 will input information such as "change the topping to fruit" as a suggestion based on the conversation with the user, and the adjustment unit 114 will adjust the recipe data to change the topping ingredient to fruit.

The adjustment unit 114 may also change the cooking utensils in the recipe data to suit the user's cooking environment. For example, recipe data that says "steam potatoes in a pot for 10 minutes" may be modified to recipe data that uses a "microwave oven," a method commonly used in the user's cooking environment, specifically, recipe data that says "steam potatoes in a microwave oven at 600W for 3 minutes." When the cooking utensil is changed from a pot to a microwave oven, the adjustment unit 114 may generate appropriate setting information for the microwave oven (microwave output, time) based on external information (such as a correspondence table between boiling time and microwave heating time) obtained from an external database, etc.

In this way, the adjustment unit 114 adjusts the recipe plan to better match the user's intentions and environment.

The element review unit 115 has the function of dividing the recipe data included in the recipe draft output from the adjustment unit 114 into elements and reviewing them according to the cooking steps of the recipe data, in order to increase the feasibility of the draft dish. In other words, at the draft dish stage, there is no certainty as to whether the dish can actually be made using this recipe, or whether it is realistically possible to create the appearance of the dish shown in the dish image. A draft dish includes recipe data showing the cooking steps and a dish image showing the appearance of the dish, but there are cases where the appearance of the dish shown in a dish image generated by image generation AI (for example, presentation on a dish, decoration) is not realistic.

The element review unit 115 therefore reviews the feasibility of each element, specifically optimizing the control of the cooking robot to be used and generating variations (expanding the elements), and generates cooking and processing information for each element that is feasible, i.e., that will result in the cooking process being successful. Element review involves actually cooking with a cooking robot and analyzing the results, or cooking is performed through simulation. The element review unit 115 can generate one or more pieces of cooking and processing information for each element.

If there is an element (cooking step) that does not produce a successful result, the element review unit 115 requests adjustment from the adjustment unit 114. In this case, the adjustment unit 114 adjusts the recipe data (for example, changes to the ingredients used, the cooking utensils used, or the appearance of the dish), and the element review unit 115 reviews the element again.

As shown in Figure 4, the element review unit 115 divides the recipe data 421A' of the dish draft adjusted by the adjustment unit 114 into elements 1 to 3, etc., and generates variations in the process of reviewing each element.

The integration unit 116 combines each successful element (cooking processing information) to generate new cooking information (integrated cooking processing information). As described above, control optimization and variation generation are performed for each element (cooking process) of the decomposed recipe data, and one or more successful derived elements can be generated for each element. The integration unit 116 can appropriately combine the successful derived elements for each element to generate multiple patterns of new recipe data as integrated cooking processing information. In this case, the integration unit 116 can generate new recipe data that meets conditions such as overall cooking time. The integration unit 116 also generates dish images corresponding to each new recipe data as integrated cooking processing information.

As shown in Figure 4, the integration unit 116 appropriately combines variations of each element to generate new dish information such as dish A'1 (integration 1) to dish A'3 (integration 3), etc.

The one or more pieces of integrated cooking and processing information generated by the integration unit 116 are output to the display control unit 113 and displayed on the display unit 140, thereby being presented to the user. When the integrated cooking and processing information is displayed, in addition to the elements (cooking and processing information) used in the integration, elements not used in the integration may also be presented as candidates. The user may interact with the agent about the presented integrated cooking and processing information and may further input suggestions. For example, a suggestion to change the elements used in the integration to other elements listed as candidates may be input. A change in ingredients or concept, etc. may also be input.

The agent unit 111 inputs the new findings into the adjustment unit 114, element review unit 115, or integration unit 116, causing them to revise the integrated cooking and processing information as appropriate.

It is also possible that the user's requests may change from their original content during the process of creating a new dish. In this embodiment, the user can input new suggestions as needed during the creation process, not only at the start of dish information generation, but also when selecting a dish draft or when integrated cooking and processing information is presented. This allows the system to work together with the user to suggest creative dishes that are closer to the user's intentions, and support the user's dish creation.

The story generation unit 117 generates a development story for the creative dish based on information stored in the storage unit 150 as appropriate during the creation process. The information stored in the storage unit 150 includes the user's dialogue with the agent (user's suggestions), the initially selected dish draft, the dish drafts that were not selected, and examples of success and failure in considering elements. The development story may be generated using text and images. The development story may also be a still image with text and images arranged in it, or a video showing the changes in the dish image from the dish draft to the creative dish. The generated development story is output to the display control unit 113 and displayed on the display unit 140 for presentation to the user. The development story may be used, for example, for advertising, etc.

The functional configuration of the control unit 110 has been specifically described above. However, the control unit 110 is not limited to the functional configuration described above and has a tool linkage function. Specifically, the control unit 110 can link with various conventional tools as appropriate when generating cooking information. For example, the control unit 110 may link with 3D CAD when generating a design in the draft generation unit 112 or the element review unit 115. The control unit 110 may also link with tools that control digital fabrication equipment such as the laser processing machine 24, cooking robot 26, and cutting machine 28, and control these devices. The control unit 110 may also link with various AI tools, such as image generation AI.

(Storage unit 150)
The storage unit 150 is realized by a ROM that stores programs and calculation parameters used in the processing of the control unit 110, and a RAM that temporarily stores parameters that change as needed.

For example, the storage unit 150 may store a cooking learning model and an ingredient pairing learning model used by the draft generation unit 112 and the adjustment unit 114. The storage unit 150 may also store basic information about the user (such as recipe data registered by the user and cooking environment information). The storage unit 150 may also store various information output during the process of creating a new dish, such as the dish draft generated by the draft generation unit 112, the dish draft adjusted by the adjustment unit 114, and the results of element review by the element review unit 115.

(Audio input/output unit 160)
The voice input/output unit 160 has a voice input unit that collects voice and inputs the voice data to the control unit 110, and a voice output unit that outputs voice. The voice input unit is realized by, for example, a microphone and collects user voice. The voice output unit is realized by, for example, a speaker and outputs agent voice.

The configuration of the information processing device 10 has been specifically described above. Note that the configuration of the information processing device 10 according to the present disclosure is not limited to the example shown in FIG. 2. For example, the information processing device 10 may be realized by multiple devices. Furthermore, at least some of the functions of the information processing device 10 may be realized by a server on the Internet. For example, each functional component of the control unit 110 may be provided on a server. In other words, each component of the information processing device 10 may be realized by an information processing system consisting of a user terminal and a server.

<<3. Operation Processing>>
Next, the operation process of the cooking information generation system 1 according to this embodiment will be described with reference to FIG.

Figure 5 is a flowchart showing an example of the overall flow of the cooking information generation system 1 according to this embodiment.

As shown in FIG. 5, first, the information processing device 10 performs an input process for user basic information (step S103). The user basic information may be input by user operation from the operation input unit 130, or may be input through interaction between the user and an agent presented by the agent unit 111.

Next, the information processing device 10 generates one or more recipe drafts using the draft generation unit 112 in response to the user's suggestions regarding the new creative dish, which are obtained from the dialogue between the agent and the user (step S106).

Next, the information processing device 10 presents one or more recipe ideas to the user on the display unit 140 (step S109).

Next, the information processing device 10 accepts the user's selection of a recipe draft and further acquires any new suggestions from the user (step S112).

Next, the information processing device 10 causes the adjustment unit 114 to perform adjustment processing on the recipe draft (step S115).

Next, the information processing device 10, using the element review unit 115, divides the recipe data included in the recipe draft into elements corresponding to the cooking process, and performs a process to review the feasibility of each element (step S118). Specifically, one or more pieces of cooking and processing information are generated for each element.

Next, the information processing device 10 performs an integration process using the integration unit 116 to integrate the generated elements (each piece of cooking and processing information) (step S121), and then displays the integrated cooking and processing information (new cooking information) generated by the integration process on the display unit 140 to present it to the user (step S124).

Next, if the user inputs new comments about the presented integrated cooking and processing information (step S127/Yes), it is determined that corrections are required, and the process returns to step S115 (adjustment processing), step S118 (element review processing), or step S121 (integration processing) as appropriate, and corrections are made. The control unit 110 determines at which stage to start corrections depending on the content of the new comments.

If there are no user-specified items (step S127/No), the information processing device 10 performs a process of registering the presented integrated cooking and processing information (step S130). The information processing device 10 may also register the integrated cooking and processing information selected by the user from the multiple pieces of presented integrated cooking and processing information.

Note that the operational processing shown in FIG. 5 is an example, and all steps shown in FIG. 5 do not necessarily have to be performed in the order shown in FIG. 5. For example, the information processing device 10 may skip steps S109 and S112 and have the control unit 110 arbitrarily select a recipe draft. Furthermore, after the adjustment processing shown in step S115, the information processing device 10 may present the adjusted recipe draft to the user, and perform the adjustment processing again if new suggestions are input. Furthermore, after the element review processing shown in S118, the information processing device 10 may present the review results to the user, and perform the element review processing again if new suggestions are input. Furthermore, before step S121, the information processing device 10 may present multiple cooking processing information (or processing results) as candidates and accept the user's selection, and in step S121, perform integration to include the cooking processing information selected in accordance with the user's selection.

The following describes each process shown in Figure 5 in detail.

<3-1. Input process of basic information>
FIG. 6 is a flowchart showing an example of the flow of the basic information input process according to this embodiment. When the cooking information generation system 1 according to this embodiment (an AI tool that assists users in creating new dishes) is launched, it first checks whether the user's basic information has already been entered. If the necessary basic information has not been entered, the system executes the basic information input process described below. This system may be executed, for example, by an application installed on an information processing device 10. The user's cooking environment may also include various cooking appliances (e.g., a laser processing machine 24, a cutting machine 28, a heating device (e.g., a pot, a microwave, a frying pan, etc.)), a camera 20, an ingredient sensor 22, a robot (e.g., a cooking robot 26), etc. that can be controlled by this system.

As shown in FIG. 6, in the information processing device 10, the display control unit 113 presents an input prompting screen on the display unit 140 that prompts the user to input information about the dish (basic information), and accepts input of the basic information by the user (step S203). The basic information may be input through dialogue with an agent presented by the agent unit 111. For example, the agent unit 111 displays the agent on the display unit 140 via the display control unit 113, and performs control to start dialogue with the user via the voice input/output unit 160.

The necessary basic user information includes, for example, the user's favorite dishes, specialty dishes (French cuisine, etc.), past cooking information (recipe data, cooking images (captured images)), and cooking environment information (main utensils used, special cooking utensils, etc.). Recipes and cooking images included in past cooking information may be tagged with recipe categories (such as the purpose of the cooking, such as for Christmas, or information about the target demographic, such as for athletes). Past cooking information may be obtained, for example, from a recipe management system used by the user, or from a database specified by the user.

Next, the control unit 110 stores the entered user's basic information in the storage unit 150 (step S206).

Next, the control unit 110 begins the learning process of the user's cooking information. First, the control unit 110 prepares the data to be used for learning. Specifically, the control unit 110 estimates the contents of the dish (step S209) and the tableware used (step S212) based on the input dish image and recipe data.

FIG. 7 is a diagram for explaining the estimation of dish contents based on the user's dish information according to this embodiment. As shown in FIG. 7, the input user's dish information 44 includes a dish image 441 and recipe data 442. The control unit 110 analyzes the dish image 411 to perform image recognition (recognition of the objects shown in the image) and estimates the dish contents by referring to the recipe data. Specifically, the control unit 110 associates each dish item shown in the dish image 411 with the recipe data. In the example shown in FIG. 7, association information 45 with the recipe data can be obtained for each image-recognized dish item, such as "Recipe 2: Fried shrimp," "Recipe 3: Meat," and "Recipe 4: Side dish." The control unit 110 also estimates the tableware used (i.e., the tableware owned by the user) from the image-recognized dish portion, such as "round plate, medium size."

Next, the control unit 110 uses the user's cooking information (recipe data, dish images) and the estimated dish contents and tableware used obtained from the dish information as input data (learning data) to learn about the user's cooking and generate a user cooking learning model 521 (step S215). Note that the input data is not limited to the above example. For example, recipe data may include information on ingredients and cooking utensils used, but if calorie information is not included, the control unit 110 may access an external ingredient database to obtain calorie information as external information and add it to the recipe data as input data. Also, as described above, recipe data and dish images may have recipe categories (purpose of cooking, genre, target demographic, etc.) added as tags, and such recipe categories are also input for learning. The learning method is not particularly limited, but machine learning such as reinforcement learning is one example. The generated user cooking learning model 521 is stored in the storage unit 150.

In this way, by preparing a learning model that has learned from the user's cooking, the recipe generation unit 112 can generate a recipe draft based on the user's cooking (for example, in line with the characteristics of the user's cooking) in response to input data (user's suggestions (purpose of cooking, genre, ingredients, etc.)).

Although the generation of the user cooking learning model 521 has been described here, in this embodiment, an other user cooking learning model 522 generated by learning the cooking of many users (chefs, etc.) can also be prepared in advance. The learning data used in the other user cooking learning model 522 is not particularly limited, but may be obtained, for example, from a recipe management system (application) used by many chefs, or from a database storing chef recipe collections.

<3-2. Recipe draft generation process>
FIG. 8 is a flowchart showing an example of the flow of a recipe plan generation process according to this embodiment.

As shown in FIG. 8, first, the agent unit 111 of the information processing device 10 inputs the user's suggestions, acquired from a dialogue with the user, into the draft generation unit 112 (step S303). The user's suggestions are matters (requests) related to the dish the user wants to make, and in this embodiment, these are assumed to be the concept and ingredients to be used. The agent unit 111 may input the suggestions (concept, ingredients to be used) obtained by analyzing the user's dialogue (content converted into text by voice recognition or text entered by the user) using natural language processing directly into the draft generation unit 112 as a prompt, or may input a prompt generated by supplementing information as appropriate.

The contents of the input data (prompt) input to the draft generation unit 112 can be, for example, "purpose of cooking (Christmas dish, party dish, etc.), ingredients (potatoes, cheese, seafood, etc.), genre (appetizer, stew, Chinese food, etc.), nutrition (low calorie, high calorie, etc.), target demographic (children, athletes, etc.), cooking equipment used (oven cooking, 3D printer, etc.), design (geometric design, gentle design, flashy design, etc.)". The agent unit 111 may supplement the details of the concept input by the user through dialogue, or may supplement the information by eliciting necessary information by asking questions such as "What kind of customer base are you looking for?" The agent unit 111 may also randomly fill in any missing information.

Next, the draft generation unit 112 uses the ingredient pairing learning model in the ingredient pairing unit 510 to output ingredients (combined ingredients) that are suitable for combination with the ingredients specified by the user (included in the above-mentioned recommendations) (step S306).

Next, the recipe plan generation unit 112 uses the cooking learning model to generate a recipe plan using the recipe learning model by the recipe plan generation unit 520 based on the inputted recommendations and ingredient combinations (step S309). As described above, the cooking learning model includes the user's cooking learning model and other users' cooking learning models. The recipe plan generation unit 520 can generate recipe plans with different levels of freshness using the cooking learning model as appropriate. The recipe plan generation process will be described later with reference to Figure 12.

Note that recommendations may include multiple items, multiple contents per item, or multiple combined ingredients. The recipe plan generation unit 112 may generate multiple input data by randomly combining the information contained in the recommendations and combined ingredients. This makes it possible to generate recipe plans that correspond to various combinations based on the user's recommendations, etc. The generated recipe plans are expected to include recipe data and dish images. A user learning model that has learned the recipe data and dish images (captured images) may generate dish images that correspond to the recipe data. Furthermore, the dish images may be generated by an image generation AI according to the generated recipe data. For example, the recipe plan generation unit 112 may generate an appropriate prompt from the results obtained by text analysis of the recipe data, and input this into the image generation AI to obtain dish images.

The generated recipe plan is displayed on the display unit 140 by the display control unit 113 and presented to the user (step S312).

FIG. 9 is a diagram showing an example of a display screen on which a recipe plan is displayed according to this embodiment. As shown in FIG. 9, a large number of recipe plans are displayed on display screen 600. Here, thumbnails of the dish images included in the recipe plan are displayed. When the user taps on any of the dish images, details of the recipe plan, i.e., the contents of the recipe, are displayed.

FIGS. 10 and 11 are diagrams illustrating the display screen that displays details of a recipe plan. When a user selects any dish image from the recipe plan list screen as shown in FIG. 9, the screen switches to a recipe plan details display screen 610a, as shown on the left of FIG. 10. The details display screen 610a displays recipe data 611a, a dish image 612a, and a suggested items display area 613a. The recipe data 611a includes ingredients (foodstuffs, seasonings), cooking steps, etc.

The user can review the presented recipe and food images, and at this point make further suggestions and revise the presented dish draft based on the initial user suggestions. An example of how to input suggestions is explained below.

For example, as shown on the right side of Figure 10, if a fruit in cooking image 612b is selected by touch operation (e.g., by circling it) and the user says, "I want to make this melon into a spring-like fruit," the agent unit 111 will recognize this and the recognized suggestion, "Make it into a spring-like fruit," will be displayed in the suggestion display area 613a. The agent unit 111 may also mark the relevant part of the recipe that corresponds to the part selected by the user (e.g., "Decorate the cake with melon").

Next, as shown on the left side of Figure 11, if the user selects the plate in the cooking image 612c by touch operation and says, "I'd like to add more decorations to this area to make it more colorful," the agent unit 111 will recognize this and display the recognized suggestion, "Add more decorations to make it more colorful," in the suggestion display area 613c. The agent unit 111 may also mark the relevant part of the recipe that corresponds to the part selected by the user (for example, "Place the cake on the plate").

Then, in the information processing device 10, the adjustment unit 114 adjusts the recipe draft in accordance with the newly input suggestions. Details of the adjustments will be described later with reference to FIG. 13. The right side of FIG. 11 shows a display screen displaying the recipe draft adjusted in accordance with the suggestions input by the user. Display screen 610d displays recipe data 611d that has been changed due to the adjustments, a new dish image 612d generated in accordance with the adjustments, and an explanation area 613d that explains the adjustments. The changed parts may be marked in the recipe data 611d.

(Generating recipe ideas with different levels of novelty)
Next, we will explain in more detail how the recipe plan generator 112 generates a recipe plan using the cooking learning model. As described above, the recipe plan generator 112 can generate a recipe plan by appropriately using the user's cooking learning model and the other user's cooking learning model. When the user's cooking learning model is used, it is expected that a recipe plan that is in line with (similar to) the characteristics of the user's cooking will be generated, and when the other user's cooking learning model is used, it is expected that a recipe plan that is different from (not similar to) the characteristics of the user's cooking will be generated, that is, at least more novel to the user.

The draft generation unit 112 can use a cooking learning model as appropriate to control the generation rate of recipe drafts with different levels of novelty, depending on the level of novelty of the recipe desired by the user.

Figure 12 is a flowchart showing an example of the process flow for generating recipe plans with different levels of freshness according to this embodiment.

As shown in FIG. 12, first, the draft generation unit 112 checks the newness level of the dish desired by the user (step S323). The newness level of the dish desired by the user may be obtained by the agent unit 111 from a conversation with the user. It may also be input by the user during the initial basic information input process.

Next, the plan generation unit 112 determines the ratio of the number of recipe plans A to D to be generated depending on the newness level (step S326). Recipe plans A to D are plans with different recipe newness levels, with the relationship being newness level a of recipe plan A < newness level b of recipe plan B < newness level c of recipe plan C < newness level d of recipe plan D. The plan generation unit 112 controls so that the higher the newness level, the more recipe plans with higher newness levels are generated.

Next, the draft generation unit 112 sets the information (prompt) about the concept and ingredients (including combination ingredients) indicated by the user as input data for the cooking learning model (step S329).

Next, the recipe plan generation unit 112 generates a recipe plan A using the user recipe model (step S332).

The recipe plan generation unit 112 also uses another user's cooking model to change the style (here, the appearance of the dish) of the recipe plan generated using the user's cooking model, generating a recipe plan B (step S335). In this specification, "changing the style of the dish" primarily means changing the appearance of the dish, such as the presentation, arrangement, and decoration of the dish. More specifically, the dish image included in the recipe plan is changed. Even with the same recipe, the impression of the dish can change if the presentation, decoration, tableware used, etc. are different. The appearance of such dishes also reflects the chef's characteristics, and by changing it using another user's cooking model, it becomes possible to create a new dish (that is less similar to the user's dish).

The recipe plan generation unit 112 also generates a recipe plan C by changing the style of the recipe plan generated using the other user's recipe model using the user's recipe model (step S338).

The recipe plan generator 112 also generates a recipe plan D using other users' recipe models (step S341).

In the generation of the recipe plans A to D described above, a large number of recipe plans A to D are generated. The plan generator 112 may generate recipe plans A to D according to the ratio determined in step S326 above.

Then, the plan generation unit 112 outputs the recipe plans A to D (the number of recipe plans corresponding to the ratio) according to the determined ratio (step S344). Each recipe plan is displayed on the display unit 140 by the display control unit 113.

The above describes how to generate recipe drafts according to newness levels. Note that the "newness level of the dish desired by the user" is not necessarily required, and the control unit 110 may set the newness level at will.

<3-3. Adjustment processing>
Next, the adjustment process according to this embodiment will be described in detail. In the adjustment process, adjustments are made to the recipe draft. In this embodiment, the user can input new suggestions after viewing the presented recipe draft. Since a user's ideas may change during the creative process, this embodiment allows the user to input new suggestions to a recipe draft generated using a cooking learning model based on the suggestions initially input by the user, thereby realizing collaboration between the user and AI. The information processing device 10 regenerates (modifies) a recipe draft using the cooking learning model based on the new suggestions. Note that by generating and presenting multiple recipe drafts based on the initial suggestions and allowing the user to select at least one from these, it is possible to narrow down the candidates to those that more closely match the user's intentions. This also reduces the amount of calculation required in subsequent processes.

FIG. 13 is a flowchart showing an example of the flow of adjustment processing according to this embodiment.

As shown in FIG. 13, if the user provides new instructions for the recipe draft (step S403/Yes) and these are changes to ingredients (step S406/Yes), the adjustment unit 114 uses the ingredient pairing learning model to output new ingredients that are suitable for pairing with the changed ingredients in response to the ingredient change (step S409). If the ingredient change is not a specific ingredient request (for example, "I want to change to a spring-like fruit," "I want to use ingredients with a good texture," "I want to increase the variety of vegetables," etc.), the adjustment unit 114 may determine ingredients using an appropriate database or AI (such as a pre-trained learning model for determining ingredients) in response to an abstract request such as "spring-like fruit."

Next, the adjustment unit 114 uses the ingredients used, the combined ingredients, and the user's new comments as input data and performs an adjustment process on the recipe data in the draft dish using the cooking learning model (step S412). Comments other than changes to ingredients include, for example, comments on the cooking process or cooking contents. The adjustment unit 114 also performs an adjustment process on the appearance of the dish using the cooking learning model in accordance with the user's new comments (for example, comments on the layout, design, etc. of the dish) (step S415). Specifically, the adjustment unit 114 performs a conversion process on the dish image. The adjustment unit 114 may also convert the dish image to correspond to the revised recipe data.

Finally, the adjustment unit 114 adjusts the recipe data to accommodate the user's cooking environment (step S418). For example, the adjustment unit 114 assigns cooking robots, laser processing machines, etc. to the cooking steps of a recipe based on the information about the user's cooking environment input as basic information. Figure 14 is a diagram showing an example of adjustment according to the user's cooking environment according to this embodiment. Here, we will explain using the recipe for "Mushroom Farci" generated based on the user's input that "I want to make a beautiful stuffed dish using vegetables."

As shown in recipe data 620 in the upper row of Figure 14, the cooking steps of the recipe are "1. Remove the stems from the mushrooms, 2. Add a pattern to the surface of the mushrooms, 3. Stuff the mushrooms with a filling (cheese, nuts, etc.), 4. Arrange on a plate and garnish with leaves," and if the user's cooking environment is equipped with a cooking robot, laser processing machine, etc., the adjustment unit 114 assigns the cooking robot, etc. to the cooking steps. Specifically, as shown in recipe data 621 in the lower row of Figure 14, the recipe data 620 is modified to "1. The robot cuts off the stems from the mushrooms, 2. The robot engraves a pattern on the surface of the mushrooms with a laser, 3. The robot stuffs the mushrooms with a filling (cheese, nuts, etc.), 4. The robot arranges on a plate and garnishes with leaves."

<3-4. Element review processing>
Next, the element review process will be described. Fig. 15 is a flowchart showing an example of the flow of the element review process according to this embodiment. Here, the feasibility of cooking is reviewed for each element of the recipe draft generated by the draft generation unit 112 or the recipe draft adjusted by the adjustment unit 114.

As shown in FIG. 15, first, the element review unit 115 of the information processing device 10 considers each cooking step of the recipe data included in the recipe draft as an element (step S503).

Next, the element review unit 115 performs a process to review the feasibility of the cooking process for each element (step S506). When reviewing elements, the content is derived (specified) (for example, in the case of "cutting carrots into the shape of a flower," variations of the flower shape are generated), and a cooking simulation is performed, or actual cooking is performed using a cooking robot, etc. Specific examples of such element review will be described later with reference to Figures 16 to 19.

Next, if there are any elements that are all unsuccessful, the element review unit 115 requests the adjustment unit 114 to adjust the recipe data (step S512) and performs the element review process according to this embodiment again.

Next, a specific example of element consideration will be explained. For example, in the case of a "Mushroom Farci" recipe as shown in Figure 14, recipe data 621 is divided into four elements (elements 1 to 4) according to the cooking process: "1. The robot cuts off the stems of the mushrooms," "2. The robot engraves a pattern on the surface of the mushrooms with a laser," "3. The robot stuffs the mushrooms with a filling (cheese, nuts, etc.)," and "4. The robot arranges the food on a plate and garnishes with leaves." The consideration of these elements will be explained in order below.

(Consideration of element 1)
The examination of the cooking process of element 1, "The robot cuts off the stems of the mushrooms," will be described with reference to Fig. 16. Fig. 16 is a flowchart showing the flow of the examination process of element 1.

As shown in FIG. 16, first, the element review unit 115 checks whether the method for using a robot to cut off mushroom stems has already been learned (step S523). If it has already been learned (step S523/Yes), processing proceeds to step S532.

If the model has not yet been learned (step S523/No), the element review unit 115 inputs the 3D model of the mushroom into the simulator (step S526) and learns how to remove the stem from the mushroom using the robotic hand (cooking robot 26) in the user's environment (step S529). The 3D model of the mushroom may be obtained from an external source, or the mushroom may be captured in 3D using the camera 20. For example, the element review unit 115 can use a simulator corresponding to the robotic hand in the user's environment to learn how to remove the stem from the mushroom using machine learning such as reinforcement learning, thereby exploring how to remove the stem from the mushroom more efficiently (simulation optimization). The learning model generated here (an example of a cooking and processing learning model) can be used when resimulating how to remove the stem from the mushroom.

Next, the element review unit 115 obtains the cooking time required (step S532). For example, the element review unit 115 obtains the cooking time required for the successful axis selection method.

Next, if the simulation results show that all cooking and processing plans (cooking and processing information) for the cooking step of element 1, "removing mushroom stems," are unsuccessful despite being considered in the user's cooking environment (step S535/Yes), the element review unit 115 requests the adjustment unit 114 to adjust the recipe data (step S538). The adjustment unit 114 may make adjustments, such as changing the recipe to one where removing mushroom stems is a task that is difficult for the cooking robot 26 to perform and therefore a task that must be done by a human, changing the recipe to one that does not require removing the stems, or changing the ingredients to ones whose stems are easier to remove. Note that if any of the cooking and processing plans for the cooking step of element 1 are successful (step S535/No), the processing ends. In the integration process described below, the successful cooking and processing plans are used when integrating the elements.

(Consideration of element 2)
The examination of the cooking process of element 2, "carving a pattern on the surface of the mushroom with a laser," will be described with reference to Fig. 17. Fig. 17 is a flowchart showing the flow of the examination process of element 2.

As shown in FIG. 17, first, the element review unit 115 checks whether engraving a mushroom with a laser cutter (laser processing machine 24) has been learned (step S543). If learning has been learned (step S543/Yes), processing proceeds to step S567.

If learning has not been performed (step S543/No), the element review unit 115 generates n pattern patterns (step S546). For example, the element review unit 115 generates pattern variations based on the food image output from the adjustment unit 114 using an algorithm such as style conversion.

Next, the element review unit 115 defines m sets of laser processing parameters (step S549). For example, the element review unit 115 sets the power and speed ranges for laser control as "power: minimum 40, maximum 60" and "speed: minimum 50, maximum 90," and defines m combinations of power and speed. The element review unit 115 may refer to processing examples of similar ingredients, or may define parameters starting with low power parameters for safety reasons.

Next, the element review unit 115 selects a set of pattern pattern and processing parameters (step S552).

Next, the element review unit 115 uses the robot (cooking robot 26) to set the mushroom (in a location where it can be processed using a laser cutter) (step S555), and controls the laser cutter to process (carve) it (step S558).

Next, the element review unit 115 photographs the processed food material using the camera 20 and acquires the image (step S561).

The above processing shown in steps S552 to S561 is performed for all combinations of pattern patterns and processing parameters (step S562). This results in a large number of processing results (images of processed ingredients), such as those shown in Table 2 below.

If all patterns have been examined (step S403/Yes), the element review unit 115 learns the relationships between the pattern pattern, processing parameters, and processed image based on these, and creates a learning model for predicting the processing result from the pattern pattern and processing parameters (step S564). Even with the same pattern pattern, the processing result can be clear, crushed, or thin, depending on the processing parameters. This is because the processing reaction may not be linear depending on the ingredients of the ingredients, and phenomena such as the ingredients expanding or crumbling may occur. In this embodiment, a learning model (an example of a cooking processing learning model) is used to predict the processing result from the pattern pattern and processing parameters, making it possible to consider many combinations of pattern patterns and processing parameters. The learning model can be generated by machine learning such as reinforcement learning, for example.

Next, the element review unit 115 evaluates each cooking and processing plan (a combination of pattern pattern and processing parameters: an example of cooking and processing information) based on the degree of match (whether the pattern is distorted by processing) between the pattern generated from the pattern in element 2 and the processing result (predicted by the learning model) (step S567).

The element review unit 115 also obtains the cooking time required (for each combination pattern) (step S570).

If all of the evaluations performed in step S567 above are low (step S573/Yes), the element review unit 115 requests the adjustment unit 114 to adjust the recipe data (step S576). The adjustment unit 114 changes the ingredients or the pattern, for example. Note that if there is a cooking and processing plan for the cooking step of element 2 that has a rating above a certain level (step S573/No), the process ends. In the integration process described below, cooking and processing plans that have a rating above a certain level are used when integrating elements.

(Consideration of element 3)
The examination of the cooking process of element 3, "Robot stuffs mushrooms with fillings (cheese, nuts, etc.)," will be described with reference to Fig. 18. Fig. 18 is a flowchart showing the flow of the examination process of element 3.

As shown in FIG. 18, first, if the user has instructed to restrict the filling to be stuffed into the mushrooms (step S583/Yes), the element review unit 115 sets restrictions on the filling material (step S586). Restrictions from the user may be extracted from a conversation between the agent and the user, such as "I want to make a dish for customers with allergies" or "I want to reduce the calories." Restrictions on the filling material may include, for example, controlling the ingredients and quantity. If the user has not instructed to restrict the filling (step S583/No), the process proceeds to step S589.

Next, the element review unit 115 creates multiple patterns of ingredients to be mixed as fillings (step S589).

Next, the element review unit 115 imports the 3D model of the mixed material into a simulator, and uses the simulator to learn how the robot hand fills mushrooms with the filling material according to the mixing pattern (step S592). The generated learning model is an example of a cooking and processing learning model. The learning model can be generated by machine learning, such as reinforcement learning.

Next, the element review unit 115 obtains the cooking time required (step S595).

If all of them are unsuccessful (step S598/Yes), the element review unit 115 requests the adjustment unit 114 to adjust the recipe data (step S601). The element review unit 115 can use the generated learning model to consider the feasibility (whether or not it will be successful) of each mixing pattern. Note that if there is a successful cooking/processing plan among the cooking steps for element 3 (step S598/No), the processing ends. In the integration process described below, the successful cooking/processing plan is used when integrating the elements.

(Consideration of element 4)
The examination of the cooking process of element 4, "The robot arranges the food on a plate and garnishes it with leaves," will be described with reference to Fig. 19. Fig. 19 is a flowchart showing the flow of the examination process of element 4.

As shown in FIG. 19, first, the element review unit 115 generates variations of food images in which food is served on tableware that the user normally uses (step S603). Information about the tableware used by the user has already been acquired during the basic information input process.

Next, when decorating a plate with leaves or flowers, the element review unit 115 generates variations of 3D models of the decorations (step S606). For example, in the case of leaves, many different types of leaves are often combined for decoration, so 3D models of many different types of leaves are generated. Note that the generation of variations of 3D models may be performed using a tool such as 3D CAD that is linked to the tool. Here, the case of decorating with leaves will be explained below.

Next, the element review unit 115 generates a leaf arrangement pattern (step S609). The element review unit 115 generates an arrangement pattern by, for example, combining multiple types of leaves.

Next, the element review unit 115 uses a simulator to learn possible arrangements for the robot hand (cooking robot 26) (step S612). The generated learning model is an example of a cooking and processing learning model. The learning model can be generated by machine learning, such as reinforcement learning.

Next, the element review unit 115 deletes placement patterns that are difficult to realize (step S615). The element review unit 115 can use the generated learning model to consider the feasibility (whether or not each placement pattern will be successful) of each placement pattern.

Next, the element review unit 115 generates food images by combining variations of the above food image of food served on tableware with leaf layouts of each learned arrangement pattern (specifically, arrangement patterns that are likely to be realized by the robot hand) (step S618).

The element review unit 115 also obtains the cooking time required (for each arrangement pattern) (step S621).

If all of the cooking and processing plans are unsuccessful (i.e., there is no feasible arrangement pattern) (step S624/Yes), the element review unit 115 requests the adjustment unit 114 to adjust the recipe data (step S627). If there is a successful cooking and processing plan among the cooking process plans (arrangement pattern: an example of cooking and processing information) for element 4 (step S624/No), the process ends. In the integration process described below, the successful cooking and processing plan is used when integrating the elements.

The above describes specific examples of the consideration of each element. In this way, in the element consideration process, more specific variations of the cooking process (cooking processing information) are created based on each cooking process of the recipe data in the dish draft, and a consideration is made as to whether the cooking process will be successful. Furthermore, in learning the cooking processing learning model used in the consideration of each element described above, the information processing device 10 can further improve the accuracy of generating the learning model by including failure cases in the learning data. Note that the element consideration unit 115 may present multiple cooking processing information (or processing results) for each element as candidates to the user, allowing the user to select the cooking processing information to be included in the integration.

<3-5. Integration processing>
20 is a flowchart showing an example of the flow of the integration process according to this embodiment. The integration unit 116 combines the elements reviewed by the element review unit 115 to generate integrated cooking and processing information.

As shown in FIG. 20, first, if there are instructions from the user regarding the integration conditions (step S703/Yes), the integration unit 116 changes the default conditions in accordance with the user instructions (step S706). For example, user instructions may include conditions such as a cooking time of 15 minutes or less and an evaluation rank of 3 or higher for element 2. Also, user instructions may include the user's selection of cooking and processing information to be included in the integration.

On the other hand, if there are no instructions from the user regarding the integration conditions (step S703/No), the integration unit 116 sets the default conditions (step S709).

Next, the integration unit 116 generates combination patterns of each element that satisfy the conditions (step S712).

Next, the integration unit 116 generates an integrated recipe and corresponding dish image as integrated cooking and processing information based on each combination pattern (step S715).

Finally, the integration unit 116 adjusts the procedural efficiency and safety of the integrated recipe (step S718). For example, in the case of a recipe for "mushroom farci," if the mixed filling ingredients need to be left to soak in the refrigerator for a while, the integration unit 116 can shorten the overall cooking time by adjusting the process so that this step is carried out first. Furthermore, in consideration of food safety, the integration unit 116 adds a step in which ingredients that require temperature control are temporarily placed in the refrigerator while the cooking robot performs time-consuming processing and arrangement. Knowledge of procedural efficiency and safety may be acquired through cooking learning, or by linking with an external database, etc.

<3-6. Presentation process>
Next, a process for presenting integrated cooking and processing information according to this embodiment will be described. The integrated recipe and the corresponding dish image integrated by the integration unit 116 are an example of integrated cooking and processing information.

The display control unit 113 may display each dish image as a thumbnail as an example of the presentation of the integrated cooking and processing information. Figure 21 is a diagram showing an example of the presentation of the integrated cooking and processing information according to this embodiment. As shown in Figure 21, the dish images are displayed as thumbnails on the presentation screen 640.

Details of the presentation of integrated cooking and processing information are explained below. Figure 22 is a flowchart showing an example of the flow of the process for presenting integrated cooking and processing information according to this embodiment.

As shown in FIG. 22, first, if a user instruction for a sauce condition is given (step S803/Yes), the display control unit 113 sorts the dish images according to the condition (step S806). For example, this may be in order of cooking time.

On the other hand, if no sauce conditions are specified by the user (step S803/No), the display control unit 113 sorts the dish images, for example, in the order in which they were created (step S809).

Next, the display control unit 113 displays a list of food images (step S812). A specific example is as shown in Figure 21.

Next, if a dish image is selected by the user (step S815/Yes), the display control unit 113 displays details of the selected dish (step S818).

Next, if a component (ingredient) of a dish included in the dish image is pointed to on the details display screen for the selected dish (step S821/Yes), the display control unit 113 displays details of that component (step S824).

FIG. 23 is a diagram showing an example of a details display screen for a selected dish according to this embodiment. As shown in FIG. 23, the details display screen 650 displays a dish image 651, element information display 652 for the selected dish, and a detailed display 653 of the cooking and processing pattern.

The element information display 652 displays not only the cooking and processing patterns that were adopted, but also the cooking and processing patterns that were not adopted. For example, if "Cooking Process 2" of element 2 is selected from the elements of the dish "Mushroom Farci" selected in the dish image 651, the "Pattern Pattern" and "Processing Parameter" blocks are displayed. If "Pattern Pattern" is selected, the detailed display 653 displays a large number of pattern patterns (examples of cooking and processing information) generated by element review in the element review unit 115, including those that were not adopted in the dish, as shown in FIG. 23.

Furthermore, when "Processing parameters" is selected, the detailed display 653 displays a large number of processing parameters (an example of cooking processing information) generated during element review in the element review unit 115, including those not used in the dish. At this time, an image after processing for each processing parameter of each pattern pattern (the processing result predicted by the learning model) may also be displayed. The user can confirm what the processing result will be when which pattern pattern is processed with which processing parameter.

In this way, the user can check the pattern patterns that were derived and generated in the element review unit 115 but not adopted, the numerous processing parameters that were not adopted, and the processing results according to the processing parameters. While checking these, the user can make more specific suggestions such as, "The shape will be a little distorted, but I think this color (burnt color) would be good," allowing for an efficient review cycle. In other words, the user can select any cooking and processing information from the many pieces of cooking and processing information presented, and instruct the system to generate integrated cooking and processing information that includes the selected cooking and processing information. This embodiment assumes an iterative process in which corrections are made each time the user makes a new suggestion, resulting in a final proposal.

Specifically, as shown in FIG. 22, if the user inputs a comment (step S827/Yes), the agent unit 111 acquires the user's new comment through a dialogue between the agent and the user (step S830).

The control unit 110 then summarizes the user's suggestions and presents them to the user so that the user can confirm them, and begins corrections (step S833). Specifically, as shown in FIG. 5, the adjustment process in the adjustment unit 114, the review process in the element review unit 115, and the integration process by the integration unit 116 are performed again as appropriate, and the corrected cooking and processing information is presented.

Examples of new user feedback and corrections include the following:

<<4. Other>>
The various learning models used in the cooking information generation system 1 according to this embodiment may be realized by LLMs (large-scale language models). That is, the above-described user cooking learning model 521, other users cooking learning model 522, ingredient pairing learning model, and cooking processing learning model may be configured to use LLMs as knowledge databases.

Furthermore, when training various learning models in this implementation, the information processing device 10 may use failure cases for training. This makes it possible to improve the accuracy of generating learning models.

The integration unit 116 can also set personalization conditions, which are conditions for changing the content of elements (cooking and processing information) for each individual receiving the food, in accordance with input from the user during the integration process. For example, the integration unit 116 sets personalization conditions that determine the type, amount, arrangement, or processing state of ingredients in specified elements, based on each individual's preferences or calorie intake. The integration unit 116 generates an integrated recipe that meets the personalization conditions set by the user. Furthermore, during cooking, the display control unit 113 may rearrange the configuration of corresponding elements in accordance with the personalization conditions specified by the user and present the integrated recipe.

In the above-described embodiment, as shown in Figures 21 and 23, a food image is displayed on the screen as a method of presenting the generated cooking and processing information to the user, but this embodiment is not limited to this, and the information processing device 10 may actually cook a high-ranked dish using a cooking robot 26 or the like and present it to the user.

In this embodiment, the story generation unit 117 can also generate a development story. During the process of creating a dish, various data is accumulated in the memory unit 150, such as dialogue between the agent and the user, adjustments made, and element considerations. When the dish is completed (when there are no new issues to be pointed out and it has been finalized), the story generation unit 117 picks out catchphrases and preferences that reflect the user's characteristics from the data accumulated in the memory unit 150, extracts unusual corrections made in response to the user's suggestions, re-learning, and episodes of failure and success (such as failed/successful cooking and processing methods), compiles a development story, and outputs it as text or images.

In addition, in this embodiment, the initial recipe draft is prepared taking into consideration any user comments, but adjustments, element review, and integration are performed automatically and mechanically, and the result is presented to the user. Here, dishes are often complex and require multiple ingredients, so the chef's sense is also important in creating delicious dishes. Therefore, as a method for creating new dishes through better collaboration between people and AI tools, manually collected data may be used as appropriate.

For example, when the adjustment unit 114 adjusts the "Mushroom Farci" recipe to suit the user's cooking environment, if it determines that cooking step 3 (element 3): filling and cooking step 4 (element 4): plating are difficult for the cooking robot in the user's environment to handle, or if it determines that manual intervention is better, it may make the recipe one that incorporates manual intervention. Consideration of recipe elements when manual intervention is used is carried out, for example, as follows:

When considering cooking processes (elements) that require manual labor, actual personnel are assigned and the study is carried out. The information processing device 10 may select personnel by acquiring the profiles (areas of expertise, etc.) and schedule information (free time) of assignable staff.

In considering element 3, the element review unit 115 generates n filling mixture patterns and has the assigned staff member create (cook) them. At this time, the cooking time can be obtained using a timer or the like. The element review unit 115 may also have the staff member think up filling mixture patterns. When cooking manually, people tend to proceed without recording the cooking process, so the agent unit 111 interacts with the staff member and hears from them about mixing points, recommended mixing patterns (rankings), etc., and stores them. The element review unit 115 may also take images of the fillings cooked by the staff member and create a list in order of the staff member's ranking. The staff member may evaluate whether they can create the fillings according to each mixing pattern, and also evaluate the appearance and taste of the created fillings, and input this information into the information processing device 10.

In considering element 4, the element review unit 115 creates n plating patterns and has the assigned staff create (cook) them. The element review unit 115 may also have the staff think about the plating patterns. For example, the staff may be asked to select the plate on which the food will be served. The acquisition of cooking time, interviews with staff, and storage of the cooking process are the same as in the consideration of element 3 above. The staff may evaluate whether they can plate the food according to each plating pattern, and input this into the information processing device 10.

Elements manually considered in this way are also used in the integration unit 116.

In addition, the cooking process of each staff member is also stored in the storage unit 150 and can be incorporated into the development story generated by the story generation unit 117, making it possible to visualize the staff member's contributions.

<<5. Supplementary Information>>
Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

The above-described embodiments can be used, for example, by chefs and culinary experts when creating new dishes.

In addition, one or more computer programs can be created to cause the hardware, such as the CPU, ROM, and RAM, built into the information processing device 10 to perform the functions of the information processing device 10. Also provided is a computer-readable storage medium storing the one or more computer programs.

Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may achieve other effects in addition to or in place of the above-mentioned effects that would be apparent to those skilled in the art from the description herein.

The present technology can also be configured as follows.
(1)
A process of generating cooking information using a cooking learning model based on instruction information input by a user;
Using a cooking process learning model, cooking process information corresponding to the cooking steps included in the dish information is generated;
A process of integrating the cooking and processing information to generate integrated cooking and processing information;
A process of presenting the generated integrated cooking and processing information;
A control unit that performs
Information processing system.
(2)
The information processing system according to (1), wherein the control unit generates a plurality of pieces of cooking processing information for the cooking process and determines feasible cooking processing information using the cooking processing learning model.
(3)
The information processing system according to (1) or (2), wherein the cooking and processing learning model is generated by reinforcement learning of simulation results based on multiple cooking and processing information generated for the cooking process.
(4)
The information processing system described in (1) or (2), wherein the cooking processing learning model is generated by reinforcement learning of cooking results in which cooking processing indicated by multiple cooking processing information generated for the cooking process is performed using a cooking robot.
(5)
The information processing system described in any one of (1) to (4), wherein the control unit presents multiple cooking processing information as candidates and generates the integrated cooking processing information to include cooking processing information selected by the user.
(6)
The information processing system according to any one of (1) to (5), wherein the instruction information includes a concept for the dish and information on ingredients desired to be used in the dish.
(7)
The information processing system according to (6), wherein the concept of the dish includes information on the purpose, impression, atmosphere, style, genre, theme, objective, key points, target demographic, design, or layout of the dish.
(8)
The information processing system described in (6), wherein the control unit uses an ingredient pairing learning model to generate combined ingredients suitable for combination with the ingredients based on the information of the ingredients included in the instruction information.
(9)
The information processing system described in (8) above, wherein the control unit uses the concept, the ingredients, and the combined ingredients as input data and generates the cooking information using at least one of a first cooking learning model that has learned the user's cooking and a second cooking learning model that has learned the cooking of other users.
(10)
The control unit
A process of generating cooking information by changing the style of the cooking design in the cooking information generated using the first cooking learning model using the second cooking learning model;
A process of generating cooking information by changing the style of the cooking design in the cooking information generated using the second cooking learning model using the first cooking learning model;
The information processing system according to (9) above,
(11)
The information processing system described in (9) or (10), wherein the control unit determines the proportion of the number of pieces of cooking information generated using each cooking learning model according to the set level of newness of the cooking.
(12)
The information processing system according to any one of (1) to (11), wherein the control unit generates multiple pieces of cooking information, presents them to the user, and sends the cooking information selected by the user to subsequent processing.
(13)
The information processing system according to any one of (1) to (12), wherein the control unit adjusts the cooking information in accordance with user comments and then sends the information to subsequent processing.
(14)
The information processing system according to any one of (1) to (13), wherein the control unit adjusts the cooking information according to the user's cooking environment and then sends it to subsequent processing.
(15)
The information processing system according to any one of (1) to (14), wherein the control unit includes failure case data in learning data when learning the cooking learning model or the cooking processing learning model.
(16)
The information processing system according to any one of (1) to (15), wherein the cooking and processing information is information about processing related to the appearance of ingredients.
(17)
The control unit sets personalization conditions, which are conditions for changing the cooking and processing information according to the preferences or calorie intake of the individual receiving the food, based on user input, changes the cooking and processing information to satisfy the personalization conditions, and then generates the integrated cooking and processing information.
(18)
The processor:
generating cooking information using a cooking learning model based on instruction information input by a user;
generating cooking processing information corresponding to the cooking steps included in the cooking information;
Integrating the cooking and processing information to generate integrated cooking and processing information;
Presenting the generated integrated cooking processing information;
A control method comprising:
(19)
Computer,
A process of generating cooking information using a cooking learning model based on instruction information input by a user;
generating cooking processing information corresponding to the cooking steps included in the cooking information;
A process of integrating the cooking and processing information to generate integrated cooking and processing information;
A process of presenting the generated integrated cooking and processing information;
A program that functions as a control unit to perform the above.

REFERENCE SIGNS LIST 10 Information processing device 110 Control unit 111 Agent unit 112 Draft generation unit 113 Display control unit 114 Adjustment unit 115 Element review unit 116 Integration unit 117 Story generation unit 120 Communication unit 130 Operation input unit 140 Display unit 150 Storage unit 160 Voice input/output unit

Claims (19)

A process of generating cooking information using a cooking learning model based on instruction information input by a user;
Using a cooking process learning model, cooking process information corresponding to the cooking steps included in the dish information is generated;
A process of integrating the cooking and processing information to generate integrated cooking and processing information;
A process of presenting the generated integrated cooking and processing information;
A control unit that performs
Information processing system. The information processing system of claim 1, wherein the control unit generates multiple pieces of cooking processing information for the cooking process and determines feasible cooking processing information using the cooking processing learning model. The information processing system of claim 1, wherein the cooking process learning model is generated by reinforcement learning of simulation results based on multiple cooking process information generated for the cooking process. The information processing system of claim 1, wherein the cooking process learning model is generated by reinforcement learning of cooking results obtained by performing cooking processes indicated by the multiple cooking process information generated for the cooking process using a cooking robot. The information processing system of claim 1, wherein the control unit presents multiple cooking and processing information as candidates and generates the integrated cooking and processing information to include the cooking and processing information selected by the user. The information processing system of claim 1, wherein the instruction information includes a concept for the dish and information about ingredients desired to be used in the dish. The information processing system of claim 6, wherein the concept of the dish includes information regarding the purpose, impression, atmosphere, style, genre, theme, purpose, key points, target demographic, design, or layout of the dish. The information processing system of claim 6, wherein the control unit uses an ingredient pairing learning model to generate combination ingredients suitable for combination with the ingredient based on the information about the ingredient included in the instruction information. The information processing system of claim 8, wherein the control unit uses the concept, the ingredients, and the combination of ingredients as input data, and generates the cooking information using at least one of a first cooking learning model that has learned the user's cooking and a second cooking learning model that has learned the cooking of other users. The control unit
A process of generating cooking information by changing the style of the cooking design in the cooking information generated using the first cooking learning model using the second cooking learning model;
A process of generating cooking information by changing the style of the cooking design in the cooking information generated using the second cooking learning model using the first cooking learning model;
The information processing system according to claim 9, wherein the information processing system is capable of performing the following. The information processing system of claim 9, wherein the control unit determines the proportion of cooking information generated using each cooking learning model according to the set level of newness of the cooking. The information processing system of claim 1, wherein the control unit generates multiple pieces of recipe information, presents them to the user, and sends the recipe information selected by the user to subsequent processing. The information processing system of claim 1, wherein the control unit adjusts the cooking information in response to user feedback before sending it to subsequent processing. The information processing system of claim 1, wherein the control unit adjusts the cooking information according to the user's cooking environment before sending it to subsequent processing. The information processing system of claim 1, wherein the control unit includes failure case data in the learning data when learning the cooking learning model or the cooking processing learning model. The information processing system described in claim 1, wherein the cooking and processing information is processing information related to the appearance of ingredients. The information processing system of claim 1, wherein the control unit sets personalization conditions, which are conditions for changing the cooking and processing information according to the preferences or calorie intake of the individual receiving the food, based on user input, changes the cooking and processing information to satisfy the personalization conditions, and then generates the integrated cooking and processing information. The processor:
generating cooking information using a cooking learning model based on instruction information input by a user;
generating cooking processing information corresponding to the cooking steps included in the cooking information;
Integrating the cooking and processing information to generate integrated cooking and processing information;
Presenting the generated integrated cooking processing information;
A control method comprising: Computer,
A process of generating cooking information using a cooking learning model based on instruction information input by a user;
generating cooking processing information corresponding to the cooking steps included in the cooking information;
A process of integrating the cooking and processing information to generate integrated cooking and processing information;
A process of presenting the generated integrated cooking and processing information;
A program that functions as a control unit to perform the above.