WO2005015467A1 - Life supporting system - Google Patents

Abstract

A life supporting system for managing articles in living environment in which articles hidden in a facility (e.g. a refrigerator, a drawer, a closet, a box) can also be operated through an interface means for operating the articles on a screen in conforming with the real situation. The life supporting system (100) comprises a display (119) for displaying the real situation in a room, an article operating unit (118) for specifying an article using the display (119), a means (124) for generating data for displaying the information on articles in the facility (104) on the display (119) based on the information in an article database (107) and an environmental map information data base (109), and a control means (120) for displaying the information on articles hidden in the facility (104) on the display (119) when the facility (104) is specified by the article operating unit (118).

Description

 Life support system

Technical field

 The present invention relates to a life support system and a life support system that provide life support by managing articles as an example of an object existing in a living environment, for example, a living environment.

The present invention relates to a control method for a program and a life support system, and more particularly to a life support system that facilitates a user's operation on an article to provide life support, a program for the life support system, and a control method for the life support system.

 Background art

 Making robots move objects to do some work is done in many fields. For example, gripping and transporting parts on an automated product assembly line at a factory, and transporting and managing inventory in an automated warehouse. On the other hand, today, attempts are being made to make mouth pots work at home by utilizing robot technology that has matured for industrial use. In other words, research and development of robots for home use have been actively pursued with the spectacular goal of the mouth pot supporting human life while coexisting with humans at home.

 For example, it has an entertainment-type mouth pot that behaves like a pet in an environment where pets cannot be kept, and heals human hearts, and is equipped with a large number of sensors, and detects obstacles in the room with sensors. So-called housekeeping assistance mouth pots, which automatically clean the inside of the room while receiving light, have been actually developed and commercialized. Elemental technologies include technologies that are indispensable for actually providing various types of housekeeping support, such as handling technology for freely grasping and transporting items in the home, and sensing technology for recognizing the state in the space where robots move. Etc. are being actively developed.

However, even if the above-mentioned handling technology and sensing technology have matured, it is not always possible to immediately realize housework support by a robot. because, This is because, in addition to the above-mentioned technologies, a technology for easily giving instructions to the robot, that is, an interface technology must be developed. In other words, it is necessary to have a technology that improves the usability of the user. For example, it is a technology that can easily give instructions to carry home goods from one place to another, or to throw away garbage.

 In the case of an industrial mouth pot, such operation instructions are relatively simple. For example, Japanese Patent Application Laid-Open No. 7-237159 discloses a method in which articles to be handled are limited and an instruction to load the articles on a pallet is easily provided. In this method, a virtual article diagram according to the actual situation is displayed on the monitor, and the user gives an instruction while looking at the virtual article diagram. That is, the user instructs the work procedure for loading the rectangular parallelepiped article on the pallet by moving the article on the virtual article diagram and inputting the position. Such a moving operation is performed by intuitive operations such as dragging and dropping a mouse. In the above method, the procedure of loading the articles on the pallet is completed by repeating such an instruction operation.

 The system that executes the above method sequentially accumulates the location before and after the movement of all the articles as work information by these instruction operations. Then, when the instruction is completed, the robot goes to the place before the movement for each article, grasps it, moves to the place after the movement, and releases the grasp to install the article. The work is completed when all the goods transport operations are completed.

 Therefore, in the above-mentioned system, the input operation of the load handling setting can be intuitively performed while watching the loading status of the load handling articles displayed on the monitor, thereby simplifying the operation.

 The technique disclosed in the above-mentioned Japanese Patent Application Laid-Open No. Hei 7-23771-59, in which a cargo handling procedure is set in a virtual article diagram, is to instruct a home robot to carry out an operation of holding and transporting an article. Is also applicable. For example, the present invention can be applied to a system in which a person who has difficulty carrying an article due to a decrease in muscular strength, such as an elderly person, can grasp and move an article indoors while watching a monitor screen.

However, unlike the industrial field, in which the shapes and arrangements of the articles are arranged so that uniform processing can be performed in advance, the articles are not always visible at home. It is not always arranged. Specifically, household items are not only placed on the floor and tables, but also in equipment (refrigerators, drawers, closets, boxes, etc.). In some cases, goods are hidden behind equipment. For this reason, it is difficult to apply the conventional technology as it is in a residential environment such as a home, and there is a need for a technology that can easily operate articles and the like in such facilities using interface means. Disclosure of the invention

 SUMMARY OF THE INVENTION The present invention has been made in view of power, and its purpose is to easily perform an operation via an interface unit even on an article inside a facility or an article shielded by a facility. It is an object of the present invention to provide a life support system that provides a life support by enabling the above, a program for the life support system, and a control method of the life support system.

 According to one aspect of the present invention, a display device that displays the actual situation of a living environment, and an article operation device that receives an operation instruction for designating equipment in the living environment on a display on the display device are provided. Prepare,

 By specifying the facility by the article operating device, the interior of the facility is determined based on the information on the article in the living environment and the structural information on the facility in the living environment and the space in the living environment. There is provided a life support system including a control unit that controls information of an article or an article concealed in the facility to be displayed on the display device. According to another aspect of the present invention, a procedure for displaying the actual situation of a living environment on a display device, and an operation instruction for designating a facility in the living environment is input on the display on the display device, and the facility is displayed. By being designated, based on the information of the articles in the living environment and the structural information of the facilities in the living environment and the space in the living environment, the articles inside the facilities or the articles hidden in the facilities are Provided is a life support system program for causing a computer to execute a procedure for displaying information on the display device.

According to still another aspect of the present invention, an actual situation of a living environment is displayed on a display device, and an operation instruction for designating equipment in the living environment is input on a display on the display device. When the equipment is designated by being pressed, the interior of the equipment is determined based on the information of the articles in the living environment and the structural information of the equipment in the living environment and the space of the living environment. Provided is a control method of a life support system, which controls information of an article or an article hidden in the facility to be displayed on the display device.

 In the life support system, when a user instructs equipment (table, refrigerator, trash can, etc.) on a display (image of a display, etc.) of a display device using an article operating device (mouse, etc.), the display device displays the equipment. Information on the items inside or those shielded by equipment is displayed. Therefore, while viewing the display on the display device, the user can obtain information on not only the articles existing outside the facility but also articles inside the facility or articles shielded by the facility. Therefore, it is possible to easily operate even an irregularly or randomly arranged article. Therefore, it becomes possible to perform operations on various articles existing in the living environment, for example, the living environment.

 In addition, according to the life support system program, or according to the control method of the life support system, the user looks at the display on the display device and not only displays the articles outside the facility but also the inside of the facility. Information on goods or goods shielded by equipment can also be obtained. Further, either one of the information on the article hidden inside the equipment and the information on the article shielded (hidden behind the equipment) by the equipment is displayed selectively and switchably, so that Even if there is an article both inside and in a part shielded by equipment, information on those articles can be displayed separately. Therefore, the user can easily operate the article.

ADVANTAGE OF THE INVENTION According to the present invention, a user can obtain information on an article inside the equipment or an article shielded by the equipment by a simple operation of simply specifying the equipment on the display of the display device using the article operation device. it can. Therefore, it is possible to easily obtain information not only on articles outside the facility but also on articles inside the facility or articles shielded by the facility. However, the operation can be easily performed. Therefore, it is possible to easily operate an article in a living space, for example, a living space via the article operating device (interface means). Brief Description of Drawings These and other objects and features of the present invention will become apparent from the following description in connection with the preferred embodiments of the accompanying drawings. In this drawing,

 FIG. 1 is a block diagram showing the overall configuration of the life support system according to the first embodiment of the present invention.

 2A, 2B, 2C, and 2D are explanatory diagrams for explaining the background subtraction method in the life support system, and FIGS. 2A to 2C show the background difference method, respectively. FIG. 2 is an explanatory diagram showing a camera and the like used in the room and a room.

 FIG. 3A is a conceptual diagram showing a configuration and description contents of article data in the life support system in a state before clearing, and FIG. 3B is a diagram showing a configuration of the article data and examples of description contents. A conceptual diagram in the state after clearing is shown,

 FIGS. 4A and 4B are schematic views of the environment of the living support system taken at two different times, respectively.

 FIG. 5A is a diagram of an actual situation for explaining the environment map information database in the life support system, and FIG. 5B is a diagram of a three-dimensional model for explaining the environment map information database in the life support system. FIG. 5C is a diagram of the plane model of FIG. 5A for explaining the environment map information database in the life support system, and FIG. 6 is an example of data of the environment map information database in the life support system. FIG.

 FIGS. 7A and 7B are diagrams showing examples of equipment and equipment attribute data in the life support system, respectively.

 FIG. 8 is a diagram showing, in a table form, equipment operation commands stored in an equipment operation information storage unit in the life support system,

 9A is a conceptual diagram showing an example of an operation screen in the life support system, and FIG. 9B is a conceptual diagram showing an example of a screen map corresponding to the operation screen in FIG. 9A.

 FIG. 10A, FIG. 10B, FIG. 10C, FIG. 10D, and FIG. 10E respectively show examples of display forms of articles stored inside the equipment in the life support system. FIG.

FIG. 11 is a diagram showing the correspondence between the condition for switching the display mode inside the facility and the display mode in the life support system, FIGS. 12A and 12B are diagrams showing an example of the equipment in the life support system and the equipment behind it, a place on the floor surface, and a display form of articles, respectively.

 Figures 13A, 13B, 13C, and 13D are conceptual diagrams showing examples of hierarchical screen maps corresponding to the environment of Figure 12A, respectively.

 FIG. 14 is a perspective view showing a configuration of a mouth pot in the life support system. FIG. 15 is an example of a list of mouth bot control commands stored in a robot control command database in the life support system. Is a tabular diagram showing

 Fig. 16 is a diagram showing an example of the confirmation screen in the life support system. Fig. 17 is a state in which the information written on the electronic tag placed on the bottom of the plastic bottle is read by the tag reader of the refrigerator. FIG.

 Fig. 18 is an explanatory diagram in the case of having the operation program of the robot arm and the hand of the mouth pot as the equipment attribute data.

 FIG. 19 is a diagram showing an example of an operation program of the robot arm and the hand of FIG. 18;

 FIG. 20 is an explanatory diagram for explaining where in the equipment the article is to be placed when moving into the equipment. BEST MODE FOR CARRYING OUT THE INVENTION

 Before continuing the description of the present invention, the same parts are denoted by the same reference numerals in the accompanying drawings.

 Before describing the embodiments of the present invention, first, various aspects of the present invention will be described.

 According to a first aspect of the present invention, a display device that displays the actual situation of a living environment,

An article operation device for inputting an operation instruction for designating facilities in the living environment on a display on the display device;

By specifying the facility by the article operating device, the article or the article inside the facility can be specified based on the information on the article in the living environment and the structural information of the facility in the living environment and the space in the living environment. There is provided a life support system including control means for controlling information of an article hidden in the facility to be displayed on the display device. According to a second aspect of the present invention, an article storing information of the article in the living environment:

 An environment map information database that stores structural information of the facilities in the living environment and the space in the living environment;

 The control means is connected to the display device, the article operation device, the article database, and the environment map information database respectively, and when the equipment is designated by the article operation device, the article database and the environment map information Referring to a database, based on the information of the articles in the living environment and the structural information of the facilities in the living environment and the space of the living environment, the articles in the facilities or the articles hidden in the facilities are determined. A life support system according to the first aspect, wherein information is controlled to be displayed on the display device.

 According to a third aspect of the present invention, when the article operating device designates one of a plurality of facilities in the living environment by the article operating device, the control means includes information on goods in the living environment and the living environment. Based on the structural information of the facilities in the environment and the space of the living environment, the facility and any one of the facilities behind it can be freely selected, and the interior of the selected facility can be switched. The life support system according to the first aspect, wherein information of an article or an article hidden in the facility is controlled to be displayed on the display device. '

 According to a fourth aspect of the present invention, when the control unit specifies one of a plurality of facilities in the living environment by the article operating device, the information on the goods in the living environment and the life Based on the structural information of the facilities in the environment and the space of the living environment, the facility and any one of the facilities behind it can be freely selected, and the interior of the selected facility can be switched. A life support system according to a second aspect, wherein control is performed to display information on an article or an article hidden in the facility on the display device.

 According to a fifth aspect of the present invention, a sensing unit for detecting information on the article, a database control unit for updating information on the article based on a detection result of the sensing unit,

The life support system according to any one of the first to fourth aspects, further comprising: I will provide a.

 According to a sixth aspect of the present invention, an article moving device for moving an article,

 When the article and the moving location of the article are input as an instruction operation by the article operating device, an article moving control unit that moves the article to the moving location by the article moving apparatus;

 The life support system according to any one of the first to fourth aspects, comprising:

 According to a seventh aspect of the present invention, the article operating device displays a force sol for operating an item on a display of the display device,

 When the cursor is placed on the facility on the display of the display device, the control unit pops up information on an article inside the facility or an article concealed by the facility either alone or together with the structural information of the facility. A life support system according to any one of the first to fourth aspects, wherein the life support system is displayed in an upload list format.

 According to an eighth aspect of the present invention, the article operating device displays a cursor for operating an article on a display of the display device,

 At least one of the facilities in the living environment is provided with photographing means for photographing the inside of the facility or an area concealed by the facility,

 When the cursor is placed on the facility on the display of the display device, the control means displays an image of an article taken by the photographing means and inside the facility or in a region hidden by the facility. The life support system according to any one of the first to fourth aspects of the present invention is provided.

 According to a ninth aspect of the present invention, the article operating device displays a cursor for operating an article on a display of the display device,

 At least one of the facilities in the living environment is provided with a transmission device that makes the inside of the facility or an area hidden by the facility visible.

When the cursor is placed on the facility on the display of the display device, the control means is configured to transparently display an article inside the facility or an article concealed by the facility by the transmission device. 4. The life support system according to any one of the aspects 4 to 4 is provided. According to a first embodiment of the present invention, the facility includes a storage body that performs storage or removal of articles.

 The life support system according to any one of the first to fourth aspects, further comprising control means for causing the storage body to execute the operation when the storage body is designated by the article operating device. .

 According to the eleventh aspect of the present invention, the storage body includes a door that can be opened and closed, and the control unit opens or closes the door of the storage body when the storage body is designated by the article operating device. A life support system according to a tenth aspect, wherein the life support system is closed.

 According to the 12th aspect of the present invention, one of a plurality of display forms is selected based on at least one of the attribute of the facility, the attribute of the article, the attribute of the user, and the operation content, and the article information is selected. The life support system according to any one of the first to fourth aspects, wherein control is performed so as to further include an in-facility display means for displaying on the display device. According to the thirteenth aspect of the present invention, a procedure for displaying the actual situation of a living environment on a display device, and an operation instruction for designating a facility in the living environment on a display on the display device is input to the facility. Is designated, based on the information on the articles in the living environment and the structural information on the facilities in the living environment and the space in the living environment, the articles inside the facilities or the articles hidden in the facilities. And a procedure for displaying information on the display device by a computer.

 According to the fifteenth aspect of the present invention, further, when one of a plurality of facilities in the living environment is designated, information on articles in the living environment, the facilities in the living environment, and the living Based on the structural information of the environment space, the facility and any one of the facilities behind it can be switched freely and can be selected and hidden in the article inside the selected facility or the facility The life support system program according to the thirteenth aspect, further comprising a procedure of controlling information of an article to be displayed on the display device.

According to the fifteenth aspect of the present invention, the actual situation of the living environment is displayed on a display device, and an operation instruction for designating facilities in the living environment is input on the display of the display device. By specifying the equipment by being forced, based on the information of the goods in the living environment and the structural information of the equipment in the living environment and the space of the living environment, the goods or Provided is a control method for a life support system, which controls information of articles hidden in the facility to be displayed on the display device.

 According to the sixteenth aspect of the present invention, further, among the plurality of facilities in the living environment,

When one facility is designated, any one of the facility and the facility located behind it is determined based on the information on the goods in the living environment and the structural information on the facilities and the space in the living environment in the living environment. 15th mode, further comprising the step of: controlling the display device to freely select one of them and displaying information of an article inside the selected facility or an article concealed by the facility on the display device. The control method of the life support system described in (1) is provided.

 Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

 First, the overall configuration of the life support system according to the present embodiment will be described, and then the operation of the life support system (specifically, the article management system) will be described along with specific examples.

 Overall configuration of one life support system 1

 FIG. 1 is a block diagram showing an example of the overall configuration of the life support system 100 in the present embodiment. The life support system 100 is roughly divided into four subsystems, namely, an environmental management server 101 (hereinafter sometimes simply referred to as a server) and an article as an example of an object in a living environment. It is composed of a robot 102 as an example of an article moving device, an operation terminal 103, and equipment 104.

 In the following, the configuration and operation of each subsystem will be described in the order of the environmental management server 101, equipment 104, operation terminal 103, and robot 102.

 Environmental management server

The environmental management server 101, which is the first subsystem, includes a first sensing unit 105 for grasping a situation in a living environment, for example, a living environment (hereinafter, simply referred to as an environment), and a first sensing unit 105. An article search / management means 106 for managing and searching for objects such as articles and moving objects (for example, people and mouth pots) existing in the environment based on the situation grasped by being connected to the Articles and moving objects connected to the management means 106 An article database 107 for storing data for management and search of the environment, and an environment map information managing means 108 connected to the first sensing section 105 for storing information on the entire environment other than the articles. An environment map information database 109 connected to the environment map information management means 108 and storing data for managing information of the entire environment, an article database 107 and an environment map information database 109, etc. 1st sending and receiving unit 110, which receives inquiries about stored data from the outside and sends information to the outside in response to it, etc., article search and management means 106, and environment map information management means 108 And the first transmission / reception unit 110, respectively. For example, an inquiry about data stored in the article database 107, the environment map information database 109, or the like is externally sent to the first transmission / reception unit 110. Received A first control means for performing predetermined operation control and performing operation control so as to transmit information from the first transmission / reception unit 110 to the outside based on a result of the predetermined operation control. It has. As will be described later, reference numeral 99 denotes an input device such as a keyboard, a mouse, and a touch panel that can be manually input by a user, and is connected to the article search / management means 106 and the environment map information management means 108. Based on the manually input information, the system manages and searches for objects, such as articles and moving objects, existing in the environment, which are information stored in the article database 107, and obtains information on the entire environment other than the articles. So that they can be managed.

 Note that the living environment in the present embodiment means an environment in which people and goods exist with mutual relations, such as a house and an office.

Environmental map information includes structural information on a room (a “space” formed by walls, floors, and ceilings) as an example of the environment, and furniture and large appliances (refrigerators, microwave ovens, It consists of structural information such as “equipment” 104) such as machines and dishwashers, and is attached to the space and equipment 104, respectively. Structural information is defined as a surface on which other objects (referred to as articles) can be installed at least inside the space and above it, inside the equipment and above it (for example, on the floor and equipment in a room). Means the area information (for example, the position coordinates of the vertices of the circumscribed polygon on the surface). The area information means information on an area represented by coordinate system information and display information based on a shape or the like. In addition, a small box is an article. Since the product can be stored or installed, it can be treated as equivalent to the equipment. Hereinafter, each component of the environment management server 101 will be described in order.

 <First sensing part>

 The first sensing unit 105 includes all monitored objects existing in an operation environment (for example, a house, an office, a store, and the like) as an example of the environment, that is, articles, furniture, people existing in the environment, and a robot 1. It constantly monitors the position and state of 02 and the like. In addition, the first sensing unit 105 detects when a new article is newly brought into the environment by a human, a robot 102, or the like.

 Although the specific configuration of the first sensing unit 105 is not particularly limited, for example, a device using an image sensor, a device using an electronic tag, and the like can be suitably used. Here, a device and a method using an image sensor and a device and a method using an electronic tag will be described as specific examples of the device and the method for detecting an article or a moving object.

 First, a method using an image sensor will be described. The type of image sensor used here is not particularly limited, but a camera (image sensor) 105 A as an example of a photographing means is preferably used for efficiently monitoring a wide area such as indoors with a small facility. Can be. That is, as shown in Fig. 2D, the camera 105A is fixedly installed on the ceiling or wall of the indoor room 104Z, and the captured images are used to detect objects. May go.

 As a general method for detecting an article using an image of a camera 105A installed in an environment, a background subtraction method is known. The background subtraction method is a method in which a model image as a background is prepared in advance, and a difference between a current input image and the model image is obtained to obtain an object from the image. The first sensing section 105 according to the present embodiment aims to detect and monitor articles and moving objects in the environment. Therefore, if there is no environmental change, for example, a single image in which there is no article or the like in the environment can be used as the model image. On the other hand, if the environment fluctuates greatly, an image obtained by averaging images taken continuously over a certain period of time may be used.

2A to 2D are assistance diagrams for specifically explaining the background subtraction method, and FIG. 2B is a diagram illustrating an example of the model image. Figure 2A shows the force that captured the image in Figure 2B FIG. 2C is a diagram showing an input image taken at a certain point in time using the same camera 105A as the camera 105A, and FIG. 2C shows the model image of FIG. 2B from the input image of FIG. 2A. FIG. 9 is a diagram showing an example of a background difference image obtained by subtraction. As can be seen from FIG. 2C, only the part having a difference between the input image and the model image emerges in the background difference image. As described above, in the background subtraction method, it is possible to detect articles in the environment by extracting only the portions that emerge from the background difference image. FIG. 2D is an explanatory diagram showing a relationship between the first sensing unit 105 including the camera 105A used in the background subtraction method and the room 104Z. The number of cameras used may be one, but if two or more cameras are used, it is possible to acquire the shape and posture information of the article using stereoscopic three-dimensional measurement technology. .

 Therefore, in this example, the first sensing unit 105 is connected to a camera (image sensor) 105 A and a camera 105 A so that the background subtraction method can be performed and the operation result is searched for an article. · It has an arithmetic unit 105B that can be output to the management means 106 and the environment map information management means 108.

 Next, a method for detecting an article using an electronic tag will be described. In recent years, methods for detecting the position of an article or a moving object using an electronic tag have been developed. In the present embodiment, almost all of these methods can be applied, and for example, the technology disclosed in Japanese Patent Application Laid-Open No. 2000-35702 can be used. Specifically, using the three tag readers installed in the room, the position of the article and the location of the article using the three-point survey method based on the radio wave intensity of the RFID tag (with the same article ID) attached to the article ID can be detected.

 Here, the electronic tag 80 is, as shown in FIG. 17, a device composed of an IC 80A for storing data and an antenna 80B for transmitting data wirelessly. By using the device 81, information can be written to the IC 80A of the electronic tag 80, and information written to the IC 80A can be read. In FIG. 17, an electronic tag 80 is arranged on the bottom surface 82 A of the pet bottle 82, and a reader / writer (an example of a tag reader) 81 of the refrigerator 104 B uses an IC 8 of the electronic tag 80. This shows a state in which information written in the OA is read.

The tag's IC 8 OA contains attribute data that characterizes the item, that is, the type of the item. It is possible to store data such as type, date of manufacture, shape, weight, and product image. By storing such data in the IC tag OA of the electronic tag and making the data freely accessible, more advanced article management becomes possible. By doing so, for example, the shape and weight can be used for gripping and installing articles, the date of manufacture can be used for quality control, and the type of article can be used for searching for items. This will bring great benefits to users. Also, only the product code (similar to a barcode) standardized in the industry is stored in the IC 8 OA itself of the electronic tag, and communication between the product code and the attribute data, such as an external server or the Internet, is stored. It may be configured to use a means to inquire the manufacturer of the attribute data of the article. In addition, it is also possible to keep a history of past positions (other attribute data) in the IC 8 OA of electronic tags, and use the time and position and other attribute data to check for existing items in the past. is there.

 Therefore, in this example, as shown in FIG. 17, the first sensing unit 105 is connected to the electronic tag 80 including the IC 8 OA and the antenna 8 OB, and to the wireless connection with the electronic tag 80. Possible and article search · Management means 106 and environmental map information management means 1

08 and a reader / writer 81 capable of outputting the same.

 As described above, the method of detecting an article and a moving object using a camera and an electronic tag, respectively, has been described as specific examples of the sensing technology. However, the first sensing unit 105 may, of course, use other methods. It may be. The first sensing unit 105 includes at least one of a camera, an electronic tag, and other sensors.

 When a new article or a moving object is detected by the first sensing unit 105, the information (for example, attribute data of the new article or the moving object) is stored in an article search / management unit 106 described later. (Example of database control means) Registered / updated in the article database 107 via.

 Ku Articles Database>

The goods database 107 is a database that stores data such as what kind of goods (including equipment), when, and where. The details will be described below with reference to the drawings. FIG. 3A and FIG. 3B are conceptual diagrams showing an example of the data structure of the article database 107 and an example of the description. Fig. 3B shows the contents of the article database after a certain time has elapsed since Fig. 3A.

 In the present embodiment, each item data constituting the item database 107 has the following six attributes: 1) item ID, 2) item name, 3) person mainly used (owner), 4) Time, 5) Location, 6) Item image.

 1) Article ID: ID for distinguishing individual articles. If the same kind of goods are physically different, they need to be treated as different goods. Therefore, different IDs are assigned to the same kind of goods. For example, if there are two PET bottles, two goods IDs “D # 0001” and “D # 0002” are attached to each.

 2) Article name: Name indicating the type of the article. Unlike the article ID, if the type is the same, the name will be the same. For example, plastic bottles and pizza.

 3) Person who mainly uses (owner): The person who mainly uses the product or the owner, and this is listed. This makes it possible to preferentially display goods to be used according to the user. For example, father, mother, child.

 4) Time: The time when the item was most recently operated (used or moved). For example, "2002/10/10 10:00" means 10:00 am on October 10, 2002.

5) Location: The place where the article moved when it was most recently operated (used or moved). The location is specified by the ID number of the environment attribute data 601 or the facility attribute data 602 registered in the environment map information database 109 described later (see FIG. 6). Also, for items whose spatial position is difficult or impossible to identify using only the ID number, the coordinate value of the item is set. For example, if the location is “refrigerated room” or “freezer room”, it is possible to specify that the article exists in them alone, so no coordinate value is specified (for example, “refrigerated room” Is “Cold—room # 0001”, and “Freezer” is “Freezer # 0001”.) On the other hand, "floor""i 1 oor # 000 If the specified location covers a wide area as in `` 1 '' and the location of the specific article cannot be specified by the location name alone, the coordinate value for specifying the location is assigned (for example, For "D # 0001", "floor # 0001 (X1, y1, 0)", for pizza "F # 0001", "floor # 0001 (x2, y2, 0)", etc.). It is preferable that the initial setting of the location of the article, the update when the article moves, and the provision of the coordinate value as additional information are automatically performed by the calculation section 105B of the first sensing section 105. However, it goes without saying that it may be done manually. The determination as to whether or not to provide a coordinate value when indicating the location of the article may be determined based on the performance of the robot 102 that grips and transports the article. If the performance of the robot 102 is very low, for example, when grasping an article in the refrigerator compartment, if the exact position coordinate value of the article is required, the location (coordinate value) of the article in the refrigerator compartment is given. You may do so.

 6) Article image: An image of the article.

 Here, an example is shown in which six attributes are provided to distinguish the characteristics of each article, but other attributes may be provided as necessary. For example, attributes such as the three-dimensional shape of the article and the exact position / posture data (not shown) of the article may be added. By providing such an attribute, the robot 102 can more easily grip the article. In addition, the time “place” article image can be recorded for each time and stored as a history. As a result, it is possible to instruct the position of an article at a certain point in the past or to display an image.

 Here, it is preferable that at least necessary article attributes are article ID, time (time), and position (location). Once the product ID is known, the manufacturer can be queried for other attributes using the Internet. It is desirable to have the “owner” attribute, but it is not necessary for some applications.

 In addition, other attributes include:

7) Equipment attribute data: Structural information when the article is equipment: This will be described later as equipment attribute data in the environment map information database 109. Next, how the contents of the data are changed over time while comparing the contents of the article database 107 with the actual situation will be described. FIGS. 4A and 4B are schematic diagrams in which a state of a certain environment (for example, one room 104Z) is photographed at two different times. Here, it is assumed that FIGS. 4A and 4B correspond to FIGS. 3A and 3B, respectively. That is, it is assumed that the database storing the article data existing in the room 104Z, which is an example of the environment at each time, matches the database shown in FIGS. 3A and 3B, respectively. In each figure, 104A is a table, 104B is a refrigerator, 104C is a freezer, 104D is a refrigerator, 104E is a microwave oven, 104F is a wastebasket, 104G is a recycle bin, 104H is a floor, 104H J is the wall, 104 K is the ceiling, 104 L is the door, 104 M is the cupboard.

 Figure 3A shows, as an example, the contents stored in the database at 9 o'clock on October 10, 2002. The database contains seven items: PET bottles, pizza, notebooks, bananas, paper waste, ice cream, and milk. Of these, five items, such as plastic bottles, pizza, notebooks, bananas, and paper waste, are scattered on the floor 104H, as can be seen in the example of Fig. 4A. Therefore, as shown in Figure 3A, the value of the location of each article in the database is "floor".

Not only “flor # 0001” but also the position coordinate value on each floor 104H is given as additional information.

 On the other hand, the remaining goods, ice cream, and milk are stored in the freezer compartment 104C and the refrigerator compartment 104D (not explicitly shown in FIG. 4A). The value of the location of each article is described as “freezer # 0001” which is a “freezer room” and “Co 1 d-room # 0001” which is a “refrigerator room”.

 Next, suppose that the following environmental changes occurred at 10 o'clock on October 10, 2002 S. That is,

1) The user is instructed to clear the items (five items of plastic bottles, pizza, notebooks, bananas, and paper waste) on the floor 104H (the clearing destinations are as shown by the curves with arrows in Fig. 4A). )). In other words, PET bottles are placed in the recycle bin 104G. Pizza and bananas are placed in Refrigerator 104D. The notebook is placed on the 104A tape. Put the waste paper in the wastebasket 104F. Receiving that instruction Then, the robot # 02 performed the article clearing.

 2) The user eats and drinks ice cream and milk in the freezer compartment 104C and the refrigerator compartment 104D, and the ice cream and milk disappear.

 Figure 3B shows the state of the database at 20:00 on October 10, 2010, a little after the environmental change.

 As shown in FIG. 4B, the five articles, PET bottles, pizza, notebooks, bananas, and paper waste, which were scattered on the floor 104H at the time of FIG. The value of the database location has been rewritten accordingly (see Figure 3B). In addition, ice cream and milk, which disappeared after being eaten, are deleted from the database and are not in the database.

(See Figure 3B). This deletion process may be performed manually, or may be automatically deleted by reading a sensor such as a tag with a reader / writer. Also, since food is virtually extinct from the real world when someone eats it, the place data may describe the person who ate the food instead of describing the place.

 If the ice cream or milk does not disappear even after eating, for example, if you eat and drink only half, the value of the original place is displayed as it is and does not disappear. Thus, when an article moves or disappears from the environment, the data content of the article database 107 is updated successively, and the latest information is always kept in the article database 107. The above is the description of the contents of the article database 107.

 <Article search ■ management means>

The article search / management means 106 uses the first sensing unit 105 and the input device 99 to retrieve information on the articles obtained by the user's manual input for all articles placed in the environment. If there is an inquiry about goods etc. stored in the goods database 107 or from outside the environmental management server 101 via the first transmitting / receiving unit 110 and the first control means 111, Necessary information according to the content of the inquiry is retrieved from the article database 107 by the article search / management means 106 and sent to the inquiry destination via the first control means 111 and the first transceiver 110. Perform the process of sending. Access to the article database 107 is permitted only to the article search and management means 106, and writing and reading are performed simultaneously for the same article data. It is managed not to be. When a request for registration / update of article information is received from the robot 102 and the operation terminal: L 03, the article database 107 is registered / updated via the article search / management means 106. It is also possible to search for the location of the item by using the operation terminal 103 described below and specifying a search key to narrow down the attributes of the item that you want to know, such as the date and the type of the item. is there.

 <Environmental map information management means>

 The environment map information management means 108 manages the map information in the room as an example of the environment. 5A to 5C are conceptual diagrams showing an example of the environment map information database 109 in comparison with the actual situation. FIG. 5A is a diagram showing the actual situation, and FIG. 5B is an environment map. FIG. 5C is a diagram showing a simplified model of the actual situation as a loop information database 109, and FIG. 5C is a diagram showing the plane of the actual situation further simplified. The environment map information database 109 may be represented as three-dimensional data as described above, or may be more simply planar data. The data should be created according to the purpose of the map and the time required to create the map.For example, if it is necessary to create a three-dimensional model in a very short time, model the three-dimensional object with the smallest cuboid that covers it And so on. For example, the model in Figure 5B is such an example. In FIG. 5B, the table 104A located at the center in FIG. 5A is modeled as a rectangular parallelepiped. The same applies to plane data. In the model shown in Fig. 5C, the taper located in the center is represented by a rectangular area (rectangular area of the hatched part in Fig. 5C) that is orthogonally projected onto the plane, and this area cannot be moved by robots. It is defined as an area. For convenience of the following description, the X-axis (direction along one side of the room floor) and the Y-axis (along the other side orthogonal to one side of the room floor) shown in FIGS. The position coordinate system in the real world composed of one Z axis (room height direction) is called the real world coordinate system.

 <Environmental map information database>

 FIG. 6 is a diagram showing an example of data in the environment map information database 109. The environment map information database 109 is roughly composed of two elements, environment attribute data 601 and equipment attribute data 602.

The environment attribute data 6001 is, in short, detailed data of a room as an example of an environment. In the present embodiment, as an example, the environment attribute data The floor data “floor # 0001” and “floor # 0002J” are recorded for the two floors (note that the second floor data “f1 oor # 0002” is not shown). The floor data describes position coordinates (position coordinates in the real world coordinate system) of vertices (corners) when the floor is a polygon. Is attached to each surface. For example, in the case of data for a square floor, as shown in Figures 6 and 5A,

 ((XI, Y1, 0),

 (X 2, Y 2, 0),

 (X 3, Y 3, 0),

 (X4, Y4, 0), 0)

It is expressed. However, the lowest floor height in the room is set to 0 as a reference for coordinate values. The first four coordinate values indicate the coordinates of the vertex of the floor, and the last number “0” indicates the material of the floor. The floor material is, for example, “0” is flooring, “1” is tatami, “2” is carpet, etc., and corresponding numbers are determined in advance for each material. If there are multiple floors with different heights in a room, these floor data need only be prepared for the number of floors.

 The equipment attribute data 602 lists the equipment 104 existing in the environment (specifically, the room) configured by the environment attribute data 601. Here, the facilities 104 are household goods which are not moved and used in a normal state, such as furniture and large home appliances.

In the example shown in FIGS. 5A to 5C and FIG. 6, in a room as an example of the environment, a facility 104 such as a table 104A, a freezer compartment 104C, a refrigerator compartment 104D, and trash cans 104F and 104G is provided. Since they exist, the respective data are stored in the environment map information database 109, and those attributes are stored in the equipment attribute data 602. For example, the table 104A stores the position coordinate values of the respective corners of the surface 1 and the surface 2 as the position coordinates. In the freezer compartment 104C, position coordinate values of respective corners of the surface 1 and the surface 2 are stored as position coordinates. In the refrigerator compartment 104D, the position coordinate values of the respective corners of the surface 1 and the surface 2 are stored as the position coordinates. The trash cans 104F and 104G also have their location coordinates as Is stored. Note that originally, the freezing compartment 104C and the refrigeration compartment 104D are integrally called a refrigerator 104B.However, in the present embodiment, the facilities are distinguished by location where goods can be stored or installed. Instead of treating the refrigerator 104B as one facility, the freezer compartment 104C and the refrigerator compartment 104D are distinguished as one independent facility.

 The equipment attribute data 602 includes, as attributes of each equipment, data of a plurality of surfaces when the surface of the relevant equipment 104 is approximated by a polyhedron, types of the relevant equipment 104, and installed on the available surface of the relevant equipment 104. Main article shapes and postures are stored. The surface data of the equipment describes the coordinate values of the vertices of the surface (position coordinate values in real world coordinates), and a flag indicating whether or not an object can be installed on the surface has a flag for each surface. It is attached. For example, if the number of vertices is 4

 ((XI I, Y 11, Z 11),

 (X 12, Y 12, Z 12),

 (X 13, Y13, Z 13),

 (X 14, Y 14, Z 14), 1)

It becomes. The first four coordinate values indicate the position coordinate values of the four vertices, and the last number “lj is a flag that means that the article can be set. Depending on the type of equipment, this flag can be switched depending on the situation, such as when the door is open and the surface on which goods can be installed is exposed or closed. Whether the surface on which articles can be installed is not exposed, etc. Figures 7A and 7B are auxiliary figures to show such a typical example.

 In the examples of FIGS. 7A and 7B, the attribute data of the equipment related to the freezing room 104C is shown. That is, FIG. 7 shows the attribute data when the door 104-1 of the freezer compartment 104 is closed.

 ((X 21, Y21, Z 21),

 (X 22, Y 22, Z 22),

 (X 23, Y 23, Z 23),

(X 24, Y 24, Z 24), 0) It becomes.

 On the other hand, in FIG. 7B, the attribute data when the door 104C-1 of the freezer compartment 104C is open is-((X21, Y21, Z21),

 (X 22, Y 22, Z 22),

 (X 23, Y23, Z 23),

 (X 24, Y 24, Z 24), 1)

It becomes. From these figures, it can be seen that the last value of the flag changes according to the opening and closing of the door 104C-1 of the freezing room 104C. That is, when the door 104C-1 of the freezer compartment 104C is closed, the article cannot be stored inside the door 104C-1 as it is, and the flag is set to 10 ". On the other hand, when the door 104C-11 of the freezer compartment 104C is open, the state is such that the articles can be stored therein as it is, so that the flag is set to “1”. In the freezer compartment 104C shown in FIG. 7B, the surface 104C-2 on which the articles are placed protrudes to the front when the door 104C-1 is opened so that the articles can be taken in and out using the robot 102. Such a mechanism may be provided. In this case, the coordinates of the four corners (X21, Y21, Z21), (X22, Y22, (Z22), (X23, Y23, Z23) and (X24, Y24, Z24) are given. The mouth pot 102 carries out articles only when the surface 104C-2 is protruding (that is, when the door 104C-1 is open). It is also possible to perform the operation of placing an object on the surface 104C-2 and the operation of removing the object installed on the surface 104C-2 while referring to the coordinate values of the surface 104C-2. It is. When the door 104C-1 is closed, the surface (article installation surface) 104C-2 is stored in the freezer compartment 104C. Accordingly, the actual coordinate value of the surface 104C-2 changes. However, when the door 104C-1 is closed, the robot 102 does not insert or remove an article into the freezer compartment 104C, so that the coordinate values described as the equipment attribute data 602 are not changed. Leave as it is.

In this embodiment, it is determined whether or not an article can be installed as the equipment attribute data 602. Although only the identification flag is shown, other information may be added as needed. For example, similarly to the environmental attribute data 6001, the surface material may be added. Also, a trajectory of the approach of the robot hand 202 to the surface for placing an article on the surface or removing an object from the surface may be added. Further, a program for moving the mouth bot hand 202 can be stored and used. For example, a standard program specification for moving the robot arm 201 is determined in advance, and when using a robot 102 capable of arm control according to the specification, one of the equipment attribute data 602 is used. The program stored as a unit may be downloaded to the robot 102 as appropriate, and the downloaded program may be used to move the robot 102. This eliminates the need for the robot 102 to prepare individual gripping control programs for all equipment, and reduces the memory capacity for storing programs.

Fig. 18 shows the operation program of the robot arm 201 and the hand 202 of the robot 1◦2 (operation to open the door 104E-1 of the microwave oven 104E-1) as equipment attribute data. It is an explanatory view of a case. FIG. 19 is an example of an operation program of the robot arm 201 and the hand 202 of FIG. As the operation of opening the door 104 E-1 of the microwave oven 104 E as equipment, three operations are described and stored as a part of the equipment attribute data 602. Specifically, (i) an operation in which the arm 201 enters from the front of the microwave oven 104 E and moves to a position just before the microwave oven 104 E, and (ii) the hand 202 is directed upward. Move upward to handle 1 0 4 E-2 and grab handle 1 0 4 E-2. (Iii) Move forward with door 1 4 4 holding handle 1 0 4 E-2. The operation of opening E-1 is made up of. As described above, since the equipment has an operation corresponding to the structure unique to the equipment, it is possible to maintain the control of the mouth pot 102 in a general-purpose manner. The movements described in Fig. 19 are the coordinates of the tip of the mouth pot arm 201, the traveling vector of the arm 201, and the movement miracle of the tip of the arm 201 (movement miracle such as movement (iii)). In the case of linear approximation), the direction of the hand 202, and the operation of the hand 202 after the movement is completed. Note that the coordinates in Fig. 19 are all coordinate systems defined in the microwave oven, and the mouth pot 102 uses the robot's own coordinate system based on its own position / posture and the position / posture of the microwave oven 104E. To Convert and execute operation.

 First control means of environmental management server>

 The first control means 111 of the environment management server 101 is a part for controlling the entire environment management server 101. As described above, the main control contents include the following.

 That is, when there is an external inquiry via the first transmission / reception unit 110 regarding various data in the server 101, the first control means 111 sends the article search and management means 1 according to the contents. A request for reference to the data is issued to the environment map information management means 106 or 06.

 Further, the first control means 111 sends the result transmitted from the article search / management means 106 or the environment map information management means 108 in response to the request to the first transmission / reception unit 110. Send to the inquiry source via

 Further, the first control means 111 interprets the request for registration / update of various data in the server 101 sent from the outside via the first transmission / reception unit 110, and interprets the content. Accordingly, the data registration / update request is issued to the article search / management means 106 or the environment map information management means 108.

 One facility—

 The facility 104, which is the second subsystem of the life support system 100 of the present embodiment, is a facility having a place for storing or installing articles for a certain purpose (for example, a storage or installation body). is there. Here, "with purpose" means, for example, "save" in a refrigerator or "warm" in a microwave oven. In addition, the term “storage” is generally used to mean storage or the like, but the storage in the present embodiment also includes temporarily putting an article in a place where the above-mentioned purpose is applied. Therefore, putting food in a refrigerator or microwave oven is also called storage. The installation also includes temporarily placing the article in a location for the purpose.

As shown in FIG. 1, the equipment 104 has, as its basic configuration, an equipment operation information storage unit 122 for operating the equipment 104 in response to an operation instruction from the outside, and the equipment 1 A fourth sensing unit 1 2 3 for acquiring raw data belonging to another article in 04, and receiving the operation instruction from outside or transmitting the result of performing the operation to the instruction source. The fourth transceiver 140 controls the fourth transceiver 140, the fourth transceiver 140, the facility operation information storage 122, and the fourth sensing unit 123, respectively, for example, when the fourth transceiver 140 receives an external operation instruction. The fourth control unit 121 controls the operation of the facility 104 and transmits the result of the operation according to the operation instruction from the fourth transmission / reception unit 140 to an instruction source.

 <Fourth sensing part of equipment>

 The fourth sensing unit 123 is similar to the first sensing unit 105 of the environmental management server 101. That is, the fourth sensing unit 123 is a device that performs sensing in order to grasp the situation in each of the facilities 104, and transmits the sensed information and the structural information of the facility 104 to a predetermined device by using the fourth control unit 121. It is connected to the. The fourth sensing unit 123 constantly monitors the positions and states of all the monitoring objects existing in the equipment 104 in which the fourth sensing unit 123 is located, that is, the articles. In addition, the fourth sensing unit 123 detects, when a new article is newly brought into the facility 104 by a person or a mouth bot 102 or the like, it is also detected. However, it is assumed that the sensed information and the structural information of the equipment 104 are stored in the article database 107 and the environment map information database 109 of the environment management server 101 via the network 98. Although the specific configuration of the fourth sensing unit 123 is not particularly limited, for example, similarly to the first sensing unit 1 セ ン シ ン グ 5, it is preferable to use a device using an image sensor, a device using an electronic tag, or the like. Can be. By constructing the fourth sensing unit 123 as an example with a camera 123A (see FIG. 10C), an intuitive Graphical Interface (GUI) using actual images in the facility 104 can be realized. Can be.

 <Equipment operation information storage unit>

The facility operation information storage unit 122 mainly stores a command (facility operation command) for remotely controlling the facility 104 from outside. FIG. 8 is a diagram showing the equipment operation commands stored in the equipment operation information storage unit 122 in a table format. The descriptive information in the table is, in order from the left column, a facility ID for distinguishing facilities existing in the environment, a facility operation command name for controlling the facility from outside, a processing procedure corresponding to the command, It is the return value of the result of processing. Here, as an example of equipment, is distinguished by facilities ID "Co 1 d_r o om # 0 001" (the refrigerating compartment), "F ree _{Z er} # 0001" (freezer compartment), "M icro wa ve # oven # 0001 "(microwave oven), and the operation instruction command is shown for each. Next, the meaning of the description information will be described using those examples.

 In the first two examples, refrigeration compartment 104D and freezer compartment 104C,

 • "door #operi"

 ■ "d o r #c l o s e"

The two are prepared. The refrigerator compartment 104D and the freezer compartment 104C transmit these commands from an external device (for example, the robot 102, a personal computer or an operation terminal such as a PDA (Personal Digital Assistance) or a mobile phone) to the refrigerator compartment 104B-2 (the 4) It functions as an example of the transmission / reception unit 140.) When the reception is performed by the equipment, as shown in the processing procedure of FIG. It is performed under the control of B-1 (which functions as an example of the fourth control means 121). Therefore, the doors 104D-1 and 104C-1 of the refrigerator compartment 104D and the freezer compartment 104C are connected to the refrigerator compartment automatic opening and closing mechanism 104B-3 and the refrigerator compartment door automatic opening and closing mechanism 104B-4 by the refrigerator control means 104B. The operation is controlled by -1, and each is automatically and independently opened and closed. Next, when the processing of each equipment operation command is completed normally, `` Ac k '' is given to the external device that is the command source, and when the processing of the equipment operation command fails, the command source is “Nack” is returned as a return value to the external device from the refrigerator transmitting / receiving section 104 B-2 by the refrigerator control means 104B-1. If the equipment 104 is a refrigerator compartment 104D or a freezer compartment 104C, it is desirable that the inside of each compartment can be seen through without opening each door. You can prepare two commands, “door # transparent #on” and “idoor # transparent # off”. A door that can be switched between transparent and non-transparent is, for example, a refrigerator door with a liquid crystal shutter or a blind. It can be realized by switching between transparent and non-transparent by 1.

 In the third example, the microwave oven 104E,

 "D oor #open"

 "D oor #c lose"

 "Wa rm # s t a r t"

 "W a rm # e n d"

 "Is s #o b j e c t #i n"

There are five types available. Of these, "dor # open" and "dor # c1ose" are the same as the refrigerator compartment 104D and the freezer compartment 104C, and therefore will not be described.

When “warm # start” is received as an equipment operation command from an external device (for example, the mouth pot 102) by the microwave transmission / reception unit (which functions as another example of the fourth transmission / reception unit 140), FIG. As shown in the processing procedure, the heating is started under the control of the microwave control means (which functions as another example of the fourth control means 121). At this time, if there is an article in the microwave oven and the warming process can be started, “Ac k” is sent to the command source, and otherwise, the external device that is the command source is sent to the external device. “Nack” is returned from the microwave transmitting / receiving unit by the microwave control means as return values. When the equipment operation command “wa rm # end” is received from the external device by the microwave transmitter / receiver, it is checked by the microwave control means whether the heating has been completed, and if the heating has been completed by the microwave control means, “True” If the heating process is still in progress, “F a 1 se” is returned from the microwave transmitting / receiving unit as a return value to the external device that is the source of the command. Upon reception facility operation command to "i _s #object # in" from an external device in a microwave oven transceiver, whether there is an article in the compartment of the microwave oven examined by electronic range control means, the microwave oven control unit, if If there is an article, “True” is returned. If there is no article, “Fa1se” is returned from the microwave transmitting / receiving unit as a return value to the external device that is the source of the command. At this time, the presence or absence of the article can be confirmed by using an image sensor and a weight sensor, or an electronic tag sensor if an electronic tag is attached to the article. As for equipment 104, refrigerator compartment 104D, freezer compartment 104C and electric

The examples of the facility operation commands are briefly described by giving three examples of 04 E. However, the necessary facility operation commands may be prepared according to the function of each facility 104. Also, when the manufacturer of the facility 104 prepares a new facility operation command for the facility 104, the new command is written to the storage means of the facility 104 using any storage medium, or If the relevant equipment 104 is connected to the manufacturer via the external network 98, the equipment operation command is sent to the equipment 104 via the network 98, written into the storage means, and used as a new operation command. It can be possible.

 —Operation terminal

 The operation terminal 103, which is the third subsystem of the life support system 100, is a terminal device for the user to instruct the operation of articles in the environment.

 As shown in FIG. 1, the operation terminal 103 has a basic configuration in which an article movement instruction is performed by a user's operation of designating an article, moving the article to a predetermined place, or the like. An operation device 1 18 for inputting an operation instruction such as an operation instruction, a display device 1 19 for displaying an operation screen, and a display on the operation screen of the display device 1 19 for indicating an article or a location in the equipment. An in-facility display unit 124 that generates a form and superimposes and synthesizes a balloon image as shown in FIGS. 10 to 1OC and FIG. 12 on an image of the real environment, and an article operation device 1 3rd transmission / reception unit 14 2, 3rd transmission / reception unit 14 2, material operation device 1 18, and display means in equipment that send the instruction content of goods operation input at 18 to the environmental management server 101 1 and 4, respectively, for example, an operation to specify an article from the article operation device 1 18 or Made more structure and the third control means 1 2 0 controlling the operation to perform by connexion article movement instruction to the operation or the like to move the article in place.

 <Article handling device>

In the article operation device 118, the user gives an instruction for article operation while referring to the image displayed on the display device 119. For example, a real-time video image of the actual situation taken by a camera or the like is displayed on the display device 119 as it is, and the user operates the article while watching the video image. For example, a pointing device (mouse, Touch panel, line-of-sight input) and keyboard cursor movement are used. 4A and 4B were used to explain the contents of the article database 107 described above.On the other hand, these figures also show real-time images of actual situations taken with cameras or the like. Corresponding (excluding lead lines and reference symbols). That is, FIGS. 4A and 4B show examples of operation screens before the article operation (before article clearing) and after the article operation (after article clearing), respectively.

 Here, it is assumed that the following operation has been performed as an example of an operation of designating an article and an operation of moving the designated article to a predetermined location using the operation screens of FIGS. 4A and 4B. That is,

 1) Move the mouse cursor to the item you want to operate on the screen.

 2) Determine the item by dragging, move the cursor to the position you want to move and drop it.

 The above operation is a very simple operation consisting of two steps. The curve with the arrow in Fig. 4A is an example of the movement locus of the mouse cursor during drag and drop. For example, if you want to place Notebook S # 0 0 0 1 on the tape, first move the mouse cursor to Notebook S # 0 0 0 1 and drag it, then move the cursor over the table in that state, Drop the mouse.

 By the way, in order to convert such an operation into instruction data described below, it is not enough to simply display a real-time image of the camera on the screen as it is, but the following processes 1) and 2) are required. . That is,

 1) Recognition processing of what force the article indicated with the mouse is on the screen, and where the drop location is (operation outside the equipment)

 2) To indicate an article in an area that cannot be seen by the camera image, for example, an article inside the facility 104 or an article hidden behind a shadow, or move the indicated article to an area that cannot be seen by the camera image. In such a case, the process of specifying the location (operation of the inside of the equipment) is required. Next, embodiments of these two points will be described.

 Operation outside the equipment>

First, how to indicate items and places other than inside the equipment (such as the interior of the collection room) will be explained. For example, in the previous example of moving a notebook, the user Move the mouse cursor to the location of "Notebook S # 0001" and perform an operation to tick "Notebook S # 00 01". At this time, in order for the system to understand that the object to be moved is “Notebook S # 0 001”, screen position data of “Notebook S # 0001” corresponding to the display screen on a one-to-one basis is necessary. Become. The simplest method of expressing the position data is a method using article mask data that has a one-to-one correspondence with the display screen. Here, “corresponding one-to-one” means that the coordinate values correspond.

 When the position of an article on the display screen is specified by a coordinate value, the same coordinate value is then referred to in the article mask data. In the article mask data, a pointer to the article data is described as an attribute immediately at the coordinates where the article exists. By following the pointer and referring to the article data, it is possible to know what the article on the display screen is.

 The problem here is how to create the article mask data. If the camera image for detecting an article using the background subtraction method described above is used as it is in a display screen for article operation, the result of the background subtraction method may be used. This is because the position of the article in the screen is detected as a result of the background subtraction, and can be used as it is as the mask data.

 On the other hand, the process of recognizing the moving location is performed, for example, using a screen map in which the video of the operation screen and the coordinates correspond in advance. 9A and 9B are conceptual diagrams showing examples of an operation screen and a corresponding screen map. FIG. 9A shows an operation screen, and FIG. 9B shows a screen map. As can be seen from this example, the screen map indicates where on the operation screen the entities corresponding to the environment attribute data 601 and the equipment attribute data 602 specified in the above-described environment map information database 109 exist. This is the data to be distinguished. The screen map can be stored, for example, as image data. For example, a screen map is image data in which each pixel is represented by an integer value, and a pair of the value and the location is

 0: Nothing

 1: Floor `` f l oor # 0001 ''

 2: Table "t ab l e # 0001J

3: Cold room `` cold om # 0001 '' 4: Freezer “freezer # 000 1”

 5: General waste bin (waste basket) “wastebaskeet # 000 1”

 6: Recycling bin "wastebasket # 0002"

 7: Cupboard "c u p b o a r d # 0001"

It should be decided in advance such as. To determine what exists at the appropriate location (pixel) at the specified location on the operation screen, check the value of the pixel at the same location in the corresponding environment map information database 109. I do. However, in this determination process, first, a process for checking whether or not the article is in force is performed, and after it is confirmed that there is no article in the place, it is not necessary to check what exists in the specified place. Not a horse.

 The examples given here include refrigerators and freezers that have an internal structure, but in order to specify more specific items and locations inside them, it is necessary to perform operation processing inside the equipment described below. is there. Of course, if the place to store the goods can be at any position in the refrigerator compartment and the freezer compartment, the method of indicating the place described here is sufficient.

 The equipment 104 to be registered in the environmental map information database 109 is hardly moved once in principle once installed. Therefore, if it is difficult to automatically create the environmental map information database 109, it may be created manually. Of course, if the three-dimensional model of each facility 104 is known and the location of the installation can be calculated accurately, the creation of the environment map information database 109 can be automated.

 <Operation inside the equipment> ''

 When an article or place to be indicated on the screen is inside the facility 104 or is hidden behind the facility 104, such an article or place does not appear in the camera image, and the instruction cannot be performed as it is. Therefore, next, an example of a simple method for designating an article and a place where the article 104 is located will be described.

 First, regardless of whether an article to be operated is dragged or not, whether or not the location designated by the mouse cursor using the mouse as an example of the article operating device 118 is a facility is, as described above, the display device 1 19 You can find out by using the screen map (see Figure 9B) displayed on the screen.

Then, the article operation device 1 1 8 was designated by a mouse cursor using a mouse as an example. If the location is the facility 104, under the control of the third control means 120, the internal structure of the facility 104 and the articles inside it are displayed in some form by the facility display means 124. The user can indicate the position and the article inside the equipment with the mouse cursor. The structure and goods inside the equipment can be known by referring to the environmental map information database 109 and the goods database 107. As described above, in the article database 107, for example, when articles are stored inside the facility 104 or installed at the top of the facility 104, the article database 1 07 The information of the article storage or installation location has been updated. Therefore, under the control of the third control means 120, what kind of articles are stored or installed in each facility 104 is determined by the third transmission / reception unit 142, the network 98, and the first transmission / reception unit. It can be easily grasped only by referring to the article database 107 via the 110, the first control means 111, and the article search / management means 106. As a display form of the structure and articles inside the facility 104 by the in-facility display means 124, for example, the following can be considered.

 (1) First display format

 In the first display mode, as shown in FIG. 10A, each article image inside the equipment is displayed for a certain time, for example, in a pop-up display. The specification of a specific article may be performed by moving the mouse cursor indicated by an arrow to the display position of the specific article while each article is displayed for a certain period of time, or of course, using another method. It does not matter. FIG. 10A shows an example in which, when the mouse cursor indicated by an arrow is moved to the freezer compartment, an image of the ice cream, which is an article stored therein, is displayed in a pop-up. This display example is suitable for moving an article in the equipment to another. Of course, characters may be used instead of the image. In the first display mode, when a product is moved into the facility, the location in the facility is left to the life support system (specifically, the following method).

In other words, the search operation to determine where to place the article when moving the article into the facility will be described. The first control means 111 searches for a place where the article can be installed, based on the information of the environment map information management means 108 and the article search management means 106. The position and size (shape) of the already installed articles are the article search management means 10 6, it is possible to find a place that is not already occupied by another item in the environment map (for example, the second shelf of the refrigerator) managed by the environment map information management means 108. it can. FIG. 20 is a diagram showing a method of obtaining an installation possibility. This example shows the case where the equipment to be installed is the refrigerator compartment 104D and the items (food dishes) on the second shelf 104D-2 from the bottom are to be installed. First, equipment attribute data 6 0

Calculate the size of the second shelf 1 0 4 D—2 from the bottom of the refrigerator compartment 10 4 D from 2 and prepare an image 10 4 D— 2 0 (all pixel values are 0) of that size . Then, the position and shape of the items (the PET bottles and bananas in Fig. 20) placed on the second stage from the bottom of the refrigerator compartment 104D using the item search management means 106 on top of them Is obtained, and the image of the second shelf 104D-2 is filled with a pixel value of 1 on the image 104D-2 (see the filled portion 104D-3). Then, the remaining portion of pixel 0 is a place where the article can be placed. Further, a template 104D-4 of the size of the article (dish) to be installed is prepared, and the template 104D-4 is accommodated in the pixel 0 area. (The trajectory is indicated by 104D-6.) Then, the location 104D-5 where the article (dish) to be installed fits is determined and set as the installation position.

 (2) Second display format

 In the second display mode, hierarchical display is performed for each type as shown in FIG. 10B. If the equipment 104 indicated by the arrow indicated by the mouse cursor is the refrigerator compartment 104D, the food categories such as meat and vegetables are displayed first, and then the food corresponding to the category selected hierarchically Is displayed. For example, pork and chicken are displayed for meat, and carrots and lettuce are displayed for vegetables. This second display mode is suitable for searching for desired items without making the structure of the equipment visible to the user (operator).

 Here, some of the equipment 104 may have a plurality of storage locations or installation locations (such as shelves), but the storage location or installation location may be displayed in a hierarchy. When displaying an image of an article as in the first display mode, a complicated screen is expected if a large number of articles are present in the facility. Therefore, it is effective to display hierarchically as in the second display mode.

If the equipment 104 indicated by the mouse cursor indicated by the arrow is the refrigerator 104B, the foods (articles) are displayed in order of decreasing expiration date, such as by 10 pieces. The display may be performed by changing the color or the size of the character. In addition, it is also possible to devise the display so that it is easy to see as appropriate, such as displaying the refrigerator 104B from the one that has been put in and out frequently. Displaying according to the number of times of putting in and out means, for example, in the case of a responsible article, the more the number of times of putting in and out, the more the quality deteriorates, so it means that such a thing is displayed more conspicuously and better. It also means to give priority to what is used. In addition, it is possible to input user information, and to filter displayed articles according to power and preference as to whether or not the item belongs to the user.

 (3) Third display format

 As shown in Fig. 10C, when a camera 123A is installed in a refrigerator compartment 104D, which is an example of equipment, to sense goods (for example, using the fourth sensing unit 123) When pointing to the refrigerator compartment 104D with the mouse cursor indicated by an arrow, the image in the refrigerator compartment 104D is displayed, and the same technology as that for pointing the goods and places on the floor and table is used. An article or location on the image in the refrigerator compartment 104D is indicated by a mouse cursor. At this time, operability is further improved by enlarging and displaying the image in the refrigerator compartment 104D to be displayed. Also, if the door inside the refrigerator compartment 104D is closed, the interior will be dark and it will not be possible to take a picture with the power of the camera. Is desirable. Furthermore, by displaying the photographed image of the refrigerator compartment 104D on the door of the refrigerator compartment 104D on the screen, it is possible to intuitively operate the instruction with the same appearance as usual.

 (4) Fourth display format

 In the fourth display mode, as shown in FIG. 10D, when an instruction operation such as overlaying a mouse cursor indicated by an arrow on the equipment 104 is performed, if the equipment 104 is a storage body, the door is operated. By automatically opening the key, the inside of the equipment is displayed directly on the display screen. As described above, an operation instruction command is prepared for each facility (for example, a storage body), and such processing can be realized using the operation instruction command.

Specifically, as shown by an arrow in FIG. 1 OD, when a desired equipment (for example, a storage body) (for example, a refrigerator) is instructed by a mouse cursor with the article operating device 118, the third control means 1 2 0, the setting is performed via the third transmitting and receiving The command “door #ope nj” is sent to the fourth control means 121 via the fourth transmission / reception unit 140 of the refrigerator 140B having “C o 1 d—ro om # 0001” (refrigeration room), which is an example of the storage 104. Then, the door of the refrigerator compartment is opened, and the interior of the refrigerator compartment is directly displayed on the display screen, and when it is determined that the instruction of the article or the storage place has been completed by the article operating device 118, the third control means 120, (3) A command of “door #clos eJ” is transmitted to the fourth control means 121 via the fourth transmission / reception unit 140 of the refrigerator 140 B via the transmission / reception unit 142 and the network 98. Since the doors are opened and closed, facilities that do not control temperature and humidity, such as closets and shoe boxes, are more suitable than refrigerators.

 (5) Fifth display format

 As shown in Figure 10E, in the case of equipment that controls temperature, humidity, etc., such as a refrigerator compartment or a freezer compartment, some or all of the walls are made walls that can be switched between transparent and non-transparent. A fifth display mode in which the wall surface is transmitted and non-transmitted at the same timing as the opening and closing of the door described above may be adopted. Install such a transparent / non-transparent door in the refrigerator or freezer, and use the transparent command “dor #t r a n s p a r en t #on” or “d oor

By preparing a non-transparent command of # transparent # of fJ, each of the display modes can be realized by using such a command. It is also effective to switch these display modes according to the situation or the type of equipment. For example, when the dragged item is placed in the facility, it is displayed in the first display mode, while when the user designates the item in the facility, it is displayed in any of the second to fourth display modes. In order to realize such a switching process, a display form table as shown in FIG. 11 is prepared in the storage section of the in-facility display means 124, and the in-facility display means 124 performs a user operation based on this table. And the display mode is switched according to the equipment. The numbers of the display forms (forms 1 to 5) in FIG. 11 correspond to the first to fifth display forms in FIGS. 10A to 10E, respectively. Alternatively, the display form may be changed according to the person who operates, the equipment to be operated, or the article to be operated. That is, the display mode may be appropriately selected according to various attributes. In addition, it is also possible to select and display the goods that exist in the equipment according to the user. To do this, use the "Mainly Using the data of `` users and owners '', if the user is a `` child '' and instructed the storage, control to give priority to the foods often consumed by children, such as `` bananas '' can do. Also, by displaying only the items owned by the user, it can be expected that the display will not be bothersome. Here, for example, the user is able to use the system by logging in to the life support system, and the user is determined by the session of the mouth login. It is possible to do so. Of course, it is also possible to use authentication technologies such as fingerprint recognition and iris recognition. In addition, when not displayed in the article database of FIGS. 3A and 3B, but using a tag reader for sensing, the expiration date of each article can also be recorded in the article database. It is also effective to preferentially display items with earlier expiration dates by using.

 Next, a description will be given of a display mode in a case where an article or place to be instructed on the screen is hidden by the facility. The area hidden by the equipment can be indicated by a mouse or the like, similarly to the inside of the equipment described above. However, in this case, thereafter, it is necessary to indicate the position in the depth direction from the line of sight of the camera 105A, such as whether it is inside the facility, behind it, or inside the facility behind it. One way to achieve this is to consider the other equipment hidden behind it, the floor of the room, etc. as part of its own structure for the equipment in the front, and 3 The display mode can be applied as it is.

 FIG. 12A shows an example in which this is realized in the pop-up list format of the first display mode. The user is an unspecified person, and the user moves the mouse cursor using a mouse, which is an example of the article operation device 118, to the cupboard 104M in front of the operation screen of the display device 119. When adjusted (or clicked), under the control of the third control means 120, the in-facility display means 124 allows the front cupboards 104 M, and the cupboards 104. Behind M (hidden behind cupboard 104 M, respectively) The contents of the storage room 104 B and floor 104 H are popped up, and the items stored or installed in each are displayed. It is possible to specify with a mouse.

In addition, FIG. 12B shows that when the user is a “child”, the child moves the mouse cursor with a mouse, which is an example of the article operation device 118, in front of the operation screen of the display device 119. The following is an example of the display when matching (or clicking) the cupboard 104M in. Here, of the items popped up in Fig. 12A, only items that are mainly used by children are displayed.

Here, a method is needed to know how much equipment is hidden behind the point indicated by the mouse. As described above, there is a method using a screen map (see Fig. 9B) as a method for checking whether the point indicated by the mouse is a facility. Here, this is extended and a sufficient number of screen maps (hierarchical screen maps) are prepared in the in-facility display means 124, and under the control of the third control means 120, the in-facility display means 1 According to 24, all equipments are arranged on each screen map so that multiple equipments do not overlap in one screen map. Then, the facility power ^s including the point indicated by the mouse on each screen map is present in the depth direction of the camera's line of sight, and the plurality of facilities and the floor hidden by any of the plurality of facilities are: It will be displayed as the contents of the pop-up list. FIGS. 13 to 13 are hierarchical screen maps corresponding to the environment of FIG. 12A. Fig. 13A is the screen map of the innermost part (floor and wall screen map), and Fig. 13B is the screen map of one nearer from the back (the floor and wall, and the screen map of the refrigerator with two wastebaskets in addition to the floor). 13C is the screen map of the foreground (screen map of the cupboard in addition to the floor and wall), and Fig. 13D is the screen map of the foreground (screen map of the table in addition to the floor and wall). . As can be seen from Figs. 13A to 13D, the equipment does not overlap in each screen map.

In addition, the display inside the equipment behind the point indicated by the mouse is displayed on another switch-like input device (for example, wheel ^ <mouse wheel or the up and down arrow keys on the keyboard), which is an example of the article operation device 118. Each time the user inputs a new one, the screen map can be selectively changed from the near side or the back side by the in-facility display means 124 under the control of the third control means 120. For example, in the example of FIG. 12A, when the mouse cursor is pointed at the cupboard 104M by the in-facility display means 124 under the control of the third control means 120, the cupboard 104 The contents of the items stored in M are displayed in a pop-up.Next, press the “Up” arrow key once to display the contents of the items stored in the refrigerator 10 The content pops up, and again pressing the “Up” arrow key will pop up the content of the item placed on floor 104H. , Etc. Then, under the control of the third control means 120, the in-facility display means 124 presses the "up" arrow key once again, thereby again displaying the articles stored in the cupboard 104M. You can return to the content. Conversely, under the control of the third control means 120, the in-facility display means 124 can display the contents in a pop-up order in the reverse order with the "down" arrow key. In addition, after dropping the item in the pop-up display, or dragging the dropped item to the destination (pop-up display), the display in the facility is controlled under the control of the third control means 120. With the means 124, the pop-up display can be erased.

 <Third control means of operation terminal>

The third control means 120 receives the above-mentioned operation instruction from the article operation device 118, generates instruction data, and sends the instruction data to the second transmitting / receiving unit 141 of the robot 102. The transmission is performed via the third transmission / reception unit 142 and the network 98. The instruction data is data from which an action plan of the robot 102 is created by the action plan creation means 117 of the robot 102. This instruction data has the following two values: (article to be operated, destination). For example, when a notebook is moved on a table, "notebook S # 001, table" is the instruction data. As the destination, only the location registered in the environment attribute data 601 or the facility attribute data 602 registered in the environment map information database can be designated. If the destination cannot be specified only by the name of the destination because the destination has a certain extent, the specific coordinates are added to the destination and the specific coordinate values are added to the real world coordinate system (the actual position based on the environment). This is a coordinate system that expresses the coordinates, and may be specified by the coordinate values of the coordinate system shown in Figs. 5A to 5C. For example, when an article is placed on a fixed place on the floor, this is specified as "article, floor (xl, y1, 0)". The coordinate values in the real world coordinate system of the location designated on the display screen are stored in the environment map information database 109, for example, the information of the three-dimensional model version environment map shown in FIG. 5B. It can be calculated by control means from the camera 105A parameters (position, posture, angle of view, etc. of the camera 105A) that are displaying the image. Since such a calculation method is a basic well-known technique of computer graphics, its description will be omitted. If the three-dimensional model and the camera parameters are known, Of the environment map information database 109 can be obtained by calculation. Bite potoo

 The robot 102, which is the fourth subsystem, plays a role of actually holding and carrying articles in the environment in the life support system 100 according to the present embodiment. As shown in FIG. 1 and FIG. 14, the mouth pot 102 has, as a basic configuration, an article 400 that detects an obstacle around the mouth bot 102 and grips the object. (For example, an obstacle sensor) that obtains information of the robot, a gripper that grips the article, and a movement that plans the movement of the robot using the environment map information database. Move the plan creation means 1 1 4, the action plan creation means 1 1 Ί for drafting the action plan of the robot 102 according to the contents of the instruction to execute the instruction from the user, and the robot 102. The drive unit 1 15 to be operated, the first transmission / reception unit 110 of the environmental management server 101, the third transmission / reception unit 142 of the operation terminal 103, and the fourth transmission / reception unit 14 of the facility 104 0, a second transmission / reception unit 14 4 1 that transmits and receives various data via the network 98, a sensor 1 1 2, a second transmission / reception unit 1 4 1, and a grip unit 1 1 3 The second control means 1 1 6 (article movement control means) for controlling the movement of the mouth bot 102 by controlling the movement plan creating means 1 1 4, the action plan creating means 1 17 and the driving section 1 15 respectively. ).

 When the robot 102 grasps and transports the article, the operation terminal 103 (specifically, the article operating device 118, the third control means 120, and the third transmitting / receiving unit 144) is used. The user issues an instruction, and the instruction data obtained by encoding the instruction content is transmitted to the network.

It is transmitted to the second transmission / reception unit 14 41 of the robot 102 via 98. robot

When the second transmission / reception unit 14 1 of 102 receives the instruction data, under the control of the second control unit 1 16, the action plan creation unit 1 17 sends the robot 10 2 itself from the instruction data. A list of robot control commands for action is generated, and the second control means 1 16 of the robot 102 processes the robot control commands in order to perform the gripping and transporting operation of the article 400. Execute.

The robot control command referred to here is a command for performing gripping by the robot 102, movement of the robot 102, and control of the facility 104 related to the operation of the robot 102. When divided, mainly ■ grip

 'Release

 • Equipment operation

There are four types. Hereinafter, these will be described.

 “Move” is represented by “moVe, coordinate value” or “moVe, equipment ID”. Upon receiving this command, the robot 102 moves from the current position to the position specified by the coordinate value or the device 104 specified by the device ID. Here, the coordinate values are specified in the real world coordinate system, and the movement route is planned by the movement plan creation means 114. When moving to the facility 104, the movement plan creating means 114 creates a route that approaches the facility 104 to a predetermined distance. The coordinates of the location of the facility 104 can be obtained by referring to the facility attribute data 602 in the environment map information database 109 via the network 98.

 “Kibamo” is represented by “grab, article IDJ. When this command is received, the robot 102 grasps the article 400 specified by the article ID. The location of the article 400 is After grasping by referring to the article database 107 described above via the network 98, a grasp plan as an example of the action plan is created by the action plan creating means 117, and based on the created grasp plan, Then, a gripping plan is performed by the gripping units 113 to grip the article 400.

 "Release" is represented by "re1ease". Upon receiving this command, the robot 102 releases the hand 202 constituting the gripper 113, and releases the article 400 held by the hand 202.

“Equipment operation” is represented by “the robot's own ID, equipment ID, and equipment operation command J. When this command is received, the robot 102 specifies the equipment 104 specified by the equipment ID. The obtained equipment operation command is transmitted via the network 98. The equipment operation command is an operation instruction command received by the individual equipment 104 from the external device, and each equipment 104 is connected to the network 98. When the operation instruction command is received via the control instruction unit, a process corresponding to the operation instruction command is performed under the control of the respective control means. When a facility operation command is sent to a certain facility 104, the received facility 104 processes the command in accordance with the operation instruction command, and then sends the processing result from the facility 104 to the robot 102 itself. This is to get a reply via 8. The content of this reply allows the robot 102 to check and confirm whether or not the force has been processed by the facility 104 in accordance with the operation instruction command.

 Above, four types of robot control commands have been described as typical examples. However, the robot control commands are not limited to these four types, and needless to say, may be increased or decreased as needed.

 FIG. 14 is a schematic diagram showing an example of the robot 102. Hereinafter, in FIG. 14, the direction in which the tip of the arm 102 faces is the front of the mouth pot 102, and the robot

Each means or each part of 102 will be described.

 Drive unit>

 Two driving units 1 15 are provided on each side of the pot body 102 A, and a total of four wheels 1 15 A and four wheels 1 15 A or at least two wheels 1 15 on both sides And a driving device such as a motor for driving A. In the present embodiment, an example of wheels is shown as the drive unit 115. However, an optimum device or mechanism may be selected for the drive unit 115 according to the place or environment where the robot 102 is used. For example, when moving on uneven ground, a crawler-type multi-legged drive may be used. In addition, when the entire area of a house including a room, which is an example of an environment, is set as a movable range, the driving unit 1 15 is not necessarily required when the gripping unit 113 including the arm 201 and the hand 202 is an example of an environment. is not.

 <Sensor>

The sensor 112 detects an obstacle or the like in the vicinity of the robot 102. In the present embodiment, the sensor 112 functions as an example of the ultrasonic sensor 112a and the visual sensor, and the robot body 1 It is composed of a stereo camera 111b arranged on the front of 02A, and a collision sensor 112c arranged on the front and back of the robot body 102A. The ultrasonic sensors 111a are attached to the front, back, left, and right sides of the robot main body 102A at three force points each, and measure the time it takes to emit ultrasonic waves and receive the reflected waves. And the distance from the ultrasonic sensor 1 1 2a to the obstacle It calculates the approximate distance. In the present embodiment, a short-range obstacle is detected before the collision by the ultrasonic sensor 112a. The stereo camera 1 1 2b inputs the surrounding situation as an image, and performs processing such as recognition on the image by the second control means 1 16 so that it can judge the presence or absence of obstacles and more accurately identify the object to be grasped. Information is obtained by the second control means 116. The collision sensor 112c is a sensor that detects that the sensor 112c has an impact with a predetermined force. The collision sensor 112c detects that the robot has come or that the robot 102 itself has hit while moving.

 <Movement planning method>

 When receiving the robot control command to move the robot 102 to the designated place, the movement plan creating means 114 sends the movement route from the current position to the designated place via the network 98. It is created using the environment map information database 109 acquired from the environment management server 101. Naturally, if there is an obstacle between the current position and the destination, a route is needed to avoid it. Since the area in which 2 is movable is described in advance, the movement route in that area may be created by the movement plan creating means 114. Once the movement route is created by the movement plan creation means 114, the sensor detects an obstacle after the robot 102 starts to move under the control of the second control means 116. In this case, a new route for avoiding the obstacle is recreated by the movement plan creating means 114 each time. In creating a moving route, the Dijkstra method, which is the most general method, is used.

 <Grip part>

The gripper 113 is a device or mechanism for gripping an article. In the present embodiment, an arm 201 composed of multiple joints as shown in FIG. 14 and a hand disposed at the tip of the arm 201 are provided. It is composed of 202. When a gripping position is instructed by a robot control command described later, the gripper 113 moves the tip of the arm 201 to that position, and performs a gripping operation of the hand 202. The arm control for moving the hand 202 to the gripping position may be performed by the gripper 113. The gripper 113 also performs a release operation of the hand 202 when instructed to release by the robot control command. Robot second control means>

 The second control means 1 16 parses the list of robot control commands sent from the external device via the network 98 and the second transmitting / receiving section 141, and sequentially executes the robot control commands. To go. If the sent robot control command is the above-mentioned instruction data, the contents are converted into action plan creation means 11 1 in order to convert the instruction data into an executable pot control command of the robot 102. 7 and receive the processed result and execute the robot control commands in order.

 Action plan creation means>

 The action plan creating means 1 17 can be operated by the robot by simply performing a simple operation of a user designating an article and moving the designated article to a predetermined place using the article operating device 1 18 of the operation terminal 103. This is a means provided so that a work instruction to 102 can be given. Specifically, when the robot 102 receives the instruction data from the operation terminal 103 via the network 98, the action plan creating means 117 may, if necessary, change the second control means 116. While referring to the robot control command DB (database) 90 connected to the CPU, a list of robot control commands for the mouth pot 102 to execute the continuous operation is generated based on the instruction data.

 Here, in the mouth bot control command DB90, a list of robot control commands when the robot 102 performs the equipment operation is stored in advance for each equipment. The reason is as follows.

As described above, the instruction data contains only two pieces of data, that is, two pieces of data of “article to be operated and destination”. If the target article is in a state where it can be grasped immediately before the mouth pot 102 or the moving place is a spatially open place, the robot control command DB 90 is not particularly necessary. However, in reality, the target article is rarely in front of the robot 102, and usually the mouth pot 102 (or the gripper 113) must move to the vicinity of the article to be operated. No. If the article is inside a facility closed by a door, the door must be opened, the article grasped, and then the door closed. Furthermore, depending on the equipment 104, more complicated processing may be required after storing or installing articles. However, these It is difficult to say that the system is easy to use if the user gives instructions one by one using the operation screen. If possible, the simple instruction operation just to generate the instruction data described earlier, that is, selecting the article and specifying the storage or installation location, the instruction to the robot 102 is mainly performed by only two operations. Is desirable.

 Therefore, knowledge data for creating a robot control command for causing the robot 102 to perform a predetermined operation including the operation of the equipment 104 from the instruction data generated by the simple instruction operation is required. Become. The mouth pot control command DB90 stores the knowledge data.

 FIG. 15 is a tabular diagram showing an example of a list of robot control commands stored in the robot control command DB90. The figure includes a table for two different facilities (refrigerator and microwave oven). In the leftmost column of each table, the ID of the facility 104 operated by the robot 102 is described. "Location attribute" in the right column indicates the source or destination,

 • Source: If you want to take out the goods while they are stored or installed in the facility 104 where the goods are located

 • Destination: This refers to the case where goods are stored or installed in a certain facility 104, and if necessary, some processing is performed on the goods stored or installed using the various functions of the above-mentioned facility 104. I do. The rightmost column shows the robot control command list corresponding to the location attribute.

 As an example, a robot control command list in the case where the equipment ID is “C o 1 d—ro o m # 0 0 0 1” (refrigerated room) and the attribute of the location is the movement source will be described. This is because, as a result of referring to the article database 107 regarding the article that is included in the first value of the instruction data, it is stored in “C o 1 d_ room # 0 0 0 1” (refrigerated room) It is a list of robot control commands in the case. The three commands here are in order

 • Open the door of “C o 1 d—ro o m # 0 0 0 1” (refrigerator),

 -Grab the item and take it out

 • "Close the door of Co 1d_room # 0 0 0 1 J (Refrigerator)

Operation. Here is the second Lopot control command, "Grab, $ object"

The “$ object” in the above means that the ID of the article to be operated is included. Information that changes in value depending on the situation is treated as a variable by prefixing a $, and when the article to handle it is specifically determined by the instruction data, a value is set for the variable. I do. By doing so, generality can be given to the robot control commands.

 Although a very simple example has been described here for understanding the contents of the mouth pot control command, a further necessary robot control command may be added for practical use. For example, the command list of the destination of the microwave oven is "Robot # 0001, Microwav e # ovene # 0001, doro

# ope ri "

 ": Rel e a se, $ obj e c tj

 "Robot # 0001, Microwa v e # ovene # 0001, doo r

# c 丄 o s e ”

 "Robot # 0001, Microwa v e #ovene # 0001, wa rm

# s t a r t "

Although these commands are composed of the following four commands, in reality the microwave oven already contains objects, it is not possible to put any more objects. Command to check if there is

 "Robot # 0001, Microwa v e #ovene # 0001, is #object #iri"

Is preferably provided. When this command returns "TRUEJ, that is, if there is an object in the microwave oven, instead of executing the subsequent command, the user receives the object in the microwave oven. You may tell them you are in, and end the process.

In addition, if this command sequence is applied to an item to be put into the microwave oven, there is a problem in that the process of "warming" is performed for any object. For this reason, for example, the microwave oven may be provided with a mechanism for recognizing the contents of the article and switching the specific heating method of the “warming” process accordingly. For example, If the microwave oven has the functions of "warm" and "thaw" as the detailed functions of the "warm-up", the thing put in the microwave oven can be used for any method, for example, image processing or object. The electronic tag may be recognized by a reader / writer arranged in a microwave oven or its vicinity, or the like, and the “heating of the dish” and the “thawing” may be appropriately switched according to the result. Of course, another method may be used for this switching. For example, if the microwave oven does not have the recognition function, the robot 102 may have the function, and the robot 102 may recognize the contents of the object and send the result to the microwave oven. .

 As described above, in the present embodiment, using such a robot control command DB 90, a series of robot control command lists for realizing the instruction data is generated and executed by the action plan creating unit 1 17. .

 It should be noted that, after the generation of the mouth pot control command list is completed, execution may be performed without confirming the user, but the content instructed by the article operating device 118 is incorrect, and the user You may not even notice. For example, in the environment shown in Fig. 4B, the user tried to put pizza in the refrigerator in a microwave oven and heat it.However, due to a user's misoperation of the mouse, the pizza in the refrigerator room was placed in the freezer compartment immediately below the microwave oven. In some cases. As a result, you may not be able to make hot pizzas forever. In order to prevent such a situation from occurring, the action plan of the robot 102 created by the action plan creation means 117 is presented to the user before execution, so that the user is prevented from erroneously operating the article. It is preferable to prompt the user to confirm whether or not there is.

Figure 16 shows a screen for confirming the instruction to move the goods. In this example, the moving object "pizza", the moving source "refrigerator", the moving destination "microwave oven", and the processing at the moving destination "warm" are displayed by characters. In order to further clarify the source and destination, a dotted arrowed curve from the source to the destination is shown. If the source or destination is not equipment, nothing may be displayed. For example, if the pizza is on the floor, the source is not a facility, so it is not displayed, and only the "microwave oven" at the destination need be displayed. Of course, this confirmation screen can be displayed in characters such as equipment or goods, Key display may be performed.

 Furthermore, if an error has occurred, a correction may be made in this confirmation processing. For example, in the above example, if the item to be moved to the microwave is “frozen meat” instead of “pizza”, then the processing in the microwave should be “thaw” instead of “warm”. As shown in Fig. 15, if the last command in the robot control command sequence at the destination of the microwave oven is only "warm" or "warm #start" as shown in Fig. 15, execute it. Resulting in. Therefore, even if the item put in the microwave is “frozen meat”, the microwave oven “warms” the “frozen meat” if the user does nothing. When such a processing error occurs, it may be corrected by the user before executing the processing.

 As described above, the living support system 100 based on FIG. 1 is composed of four subsystems: an environment management server 101, a robot 102, facilities 104, and an operation terminal 103. The subsystems are configured to exchange information with each other via a wireless or wired network 98. However, a configuration in which the operation terminal 103 is attached to the environment management server 101, the facility 104, the robot 102, or a plurality of locations thereof may be adopted. Also, the configuration may be such that a plurality of robots 102 work in parallel while cooperating, instead of one robot. In FIG. 1, for the sake of simplicity, only one facility 104 is shown.If there are a plurality of facilities, the plurality of facilities 104 are included in the life support system 100. Each is incorporated.

In the above description, the mechanism of the operation of the system when the user instructs only one operation of moving an article from one place to another and executes the operation is described. In practice, however, the user may actually want to give more than one instruction. In this case, if processing of one instruction is completed and the next instruction is input, the user cannot leave the system until all desired instructions have been issued; This makes the system difficult to use. Therefore, it is desirable that the system be designed to accept all the operations that the user wants to perform at once and process them sequentially. This allows, for example, if you want the system to heat the pizza in the refrigerator and then eat it, The following two instructions (not shown),

 ■ Put the pizza from the storage warehouse in a microwave

 ■ Carry the microwave pizza to your place (for example, a table)

You only have to put the first. That is, after the user issues the above two instructions, the system automatically executes the above operation as it is, for example, just waiting at the table. As a result, the user can do other things until the hot pizza comes to his place, for example, sitting at a table, and can use his time more efficiently.

 Further, at that time, it is preferable to provide a scheduling function for efficiently performing a plurality of processes. For example, when taking out multiple types of articles from a facility and moving them to different locations, only the process of taking out the multiple types of articles is performed at once, and scheduling is performed so that all processes are efficient. May be. For this purpose, it is not necessary to limit the number of arms 201 of the working robot 102 to one, and a multi-arm configuration may be employed so that a plurality of articles can be handled simultaneously at a time. The above-described embodiment and its modified examples can be realized using a computer program. A life support system program that allows a computer to execute the life support system according to the present invention includes a computer program that executes some or all of the operations of the above-described embodiment and its modifications. It is to be noted that, by appropriately combining arbitrary embodiments and modified examples among the various embodiments and modified examples, the effects of the respective embodiments and modified examples can be achieved.

 While the present invention has been fully described in connection with the preferred embodiments thereof with reference to the accompanying drawings, various changes and modifications will be apparent to those skilled in the art. It is to be understood that such changes and modifications are intended to be included therein without departing from the scope of the invention as set forth in the appended claims. Industrial potential

As described above, the present invention relates to a living support system for managing articles in a living environment such as a house or an office, for example, a living environment, and a control program therefor. Useful. In particular, various electronic devices such as a computer and a portable information terminal (such as a mobile phone) are used as an article operation device to instruct an operation on the article, and the article is moved by an article moving device such as a robot based on the instruction. Useful for systems and programs.

Claims

The scope of the claims 1. A display device (1 19) that displays the actual situation of the living environment,  An article operation device (1 18) to which an operation instruction for designating equipment (104) in the living environment is input on a display on the display device;  By specifying the equipment by the article operating device, based on information on the article (400) in the living environment and structural information on the equipment and the space in the living environment in the living environment, A control system (120) for controlling information of an article inside the equipment or an article hidden in the equipment to be displayed on the display device.  2. An article database (10 7) that stores information on the articles in the living environment;  An environment map information database (109) that stores structural information of the facility in the living environment and the space in the living environment;  The control means is connected to the display device, the article operation device, the article database, and the environment map information database, respectively, and when the equipment is designated by the article operation device, the article database and the environment map. An article inside the facility or an article concealed by the facility based on information of the article in the living environment with reference to the information database and the structural information of the facility and the space in the living environment in the living environment. The life support system according to claim 1, wherein the information is controlled to be displayed on the display device.  3. The control means, when one of a plurality of facilities in the living environment is designated by the article operating device, information on the articles in the living environment, the facilities in the living environment, and the living environment. Based on the structural information of the space, any one of the facility and the facility behind it can be freely selected, and information on the articles inside the selected facility or the articles hidden in the facility 2. The life support system according to claim 1, wherein the control is performed so that is displayed on the display device. 4. The control means, when one of a plurality of facilities in the living environment is designated by the article operating device, information on the articles in the living environment and the living environment Based on the structural information of the space in the facility and the living environment, the device and any one of the facilities located behind the facility can be freely selected, and the articles inside the selected facility or the 3. The life support system according to claim 2, wherein control is performed so that information on the article hidden in the facility is displayed on the display device. 5. A sensing unit (10 5) for detecting information on the article;  Database control means (106, 111) for updating information of the article based on the detection result of the sensing unit;  The life support system according to any one of claims 1 to 4, further comprising: 6. An article moving device (10 2) for moving articles;  When the article and the moving location of the article are input as an instruction operation by the article operating device, an article movement control unit (1 16) that moves the article to the moving location by the article moving apparatus;  The life support system according to any one of claims 1 to 4, comprising:  7. The article operation device displays a cursor for article operation on the display of the display device,  When the cursor is placed on the facility on the display of the display device, the control unit pops up information on an article inside the facility or an article concealed by the facility either alone or together with the structural information of the facility. The life support system according to any one of claims 1 to 4, wherein the system is displayed in an applet format.  8. The article operation device displays a cursor for article operation on the display of the display device,  At least one of the facilities in the living environment is provided with photographing means (105A) for photographing the inside of the facility or a region concealed by the facility,  When the cursor is placed on the facility on the display of the display device, the control means displays an image of an article taken by the photographing means and inside the facility or in a region hidden by the facility. The life support system according to any one of claims 1 to 4, wherein the control is performed in such a manner. 9. The article operation device displays a cursor for article operation on the display of the display device, At least one of the facilities in the living environment is provided with a transmission device that makes the inside of the facility or an area hidden by the facility visible.  The control means, when the cursor is placed on the facility on a display of the display device, causes the transmissive device to transparently display an article inside the facility or an article concealed by the facility. The life support system according to any one of to 4. 10. The equipment includes a storage body (10 4) for storing or removing articles.  5. The storage device according to claim 1, further comprising a control unit configured to cause the storage unit to execute the operation when the storage unit is designated by the article operation device. 6. Life support system.  1 1. The storage body has an openable and closable door,  10. The life support system according to claim 10, wherein the control means opens or closes a door of the storage body when the storage body is designated by the article operation device.  12. Based on at least one of the attribute of the equipment, the attribute of the article, the attribute of the user, and the operation content, select one of a plurality of display forms and display the article information on the display device. The life support system according to any one of claims 1 to 4, wherein control is performed so as to further include an in-facility display means (124).  1 3. Procedure for displaying the actual situation of the living environment on the display device (1 1 9)  By inputting an operation instruction for specifying the facility (104) in the living environment on the display on the display device and specifying the facility, the article in the living environment is designated. Based on the information of (400) and the structural information of the facilities in the living environment and the space of the living environment, information of the articles inside the facilities or the articles hidden in the facilities is displayed on the display device. A program for a life support system that causes a computer to execute the steps displayed on the screen. 14. Further, when one of a plurality of facilities in the living environment is designated, information on articles in the living environment and structural information on the facilities and the living environment in the living environment are displayed. And any one of the equipments behind the equipment and the equipment behind the equipment can be freely selected based on the information on the articles inside the selected equipment or the articles hidden in the equipment. It further includes a procedure for controlling the display. 14. The living support system program according to claim 13, wherein:  1 5. The actual situation of the living environment is displayed on the display device (1 19).  By specifying the facility by inputting an operation instruction for designating the facility (104) in the living environment on the display on the display device, the article (400) in the living environment is designated. Based on the information of the equipment and the structural information of the equipment in the living environment and the space of the living environment, so that the information of the article inside the equipment or the article hidden in the equipment is displayed on the display device. How to control the life support system. 16. Further, when one of the plurality of facilities in the living environment is designated, information on the articles in the living environment and the structural information on the facilities and the living environment in the living environment are displayed. And any one of the equipment existing behind the equipment can be freely selected based on the information, and information on an article inside the selected equipment or an article hidden in the equipment is displayed on the display device. 16. The control method for a life support system according to claim 15, further comprising controlling to display.