JP6794493B2 - Storage device - Google Patents

Description

Embodiments of the present invention relate to storage devices.

Conventionally, in a storage device such as a refrigerator, when a door (swing door) is opened or closed, an image pickup device such as a camera attached to the door images the inside of the storage device, and the image obtained by the imaging is presented to the user. It has been known.

Japanese Unexamined Patent Publication No. 2001-317858

However, in the prior art, there is a problem that an image including an area to be presented to the user cannot be obtained depending on, for example, the angle of the door at the time of opening and closing.

An object to be solved by the present invention is to provide a storage device capable of improving the quality of an image in the storage device presented to the user.

The storage device of the embodiment is attached to a main body portion having a storage chamber for storing an object, a hinged door rotatably provided in the main body portion to open and close an entrance formed on the front surface of the storage chamber, and the hinged door. The imaging unit includes an imaging unit that images the storage chamber, and the imaging unit starts imaging when the amount of change in the increasing direction of the angle formed by the front surface of the storage chamber and the hinged door exceeds a threshold value, and the angle increases. Imaging is stopped when the amount of change in direction falls below the threshold value.

Front view of the refrigerator of the embodiment. Top view of the refrigerator of the embodiment. Top view of the refrigerator of the embodiment. The figure which shows an example of the captured image corresponding to each camera position of embodiment. The figure which shows an example of the structure of the image processing apparatus of embodiment. The figure for demonstrating the angle of the hinged door of an embodiment. The figure for demonstrating the correction with respect to the captured image of an embodiment. The figure for demonstrating the correction with respect to the captured image of an embodiment. The figure which shows an example of the 1st composite image of an embodiment. The schematic diagram which shows the 1st composite image of embodiment and an example of a lack area. The conceptual diagram which shows the method of generating the 2nd composite image of an embodiment. The figure which shows an example of the structure of the image processing apparatus of embodiment. The schematic diagram for demonstrating the 1st reliability of an Embodiment.

Hereinafter, embodiments of a storage device, an image processing device, an image processing method, and a program according to the present invention will be described in detail with reference to the accompanying drawings. In the following description, a refrigerator will be described as an example of the storage device, but the present invention is not limited to this.

(First Embodiment)
FIG. 1 is a front view of the refrigerator 1 of the present embodiment. As shown in FIG. 1, the refrigerator 1 includes a hinged door 10 and a main body 20.

The "swing door" means that one end (one side edge of both sides of the door) is fixed by a hinge or the like, and the other end (door) is centered on the fixed part. Refers to a door (door) in which the other side edge of the two side edges of the door moves in an arc.

As shown in FIG. 1, the main body 20 has a storage chamber 21 for storing an object. FIG. 1 illustrates a state after the hinged door 10 is opened so that the entire entrance of the storage chamber 21 is completely exposed in the front view. Although not shown here, the hinged door 10 is provided with a grip portion for the user to open and close the hinged door 10. In the example of FIG. 1, the hinged door 10 is composed of one door, but is not limited to this, and may be, for example, a double hinged door composed of two doors, for example, a type that opens up and down. It may be a door. Further, in the example of FIG. 1, a door pocket 11 for storing a container such as a PET bottle is provided on the rear surface (the surface on the storage chamber 21 side) of the hinged door 10.

Further, in the example of FIG. 1, the hinged door 10 is a camera that changes the positional relationship with the storage chamber 21 according to the opening or closing operation of the hinged door 10 and images a plurality of the storage chambers 21 (continuously imaging). 30 is attached. In the present specification, "imaging" refers to converting an image of a subject (imaging object) imaged by an optical system such as a lens into an electric signal. The camera 30 may be any camera having a function of capturing a moving image, and may be composed of, for example, a visible light camera or an infrared camera using CCD or CMOS. In this example, the camera 30 can be considered to correspond to the "imaging unit" of the claim. In this example, the camera 30 is externally attached to the hinged door 10, but the present invention is not limited to this, and for example, the camera 30 may be built (embedded) in the hinged door 10.

In the present embodiment, the image pickup by the camera 30 is performed when the hinged door 10 is opened and closed. Further, it is preferable that the installation location of the camera 30, the optical axis direction of the camera 30 at the time of installation, the angle of view, and the like are determined according to the range of the storage chamber 21 presented to the user. For example, when presenting at least the entire entrance of the storage chamber 21, the optical axis direction and angle of the camera 30 can be determined so that at least the entire entrance of the storage chamber 21 is imaged when the hinged door 10 is opened and closed. preferable. Further, in the present embodiment, the case where the number of cameras 30 is one will be described as an example, but the present invention is not limited to this, and the range of the storage chamber 21 presented to the user by using, for example, a plurality of cameras 30 is limited. It may be in the form of imaging.

FIG. 2 is a plan view of the refrigerator 1 when viewed from above. FIG. 2 shows an example in which the camera 30 is installed at a position of about two-thirds of the total length of the upper edge of the door from the root (the part corresponding to the rotation axis of the door) at the upper part of the hinged door 10. ing. In the example of FIG. 2, the camera 30 is installed with the camera optical axis oriented in the direction of an angle θ with respect to the normal direction of the door surface (front surface or rear surface of the door).

FIG. 3 is a plan view of the refrigerator 1 from above, and when the hinged door 10 is opened until the entire entrance of the storage chamber 21 is completely exposed in the front view, the image taken by the camera 30 is taken. It shows how it is done. In the example of FIG. 3, when the opening operation is performed, the camera 30 takes an image while moving in the order of position P1 → position P2 → position P3. The camera 30 images the right side of the entrance of the storage chamber 21 at the position P1, images the vicinity of the center of the entrance of the storage chamber 21 at the position P2, and images the left side of the storage chamber 21 at the position P3, from the start of the opening operation. By the end, the entire entrance of the storage chamber 21 will be imaged.

FIG. 4A represents an image captured image corresponding to the position P1 (an image obtained by imaging with the camera 30 at the position P1). Further, FIG. 4B represents an captured image corresponding to the position P2. Further, FIG. 4C represents an captured image corresponding to the position P3.

FIG. 5 is a diagram showing an example of the configuration of the image processing device 100 mounted on the refrigerator 1. Here, the case where the image processing device 100 is mounted on the refrigerator 1 will be described as an example, but the present invention is not limited to this, and the image processing device 100 may be provided outside the refrigerator 1, for example.

As shown in FIG. 5, the image processing device 100 includes a camera control unit 110, an acquisition unit 120, a generation unit 125, and a display control unit 160.

The camera control unit 110 controls the start or end of imaging by the camera 30 based on the signal from the sensor 40. In this example, the sensor 40 is an angle sensor that detects the angle φ of the hinged door 10 at a predetermined cycle. As the angle sensor constituting the sensor 40, various known configurations can be used. In this example, as shown in FIG. 6, the angle φ represents the angle formed by the front surface of the main body 20 and the hinged door 10. If the angle φ is “0 degrees”, the hinged door 10 is in the closed state, and if the angle φ is “90 degrees”, the hinged door 10 is opened so that the entire entrance of the storage chamber 21 is exposed in front view. It can be considered as a state. Each time the sensor 40 of the present embodiment detects an angle φ, a signal indicating the detected angle φ (may be referred to as an “angle signal” in the following description) is acquired by the camera control unit 110 and described later. It is transmitted to the unit 120.

Returning to FIG. 5, the description of the camera control unit 110 will be continued. As described above, in the present embodiment, the imaging by the camera 30 is performed when the hinged door 10 is opened and closed. More specifically, it is as follows. For example, when the angle φ indicated by the angle signal acquired from the sensor 40 in the predetermined cycle changes from “0 degree” in the increasing direction, the camera control unit 110 determines that the opening operation has started, and the camera 30 determines that the opening operation has started. Controls to start imaging. After controlling to start imaging by the camera 30, the camera control unit 110 completes the opening operation when the amount of change in the increasing direction of the angle φ indicated by the angle signal from the sensor 40 becomes equal to or less than the threshold value. It is determined that the camera 30 controls to end the imaging. After that, when the amount of change in the decreasing direction of the angle φ indicated by the angle signal from the sensor 40 exceeds the threshold value, the camera control unit 110 determines that the closing operation has started, and controls to start imaging by the camera 30. .. After controlling to start imaging by the camera 30, the camera control unit 110 determines that the closing operation is completed when the angle φ indicated by the angle signal from the sensor 40 indicates “0 degree”. Control is performed to end the imaging by the camera 30.

Next, the acquisition unit 120 shown in FIG. 5 will be described. Each time the image is taken by the camera 30, the acquisition unit 120 acquires the image obtained by the image pickup and the image pickup time indicating the time when the image pickup is performed from the camera 30 to the detection unit 131 described later. Supply. Further, in this example, the acquisition unit 120 acquires the angle signal from the sensor 40 at the predetermined cycle and supplies it to the detection unit 131 described later.

For example, the camera 30 does not provide the above-mentioned camera control unit 110, and the camera 30 constantly performs imaging, and the acquisition unit 120 opens or closes the hinged door 10 based on the angle signal from the sensor 40. The captured image and the captured time may be acquired from the camera 30 only when it is determined that the image is performed.

Next, the generation unit 125 shown in FIG. 5 will be described. The generation unit 125 synthesizes a plurality of captured images obtained by imaging by the camera 30 when at least one of the opening operation and the closing operation of the hinged door 10 is performed. The specific contents will be described below. As shown in FIG. 5, the generation unit 125 includes a first generation unit 130, a storage unit 140, and a second generation unit 150.

When at least one of the opening operation and the closing operation of the hinged door 10 is performed, the first generation unit 130 has a plurality of so that the same subject included in the plurality of captured images obtained by the image captured by the camera 30 overlaps. A first composite image (panoramic image) is generated by synthesizing the captured images. Hereinafter, a specific description will be given. In the present embodiment, the first generation unit 130 includes a detection unit 131, a correction unit 132, and a synthesis unit 133.

The detection unit 131 detects the relative position of the camera 30 with respect to the storage chamber 21. In this example, the detection unit 131 can be considered to have a function corresponding to the "position detection unit" of the claim. More specifically, it is as follows. In this example, the detection unit 131 provides installation information such as a predetermined mounting position of the camera 30 (in this example, the position of the camera 30 mounted on the hinged door 10) and a direction of the camera 30 or a memory (not shown). It may be an external device). Further, for example, the detection unit 131 can also acquire environmental information indicating the imaging environment such as the brightness at the time of imaging and the shutter speed from the camera 30. Further, for example, the detection unit 131 can acquire the specifications of the camera 30 such as the resolution and the angle of view of the camera 30 from the camera 30 or a memory (which may be an external device) (not shown).

For example, the detection unit 131 defines a coordinate system with the position of the base of the hinged door 10 as the origin, the angle φ indicated by the angle signal supplied from the acquisition unit 120 described above, and the mounting position of the camera 30 indicated by the installation information described above. Therefore, the coordinate value of the camera 30 in the coordinate system can be calculated, and the calculated coordinate value can be detected as the position of the camera 30 (the position of the camera 30 relative to the storage chamber 21). In the present embodiment, the detection unit 131 determines the position of the camera 30 at the time of imaging (more specifically) based on the information (captured image, imaging time and angle signal) supplied from the acquisition unit 120, the above-mentioned installation information, and the like. Detects a plurality of imaging times and a plurality of camera positions corresponding to one-to-one). Then, the detection unit 131 supplies detection information associated with the captured image obtained by imaging at the plurality of detected positions (position of the camera 30 at the time of imaging) to the correction unit 132 described later. To do.

Further, for example, the detection unit 131 detects (determines) the start and end of the opening operation of the hinged door 10 or the start and end of the closing operation based on the angle signal supplied from the acquisition unit 120 at the predetermined cycle. You can also do it. For example, when the opening operation is performed (every time a series of operations from the start to the end of the opening operation is completed), the detection unit 131 transfers the above-mentioned detection information corresponding to the opening operation to the correction unit 132 described later. It can also be supplied. Further, for example, when the closing operation is performed (every time a series of operations from the start to the end of the closing operation is completed), the detection unit 131 applies the above-mentioned detection information corresponding to the closing operation to the correction unit described later. It can also be supplied to 132.

In addition, for example, the detection unit 131 may be in a form that also has the function of the acquisition unit 120 described above. That is, the acquisition unit 120 may not be provided.

Further, the method of detecting the position of the camera 30 is not limited to the above, and various methods can be considered. For example, a gyro, GPS, or the like can be attached to the camera 30, and the position of the camera 30 can be obtained based on the information obtained by the gyro, GPS, or the like. Further, for example, a marker or the like can be arranged in the imaging range of the camera 30, and the position of the camera 30 can be obtained from the position and size of the marker in the captured image. The position of the camera 30 may be obtained by a method other than the method illustrated above. In short, the detection unit 131 may have a function of detecting the position of the camera 30 (the position of the camera 30 when imaging is performed).

The correction unit 132 is a reference indicating a reference position among the positions of the camera 30 that change with time (change with the passage of time) when at least one of the opening operation and the closing operation of the hinged door 10 is performed. For the captured image corresponding to a position other than the position, correction is performed to align the scale of the subject included in the captured image corresponding to each position. In the present embodiment, the correction unit 132 refers to the detection information supplied from the detection unit 131, specifies a position other than the reference position among the plurality of positions included in the detection information, and corresponds to the specified position. Correct the attached captured image. Hereinafter, a specific description will be given.

As an example, the opening operation shown in FIG. 3 is assumed. In this example, in the series of opening operations, the position of the camera 30 changes with time from P1 to P2 to P3. Therefore, in the detection information corresponding to the opening operation, the images taken at the positions P1 and P1 are used. The obtained captured image (hereinafter referred to as "captured image Ip1") is associated with the captured image obtained by imaging at the position P2 and the position P2 (hereinafter referred to as "captured image Ip2"). Is associated with, and the position P3 and the captured image obtained by imaging at the position P3 (hereinafter, referred to as "captured image Ip3") are associated with each other. In the following description, when the captured image Ip1, the captured image Ip2, and the captured image Ip3 are not distinguished from each other, they may be simply referred to as "captured image Ip".

Here, when the same object inside the storage chamber 21 is imaged at positions P1, P2, and position P3, the distance to the object differs depending on the position, so that the object is included (reflected) in each captured image Ip. The size of the object is also different. Therefore, it is difficult to simply connect the captured image Ip1, the captured image Ip2, and the captured image Ip3. Therefore, in the present embodiment, the reference viewpoint position (reference position) is set, and the captured image corresponding to the position other than the reference position is corrected. The reference position is not limited as long as the storage chamber 21 is an observable position, unless it is an unobservable place such as the back side of the refrigerator 1. However, it is not preferable that the first composite image (panoramic image) described later is significantly different from the landscape to be presented to the user. Therefore, in the present embodiment, the optical axis of the camera 30 is orthogonal to the entrance surface of the storage chamber 21. The position of the camera 30 (P2 shown in FIG. 3) is set as a reference position.

For example, as a correction of the captured image corresponding to the position P1, as shown in FIG. 7, the distance L from the entrance of the storage chamber 21 is equal to the position P2, and the optical axis of the camera 30 at the position P1 and the storage chamber 21 It is also possible to correct as if the image was taken at the position P1'where the optical axes of the camera 30 intersect at the intersection c1 with the entrance. As shown in FIG. 8, the correction can use a simple geometric transformation such as an affine transformation. The correction is not limited to the above method, and any correction may be made as long as the scales of the objects appearing in the captured image corresponding to each position are the same. As described above, the correction unit 132 corrects the captured image corresponding to each of the position P1 and the position P3 other than the reference position P2. Then, the correction unit 132 includes a group of corrected captured images (in the following description, the corrected captured image may be referred to as a “corrected image”) and a reference image (this example) showing the captured image corresponding to the reference position. Then, the captured image Ip2) is supplied to the synthesis unit 133 described later.

The compositing unit 133 shown in FIG. 5 synthesizes the reference image and the corrected image supplied from the compensating unit 132 to generate a first composite image. In this example, the compositing unit 133 compares each of the reference image and the plurality of corrected images included in the corrected image group (in this example, the corrected image corresponding to the position P1 and the corrected image corresponding to the position P3). , The same range, that is, the area where the same object is imaged is overlapped. As a result, as shown in FIG. 9, the first composite image in which the image area is extended to the outside of the range of the reference image is generated. At this time, if the images closest to the reference position are combined in order, it is easy to generate a higher quality first composite image. In the present embodiment, each image is corrected and then combined, but a method of correcting while combining is also conceivable. For example, it may be a method of deforming each image (captured image corresponding to a position other than the reference position) so as to optimally overlap with the reference image. Specifically, while trying various deformations, the deformation that optimally overlaps with the reference image is searched, the deformation that optimally overlaps is determined, and then the composition is performed. The method in the present embodiment described above can be considered to be a method of determining this optimum deformation by utilizing the position of the camera 30.

The first generation unit 130 described above generates a first composite image when the hinged door 10 is opened or closed (every time the hinged door 10 is opened or closed) and is stored. While storing in 140, the first composite image and the detection information used for generating the first composite image are supplied to the second generation unit 150 described later. The storage unit 140 stores the first composite image generated by the first generation unit 130 in chronological order.

Next, the second generation unit 150 shown in FIG. 5 will be described. When the first composite image generated by the first generation unit 130 described above does not include a part of the target area indicating the area to be presented to the user in the storage chamber 21, the second generation unit 150 is in the past. A part of the target area is complemented by using the first composite image, and a second composite image to be presented to the user is generated. Hereinafter, a specific description will be given. In this example, the target area is the entire entrance area of the storage chamber 21.

For example, when the hinged door 10 cannot be opened until the entire entrance of the storage chamber 21 is completely exposed and the entire entrance of the storage chamber 21 cannot be imaged, this opening operation (hereinafter, referred to as "non-fully opening operation"). The first composite image (the first composite image generated when the non-fully open operation is performed) corresponding to (may be) does not include (not show) a part of the entire entrance of the storage chamber 21. It becomes. Similarly, for example, when a closing operation (hereinafter, may be referred to as "insufficient closing operation") is performed starting from the position of the hinged door 10 which is not opened until the entire entrance of the storage chamber 21 is completely exposed. The first composite image corresponding to this insufficient closing operation (the first composite image generated when the insufficient closing operation is performed) is an image that does not include a part of the entire entrance of the storage chamber 21.

Here, the hinged door 10 is opened until the entire entrance of the storage chamber 21 is completely exposed, based on the position of the camera 30 (the position of the camera 30 at the time of imaging) that changes with time in the above-mentioned non-fully open operation or insufficiently closed operation. Of the image area of the first composite image generated when the image is generated (in the following description, it may be referred to as a "panoramic area"), it is out of the range of the first composite image corresponding to the non-fully open operation or the insufficiently closed operation. Area (hereinafter, may be referred to as "insufficient area") can be obtained.

In the present embodiment, when the second generation unit 150 acquires the first composite image and the detection information from the above-mentioned first generation unit 130 (every time it is acquired), the position of the camera 30 included in the detection information. Therefore, it is determined whether or not the first composite image has a panoramic view area. From a different point of view, the second generation unit 150 does not have an open operation corresponding to the first composite image acquired from the first generation unit 130 from the position of the camera 30 included in the detection information acquired from the first generation unit 130. It can also be considered that it is determined whether or not the fully open operation or the closing operation corresponding to the first composite image is the insufficient closing operation.

The second generation unit 150 determines that the first composite image (may be referred to as the "latest first composite image" in the following description) acquired from the first generation unit 130 does not have a panoramic view region. If so, the above-mentioned shortage area is obtained from the position of the camera 30 included in the detection information (detection information acquired from the first generation unit 130) used for generating the latest first composite image. FIG. 10 is a schematic diagram showing an example of the latest first composite image and the missing region in this case.

Then, the second generation unit 150 has a past having a panoramic view region from the past first composite image (first composite image generated before the latest first composite image) stored in the storage unit 140. (Preferably, the first composite image in the past that has a panoramic area and corresponds to the time closest to the current time) is selected, and among the selected first composite images in the past, the above-mentioned By complementing the region corresponding to the insufficient region of the above with respect to the latest first composite image, a second composite image in which the image region is expanded to the outside of the range of the latest first composite image is generated. FIG. 11 is a conceptual diagram showing a method of generating a second composite image in this case. The second composite image generated in this way becomes an image presented to the user.

Even if the first composite image acquired from the first generation unit 130 has a panoramic view region, for example, an object that is taken in and out of the user's hand or the storage chamber 21 at the time of imaging invades the imaging range of the camera 30. It is assumed that a part of the entire entrance of the storage room 21 is hidden. In this case, for example, the second generation unit 150 has a plurality of positions (positions of the camera 30 at the time of imaging) included in the detection information acquired from the first generation unit 130, and a plurality of positions corresponding to one-to-one. The distance to the subject is estimated by a method such as motion stereo using the captured image, and the first complement target area indicating the area in which the object existing in front of the entrance of the storage chamber 21 is imaged in the panoramic area. Can be sought. Then, the second generation unit 150 selects the past first composite image in which the entire entrance of the storage chamber 21 is captured from the past first composite image stored in the storage unit 140, and the selected past A second composite image can also be generated by complementing the region corresponding to the above-mentioned first complement target region in the first composite image of the above with respect to the latest first composite image.

Further, even if the first composite image acquired from the first generation unit 130 has a panoramic view area, a part of the entire entrance of the storage chamber 21 is not captured because the moving speed of the camera 30 at the time of opening and closing is high. It is also assumed that this is the case. In this case, for example, the second generation unit 150 did not image a part of the entire entrance of the storage chamber 21 in the panoramic area based on the plurality of positions included in the detection information acquired from the first generation unit 130. It is possible to obtain the second complement target area indicating the area (which can be considered as the area where frames are dropped). Then, the second generation unit 150 selects the past first composite image in which the entire entrance of the storage chamber 21 is captured from the past first composite image stored in the storage unit 140, and the selected past The second composite image can also be generated by complementing the region corresponding to the above-mentioned second complement target region in the first composite image of the above with respect to the latest first composite image.

In short, the second generation unit 150 is a target area (in this example, the entrance of the storage room 20) in which the first composite image generated by the first generation unit 130 described above indicates an area of the storage room 21 to be presented to the user. When a part of (whole) is not included, a second composite image for presenting to the user may be generated by complementing a part of the target area by using the first composite image in the past.

Next, the display control unit 160 shown in FIG. 5 will be described. The display control unit 160 controls the display of the second composite image generated by the second generation unit 150 on the display unit (which may be composed of, for example, a liquid crystal display) 50 that displays the image. In this example, the display control unit 160 controls the display unit 50 to display the generated second composite image each time the second composite image is generated by the second generation unit 150, and the display unit 50 displays the generated second composite image. The displayed second composite image is updated with the latest second composite image. Further, in this example, the display unit 50 is provided on the front surface of the hinged door 10, but the present invention is not limited to this, and the installation location of the display unit 50 is arbitrary. For example, the display unit 50 may be provided outside the refrigerator 1. For example, the display unit 50 may be mounted on an external device (for example, a TV, a PC, a mobile phone terminal, etc.) that can be connected (communicated) with the image processing device 100.

The timing of displaying the second composite image is arbitrary, and it may be displayed at all times or in response to a request from the user. For example, the user can request the display of the second composite image via button operation, voice, mail, remote control, or the like. Further, for example, the second composite image may be displayed at regular intervals.

The image processing device 100 described above has a hardware configuration including a CPU (Central Processing Unit), a ROM, a RAM, a communication I / F device, and the like. The CPU is ROM for the functions of the above-mentioned units (camera control unit 110, acquisition unit 120, first generation unit 130 (detection unit 131, correction unit 132, synthesis unit 133), second generation unit 150, display control unit 160). It is realized by expanding and executing the program stored in the RAM on the RAM. Further, the present invention is not limited to this, and at least a part of the functions of the above-mentioned parts can be realized by a dedicated hardware circuit (for example, a semiconductor integrated circuit or the like). Further, for example, the above-mentioned storage unit 140 can be realized by a ROM, an external storage device (for example, an HDD), or the like.

Further, the program executed by the image processing device 100 described above may be provided by storing it on a computer connected to a network such as the Internet and downloading it via the network. Further, the program executed by the above-mentioned image processing device 100 may be provided or distributed via a network such as the Internet. Further, the program executed by the image processing apparatus 100 described above may be provided by incorporating it into a non-volatile recording medium such as a ROM in advance.

In the prior art, there is a problem that an image including an area to be presented to the user cannot be obtained even when a user's hand or the like invades the imaging range at the same time as opening and closing the door or when the moving speed of the door is high. Occur. That is, the conventional technique has a problem that the quality of the image in the storage device presented to the user is low. On the other hand, as described above, in the present embodiment, when the hinged door 10 is opened or closed, a plurality of captured images obtained by imaging by the camera 30 are combined to form a target. The entire area (in the present embodiment, the entire entrance of the storage chamber 21) can be represented. More specifically, in the present embodiment, when the hinged door 10 is opened or closed, a plurality of imaging images are performed so that the same subject included in the plurality of captured images obtained by imaging by the camera 30 overlaps. A first composite image in which the images are combined is generated. Then, when the generated first composite image does not include a part of the target area indicating the area to be presented to the user in the storage chamber 21, a part of the target area is used by using the past first composite image. To complement and generate a second composite image to present to the user. Therefore, for example, even if the user does not open the hinged door 10 until the entire entrance of the storage chamber 21 is completely exposed, the second composite image in which the entire entrance of the storage chamber 21 is captured for the user. Can be presented. That is, according to the present embodiment, it is possible to achieve an advantageous effect that the quality of the image in the refrigerator 1 presented to the user can be improved.

(Second Embodiment)
Next, the second embodiment will be described. The description of the parts common to the first embodiment described above will be omitted as appropriate.

FIG. 12 is a diagram showing an example of the configuration of the image processing device 200 of the second embodiment. As shown in FIG. 12, the image processing device 200 further includes a setting unit 170.

In the present embodiment, the detection unit 131 has a function of detecting the relative speed of the camera 30 with respect to the storage chamber 21 in addition to the function described in the first embodiment described above. In this example, the detection unit 131 can be considered to have a function corresponding to the "speed detection unit" of the claim. In the following description, the relative speed of the camera with respect to the storage chamber 21 may be referred to as "moving speed of camera 30" or "moving speed". In this example, the detection unit 131 determines the moving speed corresponding to the position of the camera 30 at the time of imaging. For example, the detection unit 131 utilizes the positions of the camera 30 (a plurality of camera positions corresponding to a plurality of imaging times and one-to-one) detected in the same manner as in the first embodiment described above, and the detection unit 131 of the camera 30 at the time of imaging. It is also possible to obtain the moving speed corresponding to the position. Further, the present invention is not limited to this, and for example, the camera 30 may be equipped with a gyro, an accelerometer, or the like, and the moving speed may be obtained by using the signals output from these instruments, or may be obtained by other methods. Absent.

For example, the detection unit 131 generates detection information corresponding to the opening operation when the hinged door 10 is opened, and generates detection information corresponding to the closing operation when the hinged door 10 is closed. can do. The detection information in this example is composed of information in which the captured image and the moving speed are associated with each of the plurality of detected positions (positions of the camera 30 at the time of imaging).

In the present embodiment, the first generation unit 130 generates a first composite image when the hinged door 10 is opened or closed, and the detection information used for generating the first composite image is used for the first composite image. 1 Supply to the setting unit 170 together with the composite image. When the setting unit 170 acquires the first composite image and the detection information from the first generation unit 130 (every time it is acquired), the setting unit 170 obtains the first composite image and the detection information based on the acquired first composite image and the detection information. For each of the plurality of regions, the first reliability that indicates a lower value as the moving speed when the captured image corresponding to the region is obtained is set is set. Hereinafter, a specific description will be given.

In the following description, the first reliability is a value within the range of 0 or more and 1 or less, where "1" is the most reliable and "0" is the least reliable. Further, the moving speed of the camera 30 at the position p of the camera 30 at the time of imaging is referred to as Vp. Here, the captured image when the camera 30 is stationary (Vp = 0) becomes an image without blur if the object (subject) is in focus (focus). Bokeh is an aspect of image deterioration that occurs in the process of obtaining an image by imaging, and refers to an event that occurs when a group of light rays to be imaged (concentrated) spreads on an image plane at one point. On the other hand, as the moving speed Vp of the camera 30 increases, motion blurring can be seen in the captured image, so that the image quality deteriorates and the image becomes less reliable. That is, as the moving speed Vp of the camera 30 increases, the first reliability decreases (see FIG. 13). The first reliability can be expressed by, for example, the following equation 1.

In Equation 1 above, Sp represents the first reliability and Vth represents the threshold speed. This corresponds to determining that the captured image at that time is a highly reliable image when the moving speed Vp is a constant speed Vth or less. Vth may be set according to the frame rate, shutter speed, or the like of the camera 30, and when the frame rate is high or the shutter speed is fast, Vth can be set to a large value. The method for calculating the first reliability Sp is not limited to the above method, and it can be designed so that the first reliability Sp decreases as the moving speed Vp increases. The first reliability Sp can be expressed by, for example, the following equation 2.

In the above, the blur of the image is indirectly evaluated from the moving speed of the camera 30, but the present invention is not limited to this. For example, the amount of blur in the image is directly measured, and the reliability indicating a lower value is calculated as the amount of blur is larger. It may be in the form of From a different point of view, this reliability can be considered to be synonymous with the first reliability, which shows a lower value as the moving speed increases (the larger the amount of blur).

In the present embodiment, when the setting unit 170 acquires the first composite image and the detection information from the first generation unit 130, the setting unit 170 refers to the acquired detection information for each of the plurality of captured images included in the detection information. First, the first reliability Sp corresponding to the moving speed Vp associated with the captured image is calculated and assigned to the captured image. As a result, the first reliability Sp corresponding to the moving speed Vp associated with the captured image is assigned to each of the plurality of captured images that are the source of the first composite image acquired from the first generation unit 130. Then, the setting unit 170 acquires the detection information used for generating the first composite image (acquired from the first generation unit 130 together with the first composite image) among the first composite images acquired from the first generation unit 130. A plurality of regions corresponding to one-to-one with a plurality of captured images included in the detection information) are specified, and a first reliability Sp assigned to the captured image corresponding to the region is set for each of the specified regions. To do.

As described above, for each of a plurality of regions in the first composite image acquired from the first generation unit 130 (a plurality of regions corresponding to one-to-one with the plurality of captured images that are the source of the first composite image). The higher the moving speed of the camera 30 when the captured image corresponding to the region is obtained, the lower the first reliability Sp is set.

In the first composite image acquired from the first generation unit 130, among the plurality of regions corresponding to one-to-one with the plurality of captured images that are the source of the first composite image, the portions that overlap each other are, for example, It is also possible to set the first reliability Sp1 of the portion by multiplying the first reliability Sp1 of each of the plurality of regions overlapping each other (for example, a product in which each first reliability Sp1 is weighted). For example, the first reliability Sp1 of the portion can be set by calculating the average value of the first reliability Sp1 of each of the plurality of regions overlapping each other.

Further, for example, the concept of the distance from the camera 30 to the image pickup target can be added, and the first reliability Sp1 can be assigned to each pixel in the captured image. The distance to the image pickup target may be directly measured using a stereo camera or a range finder, or may be estimated from the captured image by a method such as motion stereo. Further, when the distance to the imaging target is not directly used, there is a method of using the distance from the camera 30 to the entrance of the storage chamber 21.

Here, La is the distance (distance to the entrance of the storage chamber 21) of any pixel a in the captured image Ip corresponding to the position p of the camera 30 at the time of imaging in the real space. The larger the La, the smaller the movement of the subject in the captured image and the less likely it is that blurring will occur. Therefore, it is designed so that the larger the La area, the higher the reliability. On the contrary, the smaller the La, the more likely it is that blurring will occur. Therefore, it is preferable to design so that the smaller the La, the lower the reliability. Assuming that the reliability using the distance corresponding to the pixel a is SpL (a), this reliability SpL (a) can be expressed by, for example, the following equation 3.

In the above equation 3, Lth is an arbitrary positive constant, and the distance beyond this distance is such that blurring is unlikely to occur. The reliability SpL (a) may be calculated by a method other than the above. From the reliability SpL (a) and the reliability calculated from the moving speed corresponding to the captured image including the pixel a (for example, the reliability calculated by the above formula 1 or the above formula 2) SpV, the pixel a The corresponding first reliability Sp1 (a) can also be calculated. The first reliability Sp1 (a) can be expressed by, for example, the following equation 4.

In addition to the above equation 4, for example, a method of obtaining and normalizing the sum of SpL (a) and SpV can be considered, and methods other than these may be used. Then, using the first reliability Sp1 (a) assigned to each pixel included in each of the plurality of captured images that are the source of the first composite image, the first reliability for each pixel in the first composite image is used. The degree Sp1 can also be obtained.

When the setting unit 170 sets the first reliability for each of the plurality of regions in the first composite image acquired from the first generation unit 130 (every time), the setting unit 170 sets the first reliability. The composite image is stored in the storage unit 140, and the first composite image for which the first reliability is set and the detection information used for generating the first composite image are supplied to the second generation unit 150. The storage unit 140 stores the first composite image in which the first reliability is set in chronological order.

Similar to the first embodiment described above, the second generation unit 150 uses the past first composite image when the first composite image acquired from the setting unit 170 does not include the entire entrance of the storage chamber 21. The second composite image is generated by complementing the above-mentioned shortage region. Further, the second generation unit 150 of the present embodiment has the first reliability set for each of the first composite image (latest first composite image) acquired from the setting unit 170 and the past first composite image. A second composite image is generated based on. More specifically, in the second generation unit 150, the first reliability of the plurality of regions in the latest first composite image is higher than the first reliability of the corresponding region in the past first composite image. The region showing a low value is replaced with the corresponding region in the past first composite image. Thereby, the advantageous effect that the image quality of the second composite image can be further improved can be achieved.

(Third Embodiment)
Next, the third embodiment will be described. The description of the parts common to each of the above-described embodiments will be omitted as appropriate. Since the basic configuration of the image processing apparatus according to the third embodiment is the same as that of the image processing apparatus 200 according to the second embodiment (similar to the configuration of FIG. 12), detailed illustration is omitted.

In the present embodiment, the setting unit 170 sets a higher value for each of a plurality of regions in the first composite image as the time (imaging time) when the captured image corresponding to the region is obtained is closer to the current time. The second reliability shown is further set, and the second generation unit 150 further sets the first reliability and the second reliability set for each of the first composite image and the first composite image in the past. 2 It differs from the above-described second embodiment in that a composite image is generated. Hereinafter, the specific contents of the third embodiment will be described.

Similar to each of the above-described embodiments, the detection unit 131 generates detection information corresponding to the opening operation when the hinged door 10 is opened, and closes the hinged door 10 when the hinged door 10 is closed. Generate detection information corresponding to the operation. The detection information in this example is composed of information in which the captured image, the moving speed, and the imaging time are associated with each of the plurality of detected positions (positions of the camera 30 at the time of imaging).

Further, as in the second embodiment described above, the first generation unit 130 generates a first composite image when the hinged door 10 is opened or closed, and is used for generating the first composite image. The detected detection information is supplied to the setting unit 170 together with the first composite image.

When the setting unit 170 acquires the first composite image and the detection information from the first generation unit 130, each of a plurality of regions in the first composite image is based on the acquired first composite image and the detection information. The first reliability is set to indicate a lower value as the moving speed increases when the captured image corresponding to the region is obtained, and the imaging time when the captured image corresponding to the region is obtained is currently set. The second reliability, which indicates a higher value as the time approaches, is set, and the third reliability defined according to the set first reliability and second reliability is set. The method of setting the first reliability is the same as that of the second embodiment described above.

The second reliability is a value within the range of 0 or more and 1 or less, and "1" indicates that the reliability is the highest (indicating that the imaging time is the closest to the current time). "0" indicates the lowest reliability. In this example, when the difference between the imaging time and the current time is equal to or greater than the threshold value, the second reliability is set to "0".

In the present embodiment, when the setting unit 170 acquires the first composite image and the detection information from the first generation unit 130, the setting unit 170 refers to the acquired detection information for each of the plurality of captured images included in the detection information. The second reliability, which indicates a lower value as the difference between the imaging time associated with the captured image and the current time increases, is calculated and assigned to the captured image. As a result, a second reliability corresponding to the imaging time associated with the captured image is assigned to each of the plurality of captured images that are the source of the first composite image acquired from the first generation unit 130. Then, the setting unit 170 has a one-to-one correspondence with a plurality of captured images included in the detection information used for generating the first composite image among the first composite images acquired from the first generation unit 130. The region is specified, and the second reliability assigned to the captured image corresponding to the region is set for each of the specified regions. As described above, for each of a plurality of regions in the first composite image acquired from the first generation unit 130 (a plurality of regions corresponding to one-to-one with the plurality of captured images that are the source of the first composite image). The closer the imaging time when the captured image corresponding to the region is obtained to the current time, the lower the second reliability is set.

Then, the setting unit 170 is used for each of a plurality of regions (a plurality of regions corresponding to one-to-one with the plurality of captured images that are the source of the first composite image) in the first composite image acquired from the first generation unit 130. In addition, a third reliability defined according to the first reliability and the second reliability set in the area is calculated, and the calculated third reliability is set in the area. In the present embodiment, the setting unit 170 obtains the third reliability by multiplying the first reliability and the second reliability, but the present invention is not limited to this, and for example, the first reliability and the second reliability are each determined. The third reliability may be obtained by weighted multiplication, for example, the weighted average value of the first reliability and the second reliability may be calculated to obtain the third reliability, for example, the first. The sum of the 1st reliability and the 2nd reliability may be normalized to a value in the range of 0 to 1 to obtain the 3rd reliability.

When the setting unit 170 sets the third reliability for each of the plurality of regions in the first composite image acquired from the first generation unit 130, the setting unit 170 and the first composite image for which the third reliability is set and the corresponding The detection information used for generating the first composite image is stored in the storage unit 140 in association with the detection information, and is supplied to the second generation unit 150.

In this example, the difference between the imaging time and the current time included in the detection information associated with the past first composite image stored in the storage unit 140 increases with the passage of time. Each time the setting unit 170 stores the first composite image for which the third reliability is set in the storage unit 140, the setting unit 170 is the difference between the imaging time and the current time included in the detection information associated with the past first composite image. It is also possible to update the third reliability set for each region in the past first composite image according to the above. If the difference between the imaging time and the current time is equal to or greater than the threshold value, the third reliability can be set to "0".

Similar to the first embodiment described above, the second generation unit 150 uses the past first composite image when the first composite image acquired from the setting unit 170 does not include the entire entrance of the storage chamber 21. The second composite image is generated by complementing the above-mentioned shortage region. Further, the second generation unit 150 of the present embodiment has the third reliability set for each of the first composite image (latest first composite image) acquired from the setting unit 170 and the past first composite image. A second composite image is generated based on. More specifically, in the second generation unit 150, the third reliability of the plurality of regions in the latest first composite image is higher than the third reliability of the corresponding region in the past first composite image. The region showing a low value is replaced with the corresponding region in the past first composite image.

That is, in the present embodiment, in addition to complementing the insufficient region, the image quality of the second composite image is improved by generating the second composite image by comprehensively considering the deterioration of the image quality and the imaging time. The advantageous effect of being able to be achieved can be achieved.

Although the embodiments of the present invention have been described above, the above-described embodiments are presented as examples, and are not intended to limit the scope of the invention. The above novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The above novel embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

(Modification example)
A modified example will be described below.

(1) Modification 1
In each of the above-described embodiments, the first generation unit 130 generates the first composite image when the hinged door 10 is opened or closed, but the first composite image is not limited to this, for example, the first generation unit 130. May be in a form in which the first composite image is generated when the opening operation is performed and the first composite image is not generated even when the closing operation is performed. Further, for example, the first generation unit 130 may be in a form in which the first composite image is generated when the closing operation is performed and the first composite image is not generated even when the opening operation is performed.

Further, for example, the first generation unit 130 is used when both the opening operation and the closing operation are performed (a series of operations from the start of the opening operation to the end of the closing operation (referred to as "opening / closing operation" in the following description)). (Every time) is performed, the first composite image may be generated. In short, the first generation unit 130 may be in a form of generating a first composite image when at least one of the opening operation and the closing operation of the hinged door 10 is performed.

Here, for example, it is assumed that the first composite image is generated when the above-mentioned opening / closing operation is performed. In this embodiment, for example, the position of the camera 30 during imaging in the open operation (referred to as "first position" for convenience of explanation) and the position of the camera 30 during imaging in the closed operation (for convenience of explanation, "second position"). When they are close to each other (referred to as "position") (including "match"), the same region (referred to as "overlapping region" for convenience of explanation) is imaged at each of the first position and the second position. In such a case, it is necessary to select the image used for generating the first composite image in units of frames or regions. As the selection method, a region in which the overlapping region is captured in the captured image corresponding to the first position and a region in which the overlapping region is captured in the captured image corresponding to the second position are selected. It is preferable to compare and select the one with higher reliability. For example, as in the third embodiment described above, the moving speed (blurring amount) at the time of imaging and the reliability (third reliability) according to the imaging time are set to the captured image corresponding to the first position and the second. It is also possible to set each of the captured images corresponding to the position and select the region having the higher reliability.

In this example, the value of the information is particularly higher in the captured image after the imaging time even in the same imaging range. Specifically, the general usage of the refrigerator 1 is a flow of (1) opening the hinged door 10, (2) taking in and out the stored items, and (3) closing the hinged door 10. Since the state of the storage room 21 changes before and after 2), the state at the time of (3) is the information that the user really wants to know. That is, it is preferable to design the captured image after the imaging time so that the value is higher and the reliability is high even if the image quality is deteriorated such as the image is slightly blurred. Further, the estimation of the same imaging range can be estimated by calculating the imaging range from the position of the camera 30, a method of comparing captured images and calculating a similar region, or a method of using a marker or the like. But it can be estimated.

(2) Modification example 2
In each of the above-described embodiments, the camera 30 is provided in the hinged door 10, but the present invention is not limited to this, and for example, the camera 30 may be provided in the main body 20. In this embodiment, the camera 30 changes the positional relationship with the door pocket 11 (see FIG. 1) according to the opening or closing operation of the hinged door 10, and continuously images the door pocket 11. That is, in this form, it can be considered that the door pocket 11 corresponds to the "storage room" of the claim. Also in this form, the processing related to the generation of the first composite image and the second composite image can be considered in the same manner as in each of the above-described embodiments. This is because the camera 30 moves to take an image of the storage chamber 21 and the camera 30 takes an image of the hinged door 10 that moves stationary, from the viewpoint of relative movement. If the camera 30 (stationary camera 30) attached to the main body 20 is observed from above the moving hinged door 10, the camera 30 is observed to be moving, and the observer on the hinged door 10 observes it. It is not possible to distinguish which is moving. By the way, the position of the camera 30 (the position of the camera 30 at the time of imaging) included in the detection information in this case is a position on the coordinate system that changes with the movement of the hinged door 10. By doing so, it becomes possible to think in the same manner as the position of the camera 30 in each of the above-described embodiments. The method of estimating the position of the camera 30 is basically the same as that of each of the above-described embodiments. However, although it is impossible to attach an instrument such as a gyro or GPS to the camera 30, it is possible to estimate the position of the camera 30 by attaching an instrument such as a gyro or GPS to the hinged door 10.

In short, the camera 30 may be in a form in which the positional relationship with the storage chamber (storage chamber 21 or door pocket 11) changes according to the opening or closing operation of the hinged door 10 and a plurality of storage chambers are imaged. Further, a plurality of first generation units 130 are used so that the same subject included in a plurality of captured images obtained by imaging by the camera 30 overlaps when at least one of the opening operation and the closing operation of the hinged door is performed. Any form may be used as long as it is in a form of generating a first composite image obtained by synthesizing the captured images of. Further, when the first composite image does not include a part of the target area indicating the area to be presented to the user in the storage room, the second generation unit 150 uses the past first composite image to indicate the target area. Any form may be used as long as it complements a part of the above and generates a second composite image to be presented to the user.

(3) Modification 3
In each of the above-described embodiments, the refrigerator 1 has been described as an example of the "storage device" of the claim, but the present invention is not limited thereto. For example, a cupboard having a hinged door, a cabinet having a hinged door (a shelf for organizing documents), a refrigerating machine having a hinged door, and the like may be used.

It should be noted that each of the above embodiments and modifications can be arbitrarily combined.

1 Refrigerator 10 Swing door 11 Door pocket 20 Main body 21 Storage room 30 Camera 40 Sensor 50 Display 100 Image processing device 110 Camera control 120 Acquisition unit 130 First generation unit 140 Storage unit 150 Second generation unit 160 Display control unit 170 Setting Department

Claims (6)

A main body with a storage room for storing objects ,
A hinged door that is rotatably provided on the main body and opens and closes an entrance formed on the front surface of the storage chamber, and an imaging unit that is attached to the hinged door and images the storage chamber are provided.
The imaging unit
Imaging is started when the amount of change in the increasing direction of the angle formed by the front surface of the storage chamber and the hinged door exceeds the threshold value, and imaging is stopped when the amount of change in the increasing direction of the angle is equal to or less than the threshold value .
Storage device. A main body with a storage room for storing objects ,
A hinged door that is rotatably provided on the main body and opens and closes an entrance formed on the front surface of the storage chamber, and an imaging unit that is attached to the hinged door and images the storage chamber are provided.
The imaging unit
The small directional change amount reduced in the angle of the housing chamber front between said hinged door starts when an imaging exceeds the threshold, stopping the imaging when the decreasing direction of the variation of said angle is less than the threshold value,
Storage device. The imaging axis of the imaging unit is inclined toward the rotation axis of the hinged door.
The storage device according to claim 1 or 2 . As the hinged door opens, the imaging unit captures a portion of the storage chamber that is farther from the rotation axis of the hinged door.
The storage device according to any one of claims 1 or 3 . The imaging unit is installed so as to image the entire storage chamber from the start to the end of the opening operation of the hinged door.
The storage device according to any one of claims 1 to 4 . A generation unit for synthesizing a plurality of captured images obtained by imaging by the imaging unit is further provided.
The storage device according to any one of claims 1 to 5 .

Images