WO2024161211A1

WO2024161211A1 - Method for taking pictures with a device, equipped with cameras facing in different directions, and associated device

Info

Publication number: WO2024161211A1
Application number: PCT/IB2024/000048
Authority: WO
Inventors: Zhihong Guo; Liang Han
Original assignee: Orange SA
Current assignee: Orange SA
Priority date: 2023-02-03
Filing date: 2024-02-02
Publication date: 2024-08-08
Anticipated expiration: 2025-08-03
Also published as: WO2024159546A1

Abstract

This invention relates to a device (10) for taking a picture of a physical entity (20) equipped with a first camera (11) and a second camera (12) facing in different directions, said device being adapted for: - taking (100) a first sample picture with said first camera (11) and a second sample picture with said second camera (12); - estimating (200) a light level for said first sample picture and a light level for a second sample picture; - comparing (300) the light level for said first sample picture with the light level for said second sample picture; determining (400) an action for taking said picture, according to the result of said comparison.

Description

METHOD FOR TAKING PICTURES WITH A DEVICE EQUIPPED WITH CAMERAS FACING IN DIFFERENT DIRECTIONS, AND ASSOCIATED DEVICE

TECHNICAL FIELD

The invention relates to a method for taking a digital picture with a device equipped with at least two cameras facing in different directions. It applies in particular to mobile device, like mobile phones, embedding a front and a rear camera.

BACKGROUND OF THE INVENTION

During the last decade or so, it has become usual for most people to take digital pictures, especially with mobile devices like mobiles phones (including smartphones). These digital pictures are aimed in capturing the everyday life of the users, and include sceneries or portraits including selfportraits.

Modem devices are equipped with high-quality cameras and embedded post-processing software application, so that the quality of the taken pictures are rather good, despite the fact that most users are not knowledgeable of photographic techniques.

However, under certain conditions, the quality may still be poor, in particular when there exists an important difference between the brightness of two elements in the scenery the user wish to take. In particular, when an object of interest (e.g. a person of whom a portrait is to be taken) is in the foreground, it may well have a different brightness of the background. In such a case, the camera may automatically adjust the photographic parameters (aperture, shutter speed...) on the background so that the object of interest may be overexposed or, on the contrary, underexposed. Under certain circumstances, the preview displayed on the screen of the device is not sufficient for the user to get aware of this default before s/he takes the picture. In some cases, the screen is not visible, or too far (when taking a self-portrait, for instance) or the preview may not sufficient to get a good idea of the resulting quality.

In particular, a self-portrait may be taken by putting the device at a remote place (and directed to the user). The picture is taken when a countdown lapses or when a remote control is triggered. In such a situation, the user has no easy means to determine in advance the quality of the resulting picture.

Patent application US2014/0211041 discloses a method for automatically applying a white balancing correction on a first image captured by a first camera, by analyzing this first image and a second image taken by a second camera.

Patent application EP2840776 discloses a mobile terminal having a first and a second cameras, wherein the amount of light entering the first and second cameras is used to perform a compensation correction on an image captured by the first camera.

Patent application US2010/149372 discloses a process where first image data are captured by a first camera and second image are captured by a second camera around the same time and the first image data are processed (for instance color balanced) using information that is derived using the second image data.

Patent application US2008/231726 discloses a device having a first camera capturing a first image, a second camera capturing a second image, wherein the white balance of the first image can be corrected based on the second image.

All these prior art solutions rely on a post-processing digital operation, where the digital version of a first image taken by a first is corrected and/or compensated based on data from a second image taken by a second camera. In other words, a main image (here the first image) is digitally corrected, after having been taken by a first camera, using data from a sample image (here a second image taken by a second camera). . . .

There is therefore a strong need to improve the existing methods for taking pictures, in particular self-portraits.

SUMMARY OF THE DISCLOSURE

An object of the present invention aims in alleviating at least partly the above-mentioned drawbacks.

According to a first aspect, there is provided a method for taking a picture of a physical entity with a device equipped with a first camera and a second camera facing in different directions, said method comprising

- taking a first sample picture with said first camera and a second sample picture with said second camera; estimating a light level for said first sample picture and a light level for said second sample picture; comparing the light level for said first sample picture with the light level for said second sample picture; determining an action for taking said picture, according to the result of said comparison.

Preferred embodiments of this method comprise one or more of the following features, which can be taken separately or together, either in partial combination or in full combination. said action comprises triggering an alarm intended to the user of said device; said action comprises adapting parameters of a camera chosen between said first and second cameras, to enable taking a picture according to the adapted parameters with said chosen camera; said action further comprises taking a picture according to the adapted parameters with said chosen camera; adapting parameters comprise adapting an aperture and/or a shutter speed of said chosen camera; said light levels are estimated according to the luminance of pixels of, respectively, said first and second sample pictures; said comparison comprises comparing a ratio of said light levels with a high threshold and a low threshold, and wherein an action is determined only when said ratio is above said high threshold or below said low threshold; said first and second sample pictures are taken substantially at a same time; said device is a mobile phone, and said first and second cameras are a front and a rear cameras embedded within said mobile phone;

- the method is used for taking a self-portrait of a user of said device.

According to another aspect, it is provided a computer program comprising code instructions for executing the methods as previously described.

According to another aspect, it is provided a device for taking a picture of a physical entity equipped with a first camera and a second camera facing in different directions, said device being adapted for:

- taking a first sample picture with said first camera and a second sample picture with said second camera; estimating a light level for said first sample picture and a light level for a second sample picture; comparing the light level for said first sample picture with the light level for said second sample picture; determining an action for taking said picture, according to the result of said comparison. Further features and advantages of the invention will appear from the following description of embodiments of the invention, given as non-limiting examples, with reference to the accompanying drawings listed hereunder.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 illustrates an example context for applying a method for taking a picture of a physical entity.

Fig. 2a and Fig 2b show two situations where the method for taking a picture of a physical entity bring advantages.

Fig. 3 illustrates a flow-chart corresponding to a method for taking a picture of a physical entity.

Fig. 4 shows a flow-chart corresponding to details of possible embodiments of this method.

DETAILED DESCRIPTION OF EMBODIMENTS

A method is proposed for taking a picture of a physical entity using a device. This device is equipped with at least two cameras facing in different directions, and especially directed in substantially opposite directions to each other in some embodiments.

An example of such device is mobile phones, or smartphones. Most recent mobile phones are equipped with at least one rear camera and at least one front camera. Such front and rear cameras are directed in opposite directions, generally perpendicularly to the plane of the mobile phone.

In general, the user can choose which camera he wants to use for taking a picture.

The front camera is the one facing the user when s/he uses the smartphone, i.e. where the display is. The back camera is the one usually used for regular photos. It is positioned on the back of the device, opposite to the side with the display. Of course, picture shall be understood here as digital picture, digital image or digital photography, these 3 terms being considered here as equivalent. A digital picture, or photography, is an array composed of picture elements, also known as pixels. Color values, or grey levels, may be assigned to each pixel of a digital picture. Further information on digital photography, is accessible to the skilled person, in large range of published documentation or on the web. The related Wikipedia pages may constitute entry points to this literature, like e.g. : https://en.wikipedia.org/wiki/Digital_image, or https://en.wikipedia.org/wiki/Digital_photography

Similarly to analog photography, a digital picture may be characterized with a set of parameters of the digital camera used for taking this digital picture.

The camera parameters comprise the aperture and the shutter time, the ISO sensibility, the focal length, etc.

In photography, the aperture is adjusted to control, together with other parameters, the quantity of light reaching the image sensor. In combination with variation of shutter speed, the aperture size will regulate the film's or image sensor's degree of exposure to light. Typically, a fast shutter will require a larger aperture to ensure sufficient light exposure, and a slow shutter will require a smaller aperture to avoid excessive exposure.

The ISO sensibility correspond to a signal gain of the image sensor of the camera.

Here again, the literature on these various camera parameters is very rich, and a good first pointer could be the related Wikipedia page: https://en.wikipedia.org/wiki/Film_speed

Figure 1 depicts a device 10 comprising at least a first camera 11 directed in a first direction and a second camera 12 directed in an opposite direction or in a substantially opposite direction.

According to embodiments, the device 10 is a mobile device like a mobile telephone or smartphone. Such a mobile telephone may be a larger format device like tablets, for instance. Other devices may also be considered as possible embodiments, like laptop computers, loT (Internet of Things) devices, etc.

In case the device is a mobile phone, the second camera may be a front camera (i.e. facing the user when the user uses the smart phone), and the first camera may be a rear camera. Usually, the rear camera is used for taking digital picture, whereas the front camera may be used either for self-portrait while holding the mobile phone or for video-conference for instance.

The first camera 11 may be directed towards a physical entity 20 in order to take a digital picture of it (or including it), i.e. it is directed in a direction so as to enclose the physical entity within this picture to be taken. The physical entity may be a human being, an animal or an object. According to embodiments, it may be the user of the device 10 indeed, who aims in taking a self-portrait (or “selfie”).

Accordingly, the parameters of the first camera 11 are set so as to include the physical entity 20 in the area that will be photographed. At least some of these parameters are set by the user, others may be automatically set by the camera.

Typically, the second camera 12 is placed at an opposite side of the mobile phone, even though other arrangements are possible as well.

The first camera 11 and the second camera 12 are directed in substantially opposite directions (to each other).

More precisely, only a side 21 of the physical entity 20, that is facing the first camera 11, is represented in the digital picture. This digital picture comprises also a representation of a background 40 constituted by the space area located behind the physical entity 20, the latter forming a foreground.

Turning now to Figure 3 that depicts an example high-level flowchart of a proposed method, a first step 100 comprises taking a first sample picture with the first camera 11 and a second sample picture with said second camera.

According to embodiments, a specific mode shall be selected by the user in order to deploy the method of taking a picture according to embodiments. Other modes may correspond to legacy method of taking a digital picture.

When the proposed method is used (for instance when the appropriate mode is selected by the user), this step 100 may be triggered as soon as the user triggers the taking of a digital picture. The subsequent steps 200-500 may be chained in a transparent way, without the user doing any other actions.

The sample pictures are used for estimating light levels only, and will not constitute any output to the user. So, they may be discarded at the end of the method or even after step 200.

Preferably, the first and second sample pictures are taken simultaneously, or substantially simultaneously. As they aim in allowing a comparison of the lighting conditions in the first direction and in the opposite direction, an objective is to minimize the differences in the photographic conditions, especially the time at which they are taken.

Once the sample pictures taken, a step 200 consists in estimating a light level for both the first sample picture and the second sample picture.

In particular, the light level may be based on the data of the whole digital picture,

According to an embodiment, the light level is estimated according to the luminance of pixels of said first and second sample pictures. All the pixels of the digital picture can be taken into account (providing then an overall light level), or, alternatively, a part of the digital picture can be considered, like a “region of interest” (Rol).

For example, the light level may be the average of the luminance level of the pixels of the sample digital picture (or part of them).

The luminance level of a pixel can be computed as the mean value of the different channels of this pixel.

In a RGB model, each pixel is associated with a red channel, a green channel and a blue channel, each having a value, respectively VR, VG and VB so as to define a particular color. Then the luminance L can be defined as:

L = (VR + VG + VB) However, other formulas can be used. For instance, different weights may be assigned to each channel so as to reflect the sensibility of the human eye to the different channel.

In fact, the proposed method is independent on the way to actually compute the luminance of a pixel.

Also, other ways of determining a light level of a picture can also be used (rather than computing a luminance). In particular, the light level may measure the colorimetry of the sample pictures. For instance, a difference of the Red/Green/Blue balance may appear in both pictures, and the light level may measure this difference, so as to trigger actions in subsequent steps. For instance, when taking outdoor digital picture, it may happen that the setting sun illuminates a physical entity with a red light, whereas the background would be darker and with a more neutral colorimetry. Such a situation may then be also detected.

Once the light level determined for the first and for the second sample pictures, both levels are compared at step 300.

The light level of the first sample picture and the light level of the second sample picture can be compared so as to detect an overexposure or underexposure situation, which would result in a bad quality of the resulting picture.

According to embodiments, this step 300 may comprise comparing a ratio of the light levels with a high threshold and a low threshold. For instance, if Li is the light level of the first sample picture and L2 is the light level of the second sample picture, then a ratio L₁/L₂ can be compared with a high threshold TH and a low threshold TL:

One assumes that the first camera is used to take the final picture, as depicted in the example of Figure 1.

In such a case, the light level L2 represent the quantity of light 32 received by the second camera 12. This quantity of light is the same, or substantially the same, as the quantity of light 31 received at the side 21 of the physical entity, which face the first camera 21. The light level Li represent the quantity of light received from behind the physical entity 20. Several objects may be present in the picture, but the main contribution would be provided by the background 40.

Then, one understands that if the difference between the light levels of the first and second sample pictures is too high, this may result in an overexposure or underexposure of the visible side 21 of the physical entity 20.

Figures 2a and 2b show example of, respectively, an underexposure situation and an overexposure situation.

In Figure 2a illustrating a case of underexposure, the visible side 21 may be too dark, compared with the background 40. On the contrary, in figure 2b illustrating a case of overexposure, the visible side 21 may be too bright compared with this background 40. In both situation, details of the visible side may disappear or may not be visible enough, resulting in a poor photo quality.

In both situation, embodiments of the proposed methods bring advantages by automatically avoiding or reducing the under/over-exposure effects, or, at least, alarming the user of such a situation for him/her to take the required actions (e.g. moving the device, or the physical entity, etc.)

Figure 4 shows details of this comparison step 300, as well as the subsequent step 400, according to embodiments.

In step 310, a ratio of the light levels is computed. This ratio may be LI/L₂ or ^2/^1- Then this ratio is compared with thresholds TH and TL.

Then, 3 situations may occur:

> T_H (case 320)

1-2 - < T_L (case 340)

T_L < - < T_H (case 330)

1-2

Case 320 corresponds to an underexposure of the physical entity 20;

Case 340 corresponds to an overexposure of the physical entity 20;

Case 330 correspond to a normal situation, wherein no underexposure or overexposure is detected. The thresholds TH, Tr may be automatically set by the device 10. They may be fixed or set according to a type of photography to be taken. For instance, if the user selects a mode “portrait”, the thresholds may be different that if s/he selects a mode “landscape”.

The relationship between the two thresholds may be as T_L = — , but TH other embodiments are also possible.

As an example, TH=8 and TL=0.125

The proposed method further comprises determining, step 400, an action for taking the picture, according to the result of the comparison.

In particular, in case overexposure or underexposure is detected, automatic adaptation and/or alarm can be triggered.

According to embodiments, the action may comprise triggering an alarm intended to the user of said device. The alarm may comprise a sound alarm and/or a visual alarm. Visual alarm may comprise detailed information, like, for instance, indication about how far the ratio trespasses the relevant threshold. Typically, orange and red levels may be used to easily shows this extent.

This alarm may not block the process to pursue and the final picture to be taken.

Alternatively or complementally, automatic adaptation may be triggered.

The automatic adaptation aims in avoiding or reducing the effects of the underexposure/overexposure situation.

Automatic adaptation comprises adapting parameters of the camera which is chosen for taking the final picture (for instance the first camera 11).

Following such an automatic adaptation, the action may further comprise triggering a (digital) picture to be taken, at step 500, with the camera chosen for taking the final picture, hence taking into account the potential automatic adaptation.

At step 500, the picture can be either taken manually, upon detecting a triggering action from the user, or automatically after that the parameters of the camera have been adapted. The latter case is advantageous in that the whole process (steps 100 to 500) may be triggered by a single user action on the device, the optimization of the camera being then performed transparently for its user.

Turning to the embodiments depicted in Figure 4, steps 420 and 440 correspond to the parameters’ adaptation in case of, respectively, detection of underexposure (case 320) or overexposure (case 340).

When no underexposure or overexposure is detected (case 330), no parameters’ adaptation is required, and the process may pursue to step 500, where a picture is taken.

Parameters which can be adapted comprise the aperture of the camera, the shutter speed of the camera, etc.

The camera’s aperture or the shutter time/speed can impact the light of the photo. Mobile cameras usually use a fixed aperture and let the shutter speed be dynamically set according to the photo to be take and to the amount of light entering the camera.

According to embodiments, this dynamic setting takes into account, in addition, the amount of light received by the other camera of the device.

According to embodiments, only parameters set dynamically by the user are adapted. If one parameter is preset (e.g. aperture), and the user chooses the shutter speed, only the shutter speed will be automatically adapted in case of detection of overexposure or underexposure situation.

For instance, in case underexposure situation is detected, the shutter time can be extended so as to have more light captured, and so as to reduce, then, the balance of lights between the physical entity 20/21 and the background 40.

For instance, the device can increase or decrease the aperture to an adjacent level in a preset list. Such a preset list may be as follows:

Moving to an adjacent level corresponds to increasing or decreasing the light by a 2 factor.

Example scenarios may be as follows:

In a first scenario, the initial parameters are set as: ISO=100, aperture=f/1.8, and the user sets the shutter time at l/125s.

If an underexposure situation is detected, a longer shutter time is needed. The shutter time is automatically adapted to l/60s.

In a second scenario, the same parameters are initially set but an overexposure situation is detected. Then, a shorter shutter time is needed, and the shutter time is adapted to l/250s.

In a third scenario, the ISO is initially set to 100, the shutter time to l/125s. The user sets the aperture to f/2.0.

If an underexposure situation is detected, the aperture can be adapted automatically to f/1.4

In a fourth scenario, with the same initial parameters, an overexposure situation is detected. Then, the aperture is automatically adapted to f/2.8

In case the adaptation is not possible or not sufficient, the alarm may comprise indication for the user to move the device 10. By moving the device, the balance between the light levels for the first and second cameras will move and may become more favorable. For instance, if the physical entity 20 sits between the sun (or other important source of lights) and the device, moving the camera to either side will change the direction of the camera that will not face the sun anymore. The proposed method may be used for taking digital picture with either the first 11 or the second 12 camera of the device 10.

An example use case would be to put the device 10 at a remote place, for instance directed to the user in order to take a self-portrait (or selfie). The picture is taken when a countdown lapses or when a remote control is triggered.

The digital picture is then typically taken with a first camera 11 corresponding to the rear camera (i.e. “main” camera) when the device 10 is a mobile phone.

However, the proposed method applies for taking digital picture with either the first or the second camera of a device (so, with a front camera of a smartphone as well).

The invention has been described with reference to preferred embodiments. However, many variations are possible within the scope of the invention. For instance, a detailed example of a mobile terminal with front and rear cameras directed in opposite directions has been described, but the present invention is not limited to such an embodiment and can be implemented with any device having two cameras which are facing different directions, and thus may take two sample pictures from different directions (not necessarily opposite to each other thus), from which substantially different light levels can be estimated in order to detect an overexposure or an underexposure situation affecting the subject of picture to take.

Claims

1. Method for taking a picture of a physical entity (20) with a device (10) equipped with a first camera (11) and a second camera (12) facing in different directions, said method comprising

- taking (100) a first sample picture with said first camera (11) and a second sample picture with said second camera (12); estimating (200) a light level for said first sample picture and a light level for said second sample picture; comparing (300) the light level for said first sample picture with the light level for said second sample picture; determining (400) an action for taking said picture, according to the result of said comparison.

2. Method according to the previous claim, wherein said action comprises triggering an alarm intended to the user of said device (10).

3. Method according to any of the previous claims, wherein said action comprises adapting (420, 440) parameters of a camera chosen between said first and second cameras, to enable taking (500) a picture according to the adapted parameters with said chosen camera.

4. Method according to claim 3, wherein said action further comprises taking a picture (500) according to the adapted parameters with said chosen camera.

5. Method according to claim 3 or 4, wherein adapting parameters comprise adapting an aperture and/or a shutter speed of said chosen camera.

6. Method according to any of the previous claims, wherein said light levels are estimated according to the luminance of pixels of, respectively, said first and second sample pictures.

7. Method according to any of the previous claims, wherein said comparison comprises comparing (310) a ratio of said light levels with a high threshold and a low threshold, and wherein an action is determined only when said ratio is above said high threshold (420) or below said low threshold (440).

8. Method according to any of the previous claims, wherein said first and second sample pictures are taken substantially at a same time.

9. Method according to any of the previous claims, wherein said device is a mobile phone, and said first and second cameras are a front and a rear cameras embedded within said mobile phone.

10. Method according to any of the previous claims, for taking a self-portrait of a user of said device (10).

11. Computer program comprising code instructions for executing a method according to any of the claims 1 to 9.

12. Device (10) for taking a picture of a physical entity (20), said device being equipped with a first camera (11) and a second camera (12) facing in different directions, said device being adapted for:

- taking (100) a first sample picture with said first camera (11) and a second sample picture with said second camera (12); estimating (200) a light level for said first sample picture and a light level for a second sample picture; comparing (300) the light level for said first sample picture with the light level for said second sample picture; determining (400) an action for taking said picture, according to the result of said comparison.

13. Device according to claim 12, wherein said action comprises one of triggering an alarm intended to the user of said device (10).

14. Device according to claim 12 or 13, wherein said action comprises adapting (420, 440) parameters of a camera chosen between said first and second cameras, to enable taking (500) a picture according to the adapted parameters with said chosen camera.

15. Device according to any of claims 12 to 14, wherein said device is a mobile phone, and said first and second cameras are a front and a rear cameras embedded within said mobile phone.