ES2979319T3 - Handling multiple HDR image sources - Google Patents

Abstract

In order to enable the versatile combining of images or videos needed now that HDR videos of considerably different luminance characteristics are emerging, the apparatus (301) for combining two images or two image videos (Im_HDR, Im_LDR), one of them being a high dynamic range image or video, the apparatus comprising: - a dynamic range setting unit (302) arranged to set a combination luminance dynamic range (CombRng), which is characterized by at least one maximum luminance (LMC) which is determined based on at least one of: a maximum luminance of at least one of the two images or the two image videos, and a peak brightness of a display for reproducing the two images or the two image videos, the dynamic range setting unit further comprising a luminance anchor determining unit (303), arranged to determine an anchor luminance (anc) in the combination luminance dynamic range (CombRng), - a color transformation unit (310), arranged to perform at least one color transformation, luminance in at least one of the two images or videos, wherein the colour transformation unit (310) comprises a source anchor luminance reading unit (311) arranged to read at least one source anchor luminance (L_SA1) from a first source (350) providing a first image or video (Im1_LDR) of the two images or videos, and wherein the colour transformation unit is arranged to establish a colour transformation (FF_1) to be applied to the first image or video, which colour transformation depends on the value of the source anchor luminance (L_SA1) by having the property that the source anchor luminance (L_SA1) is assigned to an output luminance in a proximity of the anchor luminance (anc); and - an image combining unit (320) arranged to combine the two images or two image videos to form at least one output image (Im_o). (Automatic translation with Google Translate, without legal value)

Description

DESCRIPTION

Handling multiple HDR image sources

FIELD OF INVENTION

The invention relates to methods and apparatus for correctly handling combinations of multiple images from different sources and with different luminance characteristics (or videos of temporally successive images) of which at least some have a high dynamic range.

BACKGROUND OF THE INVENTION

Until a couple of years ago, all video was encoded according to the so-called Low Dynamic Range (LDR) philosophy, also recently called Standard Dynamic Range (SDR). That meant, whatever the captured scene, the maximum of the code (e.g. 8-bit luma Y'=255; or 100% voltage for analog display gating) should, by standardized definition, correspond to a representation of the color white on an LDR display with a maximum display brightness PB_D (i.e. the brightest white color the display can represent) by standard agreement of 100 nits. If people bought a real display that was a bit darker or brighter, the viewer's visual system was supposed to adapt so that the image would still look appropriate, instead of, say, too bright (in case one has, say, a night scene in a horror movie). This is the relative rendering paradigm, which always maps the maximum luminance color in the encoded image to the maximum brightness of the display, whatever that may be. This would work if the actual available display on the rendering or video/image consumption side is actually not much brighter than 100 nits, but can produce undesirable results if the display is much brighter, say 10 times brighter, i.e. has a PB_D = 1000 nits.

Of course, this is a very strictly defined TV system colorimetry in order to get them all to work in a coordinated manner, as making a practical show typically meant keeping tight control of the scene lighting settings, since even in perfectly uniform lighting, the reflection from various objects can already give a contrast ratio of 100:1, and then there is still the difference in illumination of various regions. Typically one would illuminate and expose (by choosing an iris setting), so that the white colors in the (well-lit part of the) scene were roughly mapped to the white code, i.e. the maximum luma code corresponding to a maximum PB_C encoding brightness, and one could usually clip to that PB_C luminance value for even brighter objects or regions in the scene. Typically also cameras, especially the first generations of digital cameras from the early 2000s, had trouble capturing very bright and quite dark areas simultaneously, i.e. for example a portion of a scene viewed outside a room or car window would typically be clipped to white (giving additive red, green and blue color components R=G=B=max., corresponding to their square root luma code values R'= G'=B'=255).

However, recently both high dynamic range cameras (and, more importantly, new ways of HDR image capture or imaging in general) and HDR displays started to appear, and the rest of the video and/or imaging technology, e.g. image coding, image compositing, etc., has to be kept coordinated with these new technical level requirements. Note that if in this application a dynamic range is specified first of all with a peak brightness (i.e. the brightest represented luminance) only, we assume that the lower end of the range is pragmatically zero (while in practice it may depend on viewing conditions, such as front plate viewing or cinema screen light reflection, e.g. 0.1 nit), and that those additional details are irrelevant for the particular explanation. In case nothing technically specific is done for darker image colors, one could say that there is no particular reason why a higher dynamic range display, which would then be simplified to a higher maximum display brightness in the first place, should render darker pixel colors differently than a lower dynamic range display, like in particular an SDR display. Note also that there are several ways to define a dynamic range, and that the most natural one typically used in the explanations below is a display-rendered dynamic range of luminance, i.e. the luminance of the brightest color vs. the darkest one (of course in practice psychovisual aspects may also be involved, e.g. a single bright pixel in a dark scene may not be that impressive, but for some technical explanations below this can be discarded, and only the design of technical elements that fit what is needed for humans is made clear). An HDR image can be defined as any image that can contain pixel colors, and as stated in particular, brighter pixel colors, which cannot be encoded in an SDR image encoding. Since the colorimetry of such SDR image or video encodings was fixed (with the Rec. 709 video OETF corresponding to the sRGB OETF for still photos), it could be mathematically determined that the 8-bit luma encoding could accommodate a dynamic range of only 1000:1, i.e. from 0.1 nit to 100 nit, and no more. So if one wanted to be able to encode scene luminances that are to be displayed e.g. 2x brighter than the normal white SDR scene, i.e. up to 200 nits, one would need a new way of HDR image or video encoding (the new SMPTE 2084 is an example of an OETF that allows much brighter scene luminances, or at least how they are to be represented on a display under normal TV viewing conditions, e.g. up to 10,000 nits, to be encoded as lumas, even as 10- or 8-bit lumas, which could then be handled as "normal" lumas, e.g. for video compression like HEVC compression). The human visual system can process much higher dynamic ranges, and they also exist in several practical HDR scenes in the world (e.g. a sunlit scene viewed from inside a cave, or a dark room with a small window), so there is a need for higher dynamic range video drive chains, to increase the "liveliness" of images. Typically indoor lighting can be one-hundredth of outdoor lighting, so knowing that black objects reflect about 1% of incoming light, this would already imply the need for a dynamic range (DR) of 10000:1. But, if humans are sensitive to the blackest one percent of colors in a night scene in dark vision, a DR of 1,000,000:1 of the encoded image content may be in order (which some cameras can achieve and e.g. measure logarithmically). Of course, if for example a large cave is viewed from the inside, then in principle in some situations one might artistically choose to render that surrounding cave shape as minimal black clipped, but in other scenarios one would actually want or need all the different grey values encoded from the brightest to the darkest pixel region in the image, for example when some action is happening in the cave (and some partial external views in some cave shapes can look quite strange when clipped to black, due to the psychological detection inversion of foreground and background looking like torn paper cutouts, so one could avoid that by rendering more dark colours inside the cave). So a rendered display range of 0.01 to 10,000 nit may in practice be a good HDR range, but as said, encoding or rendering even a couple of 2x factors above 100 nit PB would already qualify as HDR technology, with a corresponding need to look carefully at how it should be handled differently from legacy SDR video philosophy. Note also that a code system by itself does not natively have a dynamic range, unless a reference display is associated with it, stating that e.g. R'=G'=B'=Y'=255 should correspond to a PB of 100 nit, or 1000 nit, etc. A common misconception from earlier times is that the dynamic range of an encoded image is closely related to the number of bits used. While that would be true for linear encodings, such as for example a camera's ADC which needs more bits to span the larger photoelectron variation range of the pixel well, and while it may be nice to have at least some more encoding precision, i.e. bits for higher dynamic ranges, the number of bits needed also depends on the chosen form of the luma code mapping functions, which is also called the Opto-Electrical Transfer Function (OETF). Thus, a 10-bit Y'CbCr encoded image could be an HDR image, with colours encodable down to a PB_C value, as well as an SDR image with high precision. What kind of image it is, i.e. how the lumas should be represented on a display with high enough PB_D to be able to display all the encoded lumas as corresponding luminances, will usually be determined by reading metadata, such as for example the encoded PB_C value in nits of the received image. The PB_C can also be interpreted as the PB_D of an ideal reference display for which the image has been color graded, i.e. on which display it would look optimal, rather than too dark or too bright.

Therefore, an image encoding that is capable of encoding images with luminances to be represented of, for example, up to 1000 nit is a good starting point for having good quality HDR, and the reader can take such a configuration into account when needing to summarize a part of a teaching below, unless other details are mentioned.

So in practice there are scenes that can have a very high dynamic range (e.g. an indoor shot with objects as dark as 1 nit, while simultaneously viewing through the window sunlit objects with luminances above 10,000 nit), and since displays are getting better (a couple of times brighter than 100 nit, with 1000 nit appearing currently, and PBs of several thousand nits predicted), one goal is to be able to render these images beautifully, though - and that's important - not exactly identical to the original, but at least very natural, or at least pleasing. So HDR is not simply about brighter and darker colour, but also about all the luminances in between - i.e. it's in fact about psychovisual appearances for humans ultimately. Of course, technically, for some HDR handling technologies, such as mere encoding, one could formulate the necessary appearance in terms of technical luminances to be represented on a display, because that is what needs to happen when driving the display correctly, in order for a human to see the colors appearing correctly. But the applicant wants to emphasize (so that the reader understands the whole story below and what it means) that when one wants to design a versatile HDR technical system, one needs to abolish the inveterate paradigm of the 1-to-1 direct connection view of mere encoding. In Rec. 709, a broadcaster could see on his standard (only existing) SDR monitor what luminances he was capturing, seeing how they were rendered on his monitor, and rendered on all the TVs at home, which would be the same everywhere, because there was only the unique flavor of a 100 nit SDR PB_D display. The only thing still needed to be done was to convert the luminances to be represented (as measured by the camera) into luma codes (because such codes would make better use of the technical details of the available transmission technology), by roughly applying a square root function, and then on the receiver side the inverse function (EOTF) would ensure that a viewer watching the decoded content on substantially the same TV in the same viewing environment as on the creation side would experience the program as intended. That's still the vision of HDR10. It uses a different EOTF than Rec. 709, but still the reference display-rendered colors that were displayable on the creation side are still rendered with identical luminances on the receive side, simply by applying a fixed mathematical equation to the received lumas. But the real HDR world is highly variable, with different viewers having different displays with different PB_Ds, different number of lamps on in their viewing rooms, etc. And that was exactly what HDR movies encoded with HDR10 were criticized for: because HDR movies contain many pixels that are relatively dark, they will be difficult to view under bright surround lighting (although the important action often takes place in the relatively darker parts of the video). And HDR10 offers no solution for that, hoping that display manufacturers will sort it out. However, the applicant thinks that in general it is a complicated issue, dependent on the type of HDR scene image, and there should be a mechanism to allow the content creator to have a say in the ultimate display representation (variable, i.e. not some fixed mathematical inverted directly connected 1-to-1 between some luminance and luma representation of the desired colors) of the HDR image colors.

The reader should also understand that because a viewer is typically viewing content in a different situation (sitting in a dimly lit living room at night, or in a dark house, or in a movie theater, rather than standing in a bright African landscape), there is no identity between the luminances in the scene and those ultimately displayed on the TV (or other screen). In reality, no TV can or will be able to accurately represent at least some objects like, say, the billion-nit sun, nor does the viewer want to be blinded by it (when viewing a small screen, the angles or glare determined thereby are not the same as in a 4*pi solid angle surround environment, so care must be taken with that as well). The applicant solved this translation of the appropriate luminances to be displayed from the relative RGB values captured by a camera by defining a master HDR color grading, to begin encoding and communicating. This set of graded HDR images can be created as desired by an artist (it should be clearly obvious to the reader that while we may mention the artist, what the technical aspects of this invention should be, allowing the artist to do the communicating and controlling of his requirements), if he has a reference monitor available for example, he can sit in a typical viewing environment and perfectly grade his HDR images for a 5000 nits PB display. But that's only half the story, namely being able (at all, under a new set of video requirements) to consistently determine an artistically pleasing looking image for an HDR scene (and being able to faithfully represent that image if one views it on exactly the same 5000 nits PB_D display under the same viewing situation constraints). What should happen if a consumer receiving 5000 nits HDR images only has a 1000 nits display will be explained below (display tuning). That's a completely different question, far from something that can be handled with a direct 1-to-1 communication approach, and one can start to ponder what that single reference system should be with a PB_D reference display.

If this request is talking about "high dynamic range" images, it means that one of them has image content suitable for display on at least a 1000 nits PB_D display (of course, content can also be displayed on a 100 nits or 500 nits display, but then it's not as perfect as it could be - for example lamps will be dimmer than ideal). Therefore, in this image there will be content that makes it visually interesting when displayed on a higher PB_D display, i.e. there are some image objects that are much brighter than average (or, conversely, compared to the brightest color that can be normalized as Y=1; at least some objects that are much darker, than typical for SDR images). For example, an indoor scene might have brightnesses going up to 500 nits or even higher in the original scene, and depending on whether one has a well-lit greenhouse or a narrow tunnel region a couple of nits or less, and outdoor brightnesses might be in the real world several thousand nits. On a display, one might then render the interior below 100 nits and use whatever the display has available above that optimally for outdoor objects, e.g. up to 500 nits if one has a "dull day" movie or show, and up to 1500 nits if it's sunny outside, in case one has a display that can render up to 1500 nits or higher (PB_D >= 1500 nit). Thus, even in rendering alone, there may be a ratio between the brightest and darkest image regions of (1500/2)/(100/2) or at least 500/100 (this being typically definable as regions that are differently illuminated, i.e. midpoints of regions that still have intraregional contrast of up to 100:1 and typically 30:1 due to object reflection). If this is coupled with luma codes via a continuous function, those lumas will also be separated, but will be less useful for specifying content (depending on the shape of the OETF and possibly the technical requirements). When this application mentions lower dynamic range (than HDR images) or low, we are talking about images (perhaps a different graded look of exactly the same original scene image captured by the camera, i.e. the HDR image of the same scene) that have (or are in fact associated with a PB reference display) a peak brightness that is at least one stop (a multiplicative factor 2) lower, or typically at least 2-3 stops (4x or 8x, or more). An SDR image would fit the standard prescribed PB_C of 100 nit. The contrast ratio of all objects in those is usually not much more than 100:1 or up to 1000:1 at most, i.e. you typically see a much more uniform luminance or luma histogram.

A second thing that is needed if the dynamic range of image pixel luminance (which should ultimately be represented) for which luma codes need to be defined is known, is how to actually distribute the codes, i.e. with which code mapping function to associate a corresponding luma code Y' (not just 1023 in a 10-bit encoding which can usually be mapped to the PB_D of the associated reference display, but for example with what luminance the luma code 743 should correspond) with an object or pixel luminance Y, or vice versa.

Over the past 2-5 years, a couple of companies have proposed different ways of encoding HDR video (i.e. moving pictures, or in other words, sets of temporally successive images). There has been a bit of a rush towards the first products, expected in 2016, so despite the complexity of encoding and handling HDR images, some corners may have been cut. This has led to different encoding techniques, which were even based on different basic philosophies. Not finally settling on this would mean that all image handling systems would have to be able to deal with the differences, and that could get quite complex (certainly compared to the simplicity of single colorimetry in the SDR video era, 5 years ago).

Dolby Laboratories Inc. started with a two-layer system (i.e., a base image and a correction image are needed to make an HDR image, in line with scalability concepts in video compression), which is somewhat more expensive to put on cheap chips, but has also designed a single-layer coding and contributed a basic EOTF, the so-called Perceptual Quantizer (PQ), recently standardized as SMPTE ST. 2084.

The technology is explained very well at: https://www.smpte.org/sites/default/files/2014-05-06-EOTF-Miller-1-2-handout.pdf.

The idea behind this was that the classical square root (or more exactly Rec. 709) optoelectronic transfer or conversion function OETF (the inverse of the square power EOTF), which was conventionally used to convert camera scene luminances to luma codes, is not suitable for the high contrast ratios between different regions that are likely to occur in HDR images, no matter how many bits are used (or at least for pragmatic numbers like 14 bits or less, for the high dynamic ranges that may occur in real life, like master gradations for 20 stops or more scenes). But this turned out to be a good inspiration. Although originally designed due to an accidental physical behavior of CRT electron guns, the square root function closely followed how human vision converts scene luminances into visual luminance codes, and so was a good way to make better use of the codes (with a next code corresponding to a next psychovisual luminance impression, so not too many codes are wasted on what can't be easily perceived anyway, nor, worse, no regions where there are too few codes to accurately encode smooth, precise luminance gradients). Now one can generalize this curve giving for each successive luminance interval (e.g. one stop brighter) approximately an equal amount of luma codes up to a maximum encodable brightness, which Dolby normally considers to be 10,000 nits (which should indeed be sufficient for pragmatic display rendering because some people already complain that it is too bright, although by experiment one can also find that one can make impressive pictures even with large and even flickering regions above 20,000 nits, and everything of course depends on how the luminances of various picture objects are allocated over the possible interval, and one does not always need to fill that up to PB for every picture type all the time; and those who complain about painfully bright pictures which might be dangerous should realize that the full moon at night is also 5000 nits, and nobody has ever gone blind by looking at the moon).

So Dolby standardized on a reference EOTF, and one can use its inverse being the OETF (or roughly the inverse in case one wants more behaviors) to define lumas for all objects/pixels in the image, and then one can put those codes (typically classically color-coded as YCbCr, but one could also use RGB encoding), for example, on a Blu-Ray disc, after applying regular HEVC encoding techniques. One then has a master HDR rating on the BD, (Blue-Ray Disk), which can be rendered, any way the artist wants, for example, by a 5000 nits TV that can decode it. Thus one has a mechanism to provide end-to-end, 1-to-1, for displaying HDR (only) images of some scene on a consumer display, at the consumer premises. Of course, when it is said below, for example, via a BD, the expert reader will understand that all other image or video communication mechanisms, such as, for example, satellite transmission, Internet video delivery, etc., are also understood, since the concepts presented have a wide applicability. This EOTF 2084 has also been chosen as the luminance-to-luminance mapping function for HDR10 encoding.

The British broadcaster BBC has designed a different way of encoding HDR video. They came up with a standardized OETF (because they started from the camera view rather than the display which standardizes an EOTF from a reference display), which is a bit different in form. Their vision and technology is described very well at: http://down-loads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP283.pdf

They define an OETF that closely follows the classical Rec. 709 square root OETF function shape down to a mid-gray (Y'=0.5) of the classical 100 nit Rec. 709 OETF, and for higher luminances it becomes logarithmic, in order to be able to squeeze more bright colors over the white object color of the scene in the available codes. It was inspired by what cameras were already doing for a long time when they needed to make a good LDR output image for a scene that was too HDR. Namely, instead of a luminance contrast ratio of about 200:1 pixels, one had for example some regions that were too bright, and one tried to squeeze out brights that are 4 times brighter than the white of the scene that should be mapped to Y'=255, hence the rendered luminance Y=100 nit. Although the white objects in the scene below become greyish, it can sometimes be psychovisually acceptable because the brain also sees and interprets the brighter pixel regions in the total image. One can always map such bright scene luminances to display PB by simply reducing the camera exposure (e.g. by changing the iris), but then the newsreader's face may become too dark. So what the cameras did was to follow the curve up to, say, Y'=80% or some other inflection point, and from there use a curve that has a small slope, so it can represent a good number of higher scene luminances in the remaining 20% of the codes. Or one could use some camera-specific log curve, which reduced the luminances of the darker pixels a bit (so maybe darkening faces a bit, which shouldn't necessarily be a problem in a high contrast scene, because the viewer understands that the person is standing in a relatively dark region (possibly shadow), and the brightness could be adjusted appropriately, on a brighter display, anyway), but freeing up some more code for the really bright pixel colors which are then usually rendered a bit more coarsely, e.g. with soft clipping. This way there is a balance between colors that need to be displayed bright enough, and a still reasonable visual quality for brighter colors. But the problem was that every camera did this logarithmic mapping of captured relative scene luminances (of course, in essence, absolute) to luma codes in a different way - which was still reasonable when cameras were still just extra-good LDR cameras, but more problematic for a camera that is actually a good HDR camera, which is supposed to capture complex HDR scenes of maybe 20 stops. So the BBC wanted to define a fixed curve for all practical purposes. Still, applying that fixed OETF curve as a luminance mapping curve to the master HDR graded image as input would result in a reasonable SDR version of it, and if you transmit this image, then legacy displays can directly render the SDR image, and HDR displays can apply the fixed inverse curve to reconstruct the master HDR image. And one advantage of the BBC's approach is that little needs to be changed in the technology, as everything works pretty much as it used to (but that might be too tricky and come at a price, during production or at least some video combinations).

They may have a number of different starting points, for example Dolby may have been more focused on the professional film market, whereas the BBC may have been more focused on practical TV requirements such as field productions of shows, deployed TV crews, etc. But one important difference that is interesting for this patent application is that the BBC considered that pixel luminances (or indeed colours) should be represented relatively, whereas Dolby considered that they should be represented absolutely. That's an important difference in philosophy, because it determines whether you stand on the scene-referential side or the screen-referential side of the gap that needs to be crossed to arrive at the optimal representation. It's a big difference about what luminances are to be encoded in two images that need to be combined, but it goes even further - it may influence how such an alternative philosophy would dictate that combining may be necessary, from the point of view of focus and essential properties.

So the BBC addressed the problem by defining a basic OETF, which defined master-grade (or at least camera-captured, with also some appearance decisions like choosing an exposure, and maybe a gamma curve) relative HDR luminances and lumas, and the appropriate display luminances to be rendered should be calculated by applying some gamma system, which would depend on both the PB_D display and the brightness of the viewing environment. That would mean that if a display manufacturer got hold of BBC-encoded content, which may contain luminances up to 12 times the white color level of the scene, but perhaps not depending on the actual scene content and the camera iris settings, it could do one of two things. You could use BBC's philosophy relatively, and always map PB_C code to PB_D display, whatever PB_D display is (and then maybe apply some gamma brightness and contrast modification function, depending on what PB_D one actually has, but at least white would be rendered to white, which has at least one advantage which is that if there are no such overly bright regions but only relatively darker pixel colors like in a foggy scene, it is established in an easy calibrated way that such darker coded colors will also be rendered darker than PB_D, but on the other hand brighter parts of the scene will be rendered with varying luminance PB_D; which would seem to be a nice property for combining image data, but is it?). Or you could pretend that this encoded image luminance data is good around or to exactly some absolute luminance PB_D value, say 1000 nits, and then do your own internal degradation in case the TV has say only 500 nits Pb_D (via blind gamma mapping). Dolby however felt that one should specify in absolute terms what luminances should be rendered, at least if you have a high quality 5000 nit Sim2 or Pulsar display, and how this image was artistically generated doesn't matter as long as you can define the necessary object luminances along the EOTF master luminance reference range 0-10000 nit. So for example if one has a dark face in a horror movie, in the absolute procedure one would typically say that this image should be rendered at say 15 nits, which can be rendered on both low and high dynamic range displays. However, in a relative procedure, the brightness of this face would be scaled, even if one didn’t go so far as to have 15% of 100 nits also mean 15% of 5000 nits, since a PB-dependent gamma function of the display would correct for that somewhat, but still, it wouldn’t be determined so precisely at what luminance the face would finally be rendered on a display. That makes one think about what the artist would have liked, if on the rendering side so many changes can happen, so there is something to be said for a more accurate absolute rendering (if the full dynamic range of a display is used, of course the viewer’s sensitivity would adapt to that, but typically a subrange of an HDR display would be used if the content required rendering down to only, say, 900 nits, at least for, say, 99% of the image pixels). Still, absolute luminance encoding does not have a complete answer either, because if the PB_D of the actually present is lower than the PB_C of the encoding, i.e. the luminances needed for the image to be optimal, then some degradation strategy is still needed, which had better not be too blind and against the appearance reclassification needs of the particular HDR scene or scene type. The reader can imagine that such different philosophies lead not only to different codes determining EOTFs, but probably also to different handling of images, i.e. how the luminances of the object pixels are to be assigned to various regions along that curve or its corresponding reference luminance range. And if that already leads to difficult discussions when one has the relatively simpler problem of mapping some input image with a particular PB_C to some lower PB_D display, which one could still imagine to be something designable at will (for example, if one uses what we will call an r-curve, because of its shape it increases brightnesses a bit, and then for brighter luminances it starts to get a smaller and smaller slope, effectively compressing those into smaller subintervals of the SDR luminance interval, i.e. like the first quarter of a circle or ellipse), since one could stretch, for example, the boost part for darker colors of the HDR to SDR luminance mapping curve to boost darker colors a bit more, which would increase the brightness of some typical object in that part of the input luminance interval or the curve, say someone's coat, then the reader can imagine that things can become more complex when one wants to harmonize the brightnesses of objects in different subintervals. images, of different dynamic range, generally illuminated differently and potentially encoded differently.

The applicant like Dolby also thinks that it is better to define a master EOTF, which we define as 0-5000 nit. One reason is that a captured image of a bright African scene is fundamentally different in absolute terms from that of a Norwegian autumn, so while one can always apply all sorts of mathematical colour transformations (affecting at least luminance) to ultimately produce the appropriate brightness appearance, why not already encode this difference in luma values and their corresponding absolute luminances in the reference luminance range? (Advantageously, one could even design such lumas to already correlate to psychovisual brightnesses, then one could more easily encode the appropriate additional curve shapes by grading from the master initial HDR image corresponding images of different dynamic range, which neither HDR10 nor BBC have.) But most importantly, in the end there is only the ultimate representation of pixel colors (even if that can happen on displays of different PB), and without at least an associated reference display (defined not only by a peak brightness, but also by its grayscale behavior like an EOTF), the image is just a set of numbers (which can hopefully be decoded correctly), not much more. It's all about the ultimate experience of the HDR image, and not so much about how it was designed up to that point - and that's also what graders see on their reference display(s). One must be able, on the creation side, to make an unambiguously defined initial image. So we need to start by standardizing a reference EOTF that fixes this (otherwise, for example, the digital luma numbers on the computer used for color grading are undefined), and then more systems can be built on top of that (even if one wants to build some relativity back into the system, at least that can be controlled relativity, determined based on some commonly acceptable reference luminance measurement system).

Now, having summarized above some basic ideas of background HDR needed for the novel, we can move on to a more practical situation, of having available different images for which the creators decided for some reason that they had to be encoded by one procedure or another.

The difference in behavior of Dolby EOTF vs BBC EOTF (the present patent applicant designed an EOTF that is very similar to Dolby PQ, or you can use Dolby PQ) can be seen in Figure 1. OETF 113 is the PQ EOTF (we've shown only the part up to 1000 nits, because if we show the graph up to 10,000 nits the other two curves become unclear). BBC OETF 112 is a relative curve, so there is some debate about what absolute luminances it should encode, but we assume it will typically go up to 500 nits (or maybe 1000 nits, but then it may start to show errors and become less appropriate). The Rec. 709 curve can only encode absolute luminances up to 100 nits, and we need to remember that for various reasons in the distant future there might still be a lot of SDR content (which for example was not easily converted to some HDR version). On the y-axis we have shown relative lumas, because we want to avoid the bit depth discussion, which is not that relevant for this application (we will assume that all encodes will use 10 bits for all three color channels, so the maximum luma 1023 becomes 1023/1023=1.0). It should be obvious that if we get an input luma of 0.5 (i.e. 512 or 128 in 8-bit) that can mean anything as a corresponding luminance, depending on the encoding procedure used (somewhere around 0.7 could be 10 times different, so just doing a pixel luma replacement switch might suddenly produce a very dark or very bright replaced pixel). In Figure 1B we show how the curves differ if we also define them at a relative input luminance of 0-1.0. It should be clear that if one starts with the lumas of an encoded image, the luminances to be represented can be calculated from the inverse curves.

Now, ignoring all the practical issues such as color errors after various encoding qualities, chip redesign costs, etc., it can be said that all three encoding mechanisms work well if only a single set of HDR images need to be communicated, which is what the first phase of research, development and standardization focused on.

But practical HDR management systems need more, otherwise users are left with loose ends and questions, which you can't do as a systems designer.

Applicant has designed an encoding system, which can not only handle the communication (encoding) and use (ultimately proper rendering) of a single standardized HDR video, for a single typical type of display in the field (everyone who needs to have a 5000 nit PB display, or perhaps everyone who does not already have a legacy 100 nit TV can just buy a 1000 nit PB TV), but which system can handle multiple possible display types with multiple PB_Ds in the field, by being able to compute an optimally (re)sized image for each particular display, which Applicant calls tuning or display tuning. Its main features, and particularly the decoder 251, are summarized in Figure 2 (to the extent necessary for this application).

The applicant had the approach that any final appearance optimisation depending on the viewing environment could be done on the display side, though not with a final gamma as proposed by the BBC, but by a mechanism which (though calculated by a receiver, e.g. inside a TV or BD player) is not solely determined by the TV (i.e. its designer), and not only based on receiver-side factors such as actual display peak brightness and display surround brightness, but also depends on the picture content itself, as images are complex, and so is their optimisation to the peculiarities of the rendering side. This crosses the old and difficult gap between the creation and the use of content. The receiving side may be the only side which really knows what display capability is actually available, however the creation side is the only side which still definitively knows what the content is about, such as the artistic desiderata. If you can't sell the filmmaker or DOP on all the TVs to optimize the color look on the receiving side, you can at least let the creator specify the intentions parametrically, at least approximately, with the image creator being able to mathematically specify: "if I had a display of this PB_D, I would at least prefer to render the various pixel luminances like this and this."

The applicant's engineers argued that the image content-dependent part could be handled by allowing an artist's color grader (although automatic image analysis functions, i.e. determining luminance mapping functions based on established luminance properties of current images, could also be used) to specify the optimal color transformations according to him that would at least shift the object pixel luminances to (simply) two aspects corresponding to a different dynamic range or in fact PB_C, at two extremes of the range of possibilities (of the maximum brightnesses of the display). Thus the HDR aspect image should correspond to a rather high PB_C that the applicant chose of 5000 nits (although the applicant can use another PB_C just as well), and the low DR image can also be a sDr rating of 100 nits, since that would satisfy for example 90% of the initially existing market of people using legacy SDR displays. So for a particular scene the color grader might say "if I only have 100 nit PB, I should set the lamps in the image that I'd prefer to see rendered to 4000 nit in order to have lamp objects appear very bright in the rendered image, but that's impossible now at 95-100 nit when the first thing I want is for there to be enough luminance headroom in the display's luminance range for lower luminance objects in the room to be bright enough, i.e. between say 10 and 90 nit, so maybe I'll now set all the lamp pixels to exactly 100 nit." The same can be done for example to make a night scene appropriately dark, and for street lights to be bright enough, for both a 5000 nit and a 100 nit display rendering. The reader should pause for a moment to reflect that it is not so trivial to do a night grading that looks appropriately night-like, i.e. of the correct luminance for objects (e.g. a black person sitting in a poorly lit, shadowed area), and then in both HDR and SDR (especially with other constraints such as e.g. IC complexity), but it can be done, at least if you are not a perfectionist, then reasonably well (of course, if the display roughly follows the specs, and does not render a completely different image by itself).

Having obtained these two graded images (M_HDR and M_LDR, whose luminance distribution appearance is usually determined by a human), one for the situation where one has a high quality HDR display to render the images, and one for legacy 100 nits SDR displays, the applicant wanted to avoid encoding and communicating two whole sets of images, i.e. a need for dual encoding, which would need two channels etc. And the applicant also wanted the two different DR graded images to be encodable with existing MPEG-like DCT-based encodings, so we decided to have one of the two video image sequences be parametrically defined with color transformation functions (F_L), which would be used to compute the images from the set of images actually being communicated (e.g. but not necessarily the SDR image sequence). Thus, we may communicate via some video communication system 250 (the skilled reader understands that our system may be employed in many systems, e.g., an over-the-air transmission to a receiver which is a computer either from a professional video delivery service, or between consumer appliances connected in an internal network, or a professional secured cable-based communication system, the receiver being a professional appliance in a cinema, or an Internet-based system, etc.), e.g., the master HDR images M_HDR, such as the HEVC S encoded video signal encoded by the video encoder 203. The encoder also adds all parameters defining the color transformation functions F_L as metadata, e.g., SEI messages or the like. Or vice versa, we may transmit the master LDR classification M_LDR, together with functions allowing higher dynamic range images, such as M_HDR, to be recalculated on the receiving side.

Thus, on the creation (or transcoding) side there will be, for example, some color grading apparatus 201, containing a color transformation set 202, arranged to determine and apply various color transformation functions F_L as desired, to obtain a master LDR aspect image M_LDR from the previously created master HDR image M_HDR, e.g. 5000 nit PB_C typically (which may be done in the same color grading apparatus at the same time, or may be otherwise determined, beforehand). As an example of image processing, the reader may refer to WO2014/056679, the application teachings of which are incorporated herein in their entirety for countries which permit it. As an example of a transformation, one can get an image from a function that does not affect chromaticities, but only pixel luminances (all discussions below will, for simplicity of understanding, focus on luminance transformations) that bends higher luminances down a bit and boosts darker colors on a relative HDR input of 0-1.0 (x-axis) to the LDR output plot of the function, so that they will look bright enough on the relatively darker 100 nits display (an r-shaped curve). But more complex functions that customize and potentially only locally adjust some object luminances, and also transform chromaticity aspects like saturation, are also possible, as long as two related optimal looks can be made: an LDR image and an HDR image.

Now the next thing that can be done with this encoding technology is that if one does not really need to recalculate 5000 nits M_HDR image(s), because one needs correctly graded medium dynamic range M_MDR images that have a correctly graded appearance for a actually connected display 280 of peak brightness PB_D= 1500 nits for example, then with our display tuning algorithms one can obtain the correctly graded M_MDR images, from the received M_HDR (or M_LDR in other embodiments/applications) images. The receiving side has a classical HEVC decoder e.g. 260, which outputs a normal (e.g. linear RGB or YCbCr) M_HDR image, and the required color transformation functions (F_L or its inverse shape mirror function) to change M_HDR to M_LDR. Based on all of this information, our color tuning assembly 271 in a color processing assembly 270 (which we assume for this illustrative embodiment is separate from a TV, e.g. within a set-top box, or a server of a video-on-demand provider, etc., but may also reside within a TV) can not only determine the necessary optimal functions F*_L to determine the required M_MDR picture(s) from the received picture(s), but also ultimately output the M_MDR medium dynamic range picture(s). The display adjustment algorithms to obtain a suitable F*_L function for SDR to MDR luminance transformation based on the SDR to HDR luminance transformation function(s) F_L can be quite complex depending on the desired embodiment variant, for example to take into account details of the viewing environment, but for the understanding of the present applications all such details are beyond what is necessary, so the reader may simply note that there will be some luminance function F*_L to apply to obtain from any incoming encoded image an MDR image, for example let's say a gamma function.

While this mechanism now allows for optimal HDR image rendering not only for a single possible type of HDR display (e.g. everyone needs to buy a 1000 nits PB_D display, and no one can buy a 3000 nits PB_D display and still get a nice image, at least not if one cares about colorimetric accuracy), but for any possible display, this additional technical expertise is still not enough for a fully flexible HDR handling ecosystem. So far, the outlined technology still only focuses on getting a single HDR set of video images, from a single captured HDR scene, correctly to the various consumer sites, and a way to render it properly on a single available TV. That was already a daunting task to prepare for the new HDR video technology, so other aspects were delayed to be dealt with later.

But today video programming is more complex. One may want to get feeds from all sorts of different sites, and it has become easy to make video content (even for consumers), with simple handheld devices (even a life news feed from a helicopter, or some sporter filming his actions with a GoPro strapped to his helmet), and communicate it easily through various media such as, for example, the Internet. And, in fact, at least theoretically, because wide consumer deployment may still need to take off, televisions are becoming more like computers, with Internet connections, and capable of using or co-supplying various data sources, presented as visual information. For example, we expect various uses of video, e.g., one may be watching a news program in the back and at the same time playing some video game in some rendered area of the screen, whether it be an area of the same screen or on an adjacent screen.

That problem started in the LDR area, but it was much less difficult, because all videos were defined in the same Rec. 709 colorimetry, and more or less similarly lit and exposed. And even computer graphics of, say, a game were defined similarly. But nowadays HDR video or images can be anything, because that is what one would like to cater to, so if one does not find a good frame strategy to handle and in particular combine multiple (partial) image contributions, one could expect chaos in the future. As said, in the past all video was defined in the same LDR frame, i.e. all peak luma codes corresponded to the same rendered luminances of 100 nit, and luminances corresponding to lumas below the peak could be decoded by means of the only existing Rec.

709 EOTF.

There was still some concern (especially on complex productions, or outside of broadcast and other field productions), and a need for careful handling, that cameras might have some "minor" setting adjustments. The most important were typically white balance (i.e. the chromatic appearance of neutral colours and whites, i.e. whether white objects looked too blue or too yellowish, which might become apparent when switching between different camera feeds) and exposure settings. If one were to lower the exposure of a capture of the same scene from one of the cameras, that camera would quickly be seen as giving an image that was too dark. Especially for professional multi-camera productions of the same program that was viewable, like in a news program that cuts to a live report, one might expect a different capture (especially if it's an amateur mobile phone video), or at least, that was a capture (either temporally intercut or picture-in-picture) of a totally different looking scene (e.g. a night scene of Cairo, instead of a well-lit studio in the US). Now, there are two situations one must discriminate. Of course, if one switches between a night scene and a midday scene of a desert, the viewer would expect some difference in the average brightness of the scene, or of the luminances of the objects in the scene, although it should still be a reasonable difference - e.g. the desert scene doesn't startle the viewer, or make him squint. But on the other hand, one can make any scene look nighttime, or bright, by playing with the combination of the camera exposure and the final brightness of the rendering screen. In the SDR era that sometimes led to strange images, if a camera man had to make an inconvenient exposure choice. For example, someone standing far from the window in a deep room would be much darker than the objects near the window. The cameraman wanted to expose the person in the darkest area, so that it would look nicely colorful, which could result in the left side of the image, and all the objects in the room there, being totally clipped to white, causing the viewer to have to see an ugly “half image.” But at least the colorimetry was reasonably defined. One knew that the colors would come out like that, and that would be the same regardless of whether such images were blended or rendered solely. In the HDR era, one wanted to solve these problems by making all regions encodeable instead of clipped to white, for example by reducing the exposure based on how much brighter exactly the bright parts of the room are that correspond to the luminance of the actor or presenter in a particular scene. But that would unpleasantly come back as a problem, in case one wants to harmoniously mix video content, therefore, there is a need for the technology of the following realizations.

Also, more complex and recent cameras got additional settings, such as, for example, a black gamma. It might not be easy to perfectly match two different cameras (if they are from the same brand, one could at least, for example, turn off or set similar parameters for all the "exotics"), but at least it was the job of the camera shader, who is an engineer with a special eye for color differences (or technical director) to remotely control all those colorimetric settings. That frees up the actual camera operator to focus on framing, which is already a difficult enough task, especially for some productions with actors jumping around the scene, etc. In any case, in the SDR framework, even if one did things a bit wrong, the fact that the content creator made sure that at least his main subject was reasonably lit (i.e. at least 10% white, or about 1 stop below middle gray as a luma or visual lightness impression), combined with the limited 100:1 range roughly corresponding to various diffusive reflectances, any incorrect combination with some object luminance incorrectly positioned compared to other object luminances would usually not be too disharmonious (at least critical viewers might perceive that something is wrong, e.g. maybe blacks would become a bit more milky gray, but nothing would look really bright, or it would flare with large brightness differences, as something that might happen in HDR production).

But now, in the emerging era of HDR, there is no such thing as a 0-100 nit LDR reference range, nor do cameras (and other systems) act relatively simply in that range (i.e. they typically perform non-extreme color transformations). As you can see in Figure 1, even if in the future we choose to use only one of the possible HDR codecs, which is not obvious, there will still be some content defined by the LDR codec. And a difference between 5000 nit and 100 nit is so extreme that it cannot be called a “minor” appearance change (sometimes only professionals notice it), but rather a very noticeable and potentially even extremely irritating variation (and we can expect that if various devices like TV in the future would want to act according to their designer’s preferences throughout that vast luminance range of that vast color space, that variability may become even more extreme, and not as desired).

The problem may not be so obvious even if one is watching a single program from a single source. For example, if someone is watching HDR content from a tall PB_D display in a dark environment, the colors may be too bright, but at least their eyes can adjust. The differences always become very clear if one has some reference to judge an image, for example in a picture-in-picture (PIP) presentation of two videos.

Since images nowadays can come from everywhere, not necessarily under the control of a director who decides on the look of the images nor displayed by themselves in a controlled frame, but only potentially joined together sometimes for the first time on the screen side (e.g. a dedicated commercial is inserted), a good framework to coordinate the combining of several original HDR images is urgently needed.

So, as said, most of the (new) HDR technology is just focused on being able to communicate HDR image information, by communicating clearly defined pixel lumas, which can then be converted into rendered luminances. If one has a display with a PB_D lower than the pB_C of the content, one would have to do something, since one cannot render the luminance down to PB_C on that display. One could do linear compression (white-to-white mapping relatively, i.e. rendering the pixels in the image with PB_C encoded luminance actually on the display with PB_D luminance), but then in all likelihood the rendered image will look suboptimal (e.g. some of the darker parts of the image may look too dark, maybe even with the action being invisible). So one could try to do a smarter luminance mapping, which keeps the darker luminances reasonably visible, for example with an r-shaped curve. General gamma mapping techniques would be possible, for example inside a TV, but these would be blind, i.e. they would not adapt to the details of HDR images and would therefore be very suboptimal.

US-20150245004 teaches that two images may be mixed, of which at least one may be an HDR image and the other may be an SDR image. The application describes that there may be a mismatch in color space (i.e. the width of the RGB triangle, because other additive primaries are used in a DCI P3 color definition than in a usual sRGB video definition, but that chromatic aspect is irrelevant for the present explanation of the application, since it may be chosen in addition to what is of primary essence here, namely the luminances of the pixels of the image and the corresponding dynamic range of the image coding), and a mismatch in dynamic range. However, that application is not very detailed about how the video mixing should or could occur. [0029] mentions that the mixing may occur in the color space of the display device, i.e. the dynamic range which in the designation of this application ends in PB_D. Furthermore, the processing appears to be related to conversion to a different transfer function TF, or gamma correction [0027]. Finally, there is a very generic teaching [0036] that the compositing set can evaluate the dynamic range of the video content, and presumably that corresponds to the non-detailed teaching that statistical analysis of the input images can be performed [0028]. The person skilled in the art would interpret this in light of existing knowledge. In fact, there has been a teaching in SDR video optimization, which analyzes the statistical properties of the input video, namely the endpoints of the luminance histogram, and optimizes that particular histogram for the available display range. For example, due to various reasons it's not uncommon for black or dark pixels in a captured image to be brighter than they could (or perhaps should) be, say a luma of 30 instead of 0 (again ignoring for a moment complex details about overflow protection intervals, display light leaks, lighting-dependent visibility issues, etc.). One could then stretch the histogram so that the lowest image luma is 0 instead of 30, resulting in nicer looking images due to increased contrast of image structures and better blacks, which can be especially nice for dark surround viewing.

Actually, when one needs to prima facie design a video mixer, without having invented more taught details, the expert might have the following two options to consider as explained with Figure 13 (apart from mixing luma codes, which as explained with Figure 1 is, although nice and simple for pure mixing of SDR images, not so obvious or smart when you have differently defined images). If a teaching says you need to convert due to different transfer functions, the expert would think of a natural design mixing system as in Figure 13A. In this mix you have a range of natural luminances (1301), for example, up to 50,000 nit to be large enough and in fact, whatever the OETF forms of code definitions, it converts them to luminances. Both images contained luminances (physically uniquely defined), of course, they could be mixed in principle, but the question is whether that would be enough, given practical psychovisual and artistic considerations. Secondly, if we map to a common preset display dynamic range (1302), with a PB_D of say 700 nit [so for simplicity we now skip the OETF stage and reason purely about the corresponding luminances], then another natural mapping (the relative mapping paradigm for example) that one could consider is to scale (based on "statistical properties of images") the images towards the display's capacity (i.e. the image's dynamic range towards the display's dynamic range). The natural way to do this is to stretch the ranges (i.e. corresponding to a white-to-white mapping), for example linearly. Some mapping function in the form of gamma could also be applied, but the question then becomes what that gamma coefficient should be and why. But this creates results that are already exactly what in several cases we would like to avoid. If you stretch, for example, an SDR image of a snowy landscape to 700 nits, it can become annoyingly bright.

Furthermore, while these natural mapping systems are no longer perfect just because of considerations about their endpoints, like where the brightest color should be mapped, such techniques completely ignore what, as said, is probably the most important aspect of HDR, namely the (precisely controlled) distribution of all objects luminances between the endpoints of the dynamic range (and at present that still seems to be a lack of understanding by many, if not most experts, who seem to insist that the hDr is simply about the white luminance value divided by black that any technology can achieve, even though often an image that is correctly rendered on a lower dynamic range display can look better than a poorly controlled image on a higher DR display, e.g. contrast and HDR effect). That dynamic range is not just about technical endpoints can be easily understood by focusing on the example of having snow on an HDR master rating of PB_C=5000 nit. The grader, for example, would assign the brightest part of the snow to the sun at 800 nits, while another grader may put it at only 550 nits, so essentially this looks like a lower dynamic range situation since most of the codec's capacity is not being used for this image. In SDR leveling, one can assume that typically the grader will put the brightest pixel of the snow at 100 nits. For MDR graded images, one can give some thought to what luminance the brightest pixel should fall at, even when only one image is being rendered rather than a possibly very complicated mix of multiple image contents. So how do you know what final (blend) luminance a snow pixel should ultimately be rendered at, whether it comes from an SDR image, where it is relatively encoded at most 1.0, or from "some" HDR image encoding, where, even ignoring the OETF shape because that may not be the most important determining factor, the snow pixel has a somewhat arbitrary-looking relative luminance at first glance of, say, 0.126.

Figure 14 makes clear another important point that is overlooked by procedures that simply focus on the dynamic range of any technical system, be it some encoding corresponding to a theoretical reference display or a real display, etc., as something with an extent defined primarily by the endpoint, such as, for example, how blind luminance transformations would work for gamut mapping. Especially in HDR, the illumination of objects does not need to be strictly controlled as, for example, in an SDR news studio, but can in principle be anything. If one walks in the sun past an empty spot in the forest, for example a grassy area, one can visually perceive that the interior of the forest may already appear black. That is of course not at all because it is black, because the trees there are a brown color like everywhere else, and the undergrowth is just as green, but because the local illumination is, for example, 1/20°, or even less, that the local illumination is the free spot without trees, which receives the full light from the sun and the sky. And 1/20th starts to look to humans like psychovisual black, until you take a closer look.

An object like a t-shirt can therefore have different luminances, even on the same film (in other words, it could fall all over the place in the range of, say, 1000 nit PB_C_HDR; note that a grader might decide to restrict the variation in their master grade a bit, but we account for that with a natural almost 1-to-1 relationship with the relative luminances captured by the camera from the original scene). In the sun, the t-shirt will be -5x brighter than in an initial half-shade, where the t-shirt is mostly illuminated by, a considerable solid angle of the blue sky. In fact, 100 nit would also be artistically what one might expect for a "dull day" on a good HDR display, while 500 nit might appear as if it were actually illuminated by the sun. The further the person goes into the forest, the more area of the sky that could illuminate him is blocked by the leaves of the trees, so the luminance of the particular object, in this case the person's t-shirt, would be, say, 100 nit, ergo 50x darker than the bright t-shirt, ergo it would probably also look somewhat blackish in a rendering of this particular HDR scene. Perhaps in a cinematic HDR grading for home viewing, the luminances can be graded more conservatively, say half the given values, but the distribution of the various luminances can still be the same point. So a question now would be, if one with any of the above relatively static blending forms were to interleave these various renderings of t-shirts into a running video (with their own luminance details), would that always "automatically" do reasonably well, and if not, what should be done next?

Another interesting previous technique is the following:

US2015/0042890 teaches video equalization including performing equalization such that a sequence of images has a dynamic range that is constant to a predetermined degree, where the input video includes both standard and high dynamic range videos and images. The equalization is performed with a common anchor point (e.g., 20% gray level or log mean luminance). US2013/0328907 teaches a method of transforming image data for display on a target display. A sigmoid transfer function provides a free parameter that controls the mid-tone contrast. The transfer function may be dynamically adjusted to accommodate changing ambient lighting conditions. The transformation may be selected to automatically adapt the image data for display on a target display in a manner that substantially preserves the creative intent embodied in the image data.

WO2014/130213 teaches systems and methods for superimposing a second image/video data onto a first image/video data described herein. The first image/video data may be intended to be displayed on a display with certain characteristics, for example, HDR, EDR, VDR or UHD capabilities. The second image/video data may comprise graphics, subtitles, text, advertising or any data that is desired to be superimposed and/or composited onto the first image/video data. The second image/video data may be appearance mapped according to image statistics and/or characteristics of the first image/video data. Furthermore, such appearance mapping may be done according to characteristics of the display that the composite data is to be rendered. It is desired that such appearance mapping render composite data that is visually pleasing to a viewer, rendered on a desired display. US2015/256860 teaches a method for fusing high dynamic range video data and graphics. At a video receiver, a display management method uses metadata to map input video data from a first dynamic range into the dynamic range of available graphics data. The remapped video signal is mixed with the graphics data to generate a composite video signal. To alleviate perceptual tone mapping jumps during video scene changes, a metadata transformer transforms the metadata into transforms so that at a television (TV) receiver the metadata values pass seamlessly between consecutive scenes.

SUMMARY OF THE INVENTION

The above problem of needing a method to reasonably coordinate the combining of images with (potentially very) different dynamic range can be solved by having an apparatus (301) for combining two images or two videos of images (Im_HDR, Im_LDR) from different sources and with different maximum luminance, one of them being a high dynamic range image or video, the apparatus comprising:

- a dynamic range setting assembly (302) arranged to set a dynamic range of combination luminance (CombRng), which is characterized by at least one maximum luminance (LMC) that is determined based on the maximum luminances of the two images or the two image videos,

the dynamic range setting assembly further comprising a luminance anchor determining assembly (303), arranged to determine an anchor luminance (anc) in the dynamic range of combination luminance (CombRng);

- a colour transformation assembly (310), arranged to perform at least one luminance transformation on at least one of the two images or videos, wherein the colour transformation assembly (310) comprises a source anchor luminance readout assembly (311) arranged to read at least one source anchor luminance (L_SA1) from metadata of a first source (350) providing a first image or video (Im1_LDR) of the two images or videos,

and where the color transformation set is arranged to establish a luminance transformation (FF_1) to be applied to the first image or video that produces a first output image having first output luminances, which luminance transformation depends on the value of the source anchor luminance (L_SA1) by having the property that the source anchor luminance (L_SA1) is mapped to an output luminance in the vicinity of the anchor luminance (anc); and

- an image combining assembly (320) arranged to combine output luminances of the first output image with luminances of the other of the two images or two image videos to form at least one combined output image (Im_o).

First of all, for the avoidance of doubt, by image blending we mean various possible options for bringing together spatially or, in case of video, possibly also temporally, image content. Blending should be understood in the original broad Latin sense as "joining two together", and not in the more limited sense of, for example, blending, which we reserve in this text for the weighted addition of two sources per pixel. The skilled artisan understands that the various embodiments of this invention address the need for good blending (harmonious luminance or coordinate matching) of video which occurs, inter alia, in various picture-in-picture combinations (and in general today this includes spatially interleaving at least some pixels of the first image with part of the second image, e.g., by a temporally changing amount corresponding to flipping the first image as if it were a page and displaying pixels of the underlying image, but this invention will ensure that the colors of the pixels, and in particular their perceived brightnesses, will correspond correctly), blending combinations such as a temporal fade or interleaving such as, for example, a cut to a commercial at regular times during a movie, etc. Whether when replacing a spatial region of one image by (all or part of) a second image, or when temporarily displaying images from another video sequence between or after images from a first video, one should not have too much brightness mismatch (i.e. one of the images should not appear excessively brighter than the other, e.g. with respect to its main histogram lobe or key objects), especially when there is also flashing content, etc. In any such scenario, one can set a combining luminance dynamic range (CombRng), which is the dynamic range of the combined image (empty representation), to be filled with combined image data (e.g. the larger of the two ranges could be taken for interleaving or time-sequence).

It should be clear to the skilled person what a high dynamic range image is and why, namely an image which is not a legacy LDR image defined for a peak brightness of 100 nits, i.e. which cannot be combined in a classical LDR manner (at first glance there seems to be some mismatch between the image object or pixel luminances defined in the two different image coding frames). It should be clear to the skilled reader that if the apparatus outputs only 2 (or for example 5) still images, then typically the output image (Im_o) will be a still image. In case two or more videos of successive images are combined, an output video will be generated, i.e. a succession of those images in some form, for example, in case each video has one image for each time instant, the resulting video may be an alpha-blended mixed video with exactly the same number of images for corresponding time instants as the two original image sequences, or the resulting video may have images for different time instants, for example having a length of 100% of the first video, 50% of the second video, with its various images assigned in the mix to some of the time instants.

In order to keep the teachings of this patent readable and concise, we will pretend that the embodiments only make transformations in luminances (the reader may, when reading, pretend that the images are grayscale images), as the teachings are mainly about how to handle differences in dynamic range of luminance and the brightness impressions this would give to a viewer. However, of course, the skilled reader understands that in reality color transformations will typically be performed, for example, he or she can understand both how a luminance corresponds to a linear R,G,B color definition, and how a pixel with a desired luminance can be created by transforming its color, for example, into linear RGB as in WO2014/056679. In case for the sake of conciseness we say below something for video, the skilled person will understand that it can also be applied to still images. In some cases, video may, due to real-time aspects, desire simpler ways of handling, so elegant quick solutions for mixing are presented. Some of our best performing implementations will work with human artists, as humans are better able to judge what exactly HDR images are about and what they need in various mixed presentations, but other implementations will work with automatic annotation using image analysis algorithms.

The apparatus must be able to establish a more suitable blending luminance dynamic range, which is a new dynamic range that is part of the reason why a very complex blending problem becomes a more manageable one. As will be explained below, this need not simply be one of the dynamic ranges that already exist, i.e. where it has already been invented and defined, namely neither the dynamic range of any of the source images necessarily, nor the dynamic range of the final display on which the content is to be rendered (if even that is known during blending, which is not true for all embodiments), the intended display (which may not be known in some embodiments, although some apparatus on the creation side may have a reasonable estimate of what may be a good reference display, which is a typical representative of the typical display expected in the field, in consumer installations). That blending luminance dynamic range must be such that the blended presentation has the best visual quality or impact on the viewer. That may depend, of course, on various aspects of the content to be defined. If the main content (which we will call the second image or video, because that makes the statement easier to read) is for example a movie, then the movie creator would not like his movie to be disturbed too much. For example, it should not be so that after a bright commercial, the viewer can no longer see well what is happening in a dark scene of the movie. In such a scenario, as there will usually be one of the two videos which is distorted in its luminance properties to make it more harmonious with the other input video and thus the combined video, the movie is the most important (or main) video, for which the luminances should be left largely undistorted (where possible). But also the commercial producer or sponsor would like his commercial to come out in the final rendering as well as possible. The color transformation set (310) and/or the combination set (320) can enforce a policy that the commercial cannot overtake the movie. Still, e.g., an LDR commercial should not be unnecessarily darkened either. For example, a face may not look very pretty if, when compared to another related object in the movie, it looks almost black. As we said above, the forest may look black compared to very bright sunlit regions, if the human visual system adapts its interpretation of gray values in that bright area. So, if there are important (large) areas in the HDR movie or stream (or game, etc.) that are very bright, say 2000 nit, and especially when they are closely adjacent to the matched SDR pixels, then an SDR object of only about 20 nit might look blackish, so some brightening may be desired before matching. Also, the visibility of, for example, facial features or other object details is reduced with the average luminance and contrast of the object. Although we experience that even with single images some people may complain that in some situations the object brightness distribution (e.g. pixel luminance histogram of an HDR rendering) is not completely optimal, e.g. too bright in some scenario, visual adaptation and the human brain are usually relatively forgiving if a single image is rendered, as the brain can determine and adjust to what all objects in that single image should be. But if an object is now spatially composited into another image, the viewer is bound to notice differences as serious, even if some content is in one frame. Because the brain judges part of the images content as a reference for the scene brightness distribution, and usually scenes are not as disconnected as a PIP, but have a smooth lighting variation. As will be shown, it is not so easy at first glance to tell what a good brightness combination is. At least, that can't be done during the creation of the separate videos, since, for example, the movie creator doesn't know which local commercial will be inserted (say, by a cable operator), or even, which second image the viewer will select from some image source in, say, a PIP while watching the movie (perhaps streamed by his home computer, or central home content management appliance). Also note that the visual importance of some say the relatively bright object depends on, among other things, its size. By having both images available, the appliance can make smarter blending decisions. It may need to reduce the maximum renderable luminance (MRL), for example, to make the HDR content look more like the first image, if it's an SDR image. That may depend on looking at other pixel properties or object luminances (where an object is an intelligently grouped set of semantically related pixels). For example, if the brightest objects are lamps, one can allow reducing the brightness of those lamps in the combining dynamic range (CombRng) for some time. That is not the most spectacular HDR, but the HDR image can still be used to render very bright lamps, and especially for connected displays with a lower maximum brightness (LM_MDR) than that of the combined image (LMC), the brightest encodeable lamps cannot be rendered exactly, i.e. with the codec luminance, anyway. So one still has a high dynamic range movie, but then with a dynamic range that is more appropriate for the situation of a combination.

In fact, one can think of two typical alternative scenarios. Generically, one can state that the dynamic range of the combination, i.e. at least its upper luminance (LMC), is defined by the device analyzing the needs of, on the one hand, the input of at least two images and, on the other hand, if something regarding that can already be determined, the dynamic range (at least the maximum brightness) of the intended display on which the combination of images is to be represented (i.e. the HDR needs of the properly presented image content of a typical HDR display). Therefore, some embodiments will need to determine the CombRng based on the two images alone, if nothing or not much can be said yet about how they should be adjusted for use (but in some embodiments there may already be an adjustment towards some expected typical use: for example, if the master HDR images of e.g. PB_C=5000 nit will be displayed primarily on lower dynamic range displays, and are then expected to be considerably interleaved with SDR image material, one may consider already creating, e.g. for video compression for communication, the combined or combinable images (on the receiving side, meaning some image information may still need to be luminance transformed) in a lower dynamic range than the PB_C=5000 nit, e.g. 1.5x the expected maximum brightness of the consumer side display of e.g. 1000 nit, ergo the combining dynamic range will then have a PB_C of e.g. (1500 nit). A typical first example will then have an HDR movie, i.e. with the largest dynamic range of the two successive frame input videos, and the SDR content needs to be luminance transformed, or at least appropriately annotated by putting its one or more source anchor values into co-communicated metadata, so that it reasonably harmonizes (or can be harmonized at the final presentation site) with the HDR movie, being the main content. However, an alternative typical scenario may already pre-transform the HDR content to a lower dynamic range, which is more in line with the intended typical display rendering and/or SDR content insertion. If one has a real display for rendering of say 750 nit PB_D, such a system can for example transform (or annotate with anchor luminances to determine the luminance transformations) the luminances of both videos (which can happen at totally different sites and production times - that's the power of consistent anchoring) to make them harmonious into a DR blend of say 1500 or 2000 nit PB_C, or maybe even 1000 nit PB_C. This blended or blendable content can then be easily degraded to the defined display dynamic range PB_D = 750 nit needed.

Especially since the applicant designed a framework where the (various) ultimately (reclassified) different dynamic range appearance images can be communicated not actually as real coded images (i.e. with the final pixel colors fixedly defined in the color components of typically DCT-compressed images), but as functions for computing the various possible pixel luminances of such a secondary image from any primary image, those functions can be redefined and thus more complex definitions can be communicated (e.g. specifications of how image luminances should be computed for at least some special image objects in various situations, e.g. various combination dynamic ranges). Classical image or video coding only encodes and communicates "the" image to be used, i.e. with its pixel luminances as they should (!) be rendered, but the applicant may convey a completely different image (i.e. with the necessary spatial structure and texture of the objects, but not the correct colorimetry, in particular not yet the correct luminances of the objects or pixels in the image), and then a prescription of a procedure how to finally create the correct luminances for the final image, which in this application will be a combined image, defined with all its pixel luminances in the combined luminance dynamic range CombRng.

Similarly, there may be reasons to for example increase the minimum (LmiC) of the combining interval compared to the minimum (LmiH) of one of the images which is a dynamic range image (with dynamic range Im2_Rng) - for example because with combining a relatively bright LDR image (e.g. if it has not been darkened very precisely) the overall (or local for at least some dark parts of the HDR image) brightness of the combined output image is such that the darkest parts, which come exclusively from the HDR image, cannot be seen very well. I.e. by transforming them into a CombRng with brighter blacks they can be adequately brightened (that will happen due to the choice of the combining interval, and also there is no need to perform any clever processing of the HDR image per se, i.e. it will be adequately brightened even by a simple color transformation to the CombRng).

Thus, the reader can understand how at various points in the image handling chain, for example still some content creation site, where the final content is defined, by devices typically under the control of content providers, before entering the image consumption part of the image chain, or some image consumption site (e.g. mixing of two types of content solely under the responsibility of the end user, e.g. by the end user using his remote control or computer control), a suitable dynamic blending range can be set, either by a human, potentially semi-automatically or automatically. In general, such an apparatus will look at both the content of the image (i.e. the HDR quality, i.e. what PB_Cs the various contents have, but also the distribution of the various luminances of objects or pixels in that dynamic range ending at PB_C, and what kind of bright objects [e.g. HDR effects] exist, and whether they can be easily distorted by luminance without creating very objectionable modifications in appearance), as well as the needs of the end use of these images, i.e. typically the display rendering (if, for example, the images are to be supplied to an installed base of SDR display consumers, then a lower CombRng may be in order). When done automatically, an algorithm will use heuristic models that mimic what human combiners (i.e. those doing the combined images and/or the metadata for that, e.g. specifying the LMC value and at least one anchor point, which will be a source anchor point of at least one of the two images or videos to be ultimately combined) will use as a methodology. On the dark side of CombRng, a judgement will be made as to which darker colours will still be relatively visible, modelling such visibility based on aspects such as e.g. glare from brighter areas of brighter image objects, expected screen leakage light, and ambient light-dependent masking of darker colours due to typical reflections from the display faceplate, etc. The new element in this model compared to the single screen and the typical expected viewing environment (e.g. home theatre, dimmed lights) is the introduction of secondary image content. But in general, since this setting of the CombRng usually comes first (before determination of a suitable anchor), this estimate can be rough (e.g. not checking precisely where the SDR content is introduced and whether the brightest part of the SDR image, even potentially illuminated, falls geometrically next to a dark part of the film, hence potentially considerable masking, although some embodiments might take all that into account also when setting the upper and lower luminance of the CombRng, e.g. in a direct or iterative determination, the latter by iteratively checking what a good anchor luminance would be, and then correspondingly suitable CombRng limit luminances).

Since in several applications some approximate blending may already be suitable (e.g. if the movie viewer wants to ultimately enjoy his movie, he should not interrupt it or blend it with any additional images, but if he does, he should recognize that there will always be some distortion in the colorimetry of that main video content, even if it is only psychovisually in the appreciation of the total image impression of the blended picture, and not actually a change of the movie luminances, compared to how they were before blending), it may already be suitable to quickly calculate some heuristic lower luminance LmiC, based on the typical content. For example, if the histogram shows that most of the SDR content falls between 10 nit and 100 nit, with about half of the combined image display area occupied by SDR pixels with luminance higher than 50 nit, the set may determine that it does not need deeper blacks than, say, 0.01 or even 0.1 nit for that combination (because the typical display heuristic algorithm judges that darker image structures will not look good under such conditions). What makes this setting of a blending range so interesting is that (although the creator of the original HDR movie content may have made a movie with ultra-blacks as sharp as 0.0001 nit, for the case where the movie is enjoyed in a dark room on, say, an OLED display that can render very deep blacks), the blending apparatus (e.g. an STB, or even the TV itself), can now decide to lighten the darker colors of the HDR movie a bit, by determining an appropriate luminance mapping conditioned between the HDR movie (or more generally the main content) and the blending dynamic range, and in particular the lower part of those dynamic ranges, before putting the pixels of the HDR movie into the blended image. On the bright side of the luminance range, algorithms will typically involve determining how much contrast cropping or compression for an original HDR image is still appropriate (which will be different for example for lamps that don't need to have any internal structure rendered, compared to sunlit clouds, where ideally one would have enough contrast in the final rendered image and thus any image encoding for that [so that ideally determining the final image is reasonably simple based on that encoded received image which in its encoded luminances already mostly specifies what the final rendered luminances of all pixels should be], ergo less compression for the brighter image colours, by defining those bright areas with a larger subset of luma codes either in the merged image or in the pre-merge-defined HDR image before having actually transformed it into the merged image pixels). So depending on the needs, for example an inserted commercial (for the expected movie maker e.g. when he knows on a particular channel his movie, so this potentially reclassified definition of motion view will be offered together with the commercials) which should not appear annoyingly dark, it may be necessary to squeeze the clouds into a somewhat smaller region of bright luminances not far above the brightest luminances of the SDR image, at least when that SDR image is transformed for blending. In order to be able to further adjust this image in the most appropriate way (and generically, i.e. at least some part of the SDR image and some important part or object of the HDR image typically) and for various inserted e.g. SDR commercials, ideally the HDR image specifies a number of important luminances for those clouds, e.g. 3, between which the blending apparatus can shift, typically dimming, the various cloud areas (thus also reducing contrasts of e.g. a dark grey part of a thunder cloud). But this is where anchor luminances will come into play as taught below, with the present part of the teaching explaining that the combining interval should of course not end up with such a low LMC value that it starts to destroy the look of the main image (for example if the movie is about storm clouds, which typically have different greys if some parts of the clouds are strongly illuminated, e.g. the sunlit edges of the clouds, and the bellies of the clouds are not illuminated by the sun but only by ambient light, so they can be much darker, then a central aspect of this movie will be lost if the CombRng is such that these clouds have to be compressed into an upper sub-interval of the CombRng with only e.g. 10 lumas, because even with a strong functional luminance stretching as an attempt at correction by a display, this can never give a good HDR representation of those clouds, as they should deserve). But otherwise, the blending apparatus may decide to downgrade the brightness and contrast quality of the HDR image, in order to harmonize the HDR image (especially when dealing with less demanding HDR scenes, such as, for example, a football match with some content in the sun and some in the shade) with typical or real images of lower dynamic range with which it will be blended.

Of course, although a dynamic range (at least a maximum, and possibly also a precisely specified non-zero minimum) is a very important (initial) determining property for the final look of an image, but this still does not fully finalize how object or pixel brightnesses will be located within that range. Simple image-handling procedures will pretend that all that is needed is an upper and lower bound of some dynamic range, and then do some mapping (which is often not much smarter than mere linear compression, i.e. mapping white input content to the white final range, and black input content to the black final range). And that can even vary considerably too, especially when one starts using the full potential of large dynamic ranges, with very critical scenes, both in terms of the luminances needed to be representable, for example, simultaneously many deep blacks in a cave, and many sunlit colors seen through a small crack outside, and even objects that need critical control of their intra-object contrast, such as a condemned person through fog. But as said above, the inventor thinks that a good control of the HDR image appearance, and especially the appearance of the combined images, is also important for a good control of all or at least many, or at least the most important, luminances between the CombRng boundaries, so something else is also needed to have good HDR image handling systems, in particular such apparatuses that can, without more difficult human intervention, create finally appropriate looking combined images, for any situation that may occur in practice (since you can't sell the same Hollywood director on every STB or TV, to determine the appropriate final combined luminances for the viewer). A fast mechanism is needed, that can still be well adapted by content creators, depending on how critical or complex their needs are, in order to obtain a reasonably good quality result in at least most practical cases of image content combining.

So, the second component to sufficiently control the blending, is that the apparatus determines an anchor luminance (anc). That can be several things, but it is usually a semantically relevant luminance, of a semantically important object. For example, it can be a particularly typical important object lit in a particular way, which gives luminances in a sub-interval of the total HDR image range. It will be a luminance that can be expected to be good, in some blending interval, to determine other luminances around it (i.e. this luminance determines whether the totality of luminances are of appropriate brightness, neither too dark nor too bright). In other words, if the anchor luminance is correctly represented, whatever is correct for a particular blending interval, then the other luminances will not be bad either (and with Figure 16 we will illustrate how this can be controlled with more technical components to be better and better, according to the desire of the creating artist).

Figure 15 shows a typical HDR scene, with two regions of considerably different illumination, namely a dimly lit barn (Ins) (see scene geometry in Figure 15a) and a sunny outdoor region (Outs) with considerably brighter illumination. For a content grader to make a suitable-looking HDR image for typical TV viewing (dim environment) from the physical scene luminances, it may be sensible to grade the indoor luminances as in an SDR image, i.e. down to 100 nits for the brightest indoor object. Real-world outdoor luminances would be roughly 100 times brighter, up to 10,000 nits, but that would be experienced as too bright for TV rendering (even if the viewer typically had a high-end 10,000-nit PB_D display). Therefore, the grader may, for example, choose to render this scene with the brightest HDR outdoor luminances (in the outdoor pixel luminance subrange SDROU) of the master HDR range (DRH) of up to 1200 nit for the brightest object (except perhaps some small specular reflections on metal), and with a typical average outdoor luminance of 250 nit.

The power of having an anchor luminance is illustrated in Figure 15C. In this example we have simplified it and assumed that the darkest luminances, from the SDRIN internal pixel luminance sub-interval, can be represented with equal luminances in all situations (they will form a stable and unchanged set of luminances in the blending interval, which of course is not always the case; for example, if the SDR commercial contains many bright colors, the apparatus can use the lower anchor point AncS_ins characterizing the internal pixels of the HDR master source image material, to raise the darkest pixels a little bit, but we will clarify the basic principles of our realizations with a scenario where only one anchor point will be used to correctly position the luminance of the pixel luminances of at least this first HDR master image, in the blending interval (i.e. with the second anchor point AncS_outs of the bright outdoor pixel set). First of all, although any luminance of an interesting object can be used, for example with a typical light reflection spectrum leading to a brighter appearance, the lower anchor point AncS_ins characterizing the internal pixels of the SDR master image, can be used to raise the darker pixels. For a typical gray value, it is usually not a bad idea to use some average brightness anchor position in a sub-lobe of the histogram of all image luminances corresponding to a reasonably similar illumination region, when one wishes to determine the luminances of pixels around such a value. The reader can imagine that below with Figure 16, because if for some reason the apparatus wishes to "re-light" such a region to, say, more light, then the luminances around it can be continuously illuminated by the anchor point, just as real reflective objects would become brighter if the illumination falling on them were increased. Note that for automatic determinations when calculating a representative ANC value, care must be taken not to cause, for example, very bright sunlit edge luminances of clouds to deviate too much from a reasonable average result, so typically if humans can determine and communicate an anchor luminance along with the encoded images, that would be able to produce the best results. Thus, the reader should understand that there are two types of ANC value, namely firstly the result (i.e. a position in the blending range where the luminances of images that correspond to the same semantic meaning of the ANC definition, i.e. are close to said ANC luminance value, but in the dynamic range of the source (differently illuminated and differently encoded), should ultimately fall when rendered or written to the blended image), and secondly a (corresponding) ANCS value for all or at least most of the input images (in case some images are missing an annotated ANC, the apparatus should estimate some value, e.g. an SDR value which should not be too unreasonable for most typical SDR content). If such an anc value is defined (even one that is semantically different from the blending one, but can be related to the blending one, for example by multiplying it by a factor of 4), then the input content can be harmonized with the blending frame and thus with all other image content.

The reader can see how the ANC value (i.e. the ANC value resulting from the image blending and the range being prepared before the actual blending of the two images) is coordinated with the determined blending luminance dynamic range, i.e. typically at least its maximum brightness, e.g. for a top quality HDR blend PB_CS=2000 nit. If the dynamic range setting set (302) chose the top quality blending range (CmbRngSup), we have available a high quality dynamic range which is close to the dynamic range of the master HDR content. It therefore makes sense to also position the resulting anc value (of the semantic kind: "position where on average bright, second sublobe and outer colours should fall in the blended image") at 250 nit (where the apparatus follows the typical rules for the rendering of bright regions in TV films, which was also the basis that led to the selection of 250 nit (approximately) by the master grader for the master HDR image). This blending situation will lead to most of the relevant HDR luminances, also from outer pixels, being represented well as intended by the content creator, i.e. with the same luminances as are encoded in the master input image, and only for example specular reflections need to be toned down a bit to fall below 2000 nit. This is an example of the first type of scenario above: the blending range closely follows the original master HDR video content, which is leading and represented substantially without luminance distortions, and then only the SDR content needs to be harmonized (which will be done by looking at the corresponding 1 or more ANC values for the SDR image).

For a lower quality blending interval (Com-RngInf), the apparatus can already take into account necessary quality reductions, also for the HDR master image. Here, the apparatus may need to reduce the position of the Anc_Outs value, which it can for example typically do with a heuristic estimation of inter- and intra-regional contrast needs (even without looking at a particular image and scene, some global values for good anchor luminance positions can already be determined, which should work consistently for the next different HDR shots of the film). The resulting second position Anc_Outs2, depending on how much of the span remains above 100 nit or in other words above the lowest luminances of SDRIN (e.g. in stops, 10x 3 stops being to position all HDR effects, whatever they may be, i.e. reflecting objects in bright sunny lighting, but also lamps or specular highlights, and typically also taking into account the values that customers would like to see for good quality HDR images, i.e. no large object areas that are too bright, nor too dim) at a luminance position in the CombRng such that the bright region or parts of it have sufficient contrast above 100 nit (to have a really sunny look for example, a factor 5-10 can be a good value, and this can be formulated as the division of the two anc values), however there is also enough room for even brighter HDR effects, such as small specular reflection areas in the bright sunny lighting. objects, flames or explosions, or laser beams, and whatever the movie may actually contain to be graded and rendered near that 1000 nit PB_C_Inf upper limit. Some automatic procedural embodiments may work by calculating error estimates. For example, textures in regions may be analyzed, with complexity indicating the need for greater or lesser intraregional contrast.

Thus, if only the two images are known, for example the movie the filmmaker is creating (annotating with at least one anc value, and maybe already the luminance mapping to a typically suitable CombRng) and a typical average variant of the expected SDR content, then the apparatus can calculate at least one error for the luminance distortion compression of the HDR image, and the distortion or disharmony of the SDR image, in its rebrightening before the combination, as illustrated by Figure 17. Such distortion errors can be calculated both for typical average scenarios, for example with a probability of pixel luminances for one or more typical HDR scenes, evaluating how such a scene would need to be distorted (mainly in the brightest areas of the image), when a downgrade to a CombRng with PB_col, PB_Co2, etc. has to occur, and/or for real images, i.e. taking into account the remaining pain if a real (optimal) luminance mapping is applied to map all the images to the corresponding image. object or region luminances of the native image representation and their dynamic range, to the recolored image in the blending range. In Figure 17 we see schematically clarified how an E_cmp error can be associated with the compression of at least one part or object of the image, for example, cloud structures. Various practical implementations that mimic the behavior of the human grader can, for example, extract a texture, and by computing texture measures and other spatial and/or statistical measures of the local region, estimate how problematic a downgrade to a smaller sub-range would be. For example, for clouds, the texture analyzer would find that there are no sharp boundaries, and that cloud appreciation would occur mainly by the (smooth) distribution of the (many) gray values, and not by typical aspects of the geometric shape (e.g., the linear pattern of a kilt texture would allow more compression or posterization in fewer luma codes). That is, if the many fine smoothly graded distributed grey values are mapped to a more limited set of incorrect brightnesses, some error would quickly be expected to occur, which would deter the apparatus from doing this to too large an extent, particularly if the main picture is of critical colorimetric importance, because for example the consumer should be able to enjoy the fine graded artistic colour quality of the film, which should not be reduced to the often extreme and garish rough grading of for example some commercials (in addition to the main concept of providing anchor luminances, some embodiments will allow the content creator to indicate with additional metadata how much distortion he would allow between or around the anchors, for example that the interval between AncS_outs/10 and AncS_outs*10 should preferably not be compressed or stretched by a factor of for example 2 or 3).

Figure 17 shows a more concretely illuminating example of how errors can be formulated to arrive at luminance harmonization in a realization based on such error equations. The HDR barn image and an SDR commercial for a soft drink are to be mixed, and in addition to their ranges we show histograms of the luminance distributions of the scene images (hist_HDR respectively the histogram of the low dynamic range commercial image hist_SDR, with in the horizontal direction the counts N(Li) of the number of pixels in the total number of pixels in the image having a luminance equal to some Li value). Compression of the HDR image into the smallest upper subrange of the CombRng down to PB_comb, e.g. being 1000 nit, leads to an error E_cmp. This error can be balanced by, for example, an error for brightening the SDR content, or an error corresponding to the incompatibility of this CombRng with typical expected display ranges. For example, if everyone in the world is considered to have a 1000 nit PB_D display anyway, this E-cmp error can be normalized to zero, because that modification of the master content would have to happen anyway. But still, a CombRng of, say, 1500 nit can be better, i.e. have a negative error, indicating that it is of higher quality than a 1000 nit combi, if the visual quality of the film content is heavily weighted. Apart from the errors that can be calculated to bias the brightnesses of the SDR content, which may be less important (because on the one hand, in an SDR paradigm, viewers are assumed to be able to quickly adapt to different brightnesses, but on the other hand, the brightness of the potentially "incorrectly" rendered SDR image, e.g. judged by many expert or non-expert viewers as too bright when rendered on its own on an HDR monitor with the brightest pixel of, e.g. 1000 nits, for this particular image, which is now judged not by itself but in relation to the master HDR image), the following error can be assessed. An SDR luminance mismatch error (E_MM) can be determined, and this can be done for example as simple as the difference between the Anc_HDR value of the CombRng, i.e. also where the source image value and the HDR image value (AncS_outs) are ideally mapped to, and the actual mapping to luminance (D_AncSDR) of the SDR source anchor luminance AncB_SDR. That is, the best harmonization in this example according to the heuristic would occur if D_AncSDR is equal to Anc_HDR. The reason this works is because this type of HDR image contains a sufficiently large area of bright outer pixels (which is why your content creator defined a higher anchor luminance, AncS_outs). Since the viewer when viewing this HDR movie scene is already accustomed to brighter luminances, the introduction of approximately equally bright pixels for the SDR commercial (e.g. PIP) is much less objectionable than when the user was only viewing dark luminances, e.g. 5 minutes in a cave or basement (in which case such a bright commercial introduction would likely scare the viewer, or at least significantly spoil his enjoyment of the rest of the movie scene, at least colorimetrically). This system works especially well if the creator of the SDR content uses a particular type of its anchoring, i.e. it can be so simple that it uses only two types: above average brightness and below average brightness. Regardless of how the luminances are actually distributed in the range of 0.1-100 nits of SDR luminances, the creator can indicate what type of SDR image it is and thus how it should be represented, in particular according to this invention, how it should be harmonized with various possible sub-ranges of an HDR image. For example, the SDR image can be the normal content of a dull or dark scene ("below average brightness"), or it can be lumas that are supposed to represent a sunny scene. We assume that the creator of the commercial wants it to be shown as a "sunny" scene, with colors to make it look bright and vivid - what exactly bright and vivid is in a combination, of course also depending on what the colors of the other image are.

The reader can already understand the power of anchor luminance in coordinating the images to be combined. Even in such a simple system, and assuming that the commercial content creator has an interest in his image becoming “bright enough” (but probably never has an interest in annoying the viewer with representations of his part of the total image content that are too bright), but of course there is also a (more, equal or maybe in some scenarios less) important image quality desideratum of the HDR movie creator as well, the combination can already work reasonably right out of the box in case the HDR movie only has “darker” content in case the HDR content creator only includes a lower AncS_ins. Then the apparatus understands that at least this scene in the movie has no (too many, important) bright regions and thus the viewer will adapt to darker content, for example in this simple non-limiting lightening example only in the SDRIN of the SDR sub-interval. Thus the apparatus can make a harmonious blend if in that situation it maps the AncB_SDR (although it is assumed to be "bright SDR content") to the lower Anc_ins, or in their vicinity. By proximity we mean a fixed in-apparatus or typically customizable multiplicative fraction of the anc value above or below the anc value, e.g. between 1/3 or 1/2 and 2 or 3 times respectively. The extent of proximity can be determined (either at runtime by the blending apparatus, or on an authoring side, e.g. with explicitly communicated proximity luminance limits) based on various embodiments of aspects such as e.g. allowed luminance errors of objects when mapped, expected type difference or variability of anchors, etc. But clearly, if AncB_SDR is for example 20% of 100 nits, then if even mapped to 3x 25 nits (i.e. the upper limit position of the Anc_ins neighborhood), then the match is much more harmonious (since perfectionism is usually not needed to be achieved, but rather a pragmatically reasonable working system, balanced between appearance accuracy and complexity of realizing the system in practice) than with a linear stretch where the SDR content is bluntly mapped from white to white, i.e. from 100 nits to 1000 nits, which would make the average of AncB_SDR 200 nits (!), which would likely make all objects in the SDR image look too bright to be pleasing (note that in principle anc could be specified even below the actual luminances of the histogram, e.g. the luminance of the SDR image). mode, i.e. the most frequent luminance value in the histogram, but in any case, actual luminances will become excessively bright in a similar manner).

Coming back to Figure 15B, we interestingly show there how the anchors, and their positioning at different relative positions in any CombRng, will relate to (influence) the various luminance mappings. The TL_CS function is the luminance mapping to map luminances of the original master HDR image to luminances in the CombRng (i.e. ready to do image blending, be it pixel replacement, alpha blending, etc.). The applicant has found it very useful to perform luminance transformations on a relative axis system, i.e. both input luminances and output luminances or lumas ending in 1.0 (in case lumas are used as vertical coordinates, the mapping is defined taking into account the OETF, e.g. some fixed OETF definition like SMPTE 2084 or Rec. 709 for backwards compatible image communications). For lumas, that's very useful, because it bypasses the discussion about how many bits the vertical axis should be quantized (1.0 is just the highest luma code value, i.e. the brightest encodable color). For (e.g.) input luminance axes, one could still ponder whether or not this is a carryover from relative luminance handling, but in any case the necessary curves can be defined for any image transformation needed for any dynamic range (since this version can always be mathematically equated to its corresponding absolute luminance frame). We see that if we want to keep the absolute luminances of the darkest pixels identical, in a relative axis system with lower maximum brightness, this corresponds to raising that part of the TL_CS luminance mapping function above the diagonal, and below one can clearly see how the remaining upper subrange of the CombRng leads to some compression, which is not too excessive for the higher quality combining range. But for the lower quality blending range, the compression of the brighter HDR objects must be more severe, as also seen from the shape of the luminance mapping function corresponding to that lower blending range, namely TL_CI. In any case, this situation of a maximum brightness of a blending range, and a good anchor point, makes it clear that generically, whatever the HDR shot, some compression of the brightnesses will need to be done, which will generally have approximately this shape (the precise possibilities of the realizations are detailed below). It is also seen by combining the two functions, that if one needs to relatively transform from the upper to the lower CombRng, one would need to relatively brighten the darker pixels, but that visually they correspond first of all to a compression of the brighter HDR regions (the relative percentage subrange is, in addition to the total range available, a good measure of the encoding or rendering quality of a part of an image). In any case, it is the way the applicant defines his luminance transformations in his most typical real-life realizations, and in particular the very useful property that we will elucidate with Figure 16.

HDR video handling (in fact just encoding) methods like HDR10, which simply consider fixed representation of HDR images, do not even need to communicate a fixed luminance transformation function (since they can put the entire HDR object brightness distribution into the image luminances, which would not need communication of an OETF when transmitting lumas, if the OETF was chosen to be fixed). Technologies that take a simplistic view of the HDR object luminance distribution, i.e. its reclassifiability to other dynamic ranges, like the BBC's Hybrid-loggamma, can use a fixed function to communicate, e.g. as a LUT.

The applicant wanted both the content creator and the user (be it the TV manufacturer and/or the end viewer) to be able to unleash the full quality of HDR (hierarchically) - i.e. design a system where the content creator can indicate their wishes for the HDR image content incrementally (from coarse to fine, depending on how critical it is, or how much time and budget they want to spend), and the video consuming side can also decide up to what precision they want to follow these specifications, or if the end system wants to impart some micro flavor of color handling in the images, e.g. by deviating a bit from the object contrasts as specified by the fine grading function(s).

Figure 16 shows an example of the power of anchors, when combined with one of the requesters' preferred systems to allow, for example, a human grader to hierarchically specify their luminance reclassification needs, with a set of consecutive functions.

Consider that the input and output ranges are perceptually uniformly (roughly logarithmically) quantized. This would create a set of visually equidistant brightness levels, e.g. ultra-ulta-dark, ultra-dark, very-dark, dark, etc., up to extremely bright. You can then imagine that a grader can define how an image looks, by roughly equating the object luminances to one of these levels, e.g. a good quality HDR image should have the lamp luminance in the extremely bright luminance subrange, e.g. between 90% and 100% of PB_C. Of course, a lower dynamic range of the output, here the luminance combining range (L_Cmb_out) may not extend to a level that appears as extremely bright luminances, but then the grader would resort to mapping those lamp pixel luminances to the highest available level, e.g. very bright. This would be part of the form of the luminance mapping function, and the reader can understand how a grader (or also an automatic system using image analysis heuristics) could arrive at a complete luminance mapping function form.

Now, just defining two such axes would be a prima facie typical (albeit rather blunt) way of mapping defined luminances over a dynamic range, to output luminances over say a second, smaller output dynamic range (which would amount to a function which is the diagonal in this graph). In fact, "half-reasonable" output luminances would be formed, because this strategy would map white from any source image to the brightest possible (codable or renderable) color of the output dynamic range, and black to black, which due to the logarithmic definition would also reasonably distribute gray values between black and white from the input to the output image.

But this gives rather unspectacular images with the wrong brightnesses and contrasts, let alone allowing an artist to well match the artistic needs to the composition of any particular HDR scene (although the brightest and darkest pixel in our barn scene at e.g. a PB_C=5000 nit encoding may have the same value as in an image of say a desk lamp in a dark room with few objects in the back, of course the image composition and semantics of these two images will be very different, resulting in very different needs regarding classification and reclassification of the various objects or pixel luminances).

Thus, the applicant either for exclusive use of the image, i.e. reconstruction to an approximation of the original master HDR image of say 5000 nits, or for optimal display adjustment to obtain an optimal MDR image for driving e.g. a PB_D display of 750 nits, invented a luminance mapping definition system consisting in being able to specify at least two consecutive functions. First, a coarse reclassification function F_CRS_MDR globally remaps the luminances of all perceptual subintervals, in order to have a better initial appearance of the current HDR image and its details. We see a roughly r-shaped curve, for which the brighter half of the HDR scene pixel luminances are compressed into a somewhat small subrange of output luminances, e.g. the luminances of an SDR image to drive an SDR display, or to better match the teachings of blending, a 1000 nit MDR image, when the master HDR image had a PB_C of 5000 nits. Depending on the needs of the scene, the grader can adapt that shape. For example, if there are only a few specular reflection spots from a couple of pixels, or light bulbs in that upper half of luminances, the grader could in principle even define a function that clips to 1.0 for the upper half of input luminances, since that would not significantly degrade the visual quality of that particular scene. But if the upper half contains sunlit clouds, even slightly lowering the local slope of the curve could lead to significant quality errors. The reader should keep in mind that we are talking about relative luminance axes, so if the output is for example an SDR interval, there might not be many luma codes and corresponding luminances available to faithfully represent at the same time very dark luminances, medium luminances and very bright luminances (consider for example an image of 3 differently lit regions, with for example a dark kitchen in the 3D areas closest to the viewer where a black person is hiding, a medium room that is normally lit and again sunny objects seen through the windows, as schematically shown in Figure 16b; and another typical 2-region example is shown in Figure 16C, with a bright shop window on a street at night, i.e. different semantics than the barn example, but with subintervals that could be treated similarly, or not if the content creator so decides).

Figure 16C can be made "simple" enough (HDR affects complexity), to be sufficiently graded, and by fine-tuning the luminance mapping functions, with only a coarse grading function specified F_CRS_MDR.

But for a more complex 3-region scene (R1 indicates bright outdoor daylight with objects like houses being visible through windows, the middle region R2 is normally or dimly lit, i.e. with typical SDR luminances, and R3 has the lights off, i.e. with dark luminances) it may be more difficult to reasonably map all luminance subintervals and the corresponding object brightnesses to the small subintervals of especially the lower dynamic ranges (like e.g. the SDR range). In such a case the grader may find it optimal to use an additional curve, namely a fine gradation curve F_FINGR, to be applied to the relative luminances resulting from the coarse gradation, i.e. which in effect deviates equivalently from that coarse curve shape as shown. But such a strategy can also be applied to "simpler" hdr scenes like the internally lit nighttime shop window in Figure 16C. In order to make the shop window and its object have impressive enough contrast and brightness to make it look really bright (in a coordinated way, but by rendering this single image alone, both on e.g. a 5000 nit HDR display, and on a 1000 nit HDR display, and still reasonably approximate and nice enough as far as the limited display capability allows, on a 100 nit SDR PB_D display), and also to make pixels of relatively dark nighttime urban objects (at least in the master HDR grading) bright enough and visible, the grader can create an F_CRS_MDR function that is e.g. relatively flat at the top end. This can provide a reasonable overall look for both the shop window and the dark street, i.e. a reasonable overall look for the image. But the shop window may contain specific objects which don't come out right with respect to one aspect, be it their luminance, contrast or color, e.g. a flame in the shop window, or maybe some strongly lit part of the mannequin's white collar or shirt, etc. Thus the grader may define a correction curve (i.e. the fine grading curve F_FINGR), which at some part of the luminance range, e.g. corresponding to the luminances of that shirt, for example, reduces the highlights, e.g. to get more intra-object contrast, or colorfulness or whatever the function performs as an effect (there may even be several spatially localized luminance mappings involved to avoid shifting other objects with similar luminances in other regions of the scene, but explaining that further complexity of our system is beyond the needs to understand the present application).

Interestingly, the coarse function quickly gets all object luminances already approximately correct, with an easy action of the grader (for example, in one of our embodiments you only need to turn two dials determining the slopes of the top of the luminances starting from 1.0, and the bottom of the function starting from 0.0, and our encoder then automatically determines a smooth coarse gradation curve, which can then be used by the various HDR image combining apparatus of the present invention and application). But then the "real gradation" can begin, by adjusting the luminances of various image object pixels, and consequently the corresponding intra- and inter-object contrasts, allowing the grader to determine the shape of the fine gradation curve as desired.

A normal decoder, or display tuner, would just use these two curves as is. On the encoding direction (in case we use a backwards compatible system that communicates the HDR image information actually as legacy display renderable SDR images), first the coarse grading function is applied, and then the fine grading function which creates the corresponding optimal looking SDR image to this particular HDR scene master grading image, which was the input to the encoder. On the decoding side, in order to reconstruct the approximation of the master HDR image from the received SDR image(s), the inverse functions of exactly this coarse and fine grading function are used, because that was actually the link between these two optimal functions, which were specified to be optimal under the given typical rendering situations. That is, have an HDR display, then render the HDR image, and when you have an SDR display viewed under typical viewing conditions, then render the SDR images of the pair (and when you have a display with PB_D that deviates considerably from these two PB_C graded images, then use our display tuning to create the intermediate dynamic range appearance image, which properly corresponds to the master HDR image, i.e. closely enough approximates its object luminance appearances, at least as far as lower display capabilities allow).

But now we have a situation where we need to merge content, i.e. there may generally be a need to modify the luminances of at least one, and perhaps all images, again, to make them harmonious. The division of the classification needs into such two curves is very powerful, because it allows to separate object-related needs, such as sufficient local contrast, from the overall compression of the various subranges in the available output dynamic range (even if optimized for merging).

Thus, the coarse function defined by the anchor luminance, even if it only consists of linear segments, can now be used as an alternative coarse mapping of the subintervals, because apparently that was a better gradation for the blend, than the original coarse function F_CRS_MDR for the single representation of, say, the HDR movie, in a dynamic range with substantially the same maximum brightness as the selected blending range. Now, various embodiments could correlate the shapes of various subsegments of the F_CRS_MDR curve, but usually that will not be so necessary. What is important is that the blending apparatus can apply the fine gradation details to the various newly mapped optimized levels (i.e. bring various semantic subregions of the master HDR image, such as clouds or cave interiors, to the various subintervals of the CombRng), while essentially maintaining the fine gradation curve shape. The deviation curve can therefore be shifted to the new position (along the optimal coarse curve for mapping input HDR luminance to the blending interval, which is F_Cmb), by making a new, now optimal fine gradation curve for blending F_FINCMB. This can be done, for example, by simply equalizing the master luminances, i.e. by shifting in vertical lines the multiplicative deviation amounts.

By now, the reader should begin to understand that such anchor-based systems are very powerful in harmonizing various contents, even if they are complex in terms of luminance degradation data, but still in a relatively simple and ergo-feasible way.

We will now first describe a simpler embodiment where the apparatus can autonomously determine said anchor luminance in the HDR image, and in a related manner in the combining range. Ultimately it is important to have the anchor luminance in the CombRng, but this can be easily equated in some embodiments to where it is in the HDR image (and less easily in other embodiments, in which case the HDR image content creator can explicitly code it, such as L_S2A1). For example, we will clarify the principles with an important example of having a face (of course humans are important, so in many images or videos there will be at least one actor or presenter etc., and they can usually be reasonably well lit by the content creator, although, in the area of <l>D<r> it was never precisely specified what reasonably well that would be, nor was it really necessary). But the skilled reader should understand that our teachings about anchor luminance(s) are generic, so there may be other anchor luminances for other images, images that may not even have a face, such as, for example, a medium gray luminance. The reader should understand that in the LDR era a middle gray was a unique and very precise thing (the middle of the encoding, which would normally be represented as an average looking gray, and corresponds in the original scene to an object reflectivity of about 18% of the incoming white or illumination, which also due to the relative white-on-white representation would correspond to about 18% of PB_D), but it is not totally unique to HDR (because there can be a middle gray reflective object in a relatively darkly lit image, or even a dark region of an image, and the same gray object can reside in a relatively brightly lit image or region, especially if one doesn't want to force HDR content into the LDR straitjacket again, with very precise way of lighting and capture, but one wants liberally light scenes, liberally art-grade content, etc.).

It is advantageous if some of those anchor luminances are standardized (so that each device quickly understands what each specific luminance situation of any given image input is about), but even if a first image is specified with a different kind of anchor than the second image, or an image's anchor is different from which anchor (if only one) the device decides is optimal to specify in the CombRng, then those various specifications can still be roughly related. For example, if a black is not precisely specified (and of course, just as in HDR there can be multiple whites and brights, there can be multiple blacks), the device may assume, depending on the situation (i.e. the PB_D or other capabilities of the display, and possibly the viewing environment), that it is supposed to be rendered, say, 2.5 stops, or 4 stops below what the device decided was its primary middle gray (whether that is the first anchor luminance, and/or an auxiliary secondary). Blacks can be described (i.e., encoded into video) and manipulated, e.g., rendered, to a lesser or greater degree of accuracy. If a content creator cares about what is happening in the blacks—for example, because there is some critical action by a criminal hiding in the bushes that needs to be “half-perceived”—i.e., neither too conspicuous nor overlooked by the viewer—then the creator should annotate the image or video with additional descriptive metadata for that subgamut or color management regime. And in particular, the creator should define at least one anchor luminance for the blacks. If you don't care or care less, for example because in the present HDR scene there is only some background texture like photos in a dark basement, which could be rendered with more or less equal semantic quality whether they look good or not, then the creator can rely on typical receive-side or mid-side optimizations of blacks, where the rendering apparatus can render them with various luminance levels, as long as it still looks reasonably black (i.e. deep black, good black or milky black).

So, written with a simple example to make the user easily understand the main points of our invention, the anchor luminance (anc) in the combined luminance dynamic range (CombRng) specifies what facial luminance should be in the combined (output) image, in order for it to look appropriate. That will of course depend on whether the face is well lit, strongly lit, or in the shadows. That prevents the actor or person in, say, the commercial, or the BD commentary, or the videophone conversation, etc., from being excessively brighter than the one in the movie. Of course, the apparatus has means to not necessarily exactly place the two face luminances, but instead represent one in an offset luminance (e.g. if the actor in the main HDR movie is in the dark on purpose, but the rest of the HDR movie is (much) brighter, the commercial need not necessarily dim to the luminance details of that movie actor; note that some embodiments of the combination apparatus could additionally decide by using heuristics, such as, for example, how long the dark movie scene has already taken place, e.g. by keeping the elapsed time since the previous luminance considerably different or the type of, e.g., a daylight scene, and some embodiments could even annotate the movie luminances with additional metadata, e.g. the anc at the beginning of a movie scene having a second data element indicating how long the dark scene will last, but we will describe the essential elements of the embodiments here assuming that some content creators will not want to go through the trouble of making many metadata. annotations, and will just do the simple act of clicking somewhere on the image, or other representation of it such as a histogram, with a pointer, to set the current anc value, which of course in user interfaces can be displayed, for example, by showing all luminances in the image in a range or neighborhood around the anc value in a red pseudo-color, as a check on human selection of the anc value).

Thus, the apparatus knows where a good facial rendering luminance (the anc of this clarification example) should actually be, depending on the details of the situation (we assume for the moment a Caucasian face with a wavelength-averaged spectral reflectivity of about 36%, and of course other skin types of, say, 5% reflectivity will, under that local illumination, fall into related CombRng luminance positions). The reader may not bother with further details and more easily understand the various key aspects by assuming that the CombRng is in the present explanation only the dynamic range of the HDR image (Im2_Rng in Figure 4), and the apparatus knows where the most important facial luminance is, along that range. As explained above, various alternative determinations of the optimal CombRng may occur in various embodiments of the apparatus, depending on at least some of, on the one hand, the luminance characteristics (encodable luminance range, content actually in the range, etc.) of the input images, insofar as these are already known or can be estimated, or, on the other hand, the actual stated or expected use of the image content, e.g. the luminance capabilities of a display on which the content is to be rendered in a viewing environment (i.e. in fact the typical appearance of generic, average or test images, and the particular present images). This is already complicated, because faces tend not to have just one color or luminance (let alone that in some movies faces may be painted blue, etc.). If you study various faces even in LDR images, you see that they can contain pixel lumas even up to the limits of the LDR range in principle (i.e. in a highly contrasty capture or classification at least some pixels of the darkest part of the face are close to zero, e.g. in a backlit capture, and highlights are clipped to 255; sometimes, e.g. with consumer content, you even see half of the face being consumed as clipped white), although many faces have less contrasts between the normally illuminated region and the shadow region, i.e. they are more normal. Here we can divide the content into well-lit vs. specially-lit images, e.g. in real-life field productions (nowadays even images made by non-professionals are incorporated, e.g. in a news show, and if it is a night scene, faces can be poorly lit). One could say that in such complex situations one could also forego an accurate rendering or blending, but on the other hand one could also say that the more abnormally (deviating from good shooting practice) some image content is shot in the first place, the more one can benefit from regularizing it later on, e.g. by using an appropriate Anc value (e.g. if a cameraman in the field is in an unfavorable situation, e.g. when he cannot illuminate the person who for some reason needs to be in a darker part of the scene, and he only has a camera with lower DR capability, he may need to non-perfectly optimize his shooting images, but at least with the Anc mechanism he can quickly annotate (exotic) details of this image, e.g. as simple as with a style on his camera screen, or on his portable computing device before communicating the content to e.g. the production house, etc.). It is expected that with the emergence of better cameras and other image handling devices, for example cameras that can capture dark night scenes as if they were daytime scenes, that many years into the future the complexity of image handling (HDR) will only increase, either because there will be various standards and levels of production, for example also some people still supply news content with an old poor quality mobile phone. So the best solution would be to make a technology that can handle all such situations well. For example a gangster movie can have harsh lighting, for example cinenoir, with contrasting faces and strong highlights, for example in the hair or on a bald face. Talk shows can be lit in a less contrasting way, as sometimes for the layman the shadows are almost imperceptible, and one can light the women with a more pleasant and even lighting. It should also be noted that it is not the original 4:1 illumination ratio (brighter facial half vs. darker) of the face in the scene when captured that is important, but rather what is in the encoded image (how the contrast is graded). This contrast may usually have undergone some RAW luminance in-camera for luma mapping, and if the image is a scan of a celluloid image or film, the final contrasts may depend on such things as development, push, etc.

We have identified this aspect as there being a range of facial colours, for example in the clarifying example in Figure 4 the colours of the face in the LDR image are softly lit (hence a small range of luminances R_f), and the face in the HDR image was more brightly lit because it was, say, a horror movie, leading to a larger range R_f2; for example a person might be walking down a dimly lit hallway with spotlights, causing their face to become darker and brighter over time, around an average value). What is relevant is also how, geometrically, the face is lit. If there is, for example, a small white spot on a bald head, and this image is inappropriately, too brightly, blended with the other HDR movie in the CombRng, one might hear a comment that it looks like he has a lamp sticking out of his head (and the viewer can make such confusing recognitions when such a person blinks rapidly across the image, and potentially misses part of the story, or at least gets distracted by the wrong part of the image). But a small dot of a couple of pixels is not necessarily a major problem yet. That can usually be ignored for typical face color determination. When rendered on HDR displays, it may look like that person is illuminated by an extremely bright lamp, but at least the dot is small (say a couple of pixels). If the same thing happens to his entire nose, or his eyes, or the entire half of his face, then that might be more disconcerting.

So by being able to indicate a facial luminance (e.g. L_SA1 in the first image, let's say LDR) in the content, the creator can also easily and quickly indicate not only where there is a facial luminance, but also potentially what kind of luminance distribution it is. For example, if there is only a small light in her hair, she can use the average of the darkest pixels of the actual face as L_SA1, and some pixels that get brighter will look fine. If on the other hand half (or more than half) of the face is strongly illuminated, she can use that average luminance value or another characteristic luminance value as L_SA1, and then the rest will be darkened, and that will be a typically better way to blend, whatever the content this face will be blended with. In this case, the creator can still indicate with additional metadata that the face is actually an atypically lit face, e.g. X stops above the normal lighting. Such information can be advantageously used by the combining apparatus to judge at what luminance (above or below the previous position in the CombRng) this face should be represented (approximately). The normal illumination of a Caucasian (white) face can be taken from the reflection spectrum, which averaged over the wavelengths gives a reflectivity of 36%. So that is one stop above 18% of middle grey (which makes these two values identifiable). As stated above, in the classical LDR framework one would also represent this middle grey at about 18% of the maximum brightness of the screen, e.g. 12%. Of course, other races should be a bit darker depending on their skin type, because otherwise it can be difficult to actually detect the actor's skin type on film for those who want to, and that could lead to confusion. For example, a black skin may reflect as little as 5%, i.e. 3 stops darker than the typical Caucasian one. As for rendering, the face can be made to be slightly brighter in the LDR range than 36% of the maximum brightness of 100 nits, to have very "sunny" faces. For example, the imager can indicate in a first field of the metadata the luminance code of the "average" or reference luminance (via EOTF lumas can be calculated in absolute luminances) of this face object, e.g. 853, or 80% in a normalized luma encoding, and can indicate in a second field that this is a "2SB" type face luminance, meaning it is illuminated two stops brighter than the rest of that locale in the scene and its image, etc. The reader understands that the same thing can be achieved by different encoding mechanisms, for example one can also encode a second anchor point L_S2A1pkS which is a luminance k stop brighter still containing a face color (e.g. the brightest, or 95th percentile, or the brightest still in the face and not a hair highlight), and a third anchor point L_S2A1mkS which specifies some darker extreme of this face in this HDR image or video shot. Anchor points of various types can therefore be indicated in a standard understandable way (to be agreed upon by various standards implementing the system), for example either with a set of enumerated types such as “Face36”, “Facet8”, “Face72”, “Face5”, “MiddleDimGrey”, “MiddleBlack”, “MiddleDeepBlack”, or a hierarchical, sequential annotation of data can be used, e.g. “Face” “1 stop up” “Dim” (where, then, “face” means the 36% reflectance of the average amount of light residing in that part of the image under a particular type of illumination, and more importantly, a particular subrange of the display’s dynamic range to be used to render those pixels, to give the correct appearance of the rendered scene – “Dim” would be the middle bright colours, which are, for example, typically rendered, on most or all displays, in the range of 10-100 cd/m2). nits, below which there should be at least one region of blacks, and preferably a region of "Blacks" that are slightly worse lit, i.e. the dim illumination value divided by up to 10 (or 1-10 nit luminances typically), and a region of deep blacks, up to 1/100th of the normal dim illumination (i.e. 0.x up to 1 nit luminances), which in HDR artistic scene rendering would typically be used for example for the deep gloomy landscape seen in a castle when the hero walks through it simply with a candle. Above dim lighting, one should already be able to get excellent results, creating a well controlled high quality HDR landscape, if one adds a region for the "Brights" (where sunny outside would be rendered, say, about 10x for TV watching, where in real scenes it would be 100x brighter, i.e. luminances of 100-1000 nit), and one for the "Ultra-Brights", where one could render, say, lamps, explosions, etc., i.e. the effects that still make it interesting to have high-end HDR displays, with an upper subrange of say 1000-10,000 nit. One control should be enough, either for display tuning (to finally close the gap between scene-referred and screen-referred image encoding, after absolute encoding of the image structure by reintroducing again some correct level of display ratio), or as in this application for correct image blending, allowing for precise differentiated control across 5 scenes. Actually, if you do a study of HDR scene types, that exist in real life or that artists might make, you often see two region images, like for example the shop at night in Figure 16C (a uniformly lit region image is of course normally SDR-encodable), and you are already somewhat less likely to find three region images like Figure 16b, and when you need the full power of the 5 different lighting regimes, you are making a really complex HDR scene (maybe in some movies it only appears at the beginning like a commercial showing the latest HDR possibilities). But the reader will now understand that our principles are not limited to smaller or more complex realizations. If the content grader specifies only one anc value, then harmonization of several image contents with that anc value is already possible. This would normally happen if there is only one critical object in the image, for example the only actor or presenter (but as said, although this actor, even when highly contrast-lit, will only have a typical illumination and luminance, it can theoretically be anything between 0 nit and PB_C). If nothing is specifically needed for the anchor type, then the blending apparatus is supposed to work with a normal anchor related to a middle gray value, and of a normal rendering luminance, typically "Dim". Therefore, content creators who annotate only 1 anc value, should generally use a representative value for a mid-gray position (in the local illumination of that area of the scene and their image) of their actor, e.g. in a bimodal histogram residing in the sun, and then annotate the type as e.g. “Bright”, so that even with this simple small amount of information, the blending apparatus can very well judge what to do (e.g. in case the movie doesn’t bother too much, it may decide to render the “anyway” bright actor in the commercial as a dim PIP within the movie, or to allow for an impression of brightness, it may render the bright actor somewhat above the dim level, e.g. at (bright+dim)/2, e.g. at (30+300)/2 nit; if the standard allows for additional metadata to be communicated, and this metadata is populated, content creators such as e.g. the commercial may indicate that they do not want to deviate from this). its preferred "bright" representation too, but on the other hand, for various reasons, the devices could ignore it, for example, because the end viewer has indicated with the user interface that he wants to enjoy the film, i.e. the lowest disturbance errors of the bright inserted content should be given preference in the film histogram).

Of course, if the grader notes further values, he can, for example, specify his "Deep Darks", and then the blending apparatus can take that into account in its final blend, whether already prepared for eventually renderable bad blacks in a bright viewing environment, or not (i.e. with the blending still being done in a theoretically perfect frame of an ideal rendering system before tuning the display to a real system).

Thus, having both the anchor luminance (anc) indicating where to roughly place the face color (we assume for simplicity of explanation for this embodiment that the HDR face colors were already at that luminance level, so getting the face colors from the HDR image is then trivial, with no need for complex luminance mapping considerations), and a way to identify where at least the main face color is in the LDR image (source anchor luminance L_SA1), the apparatus can relatively simply put all face colors from both correctly combined images into the combination image as output image. And even for the simplest embodiments, the surrounding colors (of the face to begin with) and all other colors in the image will relatively nicely automatically fall around that anchor luminance (anc). The philosophy for the other colors was also relatively simple, as it was not necessary to elaborate it in further detail for those technical rendering scenarios. If all other colours fell within a range of about 2-2.5 stops around mid-grey (e.g. white is typically 90% or 5 times brighter in linear luminance) that would give a good picture. Prints often couldn't get very deep blacks, and also practical LDR television was limited to typically 32:1 (peak white vs. still reasonably well discriminable deepest blacks), in view of the light reflecting off the front glass of the TV. Being 5 stops from the image object content meant that if one made sure that the darkest colours and in particular the important blacks were 2-2.5 stops from mid-grey, one would also be reasonably well off the darker side of a well-lit LDR scene. All of this in principle means nothing anymore in handling HDR images, in particular for compositing and rendering, so an improved mechanism is needed, and in particular in view of the complexity (images can contain almost anything), some human guidance of what is semantically relevant, and therefore what would be a good match. More complex implementations can make more intelligent coordinated decisions of how other object luminances can be intelligently placed around the main object as placed by the anchor luminance, which in this simple to understand example is a face. One could annotate for example an anchor point which is a boundary of an interval, etc. For the avoidance of doubt, when we refer to video, we refer only to a temporal succession of images, and we do not limit ourselves to the fact that a particular strategy must be applied in a similar way for a whole movie, but may be applied only to a part of that movie, for example a shot of images of a same scene (i.e. for example the CombRng may be different at different moments of time, certainly if a new commercial with different object luminance characteristics is later combined). Since most of the features of the embodiments of our invention will be similar at least for spatial combinations, whether still images or videos are used, to avoid tedious less readable formulations, we can use in cases where the distinction of one of the two embodiments, video or image, is not necessary in the clarification, and the reader should understand that he can read the other as well. The determination of a function mapping a first luminance (source anchor) to a second luminance (combination interval anchor and), should be sufficiently clear to the reader for several embodiments. As said, one could, for example, construct the color transformation set of the combining apparatus to be as simple as making a two-part linear function that raises the particular input luminance (source image) from the diagonal to where it needs to be, and then connecting lines to the maximum and minimum of the input and output range (0, and [PB_C_image_1; LMC]), but if one has a particularly shaped function for reprocessing image luminances to make them more appropriate for different dynamic ranges or brightnesses, etc., one could also, for example, linearly scale that function so that the point on it for ancS falls on the output value of anc on the Y axis. It should also be clear how one could construct similar systems that do not need to use exact anchor placement, but work with a (not too large) neighborhood around that prior value. Usable proximity may be set as any combination between what the content creator may have indicated in the source metadata (e.g. this anchor may be rendered with a 20% deviation from its exact value) and what the blending apparatus may decide depending on the situation (as said, although for example a face in the business calculation may come out of the calculation to be rendered at, say, 150 nits, the blending apparatus may, according to various program rules, decide to deviate from that a bit - e.g. it may decide that PIPs that want to be placed in the very important center position of the image will be brightened a bit, and for example to the extent that the blending apparatus, which may be the only apparatus that has all the information of all the parts of the image to be blended, and perhaps even the rendering system and environment in which the blended image will be displayed, judges to be optimal or reasonable for the current total presentation of the content of the blended image). Unless otherwise specified, a factor of 2x below or above the ideal anchor position may be a pragmatically reasonable proximity.

Finally, we would also like to mention that because Applicant's embodiments may operate with functions that specify the final look of the images, in various embodiments it is not necessary for an actual apparatus (e.g. the combining apparatus) to actually make the combined image. For example, if the combining apparatus is an STB, it could send a fully optimally combined image to a TV, e.g. HDR10 encoded over HDMI if that standard is sufficient for what is needed for that image, and then the TV directly renders it as a dumb display. But the STB could also judge and prepare the situation, and then send all the necessary data: at least some image data, and some transformations (which when converted to the optimal transformations FF to calculate the combined image should be sufficient, but some other embodiments could also send anchor values, maybe even more anchor luminance values determined by the STB to the TV). In that case the TV has all the information to perform the blending uniquely, i.e. in a PIP blending it will not only do pixel selection, but apply FF_1 to the corresponding received SDR image pixels before storing them in the blended image buffer, and FF_2 to the luminances of the HDR image pixels, to get the correct luminances of those pixels in the blended image. The simplest embodiments of a blending apparatus may determine only a good anchor position. For example, depending on what the TV is displaying, how it is configured (e.g. via a user brightness control), etc., it may only determine a dim-light average gray anchor value, which under current conditions should render good looking images, e.g. anc_1D = 30 nit. Of course, this won't produce the best control over image blending, but if a first piece of content indicates with its anchor (e.g. a black average luminance value) that the action is happening on a dark night, and the second piece of content indicates that it's a bright, flashy commercial, then this simple blending apparatus implementation should already be able to reasonably harmonize both images around its anc_1D value. Of course, better, more complex blending apparatus implementations will have judged what a good blending situation would be for various types of content, and will have specified optimal anc values for the preceding brightness type's anc value (deep black to ultra-bright). The device will then be able to powerfully match any incoming content, noting its situation type or HDR effects indicated by examining its source values (and potentially additional metadata such as regrading functions, e.g. indicating how blacks can be made grey if necessary), to those different lighting subranges of the CombRng, whatever the situation (e.g. if the matching is done or prepared for expected additional handling, e.g. a rendering system that has low brights, e.g. there is not enough peak display brightness to dramatically render the ultra-highlights sufficiently brighter than the brights, in which case the matching may swing towards some form of posterization for the brightest image pixels - that would not be the theoretical best matching of image pixel colours per se, but a good matching for such lower quality systems). So the anchor luminance in the blending range is typically determined to be at least a good luminance to determine the other luminances around it, and is determined by the blending apparatus (based on generic needs for good quality image rendering and/or actual or intended content specifications and/or display capabilities), and the source anchors do a similar thing in the dynamic range of the source image, and are typically determined according to what the creator thinks is important for their images, i.e. at least what specific important pixel regions and objects around them (at least luminance-wise) should be rendered and used well controlled, in the various possible usage scenarios, in particular image blends. One can understand how to define a suitable anchor luminance for a specific object, i.e. a spatial and temporal subpart of images, and it should be clear that anchors may need to be redefined when the film switches to a new (take of) image(s), of a different scene, e.g. the first scene taking place in the living room of a naturally lit house on a rainy day, and the next scene in the basement lit by a single practical light, namely a light bulb in the middle of the basement (both scenes will have similar objects which can be indicated by an anchor, e.g. the actor's shirt, but of course they will have different graded luminances in the master images, and need to be represented differently in the final MDR images for variable displays).

It is advantageous when the color transformation assembly (310) is arranged to determine the color transformation (specifically the luminance transformation) (FF_1) such that an output luminance (LF_1_o), which is determined as a result of applying the color transformation (FF_1) to an input luminance of a pixel color of the first image or video (Im1_LDR) that is equal to the at least one source anchor luminance (L_SA1), is equal to the anchor luminance (anc). Therefore, one way to coordinate the luminances of the at least two images (or videos) to be combined is to design the combining strategy so that the anchor luminances chosen for all images are the same and are set to the appropriate anchor luminance (anc) value in the dynamic blending range of the output image (Im_o) in which the final image blending (e.g., a PIP or an alpha blend) will occur. The skilled artisan understands how the apparatus can compute a function having as property: FF_1(L_i=L_SA1)=anc, and then a mapping for all luminances around anc respectively L_<s>A 1. In particular, since we may be working in linear luminance color space (and the intelligent choices regarding placement of the object luminance along the luminance axis have already been largely made by the decoder 251 working on at least one of the images, because the content creator created the details of the different graded images for their image dynamic range, and the corresponding regrading luminance transform functions if communicated), a linear mapping (or sometimes a simple non-linearity for unevenly distributed contrasts, such as a gamma function) may often be used. As clarified by Figure 18a, one can understand how one (with a linear attenuation factor) has to attenuate the luminances as determined by the input image encoding by some factor for darker images, and a somewhat different similar mapping occurs for input image luminances brighter than the source anchor L_SA1.

In case of a linear function, one can easily understand that the function can be determined (e.g. in a 1.0-axis normalized system with x-axis = linear input luminance of the LDR image, and y-axis is normalized luminance in CombRng), namely as L_out=FF_1(L_in)=C*(L- L_SA1)+anc. C is a contrast constant, which the apparatus can further optimize to make the two images more harmonious. It could be determined by looking at the semantics of the two images, e.g. the histogram and spatial luminance distribution (e.g. small shapes at the top of the image being lamps, etc.), average brightness ratios of the images, etc. For example, if the HDR movie consists mostly of a dark basement, in which a person walks in the shadows, with the high brightness part consisting only of a single lamp, then the relevant contrasts (of the majority of the HDR image) will not be very high. So if one has to match a very contrasty LDR commercial, one might want to lower the contrast of that commercial a bit, to be more in line with the lower contrasts that the visual system perceives, hence expects in dark environments (e.g. one could restrict the brightest part of the LDR commercial, by applying a gently sloping part of FF_1 for those brighter relative colors, whatever the semantic content is, be it sunlit outdoor objects or light bulb pixels, as in Figure 18C. On the other hand, increasing C somewhat above 1 can help increase the vividness of a commercial, even if it is not displayed at its maximum or average HDR brightness in an HDR movie that is very contrasty (i.e. one can control the contrasts in the local subrange of the hdr combrng to which all possible SDR image luminances will be assigned, etc.). Of course, the reader understands that the apparatus can also perform other FF luminance (color) transformations, e.g. a function which has relatively high contrast just below the (L_SA1,anc) point, but then starts to decrease its slope, leveling out at some threshold blacks L_out=L_b etc. In fact, in principle any function can be determined by the blending apparatus, as long as it roughly maps the source anchor luminance to the anchor luminance, but typically the resulting functions will be relatively simple (and for example a multi-linear segment function will usually already give good results, at least on less critical HDR scene images), unless the authoring side has communicated specific details for the blending apparatus to follow, such as partial luminance mapping functions to reclassify various luminance sub-intervals, or the realization of the blending apparatus has significant internal image analysis sets, which allow it to propose more beautiful looking blends (by identifying and optimizing various inter-pixel or inter-region contrasts, texture rendering quality, etc.).

Advantageously, the colour transformation assembly (310) is arranged to determine the colour transformation (FF_1) such that a luminance ratio of a second output luminance (LT2_o), which is determined as a result of applying the colour transformation (FF_1) to a second input luminance (LT1_i), divided by the output luminance (LF1_o) is a multiplicative constant (C) times a ratio of the second input luminance (LT1_i) divided by the source anchor luminance (L_SA1). As mentioned above, simpler embodiments may determine a fixed value of C for the entire (partial) luminance range (the input luminance of the image to be processed), but C may also be made to be a varying function of the input luminance L_i (C=CF(L_i)). Partial luminance ranges may be defined by the combining apparatus (again, either guided by additional metadata prescriptions received, or by itself) in various ways, for example, the extent of a main histogram lobe may be determined, or a range for luminances in the LDR image between ancS/k and k*ancS may be determined, with k, for example, equal to 4, and the remaining LDR luminances mapped to the two discontinuous partial functions of Figure 18C, etc. As an example of more variable FF functions, one can increase the contrast for brighter parts of the LDR image (above the constant L_it, e.g. as a histogram percentile etc.), in case there is important information (e.g. strongly illuminated faces), but alternatively one can also decrease the contrast for that bright region in case it is only less relevant information above the main action, so that for example the apparatus can slightly increase the average luminance of the first mixed LDR image, e.g. (e.g. by mapping to anc+d_anc), but not overemphasize the HDR movie with too bright highlights in the luminance-mapped LDR image in CombRng. This allows, among other things, to make the luminances correspond more to an expected brightness distribution calculated in the human brain. Thus, LT2_o can be selected, for example, as some percentage above LF1_o (e.g. equal to anc), e.g. 150% of LF1_o, or k stops above LF1_o, and then C1 specified for that luminance range, etc. Intelligent adaptation of such (at least one or more) subregions of what is usually the least important of the images (e.g. an SDR commercial) can greatly improve the harmonious look of the total combination.

Advantageously, the colour transformation assembly (310) comprises an offset determining assembly (312) arranged to determine on the basis of the at least one source anchor luminance (L_SA1) a luminance offset (d_anc), and wherein the colour transformation assembly is arranged to determine the colour transformation (FF_1) such that an output luminance (LF1_o), which is determined as a result of applying the colour transformation (FF_1) to an input luminance of a pixel colour of the first image or video (Im1_LDR) which is equal to the at least one source anchor luminance (L_SA1), is equal to the anchor luminance (anc) plus the luminance offset (d_anc). As stated, there may be considerable differences between such object-related luminance aspects as for example the illumination of a face (e.g. high contrast) and the relationship of the face's luminance to the rest of the image. Therefore, the apparatus may decide that it is more appropriate not to render the LDR face (or any first dynamic range image anchor object, or corresponding region and subrange of luminances) exactly at the output luminance, but rather, for example, 2 times brighter. This may be determined, inter alia, based on a type of facial illumination (which may for example typically be indicated with an anchor type encoding), for example, if the source anchor luminance (L_SA1) is determined as the average luminance of a very brightly illuminated half face, and the face is to be rendered in the final image at a large size, then the apparatus may decide to determine a negative d_anc of, for example, 50%. The heuristic considerations behind these program rules and mathematics might consider, for example, that bright regions might over-irradiate surrounding darker regions, or be too flashy and conspicuous, or just irritatingly bright compared to the rest of the image, which might be the main thing the viewer was watching, such as a movie, etc. Considerations might also evaluate the brightness composition of the HDR movie image. For example, that image might consist of a dark interior region and a brighter exterior region as viewed through an open garage door. If the PIP of the relatively much brighter LDR ad is supposed to be placed in the dark part, it should be darkened (i.e. a relatively large d_anc, say, to carry the average luminance of the face in the LDR, or additional anchor luminances characterizing the LDR image, being weighted to a single representative luminance for the LDR image, halfway between the ANC and the local average luminance of that dark region of the second HDR movie image), so that the contrast between the two images, and the local contrasts seen from there by the brain in adjacent objects in those two images, are not too excessive. But if the bright PIP is to be presented near the sunny, outdoor part of the HDR movie, it may even need to be brightened with a positive d_anc. So in general these d_anc deviations will be calculated based on what an ideal composite (of say two faces) would be in an ideal situation (e.g. if the faces were surrounded in both images by a medium grey background), and what the actual situation is for at least the main image (i.e. in this example the HDR movie, on which we pasted the LDR commercial), both globally (i.e. what kind of objects it comprises, with average luminance and size, and potentially more semantic characterizers such as internal complexity with a texture measure etc.), and locally around where compositing would occur (e.g. in a small PIP of size X,Y). So the offsets will in general be determined based on what is harmonious for the two images, and in various embodiments that will depend on the details of the images themselves (contrasts, type of source anchor position and object content in that neighborhood, and content outside that neighborhood, geometric size, etc.), and the blend (whether the PIP is for example placed in a less critical small spot at the bottom right of the image, or how the blend will shine through partial information in a blend mix, etc.). Simpler embodiments will just use a relatively simple harmonization structure, which is not so bad per se (especially if the anc+d_anc value can be optimized to mitigate for example a risk of excessive contrast, if that's judging by the commercial), but more advanced blends can more precisely see where and how exactly the second image information is placed, or even additional properties of the blend.

Advantageously, the colour transformation assembly (310) is arranged to read at least one second source anchor luminance (L_S2A1) obtained from a second source (351) delivering a second image or video (Im1_HDR) of the two images or videos, and wherein the offset determining assembly (312) is arranged to determine the luminance shift (d_anc) also as a function of the at least one second source anchor luminance (L_S2A1). We have described above a simple embodiment, wherein the luminance structure of the main image (which we call image 2 in our claim formulation) has a known luminance structure and can therefore be easily transformed and form the basic reference in the CombRng. It is a useful way of combining in case the HDR image is a main image, which should be minimally distorted, whereby it is mainly the secondary content that is transformed to it. This can happen, for example, if the HDR image is known to be of a type produced to ensure that all Lambertian reflective objects (e.g. from a studio talk show) are given a luminance that always falls within the 0-400 nits range of the Im HDR Im2_Rng dynamic range (and that will correspond to, say, a 0-300 nits range in CombRng, or the same 0-400 range in CombRng), and everything above is mostly highlights, such as, e.g. small specular reflections on metals. In that scenario, one has a specific type of HDR scene, where the low-light part is produced in a way that is largely similar to LDR TV production, and the HDR effects are mostly highlights, for which one is not too critical about their exact luminance (and not, for example, objects through a window, which should not only be seen clearly, but outside objects should perhaps even have particular luminances in order to convey some artistic mood). In this case, the problem is mostly about coordinating the LDR spot with the lighting situation of the HDR talk show, mostly the part up to 400 nits, and also coordinating to some extent with what exactly is in the bright regions (for example, one could use the above contrast mechanism with multiplicative constants C, or any general determination and application of a luminance mapping function, to brighten small (identified or assumed) highlight regions in the LDR spot to make them correspond more closely to the luminances of these highlights actually present in the HDR talk show).

But in general, HDR content can be anything. For example, your faces may not be lit to about 25% of 400 nits steadily, but instead fall all over the luminance axis if an actor is running, for example, through a dark mine corridor with sparsely placed lamps. It is therefore advantageous for everyone making any content to annotate their content, so that also the HDR image will indicate where one or more of your faces fall on its luminance axis, with at least a second source anchor luminance (L_S2A1). It is then more certain that both images will harmonize in their final combination, because the apparatus can, by reading out L_S2A1, check where the face luminances are in the second HDR image (see corresponding clarification example in Figure 4). The HDR image can be optimally transformed for whatever the best final combing situation (in particular the CombRng) is, for example taking into account that the film will most likely be shown outdoors rather than in a dark environment etc. For example, the FF function can modulate different facial illumination to a lesser degree, in particular if for example the viewing situation will be such that the representation of dark object texture will deteriorate compared to the optimal cinematic display of the film.

It is also advantageous if the colour transformation set is arranged to establish a colour transformation (FF_1) to be applied to the first image or video also based on first colour remapping functions (F1_L) specifying a change in the luminance distribution of objects in the first image or video (Im1_LDR) to map that first image or video from the dynamic range associated with the encoding of the first image or video, to a dynamic range with a maximum brightness differing by at least a multiplicative factor 2, or wherein the colour transformation set is arranged to establish a colour transformation (FF_2) to be applied to the second image or video also based on second colour remapping functions (F2_L) specifying a change in the luminance distribution of objects in the second image or video (Im2_HDR) to map that second image or video from the dynamic range associated with the encoding of the second image or video, to a dynamic range with a maximum brightness differing by at least a multiplicative factor 2. multiplicative factor 2. As explained above, in our basic HDR video or image coding framework we indicate with communicated color transformation functions (at least defining a luminance transformation, but sometimes also a necessary saturation transformation, e.g. to increase the saturation of objects that needed to be darkened in the LDR look, in order to keep them vivid enough; functions specifying the look of dynamic ranges that typically differ by at least a factor 2 in PB_C, e.g. how to transform one extreme or medium degree, e.g. 5000 nit PB_C into the other, e.g. 100 nit PB_C) how a luminance distribution of objects should change (which is in general a complex optimization towards the lower capabilities of a lower PB display or codec, taking into account the semantic details of the HDR scene) when moving from a higher to a lower dynamic range (or vice versa). That is, when transforming from a 5000 nit HDR graded master image, to, say, a 100 nit image, or some mid-dynamic range image, for example, optimal for driving a 1200 nit display. Typically, the creator can determine how they would like the look to change, for example, progressively darkening highlights and dark regions.

The power of the blending framework is that it can decide how to formulate the new kind of functions, which do not map to a different (single) optimal rendering dynamic range, but to a blending dynamic range, depending on, among other things, how precisely one wants to control the colorimetry of the blend (which itself can depend on various external parameter settings, with parameters quantifying, among others, the desire of a content creator, owner, or distributor, a viewer - are you trying to attentively enjoy the movie, for example - or even the display manufacturer). Of course, the overall look of the blend may vary depending on the particular blend and what it contains, but that doesn't mean that the regrading functions per se don't contain interesting regrading needs or desires (whether in the brightest subrange, say the top 20% of lumas, there are clouds that want careful regrading, or less important objects), which also the blending apparatus may want to track to some extent.

Now, if the simple linear functions FF mentioned above are used to map image input luminances around the anchor luminance ANC, the details of F_L may not be needed. But more sophisticated mappings can vary in linear strategy, and in particular need to do so according to what the content creator is implicitly said to need in the F_L functions for that particular image of that particular scene<h>D<r>, in that subrange of luminances. For example, if the creator likes to have the darkest luminances quickly fade to black, the color transformation suite (310) can take that into account when determining an optimal FF function - e.g. it can stretch out some of the contrast for brighter regions (around face color, for example) by more quickly clipping to black a larger percentage of the LDR luminances than the linear function would do. Thus, the apparatus may evaluate the functional form of at least one luminance transformation function from the set of functions F_L (which, for simplicity of understanding, the reader may assume to be just a function mapping from arbitrarily shaped function normalized to 1.0 input luminances to normalized to 1.0 output luminances), and use a portion of that form to design the final function form to map to CombRng. It should be noted that various desires may be given in such functions for reclassifying parts of an image, either by itself or relative to other possible luminances in other parts of other images, etc. For example, one could specify for a geometric region of an image that the luminances present there, for example, should not be increased too much, even if, for example, in the rest of that image there are pixels with the same luminance that can be greatly increased. In principle, any form of such partial function can be indicated (e.g., previous embodiments that the applicant has developed allow the specification of a subset of pixels in an image that meets the property of falling in a particular geometric region such as a rectangle, and has luminances between L_min and L_max, and then a desired mapping behavior partial function could be defined, e.g., for luminances in that region of, say, the HDR image that falls between L_min+k and L_max-l, with k and 1 constants. For example, a single or parameterized boosting function could be given for those luminances L_out=B*L_in+O, where B and O could be a function of various properties, such as LMC, or an average of the luminances of the combined output image surrounding the rectangle in which the HDR image data is placed, etc. Any split reclassification behavior can be indicated in the functional specifications before final combining, or even deconstructed after an image has been created. combined (so that at least part of the total content is combined).

It is advantageous if the dynamic range setting assembly (302) is arranged to set the dynamic range of combining luminance (CombRng) depending on the luminances in at least one of the two images or videos. Not all embodiments need or can take into account the details of the final viewing environment. For example, the output image may be mixed in some main control room for transmission to several users. The final situation may not be known, and could vary considerably (if one viewer is watching, for example, on the train, and another in the movie theater he has made in his attic). In that case, it is advantageous if the combination is optimized based on the characteristics of both (or all if more than 2) images are mixed, because that will always be relevant. Of course, the reader will understand that if well applied, the combination paradigm is even more scalable. For example, a first content creator may have mixed two image sources, but that does not mean that more content cannot be mixed in the future. And that could happen both post factum, and with the first creator already anticipating, and at least some of the blending apparatus's embodiments should cater for all of these situations. For example, the first content creator may have the first say on what should happen, so he might determine a CombRng (and/or anc) that is optimal for his content, but perhaps already anticipating that some entity later will add some image content that has some typical additional features (e.g. a ticker tape with news information at the bottom, or in a movie theater or store or museum situation, a secondary projector projecting some secondary image information in addition to or partially through the primary rendered image content, etc., which secondary content will have a dynamic range, average brightness, dynamics such as time evolution of average brightness, etc.). The secondary content blender may still decide whether it is better to follow the first content creator's cue, e.g. keep the CombRng, or perhaps fine-tune it, given that there is now additional information regarding the final blended image or its presentation. Two kinds of embodiments are possible. In a simpler class, the two images are already at this stage inseparably mixed (e.g. final PIP compositing), and the final receiving side can only optimize this total image according to its rendering details (PB screen of the display to be rendered, average surround brightness, etc.). That is, for simplicity of understanding, let's say you take this combined image and apply it as a square root function to its luminances, to get the final output luminances to be rendered. This may be sufficient for many scenarios, because the blended image may already have harmonized luminances and the viewing environment may not differ too much from the intended one.

More advanced embodiments will allow redetermination of the object/pixel luminance distributions of the two images at the receiving end, and in some embodiments may comprise redetermination of the original images, at least to some extent (e.g. their basic encoding, or at least the original luminance gradation; that is the deconstructibility mentioned above). For example, some embodiments could specify the PIP combination in some way of luminance distribution, but jointly encode the specific details of CombRng and FF functions to bring at least one of the two images to correctly harmonized luminances within CombRng. For example, one can imagine that one just puts the LDR PIP on the HDR surround image with linearly scaled luminances to 5000 nit PB, which without correct processing at the receiving end would of course give a horrible looking combination. But the receiving end gets all the information needed to make the combined rendering correct, at its side, i.e. with a harmonized PIP image. It can do this by appropriately changing the pixel luminances in that rectangle, taking into account everything it knows (i.e. for example received partial luminance mapping functions for that region, or special additional ANC values for that rectangular region, etc.; or what it can determine on its own end, e.g. that said region appears considerably bright compared to surrounding pixels, etc.). The skilled person will now have understood that these <f>F functions are not the F_L functions for reclassifying for a different dynamic range per se, e.g. not simply the functions for reasonably mapping a single LDR image appropriately to an MDR display dynamic range of e.g. 1500 PB, but rather functions with which the authoring side (or the blending apparatus at whatever location it is set) specifies that this would give a good blend for the LDR image harmonized with the HDR image, i.e. taking into account the details of the two images together.

Advantageously, the dynamic range setting assembly (302) is arranged to set the dynamic range of combination luminance (CombRng) in dependence on a maximum brightness of a display on which the at least one output image (Im_o) is to be represented, and preferably also on a brightness characteristic of a display environment. If an embodiment of our apparatus is incorporated into a final rendering site apparatus or system, for example a television or computer, then it may be appropriate to further specify the combination according to the display details. If an LDR image would appear excessively bright when viewed on its own in a dark environment, it is likely too bright for that dark viewing environment when paired with a movie, particularly since that movie may already have been optimized for that particular viewing situation (see Figure 3 for the appropriate MDR rescaled aspect image Im2_MDR that is calculated for an e.g. 1200 nit PB display viewed in a dim environment, from a 5000 nit Im_HDR that all viewers having the same content subscription or watching the same streamed program receive). The partial images can then be blended together at their appropriate MDR dynamic range (Im1_LM and Im2_HM). This will be particularly important when, for example, the darkest pixels of some image are difficult to see in brighter viewing environments, in particular if the viewer is distracted by (perhaps even spatially neighboring) bright content of the other combined images, and then having, for example, an additional anc value for the darkest pixels in at least one of the images (e.g. the main image) can significantly improve the optimization of the final combination. The skilled reader will understand that of course with the combined presentation of the two images the optimization generally needs to be different from single image optimizations, and depending on the totality of the situation, since that is what the viewer comes to perceive (of course, pragmatically one will build apparatuses that do the optimization more or less accurately based on how complex one can afford in view of the apparatus's positioned market price; some of the simpler realizations can already do a quite reasonable job). When watching a single LDR commercial that is too bright, even in a dark viewing environment, the viewer will largely adapt his vision to what is seen on the screen. But when having two combined images, it is not trivial a priori what exactly human vision will adapt to, and consequently what luminances and brightnesses the brain will see for the combination of rendered pixel luminances (it is not unlikely to start seeing the main movie poorly), hence the desire to have a system where the content creator can at least indicate the minimally necessary information and guidance (practical technical systems are always a trade-off between complexity - e.g. the amount of effort invested by a content creator and the cost of the integrated circuits that need to do the calculations - versus the need to be able to handle at least some minimally necessary color transformations so that the system behaves at least half-reasonably, rather than just poorly; and various, typically standardized, realizations will decide how far they will go with additional specific control functions in which scenarios). Thus, the skilled reader understands that it is advantageous to determine the CombRng (and also the anchor luminance typically, and also the specific forms of the FF functions if they are non-linear, or the contrast multiplier C if linear) depending on at least the capabilities of the display (its PB may be sufficient in most situations, but additional features may be incorporated, e.g. if it is not fully utilized in view of energy saving, etc.). And if apparatuses are connected to assess the brightness of the viewer's surround environment, then one may want to do more. For example, an illuminance meter can be used, but it may be better to use a camera somewhere in the room on the receiving side, e.g. connected to the display, which can be calibrated on a luminance meter for various regions of the image viewing environment, and is already in some TVs. From this, one can roughly estimate how the viewer will experience the images. Simpler or more advanced versions can be used, for example a camera that checks the various luminances in a region behind (see around) the TV up to 1 TV width on each side.

It is advantageous if the luminance anchor determination assembly (303) is arranged to determine the anchor luminance (anc) depending on at least one of: the dynamic range of combination luminance (CombRng), the luminances in at least one of the two images or videos, a maximum brightness of a display on which the at least one output image is to be rendered (Im_o), and a brightness characteristic of a viewing environment. At least one anchor luminance should also be established and, from which, the distributions of the luminances of both images can be distributed in a harmonized manner. The reader should understand that various embodiments may make the determination of CombRng and anc in any order. For example, some embodiments may first establish, e.g., depending on the capabilities of the display (actual or expected) and the brightness characteristics of the viewing environment, what would be a good CombRng, e.g., typically for the current movie situation, or even per image shooting, etc.

For example, some embodiments may, before starting a movie, analyze the first few frames of the movie, or a characteristic set of frames sampled throughout the movie. Or the movie may have jointly encoded metadata, from which the apparatus can establish, even before decoding the first frame, what a reasonable at least initial CombRng would be. For example, the metadata may specify that the average brightness of the movie is 200 nits with a lower range down to 400 nits, and with a more extreme sequence, say a shot in a desert, dropping down to (i.e., which should preferably be rendered on any MDR display that has sufficient capability to do so) luminances of diffuse objects such as sky or sand of up to 900 nits (and an average of 600 nits). The image-combining apparatus implementation may then decide that it should reserve in the CombRng (especially if dealing with multi-PB service displays) a fixed area of excess brightness, say up to 200% of that 900 nit that would be the most excessive requirement for that movie, so that the top 100% of luminances can be used for all highlights (whether small specular reflection spots, lamp, or even regions above the brightness that contain image detail, such as a brightly lit part of a scene). Of course, more advanced specifications can help determine what exactly the brightest luminance region in CombRng should be, also encoding what kind of HDR effects are typical in the program or movie, or part thereof. Working with such a pair of subranges is powerful and versatile (whether they primarily indicate content needs, or intended rendering capabilities, or both), especially if supplemented by a pair of good anchor luminances for those ranges.

Having decided on an optimal CombRng, the apparatus can then build on it to determine where the anchor luminance of, say, a face color should be. For example, if half the range is for well-lit Lambertian objects, it can then decide to use 40% of that 50% peak luminance as the front luminance point. However, some implementations can work the other way around. Taking into account all the rendering characteristics, the apparatus can determine an appropriate anchor luminance for faces or the main region of mid-gray Lambertian objects, so that it looks appropriately bright to the viewer. It can then build an appropriate CombRng around that. For example, it looks at the most demanding HDR image and sees bright objects (see lamps) up to 20 times the luminance of the face. You can then decide to set the maximum of the range to that luminance value, or consider for example that 80% of that value should be sufficient for a combined representation of two sets of images, and then the higher values in Im_HDR respectively Im2_MDR (whichever is used in the combination in the particular realization) will typically be clipped to the maximum LMC of the dynamic range of the combination. This can be very useful in case you want to harmonize content with for example lower dynamic range capabilities, especially if for example the HDR movie is not the main content, but for example a PIP on a computer screen which mainly shows other image regions with for example information etc. So simple embodiments will make an object 2x brighter than the face color, or generally any anchor luminance in any image (i.e. the first and second HDR images, or the LDR image in case an LDR image is used in the combination) twice as bright also in the respective image pixels when it is ready to be added to the combination image, or images (Im_o). Or more advanced color transformations (or combinations, some of which may also weight or change pixel luminances in their mathematical combinations) may make some luminances darker or brighter. And the apparatus may make similar considerations to arrive at the lower bound luminance LmiC of the CombRng, e.g. somewhat brighter than in at least one of the source images, so that the combined image (which may now have much brighter objects than at least one of the originals) does not look too dark or too contrasty, etc.

Other embodiments may be realized as corresponding apparatuses (either as small as part of an IC, or as large as a complete professional or consumer system) or methods, for example:

A method of combining two images or two videos of images (I m_HDR, I m_LDR) from different sources and with different maximum luminance, one of them being a high dynamic range image or video, the method comprising:

- establish a dynamic range of combination luminance (CombRng), which is characterized by at least one maximum luminance (LMC) that is determined based on the maximum luminances of the two images or the two image videos,

The establishment further comprises determining an anchor luminance (anc) in the dynamic range of combining luminance (CombRng);

- applying a luminance transformation (FF_1) on at least one of the two images or videos producing a first output image having first output luminances, which luminance transformation is set based on a value of a source anchor luminance (L_SA1) read from metadata of a first source (350) delivering a first image or video (Im1_LDR) of the two images or videos, the luminance transformation having the property that the source anchor luminance (L_SA1) is mapped to an output luminance in the vicinity of the anchor luminance (anc); and

- combine the output luminances of the first output image with luminances of the other of the two images or two image videos to form at least one combined output image (Im_o).

A method for combining two images as above, wherein the color transformation (FF_1) is determined such that an output luminance (LF1_o), which is determined as a result of applying the color transformation (FF_1) to an input luminance of a pixel color of the first image or video (Im1_LDR) that is equal to the at least one source anchor luminance (L_SA1), is equal to the anchor luminance (Anc), or a luminance shift from the anchor luminance (Anc) by a given difference (d_Anc).

A method of combining two images in which the color transformation (FF_1) is determined such that a luminance ratio of a second output luminance (LT2_o), which is determined as a result of applying the color transformation (FF_1) to a second input luminance (LT1_i), divided by the output luminance (LF1_o) is a multiplicative constant (C) times a ratio of the second input luminance (LT1_i) divided by the source anchor luminance (L_SA1). As explained above, the other luminances occurring in the images around the anchor luminance can be simply mapped around the anchor luminance, or in a more precisely determined manner.

A method for combining two images in which a color transformation (FF_2) of the second image or video of images is determined based on at least one second source anchor luminance (L_S2A1) obtained from a second source (351) outputting a second image or video (Im1_HDR) of the two images or videos. Several anchor luminances may be determined not only of the first but also of the second image, so that their various brightness ranges (e.g., well-lit, light shadows, deep shadows, lamps, etc.) may be optimally coordinated with each other in the output range of the output image CombRng.

A method for combining two images where at least one of the color transformations (FF_1, FF_2) to be applied to the respective one of the at least two images or videos is determined at least in part based on at least one received function (F1_L, F2_L) indicating how the respective image is to be transformed from the dynamic range corresponding to the one it is encoded in, to a dynamic range which is at least 2 times larger or smaller. As said, these functions determine how exactly the brightnesses of all objects in their scene should look like at any dynamic range (at least 2 times different from their native dynamic range, i.e. the PB of the reference display corresponding to the EOTF defining the Y' luma codes of the received input image). That information can therefore also be used if re-grading is needed, for example, to a somewhat lower dynamic range (e.g. CombRng is a bit lower than the MDR range of Im_MDR). This will do most of the cleverness in arriving at coordinated luminances at least for an image or video used on its own, but can be a good starting point for roughly determining an initial set of image pixel luminances mapped to CombRng for combined presentations of image content as well. Additional fine-tuning may occur, either somewhat simplistic, e.g. a heuristic linear stretching determined by the combining apparatus or procedure, or more advanced taking into account the specific required luminance coordinations in the images at their objects or luminance sub-intervals (either intelligently determined according to, e.g., object segmentation algorithms and the corresponding luminance intervals, or more approximately, e.g. by dividing the CombRng into 5 equal or non-equal sub-intervals, and likewise for the two input image luminance intervals, and their corresponding sub-intervals), as they are specified by the content creator in the shape characteristics of the functions F1_L respectively F2_L, whichever is being examined and used in any embodiment.

A method for combining two images as above, wherein at least one of the combination luminance dynamic range (CombRng) and the anchor luminance (anc) is determined based on at least one factor from the set: a property dependent on the luminance distribution of objects in at least one of the two images, information summarizing the luminance characteristics of at least one of the two images, the dynamic range of a display on which the at least one combined output image is to be rendered (Im_o), and a brightness measure for a viewing environment in which the output image is to be viewed.

All of the above can be incorporated into various systems and forms, whether consumer appliances or professional systems, some components of which may reside on servers in other countries connected via the Internet, etc.

It may be advantageous if an image is supplemented with metadata, such as a specification of a CombRng (e.g. its upper luminance LMC and its lower bound luminance LmiC), and/or an anchor luminance typical of, e.g., a face, and possibly also at least one color transformation function indicating how to map pixel luminances of the image to the CombRng.

It is also advantageous if the content creator annotates his images with at least one semantically relevant anchor luminance (and preferably of a pre-agreed type), so that when received, the image lighting options can be understood by an apparatus wishing to combine multiple images.

Advantageously, each of the methods may be embodied as a computer program capable of communicating via some physical medium, for example, a computer program product comprising software code enabling a processor to execute the code such that when the code is executed all steps of any of the above method embodiments are performed.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the method and apparatus according to the invention will become apparent from and with reference to the embodiments and embodiments described hereinafter, and with reference to the accompanying drawings, which serve merely as specific, non-limiting illustrations exemplifying the more general concepts, and where dashes are used to indicate that a component is optional, non-discontinuous components are not necessarily essential. Dashes may also be used to indicate elements, which are explained as essential, but hidden within an object, or for intangible things such as, for example, object/region selections (and how they may be displayed on a screen).

In the drawings:

Figure 1 schematically illustrates how in the (recently started) HDR era several different ways of encoding videos have been proposed (in 2015), which will lead to images with very different luminance characteristics, which are not easy to relate;

Figure 2 schematically illustrates the basic components of how Applicant's HDR encoding technology can encode an HDR image, together with a different LDR luminance appearance (or color gradation) of the brightnesses of objects in the same HDR scene of the image, and how this information is sufficient to calculate an appropriate appearance with the correct brightnesses of the objects in the image on any available HDR display at any maximum brightness (PB_MDR);

Figure 3 schematically illustrates a basic embodiment of the present HDR-capable image or video combining apparatus, showing the main assemblies;

Figure 4 schematically illustrates how the color transformation appropriate to the optimally chosen dynamic blending range occurs prior to video or image blending;

Figure 5 schematically illustrates what problems can occur if one does not carefully prepare the pixel luminances (or overall colors) of the two images, but instead combines them in a simplistic way, so that one of the possible combinations is a picture-in-picture composite;

Figure 6 schematically illustrates how the apparatus may determine some appropriate color, or at least luminance, transformation function(s) (FF) by way of example for at least one of the at least two images to be combined;

Figure 7 schematically illustrates what algorithms may be used, as developed by the present applicant's inventors, to automatically optimally artistically regrade, as desired by a content creator, an image from a first dynamic range to a different second image, of which the dynamic range (at least its maximum brightness) may differ by several factors of two;

Figure 8 schematically illustrates some more advanced examples of how the apparatus can adjust the functions (FF) to color transform the images to be combined in the combining interval (CombRng), based on the semantic details of the object luminance distribution in the HDR scene as shown;

Figure 9 schematically illustrates an example of an application of an embodiment of the apparatus and method described herein, namely, in an original multi-video production, such as, for example, a television talk show;

Figure 10 schematically illustrates a second example where one embodiment of the apparatus and method is used in a system at an intermediate node of a video communication network, such as, for example, a transmission control room to serve a local area (the reader should understand that other embodiments of combining apparatus may reside elsewhere in the video or image management chain, for example, at the premises of the video end user);

Figure 11 schematically shows how embodiments of the apparatus may determine a CombRng and assign source image luminances thereto, from a good position of a luminance value of an anchor luminance;

Figure 12 schematically shows further explanatory examples of how embodiments of the apparatus may determine a CombRng and assign source image luminances thereto, from a good position of a luminance value of an anchor luminance;

Figure 13 schematically shows how pixels from two images could be simplistically combined if the technical aspects of the present invention are not used, but which could lead to poor quality combined images;

Figure 14 schematically shows how even in an image capture procedure, in particular encoded with an HDR video encoding technical procedure, one could have very variable luminances for the same object, in particular if one wishes to use more freely the potential of HDR imaging;

Figure 15 further clarifies schematically how various combining intervals could be determined, and what that could mean in practice, in particular for the establishment of the related sensitive anchor luminance(s) (anc), and from it the typical luminance mapping functions for harmonizing the partial content of the at least two images or videos;

Figure 16 shows schematically in particular how a very powerful harmonization can be achieved in case the upstream part of the video handling chain, the video coding per se, has been specified in a particular way of coarse and fine reclassification of the various image luminances;

Figure 17 schematically shows how automatic versions can formulate the determination of the various variables based on various algorithms that determine factors such as errors, both distortion of the image content and disharmony of the content, which in the case of human classification and image annotation can be determined by the human content creator; and

Figure 18 schematically shows some additional examples of luminance transformation function realizations useful for determining harmonized luminances for image combining.

DETAILED DESCRIPTION OF THE DRAWINGS

Figures 1 and 2 have already been explained above, and Figure 2 shows how we preferably encode an HDR video, or more precisely an LDR graded image and an HDR graded image (typically 5000 nit PB) of the same scene, which can be decoded (by applying one or more luminance transformation functions to the received LDR image(s)) to produce an optimally regraded image for peak display brightness anywhere within or even outside the range of dynamic ranges spanned by the LDR and HDR grading as extremities.

Figure 3 clarifies at a high level some basic components underlying most of our HDR image, or video, harmonic combining apparatus or method implementations. We will now describe the situation with a PIP example, although the same techniques can also be used, for example, to attenuate the brightness of a classification in a coordinated manner in a temporal succession of interleaved video from different sources. For example, one can set a good CombRng level for the LDR commercial and adapt the luminances in the HDR images gradually towards it (various additional metadata could be sent regarding the temporal evolution of the various videos). The skilled reader will understand that once the situation of very different and differently defined luminances (let alone luma codes) of the two images has been converted into corresponding related similar luminances (to be combined), one can also start to blend the content in more complex ways, e.g. alpha blending, or replacing part of a face with some computer generated face structure with appropriately coordinated luminances, etc. For example, if the luminances of the face (e.g. each side of the differently lit face) mapped to the CombRng of the first image are x and y, and of the second image x+e1 and y+e2, with sufficiently small deviations from e1 and e2, then one can apply blending equations like L_out_comb = alpha*L_im1_comb+(1-alpha)*L_Im2_comb, where L_im1_comb and L_Im2_comb are respectively the pixel luminances of the first and second images after they have been color transformed (with the respective FF function to be used) to the CombRng, and alpha is a real number between 0.0 and 1.0, and L_out_comb is the final luminance for that blended pixel in the output image Im_o, i.e. the blended image having CombRng as dynamic range.

The reader can understand that if a creator has as much control as possible over the original images (in particular, he may even need to know the details of the intended rendering environment), he could of course fine-tune a combined image by any means until it is completely satisfactory according to his wishes. But the present embodiments cater for the many situations where that is not so simple or possible, for example because there is no time to do all that amount of fine-tuning manually, or some inputs or variables defining the optimal look are not yet controllable, or not even known. Something can always change in the image handling situation, so that is what more advanced HDR handling technology must cater for. Or at least our embodiments allow for more versatile applications, where at various times colorimetric fine-tuning can be done, for example according to new wishes, and some aspect information of the original image can be retained. That is important, that one retains something of at least the most important aspects of the images at each stage. Of course, the end-user could decide to totally recolor the video, but then all the content creator's efforts have been bypassed and lost, and that's actually of no use to anyone (even if, for example, a TV manufacturer wants to add some of his own particular taste to the image rendering, he shouldn't necessarily do so by ignoring everything the image is about, in particular what the content creator specified about the image; but on the other hand, the receiving side shouldn't feel like he has absolutely no say in the colorimetry of the image, even if it clearly seems to be inappropriate for the situation).

The decoder 251 is again similar to what we elucidated in Figure 2, i.e. it can handle image gradations (HDR, LDR) according to our explained HDR encoding procedure (extracting luminance mapping features, determining new MDR regraded images therefrom, etc., according to the various possible embodiments of global or local images). Of course, the reader understands that this is simply elucidating one of the possible embodiments, and the apparatus would normally be able to handle (i.e. decode and combine) all types of HDR or LDR video or images, i.e. Im_HDR could also be defined according to the PQ EOTF, or the BBC-HLG procedure, etc. In this example, however, we assume that from some (second) image source 351 (e.g. a satellite broadcast channel, or a connection to a video storage via the Internet, or even a connected camera, etc.), the second of the two images to be mixed is obtained (e.g. the main, more important program (driver) is an HDR movie that is artistically graded with color accuracy), and for example it is a 5000 nits PB HDR gradation whose images are transformed with a PQ EOTF, and then encoded by DCT, and encoded according to some HEVC profile and level. Metadata comprising F2_L functions is also communicated to downgrade this HDR image to a 100 nits LDR image. The decoder can create an optimal image for a connected display, for example a 2500 nits MDR image, Im2_MDR. This would be the correct rating for this film being viewed on its own, without combination with any other (first) image data, whether spatially or temporally interspersed.

In Figure 7 we summarize how, according to one of our embodiments, such a reclassification from a first dynamic range aspect to a second one can be done. Suppose L_in are normalized [0.0-1.0] HDR input luminances, corresponding to real luminances (to be rendered on a display when rendering the HDR image) of 0-5000 nit. L_out is an LDR luminance, normalized, but corresponding to a PB=100 nit (i.e. an SDR luminance). The function 702 that maps this HDR to this LDR image is an example of a particular F2_L color luminance transformation (one can assume for simple understanding that one simply has a grayscale image, but we have shown that one can apply the required color transformation on RGB colors this way as well). Suppose we want to derive the optimal rescaling function 703, which corresponds to the PB_MDR of an MDR display of, say, 3400 nits. The diagonal line 701 would be the required luminance mapping if we wanted to map the 5000 nits input image into an optimal image for a PB=5000 nits display, because the HDR image is already optimally scaled for that display (i.e., symbolically transformed by an identity transform). We set a derivation direction DIR, say, vertical. The scaled function for a 3400 nit PB display, should give a value F*(Ls) for any input luminance Ls that lies between F(Ls), i.e., whatever the optimally determined degradation function F(F2_L) lies at that luminance point, and Ls. We can determine this point to be Ls+FI(PB_MDR, Ls)*(F(Ls) - Ls), where FI is some function giving values between 0 and 1, and in simpler scenarios being just a function of PB_MDR, which can also be written multiplicatively as k(PB_MDR)*Ls. What function exactly is used depends on the reclassification philosophy of the implementation used, and the reader can imagine that there can be several more or less complex implementations depending on whether for example specific features of the viewing environment (or even viewer preferences) are taken into account, but the details of that are beyond what this application needs as an explanation, which is already complex enough (the reader should understand that some reclassification function can be set, and applied to the input image, to obtain the correctly reclassified image with the correct luminances of the image objects for an MDR display).

As stated, the first image can be anything, but we will clarify the principles with an LDR commercial, the principles of our realizations being similar for other content. This 2500 MDR may not be the appropriate rating (i.e. 2500 may not be the appropriate PB) when mixed with a 100 nits LDR ad, at least because of the huge difference in dynamic range of the two images, but also because of the possibly extreme distribution of object luminances in the LDR ad (i.e. 2500 nits may not be the best LMC value for the CombRng, but then another value can be chosen). High dynamic range should be understood not only as “high”, i.e. bright, but also in terms of “dynamic”, i.e. where on the luminance axis the creator has assigned the various objects and/or semantic regions in the image(s). Contrary to popular misconceptions, displaying an image at simply high brightness can still result in a rather flat LDR appearance, and even an irritatingly bright one, so tighter control over all or many grey values is desired. For example, the ad creator may have made the LDR ad extra bright and eye-catching, to give it a high visual impact on 100 nit LDR displays. But this impact may be too high even in the 2500 nit dynamic range (let alone on a 5000 nit PB_D display for a viewer who has purchased such a display), so for example CombRng taking into account both the need to display very bright lamps in the film and the extremity of the LDR rating, may for that case need to be, say, 1500 nits LMC. The apparatus may use methods such as measuring the overall contrast of the LDR image, e.g., counting how large the number of pixels above k% = say, 80% luminance in the LDR image is, how often high and extreme the average luminance is, or how the number of pixels above k% varies, etc., and from that determine a measure of extremity, and from that determine, for example, how bright the brightest parts of this LDR commercial might become when mapped to the CombRng, and, for example, determine what LMC should be (e.g., the brightest of the LDR pixels might become 300 nits, depending on the nonlinearity being used, or depending on how much brightness is needed to match the luminances of the face, and the brightest Lambertian pixels in the HDR film (i.e., identified in the lower range of that film, e.g., as 5x the coded middle gray of that main region for the HDR film) might, for example, become k times more or less, i.e. 200 or 450, and the brightness of the lamps can become 5 times 300 nits, giving a LMC=1500 nits).

The skilled reader understands that there are several ways to arrive at the CombRng (or anc), for example it may be determined primarily by the HDR movie, and then a determination of how to coordinate the LDR image with the HDR movie assigned to that CombRng (as taught in the first examples described above), or as exemplified here it may be determined primarily or largely by the characteristics of the LDR image, while protecting sufficient quality of the HDR images by determining how far the brightest HDR objects may extend above the brightest or average colors in the LDR image, or iteratively until a reasonable optimum is found for the two, etc. That all depends on the needs of the particular situation and the behavior of the particular combining apparatus embodiment. Which of the two images is of the greatest importance may be preset in various ways, for example the viewer may in some embodiments with his remote control indicate that he wants minimal disturbance of his movie by any image being combined, or content creators may give direction about their content, which the combining apparatus ultimately decides, etc.

Returning to Figure 3, a first image source 350 provides the first image(s), which we assume to be an LDR image Im1_LDR, i.e. encoded in a standard manner, e.g. according to Rec. 709 (of course, there may be a decoder assembly, not shown for this). It is possible that, in case it is a recent video, it has been annotated with its own F1_L functions to convert it to HDR, or that it has not been annotated, in case it is an old SDR video. That is, it may have associated F1_L functions in its metadata, allowing artistically optimal conversion to other dynamic ranges than its native DR of 100 nits. In case there are missing features, the combining apparatus can always apply functions and estimates that work reasonably well on average. Again, the image source 350 may be multiple sources, for example, a security camera at the front door of the viewer's home may play an image when someone rings the doorbell, etc.

As explained, according to one of several possible algorithms, a dynamic range setting set (302) analyses the situation, for example the luminance characteristics of the first and second images, and sets an appropriate combination luminance dynamic range (CombRng), in the above example with LmiC=0 nit, and LMC=1500 nit. A luminance anchor determination set (303) will determine an anchor luminance (ANC). For example, it reads from the first image source in the metadata associated with the first image a source anchor luminance (L_SA1) of the type "face luminance", this value being for example 45 nit. Knowing that for at least one of the faces in the images it seems to be important, you then need to establish what a good face color would be and in the CombRng (of course other embodiments may first decide on an anc even without looking at what source anchor luminances where specified, or when you see that the anchor luminance was for example an average gray in a bright outer subregion, you may determine for the CombRng an overall average gray and a d_anc position at which to position bright outer luminances compared to an overall neutral representation of colors in the CombRng. Depending on the complexity of the embodiment, a typical embodiment may want to check what the value of a second source anchor luminance L_S2A1 of type face color is that indicates how the face colors are currently in this shot of the movie. On a normalized luminance axis this will of course be a lower value, because the upper range is used for HDR effects like illuminated objects or regions, or lamps. Still, if a reasonable value in a diffuse object range up to, say, 400 nit would be 180 nit, and we find a value of only 30 nit, we know we have either a naturally dark face or a dimly lit face, or both. In either case, even if that face were to look darker than the commercial's, it will be much darker if we choose to put our anc value at, say, 180*1500/2500. So we might want to calculate an anc value that is somewhat closer to 30 nits, put the LDR face luminance at or around that nit value, say, 0.8*180*1500/2500, and have the HDR face colors mapped closer to that anc value as well, say, 0.3*180*1500/2500. This ensures that we have appropriately dimmed faces for the lDr image, and appropriately dark faces, as they should be according to history, for the HDR movie, and both coordinated to the optimal anchor value. Of course, the same can be done by taking into account other anchor luminances or additional luminances, such as, for example, a mid-gray for a specifically HDR-lit region.

Having set a CombRng with an anchor luminance anc, the only thing left to do generically is to optimally map the two images across that dynamic range according to a specific luminance value that is determined because it must fall at or around the anc value. One way to do this is with a linear mapping of the luminances, with an optimally determined contrast value C being the slope of the line of the luminance mapping function, and then mapping luminances by computing the linear equation until clipping occurs (and the amount of clipping can be controlled along with the C value - note that clipping can be done to some value within the CombRng, i.e. below the maximum LMC value). The down mapping to the CombRng in this example can also be done by taking into account the same mathematics that the decoder 251 uses, namely, by using at least one luminance mapping (F2_L) with a shape that encodes the behavior needed for the brightest and darkest parts of the image, as typically determined by a human grader. For example, once the CombRng is known, one could have the pixels corresponding to the second image mapped by the decoder 251 immediately from the 5000 nit HDR image Im2_HDR. The color transformation set (310) is arranged to perform all necessary pixel color transformations, in particular at least the appropriate luminance adjustments, such that the combining set (320) can apply simple algorithms (e.g., be a simple additive weighting set or pixel replacement or switching, because all colorimetric optimization has already been performed). A source anchor luminance readout assembly (311) will read at least one source anchor luminance associated with one of the two images, for example, a face luminance from the LDR image, or a good average value for blacks in the image scene, etc. As shown above, in some embodiments it may also read a number of source anchor luminances, which are characteristics of special luminance regions in the images (e.g., a special gray value, or a gray value associated with a common semantically important object, such as an average bright or dark sky luminance, etc.), from both or, in general, from all of the images. Since in some less simple scenarios it is not desired to map the luminances of the two images based on matching just a similar anchor luminance (anchored to anc), but rather it is desired to intelligently position the output luminance in comparison to anc, an offset determination set (312) is arranged to determine an appropriate luminance offset (d_anc). As exemplified, this may be, for example, because the advertisement contains a highly contrasted, brightly lit face, and the horror film comprises a muddy dark face, and, for example, if anc is determined to be close to the luminance of the dark face of the film, then a sufficiently large positive offset d_anc is desired for where to map the face colors of the LDR commercial face. The correctly intermediated luminance mapped images Im2_HM and Im1_LM will then go to the combining set (320), which can then be, for example, as simple as a pixel switch, which depending on the (x,y) position of the pixel puts the mapped HDR color (i.e. Im2_HM) into the output image Im_o, or the LDR color Im1_LM, either as linear RGB, or recalculated into a Y'CbCr, etc.

Figure 4 makes things clearer for the chosen example by showing the mappings into the dynamic luminance ranges. Im1_Rng is the 0-100 nits LDR dynamic range of the first LDR image, i.e. it has encoded luminances that must lie between the minimum luminance Lmi_L= 0 nits and the maximum luminance LM_L= 100 nits. Im2_Rng is for example the dynamic range of the HDR image with LMH=5000 nits (or it could be the MDR image optimized for display, in which case LMH would be 2500 nits in the example above). From this it can already be seen that there is no need for the CombRng to be in any way the same or close to either the Im2_Rng or the display range Di_Rng. This display range is shown for comparison, but in some embodiments it may not even be used in the blending, but instead Im_o will go to a display tuning color transform set to map the blended output image Im_o defined up to, say, 3000 nits into a dynamic display range of, say, 1400 nits, or whatever the values are (the skilled artisan understands that there may be scenarios where LMC is greater than LM_MDR, for example if the blending was performed on another set, say, a cable headend, and a particular user has an HDR display with limited PB, say, 950 nits; or, in case the blending occurs all at the viewer's location, and the viewer has a high PB display, say, 7000 nits, but is currently receiving HDR program content such as a news program with Pb = 1200 nits, or less, and an LDR image, then it may be determined that LMC is much lower than LM_MDR=7000 nits). Tuning functions can be, for example, those of the HDR movie content (i.e. F2_L), as the LDR ad will already have been sufficiently harmonized to it. Of course, more complex implementations can directly optimize both images to the dynamic range of the display. The reader will understand that there are situations where the CombRng may not even be, for example, the dynamic range of the display, for example at a video production or streaming site, where the specific details of the viewer's TV are not yet known. That is one of the complexities that the HDR market has created, and which will need to be addressed. In the LDR era all TVs were roughly the same (with PBs of around 100 nits) so video handling was simple, but now some viewers might have a 1000 nit HDR TV, others a 7000 nit HDR TV, others a legacy 100 nit SDR TV, while others like to view pictures on their I-pad or mobile phone, with a PB of say 500 nits (which still doesn't begin to divulge all the details of the content of the various pictures). It should be understood that because of their large variance alone, those display dynamic ranges may not always trivially be the most appropriate CombRng to combine the two images and render both optimally. But also, especially if we ultimately have to render on, say, a 1000 nit TV or monitor, would it be optimal if we were to upgrade and match an LDR up to 5000 nits of HDR image dynamic range 2, if we subsequently have to significantly reduce it again, and risk, especially when using the F2_L functions, losing a significant portion of the brightest LDR colors?

We explain that a bit further with Figure 5, which uses an example of PIP 501 (but similar issues occur, e.g. with masking and temporal adaptation, etc.) in a main 500 region of an HDR movie.

Classical LDR combiners can do their combining in Y'CbCr (i.e. luma 2 chrominance coordinates), non-linear R'G'B', linear RGB or in principle any color space (although that may be less typical for systems that need to work at video rate but also want to save on computing resources). Suppose we have a gray value image.

In case one simplistically changes per pixel the Y' luma codes of the pixels, that would lead to very bad results, because they are determined very differently for HDR and LDR images (basically it becomes a major problem of misunderstanding of the drive unit). For example, everything seen through the 510 window in the LDR PIP (say it's a director's commentary) may have been clipped to the maximum luma code, Y'=255, or 1.0 normalized, because it's too bright for LDR and not interesting in that single video anyway. By rendering the combined image Im_o in a luma encoding, where some of the normalized pixel lumas of the original HDR movie (e.g. a dark night scene) pixels have been replaced by the normalized lumas of the LDR image, by decoding it with e.g. PQ HDR EOTF, the PIP will look excessively bright (window colors would be rendered as e.g. PB_D=4000 nit, where alternatively in principle the outside object colors could have been well harmonized, or at least less irritatingly bright clipped colors). This will lead to 502 over-irradiation, also called light bleeding or light spreading. If this is not due to the physics of the display (e.g. low amount of backlight LEDs, interreflections on the display front plate), often glare on the human eye can also make very bright regions on an HDR display hard to see without using your hand to block the bright area. In short, the viewer doesn't like it, and would prefer the opposite, if technology made it possible. At least it should be clear that those luminances of the LDR PIP and the dark regions of the night scene in the HDR movie are not well harmonized. But the opposite can also happen. If the PIP is next to a sun in the HDR movie, or another very bright part, it might be too dim. What can also be ugly is, for example, that colors that are clearly supposed to be white, look too greyish. If, as shown on the right, we decrease the luminance of the PIP face closer to the luminance of the dark HDR face, because the luminances of the other objects in the LDR image are related to the color of the face (in contrast ratio), they will also be dimmed more appropriately. Thus, all colors should look relatively well coordinated.

One might naively think that all problems can be solved by assigning the proper LDR subrange to the PIP image (i.e. the 0-100 nits luminance subrange, as it would fall into the HDR CombRng, whatever that is), which corresponds to properly decoding it with Rec 709 maxed out at a 100 nits LDR PB, instead of the (relative) display PB of say 3000 nits, and then making those LDR pixels have luminances up to a maximum of 100 nits at the display dynamic range (assuming the CombRng is taken as equal to the available display DR). But as said, while in some situations that may be a good choice, that makes the decision dependent on what the display dynamic range actually is (viewed relatively, it becomes a random relationship). For a very bright sunlit scene, it might happen that next to the maximum LDR PIP of 100 nits, on a 3000 nits display there are adjacent objects with, for example, a luminance of 1500 nits. This makes the PIP rather disappointingly dark, and its colors will look muddy instead of vivid, which is not what the person who paid for the commercial would expect. The same might happen when blending in the dynamic range of the HDR image (Im2_Rng). It will therefore be understood that proper blending is not a trivial thing, i.e. a good framework is needed as in the present invention. Furthermore, having two HDR images, even if the EOTFs were not that different, one would still be unsure how the content creator used that available range, i.e. colorimetrically designed his total system (content production rules, etc.). Even if the P<b>of the two encodings did not differ too much, e.g. 5000 vs.

4000 nit, the creator might still have made significantly different decisions (for good reasons - e.g. to make their particular story with well-designed HDR scenes look spectacular, versus another creator making a news programme easily convertible to other dynamic ranges - e.g. for viewing on portable displays, though the images would still need to have some HDR) about, say, the lower range that Lambertian reflective objects fall into, e.g. an indoor scene. The first creator might have set those to as low as 300 nits (on the 4000 nit system), while the creator of the second image might have decided for their scene, e.g. the interior is a space station, that it goes up to 900 nits (on the 5000 nit system). Sometimes it can be as simple as whether you like your space stations bright or eerily dark. Even if they followed very strict rules regarding lighting or at least the rendering of face luminances, say 50% diffusive white (i.e. 300 or 900 nits), without the proper harmonizing faces in the mix could look considerably, even oddly, different, possibly even bright. And the same could happen with the many other colorimetrically critical objects that HDR images can contain, say whether a swimming pool looks somewhat bright and lit from within, or just dull water, or how dark a night scene should be, or how foggy a day is, or someone might even care about the luminances on an illuminated sign on the front of a drinks vending machine, especially if that relates to other objects in the scene, and the story and mood communicated, etc.

Figure 6 illustrates two possible examples of the color transformation to the CombRng. We have as input image one of the images to be combined, with relative input luminances L_in. As explained, the apparatus needs to determine a color transformation function (FF_1), which maps an output luminance L_out_comb in the CombRng to all possible input luminances. With this function, for example, loaded into a LUT, one can start processing the incoming pixel colors for successive images. A first simple linear determination yields the linear function 601, determined by the two parameters anc_Fc for the luminance of the face color L_SA1Fc in the input image, and the multiplicative contrast constant C. This function may clip to black or white, which may or may not be desirable (again, depending on how simple the deciding party, whether ultimately the content creator or the apparatus manufacturer, wants it to be). In more complex embodiments, we can improve on that. The luminance mapping feature 602 shows how we can nicely set up the desired mappings based on just a couple of important anchor luminances. That, in general, while not much harder to use, would produce much better results. Maybe this commercial LDR image had too many bright pixels, for example in the background. By applying particular photographic composition rules, one might choose to photograph the main person against a darker or lighter background, for example an average grey background, or a very light or white background, or potentially even a dark background. Often in a video that will depend on what the actual background is (for example, if the video is a report about a fashion photographer who is going to photograph his model against a white background, the video will show him speaking in front of the white walls or background to be photographed). In studios, the background may have been designed. In a weather announcement, the meteorologist may be standing in front of a representation (usually virtual, by green screen) of a photo that has been sent in by a viewer, a non-professional. Or in a movie someone might be sitting in front of a bright advertising display, e.g. an LED display, for some reason etc. This is another example of the need to coordinate, somewhere, because in the past a well coordinated LDR photo was shown in an LDR weather show, but soon viewers might send in HDR photos for shows that are still produced in an LDR manner, or within a couple of years someone might send in a very low quality LDR photo to be shown, thus, mixed in an HDR show. In any case, for curve 602 we assumed that there would be many pixels that were too bright in the LDR image to be PIP-ed. We might not only need to reduce the output values of function 602 for some whites, but also, for example, start making soft clips for the brightest input luminances. Then curve 601 would produce results that are too bright. The apparatus (301) has done this by reading three types of source anchor luminances in the input image and mapping them to the respective anchor luminances in the CombRng output image. Namely, the content creator has also specified a good reference level for diffusive white, i.e., what luminances in his image correspond to Lambertian reflective objects that are white (i.e., with a reflectivity of, say, 90%) and are averagely illuminated. The apparatus will assign for this LDR image an anc_DW luminance for the diffusive whites of the LDR image in the CombRng, which is explicitly lower than what the linear luminance transformation curve 601 would produce. Thus, the combining apparatus can now intelligently decide how to luminance transform such bright image pixels, given what it has determined to be particular needs for those brighter pixels in the combining image. Similar appropriate behavior can be specified at the dark end, e.g. in this case to avoid clipping, by specifying a priori in the content a black anchor luminance L_SA1RBk, and determining at blending time an appropriate black anchor luminance anc_RBk. The renderer may then, for example, decide to continue the function linearly to absolute black (i.e. 0.0) from this point anc_RBk, and have it rise with a very slight slope above anc_DW. Or it may optimize blacks for a specific rendering situation, with knowledge of the source black anchor luminance L_SA1RBk.

We clarify a couple of more complex possible examples with Figure 8, namely how some of our apparatus and method embodiments can not only design multi-segment functions based on various anchor luminances, but also determine the shape of portions of the luminance transformation (FF) curve based on the reclassification behavior as specified in the color transformation functions (F2_L and, if available, F1_L) as jointly communicated to the image from the authoring side in metadata.

We can assume that the function (FF) determining between the three anchor luminances happens again as before (by linear or non-linear interpolation), but that the apparatus is going to determine the luminance transformation behavior of the advanced luminance transformation function 810 in Figure 8B based on the knowledge obtained about the gradation by the content creator from the received metadata (F2_L). F2_L is now the luminance transformation (for the second image, but the same can be understood for all other images to be combined into the combination output image) of the second image, from its native dynamic range (DR1) to another pre-specified dynamic range (DR2), i.e. the input luminances L_DR1_im2 are related to the output luminances L_DR2_im2, which is another dynamic range aspect for that same photographed HDR scene. We can see from the graph that there is a main part (between anc_DW and anc_RBk), which is well represented (will be represented) with sufficiently bright and contrasty luminances (for the blending situation), but that blacks (below L_SA1RBk) can be easily discarded at least for some aspects corresponding to some dynamic ranges, because the function quickly drops to zero, and clips. That is, in this example there will be no very important objects in this sub-range of the darkest luminances. The apparatus can use this to determine its appropriate function 810, e.g. by quickly dropping the function for these dark blacks, and it can decide this based on how dark or contrasty you want the blend, or at least the second part of the image in that blending image to be (e.g. based on how many pixels with luminance below L_SA1RBk there are in that second image, or after appropriate blending, e.g. scaling, in the output image Im_o, etc.). That is, the combining apparatus analyzes the reclassification behavior of two received classified images' encoding (i.e., how darker pixel luminances are handled when reclassifying from a master PB_C classified image of, say, 5000 nits, to a SDR classified of 100 nits, in that part of the received luminance mapping function shape), and determines a partial shape that is similar to some degree.

We have also illustrated a possibility for the brightest pixels. The discontinuity in the F2_E curve indicates that there are clearly two regions, e.g. the Lambertian objects in the indoor scene and the bright colors representing lamps (or similar situations). Especially if we have lamps, that means we can more freely assign their final luminances, depending among other things on the dynamic range we have available in the MDR display and/or the CombRng. Thus, the apparatus can determine an appropriate average level L_RefB for those pixel luminances and finalize the curve shape 810 as such. Figure 8<a>shows how the received luminance mapping function F2_L is used to map between the master HDR and SDR gradations as seen received, with the corresponding partial bright (802) and partial dark (801) function shape and behavior.

The skilled reader understands that the various embodiments herein will be applicable in many systems and to many combination scenarios, and with the various components in various combinations, but we clarify some additional exemplary possibilities with two Figures.

Figure 9 shows an example of a television production system. This may be a multi-camera studio production (first camera 901 and second camera 902), and some of the live content may come from location, etc. It may also be a field production, with an OB van, etc. The technical director must mix the camera feeds and other video, some of which may be HDR and some LDR, for example, and the HDR videos may of course be contrasted with the LDR videos which are of only one type (the single universal Rec. 709 standard that existed), or they may be of various types and characteristics. He wants to determine the optimal mix, in a mix determining apparatus 903, which may reside in a production control room. Although it determines a combination, we show in this example that this combination need not necessarily result in a fixedly defined Im_o (rather, the technical director need only determine a suitable combination, but the details of that combination may be communicated in various ways to various receiver sites further down the video communication pipeline, with the receivers even wanting to re-determine the combination into a different one, e.g. slightly different from the one preferred by the technical director; i.e., the combination may be communicated as approximate image and function and other metadata, prior to the combination). In that case, an image signal 910 may be communicated with metadata, comprising the two images (e.g., with scaling functions to determine the decided PIP combination), and with the color transformations CombRng, anc, and possibly also FF_1 and FF_2 determined to map the images to the CombRng, so that any receiver (e.g., a broadcast receiver 904, at some intermediate node, such as, for example, in a broadcast control room) may actually apply the final combination. To allow for possible redetermination (of at least one of CombRng, anc, FF_1 or FF_2), the production side may also pass along additional metadata in the video, for example if it contains a lot of bright scenes, which need an additional 3x brightness above the average brightness of the film (i.e. this is how they want this content to be luminance mapped in the final CombRng), details regarding the luminances of objects or regions, or semantic information such as e.g. the luminance range and/or sky location, etc.

Figure 10 shows another possible embodiment, namely an intermediate combining apparatus (1001). This may be, for example, in a broadcast control room of a cable provider or the like. It may obtain the main production image(s) (as created in Figure 9 and communicated as final images or initial images with sufficient metadata to combine them appropriately according to the director), i.e. Im_Prod and, for example, a local commercial feed ImLocComm. The intermediate combining apparatus (1001) may also potentially perform a dynamic range transformation, for example, to convert a 5000 nit Im_Prod into output images encoded with PB=3000 nits, which may be more suitable for the purpose, for example, further distribution over some pre-established video communication system. The latter is of course more interesting if the 1001 apparatus outputs two separate image streams to be mixed later, but in this example, where we assume that a final suitable image stream is generated (only to be tuned by the final receivers), which will typically already have a different LMC of 5000 nit.

The reader can also understand how similarly a consumer-side system can combine HDR images or videos, for example a movie together with images received via a computer, and combined in an IC of a set-top box, personal computer, etc., or how the same can be used in video conferencing rooms, etc. The image in Figure 3 can go to a TV, one of the images can be received on a Blu-Ray disc via a BD player, etc.

Figures 11 and 12 offer a couple of possibilities for determining the CombRng. The luminance axis has been given in Log_base2 sets, because this gives a better correlation with human visual luminance and brightness impressions of the images if rendered. The algorithms for determining the various pixel luminances can also advantageously be formulated in such a logarithmic representation, indeed some of our preferred video coding embodiments also work on such perceptually uniform luminance representations. In these embodiments, the apparatus has already determined a good value for anc_FC. For example, the apparatus assumes that the LDR value of about 30 nits will be a good value, also for <h>D<r> representations. The apparatus will now analyze the luminance distribution situation of the two images, combined with the details of the combination, to arrive at good choices for LMC and LmiC. The device will have rules for this, which can be selected and varied, for example, by the viewer control settings. The first two examples are where a higher dynamic range image (Im2, shown with the thicker line) is the main image, and a lower dynamic range image (not necessarily SDR) Im1 is a PIP, for example, occupying U of area (which is one of the control parameters that can indicate the visual dominance of objects or regions of the Im1 compared to the objects in the scene of Im2. In these examples, the apparatus will want the HDR aspect of Im2 to dominate, so it will determine the CombRng and the placement of Im2 on it first, and as a guide, and then place Im1 in a harmonized manner on it. In these embodiments, the face colors have already been assigned to the optimal final face color anc_FC, so the remaining question is about the assignment of the rest of the luminances. This will be influenced by both the extremities of CombRng and the color transformation functions arising from them to coordinately adjust the luminance on the other luminances, and the variability allowed to change the shape of those functions as required by the impact of brightness. In Figure 11A, the HDR I m2 has a histogram that already fits well to a typical scene with a face when rendered around anc_FC, i.e. the face in Im2 is normally illuminated. We see from the long tail to the blacks that there are dark areas in the scene, for example this may be an unlit room somewhere behind the actor. It could be that some of the pixel colors were not visible in some renderings because they are indiscriminately dark, but at least they are encoded and calculated by the apparatus, so that any final rendering apparatus (e.g. a TV tuning its MDR interval and vendor-specific brightness processing of the darkest regions) can use this data appropriately. We further see that there is a bright luminance mode HiLaIm2, which may correspond to for example a lamp, in which we would ideally like to see some structure (e.g. how the lamp is reflected in its metal fixture). The image to be PIP-ed, Im1, has a feature with a high pixel count peak HiModIm1 near white (the brightest luminances in that code). This may be because the person (who also has a reasonably large area, as can be seen from the near mode anc_FC, where a typical face color should be in typically brightly lit images) is standing in front of a large, bright screen, e.g. a light box, which is near white or even clipped to maximum white potentially. This peak has a high chance of being too bright and disturbing in the combination. However, in this first example, because the face colors are in both images in normal positions, and there is already a reasonably large amount of brighter colors in the HDR image (both the diffusely lit object mode, e.g. below the main set of lamps, and some HLaIm2 highlights which are 100x brighter than the face), one can put the HiModIm1 in its normal position (i.e. the same number of stops above the anchor in the CombRng as is encoded in the Im1 when decoded to a linear luminance representation, which is also in this example a normal decoded value of around 100 nit because the face luminance is in its normal LDR position), because there will be some normal luminance which could also be seen in Im2, i.e. it will not be too inharmonious. This also means that the Im2 highlights can be kept in their normal positions (because no re-matching or re-coordinating is needed, as all matched colors fit the luminance distribution profile of the current HDR image, which has its highlights where they are for this image), i.e. the set can select LMC to be the upper limit of the HiLaIm2 luminance mode, or the maximum brightness of the HDR Im2 range in case it has no luminances up to PB. That is not necessarily so, but it can be a reasonable choice if no further details are known, e.g. regarding typical usage further down the line in the HDR image handling chain). In Figure 11B we have a similar situation (the same Im1 is PIP-ed), but now the Im2 has a different histogram (e.g. a later shot from the movie). Here faces should be rendered (i.e. if they were rendered on their own without any combination with other image content) relatively dark, e.g. at 4 nit. It could be for example that a person is standing in the shadows. In this example the fixture doesn't want to make the faces in the PIP that dark, but still wants a more coordinated darkness for the faces in Im1 as well. The fixture decided to take an Anc_FC setting one stop above the luminance of the faces in the HDR image, i.e. 8 nits, and render the Im2 faces one stop below Anc_FC (since they should retain their dark look, to maintain the artistic intent of this movie scene), and (depending on among other things the size of the PIP) render in CombRng the faces in the LDR commercial Im1 1 stop above Anc_FC (still clearly a brighter part of the image, but not too bright). By keeping the ratios of all object luminances in Im1 (the linear mapping via the anchor luminance anc_FC) this would already reduce the luminance of the large region HiModIm1 to less than 60 nit instead of around 100. Still, when the apparatus judges the impact of brightness (which it can judge purely on luminances, e.g. by calculating a CHL contrast between some reference color of an important part or object of Im2, e.g. the face color Anc_FC, and this HiModIm1 luminance, or also by spatial analysis, e.g. by weighing a larger, closer highlighted area of Im1 as a more serious contrast to the important, e.g. central area of Im2), it may still consider the large bright area of Im1 to be too bright compared to the rather dark look of most of Im2. So you might decide to reduce the luminance in this HiModIm1 mode by 1 stop, bringing it to 30 nits, which shouldn't be too distracting for a small PIP in the top right corner. As shown above, the apparatus can do this in a number of ways, for example by reducing the contrast factor C for the luminance transformation to CombRng above anc_FC, or some non-linear function that can be calculated based on some desired maximum LAMX position for the brightest possible pixels in Im1, potentially further determined by the positioning of some diffuse white anchor luminance, etc. Note also that the apparatus in this embodiment has also performed some harmonization processing on the black luminance subrange. Because the blacks of the HDR image are very deep (and one can assume that several displays can render this, at least when the algorithm is applied before display optimization - if running with a display connected, or even on-screen of course the apparatus can take into account what the display can actually render with sufficient visual quality, i.e. visibility), the apparatus can also deepen the blacks of Im1, to make both sets of blacks more harmonious. It can do this by stretching for example all luminances below what was encoded for Im1 as LA_Bk (some black delimiter), e.g. by stretching the local contrast by 2x. It can be seen in Figure 11B that in this example keeping the brightest regions of the HDR image can still be useful, i.e. the LMC is again determined to the maximum value needed for faithful rendering of Im2 on displays that can do so, because the problem was mainly in coordinating Im1 in a luminance subregion that is much deeper. Note that the apparatus has chosen in this example to limit LmiC somewhat above the darkest colors in Im2, i.e. X stops below anc_FC, because it judges that colors that are too dark will be ignored by the viewer in the blend anyway. Mapping color transformations to CombRng can take this into account and lighten the darker colors to bring them into CombRng.

Figure 12A gives an example where Im1 (i.e. in our simple clarification, the commercial) becomes the dominant image. Perhaps viewers do not want their movie to become small when the commercials start, but the PIP could be, for example, a second channel to watch, or even containing the initially watched movie Im2 changes to a small PIP if something important starts on a second channel that the viewer would also like to watch (e.g. this image of Im1 corresponds to the start of the news that the viewer does not want to miss because an important news item is expected). Now suppose Im1 is an SDR image, but someone (e.g. the content creator, say a commercial, may have specified the following desideratum in the metadata "render brightest luminance at 500 nit, on PB displays of at least 1000 nit", or the viewer may have chosen some user setting, indicating how dynamic or conservative they want to view the brightnesses of their content, even if SDR) has specified it to be rendered at higher dynamic ranges like very bright, since there is room for that on brighter displays. So if decoded normally according to Rec. 709 we would get the histogram Im1Nat, but in reality, we will get in the CombRng to be rendered the histogram Imladj with values for Im1 in the combination up to 500 nit (at least when the desideratum of the SDR content creator is fully followed by the combination apparatus, because for example it is set to a display mode for free content, paid for by commercial owners). Figure 12A below provides a calculation realization where for example a younger viewer has set his system to dynamic mode. In order to be able to contrast spectacularly with the large bright area coming from Im1, the apparatus needs to set (at least in this interim specifying CombRng, whatever display setting may happen to it) the bright lamps to for example 8000 nit, i.e. it needs to set LMC to 8000 nit (even when it was for example 4000 nit in the original Im2 decoding). Figure 12b is a contrasting realization to the first possibility because the viewer has set his system to conservative (because he doesn't like to see overly bright images). Now that's another harmonization consideration, where the brightness of the SDR content is primary, and bright enough, so the reader sees that this is a very different result from the CombRng's LMC derivation. Because there are already so many pixels of Im1 that are very bright, contributing to the overall high judged or experienced brightness of the combined image Im_o, the set may wish to reduce the luminances of the HiLaIm2 bright HDR mode and consequently LMC to 2000 nit only (even though the movie was, say, master-rated on a 10,000 PB_D rated screen, and communicated as 5000 nit HDR<p>B_C images, say, with an SMPTE EOTF 2084 luma assignment). Although there is now less contrast between the brightest lamps in the HDR image and the large bright region of the Im1 (2 stops instead of 5 or more), meaning the HDR movie will lose some of its impact when the PIP appears, at least the overall image is not too bright. The viewer may wish for this form of blending.

Furthermore, if the CombRng range is specified before anc_FC, the apparatus can look at various aspects of the pixel and object luminances in both images, e.g. specify one or more corresponding typical luminances. For example, it can look at a percentage of pixels in both images k times above middle gray, or above 1 times below the maximum luminance of the decoded images, and decide from these values in both images what would be good values for various typical luminances, the maximum possible luminance in the LMC combination at least to begin with. If an embodiment of the apparatus uses a typical value of the brightest object luminances for each image, it can determine how they would relate when mapped to CombRng, i.e. for example what luminance contrast they would have, and propose a more harmonized contrast. Of course, other values, such as representative middle gray values of most of the colors in the scene (middle gray may have a true reflectance of 10% in some typical scene illumination, but can also be used to indicate, for example, a mode, majority or average value, etc. of the pixels in the image), can help determine what the various interregional contrasts will be in the combined image, e.g. the brightest parts of Im2 versus the middle gray part of Im1, etc., and choose harmonious relationships between them. For example, some apparatus embodiments may be configured to always choose some typical contrast values, with the bright, dark, and average regions mapped to some typical luminance subregions of CombRng (which may of course differ depending on whether LMC is 1000 nits, 2000 nits, 5000 nits, or 10000 nits or so), and those values may be biased (e.g., mapped k stops higher in CombRng) when some kind of characteristic image is detected, e.g., night scene, well-lit news show, night in nature, etc. By determining a good ANC value for, say, a good face or gray brightness (or bright light regions), the apparatus may measure what viewing environment the viewer is sitting in. For example, the forward-facing camera in a TV may summarize the brightness characteristics of the environment into one or more characterization values, which may be used in equations to scale the anc_FC value, etc. (e.g. 1 or more stops up or down), and the contrasts needed for other luminances compared to the anchor luminance(s), etc. For example, the camera may see that the viewer (by face detection) is sitting on a bus under some local illumination, and may determine a second region to be the walls and other objects behind, generally worse illuminated. If a single brightness value of the characterizing environment has to be derived by a realization, the brightness of the surrounding walls will be valued higher in the calculation (e.g. a*surround_average_L+b*local_spot_average_L), because the local illumination will generally come from above, and will influence less of the viewer's experience of the environment (e.g. what the brain establishes as deep black in that environment, which the screen is only a part of). From these various clarifications, it should now be clear to the reader that in the various embodiments, there may be various ways of arriving at the totality of CombRng, one or more luminances and forms of luminance mapping function, and indeed, in the end, a harmonious output image for each situation.

Figure 18 simply shows some typical generic possibilities of elucidating certain color transformation functions FF to be applied to the luminances of at least one of the images or parts of the image to be blended (or some or all of them having their respective FF_x functions), and from these examples the reader can imagine other possibilities following the general concepts of our inventions and their various embodiments. For example, Figure 18a shows two simple ways of determining a function FF_1 composed of two parts (in this example, ending at the exact endpoints of the interval, which can be deviated in other scenarios). In case the luminance of the output result (LF1_o) of applying the function to an input image luminance equal to the source anchor (L_SA1) has to be exactly equal to anc, we obtain from this internal program of the apparatus the drawn function. In case the output luminance LT2_v can fall in a neighborhood, a somewhat different two-segment function similar to FF can be determined passing through the point with the value Y anc/k, where k is, for example, 1.5, 2, 3 or 4.

Figure 18B shows that complex shaped functions can be designed around this, e.g. normally/weakly illuminated middle gray anchor, with e.g. an S-curve for blacks. The combining apparatus can propose such a luminance mapping function part, e.g. by typically looking at the re-grading luminance functions (e.g. the fine-grading function) that has been communicated according to our coding principles to get one of the dynamic range images from the other, actually received one. If the combining apparatus sees S-type behavior for blacks, it can understand that there appears to be a region of important objects in the middle of the blacks, which should have good strong contrast, while the deeper blacks appear to be largely ignorable, because the function shows that they can be nearly clipped to the minimum black (0, or whatever Min-Black is). So somewhat smarter combining program implementations will follow this behavior in their given FF form, but now the function is reconfigured because the mapping is not between the 5000 nit HDR master luminance range and a 100 nit SDR range, but to say a 500 nit CombRng (so there will be a need to transform black a bit, according to similar needs of mainly making sure mids are rendered better, but it will be a different thing). Similarly, for the function part for highlights, the combining apparatus may decide to follow the shape specifications corresponding to the critical luminance subranges of image objects to a more or less precise degree. Figure 18c shows an example where the combining apparatus controls the contrast C of some "middle range" of the SDR image, e.g. to be combined, and also in this example the apparatus decided not to stretch the top of the Ff function all the way to the maximum of the CombRng (of the output image Im Cmb), as in realization that could result in too bright SDR pixel representation (the reader should understand that HDR image luminance histograms typically have very spread out lobes, with the lobe of the brightest luminances, e.g., containing only 20x5 pixels of 20 small specular patches, while the SDR image contains tightly clustered histograms, with many luminances around the maximum PB_C=100 nit, e.g., even potentially large image regions of outdoor scenery clipped to white, and certainly in a relatively dimmer or darker HDR film, those regions of combined pixels should not be as bright as PB_Comb= 2000 nit, so as not to destroy the pleasure of watching movies). It should be appreciated that in the earlier days embodiments of blending apparatus may have only a couple of such sensible pre-programmed rules in their software or hardware, but further embodiments in the future could do complicated analysis, to arrive at finally balanced image blends. Thus, several different embodiments of the system can be designed around that central apparatus that handles the correct image or video blending, for example to allow a human to specify, long before the (often unknown) blending actually happens, in an elegant and not too cumbersome way, what his content needs, by selecting a good value of the anchor luminance of his content. Several embodiments can occur in the blending apparatus, for example when the apparatus uses the same anchor type (although some apparatus might also decide to use a different anchor type and convert the harmonized mapping of the two images from their anchor types into the available luminances of the blending dynamic range). The skilled reader can understand that in practice the first situation can be incorporated, for example, in a mixing apparatus operating in a sort of slave mode, by checking that at least the main video, for example, defined what was its critical face anchor luminance, then establishing what would be a good face luminance in the blending dynamic range (e.g. already taking into account the final rendering characteristics), and then transforming the luminance of the second image (even if it does not contain a face), to be in harmony with that luminance of the important face in the first image (i.e. how it was rendered in the blending dynamic range). An example of the second strategy, where the blending apparatus is leading, could be for example if the user is looking at photos on TV and you have set some reference for that (by defining an anchor luminance as a single representative value for the typical overall brightness that photo representations should (approximately) conform to; and typically also a presentation dynamic range - e.g. you have defined a not too bright blending dynamic range (soft display) already with the photos' luminance mapped into that range, before other image data comes in and is harmoniously blended - and you may have also specified that the photos currently use mostly half of that range, i.e. all our pixels have luminances below PB_Comb_PhotView/2, although there is some room in the blending dynamic range for brighter pixels, if secondary content wants it). Blending of secondary image content (primary content being photos already being mapped under a given luminance mapping) will next happen by the appliance looking at the anchor value in the content (which can be different from the photo view's anchor type, but by a blending appliance formula calculation will be related to it, e.g. anc_face_inmix = 1.3*anc_photviewapplic), and the photo view's blending dynamic range set with luminances between e.g. 0 or k nit and PB_Comb_PhotView. It is very useful to have a single (suitable for all situations) generic HDR image content blending system that can handle multiple situations, and the determination across multiple species will depend on what is optimal for the situation and what is already known by the appliance at any given time.

For example, in case the final rendering system is not yet known, the best a mixer can do when it needs to harmonize the two image contents is to at least see how they are best blended (e.g. if one of the images has excessively more (perhaps large) high brightness regions, which need to be dimmed in order to enjoy the other image content). In that case, the final display optimization stage (e.g. on a consumer TV) may still need to perform more complex optimization, taking into account the already stated images as how they should be (generically) optimally blended. However, in embodiments where the rendering environment is already known, and in particular when it has strong desires, the mixer may balance more towards the final look (i.e. the visual quality needs of the Combrng determined by the display by a large factor, and e.g. sacrifice some of the perfect single representation of one of the images, by doing a different mapping). In case rendering capabilities are of primary importance in determining the CombRng, typically at least some typical considerations will be used regarding the expected properties of, say, HDR images (e.g. the rules of the combining apparatus software determining that likely most of a typical "generic" HDR image would reside (+-) below, say, 500 nits, and bright objects should be renderable if assigned to be displayed at luminances of, say, up to 1500 nits. If a display, for example, is only capable of only PB_D=1000 nits, that would correspond to non-optimal misrendering of good quality HDR images (i.e. bright bursts of, say, 1400 nits would be too dim). In such a situation, although the actual PB_D of the available display may have a large impact on determining the CombRng's PB_C for blending the image content prior to rendering, the blending apparatus might still decide to determine that the PB of the CombRng is somewhat higher than 1000 nits (and apply a final display adjustment of the blended image to map it to the display's rendering range), e.g. 15% higher, i.e. 1150 nits. Of course, ideally in best embodiments, the optimization of the CombRng, and also a suitable prior point in that blending dynamic range, will take into account, and balance, all known factors (i.e., ideal content rendering needs, and all practical requirements and constraints of the current HDR image blending phase). An optimal determination of a point may typically look at what is available (possible) after the optimal CombRng has been previously defined (but as taught, various other factors may be involved, such as, for example, what type of content the images semantically contain, and in some embodiments possibly even what the end viewer expects for such image content, etc.). Thus, various embodiments may set the ANC in various ways, ranging from a fixed setting in the apparatus that (whatever is specified as source image anchor types) will always relate everything to suitable face luminances, over a system that has several ANC type options and chooses the one closest to what was determined in at least one of the source images to be combined (i.e., an ANC luminance in the CombRng of the same type as indicated in at least one of the images to be combined, such as good face color, but in the CombRng, i.e., optimal for the combining situation, which should be understood to be different than for representing each image on its own), i.e., typically the main image, to an on-the-fly determination by the apparatus of what would be a sensible ANC value given all the fiscal details of the situation (content pixel luminance distribution and display capabilities, for example), etc. The latter type of implementation is good if, for example, a face color has to be optimally determined given specific rendering constraints, such as how dark the display can render dark objects, above which the luminance of the face should fall, which can be especially critical for faces in dark regions of the source images. Thus, to be clear, some embodiments of the apparatus may operate by determining some ANC value (e.g. a good average luminance), although the two incoming images may currently be defined with face-color type ANC luminances, because generically all that is needed to make the apparatus work is that they can be related by the apparatus (i.e. that the typical face luminance in an input image can be mapped into the vicinity of the "average type" ANC value of the CombRng, because the apparatus judges that sufficiently good images would be obtained if the faces are, say, 50% brighter than the particular, first and independently set good usable ANC value. Thus, the ANC types of the source image and CombRng must be related or identifiable. An ANC can optimally and elegantly summarize all the colorimetric complexities of an image, and in many situations only one coordination point would be needed for the combination anyway. For completeness, it should be obvious that an anc value in a dynamic range means not any of the endpoint luminances, but rather additional information to that. As for setting a dynamic range of luminance, the skilled reader can understand how that may involve setting an upper luminance, and for implementations that do not systematically take the lowest luminance as nit zero, also setting a lower luminance, again dependent on the balanced needs of known factors such as the amount of relevant image content in the darker regions of the image, the ability of the display system to display dark colors, etc.

The algorithmic components described in this text may (wholly or partially) be implemented in practice as hardware (e.g. parts of an application-specific IC) or as software running on a special digital signal processor, or a generic processor, etc.

It should be understandable to the skilled person from our presentation which components may be optional enhancements and may be realized in combination with other components, and how (optional) steps of methods correspond to the respective means of the apparatuses, and vice versa. The word "apparatus" in this application is used in its broadest sense, namely a group of means allowing the realization of a particular objective, and may therefore be, for example, (a small circuit part of) an IC, or a dedicated apparatus (such as an apparatus with a display), or part of a networked system, etc. "Arrangement" is also intended to be used in the broadest sense, so it may comprise, inter alia, a single apparatus, a part of an apparatus, a collection of (parts of) cooperating apparatuses, etc.

It should be understood that the denotation of computer program product encompasses any physical embodiment of a collection of commands that enables a generic or special-purpose processor, after a series of loading steps (which may include intermediate conversion steps, such as translation to an intermediate language and a final processor language) to input the commands to the processor and perform any of the functions characteristic of an invention. In particular, the computer program product may be embodied as data on a medium such as, for example, a disk or tape, data present in a memory, data traveling over a network, wired or wireless connection, or program code on paper. In addition to program code, the characteristic data required for the program may also be embodied as a computer program product.

Some of the steps required for the operation of the procedure may already be present in the processor functionality rather than being described in the computer program product, such as data input and output steps.

It should be noted that the above-mentioned embodiments illustrate the invention rather than limit it. Whereas the skilled person can easily map the presented examples to other regions of the claims, for the sake of conciseness we have not mentioned all these options in depth. Apart from the combinations of elements of the invention as combined in the claims, other combinations of the elements are possible. Any combination of elements can be realized in a single dedicated element.

Any parenthetical reference sign in the claim is not intended to limit the claim. The word "comprising" does not exclude the presence of elements or aspects not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.

Claims (15)

1. An apparatus (301) for combining two images or two videos of images (Im_HDR, Im_LDR) from different sources and with different maximum luminance, one of them being a high dynamic range image or video, the apparatus comprising: - a dynamic range setting assembly (302) arranged to set a combination luminance dynamic range (CombRng), which is characterized by at least one maximum luminance (LMC) that is determined based on the maximum luminances of the two images or the two image videos, the dynamic range setting assembly further comprising a luminance anchor determining assembly (303), arranged to determine an anchor luminance (anc) in the combination luminance dynamic range (CombRng); - a colour transformation assembly (310), arranged to perform at least one luminance transformation on at least one of the two images or videos, where the colour transformation assembly (310) comprises a source anchor luminance readout assembly (311) arranged to read at least one source anchor luminance (L_SA1) from metadata of a first source (350) delivering a first image or video (Im1_LDR) of the two images or videos, and where the color transformation set is arranged to establish a luminance transformation (FF_1) to be applied to the first image or video that produces a first output image having first output luminances, which luminance transformation depends on the value of the source anchor luminance (L_SA1) by having the property that the source anchor luminance (L_SA1) is mapped to an output luminance in the vicinity of the anchor luminance (anc); and - an image combining assembly (320) arranged to combine output luminances of the first output image with luminances of the other of the two images or two image videos to form at least one combined output image (Im_o). 2. An apparatus according to claim 1, wherein the color transformation assembly (310) is arranged to determine the luminance transformation (FF_1) such that an output luminance (LF1_o), which is determined as a result of applying the luminance transformation (FF_1) to an input luminance of a pixel color of the first image or video (Im1_LDR) that is equal to the at least one source anchor luminance (L_SA1), is equal to the anchor luminance (anc). 3. An apparatus according to claim 2, wherein the colour transformation assembly (310) is arranged to determine the luminance transformation (FF_1) such that a luminance ratio of a second output luminance (LT2_o), which is determined as a result of applying the luminance transformation (FF_1) to a second input luminance (LT1_i), divided by the output luminance (LF1_o) is a multiplicative constant (C) times a ratio of the second input luminance (LT1_i) divided by the source anchor luminance (L_SA1). 4. An apparatus according to one of the preceding claims, wherein the colour transformation assembly (310) comprises an offset determining assembly (312) arranged to determine on the basis of the at least one source anchor luminance (L_SA1) a luminance shift (d_anc), and wherein the colour transformation assembly is arranged to determine the luminance transformation (FF_1) such that an output luminance (LF1_o), which is determined as a result of applying the luminance transformation (FF_1) to an input luminance of a pixel colour of the first image or video (Im1_LDR) which is equal to the at least one source anchor luminance (L_SA1), is equal to the anchor luminance (anc) plus the luminance shift (d_anc). 5. An apparatus according to claim 4, wherein the colour transformation assembly (310) is arranged to read at least one second source anchor luminance (L_S2A1) obtained from a second source (351) outputting a second image or video (Im_HDR) of the two images or videos, and wherein the offset determining assembly (312) is arranged to determine the luminance shift (d_anc) also as a function of the at least one second source anchor luminance (L_S2A1). 6. An apparatus according to one of the preceding claims, wherein the colour transformation set is arranged to establish a luminance transformation (FF_1) to be applied to the first image or video also in dependence on first luminance remapping functions (F1_L), wherein these first luminance remapping functions (F1_L) specify a change in the luminance distribution of objects in the first image or video (Im1_LDR) to map that first image or video from the dynamic range associated with the encoding of the first image or video to a dynamic range with a maximum brightness that differs from the maximum brightness of the dynamic range associated with the encoding by at least a multiplicative factor 2; either wherein the color transformation set is arranged to establish a luminance transformation (FF_2) to be applied to the second image or video also based on the second luminance remapping functions (F2_L), wherein these second luminance remapping functions (F2_L) specify a change in the luminance distribution of the objects in the second image or video (Im2_HDR) to map that second image or video from the dynamic range associated with the encoding of the second image or video to a dynamic range with a maximum brightness that differs by at least a multiplicative factor 2 from the maximum brightness of the dynamic range associated with the encoding of the second image. 7. An apparatus according to one of the preceding claims, wherein the dynamic range setting assembly (302) is arranged to set the dynamic range of combination luminance (CombRng) depending on the luminances present in the two video images. 8. An apparatus according to claim 7, wherein the dynamic range setting assembly (302) is arranged to set the dynamic range of combination luminance (CombRng) further depending on a maximum brightness of a display on which the at least one combined output image (Im_o) is to be displayed, and preferably also on a brightness characteristic of a viewing environment. 9. An apparatus according to one of the preceding claims, wherein the luminance anchor determining assembly (303) is arranged to determine the anchor luminance (anc) depending on at least one of: the combined luminance dynamic range (CombRng), luminances in at least one of the two images or videos, a maximum brightness of a display on which the at least one combined output image is to be displayed (Im_o), and a brightness characteristic of a viewing environment. 10. A method for combining two images or two videos of images (Im_HDR, Im_LDR) from different sources and with different maximum luminance, one of them being a high dynamic range image or video, the method comprising: - establishing a dynamic range of combination luminance (CombRng), characterized by at least one maximum luminance (LMC) that is determined based on the maximum luminances of the two images or the two image videos, the establishment further comprising determining an anchor luminance (anc) in the dynamic range of combination luminance (CombRng); - applying a luminance transformation (FF_1) on at least one of the two images or videos producing a first output image having first output luminances, which luminance transformation is set based on a value of a source anchor luminance (L_SA1) read from metadata of a first source (350) delivering a first image or video (Im1_LDR) of the two images or videos, the luminance transformation having the property that the source anchor luminance (L_SA1) is mapped to an output luminance in the vicinity of the anchor luminance (anc); and - combine the output luminances of the first output image with the luminances of the other of the two images or two image videos to form at least one combined output image (Im_o). 11. A method for combining two images according to claim 10, wherein the luminance transformation (FF_1) is determined such that an output luminance (LF1_o), which is determined as a result of applying the luminance transformation (FF_1) to an input luminance of a pixel color of the first image or video (Im1_LDR) that is equal to the at least one source anchor luminance (L_SA1), is equal to the anchor luminance (anc), or equal to the anchor luminance (anc) plus a determined difference (d_anc). 12. A method for combining two images according to one of the preceding method claims, wherein the luminance transformation (FF_1) is determined such that a luminance ratio of a second output luminance (LT2_o), which is determined as a result of applying the luminance transformation (FF_1) to a second input luminance (LT1_i), divided by the output luminance (LF1_o) is a multiplicative constant (C) times a ratio of the second input luminance (LT1_i) divided by the source anchor luminance (L_SA1). 13. A method for combining two images according to one of the preceding method claims, wherein a luminance transformation (FF_2) of the second image or video of images is determined based on at least a second source anchor luminance (L_S2A1) obtained from a second source (351) providing a second image or video (Im1_HDR) of the two images or videos. 14. A method for combining two images according to one of the preceding method claims, wherein at least one of the luminance transformations (FF_1, FF_2) to be applied to the respective one of the at least two images or videos is determined at least in part as a function of at least one received luminance mapping function (F1_L, F2_L) which received luminance mapping function indicates how the respective image is to be transformed from the dynamic range for which it is encoded, to a dynamic range having a maximum brightness which is at least twice as large or small as the maximum brightness of the dynamic range for which the respective image is encoded. 15. A method for combining two images according to one of the preceding method claims, wherein at least one of the combination of the dynamic range of luminance (CombRng) and the anchor luminance (anc) is further determined based on at least one factor from the set: a property dependent on the luminance distribution of objects in at least one of the two images, information summarizing the luminance characteristics of at least one of the two images, the dynamic range of a display on which the at least one combined output image is to be displayed (Im_o), and a brightness measure for a viewing environment in which the combined output image is to be viewed.