Restoring HDR to AI-Edited iPhone Photos: The Long Version

By David W. Rose

Published ·Updated hdrimage-processingaimljpegiossystems

I sometimes use the latest AI image-editing tools to touch up photos and remove objects. The problem is that the current batch I have tried, including OpenAI and Gemini, strip the HDR layer. That is the pop you see in the brightness of the sky in an iPhone photo. Specifically, it is a layer called the gain map.

Picture formats have changed a lot over the years. While we just see a box of pixels on the screen, more stuff is getting stuffed into the data itself. One of the modern ways to keep HDR photos compatible across devices is to include a standard SDR image and an additional grayscale gain map. Its job is to tell an HDR-capable display how much brighter to render each of those base pixels. Google and Apple ended up with different ways to structure this metadata, although the formats have started to converge around ISO 21496-1.

Now if we return to the image-editing tools of the modern AI stack, they operate on the simpler box of pixel values I brought up earlier. A matrix of pixels. Neural networks love matrices. The current crop just tosses out the HDR data, so if you want to edit a street sign out of a nature photo, the result looks flat next to the unedited ones on your MacBook or iPhone.

Beach photo after HDR gain-map restoration, rendered as an Ultra HDR JPEG
A restored Ultra HDR JPEG. If you are viewing this on an HDR-capable browser and display, the sky and specular highlights sit above SDR white. If you are not, this looks like a normal JPEG.

Apple says iPhones have been capturing gain-map HDR photos since 2020, and Google introduced Ultra HDR with Android 14. It is two parts, remember? The first is an 8-bit SDR JPEG, the kind of file any software written in the last thirty years can decode. The second is a grayscale image called the gain map, stored as another JPEG, which says how many stops brighter each part of the image should be rendered if the display can go brighter than SDR white.

This is not all HDR. Other formats store HDR pixels directly, including 10-bit HEIF and AVIF. The gain-map design is more pragmatic for a JPEG because it does not require a new codec or file extension. Old software can ignore the extra data and show the SDR image.

How the AI editors work

Each AI image editor I have looked at has roughly the same shape. The incoming file gets decoded into an RGB tensor. That tensor runs through some model, maybe diffusion or a segmentation-plus-inpainting pipeline. The output tensor gets encoded back to a JPEG or PNG and sent to the user. The piece that reads metadata, applies orientation, and writes the output is outside the model itself.

If the editor preserved the gain map byte for byte, the map would no longer be correct because the SDR pixels it was calibrated against are no longer the same pixels. The gain map describes the luminance difference between the SDR image and its brighter rendition. If you change the SDR image, the stencil-like overlay of HDR gain may no longer line up. Google's own editing guide makes the same distinction: crops and rotations can transform the gain map along with the base image, but edits to the actual contents may require changing or removing it.

Two kinds of gain-map metadata

One is the Google-Adobe format. Google defined it in the Ultra HDR spec that shipped with Android 14. The gain-map parameters live in an XMP packet in the primary JPEG, in an hdrgm namespace. There is a Container directory with a Primary item and a GainMap item. The GainMap item has a Length pointing at the size of the appended gain-map JPEG, and a Multi-Picture Format (MPF) marker points to where that second JPEG starts. Chrome on a capable display can parse this and render HDR.

The second is ISO 21496-1. The current iPhone files I inspected have an APP2 segment whose payload starts with urn:iso:std:iso:ts:21496:-1, and it appears in both the primary image and the gain-map image. In my tests, Preview.app on macOS and Photos on iOS would not show HDR from the Google XMP alone. With the ISO markers, they did. Current Android guidance now recommends encoding both kinds of metadata for maximum compatibility, so this is not just an Apple workaround anymore.

For my files to render as HDR in both Chrome and Apple's photo apps, they need all of this:

All of this goes at the front of the file in a specific order, before the actual image data starts. It took a bit of trial and error to figure out.

The file structure, for one of my actual outputs, looks like this:

PRIMARY IMAGE (~223 KB):
  SOI (start of image)
  APP0 JFIF                                 16 bytes
  APP1 XMP + Container + hdrgm           1,486 bytes   <- Chrome needs this
  APP2 MPF (offset table)                   88 bytes   <- links primary to gain map
  APP2 ISO 21496-1 marker                   34 bytes   <- Preview.app needs this
  APP2 ICC_PROFILE                         604 bytes
  DQT, SOF0, DHT, SOS, image data...

GAIN MAP (~10 KB):
  SOI
  APP0 JFIF                                 16 bytes
  APP2 ICC_PROFILE                         604 bytes
  APP2 ISO 21496-1 marker + params          91 bytes   <- Preview.app needs this
  APP1 XMP + hdrgm params                  616 bytes   <- redundant, also helpful
  DQT, SOF0, DHT, SOS, gain-map pixels...

Two JPEG files concatenated, with two kinds of HDR metadata woven through the
APP segments at the front of each one.

Building Ultra HDR JPEGs without libultrahdr

Google publishes a reference C++ library called libultrahdr that knows how to produce these files. It works, and for a general-purpose encoder it is the obvious choice. I only needed one narrow operation: combine an already-encoded SDR JPEG and gain-map JPEG using the metadata layout I had tested in Chrome and Apple's apps. I also wanted to understand the bytes. For that job, a few hundred lines of Python were easier for me to inspect and change than another compiled dependency. Code is cheap now, so I wrote it myself.

The module is called direct_ultrahdr.py. It takes the primary SDR JPEG, the gain-map JPEG, and a metadata object, then emits the final Ultra HDR JPEG bytes in about six hundred lines of Python.

The pieces are:

Those constants look like this in the source:

ISO_21496_PRIMARY_APP2 = bytes.fromhex(
    "ffe2002275726e3a69736f3a7374643a69736f3a74733a32313439363a2d310000000000"
)

ISO_21496_GAINMAP_APP2 = bytes.fromhex(
    "ffe2005b75726e3a69736f3a7374643a69736f3a74733a32313439363a2d31"
    "00000000004000000000000f42400028eb29000f4240ffffe317000f424000"
    "28eb29000f4240000d5810000f42400000000a000f42400000000a000f4240"
)

The hex is the string urn:iso:std:iso:ts:21496:-1 followed by some numeric fields. I am not thrilled about shipping byte-literal constants that came out of a hex dump. But this is what the iPhone itself produces and what Preview.app accepts.

With those pieces, constructing the final file is mechanical. Insert the APP segments into the front of the primary JPEG, before the Start of Scan marker. Insert the matching APP segments into the gain-map JPEG. Concatenate. Patch the JFIF APP0 to include the four-byte AMPF marker that signals this is a multi-picture file. Yay, done.

A short sanity-check function I keep around for any file anyone sends me:

def analyze_ultrahdr(path):
    with open(path, 'rb') as f:
        data = f.read()

    first_soi  = data.find(b'\xff\xd8')
    second_soi = data.find(b'\xff\xd8', first_soi + 2)
    if second_soi == -1:
        print("Not an Ultra HDR JPEG (no second image found)")
        return

    primary = data[:second_soi]
    gainmap = data[second_soi:]

    checks = {
        'MPF in primary':         b'MPF\x00' in primary,
        'Container XMP':          b'http://ns.google.com/photos/1.0/container/' in primary,
        'hdrgm namespace':        b'http://ns.adobe.com/hdr-gain-map/1.0/' in primary,
        'ISO 21496-1 (primary)':  b'urn:iso:std:iso:ts:21496:-1' in primary,
        'ISO 21496-1 (gainmap)':  b'urn:iso:std:iso:ts:21496:-1' in gainmap,
        'ICC profile':            b'ICC_PROFILE\x00' in primary,
    }
    for name, ok in checks.items():
        print(f"  {'OK ' if ok else 'NO '} {name}")

If all six are true, the files I generate render as HDR in both Chrome and Preview.

Where the gain map comes from

I have glossed over the interesting part. You have an edited SDR image, the gain map is gone, and you want a new gain map. How do you produce one?

You cannot reconstruct the exact original gain map because it encoded the camera pipeline's HDR rendition, information that is no longer present in the edited SDR image. But you can produce a plausible gain map: one that an HDR display can use to lift the right regions, and that looks like something the iPhone pipeline might have made for an image with those pixels.

The naive version is straightforward. Compute the luminance of each SDR pixel, stretch the bright end of the range, and call that the gain map. This is roughly what synthetic inverse-tone-mapping methods do. It kind of works, but the output is generic: every sky is boosted the same amount, every reflection is boosted the same amount, and the map does not know what a face is. Apply it uniformly and skin tones also get lifted. In the iPhone files I looked at, specular highlights and bright sources get pushed a lot, midtones get lifted a little, and skin and shadows are mostly left alone.

The less naive version is a neural network trained to produce a gain map. The one I landed on is GMNet, from the ICLR 2025 paper Learning Gain Map for Inverse Tone Mapping by Liao et al. It is small, it runs on CPU, and the authors published trained weights and both synthetic and real-world datasets.

GMNet has two branches. A global branch takes a 256×256 thumbnail of the input and extracts scene-level features. From those it produces three things: a small 3×3 kernel to apply dynamically to the local branch, a channel-attention vector, and a single scalar called qmax that is the ceiling on the boost. A local branch takes the full-resolution input and runs it through residual blocks. The two branches meet when the local features are convolved with the global-branch-produced kernel, the channel attention is applied, and the result is upsampled with pixel shuffle back to full resolution. Okay, that was messy, but at the end you have the gain map, along with qmax, which tells you the GainMapMax value to write into the metadata.

The global branch sees the whole scene and decides, for example, that this is a sunset, the sky should get a big boost, and the foreground should not. The local branch has enough resolution to paint that decision onto the right pixels. The fact that the boost kernel itself is network-produced, rather than fixed, is how the same model handles a noon beach shot and a candlelit dinner.

Restored Ultra HDR JPEG of a dimly lit cafe scene, with pendant lights as the bright highlights
A cafe at night. The pendant lights are where the gain map concentrates its boost. On an HDR display the bulbs sit well above SDR white while the wood of the table, the wine, and the faces stay in their normal SDR range. On a non-HDR display or browser, this is just a JPEG.

The output of the whole pipeline is the SDR image you uploaded, unchanged; a grayscale gain-map image derived from GMNet's prediction; and the gain-map metadata (min, max, gamma, offsets, capacity min, and capacity max) that goes into the Google XMP and ISO 21496-1 payload. Those three pieces get handed to the Ultra HDR packager, and out comes a file Chrome and Preview both understand.

Did you notice the HDR photos?

I was wondering which visitors to my sites were on devices that would actually render these correctly.

Browsers expose one direct signal: the CSS media query window.matchMedia('(dynamic-range: high)'), which reports whether the browser and output device say they support high peak brightness, high contrast, and more than 8 bits per color channel. It does not guarantee that HDR mode is currently active. I also collect the WebGL renderer string. That does not tell me whether the display is HDR, but it gives me a rough device and GPU class to help interpret the results.

Among my unique visitors, about 20% hit dynamic-range: high. If I count events rather than unique visitors, HDR-capable sessions account for 40%. The rest are on hardware that cannot go brighter than SDR, on browsers without the HDR rendering path enabled, or on OS versions that have not shipped gain-map rendering yet.

Apps and uploads are messier. Some preserve HDR on upload but only render it inside their native app, not in the web viewer. Some preserve it everywhere. Meta has written about HDR photo support in Instagram and Threads, and the state of the art there is better than it was a few years ago, but it is still platform- and device-specific.

Okay, thanks for reading!