Got a pink splotch in a DNG file that only Affinity sees

[frank26080115]

I got this photo I took with a Sony A6600, I process my ARW files with DxO PureRAW first and that gets me a DNG file, and I open the DNG file with Affinity.

This usually works fine and my typical photos are more nature centric and I've never had a problem.

But this particular file has this huge pink splotch that does not show up in Windows Photo viewer or DxO, I only see it in Affinity. I am in the Develop persona but have touched none of the adjustments. (white balance does say there's a 25% pink tint but adjusting it to zero does not help)

I have uploaded screenshots and the actual DNG file.

Is this a bug with Affinity? Or with DxO PureRAW? Or is something not configured right?

DSC02092-ARW_DxO_DeepPRIME.dng

[Old Bruce]

I think it is with the lighting for the original shot. That background is far too bright and has messed up the rgb values there, resulting in the pink. I see this (rather too often) when I am using my manual lenses and forget to stop down the 5 or so f-stops after focusing, clouds will have that pink.

[frank26080115]

I've been using Affinity for a long time now and it's never done that to ARW files that have overexposed areas

It could still be DxO's fault but Windows Photo Viewer doesn't see it. I actually just tried Camera Raw (Photoshop) and Luminar 4 and it doesn't see the pink either. Only Affinity. So the question is, why only Affinity?

[firstdefence]

Rawtherapee 5.8.0 on Mac also displays pink in the brightest highlights.

See Info re Rawtherapee and the Raw tab adjustments here 

Darktable 3.4.1 displays a very faint pink haze.

 

[frank26080115]

I checked Adobe RGB 1998 vs sRGB and turning monitor calibration on and off and nothing helped

I also did a 30 second exposure pointed at my computer monitor to generate a pure white image, and went through my workflow, and it turns out pink as well, and only in Affinity.

I am looking around DxO's forums and found one person with a Sony RX100 that got purple tones, he didn't mention any other software being involved though.

I can't just magically avoid overexposure in every photo I take, and DxO PureRAW literally has no real settings other than what neural network is doing the processing and I don't really want to change that since the whole point is to use their strongest noise reduction.

 

[Old Bruce]

5 hours ago, frank26080115 said:

I also did a 30 second exposure pointed at my computer monitor to generate a pure white image,

That 30 second exposure is the problem. You probably have a massive dynamic range in the scene.

There are limits with camera sensors being able to capture extreme light dark ranges. I think your lighting setup is exceeding those limits. Once the camera's sensor has captured and stored those extreme, and therefore wrong, values there is nothing you can do. You have to go back and reshoot the scene. Try using a reflector to light up the binoculars or you may be able to use a flash or secondary light to brighten them up so the black and white fall within range of your camera sensor.

Expose the white wall (or window) so it is 4 stops over exposed and see what the binoculars look like.

[IanSG]

3 hours ago, Old Bruce said:

That 30 second exposure is the problem. You probably have a massive dynamic range in the scene.

Isn't that the other way round - everything's clipped and there's no dynamic range?  

[Old Bruce]

Weird things happen with sensors and Long exposure times. Weird things happen with Massive Dynamic Range Scenes which result in the sensor being hit with far too much light relative to the dark area. There is possibly some fold over happening with the high values.

If you look at the image in the first post the binoculars are, to my eye, over exposed by about a stop, maybe two stops.Now add that overexposure to the relatively bright wall or window. The pink area is grossly overexposed. 

I just did a check and I can easily get a shot with a range of 17 stops. That is far wider than what a sensor is expected to be able to handle. I think the reason the pink is not showing in the Finder's preview here on my Mac is due to the bright area being read as White and the Finder not trying to do the math required to correctly mix the R and G and B pixel information. There is a massive amount of overexposure there.

The following is just my opinion:

There is nothing wrong with the software (any of them), the fault is in the scene's very bad lighting. The lighting of the scene has caused some kind of overload of the sensor and the information written to the camera's file is therefore bad.

[frank26080115]

I am not trying to fix the photo. I am trying to identify a software misconfiguration or bug. The 30 second exposure test gave me a small clue as the file size is significantly smaller and so now I will be checking thing regarding DNG compression and ensuring it's not lossy.

firstdefence's clue is that open source software seems to have the same problem, which points to the DNG format specifications being read without close support from Adobe's own engineers and files decoded in a way that's technically correct on paper but fails in practice just a bit.

Still could be DxO's fault too but less likely, they are reading the DNGs right, and their product family is very impressive and likely have a lot of talent working on them.

The best thing for me to do right now is just find a DNG to TIFF converter that handles 16 bit colour. DxO debayer'ed it anyways so TIFF should be fine for additional processing at 32 bit

Bad sensor data from the sensor would have appeared in the ARW, so that's not quite it. Surely overexposure is a problem handled perfectly fine by the camera seeing as the ARW and JPG files come out fine.

My goal is not to eliminate overexposure, my goal is to have a post processing workflow that can handle overexposure. I don't want to manually edit every pink water droplet that reflected the sun later even though the scene was correctly exposed overall.

[RM f/g]

In APhoto on my Mac, I click ‘Assistant Options’ in the toolbar. Then ‘Develop Assistant’.

There I change Raw Engine from ‘Serif Labs’ to ‘Apple (Core Image RAW)’.

When I open your file now, I don't see any pink haze.

I Suppose there's a similar option in Windows.

Worth to give a try.

[firstdefence]

Just tested ON1 HDR 2021 and it opens the file like Adobe Camera Raw (ACR) nice and clean, no magenta.

[firstdefence]

1 minute ago, RM f/g said:

In APhoto on my Mac, I click ‘Assistant Options’ in the toolbar. Then ‘Develop Assistant’.

There I change Raw Engine from ‘Serif Labs’ to ‘Apple (Core Image RAW)’.

When I open your file now, I don't see any pink haze.

I Suppose there's a similar option in Windows.

Worth to give a try.

Works the same for me, I did try changing that but didn't close the file down and reopen it, another little lesson learned. So ON1 must be using Apple (Core Image Raw)

 

[David in Яuislip]

1 hour ago, frank26080115 said:

I will be checking thing regarding DNG compression and ensuring it's not lossy

I dug into this and DxO doesn't produce lossy compressed dngs from either Photolab or PureRAW. I produced a dng out of PhotoLab from a grossly overexposed image and it didn't show pink in Photo. I don't really buy the overexposed idea, an area that's just over or grossly over is still white so where does the pink from?
DSC02092-ARW_DxO_DeepPRIME.dng opens properly in CS2

[firstdefence]

Had another look at Rawtherapee now I have some time and checked out the RAW tab, I knocked the Raw White-point Correction from 1.00 to 1.01 and the magenta vanished...

[IanSG]

2 hours ago, frank26080115 said:

firstdefence's clue is that open source software seems to have the same problem, which points to the DNG format specifications being read without close support from Adobe's own engineers and files decoded in a way that's technically correct on paper but fails in practice just a bit.

I'm not sure about that - Raw Therapee's Highlight Reconstruction option removes the pink spot, which would suggest it's a known problem that AP's not handling.  Also, processing your DNG file with Adobe's DNG converter produces a file that has the same problem  That could be a case of GIGO, but I'd expect Adobe's software to comply with their own standards.

<edit>

Sorry, I'm wrong about the DNG converter!  But why would 3 different programs make the same mistake (unless they're sharing code)?

[IanSG]

23 minutes ago, David in Яuislip said:

I don't really buy the overexposed idea, an area that's just over or grossly over is still white so where does the pink from?

I agree with the "whiter than white" argument, but then how does AP detect highlight clipping?  

[David in Яuislip]

20 minutes ago, IanSG said:

how does AP detect highlight clipping?  

I suspect that a perfectly exposed white object would be shown as clipped, in 8bitspeke once you go over #ffffff it's forever white. Ansel Adams may have had a more eloquent response

Anyway, questions for the OP @frank26080115
what happens if you proces the raw file in Photo?
what happens if you export a tiff from DxO PureRAW?

DSC02092-ARW_DxO_DeepPRIME.dng opens properly in PhotoLab4

Rawtherapee, Darktable (ref @firstdefence above) and Photo have issues, is this a libraw thing?

[IanSG]

23 minutes ago, David in Яuislip said:

Rawtherapee, Darktable (ref @firstdefence above) and Photo have issues, is this a libraw thing?

Raw Therapee uses dcraw and Darktable uses RawSpeed.  I don't know about RawSpeed, but dcraw and libraw have a common ancestry.

[Affinity Info Bot]

The issue "Develop Persona: issues (highlights rendering) with DNG generated by DxO PureRaw/DeepPRIME technology." (REF: AFP-4444) has been fixed by the developers in internal build "2.1.0.1781".
This fix should soon be available as a customer beta and is planned for inclusion in the next customer release.
Customer beta builds are announced here and you can participate by following these instructions.
If you still experience this problem once you are using that build version (or later) please reply to this thread including @Serif Info Bot to notify us.