Outstanding BW Open issue for Sigma Merrill RAW (1.6x, 1.7.x, 1.8.x)

[Andy Somerfield]

To clarify the B/W issue:

 

a.) The B/W issue is nothing to do with our use of libraw (we use x3ftools now for X3F anyway - hence the improvements in 1.8.4 quality).

b.) The image is not B/W. It is a perfectly normal, 3 channel RGB image. A metadata item in the files indicates the user preference for B/W processing.

c.) The RGB -> B/W algorithm used by the proprietary processing software is not available to us.

d.) We have a B/W conversion facility in the develop persona which can be used to manually mix the RGB channels to mono.

 

Aside from automatically enabling the B/W conversion, with default settings, in the develop persona (which we are considering doing as we speak) - what else is expected here?

 

Thanks,

A

 

 

[Wumpus]

Thanks so much, @Mark Ingram

I have no problem with issues and setbacks: I'm a developer too, and many years ago, ran a Fortune 500 Customer Contact Centre for a few years, so I'm familiar with the challenges of both diverse backlogs as well as how to properly support diverse users too. I really appreciate your frankness, and am happy to support the efforts with testing if it could make things easier.

It's OK if it takes a long time to fix, as long as there's a plan in place and appropriate levels of communication. I think my biggest ongoing challenge has been the client communication I've received from support, from Day 1 as a customer (in fact, during my trial period too). The only constructive help I've received has been from speculating users, and while that's occasionally invaluable, they are certainly not the authoritative voice of Affinity. That authoritative voice of Affinity support (at least to me) has been vague, evasive, contradictory and even downright manipulative at best.

At this point, I think it would be best to deal directly with someone in development, if you might be able to appoint someone. I appreciate fully how that can be a major productivity interruption, so I will accept much longer response turnarounds, and will minimize my communication to constructive input, to respect their efforts.

This is incredible news about some progress on the Mac platform, so big congrats. I'll be patient and supportive, even if the forward movement is glacial, as long as I can minimize my friction with unexemplary product support.

[Mark Ingram]

3 hours ago, Wumpus said:

At this point, I think it would be best to deal directly with someone in development, if you might be able to appoint someone

I'm the lead engineer on Windows, and Andy who replied above is the lead engineer for Affinity Photo, so you're not going to get better connections than that. 

[Wumpus]

Excellent & thanks @Mark Ingram. I just figured you two might want to allocate a non-lead person from a relevant Dev pod, as I didn't want to presume key contributors would be helping me out, but that's fantastic.

Thank you @Andy Somerfield. The rest of this post is for you, and my apologies up front, as I'm dashing this off rather hastily, but thank you so much for your attention.

Yes, In my unaided quest for a solution, I made a post about a year ago that while many RAW developers & viewers observe the BW metadata switch, Kalpanika was one that did not, so I'm aware of that. And my source of bad information is the sparse, contradictory and incomplete support responses I have been receiving to date.

I'm not sure which Kalpanika is being used (0.56 & 0.57 handle Merrills differently, or maybe its something altogether different), but something ugly is being introduced. Maybe going to intermediate RGB rather than mixing three linear channels... Anyway, I'll stop and respect your time. The following comments are verified up to the latest Windows AP Beta 1.8.4.676 (but also since 1.6.x)

• Grabbing the blue (top) layer only should provide incredibly noise free BW images. Great also if using both Green and Blue layers (the Red layer is the typically noisy one). If I do this through the Develop Personae (as I've posted since trialling AP in Dec/2018 through Jan 2019), it's not. If anything, a person's face might turn jet black if I dial-down pure Red only: most people I photograph don't have pure R:100 G:0 B:0 Faces, so that seems unusual.
• In theory, a BW image aggregating all three component RGB layers should see naturally reduced noise (as if from a 3-stack exposure), but that's also not the case. Ordinarily, well-exposed Sigma images can be used with zero sharpening and zero de-noising, but that's not what I've been finding here for a year and a half.

However,  I agree with your points, in theory. Practice doesn't follow suit, unfortunately.

Yes, these work beautifully for X3F files taken with older Foveon sensors (as I've often indicated here, and supplied images for controls), but I've been asking to fix images captured with the 5 camera models that use the Merrill series sensors. Those exhibit these problems.

Affinity Photo sees so many frequent updates, and has an embarrassment of conveniences - it's fantastic. I've been trying to make a permanent switch to AP for my workflow, but I am still losing out on some of the biggest advantages that Foveon can provide.

Thanks so much!

[Andy Somerfield]

@Wumpus,

 

Thanks for the response.

As Mark said, we had some bugs on the Windows builds which meant you weren't seeing what we are seeing here. Those bugs have been resolved in the most recent beta. We have also made the develop persona automatically enable the default B/W conversion based on the X3F metadata tag.

If you have chance, could you please repeat your tests in that latest build.

 

A

[Wumpus]

Thank you @Andy Somerfield and @Mark Ingram.

I will test out the new version in a moment, and report back here. I just wanted to document some further testing I did earlier that might be helpful, but didn't have any chance to make a post yesterday.

I used identical settings with Kalpanika 0.56 and 0.57. I don't know if you use these public releases or have a special version for Affinity Photo.

Here's the same workflow, changing only the Kalpanika version:

STEP#1: x3f_extract_v056.exe -v -dng -color none -sgain -ocl SDIM6813.X3F   and    x3f_extract_v057.exe -v -dng -color none -sgain -ocl SDIM6813.X3F
(using camera's original white balance, and minimal Kalpanika denoising)

STEP#2: Adobe DNG Converter (12.3.0.493) using CameraRaw4.6 compatibility setting, output to DNG
(don't embed fast load data, JPG preview Medium size, Preserve Pixel Count, Don't embed original RAW, Don't use lossy compression)

Attached are Irfanview screengrabs from above workflow's DNG output, differering only in Kalpanika version

Kalpanika 0.56 (above)

Kalpanika 0.57 (above)

Opening either in any Affinity Photo Windows does not permit removing channels (such as Red) or using only one channel (such as Blue or Green) without causing image artefacts, and also loses all intrinsic benefits (reduced noise, natural microcontrast, effects from 'stacking').

STEP#3: Open in Affinity Photo 1.8.4.676. Pick WB point (I used white rolled flag middle right), adjust Exposure +0.5, Disabling Noise Removal, Develop, Channel Mixer "Grey" using "Intensity" at 120%, Merge at 100% opacity. Using any other form of colour or channel isolation did not seem (to me) as helpful in anyway, as had been suggested.

This still isn't ideal, but is far less destructive than anything else to date, and provides some panchromatic stacking benefits (but still not much noise reduction or microcontrast).

It also shows some of the differences in Kalpanika version output, particular to Merrill (aka Sigma F20) sensors. I hope these efforts were constructive in some way. I have provided the original RAW earlier, so you can follow the workflow/recipe if you like, or I can provide the two output DNGs if you prefer.

I'll try the new 1.8.4.681 now!

[Wumpus]

Wow, what a change, @Andy Somerfield and @Mark Ingram!

I need to do much more comparison work, but the new 'out-of-box' exposure and colour when opening up an X3F is absolutely the best ever. Similar with the BW exposure and contrast.

It's now mind-blowingly easy to work with X3F files for exposure and contrast and colour (normally Sigma/Foveon images require a lot of hand-holding to just get within workable range, and that's largely true for every raw developer).

Noise is possibly the only detriment though: most evident upon pixel-peeping, but still visible at 100%. Perhaps use the Kalpanika denoising? Just early thoughts...

Sigma F13 Sensor X3F opened using non-beta AP 1.8.3.641, with the usual denoising & sharpening turned off.

Sigma F13 Sensor X3F opened using AP-Beta 1.8.4.681, with the usual denoising & sharpening turned off.

I'll point out that I still need to do a lot more work on the first image to get it closer to 'normal' (which the Beta did nearly automatically, just darkening exposure 0.5). The first image required about 10 or 12 changes to get it into its current 'somewhat-close' state.

With moderate de-noising applied to the Beta, there is definitely halo-ing, and only a bit of detail begins to be lost, but it's not too bad.

 

Now that's with the F13 Sensor. Doing a couple fast tests with the F20 (Merrill) sensor, the improvements are almost night and day better. Again, there is more resultant noise than when comparing the laborious Kalpanika->Adobe DNG extractor workflow (meniotned above) to Affinity on its own, but the upside is that, as Andy says, you just open the file or turn on the BW switch.

If you have specific things you would like checked, please leave them with me as some homework. This is a giant improvement, and doesn't seem to ruin other sensor X3Fs handling (I was worried we might fix Merrill X3Fs and breaking other X3Fs). I need to try a more representative selection of sample images.

This is a huge change, and thank you very much!

[Wumpus]

BW images opened in the new AP-Beta are almost ideal out of the box, for exposure, colour balance and curves.

In some situations, it is even possible to adjust the individual colour channels like you can when converting Bayer colour images to BW, without adding artefacts. However, in others, it is still not possible to do so (ugly side effects).

In general, the BW images are much noisier, and have some haloing (even with zero sharpening applied) just through the denoising process. However, with judicious use of denoising, you can get the noise toned down enough that not too much detail is lost and the haloing doesn't jump out at you.

I think it might be a good final effort to change the current denoising (however it's being done), either to return to Affinity's denoising or Kalpanika's denoising. The first comparison is zoomed in to see the shadow noise and highlight patterns (300%). Zero sharpening, but moderate denoising as indicated in the AP menu screenshots.

The next comparison is between A) an X3F processed through Kalpanika to a DNG format, then brought into Affinity Photo Beta, and B) an X3F brought directly into the same version of Affinity Photo Beta, to compare noise handling. Note that I had to increase the denoising for the direct X3F opening and processing in the AP beta, as apples-to-apples, it was quite noisy.

As a control reference, here's a couple of other means of processing the same image. As this sample was high-ISO (don't laugh please): I just used the blue channel to minimize the noise.

Above is an export from Corel Photo X7 (not PhotoPaint), (which uses DCRaw to process the X3F), and using only the Blue channel, converted to greyscale. Zero denoising, zero sharpening.

Above is Blue channel export from Sigma Photo Pro (Foveon vendor's app). Zero denoising, zero sharpening.

A key caveat to all of these is that this photo was taken at nearly the camera's ISO limit (400: don't laugh) in noon sunlight (broad dynamic range), which is almost always extremely noisy.

To provide a broader sample of images, here's the F13 sensor in colour, zero denoising or sharpering, opened in Affinity Photo (non-Beta) (below). Note that heavy exposure and colour processing was required to get it into the ballpark of the output from Affinity Photo Beta (further down):

And above is the same image in Affinity Photo Beta, zero sharpening, but moderate denoising. Only exposure and denoising we required.

In about 5-10% of cases, the default exposure and curves weren't really anywhere close to usable, so required lots of effort to tweak something out of the opened image. However, that was also the case for about 75%-90% of the images with Affinity Photo before the beta, as well as most other 3rd-party raw processors (digikam however was much better than average), so Affinity Photo was not at all alone in that regard.

In about 10-25%, noise was quite high, and individual channels couldn't not be altered without introducing side-effects. However, the exposure and colour and curves were shockingly good at default.

While there were a lot more tests, suffice to say that in about 40-70% of the tested cases, moderate denoising was required, and still had a fair bit of noise that could never be mitigated in the shadows, but it wasn't extremely evident at 75% or lower magnification. Again, exposure, colour & curves were shockingly good at default.

In about 25% of the cases, the noise was very low (barely evident), and the RGB channels could be adjusted independently without adding artefacts. That's the ideal situation.

Here's just one example (F20/Merrill sensor taken at ISO100, zero sharpening and zero denoising, default BW conversion {3-channels to greyscale}). The exposure, colour & curves were shockingly good at default.

Apologies for the post length, but hope it gives more to work with and possibly more specific questions for subsequent tests.

😊