Jump to content
Mareck

Is 32bits linear workflow well implemented in AP?

Recommended Posts

I explain myself,

After 2 days of trials, watching tutos, topics on the forum, I finally managed to have a fonctional 32bits linear workflow in place.
I've tried tones of LUTs, OCIO layer and 32bits preview configs but, the better I had was a good viewing and a crap export.
Finally I've found a way, perhaps not a good one, but a way.
I've read @Frank Jonen talking about a gamma issue on a spécific question so I've tried the transfer fonction trail.
It lead me to add an S curve shape on top of my layer stack and this time my LUT worked as expcted and the export too.
But why do I need to do something approximative to have all working?

Infos:

-working with 32bits EXR float (full) linear render
-OCIO layer to convert from linear to LOG
-LUT layer to apply a creative Log base look
-S curve shape on top
-ICC or OCIO 32bit preview


@troy_s welcome to the discussion
 

AP_32bits_workflow_stack.jpg

Share this post


Link to post
Share on other sites

A simplified way to describe "scene linear" is how light behaves in a scene. Then it's up to you to decide how you look at that information. Scene-linear itself is neutral. All values go from 0 to 1 in floating point steps. Either in a 32-bit space or just a 16-bit space. Camera data usually comes in as 16-bit float plates that are composited with the — to be adjusted — 32-bit float CG elements in a 32-bit float space.

 

I'd wish Affinity had a checkbox to turn off ICC profiles for 32-bit spaces and leave us in a pure OCIO workflow and a clearer information of how an imported EXR was 'converted' to scene linear. The ICC toggle would need like two confirmation dialogues the first time it is set. "Are you sure? This will look horrible on non-hardware calibrated devices." and "Yes, my body is ready." 

 

Personally I think it'd be fantastic to have Serif as an ACES partner and the Affinity suite have an official "ACES mode" where everything just works and an option to screw it up manually.

 

In ACES we have several camera models and colour transforms that one day all may translate without loss between each other. To start you can translate your scene linear to ACEScg which is scene-linear more or less. If you mostly work with CG elements, it's a good environment to work in. When you export and send your shot to another facility you pick the big one, ACES2065-1. This carries your colour decisions in a large colour space. It also minimises the checking back with colourists whether you need to send them ACEScc or ACESct as they can just use it or convert it before using it. Conversion between cc and ct is a bit iffy still. 

 

To match in Affinity what you see in Modo for example. Good luck. Modo uses rec709 but not really. Its sRGB viewer LUT is somewhere in between rec709 and sRGB. Just make peace with the fact that it'll look different in comp than in Modo. I tried to add ACES transforms to Modo but the setup to get existing scenes to look right was so weird and nasty to replicate in Nuke that I gave up on it.

 

When you export to ICC from 32-Bit OCIO it's always a tradeoff since they have a fundamentally different view on colourspaces. One of the reasons OCIO exists. On a Mac you can install the OCIO package with homebrew and generate the transform you need. That'll give you a lossy LUT (all current LUT formats are lossy, ACES transforms strive to be lossless) or ICC profile that you apply to your file before converting to ICC. Sometimes you have to convert it to 16-bit first before the generated LUT looks right.

 

Gamma is still an issue in Affinity as you might have noticed the hard way. For some inexplicable reason the sliders end at 0 and 2. Maybe that is useful to someone out there. If anybody finds that necessary, fine. But Affinity really needs a sensible Gamma transform. The current one in the Levels layer is just terrible. It sounds like that adjustment layer might eventually be replaced by a good one though in the future from reading the roadmap.

Share this post


Link to post
Share on other sites
4 hours ago, Frank Jonen said:

Scene-linear itself is neutral. All values go from 0 to 1 in floating point steps. Either in a 32-bit space or just a 16-bit space. 

 

False.

 

Scene referred linear is zero to infinity, with 1.0 meaning absolutely nothing.

 

A scene referred image that was mastered under a particular series of transforms will need to replicate those transforms to achieve 1:1 between applications.

 

In the case of your file, start by taking the scene referred data from the EXR and apply the transforms in question. I believe that Affinity had a problem applying Looks via OCIO, and as such, the proper transform would need to manually stack the look into the transform chain.

 

In this instance, the output would be mastered for whatever the output destination was designed for. ACES for example, has many potential outputs. Filmic was designed for a typical 2.2 sRGB display.

 

If one accidentally appends an ICC on top of that chain, the output would become broken as one would be applying an additional series of transforms via the ICC in question.

 

To start, disable any ICCs in your chain, as well as any power functions (Awful term "gamma"). Apply the same OCIO transforms on your image as you were doing in the other application.

 

Do you see 1:1? If so, does the exported display referred format appear 1:1?

Share this post


Link to post
Share on other sites

Thanks to both of you, I wouldn't thought I'll fall in the rabbit hole again ^^

For The scene linear values I think Troy is talking about the renderer engin values wich haven't any 0 to 1 limit and Frank is talking about exported Exr linear values wich are 0 to 1 in floating point, am I right?
 

12 hours ago, Frank Jonen said:

I'd wish Affinity had a checkbox to turn off ICC profiles for 32-bit spaces and leave us in a pure OCIO workflow and a clearer information of how an imported EXR was 'converted' to scene linear. The ICC toggle would need like two confirmation dialogues the first time it is set. "Are you sure? This will look horrible on non-hardware calibrated devices." and "Yes, my body is ready."

Personally I think it'd be fantastic to have Serif as an ACES partner and the Affinity suite have an official "ACES mode" where everything just works and an option to screw it up manually.

I agree with that, even if I've read that ACEScg have problems with self reflecting colored materials (if I'm right)
 

12 hours ago, Frank Jonen said:

Gamma is still an issue in Affinity as you might have noticed the hard way. For some inexplicable reason the sliders end at 0 and 2. Maybe that is useful to someone out there. If anybody finds that necessary, fine. But Affinity really needs a sensible Gamma transform. The current one in the Levels layer is just terrible. It sounds like that adjustment layer might eventually be replaced by a good one though in the future from reading the roadmap.

Yes, completely nonsense here.
 

8 hours ago, troy_s said:

To start, disable any ICCs in your chain, as well as any power functions (Awful term "gamma"). Apply the same OCIO transforms on your image as you were doing in the other application.

 

Do you see 1:1? If so, does the exported display referred format appear 1:1?

I think it's the point here, I haven't found a way of disabling ICC at the export time. (I think it's when converting from 32bit to 8bit)
I've set my display transform to unmanaged, this way I see exactly what the stack is doing.
I've got EXR 32bit linear, followed by a OCIO layer linear to log, then OCIO log to sRGB but the export is washed out because of ICC export transformation, it's why I need to compensate with an S curve shape before the export.
(exported image without S curve)

 

AP_32bits_workflow_setup.jpg

AP_32bits_workflow_export.jpg

Share this post


Link to post
Share on other sites

That looks suspiciously like the sRGB OETF is being applied on top of the output stack.

 

Are those all of the places where ICCs are integrated in Affinity?

 

If so, alter the first one to be linear sRGB, as the transfer function / display referred encoding is being manually applied in this instance.

Share this post


Link to post
Share on other sites
2 hours ago, troy_s said:

Are those all of the places where ICCs are integrated in Affinity?

I'm not sure but I think yes. In fact I haven't found so much about ICC in affinity in the officials doc and tutorials.

I tried with linear sRGB but it's not a close 1/1 export compare to sRGB view.

 

3 hours ago, troy_s said:

That looks suspiciously like the sRGB OETF is being applied on top of the output stack.

You're right, the best I can have is by having an EOTF curve on top of my stack to counter the OETF's one from the ICC.
So now I can work with 32 bit linear exr, but it looks a bit like a hack^^.

So it answer my question, doesn't look like the 32 bit linear workflow is so well implemented here.

Share this post


Link to post
Share on other sites
On 17/05/2018 at 0:52 PM, Mareck said:

You're right, the best I can have is by having an EOTF curve on top of my stack to counter the OETF's one from the ICC.

So now I can work with 32 bit linear exr, but it looks a bit like a hack^^.

 

Can you unset the ICC setting? That OETF encode is very much coming from the ICC chain, as the entire chain would break if it were hard coded.

 

That implies that somewhere a display / monitor setting is set to something with the sRGB OETF specified in it somewhere. Finding which one it is will sort it.

 

Am I correct in estimating that when you set it to sRGB linear for both that it is very close, with a slight deviation in the lower values that is most noticeable?

Share this post


Link to post
Share on other sites

I haven't found a way to disable ICC at export time for the moment, I'll make a post about that.

I'm not sure to understand well, I've tried scene linear to linear sRGB, slight shift of the red channel (because my image is mostly red dominant) and looks like there is a compression of the values to 0/1 range.
And scene linear to linear sRGB then linear sRGB to output sRGB, no difference at all between scene linear to output sRGB.

What I don't understand is what's linear sRGB for? I mean in general use.

Share this post


Link to post
Share on other sites
4 hours ago, Mareck said:

I'm not sure to understand well, I've tried scene linear to linear sRGB, slight shift of the red channel (because my image is mostly red dominant) and looks like there is a compression of the values to 0/1 range.

There shouldn't be a compression. Possibly a clip.

Scene referred linear sRGB / 709 should be identical in terms of the buffer (exception being the footnote below) to a linear profile ICC with 709 / sRGB primaries. The problem is in the handling given ICCs are designed for display referred graphic design (relative normalized luminance) work and not photographic work in the scene referred domain.

Why the red channel would shift is beyond me unless you have an extremely intense and saturated red colour that was clipping somewhere along the dumping through the ICC chain.

The discrepancy between your physical display and the sRGB OETF is not tremendous. A typical display more than likely targets a pure power function in the DAC that ends up 2.2 - 2.4. This matches the power function portion of the sRGB OETF reasonably closely. The linear toe of the OETF is where larger differences will show up, which would be in the deeper shadow areas of your display referred output.

Quote

And scene linear to linear sRGB then linear sRGB to output sRGB, no difference at all between scene linear to output sRGB.

The point I make is that you don't want a conversion to sRGB's OETF at all. The primaries are identical, and the transfer function is already designed for 2.2 output. This means that you would want to simply generate the encoded values and "assign" the profile (aka tag not encode) to the encoded values, not "convert" (aka re-encode the data).

Viewing the OCIO transform chain under the linear sRGB profile should yield 1:1. If it doesn't, a secret sRGB transform is happening somewhere. Perhaps Affinity is rolling the final output through the operating system's chosen display ICC?

Quote

What I don't understand is what's linear sRGB for? I mean in general use.

That is the kludgy way you work in a linearized reference space with display referred applications. Your data starts off tagged as official sRGB, then would be compared against the reference, which would be linear sRGB. The primary lights are identical[1], so the transform to the reference space aligned buffer would undo the OETF, yielding display linear values.

On the way out, the reference would be identified as linear sRGB, which means linearized values using REC.709 primaries, and the output ICC (sRGB, Apple P3, etc.) would transform the primaries to the destination, and then lay the transfer function on top of that.

In this way, you essentially end up with a linearized reference for manipulation, albeit within the crippled up ICC model, subject to possible display referred / ICC assumptions.

[1] Under the ICC v2 chain, sadly the reference is enforced to be D50, which means your data isn't entirely subject to merely the transfer function, but also a chromatic adaptation transform to align the achromatic axis with D50. The opposite would happen on the way out.

Share this post


Link to post
Share on other sites

Hi Mareck, I'm attempting to replicate your workflow, but I noticed in your previous thread you were using the blender Filmic config—now it looks like you're using a Nuke config. Could I just clarify what you're actually trying to achieve? Do you want Affinity Photo to be the final destination before you export for web, I.E. are you just looking to beautify your renders? Or is the goal to perform some editing in a colour-managed environment and pass an exported EXR to another app for compositing?

Initially, I thought you were perhaps missing a step out of the workflow, which is to ensure that Photo converts from whatever colour space you were using in blender to scene linear. You would do this by appending the colour space to the filename—for example, "render Filmic Log.exr". However, it would seem blender simply exports in scene linear, except if you're compositing in the VSE where you can define the colour space (which I believe is irrelevant for your workflow). Apologies as I don't use blender very often, so I've been playing catchup here.

 

From what I can gather, what you want to be able to do is have your starting point in Photo look exactly like the preview/render in blender when using the Filmic view transform, is that correct? Normally, this is easily achievable by either:

a) Setting your view transform to OCIO managed on the 32-bit preview panel, then choosing the correct display output (sRGB) and view transform (Filmic)

or

b) Setting your view transform to unmanaged (linear light), then using an OCIO Adjustment layer to go from Linear to Filmic sRGB

Both of these options work fine (option b effectively "baking" in a transform), I've tested both and they match the preview in blender. They both bypass the typical display transform that's applied (you can see this by using the "ICC Display Transform" option on the 32-bit preview panel). This is how you would work on a 32-bit document in Photo before exporting it back to EXR for use in other software.

However, the complication arises when you want to export out of Photo to a 16 or 8-bit format—in this case, it has to include a non-linear transform using the document profile.

 

It's not pretty, but I believe I've found a solution that will allow you to match the preview in blender whilst using the ICC Display Transform—and from then on, your exports will look correct.

I've set up a similar workflow with one of my photogrammetry models and am using the default OCIO config that ships with blender (which is what I believe you were using originally). I've copied this config over to a separate folder in order to use it in Affinity Photo as well. Please note that I've only verified this with blender's standard OCIO configuration, which seems to include Filmic? I've tried piecing together this workflow by looking at the .ocio configuration file.

Try following these steps and see if they help:

  1. In blender's colour management options, set display device to sRGB, view to Default and look to None.
  2. Import your EXR document into Photo. First, go to Document>Assign ICC Profile and choose your monitor profile. For example, I profile my monitor and I'm using a custom profile called Dell-D50, so I would choose that. If you're using a standard profile, it's probably named after your display—for example, my Dell monitor's profile is named DELL UP3216Q. You might see a noticeable shift in colour. If you now A/B this against the blender preview set to sRGB with a Default view (not Filmic yet), you should find they match closely. Photo is colour managed and will correctly convert based on the document profile, but if we assign a display profile we're effectively bypassing that and presenting the numbers straight to the monitor.
  3. Next, add an OCIO Adjustment layer and go from Linear to Filmic sRGB.
  4. Now, either:
    1. Add a LUT adjustment. You'll want to find srgb.spi1d in the blender OCIO config LUTs folder and apply that.
    2. Or add another OCIO adjustment. Go from sRGB (not Filmic sRGB) to Linear. Normally a Linear to Filmic sRGB then Filmic sRGB to Linear transform would be identity—but we're instead saying that on the transform back, the source colour space is sRGB, causing the numbers to change.
  5. If you change blender's view to Filmic and the look to None or Base Contrast, you should find the result in Photo now matches blender.
  6. When you're finished editing, don't forget to flatten and convert the ICC profile to sRGB (alternatively, make sure you assign sRGB as the document profile whilst exporting).

 

Not pretty, but it appears to work.

Additionally, you may want to apply one of the looks—looks aren't exposed in Photo, but you can still apply them through the LUT adjustment. It requires altering the workflow slightly.

  1. Work back to when you applied the monitor profile.
  2. Add an OCIO Adjustment layer and this time go from Linear to Filmic Log (not Filmic sRGB).
  3. Now add a LUT adjustment and browse for the "filmic" folder in the OCIO configuration folder. Choose the corresponding LUT depending on which look you want (note that you can always find out which looks point to which LUTs by checking the .ocio configuration file):
    1. filmic_to_0-35_1-30.spi1d - Very Low Contrast
    2. filmic_to_0-48_1-09.spi1d - Low Contrast
    3. filmic_to_0-60_1-04.spi1d - Medium Low Contrast
    4. filmic_to_0-70_1-03.spi1d - Base Contrast (this will give you the same result as choosing "None" for the look)
    5. filmic_to_0-85_1-011.spi1d - Medium High Contrast
    6. filmic_to_0.99_1-0075.spi1d - High Contrast
    7. filmic_to_1.20_1-00.spi1d - Very High Contrast
  4. Finally, either add the LUT with srgb.spi1d or add another OCIO Adjustment layer and once again go from sRGB to Linear.

And that should be it. Sorry for the wall of text, but hopefully you'll be able to follow the instructions. I've attached a quick and dirty side-by-side of my photogrammetry render, I used the second method of applying a look (the very low contrast one). It's almost a 1:1 match on my screen.

ocio.thumb.jpg.4bfe0f199b575bf5cbea991fe5a510be.jpg

 

Hope that helps!


Affinity Photo Video Tutorials - Affinity Photo for iPad Tutorials

Looking for a manual/documentation? Check affinity.help for online help!

@JamesR_Affinity for tutorial sneak peeks and more

Share this post


Link to post
Share on other sites

@troy_s I'll post a screenshot for the red shift and I'll try what you said, and thanks for the precisions on linear sRGB.

@James Ritson thanks for the time you spend on that answer.
I try to set a workflow to magnify render for 8bits export but I'm definatly interessed in editing and export to an other app since I can be brought to make render for comecial video ad.

Yes I've change settings to try things and moving forward, my workflow to beautify renders is:
Import linear .exr, convert from linear to Log space with OCIO, then adding a creative LUT wich already convert from Log space to sRGB one, therefore I don't need an additional transform with ICC. I'm working in unmanaged preview since I only want the transforms I choose in my stack and the end of my stack is already in sRGB space with the LUT.

But I'll definitely try what you explained, there is a lot of usefull info, but it's a bit late here, I'll let you know how it went.

Share this post


Link to post
Share on other sites
5 hours ago, Mareck said:

Import linear .exr, convert from linear to Log space with OCIO, then adding a creative LUT wich already convert from Log space to sRGB one, therefore I don't need an additional transform with ICC. I'm working in unmanaged preview since I only want the transforms I choose in my stack and the end of my stack is already in sRGB space with the LUT.

But I'll definitely try what you explained, there is a lot of usefull info, but it's a bit late here, I'll let you know how it went.

Hi Mareck, all I can say here is that if you're looking to export from Photo to TIFF/JPEG or any other 8/16-bit format, don't work in Unmanaged. For the export to match what you're seeing on the canvas view, you must be using ICC Display Transform.

If you're simply editing in Photo and passing the EXR to another app (e.g. Nuke, Fusion, Natron, Flame, Resolve), then you can work with an OCIO Transform. Don't forget what I mentioned above about colour space conversion—Photo works in scene linear. If you don't specify a colour space on import, it will assume the document you're importing is already in scene linear. With blender exports, this is the case, but if you have an EXR in another format (e.g. aces) and you're using the appropriate configuration, you can append "aces" to the filename and Photo will convert from that to scene linear. When you're exporting, don't forget to append "aces" to the filename if you need to convert back from scene linear to aces.

 


Affinity Photo Video Tutorials - Affinity Photo for iPad Tutorials

Looking for a manual/documentation? Check affinity.help for online help!

@JamesR_Affinity for tutorial sneak peeks and more

Share this post


Link to post
Share on other sites
20 hours ago, James Ritson said:
  • Import your EXR document into Photo. First, go to Document>Assign ICC Profile and choose your monitor profile. For example, I profile my monitor and I'm using a custom profile called Dell-D50, so I would choose that. If you're using a standard profile, it's probably named after your display—for example, my Dell monitor's profile is named DELL UP3216Q. You might see a noticeable shift in colour. If you now A/B this against the blender preview set to sRGB with a Default view (not Filmic yet), you should find they match closely. Photo is colour managed and will correctly convert based on the document profile, but if we assign a display profile we're effectively bypassing that and presenting the numbers straight to the monitor.

Are you sure about this?

There are at least a few problems with this approach:

  • You will have effectively transformed pure sRGB / REC.709 primaries to the primaries of your display, essentially breaking the integrity of your data. Specifically, the data encoded in the file would be idealized REC.709 lights, and assigning a profile then rolling through an ICC chain would convert the primaries to those of your display.
  • Via the assignment, it would be identifying data as being display referred nonlinear where it is not. This would take the values between 0.0 to 1.0 of the EXR invert the transfer function when attempting to go to a linearized reference space, which @Mareck appears to be desiring. In the event that the reference were nonlinear in terms of transfer function, this would end up being a no-op, and create the illusion that it works.

Neither of which are desired in most pipelines.

The above two issues could be mitigated if @Mareck were wishing to work nonlinearly, by going to a nonlinear encoding to start with.

The problem is that Affinity appears to be performing an additional transform to the display, which may be obtained via the operating system. It would be good to hear back from a developer on this. Are there any Affinity developers out there that can confirm that there is a display transform happening in this chain that is not listed in the settings?

Share this post


Link to post
Share on other sites

@troy_s Like James said in the other thread, it wasn't planned for exporting to 8/16bit format, what surpise me but it explain why it don't works.
Do you think it's feasible to create a neutral ICC profil (no transfer curve and no color convertion)?
If so I could learn to make one and add it to affinity, that way I can choose it at export time and the problem is solved.

Share this post


Link to post
Share on other sites
On 5/23/2018 at 5:32 AM, Mareck said:

@troy_s Like James said in the other thread, it wasn't planned for exporting to 8/16bit format, what surpise me but it explain why it don't works.

It doesn't explain everything you are seeing.

Exporting to various display referred bit depths isn't something that needs to be designed to in the classical sense. The problem is that there appears to be another transform creeping into the chain. My hunch is that it is the OS level display ICC, but I would love confirmation.

On 5/23/2018 at 5:32 AM, Mareck said:

Do you think it's feasible to create a neutral ICC profil (no transfer curve and no color convertion)?
If so I could learn to make one and add it to affinity, that way I can choose it at export time and the problem is solved.

The colour science behind the ICC approach isn't that complex. At risk of gravely oversimplifying, the protocol typically goes as follows:

  1. "Undo" the display referred transfer function to get to linearized values for linear algebra, if required. Almost always display linear.
  2. Take data from primaries and adapt to D50 ICC reference, if the reference isn't ICC's D50.
  3. Convert via linear algebra to destination adapted colorimetry, using the D50 reference.
  4. Adapt back to the destination white if the white doesn't match ICC's D50.
  5. Apply the nonlinear transfer display linear to display referred nonlinear transfer function if required. Examples would be the sRGB OETF.

Assuming an image is encoded for standard D65 REC.709 primaries, and the transfer function is listed as an identity / no-op, aka "linear" values, (1) and (5) would result in a no-operation if using a half-decent linear sRGB / REC.709 profile. I say half decent because good gosh there are plenty of junk sRGB profiles out there. For the colour nerds out there, yes there would be a D50 to destination transform, which would be D65 in the case of REC.709 / sRGB, which means it most certainly is not a no-op in reality.

The problem is that ICC is protocol based, so we are always discussing a source ICC and a destination. On the way to encoding, there are sometimes three. Those might be typically almost invisible:

  1. Source is assumed or tagged "sRGB". This is the first ICC in question.
  2. Reference space ("working" space in Adobe terms) where the pixels are taken "to" for manipulation. This is the second ICC.
  3. Output or encoding transform. This could be a display or print etc. This is the third.

Affinity clearly lets you specify (2), but seems to obfuscate where (3) is coming from, or at least hide it. Your tests indicate that it is applying a transform there, which is all well and good except in the case of using OCIO, as the two systems are quite incompatible. The reasons for this begin with the lack of protocol metadata, that ICC protocol specifically requires to function, in an OCIO chain, but also covers into colour mangling with ICC's D50 requirements and other display referred considerations.

TL;DR: The linear sRGB reference space would almost, sort of, kind of work as you want, minus the D50 protocol problem.

Affinity would be wise to provide the display transform it is using from within the application, permit overriding with a selected ICC, and most importantly, allow one to turn off ICC management altogether if one chooses to use OCIO instead. Best of all worlds would be to provide clear handling at the tail end to embed / tag the display referred output with ICC information, as well as have the display referred output potentially colour managed for display via ICC protocol, versus OCIO.

If you were to hack the system by tagging the output as linear sRGB, that too would fail as you end up with the same problem. You would need to hack the display transform to linear sRGB (OMG Yuck) then tag the image as nonlinear standard sRGB. Really yuck.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...

Important Information

Please note the Annual Company Closure section in the Terms of Use. These are the Terms of Use you will be asked to agree to if you join the forum. | Privacy Policy | Guidelines | We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.