Is there a way to use blend modes to average hues or "push away" a hue from another hues?

[NullAllocationError]

This is mostly for the sake of lighting. Colored lighting looks nicer if represented with highlights tending towards the lighting's color and shadows contrasting that lighting's color; or, to put it another way:

• for the sake of highlights, I'd want to be able to change the hue to be the average (weighted average, if taking into account opacity) of the blended layer and the underlying one (so with resulting hue = (underlying layer's hue + opacity of blended layer * blended layer's hue) / (1 + opacity of blended layer) as the formula)
• for shadows/shading, I'd want the resulting hue to be like the hue of the blended layer was subtracted from that of the underlying one (so basically operating on the formula of resulting hue = underlying layer's hue - opacity of blended layer * blended layer's hue, wrapping around at the 0/360 point)

This seems like it should probably be an obvious application of blend modes, but I have so far not been able to figure it out, so I'd appreciate any help people can give. I'd sort of hoped the Hue blendmode would work for this, but it changes the luminance more than the actual hue.

[smadell]

I'm not following your math completely, but I think I get the gist of what you're after. I'll offer a very simple solution. I started with a portrait (taken from Unsplash) that seems to have a yellow-orange cast to the overall lighting. I created 2 additional layers above that: one that is completely yellow-orange, and a second that is its inverse. I set the Blend Mode of both these colored layers to Soft Light. Then, I changed the Blend Options (click the little gear icon in the Layers panel) to apply the yellow-orange layer only to the highlights, and the inverted color layer only to the shadows. I've included the results below.

Obviously, you can tweak the results by (i) changing the Blend Options curves to include more highlights, more shadows, etc.; (ii) changing the Opacity of either or both of the color layers.

Is this what you were after?

Additional thoughts...  You can use the Hue blend mode instead of the Soft Light blend mode for a more subtle result. There are fewer changes (really, no changes) to luminosity. However, the Blend Options settings and Opacity adjustment seem to be the most important factors.

Edited September 28, 2021 by smadell
Additional thoughts added