Converting 2D photos into 3D images (RGB-D input)

[Kiarian]

I received an email from NVIDIA promoting a new technique for converting a 2D photo into a 3D image. Article quotes:

'To convert a single RGB-D input image into a 3D photo, a team of researchers from Virginia Tech and Facebook developed a deep learning-based image inpainting model that can synthesize color and depth structures in regions occluded in the original view.
"Classic image-based reconstruction and rendering techniques require elaborate capture setups involving many images with large baselines, and/or special hardware,” the researchers stated in their paper, 3D Photography using Context-aware Layered Depth Inpainting. “In this work, we present a new learning-based method that generates a 3D photo from an RGB-D input. The depth can either come from dual-camera cell phone stereo or be estimated from a single RGB image."
'To show the potential of the project, a separate researcher from Google has taken the code and developed a Chrome extension that turns every Instagram post into 3D images. Developers interested in setting up the extension can follow the tutorial to set up and run the project using NVIDIA GPUs on the Google Cloud Platform'

Is this something Affinity Photo might have on the horizon? This isn't simply cutting away chucks of an image to then move in a parallax motion, the technique actually looks to give objects 3D properties.

Also see YouTube link using this technique on old historic photos. The results are amazing. The picture comes alive!

Full link as follows:

https://news.developer.nvidia.com/automatically-convert-your-photos-into-3d-images-with-ai/?mkt_tok=eyJpIjoiWmpaa1l6ZGxPREExTXpsaiIsInQiOiJ1MzBYaDJIbjBIK0JlWHptajFZdCtDWTRQUU81UHVsU0Rhb3JNZTkxMHlCaXZlZnkxTzAzWWF6cDdQRXQ2UnRObVVEMXFZU0pzbVdpckdhXC85MlVOTVIrUmJnVnNBUnlvT0p5UGZ6VklkMHFXcCs0aVI0SVZDSzFOdkdRQWlzeUsifQ%3D%3D

[wonderings]

That is some very cool stuff. Nothing useful for me but the effect is amazing. I know Photoshop has 3D capabilities, I toyed with some 3D models a while back. Does Photos have any 3D capabilities at all?

[Kiarian]

On 4/24/2020 at 3:22 PM, wonderings said:

That is some very cool stuff. Nothing useful for me but the effect is amazing. I know Photoshop has 3D capabilities, I toyed with some 3D models a while back. Does Photos have any 3D capabilities at all?

Not that I know of. Definitely something for the future though. The technology isn't too far away though, couple of years I reckon. In the future we'll be able to look behind photos!

[Andy05]

I truly appreciated that they kept those examples, which showed some flaws (like, Queen's diamonds on her dress or Einstein's hair "stuck to the chalk board).

But the results look amazing nonetheless.

[R C-R]

On 4/24/2020 at 8:40 AM, Kiarian said:

Is this something Affinity Photo might have on the horizon? This isn't simply cutting away chucks of an image to then move in a parallax motion, the technique actually looks to give objects 3D properties.

If you mean the "deep learning-based image inpainting model" it probably isn't something we will see in AP any time soon unless some third party develops a dynamic library or framework that includes what that model has learned from being trained with a very large sample of images & permits Serif to use it for free or at a very low cost.

[Kiarian]

On 4/28/2020 at 9:17 PM, R C-R said:

If you mean the "deep learning-based image inpainting model" it probably isn't something we will see in AP any time soon unless some third party develops a dynamic library or framework that includes what that model has learned from being trained with a very large sample of images & permits Serif to use it for free or at a very low cost.

Well, took a few years. 2023, and I think the AI revolution is here. Here is what Apple are doing with using FaceLit technology. Simple 2D images. I don't think it will be long until we see something like this in Affinity Photo and Photoshop.

FaceLit: Neural 3D Relightable Faces - Apple Machine Learning Research