problem with large files

[toyotadesigner]

I'm working with relatively large files: starting at ~ 120 MegaPixel, TIFF format, 16/48 bit color depth, scans from large format film, scanned @ 4.000 ppi, file size starting at ~ 650 MB. Files are stored on external FireWire 800 hard drives (WD Studio). Some file sizes even exceed 1.4 GB...

 

Currently I'm still using PhotoLine, at 16/48 bit image editor. Opening an image with this editor takes around 10 seconds to display the image. Opening the same image (no other image editor running) takes almost a minute. That's a lot of time to read in the same image.

 

What might cause this slow down? Are the image files pre-processed before AP can display them?

 

Scaling down these images i.e. from 12.900 x 8.600 pixels to 1.500 x 1.000 pixels takes ages compared to my 'old app'.

 

After scaling down i.e. to 6.000 x 4.000 pixels and then trying to apply a white balance or any other effect again takes ages. AP will not gain speed after scaling down an image... seems it keeps the original in RAM until I will close AP. 

 

OK, I have to admit that I'm running a 24" iMac 8.1 (late 2008) with 6 GB RAM, but with my 'old app' the workflow is considerably faster.

 

I can not tell you anything about the behavior of PhotoShop, because I don't use it anymore (I did not like the cloud model). 

 

Simple tasks like i.e. retouching, adding a selection, etc. behave normal, that means without any speed lag.

 

Will this be optimized in the final release?

 

I guess that AP uses the same 'IOFireWireSerialBusProtocolTransport.kext' to access images from the external drives, so where is the bottleneck? What can I do to speed AP up (except investing into a new iMac)?

 

Any suggestions? Any ideas or insights are highly welcome ;-)

[Andy Somerfield]

Hi,

 

I'm not sure what's going on with the loading speed issue - when we test here, AP loads all files we have as fast if not faster than other applications.. If you can send us the file in question, I'll happily perform the same tests on it here.

 

The performance of downsampling images depends on the resampler used. Lanczos, especially non-separable, can take a long time - but the results are vastly superior to NN or Bilinear, in most situations.

 

AP will not free the RAM used in a resample operation - because the user can always choose to "Undo" the resample operation.

 

It's worth noting though, that the memory management in AP is quite sophisticated - it will aggressively use RAM until the limit specified in Preferences has been reached - then it will dynamically move history data to disk (in idle time, if possible). It might be worth checking what you have that RAM slider set to - it defaults to the physical amount of RAM in your machine, which is usually a good bet.

 

Thanks,

 

Andy.

[toyotadesigner]

Hi Andy, thanks for the fast reply.

 

I don't have any other comparison than my 'old app'. 

 

Submitting a sample file is almost impossible with my limited bandwidth line.

 

Downsampling: Yup, I just made a comparison - the Lanczos non-separable algorithm definitely is superior compared to others, so I will accept the longer processing time.

 

The trick with the 'Undo' makes sense, so I will downsample the images, save/export them, close the original and open the down sampled image for further processing.

 

My RAM slider is set to 6.144 (on my 6 GB RAM iMac). Undo limit to 1.024, View Quality Bilinear (Best Quality), Use precise clipping enabled.

 

The internal HD has 400 GB free - should be sufficient for any image editor.

 

I know that my iMac is not the fastest, but as long as I can process my images in a reasonable time I will keep it.

[Andy Somerfield]

Hi,

 

Thanks for the reply. Although we pride ourselves on performance, quality is always "king".. If we can perform an operation "better but slower", we will nearly always take that path - our "denoise" is a good example. Many editors provide a "fast but average quality" denoise, but for most users, the result is more important than the time taken!

 

I could implement a little button to "dump history" - which would avoid you having to do the Save / Load dance to preserve memory, although, I am concerned that you encounter memory problems in any way! Your machine is easily capable of what you describe, so I will investigate the issue further..

 

Thanks,

 

Andy.

[toyotadesigner]

Hi Andy,

 

If we can perform an operation "better but slower", we will nearly always take that path - our "denoise" is a good example.

 

Excellent statement and philosophy! As I said - as long as the result will improve, I will wait a tad longer.

 

Today I used AP for more than 2 hours, loading different huge and medium size images and working with them. Over the time the loading accelerated, and I ended up with 30 seconds for a 650 MB file. Working with some TIFF files from a 24 MP digital camera was a breeze. Downscaling was fast, adding filters as well. 

 

I am not a coder, but I know that Mac OS X always had a problem to generate icons if the dimensions of an image exceeded a certain size. In finder the thumbnails from 24 MP files are generated relatively fast. The thumbnails for my huge files take ages... As I mentioned in another thread here, Graphic Converter doesn't even display thumbnails from my huge images. Nada. Or LightZone (ex LightCrafts, now free) doesn't display thumbnails from my huge files either. It appears to me that there might be a size barrier <sigh>: if an image is larger than x/y, Mac OS X needs a long breath to handle the files.

 

Never mind, this problem seems to be only at my end, all other thousands or millions of users don't face this problem. So don't waste too much time for an extra button to dump history or investigating the problem at this very moment. Time will tell where the 'pointer is bent' ;)

[toyotadesigner]

Update.

 

After playing a bit with AP last night, I discovered a nice workflow for my huge images: Load them, retouch and correct them if necessary, and then just export them with a specific size for web presentations, resample with Lanczos 3 non-separable, quality 95%. Result: awesome quality, no downscaling of the original, no extra sharpening. 

 

It's just a matter of thinking different ;)

[MEB]

Yes. The addition of Lanczos for resampling surprised me too when i first opened the program last year. Glad we have it there!

The difference is considerable.

[toyotadesigner]

Another night to 'play' with AP. Downscaling a 130 MP TIFF from a film scan with the above settings is absolutely great. However, downscaling an image from a digital camera (24 PM) is not as effective in comparison to the film based image. 

 

But I encountered a strange behavior. After retouching around 200 dirt spots on the image (scan from a film), AP started to need a long time when I tried to open a menu on top. Then I just saved the image (TIFF), closed AP, restarted AP (to release and recover memory), and opened the image. It was pure black! Shock in the night... What is weird: I could see the image in XnView MP, open it with XnView MP and with PL. Saved the image again, closed PL and opened it in AP - everything was there.

 

So it seems that AP made a mistake in saving the huge TIFF. I still had 2 GB RAM free, so I really don't know what happened.

[toyotadesigner]

Sorry if I open up this topic again, but it is definitely necessary.

 

Last night I installed the latest WD +Turbo drivers (WD_1394_64_109HPDriver) on my system (Mavericks). Writing and especially reading files was considerably faster with the +Turbo drivers. Peak reading was ~100MB/s, peak writing ~80MB/s. Let's say that the average speed meets the FireWire tech specs of 60MB/s it would mean that a file of 648 MB should need no more than 11 seconds to be loaded. But - chirp, chirp - it took 40 seconds to be displayed in AP. As a non-coder I'd say: seems the file needs to be converted before it is displayed - at least the time needed could suggest this idea.

 

And yes, AP was already up and running before I clicked to load the file, and no other image had been opened with AP.

 

Now it's getting scary: AP is still running, no image loaded. Started Affinity Designer. Pulled up Finder, right click the same file, open with AD. And guess what happened: the image is displayed in 16 seconds! What a difference!

 

So there must be something weird going on in Affinity Photo that it needs 24 seconds more to load and display the same image.

 

Last check: Saved the file as a *.afphoto file on the same external FW drive. Loaded it with AP - 40 seconds. Loaded it with AD - 15 seconds.

 

Question: could you take a deep look into this inexplicable behavior? 

 

Because I can not submit a 648 MB file, you can blow up a 48 bit file to a size of 12.900 x 8.600 pixel, save it, and then open it. Any TIFF with 16/48 bit depth will do the trick.

 

BTW, the TIFFs are either with ProPhoto or ECI V2 icc profile, I did not select to convert the icc when loading a file.

[toyotadesigner]

Forgot to mention my RAM data: