Jump to content

slow load/unload time for files into batch dialogue queue


Recommended Posts

I'm experimenting with file export right now, and I was unimpressed with how long it took to load a thousand .ARW files into the batch dialogue queue, and the fact that the CPU was at ~1% and disk (NVMe) was at 0.4% the whole time, which makes me think there's an inefficiency in code somewhere?

So I figured I'd be a good beta tester and I did a few tests on the batch dialogue input file queue and found a couple interesting things. I tested with 1000 images in .ARW and .DNG (converted from the same .ARW)

To clarify, by "batch dialogue queue" I mean opening the "New Batch Job" dialogue from within Affinity Photo via File/New Batch Job/ and selecting the [Add] button to add files to the processing queue.

1. It took 2.25 minutes to load 1000 .ARW into the batch dialogue queue.

2 It took 15 SECONDS to load DNGs into the dialogue queue (9x faster than ARW)

3 It took 1.2 minutes to REMOVE the .ARW OR the .DNG from the batch dialogue queue.

4. Most of the times (I think all but the first) The dialogue queue disappeared because focus shifted to the main affinity photo window and the batch dialogue became hidden behind it.

Platform was Windows 10/Threadripper 3960x with 256GB RAM and NVMe drive.

Andy

Link to post
Share on other sites

Out of interest what is the total file size of the 1000 .ARW files and the 1000 .DNG files?

 

Due to the fact that Boris Johnson is now our Prime Minister, punctuation, spelling and grammar will never be worried about ever again.  We now have far bigger problems to be worried about.

Link to post
Share on other sites
  • Moderators

Hey riveryeti,

I will reply to your  post whilst pretending not to be jealous of your machine :P 

I think you are definitely onto something. Ideally, we should be using whatever resources are available to us. I am somewhat surprised that the CPU and disk usage was so low. I will ask a developer to comment on this.

I would be interested to see what the average machine would do in this case. I'm sure I've seen my machine take longer to do a tenth of the files you queued up. 

Link to post
Share on other sites
  • Moderators

So I've just tried it with 100 raw files and it took 15 seconds on a much slower machine. So if I was doing 1000 I imagine it would take roughly the same amount of time, maybe a bit longer on my slower machine.

I did notice a spike in CPU though. I'm not sure if your machine should be doing it much quicker but this is what I'll try and find out.

Oh, I checked on my 2017 iMac and 100 raw files loaded instantly. As soon as I clicked ok in Finder, they were in the Sources queue.

Link to post
Share on other sites
  • Moderators

@riveryeti - I've spoken with dev and we think we may have to get our hands on one of these threadrippers. Of course I've requested it for myself ;) 🙏. However due to lockdown and everyone working from home, I don't know how soon this would happen. It might take some time if I'm being honest with you.

I really appreciate you flagging this up though. 

Link to post
Share on other sites
10 hours ago, carl123 said:

Out of interest what is the total file size of the 1000 .ARW files and the 1000 .DNG files?

 

The 1000 .ARW files total 34.7GB and the 1000 DNGs total 36.2GB. I generated the DNGs with Adobe DNG converter and the "no thumbnails" option.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Please note there is currently a delay in replying to some post. See pinned thread in the Questions forum. These are the Terms of Use you will be asked to agree to if you join the forum. | Privacy Policy | Guidelines | We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.